Instructions to use vllm-sr/vela3-flash with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use vllm-sr/vela3-flash with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("zero-shot-classification", model="vllm-sr/vela3-flash", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("vllm-sr/vela3-flash", trust_remote_code=True) model = AutoModel.from_pretrained("vllm-sr/vela3-flash", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Keep each thread's adapter gate to itself
Browse filesThe adapter gate in `vela3_adapters.py` was one object for the whole process. `active()` set its adapter name and row mask, and every `MultiLoRA.forward` read them, so when two threads called `system_one` on the same model, one thread's passes ran with the other's adapter. On vela3-edge on CPU, with one thread asking span questions in a loop and another asking the same refund question 25 times, all 25 answers came back wrong: 0.736 to 0.820 against 0.956 when asked alone.
The gate is now a `threading.local`, so each thread sees only the adapter it switched on, and a new thread starts with every adapter off. Nothing else changes: `active()`, `MultiLoRA.forward` and the graph check in `modeling_vela3.py` read the gate as before, now on the calling thread.
Tested on vela3-edge on CPU. In the same two-thread run, 0 of 25 answers now differ from the answer asked alone. One request each over three states, with the nine router questions, a hazard question, an ordinary choice and yes/no question and three span questions (PII, relation, entities), gives byte-identical JSON before and after. The change is the same in all six sizes.
- SHA256SUMS +1 -1
- vela3_adapters.py +2 -1
|
@@ -16,7 +16,7 @@ cc710bdeb492c51564600ff09b645bce2b716d1fff76e810249b0926dd2e49fe ner_head.safet
|
|
| 16 |
d2c5f2f446cf12d2f8a24dbf634452477088523b0973e1e07df244c851421efb span_decode_v2.py
|
| 17 |
e5e74fe176bfd580f86448a152c7a8e9d278645baaf155e5f2f0ee3b766c8a61 span_head.safetensors
|
| 18 |
d1c684f81bd9f4d9d2c749120162f26482a9e05984ca8865652a5ec3c18a109d span_lora.safetensors
|
| 19 |
-
|
| 20 |
9c14b2d82a95f33eb2c094f301a2f33450205b67e5cdd3cf6082e1fd827847ae vela3_config.json
|
| 21 |
b7fcefdd1a0b3e8ac22ac12338c72e76925688a989b5af6c70a104fa70122a39 vela3_labeldef.py
|
| 22 |
97d2448765a7ee5411dff44c601c3a073fdad9a9921f9818bac4135916081c20 vela3_span.py
|
|
|
|
| 16 |
d2c5f2f446cf12d2f8a24dbf634452477088523b0973e1e07df244c851421efb span_decode_v2.py
|
| 17 |
e5e74fe176bfd580f86448a152c7a8e9d278645baaf155e5f2f0ee3b766c8a61 span_head.safetensors
|
| 18 |
d1c684f81bd9f4d9d2c749120162f26482a9e05984ca8865652a5ec3c18a109d span_lora.safetensors
|
| 19 |
+
2e88d7af591e3097a735ecbbb17fa6433aaa5b803e4b810cab6231e62ce3a2ee vela3_adapters.py
|
| 20 |
9c14b2d82a95f33eb2c094f301a2f33450205b67e5cdd3cf6082e1fd827847ae vela3_config.json
|
| 21 |
b7fcefdd1a0b3e8ac22ac12338c72e76925688a989b5af6c70a104fa70122a39 vela3_labeldef.py
|
| 22 |
97d2448765a7ee5411dff44c601c3a073fdad9a9921f9818bac4135916081c20 vela3_span.py
|
|
@@ -11,12 +11,13 @@ from __future__ import annotations
|
|
| 11 |
|
| 12 |
import contextlib
|
| 13 |
import json
|
|
|
|
| 14 |
|
| 15 |
import torch
|
| 16 |
from torch import nn
|
| 17 |
|
| 18 |
|
| 19 |
-
class _Gate:
|
| 20 |
name = None # active adapter name, or None (all adapters off)
|
| 21 |
rows = None # None = every row of the batch; else a bool tensor [B]
|
| 22 |
|
|
|
|
| 11 |
|
| 12 |
import contextlib
|
| 13 |
import json
|
| 14 |
+
import threading
|
| 15 |
|
| 16 |
import torch
|
| 17 |
from torch import nn
|
| 18 |
|
| 19 |
|
| 20 |
+
class _Gate(threading.local): # per thread: concurrent requests never see each other's adapter
|
| 21 |
name = None # active adapter name, or None (all adapters off)
|
| 22 |
rows = None # None = every row of the batch; else a bool tensor [B]
|
| 23 |
|