| --- |
| license: apache-2.0 |
| base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct |
| base_model_relation: finetune |
| library_name: transformers |
| pipeline_tag: any-to-any |
| language: |
| - multilingual |
| tags: |
| - multimodal |
| - text |
| - audio |
| - image |
| - video |
| - decision-model |
| - choice |
| - qwen3-omni |
| - lora |
| - custom-code |
| - safetensors |
| --- |
| |
| # SPEV |
|
|
| SPEV is a 30B parameter omnimodal decision model that evaluates questions over text, |
| audio, images and video. It scores the supplied choices in one forward pass and |
| returns an answer with a probability for each choice. |
|
|
| ## Model |
|
|
| - **Backbone:** Qwen3-Omni-30B-A3B Thinker, with a trained LoRA adapter. |
| - **Decision head:** a shared scorer over contextual representations of the supplied choices. |
| - **Output:** per-choice probabilities and the highest-probability answer for each question. |
|
|
| ## Quickstart |
|
|
| The easiest way to get started is with the hosted API on Sprag (free through October 2026). |
| The full API docs are available [here](https://sprag.ai/docs/decisions). |
|
|
| ## Usage |
|
|
| Install the dependencies listed in `requirements.txt`, then load the model and processor: |
|
|
| ```python |
| import torch |
| from transformers import AutoModel, AutoProcessor |
| |
| model_id = "SpragAI/SPEV" |
| processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True) |
| model = AutoModel.from_pretrained( |
| model_id, |
| trust_remote_code=True, |
| dtype=torch.bfloat16, |
| device_map="auto", |
| attn_implementation="sdpa", |
| ) |
| |
| questions = { |
| "decision": { |
| "instructions": ( |
| "At the end of this conversation audio, has the most recent " |
| "speaker yielded the floor or are they continuing their turn?" |
| ), |
| "criteria": { |
| "keep_listening": ( |
| "The most recent speaker is continuing their turn; keep listening." |
| ), |
| "respond": ( |
| "The most recent speaker has yielded the floor; " |
| "another speaker can respond." |
| ), |
| }, |
| } |
| } |
| inputs = processor(audio="conversation.wav", state={}, questions=questions) |
| inputs = inputs.to(model.device, dtype=torch.bfloat16) |
| with torch.inference_mode(): |
| outputs = model(**inputs) |
| result = processor.post_process(outputs, questions=questions) |
| |
| print(result["answers"]["decision"]["choice"]) |
| print(result["answers"]["decision"]["probabilities"]) |
| ``` |
|
|
| ### Inputs |
|
|
| | Argument | Description | |
| |---|---| |
| | `audio` | Audio file to evaluate | |
| | `image` | Image file | |
| | `video` | Video file; visual frames | |
| | `state` | JSON context for the decision | |
| | `questions` | Mapping of question IDs to instructions and named criteria | |
|
|
| Media arguments are optional and can be combined. For text-only questions, supply |
| text through `state`. The processor resamples audio to the expected 16 kHz; callers |
| do not supply a sample-rate argument. Supply a video's soundtrack separately as |
| `audio` when it is part of the evidence. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| |---|---| |
| | `model-*.safetensors`, `model.safetensors.index.json` | Backbone, adapter and decision-head weights | |
| | `config.json`, `configuration_spev.py` | Model architecture and scoring configuration | |
| | `modeling_spev.py` | Transformers model implementation | |
| | `processing_spev.py`, `decision_utils.py` | Media preparation, question rendering and answer formatting | |
| | Tokenizer, processor and chat-template files | Text, audio, image and video processing | |
| | `examples/turn_policy.json` | Evaluated turn-taking question and application threshold | |
| | `requirements.txt` | Inference dependencies | |
| | `LICENSE` | Apache-2.0 license | |
|
|
| ## Results |
|
|
| | Input | SPEV p50 / p95 | OpenAI p50 / p95 | |
| |---|---:|---:| |
| | 5-second audio | 166 / 191 ms | 752 / 1331 ms | |
| | 20-second audio | 216 / 270 ms | 997 / 1326 ms | |
| | 512 × 512 image | 183 / 283 ms | 194 / 334 ms | |
| | 1024 × 1024 image | 322 / 642 ms | 373 / 763 ms | |
|
|
| Warm public-HTTPS latency including RTT, measured from the same client with concurrency one. SPEV ran on an RTX PRO6000 with native vLLM pooling. Each row has 30 measured pairs over six fixtures after three warmup pairs. OpenAI used `gpt-6-luna`; audio includes a fresh `gpt-4o-mini-transcribe` request before Decisions. These measurements cover latency, not accuracy. |
|
|
| ## License |
|
|
| Apache-2.0. See [LICENSE](LICENSE) and [third-party notices](THIRD_PARTY_NOTICES.md). |
|
|
