Text Classification
Transformers
Safetensors
qwen3_5
image-text-to-text
decision-model
typed-decisions
classification
routing
calibrated-probabilities
Instructions to use ironbcc/Seb-9B-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ironbcc/Seb-9B-v2 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="ironbcc/Seb-9B-v2")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("ironbcc/Seb-9B-v2") model = AutoModelForMultimodalLM.from_pretrained("ironbcc/Seb-9B-v2", device_map="auto") - Notebooks
- Google Colab
- Kaggle
add README.md
Browse files
README.md
CHANGED
|
@@ -22,7 +22,7 @@ picking the next action for a robot: same call, same model.
|
|
| 22 |
|
| 23 |
- **Quality.** Mean accuracy 0.801 across 17 public suites (Seb-9B 0.793, hosted Jev 1.13 0.786, Laya 0.694). Biggest gains over v1:
|
| 24 |
BANKING77 0.973, jailbreak detection 0.940, model routing 0.980. In our 3D robot-arm demos all 300 decisions matched the safety
|
| 25 |
-
rules. Against general LLMs on 420 public decision rows it is level with gpt-6-luna (high) and Claude Haiku (medium) and behind Claude
|
| 26 |
- **Speed.** About 22 ms per decision on one RTX PRO 6000 (p95 36 ms) and about 88 ms on an Apple M5 Max (p95 209 ms), up to about
|
| 27 |
87 decisions per second on a single GPU.
|
| 28 |
- **Decision Index.** 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. We are ahead of
|
|
@@ -73,22 +73,10 @@ picking the next action for a robot: same call, same model.
|
|
| 73 |
|
| 74 |
## Against general LLMs
|
| 75 |
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|---|---|---|---|---|
|
| 81 |
-
| Claude Opus | 359 / 420 | 0.855 | 0.82 - 0.89 | ahead (p = 0.0003) |
|
| 82 |
-
| Claude Sonnet (default effort) | 349 / 420 | 0.831 | 0.79 - 0.86 | ahead by 3 points (p = 0.054) |
|
| 83 |
-
| Claude Sonnet (low effort) | 343 / 420 | 0.817 | 0.78 - 0.85 | ahead by 2 points (p = 0.26) |
|
| 84 |
-
| **Seb-9B-v2** | 335 / 420 | **0.798** | 0.76 - 0.83 | |
|
| 85 |
-
| Claude Haiku (medium effort) | 334 / 420 | 0.795 | 0.75 - 0.83 | level (p = 1.0) |
|
| 86 |
-
| gpt-6-luna (high) | 327 / 420 | 0.779 | 0.74 - 0.82 | level (p = 0.29) |
|
| 87 |
-
|
| 88 |
-
Claude models are Claude Code's `opus`, `sonnet` and `haiku` aliases on 2026-10-22 (one Haiku answer was missing and counts as wrong); gpt-6-luna ran through our
|
| 89 |
-
JSON-schema labeling harness (`scripts/label_with_codex.py`, high effort). p-values are exact sign tests on the rows where exactly one of the two was right.
|
| 90 |
-
Seb-9B-v2 is level with Claude Haiku (medium) and gpt-6-luna and statistically level with Sonnet at low effort; Sonnet at default effort is 3 points ahead and Opus 6 points
|
| 91 |
-
ahead (significant). The general models are much larger and need a hosted API call per decision; Seb-9B-v2 runs locally. Rows and scripts: `opus_eval/` in the demo folder.
|
| 92 |
|
| 93 |
## Demo results
|
| 94 |
|
|
|
|
| 22 |
|
| 23 |
- **Quality.** Mean accuracy 0.801 across 17 public suites (Seb-9B 0.793, hosted Jev 1.13 0.786, Laya 0.694). Biggest gains over v1:
|
| 24 |
BANKING77 0.973, jailbreak detection 0.940, model routing 0.980. In our 3D robot-arm demos all 300 decisions matched the safety
|
| 25 |
+
rules. Against general LLMs on 420 public decision rows it is level with gpt-6-luna (high) and Claude Haiku (medium) and behind Claude Opus and, within noise, Claude Sonnet at low effort (table below). One step back: v1 is still better calibrated (ECE 0.088 vs 0.110).
|
| 26 |
- **Speed.** About 22 ms per decision on one RTX PRO 6000 (p95 36 ms) and about 88 ms on an Apple M5 Max (p95 209 ms), up to about
|
| 27 |
87 decisions per second on a single GPU.
|
| 28 |
- **Decision Index.** 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. We are ahead of
|
|
|
|
| 73 |
|
| 74 |
## Against general LLMs
|
| 75 |
|
| 76 |
+
<img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/llm-table.png" alt="Accuracy on 420 public decision rows: Claude Opus 0.855, Claude Sonnet low effort 0.817, Seb-9B-v2 0.798, Claude Haiku medium effort 0.795, gpt-6-luna high 0.779" width="100%">
|
| 77 |
+
|
| 78 |
+
Seb-9B-v2 is level with Claude Haiku (medium) and gpt-6-luna, and statistically level with Claude Sonnet at low effort; Opus is clearly ahead. The general models are much
|
| 79 |
+
larger and need a hosted API call per decision; Seb-9B-v2 runs locally and also returns a probability for every option. Rows and scripts: `opus_eval/` in the demo folder.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
## Demo results
|
| 82 |
|