ironbcc commited on
Commit
6922a5c
·
verified ·
1 Parent(s): c01ef79

add README.md

Browse files
Files changed (1) hide show
  1. README.md +29 -1
README.md CHANGED
@@ -29,7 +29,7 @@ picking the next action for a robot: same call, same model.
29
  87 decisions per second on a single GPU.
30
  - **Decision Index.** 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. We are ahead of
31
  Jev 1.13 on 22 of 37 benchmarks and in 4 of 5 areas (knowledge and reasoning is where Jev is clearly better). Newer open entries
32
- on the live board score higher, for example OneJev1.6-9B at 65.3.
33
 
34
  **3D robot arm: sorting with a person entering the workspace (PyBullet)**
35
 
@@ -102,6 +102,33 @@ Decision Index 0.3 public index, per area (chance-corrected, higher is better).
102
 
103
  The published numbers are lower for Seb-9B-v2 even though the H200 is the stronger GPU, but the hardware and the request mix differ, so read it as "lower on each card's own numbers", not "faster". Default context: Seb-9B-v2 32k, clef models 16k.
104
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
105
  ## Against general LLMs
106
 
107
  <img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/llm-table.png" alt="Accuracy on 420 public decision rows: Claude Opus 0.855, Claude Sonnet low effort 0.817, Seb-9B-v2 0.798, Claude Haiku medium effort 0.795, gpt-6-luna high 0.779" width="100%">
@@ -216,3 +243,4 @@ print(dict(zip(keys, p)))
216
  8-bit copy: `python -m mlx_lm convert --hf-path ironbcc/Seb-9B-v2 --mlx-path seb-9b-v2-q8 -q --q-bits 8`.
217
 
218
  Decision aid, not advice. Apache-2.0, like the base model.
 
 
29
  87 decisions per second on a single GPU.
30
  - **Decision Index.** 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. We are ahead of
31
  Jev 1.13 on 22 of 37 benchmarks and in 4 of 5 areas (knowledge and reasoning is where Jev is clearly better). Newer open entries
32
+ on the live board score higher, for example OneJev1.6-9B at 65.3. Size for size it is ahead of vLLM Semantic Router's d3-flash (9B, 57.79 public suite).
33
 
34
  **3D robot arm: sorting with a person entering the workspace (PyBullet)**
35
 
 
102
 
103
  The published numbers are lower for Seb-9B-v2 even though the H200 is the stronger GPU, but the hardware and the request mix differ, so read it as "lower on each card's own numbers", not "faster". Default context: Seb-9B-v2 32k, clef models 16k.
104
 
105
+ ## Against the d3 family (vLLM Semantic Router)
106
+
107
+ Decision Index public suite (chance-corrected, higher is better). At every size where both families have a model, the Seb model is ahead.
108
+
109
+ | Size | d3 model | d3 public suite | d3 official board | Seb model | **Seb public suite** | Seb lead |
110
+ |---|---|---:|---:|---|---:|---:|
111
+ | 0.8B | [d3-edge](https://huggingface.co/vllm-sr/d3-edge) | 28.82 | 20.2 | | | |
112
+ | 0.8B | [d3-lite](https://huggingface.co/vllm-sr/d3-lite) | 36.93 | 32.3 | | | |
113
+ | 2B | [d3-nano](https://huggingface.co/vllm-sr/d3-nano) | 42.22 | 38.9 | Seb-2B-v2 | **44.48** | **+2.26** |
114
+ | 4B | [d3-mini](https://huggingface.co/vllm-sr/d3-mini) | 54.90 | 52.6 | Seb-4B-v2 | **55.34** | +0.44 |
115
+ | 9B | [d3-flash](https://huggingface.co/vllm-sr/d3-flash) | 57.79 | 55.8 | **Seb-9B-v2** | **59.16** | **+1.37** |
116
+ | 27B | [d3](https://huggingface.co/vllm-sr/d3) | 65.45 | 66.3 | | | |
117
+ | 26B | [d3-turbo](https://huggingface.co/vllm-sr/d3-turbo) | 71.7 (its estimate) | none (estimate 70.2) | | | |
118
+ | 26B | [d3-ultra](https://huggingface.co/vllm-sr/d3-ultra) | 71.57 | 70.18 | | | |
119
+
120
+ The larger d3 models (27B and 26B) are ahead of Seb-9B-v2 by 6 to 12 points.
121
+
122
+ **Decision benchmarks, same harness for every model** (self-run):
123
+
124
+ | Benchmark | **Seb-9B-v2** | Seb-2B-v2 | d3-nano (2B) |
125
+ |---|---:|---:|---:|
126
+ | decisionbench.ai v4, with images | **88.42%** (947 / 1,071) | 87.30% (935) | 85.06% (911) |
127
+ | decisionbench.ai v4, text only | **87.02%** (932 / 1,071) | 86.27% (924) | 83.85% (898) |
128
+ | DecisionBench (Hanno Labs) v1, all 23,900 rows | **78.66** | 75.97 | 73.82 |
129
+
130
+ The d3 Decision Index numbers are from each model's card, fetched 2026-11-04 (0.3.2 kit, 140,178 public requests), except d3-ultra (live board, 2026-10-26) and d3-turbo (the card's own estimate). The Seb numbers are full self-run measurements with the 0.3 kit (110,520 public requests). The request lists differ, so treat the small 4B lead as a tie. No Seb model has been submitted to the official board, which adds private tests. On DecisionBench, Seb-9B-v2 was served with up to 255 candidates per request and answered every row; Seb-2B-v2 and d3-nano were served with up to 160, and the 561 larger rows count as wrong for both. We have not run the other d3 models on decisionbench.ai or DecisionBench.
131
+
132
  ## Against general LLMs
133
 
134
  <img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/llm-table.png" alt="Accuracy on 420 public decision rows: Claude Opus 0.855, Claude Sonnet low effort 0.817, Seb-9B-v2 0.798, Claude Haiku medium effort 0.795, gpt-6-luna high 0.779" width="100%">
 
243
  8-bit copy: `python -m mlx_lm convert --hf-path ironbcc/Seb-9B-v2 --mlx-path seb-9b-v2-q8 -q --q-bits 8`.
244
 
245
  Decision aid, not advice. Apache-2.0, like the base model.
246
+