ironbcc commited on
Commit
5ea1da2
·
verified ·
1 Parent(s): 9b1dd3b

add README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -17
README.md CHANGED
@@ -22,7 +22,7 @@ picking the next action for a robot: same call, same model.
22
 
23
  - **Quality.** Mean accuracy 0.801 across 17 public suites (Seb-9B 0.793, hosted Jev 1.13 0.786, Laya 0.694). Biggest gains over v1:
24
  BANKING77 0.973, jailbreak detection 0.940, model routing 0.980. In our 3D robot-arm demos all 300 decisions matched the safety
25
- rules. Against general LLMs on 420 public decision rows it is level with gpt-6-luna (high) and Claude Haiku (medium) and behind Claude Sonnet and Opus (table below). One step back: v1 is still better calibrated (ECE 0.088 vs 0.110).
26
  - **Speed.** About 22 ms per decision on one RTX PRO 6000 (p95 36 ms) and about 88 ms on an Apple M5 Max (p95 209 ms), up to about
27
  87 decisions per second on a single GPU.
28
  - **Decision Index.** 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. We are ahead of
@@ -73,22 +73,10 @@ picking the next action for a robot: same call, same model.
73
 
74
  ## Against general LLMs
75
 
76
- Same 420 public decision rows (20 from each of 21 public suites, fixed seed), same prompt content (state, question, options), each model answering blind;
77
- gold is the suite's own label. The LLMs return an answer; Seb-9B-v2 also returns a probability for every option.
78
-
79
- | | Correct | Accuracy | 95% interval | vs Seb-9B-v2 |
80
- |---|---|---|---|---|
81
- | Claude Opus | 359 / 420 | 0.855 | 0.82 - 0.89 | ahead (p = 0.0003) |
82
- | Claude Sonnet (default effort) | 349 / 420 | 0.831 | 0.79 - 0.86 | ahead by 3 points (p = 0.054) |
83
- | Claude Sonnet (low effort) | 343 / 420 | 0.817 | 0.78 - 0.85 | ahead by 2 points (p = 0.26) |
84
- | **Seb-9B-v2** | 335 / 420 | **0.798** | 0.76 - 0.83 | |
85
- | Claude Haiku (medium effort) | 334 / 420 | 0.795 | 0.75 - 0.83 | level (p = 1.0) |
86
- | gpt-6-luna (high) | 327 / 420 | 0.779 | 0.74 - 0.82 | level (p = 0.29) |
87
-
88
- Claude models are Claude Code's `opus`, `sonnet` and `haiku` aliases on 2026-10-22 (one Haiku answer was missing and counts as wrong); gpt-6-luna ran through our
89
- JSON-schema labeling harness (`scripts/label_with_codex.py`, high effort). p-values are exact sign tests on the rows where exactly one of the two was right.
90
- Seb-9B-v2 is level with Claude Haiku (medium) and gpt-6-luna and statistically level with Sonnet at low effort; Sonnet at default effort is 3 points ahead and Opus 6 points
91
- ahead (significant). The general models are much larger and need a hosted API call per decision; Seb-9B-v2 runs locally. Rows and scripts: `opus_eval/` in the demo folder.
92
 
93
  ## Demo results
94
 
 
22
 
23
  - **Quality.** Mean accuracy 0.801 across 17 public suites (Seb-9B 0.793, hosted Jev 1.13 0.786, Laya 0.694). Biggest gains over v1:
24
  BANKING77 0.973, jailbreak detection 0.940, model routing 0.980. In our 3D robot-arm demos all 300 decisions matched the safety
25
+ rules. Against general LLMs on 420 public decision rows it is level with gpt-6-luna (high) and Claude Haiku (medium) and behind Claude Opus and, within noise, Claude Sonnet at low effort (table below). One step back: v1 is still better calibrated (ECE 0.088 vs 0.110).
26
  - **Speed.** About 22 ms per decision on one RTX PRO 6000 (p95 36 ms) and about 88 ms on an Apple M5 Max (p95 209 ms), up to about
27
  87 decisions per second on a single GPU.
28
  - **Decision Index.** 59.16 on the 0.3 public index, against 57.96 for hosted Jev 1.13 and 60.70 for Jev 1.14. We are ahead of
 
73
 
74
  ## Against general LLMs
75
 
76
+ <img src="https://huggingface.co/ironbcc/Seb-9B-v2/resolve/main/assets/llm-table.png" alt="Accuracy on 420 public decision rows: Claude Opus 0.855, Claude Sonnet low effort 0.817, Seb-9B-v2 0.798, Claude Haiku medium effort 0.795, gpt-6-luna high 0.779" width="100%">
77
+
78
+ Seb-9B-v2 is level with Claude Haiku (medium) and gpt-6-luna, and statistically level with Claude Sonnet at low effort; Opus is clearly ahead. The general models are much
79
+ larger and need a hosted API call per decision; Seb-9B-v2 runs locally and also returns a probability for every option. Rows and scripts: `opus_eval/` in the demo folder.
 
 
 
 
 
 
 
 
 
 
 
 
80
 
81
  ## Demo results
82