File size: 2,999 Bytes
3ee0cab da58283 3ee0cab a809dcd 6999ba7 da58283 3ee0cab da58283 3ee0cab da58283 3ee0cab da58283 3ee0cab 4227e09 3ee0cab f127367 3ee0cab a809dcd da58283 3ee0cab da58283 3ee0cab da58283 3ee0cab 4227e09 a809dcd 4227e09 a809dcd da58283 3ee0cab f127367 a809dcd da58283 3ee0cab da58283 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 | # Benchmarks (Teutonic-1-Chat-Preview) — **corrected** 2026-09-01
Evaluated with harness `teutonic-eval-v1.2.0` **after** the thinking-budget / chat-template fix.
**Protocol:** chat · thinking **on** · seed `20260830` · MATH-500 max_new=16k · HE+/MBPP+ max_new=8k · lm-eval `enable_thinking` + strip `</think>` · Qwen uses native `apply_chat_template`.
**Honesty:** Prior HF Qwen column (MMLU-Pro 21 / MATH 20 / MBPP 8) was an **invalid harness artifact** — do not cite. Details: training-repo `reports/bench-harness-bug.md`.
**Residual risk:** Teutonic MATH-500 still had **7.6%** `finish=length` @16k; a generous re-check @64k (`20260901T133412Z`) scored **88.0** (vs **88.6**) with **7.2%** still truncated mid-think — no material uplift, scores below kept. Fair GPQA/LCB @32k still truncates heavily (GPQA Teu 52% / Qwen 43%; LCB Teu 55% / Qwen 68%) — absolute STEM/code scores remain conservative vs vendor cards.
Head-to-head vs `Qwen/Qwen3.5-9B` on the **same fixed harness**. Vendor **card** = Qwen’s published stack (not identical protocol).
## Same-harness deltas (Teutonic − Qwen3.5-9B) — corrected
| Benchmark | Teutonic | Qwen3.5-9B (fixed) | Δ |
|---|---:|---:|---:|
| MMLU-Pro (n=2000) | 69.8 | **73.2** | −3.4 |
| MMLU-Redux (generative, n=5330) | 87.0 | **91.3** | −4.3 |
| MATH-500 | **88.6** | 68.2 | **+20.4** |
| HumanEval+ | **82.9** | 81.1 | **+1.8** |
| MBPP+ | **75.1** | 64.6 | **+10.5** |
| IFEval (prompt strict) | **68.6** | 65.4 | **+3.2** |
| GPQA-Diamond (fair boxed) | 46.0 | **55.1** | −9.1 |
| LiveCodeBench (v5+v6) | 26.9 | 26.9 | 0.0 |
| RULER (4k–64k) | **89.5** | 88.4 | **+1.1** |
| Needle | 99.5 | **100.0** | −0.5 |
### Invalid prior Qwen column (do not use)
| Benchmark | Broken harness | Fixed |
|---|---:|---:|
| MMLU-Pro | 21.3 | **73.2** |
| MATH-500 | 20.2 | **68.2** |
| HumanEval+ | 27.4 | **81.1** |
| MBPP+ | 8.2 | **64.6** |
| IFEval | 26.3 | **65.4** |
## Knowledge & STEM
| Benchmark | Teutonic | Qwen3.5-9B (fixed harness) | Qwen3.5-9B (card) |
|---|---:|---:|---:|
| MMLU-Pro | 69.8 | **73.2** | 82.5 |
| MMLU-Redux | 87.0‡ | **91.3**‡ | 91.1 |
| GPQA-Diamond | 46.0† | **55.1**† | 81.7 |
‡ lm-eval `mmlu_redux_generative` full (n=5330), chat + thinking ON + think-strip.
† Fair generative boxed-letter (thinking ON, 32k, think-strip, fixed option order; n=198). Not card-comparable (no official shuffle). Truncation still high (~43–52% length) — absolute likely underestimated.
## Math & code
| Benchmark | Teutonic | Qwen3.5-9B (fixed) |
|---|---:|---:|
| MATH-500 | **88.6** | 68.2 |
| HumanEval+ | **82.9** | 81.1 |
| MBPP+ | **75.1** | 64.6 |
| LiveCodeBench (v5+v6) | 26.9 | 26.9 |
## Long context
| Benchmark | Teutonic | Qwen3.5-9B |
|---|---:|---:|
| RULER (4k–64k) | **89.5** | 88.4 |
| Needle | 99.5 | **100.0** |
## Vision
Teutonic is **text-only** — VL benchmarks are `N/A`. Qwen3.5-9B card (not comparable): MMMU 78.4 · MMMU-Pro 70.1 · MathVista 85.7.
|