# Benchmarks (Teutonic-1-Chat-Preview) — **corrected** 2026-09-01

Evaluated with harness `teutonic-eval-v1.2.0` **after** the thinking-budget / chat-template fix.  
**Protocol:** chat · thinking **on** · seed `20260830` · MATH-500 max_new=16k · HE+/MBPP+ max_new=8k · lm-eval `enable_thinking` + strip `</think>` · Qwen uses native `apply_chat_template`.  
**Honesty:** Prior HF Qwen column (MMLU-Pro 21 / MATH 20 / MBPP 8) was an **invalid harness artifact** — do not cite. Details: training-repo `reports/bench-harness-bug.md`.  
**Residual risk:** Teutonic MATH-500 still had **7.6%** `finish=length` @16k; a generous re-check @64k (`20260901T133412Z`) scored **88.0** (vs **88.6**) with **7.2%** still truncated mid-think — no material uplift, scores below kept. Fair GPQA/LCB @32k still truncates heavily (GPQA Teu 52% / Qwen 43%; LCB Teu 55% / Qwen 68%) — absolute STEM/code scores remain conservative vs vendor cards.

Head-to-head vs `Qwen/Qwen3.5-9B` on the **same fixed harness**. Vendor **card** = Qwen’s published stack (not identical protocol).

## Same-harness deltas (Teutonic − Qwen3.5-9B) — corrected

| Benchmark | Teutonic | Qwen3.5-9B (fixed) | Δ |
|---|---:|---:|---:|
| MMLU-Pro (n=2000) | 69.8 | **73.2** | −3.4 |
| MMLU-Redux (generative, n=5330) | 87.0 | **91.3** | −4.3 |
| MATH-500 | **88.6** | 68.2 | **+20.4** |
| HumanEval+ | **82.9** | 81.1 | **+1.8** |
| MBPP+ | **75.1** | 64.6 | **+10.5** |
| IFEval (prompt strict) | **68.6** | 65.4 | **+3.2** |
| GPQA-Diamond (fair boxed) | 46.0 | **55.1** | −9.1 |
| LiveCodeBench (v5+v6) | 26.9 | 26.9 | 0.0 |
| RULER (4k–64k) | **89.5** | 88.4 | **+1.1** |
| Needle | 99.5 | **100.0** | −0.5 |

### Invalid prior Qwen column (do not use)

| Benchmark | Broken harness | Fixed |
|---|---:|---:|
| MMLU-Pro | 21.3 | **73.2** |
| MATH-500 | 20.2 | **68.2** |
| HumanEval+ | 27.4 | **81.1** |
| MBPP+ | 8.2 | **64.6** |
| IFEval | 26.3 | **65.4** |

## Knowledge & STEM

| Benchmark | Teutonic | Qwen3.5-9B (fixed harness) | Qwen3.5-9B (card) |
|---|---:|---:|---:|
| MMLU-Pro | 69.8 | **73.2** | 82.5 |
| MMLU-Redux | 87.0‡ | **91.3**‡ | 91.1 |
| GPQA-Diamond | 46.0† | **55.1**† | 81.7 |

‡ lm-eval `mmlu_redux_generative` full (n=5330), chat + thinking ON + think-strip.  
† Fair generative boxed-letter (thinking ON, 32k, think-strip, fixed option order; n=198). Not card-comparable (no official shuffle). Truncation still high (~43–52% length) — absolute likely underestimated.

## Math & code

| Benchmark | Teutonic | Qwen3.5-9B (fixed) |
|---|---:|---:|
| MATH-500 | **88.6** | 68.2 |
| HumanEval+ | **82.9** | 81.1 |
| MBPP+ | **75.1** | 64.6 |
| LiveCodeBench (v5+v6) | 26.9 | 26.9 |

## Long context

| Benchmark | Teutonic | Qwen3.5-9B |
|---|---:|---:|
| RULER (4k–64k) | **89.5** | 88.4 |
| Needle | 99.5 | **100.0** |

## Vision

Teutonic is **text-only** — VL benchmarks are `N/A`. Qwen3.5-9B card (not comparable): MMMU 78.4 · MMMU-Pro 70.1 · MathVista 85.7.