Align Ekko Tiny benchmark scores

README.md

@@ -43,7 +43,7 @@ model-index:
       split: test
     metrics:
     - type: wer
-      value: 24.15
+      value: 23.88
   - task:
       type: automatic-speech-recognition
     dataset:
@@ -53,7 +53,7 @@ model-index:
       split: test
     metrics:
     - type: wer
-      value: 11.87
+      value: 11.74
   - task:
       type: automatic-speech-recognition
     dataset:
@@ -63,7 +63,7 @@ model-index:
       split: test
     metrics:
     - type: wer
-      value: 9.75
+      value: 9.36
   - task:
       type: automatic-speech-recognition
     dataset:
@@ -72,7 +72,7 @@ model-index:
       split: test_balanced
     metrics:
     - type: wer
-      value: 7.62
+      value: 7.57
 ---
 
 ![Ekko v1 Tiny — offline Danish speech recognition](cover.png)
@@ -95,7 +95,7 @@ evaluated as a separate test set.
 
 ## Benchmark
 
-**12.37% mean WER** across 26,780 rows from five Danish test sets, evaluated
+**12.14% mean WER** across 26,780 rows from five Danish test sets, evaluated
 with the leaderboard harness. Scores use the published defaults: NFKC
 normalization, spoken-number folding, Danish number-word normalization,
 lowercase, punctuation removal, symmetric filler removal, and whitespace
@@ -103,36 +103,38 @@ collapse. Lower is better.
 
 | Test set | Speech type | WER |
 |---|---|---:|
-| CoRal conversation | Spontaneous conversation | 24.15% |
-| CoRal read aloud | Read speech | 11.87% |
-| Common Voice Danish (CV 25.0 test subset) | Crowd-sourced read speech | 8.44% |
-| FLEURS `da_dk` | Read news-style speech | 9.75% |
-| FTSpeech | Parliamentary speech | 7.62% |
-| **Equal-weight mean** | | **12.37%** |
+| CoRal conversation | Spontaneous conversation | 23.88% |
+| CoRal read aloud | Read speech | 11.74% |
+| Common Voice Danish (CV 25.0 test subset) | Crowd-sourced read speech | 8.15% |
+| FLEURS `da_dk` | Read news-style speech | 9.36% |
+| FTSpeech | Parliamentary speech | 7.57% |
+| **Equal-weight mean** | | **12.14%** |
 
 The NeMo checkpoint processed the benchmark at **795.8x realtime** on one A100.
-The official run saved raw per-sample outputs, and an independent offline
-rescore reproduced 12.37% exactly.
+The official run saved raw per-sample outputs. Rescoring those same outputs with
+the leaderboard normalization updated on 15 October 2026 gives 12.14%; no
+model inference was rerun.
 
-### Comparison with models below 800M parameters
+### Comparison snapshot (9 October 2026)
 
 ![Mean WER of Ekko v1 Tiny and all twelve open-weight leaderboard models below 800M parameters](comparison.png)
 
 The chart compares Tiny's Q5_0 release score with 12 open-weight models below
 800M parameters in the [Danish ASR Leaderboard](https://huggingface.co/datasets/RyeAI/danish-asr-leaderboard/tree/f7c930a30ac2f88e68421d88fc577c0cbafa8c31/results)
-as of 9 October 2026. Each score is an equal-weight mean WER across the same
-five Danish test sets (26,780 clips). Tiny scores 12.37%, 0.27 percentage
-points lower than Svale 600M at 12.64%. Rankings may change when new models
-or reruns are added.
+as of 9 October 2026, before the normalization update. Each score is an
+equal-weight mean WER across the same five Danish test sets (26,780 clips).
+In this historical snapshot, Tiny scored 12.37% and Svale 600M scored 12.64%.
+For current scores, use the live leaderboard.
 
 ### Native GGUF quantization
 
 The F32, F16, Q8_0, Q6_K, Q5_K, Q5_0, Q4_K, and Q4_0 files are
-scored on the same current 26,780-row matrix and normalization policy as the
-NeMo result above. All 40 variant/domain outputs completed and passed exact-row
-offline scoring.
+scored on the same 26,780 clips under the earlier normalization policy. Their
+WER figures below have not yet been rescored with the 15 October 2026 update;
+they should not be compared directly with the current NeMo score. All 40
+variant/domain outputs completed and passed exact-row offline scoring.
 
-| Variant | Size | Mean WER | CPU speed |
+| Variant | Size | Mean WER (earlier scoring) | CPU speed |
 |---|---:|---:|---:|
 | **F16** | 248.2 MiB | **12.315%** | 74.4x |
 | F32 | 430.6 MiB | 12.318% | 71.8x |
@@ -152,9 +154,10 @@ reference tier.
 
 The model is available for sherpa-onnx and was evaluated on the same 26,780-row
 matrix with mandatory non-silent peak normalization to 0.95. The browser
-downloads the compact int8 export.
+downloads the compact int8 export. The ONNX WER figures below also use the
+earlier normalization policy and have not yet been rescored.
 
-| Export | Size | Mean WER | CPU speed |
+| Export | Size | Mean WER (earlier scoring) | CPU speed |
 |---|---:|---:|---:|
 | FP32 ONNX | 450.6 MiB | 12.465% | 53.4x |
 | **INT8 ONNX** | **129.0 MiB** | **12.946%** | **49.2x** |
@@ -172,12 +175,15 @@ Clean-cache packaged download and inference tests passed on macOS arm64 and
 Linux x64 with native word timestamps. The Windows x64 artifact is mapped but
 has not been exercised against this release.
 
-| Format | Path | Measured mean WER |
+| Format | Path | Mean WER |
 |---|---|---:|
-| NeMo | `ekko-v1-tiny.nemo` | 12.37% |
-| parakeet.cpp GGUF | `gguf-parakeet.cpp/ekko-v1-tiny-{f32,f16,q8_0,q6_k,q5_k,q5_0,q4_k,q4_0}.gguf` | 12.315-12.467% |
-| sherpa-onnx FP32 | `onnx-sherpa/{encoder,decoder,joiner}.onnx` | 12.465% |
-| sherpa-onnx INT8 | `onnx-sherpa/{encoder,decoder,joiner}.int8.onnx` | 12.946% |
+| NeMo | `ekko-v1-tiny.nemo` | 12.14% (current) |
+| parakeet.cpp GGUF | `gguf-parakeet.cpp/ekko-v1-tiny-{f32,f16,q8_0,q6_k,q5_k,q5_0,q4_k,q4_0}.gguf` | 12.315-12.467% (earlier) |
+| sherpa-onnx FP32 | `onnx-sherpa/{encoder,decoder,joiner}.onnx` | 12.465% (earlier) |
+| sherpa-onnx INT8 | `onnx-sherpa/{encoder,decoder,joiner}.int8.onnx` | 12.946% (earlier) |
+
+The GGUF and ONNX scores above use the earlier normalization policy; the NeMo
+score uses the current leaderboard policy.
 
 The browser loads `onnx-sherpa/runtime-config.json` with the INT8 files and
 checks its SHA-256 before using its audio and model settings.
@@ -219,7 +225,7 @@ pseudo-labelled audio.
 - The 44-token vocabulary does not contain uppercase or punctuation tokens.
 - Punctuation and capitalization require a separate restoration model such as
   `RyeAI/ekko-pnc`.
-- Spontaneous conversation remains the hardest benchmark domain (24.15% WER).
+- Spontaneous conversation remains the hardest benchmark domain (23.88% WER).
 - Noisy audio and overlapping speakers were not evaluated separately.
 
 ## License