mbCrypto commited on
Commit
6822117
·
verified ·
1 Parent(s): 409d175

Accuracy in the open: four-engine WER table incl. whisper; speed as engine ratios

Browse files
Files changed (1) hide show
  1. README.md +45 -23
README.md CHANGED
@@ -41,16 +41,15 @@ through the Q6_K copy. That is the only numerical change, and it also removes
41
 
42
  ## Accuracy
43
 
44
- FLEURS test clips (24 per language) plus a Multilingual LibriSpeech French anchor,
45
- greedy decoding, scored corpus-level with digits and years spelled out on both sides
46
- (methodology and scripts:
47
- [bench/README.md](https://github.com/martinobettucci/VibeASR-bitnet.cpp/blob/claude/asr-cpu-optimization-cztnh9/bench/README.md)).
48
 
49
- Both models measured with one runtime build (branch commit `094b53c`, portable
50
- AVX2+VNNI, deterministic — identical transcripts at any thread count). Transcripts
51
- are bit-reproducible *within* a build; absolute WER can shift a point or two between
52
- builds as float codegen changes, which is why both columns always come from the same
53
- binary.
 
 
54
 
55
  | Corpus / language | Released (1.70 GB) | **Slim (1.23 GB)** | Δ |
56
  |:--|--:|--:|--:|
@@ -63,15 +62,35 @@ binary.
63
  | **FLEURS corpus** | **13.95** | **14.25** | **+0.30** |
64
  | MLS French (read speech) | 21.97 | 22.31 | +0.34 |
65
 
66
- Four of six FLEURS languages produce **identical transcripts** under both models;
67
- German is marginally better, and the corpus delta (+0.30) comes almost entirely from
68
- French, the model's weakest language, where ±1–2 points is within sampling noise on
69
- ~700 reference words. Practical read: **no measurable accuracy cost** for the 27%
70
- size reduction.
71
-
72
- The MLS row doubles as the register check: French drops from ~35% (FLEURS,
73
- encyclopedic) to ~22% (audiobooks) for both models — the FLEURS gap is corpus
74
- difficulty, not a defect of either quantisation.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
75
 
76
  The runtime also ships opt-in decoder-level hotword biasing (`--hotwords`,
77
  token-trie logit boosting): on FLEURS-French with oracle terms it recovers ~4 WER
@@ -88,11 +107,14 @@ languages degrade sharply; this repack does not change language coverage.
88
 
89
  ## Speed
90
 
91
- Documented in the runtime repo, not here — it is a property of the code, and the
92
- code moved a lot: ~2.8× end to end versus the upstream runtime on a 4-core AVX-512
93
- VM (compute RTF ≈ 0.37 for an 8 s clip), via VNNI kernels, a register-tiled INT8
94
- GEMM, vectorised quantisation epilogues and a layout-native depthwise convolution.
95
- Tables, profiler methodology, and full reproduction scripts:
 
 
 
96
 
97
  ➡️ **[martinobettucci/VibeASR-bitnet.cpp](https://github.com/martinobettucci/VibeASR-bitnet.cpp/tree/claude/asr-cpu-optimization-cztnh9)** — "CPU optimisation on AVX-512"
98
 
 
41
 
42
  ## Accuracy
43
 
44
+ Two measurements, two questions.
 
 
 
45
 
46
+ **1. Does the slim repack cost accuracy?** FLEURS test clips (24 per language)
47
+ plus a Multilingual LibriSpeech French anchor, greedy decoding, scored
48
+ corpus-level with digits and years spelled out on both sides (methodology and
49
+ scripts:
50
+ [bench/README.md](https://github.com/martinobettucci/VibeASR-bitnet.cpp/blob/claude/asr-cpu-optimization-cztnh9/bench/README.md)).
51
+ Both models on one runtime build (`094b53c`, portable AVX2+VNNI, deterministic —
52
+ identical transcripts at any thread count):
53
 
54
  | Corpus / language | Released (1.70 GB) | **Slim (1.23 GB)** | Δ |
55
  |:--|--:|--:|--:|
 
62
  | **FLEURS corpus** | **13.95** | **14.25** | **+0.30** |
63
  | MLS French (read speech) | 21.97 | 22.31 | +0.34 |
64
 
65
+ Four of six languages produce **identical transcripts**; the corpus delta comes
66
+ almost entirely from French, the model's weakest language, where ±1–2 points is
67
+ sampling noise on ~700 reference words. Practical read: **no measurable accuracy
68
+ cost** for the 27% size reduction. The MLS row is the register check: French drops
69
+ from ~35% (encyclopedic FLEURS) to ~22% (audiobooks) for both models — corpus
70
+ difficulty, not quantisation damage.
71
+
72
+ **2. Where does the model sit in the landscape?** A second, independent suite —
73
+ 100 clips drawn across the six languages plus the MLS-French anchor, all engines
74
+ back-to-back on one host at 4 threads, later runtime build, whisper given each
75
+ clip's language code (generous: this model runs unhinted):
76
+
77
+ | Engine / weights | de | en | es | fr | fr-MLS | it | pt | **all** |
78
+ |:--|--:|--:|--:|--:|--:|--:|--:|--:|
79
+ | Upstream engine, original 1.70 GB | 16.7 | 6.6 | 5.8 | 32.7 | 21.6 | 8.3 | 10.8 | **15.75** |
80
+ | **This repo (slim), fork engine** | 13.9 | 7.2 | 5.8 | 35.4 | 21.2 | 10.2 | 10.5 | **16.03** |
81
+ | whisper.cpp small q5_1 | 8.5 | 4.6 | 7.1 | 12.9 | 16.7 | 6.6 | 8.0 | **9.92** |
82
+ | whisper.cpp large-v3-turbo q5_0 | 4.1 | 3.7 | 4.0 | 3.3 | 10.3 | 3.3 | 4.3 | **5.12** |
83
+
84
+ Read it straight: slim-vs-original is again parity (+0.28, per-language scatter in
85
+ both directions — German improves under the fork's overflow-fixed kernels, Italian
86
+ regresses), while whisper models beat this architecture on accuracy at this corpus
87
+ register. What this stack offers instead is speed shape and features: compute
88
+ proportional to clip length where whisper always encodes a fixed 30 s window (this
89
+ engine is 1.76× *faster* than whisper-small below 8 s of audio and 3.9× faster
90
+ than turbo overall), native segment/speaker JSON, and decoder-level hotword
91
+ biasing. Speed ratios and their methodology live in the
92
+ [runtime repo](https://github.com/martinobettucci/VibeASR-bitnet.cpp/tree/claude/asr-cpu-optimization-cztnh9)
93
+ and are not duplicated here.
94
 
95
  The runtime also ships opt-in decoder-level hotword biasing (`--hotwords`,
96
  token-trie logit boosting): on FLEURS-French with oracle terms it recovers ~4 WER
 
107
 
108
  ## Speed
109
 
110
+ Documented in the runtime repo, not here — it is a property of the code, and
111
+ absolute RTF is a property of the host, so speed is published as **ratios**
112
+ between engines measured back-to-back: **2.56× faster than the upstream runtime**
113
+ (paired per-clip median, 100 clips, identical weights class), parity with
114
+ whisper-small on long clips and 1.76× faster below 8 s. Achieved via VNNI kernels
115
+ with a packed-B INT8 GEMM (fused dequantisation epilogue), conv-as-GEMM
116
+ downsampling, and a layout-native depthwise convolution. Tables, profiler
117
+ methodology, and full reproduction scripts:
118
 
119
  ➡️ **[martinobettucci/VibeASR-bitnet.cpp](https://github.com/martinobettucci/VibeASR-bitnet.cpp/tree/claude/asr-cpu-optimization-cztnh9)** — "CPU optimisation on AVX-512"
120