mbCrypto commited on
Commit
409d175
·
verified ·
1 Parent(s): a1b693b

Refresh accuracy tables on the current portable build (094b53c): 4/6 languages identical, corpus +0.30; add MLS-FR anchor row and hotword biasing note

Browse files
Files changed (1) hide show
  1. README.md +33 -19
README.md CHANGED
@@ -41,28 +41,42 @@ through the Q6_K copy. That is the only numerical change, and it also removes
41
 
42
  ## Accuracy
43
 
44
- FLEURS test clips, 24 per language, greedy decoding, scored corpus-level with digits
45
- and years spelled out on both sides (methodology and scripts:
 
46
  [bench/README.md](https://github.com/martinobettucci/VibeASR-bitnet.cpp/blob/claude/asr-cpu-optimization-cztnh9/bench/README.md)).
47
- Both models measured on the same deterministic runtime build:
48
 
49
- | Language | Released (1.70 GB) | **Slim (1.23 GB)** | Δ |
 
 
 
 
 
 
50
  |:--|--:|--:|--:|
51
- | Spanish | 5.96 | 7.67 | +1.70 |
52
- | English | 6.48 | 6.48 | 0.00 |
53
- | Portuguese | 9.23 | 8.90 | −0.33 |
54
- | Italian | 9.67 | 9.52 | −0.16 |
55
- | German | 13.41 | 15.51 | +2.09 |
56
- | French | 34.63 | 30.46 | −4.17 |
57
- | **corpus** | **13.95** | **13.65** | **−0.30** |
58
-
59
- Corpus-level the slim model scores marginally *better*; per-language differences
60
- swing both ways by up to ±2–4 points on ~400-word samples, which is sampling noise,
61
- not a head-precision effect. Practical read: **no measurable accuracy cost** for the
62
- 27% size reduction.
63
-
64
- The runtime is deterministic: identical transcripts at any thread count (this was
65
- not true of the original runtime; see the repo for the fix).
 
 
 
 
 
 
 
 
66
 
67
  ## Languages
68
 
 
41
 
42
  ## Accuracy
43
 
44
+ FLEURS test clips (24 per language) plus a Multilingual LibriSpeech French anchor,
45
+ greedy decoding, scored corpus-level with digits and years spelled out on both sides
46
+ (methodology and scripts:
47
  [bench/README.md](https://github.com/martinobettucci/VibeASR-bitnet.cpp/blob/claude/asr-cpu-optimization-cztnh9/bench/README.md)).
 
48
 
49
+ Both models measured with one runtime build (branch commit `094b53c`, portable
50
+ AVX2+VNNI, deterministic — identical transcripts at any thread count). Transcripts
51
+ are bit-reproducible *within* a build; absolute WER can shift a point or two between
52
+ builds as float codegen changes, which is why both columns always come from the same
53
+ binary.
54
+
55
+ | Corpus / language | Released (1.70 GB) | **Slim (1.23 GB)** | Δ |
56
  |:--|--:|--:|--:|
57
+ | FLEURS Spanish | 5.96 | 5.96 | 0.00 |
58
+ | FLEURS English | 6.48 | 6.83 | +0.35 |
59
+ | FLEURS Portuguese | 9.23 | 9.23 | 0.00 |
60
+ | FLEURS Italian | 9.67 | 9.67 | 0.00 |
61
+ | FLEURS German | 13.41 | 13.24 | −0.17 |
62
+ | FLEURS French | 34.63 | 36.02 | +1.39 |
63
+ | **FLEURS corpus** | **13.95** | **14.25** | **+0.30** |
64
+ | MLS French (read speech) | 21.97 | 22.31 | +0.34 |
65
+
66
+ Four of six FLEURS languages produce **identical transcripts** under both models;
67
+ German is marginally better, and the corpus delta (+0.30) comes almost entirely from
68
+ French, the model's weakest language, where ±1–2 points is within sampling noise on
69
+ ~700 reference words. Practical read: **no measurable accuracy cost** for the 27%
70
+ size reduction.
71
+
72
+ The MLS row doubles as the register check: French drops from ~35% (FLEURS,
73
+ encyclopedic) to ~22% (audiobooks) for both models — the FLEURS gap is corpus
74
+ difficulty, not a defect of either quantisation.
75
+
76
+ The runtime also ships opt-in decoder-level hotword biasing (`--hotwords`,
77
+ token-trie logit boosting): on FLEURS-French with oracle terms it recovers ~4 WER
78
+ points at `--hotword-boost 5`, with no degradation when fed wrong terms. Details and
79
+ the four-condition table in the repo README.
80
 
81
  ## Languages
82