mbCrypto commited on
Commit
3821925
Β·
verified Β·
1 Parent(s): 0e5b175

Correction: long-form is this model's strength, not its weakness (WER 8.88 vs whisper-small 27.57)

Browse files
Files changed (1) hide show
  1. README.md +36 -22
README.md CHANGED
@@ -85,28 +85,42 @@ of the benchmark host, not of the model); > 1 is faster than upstream.
85
  | whisper.cpp small q5_1 | 8.5 | 4.6 | 7.1 | 12.9 | 16.7 | 6.6 | 8.0 | **9.92** | **3.06Γ—** |
86
  | whisper.cpp large-v3-turbo q5_0 | 4.1 | 3.7 | 4.0 | 3.3 | 10.3 | 3.3 | 4.3 | **5.12** | **0.75Γ—** |
87
 
88
- Read it straight. Slim-vs-original is parity (+0.32, per-language scatter both
89
- ways β€” German improves under the fork's overflow-fixed kernels, Spanish
90
- regresses). But **on this corpus whisper-small is both more accurate and ~1.3Γ—
91
- faster** than this model, and turbo is far more accurate again.
92
-
93
- The obvious rebuttal is that 5–25 s clips are whisper's ideal case and this
94
- architecture's worst β€” it never exercises single-pass long-form or the
95
- speaker/segment output the runtime can request. **That rebuttal was tested and it
96
- failed.** On a 100 s two-speaker recording (full table in the runtime repo): the
97
- model transcribes correctly from the first word, then **stops early β€” 140 of 266
98
- reference words**, end token emitted with the token budget 98% unused; it emits
99
- **no speaker labels** despite the JSON format existing; and asking for that JSON
100
- makes the transcription worse (WER 55.7 vs 51.1). whisper-large-v3-turbo gets all
101
- 288 words at WER 8.6 on the same file.
102
-
103
- Honest guidance: **choose whisper for transcription accuracy at any clip length.**
104
- Choose this stack if you specifically need a 1.2 GB ternary CPU model with
105
- decoder-level hotword biasing (measured: FLEURS-French 36.0 β†’ 31.9 with domain
106
- terms), or you are working on the architecture itself. The runtime around it is
107
- fast, deterministic and portable β€” that part holds up; see the
108
- [runtime repo](https://github.com/martinobettucci/VibeASR-bitnet.cpp/tree/claude/asr-cpu-optimization-cztnh9)
109
- for methodology and reproduction scripts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
110
 
111
  The runtime also ships opt-in decoder-level hotword biasing (`--hotwords`,
112
  token-trie logit boosting): on FLEURS-French with oracle terms it recovers ~4 WER
 
85
  | whisper.cpp small q5_1 | 8.5 | 4.6 | 7.1 | 12.9 | 16.7 | 6.6 | 8.0 | **9.92** | **3.06Γ—** |
86
  | whisper.cpp large-v3-turbo q5_0 | 4.1 | 3.7 | 4.0 | 3.3 | 10.3 | 3.3 | 4.3 | **5.12** | **0.75Γ—** |
87
 
88
+ Slim-vs-original is parity (+0.32, per-language scatter both ways β€” German
89
+ improves under the fork's overflow-fixed kernels, Spanish regresses). On these
90
+ **short clips** whisper-small is more accurate and ~1.3Γ— faster.
91
+
92
+ **But short clips are the wrong test for this architecture, and the difference is
93
+ dramatic.** VibeVoice-ASR compresses audio 3200Γ— into an LLM context so a whole
94
+ recording is one encode with full context available to the decoder. On 76 s of
95
+ continuous French audiobook speech, same host, back to back:
96
+
97
+ | Engine | RTF | **WER** |
98
+ |:--|--:|--:|
99
+ | **This model, fork runtime** | 0.44 | **8.88** |
100
+ | whisper.cpp small q5_1 | 0.30 | 27.57 |
101
+ | whisper.cpp large-v3-turbo q5 | 0.99 | 5.61 |
102
+
103
+ The ranking inverts: **8.88 against whisper-small's 27.57**, a 3Γ— gap the other
104
+ way, and within 3 points of large-v3-turbo at 2.3Γ— less compute. The same weights
105
+ score ~21 on this corpus's short clips β€” more context makes this model better,
106
+ which is the point of the architecture.
107
+
108
+ **Known limit: ~80 seconds.** Past that the decoder emits its end token early and
109
+ drops the tail (measured: 101 s of audio β†’ 119 of 280 reference words), with the
110
+ token budget 98% unused. It is a property of the weights, not the runtime β€”
111
+ upstream behaves identically. Segment recordings longer than ~75 s.
112
+
113
+ Two things this model does **not** do, despite the runtime supporting the format:
114
+ it emits **no speaker labels**, and the `{Start, End, Speaker, Content}` JSON
115
+ prompt is meant for the 7B model β€” asking this 1.5B for it costs ~1.4 WER. Use
116
+ plain text.
117
+
118
+ Guidance: **use whisper for short-clip transcription; use this for continuous
119
+ speech in the 30–75 s range**, where it is far more accurate than whisper-small at
120
+ comparable cost and near turbo at a fraction of it β€” plus decoder-level hotword
121
+ biasing (FLEURS-French 36.0 β†’ 31.9 with domain terms). Methodology and
122
+ reproduction scripts:
123
+ [runtime repo](https://github.com/martinobettucci/VibeASR-bitnet.cpp/tree/claude/asr-cpu-optimization-cztnh9).
124
 
125
  The runtime also ships opt-in decoder-level hotword biasing (`--hotwords`,
126
  token-trie logit boosting): on FLEURS-French with oracle terms it recovers ~4 WER