VibeVoice-ASR-BitNet-slim

A repack of microsoft/VibeVoice-ASR-BitNet with a redundant tensor removed. No retraining, no re-quantisation of the ternary weights — the transformer body is byte-for-byte the original.

	LM	VAE	Total
microsoft/VibeVoice-ASR-BitNet	992.9 MB	703.1 MB	1.70 GB
this repo	526.1 MB	703.1 MB	1.23 GB

What changed

The released LM GGUF carries output.weight as F16, 466.7 MB — 47% of the file, next to token_embd.weight already stored as Q6_K at 191.4 MB.

In the source checkpoint tie_word_embeddings is true, and lm_head.weight is bit-identical to embed_tokens.weight. So the F16 tensor is the same matrix a second time, at higher precision than the copy the file already holds.

llama.cpp loads LLM_TENSOR_OUTPUT as TENSOR_NOT_REQUIRED and falls back to token_embd when it is absent, so the tensor can simply be dropped. The output projection then runs through the Q6_K copy instead of the F16 one — the only numerical change in this repack.

Removing it also takes 466.7 MB off the memory read on every decoded token, which was roughly half the LM's per-token bandwidth.

What it costs

Not nothing. Moving the output projection from F16 to Q6_K is measurable. FLEURS, 24 clips per language, greedy decoding, 2 threads, numbers spelled out on both sides before scoring:

Language	microsoft/VibeVoice-ASR-BitNet	this repo	Δ
Spanish	6.47	6.47	+0.00
English	8.23	8.58	+0.34
Portuguese	8.90	8.57	−0.33
Italian	9.67	9.52	−0.16
German	14.63	14.98	+0.35
French	34.08	35.88	+1.81
corpus	14.31	14.69	+0.38

So: about +0.4 WER for −47% LM size. Two languages improve, one is unchanged, three get worse. At ~400 reference words per language a ±0.3 swing is inside the noise; French's +1.81 is roughly seven word errors and sits at the edge of it.

Take the trade if size or decode bandwidth matters to you, and don't if you need every last point of accuracy.

Bit budget

Component	Type	Weights	MB	bits/wt
transformer body	I2_S	1,310,195,712	327.6	2.00
token embedding	Q6_K	233,373,696	191.4	6.56
norms / biases	F32	144,896	0.6	32.00
total		1,543,714,304	519.6	2.69

The released file is 4.44 bits/weight overall; this one is 2.69. Note that the ternary body is packed at exactly 2.000 bits/weight, not log₂3 = 1.585 — I2_S stores four ternary values per byte and leaves one of four codes unused, which is 68 MB of padding (20.8% of the body).

Usage

Drop-in for the released model — same runtime, same flags:

./build/bin/asr_infer \
    --vae-model vibeasr-vae-encoder-i8_s.gguf \
    --lm-model  vibeasr-lm-i2_s-tied.gguf \
    --audio input.wav -t 4 --greedy

Languages

VibeVoice-ASR was trained on en, zh, fr, it, ko, pt, vi. Among EU official languages that means English, French, Italian and Portuguese are in-distribution; Spanish and German are not in the training mix but generalise usably. The other EU languages degrade sharply and this repack does not change that — it is a packaging fix, not a capability change.

Speed

Not covered here by design. This card documents the model artifact — what changed in the weights and what it costs in accuracy. CPU inference speed is a property of the runtime, and the fork this model ships with carries substantial kernel work (AVX-512/VNNI dispatch, a register-tiled INT8 GEMM, vectorised quantisation epilogues — RTF well under real time on 4 modest cores). The measured speed tables, the per-stage breakdowns, and the scripts that regenerate them live in the GitHub README:

➡️ martinobettucci/VibeASR-bitnet.cpp — "CPU optimisation on AVX-512"

The WER tables above were produced by that repo's benchmark harness (bench/run_asr.py, methodology in bench/README.md); the harness documentation is the reference for how they were scored (FLEURS slices, corpus-level WER, digit runs spelled out in the clip's language on both sides).

Provenance

Produced with tools/requant_lm_head.cpp --drop from martinobettucci/VibeASR-bitnet.cpp, branch claude/asr-cpu-optimization-cztnh9. The VAE encoder and tokenizer files are copied unmodified from the upstream repo.

Licensed MIT, as upstream.