Follow on X GitHub Hugging Face Website

Rumble-2-Flash — GGUF (Q5_K_M / Q8_0, llama.cpp)

0.5 B speech recognition model for 45 languages, converted from Reson8/Rumble-2-Flash to llama.cpp GGUF. Apache-2.0, inherited from the base model.

I converted and quantized these files with llama.cpp 9a080f4 plus a small converter patch (llama-cpp-rumble.patch) and a chat template for llama-server (chat_template.llamacpp.jinja); build_gguf.sh reproduces every file and checksum below.

I benchmarked the files before publishing on 700 FLEURS test utterances over 7 languages, against the BF16 checkpoint on the same utterances:

  • Q5_K_M: 12.10 % pooled error vs 11.65 % BF16, delta +0.45 pp (95 % CI −0.09 … +1.22).
  • Q8_0: 11.98 % vs 11.65 %, delta +0.34 pp (95 % CI 0.00 … +0.73), borderline.

Both deltas are driven by Cantonese, where the base model is already weak (42 % CER) and the quantizer changes many output tokens; no other language shows a detectable change. 302/700 transcripts differ byte-for-byte at Q5_K_M (169/700 at Q8_0), mostly punctuation and word choice. Full method, per-clip transcripts and intervals: Accuracy.

Files

One decoder + the audio encoder (mmproj); both are required.

File Size SHA-256
Rumble-2-Flash-Q5_K_M.gguf 242,414,400 B 2acc1ba209501ceb2c85241f303dfc4421ddae3e3c1655a4f66c66b8468f93d8
Rumble-2-Flash-Q8_0.gguf 346,355,648 B 64a38f3937b7db0fe37124b1ce641f0d35a2237046b555fc59cb55cc5bddfa0e
mmproj-Rumble-2-Flash-Q8_0.gguf 207,942,080 B c7aae5fd0172e36758c536bf8491aa56227d2e05d8f0f8c25dde68d6da3063cf

The auxiliary CTC head of the original repository (ctc/, 357 MB F32) is not included; llama.cpp has no runtime for it. The F16 conversion (645 MB decoder, 379 MB mmproj) is not shipped either; build_gguf.sh produces it.

Accuracy

FLEURS test, 100 utterances per language, deterministic sample (seed 1234), greedy decoding, forced language, Whisper text normalizers (CER for Mandarin and Cantonese). Paired bootstrap over utterances, 2000 resamples. Measured on an NVIDIA GB10 against the BF16 checkpoint in Transformers 5.23.

Language n BF16 err % Q5_K_M err % delta pp [95% CI] Q8_0 err % delta pp [95% CI]
English 100 7.29 7.97 +0.68 [−0.14, +1.70] 7.88 +0.59 [−0.08, +1.46]
Spanish 100 4.28 4.23 −0.06 [−0.46, +0.31] 4.23 −0.05 [−0.28, +0.15]
Portuguese 100 7.00 7.34 +0.35 [−0.22, +1.07] 7.07 +0.07 [−0.32, +0.69]
German 100 7.22 7.03 −0.19 [−0.74, +0.33] 7.29 +0.06 [−0.26, +0.42]
French 100 7.16 7.70 +0.55 [−0.24, +1.42] 6.95 −0.20 [−0.62, +0.23]
Chinese (CER) 100 6.26 6.27 +0.01 [−0.41, +0.46] 6.45 +0.19 [−0.14, +0.62]
Cantonese (CER) 100 42.33 44.18 +1.85 [−1.63, +7.16] 44.02 +1.69 [−0.30, +4.23]
pooled 700 11.65 12.10 +0.45 [−0.09, +1.22] 11.98 +0.34 [+0.00, +0.73]

Cantonese dominates the pooled delta even though its own interval is not significant: the per-utterance variance there is large. Excluding Cantonese, pooled error is 6.76 % (Q5_K_M) and 6.64 % (Q8_0) against 6.53 % for BF16 — deltas +0.22 pp [−0.03, +0.52] and +0.11 pp [−0.07, +0.33], both within noise. Absolute numbers are not comparable to the base model card's Open ASR Leaderboard run (different splits and normalization); compare rows within this table only.

Usage

llama-server -m Rumble-2-Flash-Q5_K_M.gguf --mmproj mmproj-Rumble-2-Flash-Q8_0.gguf \
  -c 1024 -np 1 --temp 0 --jinja

curl -F file=@audio.wav -F language=de -F temperature=0 \
  http://127.0.0.1:8080/v1/audio/transcriptions

language accepts ISO codes or English names; omit it to auto-detect (the reply then starts with language <Name><asr_text>). Use greedy decoding, as measured. On a phone keep -c 1024 (15 s clips stay under ~300 tokens). CPU reference on this box: 1.1 s per 9–13 s clip at idle (RTFx 10, peak RSS 1.3 GiB at -c 4096); single-stream numbers move 3–4x with background load, and phones are slower than a 20-thread ARM server — the LiteRT Android build reports 1.5 s for a 5.8 s clip.

Long audio: the endpoint does not segment; audio costs ~13 tokens/s, so -c must cover audio + transcript. A 285 s clip needs -c 8192 (3726 prompt tokens, 812 generated, same transcript as BF16); at -c 4096 the output is truncated. -c 1024 fits ~60 s.

llama.cpp llama-mtmd-cli --jinja --audio clip.wav -p "lang:en" works too. Without --jinja the embedded template is ignored and transcripts degrade.

How it was made

  • Source: Reson8/Rumble-2-Flash revision a0c34287229103b706c670d70791683e34379732 (0.5 B: 18-layer audio encoder + 14-layer Qwen3 decoder, 99,630-token vocabulary) plus llama.cpp 9a080f4 (CUDA build).
  • llama-cpp-rumble.patch fixes the three points where the stock converter does not match this checkpoint: the pruned 99,630-token tokenizer has a new fingerprint (mapped to qwen2), the config omits M-RoPE sections (written as [24, 20, 20, 0], the Qwen3-ASR value; audio and text positions are identical in every section, so this is plain RoPE), and the Transformers-5 layout names the projector multi_modal_projector.linear_1/2 and the tower model.audio_tower.* (mapped to the audio_tower.proj1/2 / audio_tower.* names the qwen3a mmproj graph expects).
  • chat_template.llamacpp.jinja adapts the vendor template for llama-server: user content stays a string (llama-server puts an audio media marker there), the endpoint's own instruction is dropped because the model was trained with audio alone in the user turn, language directives are read, and the Reson8 control tokens and {% generation %} tags are handled.
  • Quantization: llama-quantize Q5_K_M, Q8_0 and mmproj Q8_0. No imatrix: llama-imatrix feeds text, while this decoder consumes audio embeddings.
  • Reproduce: ./build_gguf.sh (downloads the source checkpoint, applies the patch, converts, quantizes, prints checksums).

Benchmark scripts

bench_select.py builds the deterministic FLEURS split; bench_release.py runs the BF16 reference and a GGUF pair through the Transformers and llama.cpp backends and writes out/bench/REPORT.md with per-clip transcripts and paired bootstrap intervals. The runs quoted above are archived in bench/ and bench-q8/, the split is data/bench_clips.json.

hf download google/fleurs --repo-type dataset --local-dir data \
  data/de_de/audio/test.tar.gz data/de_de/test.tsv      # ...see bench/REPORT.md for all seven
.venv/bin/python bench_select.py --clips 100
.venv/bin/python bench_release.py --clips 100 --llama-server /path/to/llama-server
.venv/bin/python bench_release.py --clips 100 --engines gguf --bf16-json out/bench/bf16.json \
  --gguf-decoder Rumble-2-Flash-Q8_0.gguf --out-dir out/bench-q8

Limitations

  • Read speech, 3–25 s clips. Long-form is supported (see Usage) but not scored here; conversational and code-switched audio are not covered.
  • Forced language: language identification is not scored.
  • Cantonese is weak in the base model (42 % CER) and the quant makes it noisier; treat Cantonese numbers as lower confidence.
  • Below ~0.45 GB total, accuracy degrades measurably on this 0.5 B model; that is why the small build is Q5_K_M.

License

Apache-2.0, inherited from Reson8/Rumble-2-Flash. All credit for the model belongs to Reson8; the conversion and benchmark are mine. llama.cpp is MIT-licensed by the ggml authors.

Downloads last month
-
GGUF
Model size
0.3B params
Architecture
qwen3vl
Hardware compatibility
Log In to add your hardware

5-bit

8-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for javierbits/Rumble-2-Flash-GGUF

Quantized
(10)
this model