Instructions to use javierbits/Rumble-2-Flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use javierbits/Rumble-2-Flash-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M # Run inference directly in the terminal: llama cli -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M # Run inference directly in the terminal: llama cli -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Use Docker
docker model run hf.co/javierbits/Rumble-2-Flash-GGUF:Q5_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use javierbits/Rumble-2-Flash-GGUF with Ollama:
ollama run hf.co/javierbits/Rumble-2-Flash-GGUF:Q5_K_M
- Unsloth Desktop
- Docker Model Runner
How to use javierbits/Rumble-2-Flash-GGUF with Docker Model Runner:
docker model run hf.co/javierbits/Rumble-2-Flash-GGUF:Q5_K_M
- Lemonade
How to use javierbits/Rumble-2-Flash-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Run and chat with the model
lemonade run user.Rumble-2-Flash-GGUF-Q5_K_M
List all available models
lemonade list
- Atomic Chat
Rumble-2-Flash — GGUF (Q5_K_M / Q8_0, llama.cpp)
0.5 B speech recognition model for 45 languages, converted from
Reson8/Rumble-2-Flash to llama.cpp GGUF.
Apache-2.0, inherited from the base model.
I converted and quantized these files with llama.cpp 9a080f4 plus a small converter patch
(llama-cpp-rumble.patch) and a chat template for llama-server
(chat_template.llamacpp.jinja); build_gguf.sh
reproduces every file and checksum below.
I benchmarked the files before publishing on 700 FLEURS test utterances over 7 languages, against the BF16 checkpoint on the same utterances:
- Q5_K_M: 12.10 % pooled error vs 11.65 % BF16, delta +0.45 pp (95 % CI −0.09 … +1.22).
- Q8_0: 11.98 % vs 11.65 %, delta +0.34 pp (95 % CI 0.00 … +0.73), borderline.
Both deltas are driven by Cantonese, where the base model is already weak (42 % CER) and the quantizer changes many output tokens; no other language shows a detectable change. 302/700 transcripts differ byte-for-byte at Q5_K_M (169/700 at Q8_0), mostly punctuation and word choice. Full method, per-clip transcripts and intervals: Accuracy.
Files
One decoder + the audio encoder (mmproj); both are required.
| File | Size | SHA-256 |
|---|---|---|
Rumble-2-Flash-Q5_K_M.gguf |
242,414,400 B | 2acc1ba209501ceb2c85241f303dfc4421ddae3e3c1655a4f66c66b8468f93d8 |
Rumble-2-Flash-Q8_0.gguf |
346,355,648 B | 64a38f3937b7db0fe37124b1ce641f0d35a2237046b555fc59cb55cc5bddfa0e |
mmproj-Rumble-2-Flash-Q8_0.gguf |
207,942,080 B | c7aae5fd0172e36758c536bf8491aa56227d2e05d8f0f8c25dde68d6da3063cf |
The auxiliary CTC head of the original repository (ctc/, 357 MB F32) is not included;
llama.cpp has no runtime for it. The F16 conversion (645 MB decoder, 379 MB mmproj) is not
shipped either; build_gguf.sh produces it.
Accuracy
FLEURS test, 100 utterances per language, deterministic sample (seed 1234), greedy decoding,
forced language, Whisper text normalizers (CER for Mandarin and Cantonese). Paired bootstrap over
utterances, 2000 resamples. Measured on an NVIDIA GB10 against the BF16 checkpoint in
Transformers 5.23.
| Language | n | BF16 err % | Q5_K_M err % | delta pp [95% CI] | Q8_0 err % | delta pp [95% CI] |
|---|---|---|---|---|---|---|
| English | 100 | 7.29 | 7.97 | +0.68 [−0.14, +1.70] | 7.88 | +0.59 [−0.08, +1.46] |
| Spanish | 100 | 4.28 | 4.23 | −0.06 [−0.46, +0.31] | 4.23 | −0.05 [−0.28, +0.15] |
| Portuguese | 100 | 7.00 | 7.34 | +0.35 [−0.22, +1.07] | 7.07 | +0.07 [−0.32, +0.69] |
| German | 100 | 7.22 | 7.03 | −0.19 [−0.74, +0.33] | 7.29 | +0.06 [−0.26, +0.42] |
| French | 100 | 7.16 | 7.70 | +0.55 [−0.24, +1.42] | 6.95 | −0.20 [−0.62, +0.23] |
| Chinese (CER) | 100 | 6.26 | 6.27 | +0.01 [−0.41, +0.46] | 6.45 | +0.19 [−0.14, +0.62] |
| Cantonese (CER) | 100 | 42.33 | 44.18 | +1.85 [−1.63, +7.16] | 44.02 | +1.69 [−0.30, +4.23] |
| pooled | 700 | 11.65 | 12.10 | +0.45 [−0.09, +1.22] | 11.98 | +0.34 [+0.00, +0.73] |
Cantonese dominates the pooled delta even though its own interval is not significant: the per-utterance variance there is large. Excluding Cantonese, pooled error is 6.76 % (Q5_K_M) and 6.64 % (Q8_0) against 6.53 % for BF16 — deltas +0.22 pp [−0.03, +0.52] and +0.11 pp [−0.07, +0.33], both within noise. Absolute numbers are not comparable to the base model card's Open ASR Leaderboard run (different splits and normalization); compare rows within this table only.
Usage
llama-server -m Rumble-2-Flash-Q5_K_M.gguf --mmproj mmproj-Rumble-2-Flash-Q8_0.gguf \
-c 1024 -np 1 --temp 0 --jinja
curl -F file=@audio.wav -F language=de -F temperature=0 \
http://127.0.0.1:8080/v1/audio/transcriptions
language accepts ISO codes or English names; omit it to auto-detect (the reply then starts with
language <Name><asr_text>). Use greedy decoding, as measured. On a phone keep -c 1024
(15 s clips stay under ~300 tokens). CPU reference on this box: 1.1 s per 9–13 s clip at idle
(RTFx 10, peak RSS 1.3 GiB at -c 4096); single-stream numbers move 3–4x with background load,
and phones are slower than a 20-thread ARM server — the LiteRT Android build reports 1.5 s for a
5.8 s clip.
Long audio: the endpoint does not segment; audio costs ~13 tokens/s, so -c must cover audio +
transcript. A 285 s clip needs -c 8192 (3726 prompt tokens, 812 generated, same transcript as
BF16); at -c 4096 the output is truncated. -c 1024 fits ~60 s.
llama.cpp llama-mtmd-cli --jinja --audio clip.wav -p "lang:en" works too. Without --jinja
the embedded template is ignored and transcripts degrade.
How it was made
- Source:
Reson8/Rumble-2-Flashrevisiona0c34287229103b706c670d70791683e34379732(0.5 B: 18-layer audio encoder + 14-layer Qwen3 decoder, 99,630-token vocabulary) plus llama.cpp9a080f4(CUDA build). llama-cpp-rumble.patchfixes the three points where the stock converter does not match this checkpoint: the pruned 99,630-token tokenizer has a new fingerprint (mapped toqwen2), the config omits M-RoPE sections (written as[24, 20, 20, 0], the Qwen3-ASR value; audio and text positions are identical in every section, so this is plain RoPE), and the Transformers-5 layout names the projectormulti_modal_projector.linear_1/2and the towermodel.audio_tower.*(mapped to theaudio_tower.proj1/2/audio_tower.*names the qwen3a mmproj graph expects).chat_template.llamacpp.jinjaadapts the vendor template for llama-server: user content stays a string (llama-server puts an audio media marker there), the endpoint's own instruction is dropped because the model was trained with audio alone in the user turn, language directives are read, and the Reson8 control tokens and{% generation %}tags are handled.- Quantization:
llama-quantizeQ5_K_M, Q8_0 and mmproj Q8_0. No imatrix:llama-imatrixfeeds text, while this decoder consumes audio embeddings. - Reproduce:
./build_gguf.sh(downloads the source checkpoint, applies the patch, converts, quantizes, prints checksums).
Benchmark scripts
bench_select.py builds the deterministic FLEURS split;
bench_release.py runs the BF16 reference and a GGUF pair through the
Transformers and llama.cpp backends and writes out/bench/REPORT.md with per-clip transcripts and
paired bootstrap intervals. The runs quoted above are archived in bench/ and
bench-q8/, the split is data/bench_clips.json.
hf download google/fleurs --repo-type dataset --local-dir data \
data/de_de/audio/test.tar.gz data/de_de/test.tsv # ...see bench/REPORT.md for all seven
.venv/bin/python bench_select.py --clips 100
.venv/bin/python bench_release.py --clips 100 --llama-server /path/to/llama-server
.venv/bin/python bench_release.py --clips 100 --engines gguf --bf16-json out/bench/bf16.json \
--gguf-decoder Rumble-2-Flash-Q8_0.gguf --out-dir out/bench-q8
Limitations
- Read speech, 3–25 s clips. Long-form is supported (see Usage) but not scored here; conversational and code-switched audio are not covered.
- Forced language: language identification is not scored.
- Cantonese is weak in the base model (42 % CER) and the quant makes it noisier; treat Cantonese numbers as lower confidence.
- Below ~0.45 GB total, accuracy degrades measurably on this 0.5 B model; that is why the small build is Q5_K_M.
License
Apache-2.0, inherited from Reson8/Rumble-2-Flash. All credit for the model belongs to Reson8;
the conversion and benchmark are mine. llama.cpp is MIT-licensed by the ggml authors.
- Downloads last month
- -
5-bit
8-bit