How to use from
llama.cpp
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/Rumble-2-Flash-GGUF:
# Run inference directly in the terminal:
llama cli -hf AtomicChat/Rumble-2-Flash-GGUF:
Install from WinGet (Windows)
winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AtomicChat/Rumble-2-Flash-GGUF:
# Run inference directly in the terminal:
llama cli -hf AtomicChat/Rumble-2-Flash-GGUF:
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AtomicChat/Rumble-2-Flash-GGUF:
# Run inference directly in the terminal:
./build/bin/llama cli -hf AtomicChat/Rumble-2-Flash-GGUF:
Use Docker
docker model run hf.co/AtomicChat/Rumble-2-Flash-GGUF:
Quick Links

Rumble-2-Flash — GGUF (by AtomicChat)

Atomic Chat Discord GitHub X
  • Built by the team behind Atomic Chat, the app that runs open-weight models locally and offline.

GGUF files of Reson8/Rumble-2-Flash for llama.cpp. Rumble-2-Flash is a 0.5B speech recognition model for 45 languages under Apache 2.0: an 18-layer audio encoder and a 14-layer decoder. It is built on Qwen3-ASR-0.6B (see Base model), so it runs in llama.cpp as qwen3vl with the qwen3a audio projector.

llama.cpp note. Stock llama.cpp (tested on b12521) runs these files, but its Qwen3-ASR audio path passes the zero padding at the end of each clip to the model as if it were sound. On short clips the transcript then often ends with a few words that were never said ("... cried the professor. I am not."). We wrote a fix for llama.cpp, and with it llama.cpp matches the Transformers reference within noise on every test set. We are preparing it for upstream llama.cpp; until it lands there, the patch is in the metrics repo.

The tables below are measured with the fix. On stock b12521, the same files have up to 1 point more WER, most on LibriSpeech (BF16 on test-clean: 3.13 instead of 2.09). The files are the same in both cases.

On stock llama.cpp b12521 (without the fix), the shipped files measure:

Files LS clean LS other MLS DE MLS ES MLS FR FLEURS ZH
BF16 + mmproj F16 3.13 5.68 7.00 4.92 6.80 10.45
Q8_0 + mmproj F16 3.16 5.62 6.98 5.04 6.89 10.51
AD-Q5_K_M + mmproj Q8_0 3.27 5.96 7.06 4.89 6.77 10.64

Files

You need one model file + one mmproj (the audio encoder).

WER in % (FLEURS Mandarin: CER in %), 2,200 utterances, lower is better. Each model file is measured with mmproj-…-F16.

File GB on disk bpw LS clean LS other MLS DE MLS ES MLS FR FLEURS ZH
Rumble-2-Flash-BF16.gguf 0.65 16.0 2.09 4.70 6.85 4.57 6.57 10.38
Rumble-2-Flash-Q8_0.gguf 0.35 8.50 2.11 4.70 6.79 4.63 6.68 10.31
Rumble-2-Flash-AD-Q5_K_M.gguf 0.28 6.94 2.09 4.78 6.83 4.51 6.60 10.52
stock Q6_K (baseline, not shipped) 0.27 6.56 2.20 4.64 6.84 4.51 6.66 10.44
our AD-Q4_K_M (tried, not shipped) 0.26 6.48 2.22 4.81 7.05 4.74 6.88 10.54
stock Q5_K_M (baseline, not shipped) 0.24 5.92 2.14 4.89 7.00 4.57 6.87 10.76
stock Q4_K_M (baseline, not shipped) 0.22 5.32 2.42 5.21 7.62 5.14 7.47 11.99
mmproj (audio encoder) GB on disk LS clean LS other MLS DE MLS ES MLS FR FLEURS ZH
mmproj-Rumble-2-Flash-F16.gguf 0.38 2.09 4.70 6.85 4.57 6.57 10.38
mmproj-Rumble-2-Flash-Q8_0.gguf 0.21 2.11 4.70 6.71 4.61 6.64 10.38

The mmproj rows use the BF16 model file. Bold numbers are differences to BF16 whose paired 95% interval excludes zero; all others are within the noise of BF16.

  • Recommended: AD-Q5_K_M + mmproj-Q8_0, 0.50 GB in total. Measured as a pair: 2.12 / 4.80 / 6.89 / 4.54 / 6.65 / 10.60. That is within noise of BF16 on five of six test sets, with +0.2 points of CER on Mandarin. If that matters, use the F16 mmproj (0.66 GB in total), which is within noise everywhere.
  • Q8_0 (0.56 GB with the Q8_0 mmproj) and BF16 are here for reference: Q8_0 is within noise of BF16 everywhere.
  • Below about 0.27 GB every file we tried loses accuracy. At 0.26–0.27 GB, the stock Q6_K and our AD-Q4_K_M are on par and both lose a little (LibriSpeech clean, and French for AD-Q4_K_M), so we ship neither. The stock Q4_K_M (0.22 GB) loses everywhere, up to 1.6 points of CER in Mandarin. Smaller K-quants lose more (stock Q3_K_M: 18.8% CER in Mandarin).
  • The mmproj is the larger part of the download. Its Q8_0 is within noise of F16, and F16 is within noise of F32.

Run it

Qwen3-ASR support is in llama.cpp since b8769; these files were tested on b12521. Pass the language: the model then writes the transcript only. Without it, the reply starts with language <Name><asr_text> (the detected language), as in the original model.

Server with the OpenAI-style transcription endpoint:

llama-server -m Rumble-2-Flash-AD-Q5_K_M.gguf --mmproj mmproj-Rumble-2-Flash-Q8_0.gguf -ngl 99
curl http://localhost:8080/v1/audio/transcriptions -F file=@audio.wav -F language=de

Chat endpoint, with the model's transcription options in chat_template_kwargs:

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
  "messages": [{"role": "user", "content": [{"type": "input_audio", "input_audio": {"data": "<base64 wav>", "format": "wav"}}]}],
  "temperature": 0,
  "chat_template_kwargs": {"language": "English", "digits": true, "filler": false, "verbatim": true}
}'
Option Default Effect
language unset Forces the language: English name ("Dutch") or ISO code ("nl"). The transcription endpoint's language field and a lang:xx prompt work too.
digits true Numbers as digits; false writes them as words.
filler false true keeps fillers such as "um" and "uh".
verbatim true false collapses repeated words.

Command line:

llama-mtmd-cli -m Rumble-2-Flash-AD-Q5_K_M.gguf --mmproj mmproj-Rumble-2-Flash-Q8_0.gguf --jinja --audio audio.wav -p "lang:en" --temp 0
  • llama-mtmd-cli needs --jinja: without it, the CLI ignores the embedded template and the transcripts get worse. llama-server uses the template by default.
  • Use greedy decoding (temperature 0), as the reference does.
  • Audio is resampled to 16 kHz mono. Long audio is encoded in 8-second windows, as in the reference.
  • The chat template in these files is the vendor's template adapted for llama.cpp. llama-server passes audio as a media marker inside the user text, which the vendor's template drops, so the user turn now keeps it. The template also reads the transcription endpoint's language field and removes its fixed "Transcribe audio to text" instruction, because the model was trained with audio alone in the user turn. For audio passed as typed parts (Transformers), it renders the same prompt as the vendor's, byte for byte.
  • The auxiliary CTC head of the original repo (ctc/) is not included: llama.cpp has no runtime for it.

How these files were made and measured

  • Conversion. The source is Reson8/Rumble-2-Flash @ 92ec9141, converted on llama.cpp b12521. The stock converter does not convert this checkpoint yet, and we bridged three gaps (script in the metrics repo):

    1. The vocabulary is pruned to 99,630 ids, so the tokenizer fingerprint is new. Its normalizer, pre-tokenizer and decoder are identical to Qwen3-ASR's, so it maps to qwen2.
    2. The Transformers-format config has no M-RoPE sections, and llama.cpp needs them for qwen3vl. We write [24, 20, 20, 0], the value of Qwen3-ASR-0.6B. Audio and text positions are the same in every section, so this is plain RoPE, as in the reference.
    3. The mmproj converter reads only the original Qwen layout (thinker.audio_tower.*). The Transformers layout names the projector multi_modal_projector.linear_1/2, so we map it to proj1/proj2.

    As a check, we pushed Qwen/Qwen3-ASR-0.6B-hf through the same path and compared it with ggml-org/Qwen3-ASR-0.6B-GGUF. All 310 LLM tensors and 301 of 302 mmproj tensors are byte-identical, and the metadata matches. The one exception is the generated sinusoid table, which differs by 1e-7.

  • Real types. Every tensor type of every file was read back. All rows are multiples of 256, so there are no silent type swaps.

  • Layout. The token table is 32% of the decoder and is tied to the output head. AD-Q5_K_M keeps the token table and attn_k/attn_v in Q8_0, ffn_down and the first and last block in Q6_K, and the rest in Q5_K.

  • No imatrix. llama-imatrix feeds text only, while the decoder reads audio embeddings, so a text imatrix would calibrate the wrong inputs.

  • Test. 2,200 utterances, 6.7 hours: LibriSpeech test-clean 500 and test-other 500 (EN), Multilingual LibriSpeech test DE / ES / FR 300 each, FLEURS test Mandarin 300. The language is forced and the vendor's template defaults are used (digits, no fillers, verbatim). Normalization is Whisper's English normalizer for EN and its basic normalizer for the rest; Chinese is scored as CER without spaces. Decoding is greedy, through llama-server with 8 parallel slots on one NVIDIA A10.

    • Every file is compared with BF16 on the same clips. A difference counts when its paired 95% bootstrap interval over utterances excludes zero. Two runs of the same BF16 file agree on 97–99% of transcripts, which is the floor of this setup.
    • The reference is the vendor's weights in Transformers 5.20.2 (BF16, CUDA, greedy), on 100 clips per subset.
  • Numbers vs the vendor's card. The vendor reports the Open ASR Leaderboard, which uses other test splits and its own text normalization, so absolute numbers differ from ours (FLEURS Mandarin: 5.82 there, 9.9% CER in Transformers on our harness). Compare rows within one table only.

  • All results, per-clip transcripts, tensor types, logs and scripts are in AtomicChat/Rumble-2-Flash-GGUF-metrics.

Base model & license

Base model: Reson8/Rumble-2-Flash by Reson8, released under Apache 2.0. These files keep it (see LICENSE).

The vendor's card names no base model. A comparison of the weights shows that Rumble-2-Flash is derived from Qwen/Qwen3-ASR-0.6B by the Qwen team, also Apache 2.0:

  • its 14 decoder layers match layers 0–6 and 21–27 of Qwen3-ASR's 28 (cosine 0.96–0.99 after fine-tuning);
  • its 99,562 text tokens are a subset of Qwen3-ASR's vocabulary (row cosine 0.98);
  • its audio encoder is Qwen3-ASR's, fine-tuned (cosine 0.96–0.99 per tensor).

Quants by AtomicChat.

Downloads last month
136
GGUF
Model size
0.3B params
Architecture
qwen3vl
Hardware compatibility
Log In to add your hardware

5-bit

8-bit

16-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AtomicChat/Rumble-2-Flash-GGUF

Quantized
(9)
this model
Quantizations
1 model

Collection including AtomicChat/Rumble-2-Flash-GGUF