Rumble-2-Flash for LiteRT-LM

On-device speech-to-text in a single file.

This is Reson8/Rumble-2-Flash (a 0.5B Qwen3-ASR model covering 45 languages) packaged as a self-contained .litertlm bundle for LiteRT-LM Omni. The file contains the INT8 audio encoder and projector, the 14-layer Qwen3 decoder with stateful KV cache, the token embedder, the Hugging Face tokenizer, and LlmMetadata / ExecutorMetadata. No Python and no network are needed at inference time.

  • Architecture: 128-bin log-mel front-end, 18-layer audio encoder (tf_lite_audio_encoder_hw), 14-layer Qwen3 decoder (tf_lite_prefill_decode + tf_lite_embedder), 99,630-token vocabulary
  • Streaming: 5-second sliding windows, merged across overlaps by the LiteRT-LM ASR session
  • Transcription style (fixed at export): numbers as digits (<digits>), fillers dropped (<nofiller>), repeated words kept (<verbatim>), language auto-detected. These are Rumble's defaults.

Files

File Size Backend Notes
rumble_2_flash_5s_i8.litertlm 596 MB CPU (GPU untested) INT8 dynamic-range weights, 5 s window
rumble_asr_extension.py 5 KB — litert-torch extension used for the export
omni_runner_skip_until.patch 4 KB — Adds --asr_decode_skip_until_token_id to omni_runner

One runtime override is required. Rumble-2-Flash remaps the Qwen3 vocabulary, so its <asr_text> token is 99623, while LiteRT-LM's qwen3-asr runtime defaults decode_skip_until_token_id to the stock Qwen3-ASR value 151704. Without the override every transcript is empty. The value cannot be embedded in the file: a .litertlm holds a single top-level capability section, and this model uses LlmMetadata.

  • omni_runner: pass --model_name=qwen3-asr --asr_decode_skip_until_token_id=99623 (requires omni_runner_skip_until.patch).
  • C++ API: pass a Speech2TextMetadata as the fourth argument of OmniEngine::Create (see below).

Usage with LiteRT-LM Omni

1. Download

pip install -U huggingface_hub
hf download litert-community/Rumble-2-Flash-LiteRT --local-dir rumble

2. Build omni_runner with the patch

Tested against LiteRT-LM commit 7ee9bdb. Prerequisites (Bazel, Android NDK r28b+) are in the LiteRT-LM build guide.

git clone https://github.com/google-ai-edge/LiteRT-LM && cd LiteRT-LM
git apply ../rumble/omni_runner_skip_until.patch

# Desktop (macOS / Linux)
bazel build -c opt //omni:omni_runner

# Android arm64
export ANDROID_NDK_HOME=/path/to/android-ndk-r28c
bazel build -c opt --config=android_arm64 //omni:omni_runner

3. Run

bazel-bin/omni/omni_runner \
    --model_name=qwen3-asr \
    --model_path=../rumble/rumble_2_flash_5s_i8.litertlm \
    --audio_path=/path/to/audio.wav \
    --asr_decode_skip_until_token_id=99623

--model_name must be given explicitly because it cannot be inferred from the file name. WAV input of any sample rate and channel count is resampled to 16 kHz mono.

Android (adb)

D=/data/local/tmp/omni
adb shell mkdir -p $D
adb push bazel-bin/omni/omni_runner prebuilt/android_arm64/*.so \
    ../rumble/rumble_2_flash_5s_i8.litertlm audio.wav $D
adb shell "cd $D && LD_LIBRARY_PATH=$D ./omni_runner --backend=cpu \
    --cache_dir=$D/cache --model_name=qwen3-asr \
    --model_path=$D/rumble_2_flash_5s_i8.litertlm --audio_path=$D/audio.wav \
    --asr_decode_skip_until_token_id=99623"

Android (Termux, push-to-talk)

Copy omni_runner, the two prebuilt/android_arm64/*.so files and the .litertlm into ~/rumble (binaries cannot execute from shared storage), and grant Termux the microphone permission. Then:

pkg install ffmpeg termux-api

cat > ~/rumble/asr <<'EOF'
#!/data/data/com.termux/files/usr/bin/sh
D=$HOME/rumble
LD_LIBRARY_PATH=$D "$D/omni_runner" --backend="${BACKEND:-cpu}" --cache_dir="$D/cache" \
  --model_name=qwen3-asr --model_path="$D/rumble_2_flash_5s_i8.litertlm" \
  --audio_path="$1" --asr_decode_skip_until_token_id=99623 2>/dev/null
EOF
chmod +x ~/rumble/asr

termux-microphone-record -f ~/rec.m4a -l 10 && sleep 11
ffmpeg -y -loglevel error -i ~/rec.m4a ~/rec.wav && ~/rumble/asr ~/rec.wav

C++ API

#include "omni/omni_engine.h"
#include "runtime/proto/speech2text_metadata.pb.h"
#include "runtime/proto/speech2text_session_config.pb.h"

litert::omni::OmniEngine::Settings settings{
    .backend = litert::omni::OmniEngine::Settings::Backend::kCpu,
    .cache_dir = "/data/local/tmp/omni_cache",
    .model_path = "/path/to/rumble_2_flash_5s_i8.litertlm",
};
litert::lm::proto::Speech2TextSessionConfig session_config;
litert::lm::proto::Speech2TextMetadata metadata;
metadata.set_decode_skip_until_token_id(99623);  // Rumble's <asr_text>

ABSL_ASSIGN_OR_RETURN(
    auto engine, litert::omni::OmniEngine::Create(
                     "qwen3-asr", settings, &session_config, &metadata));

Link //omni:omni_engine, //omni:omni_session and //omni/speech2text:qwen3_asr. The rest of the session API is unchanged from the LiteRT-LM Omni README.

Validation

Sample: runtime/components/testdata/audio_sample.wav from LiteRT-LM (5.8 s, English).

Runtime Transcript
Hugging Face Transformers (FP32, reference) What paintings were inside the dead person's old kingdom tombs?
.litertlm, macOS CPU What paintings were inside the dead person's old Kingdom tombs.
.litertlm, Android CPU (Snapdragon 8 Elite Gen 5) What paintings were inside the dead person's old Kingdom tombs.
  • Audio encoder vs. PyTorch: FP32 max abs diff 6.7e-7; INT8 minimum per-token cosine similarity 0.986.
  • Android CPU (Snapdragon 8 Elite Gen 5, Termux): 5.8 s clip transcribed in 1.5 s, including first-run cache creation.

Not yet validated: GPU backend, NPU, long-form audio across many windows, and non-English accuracy.

How it was converted

The built-in qwen3_asr extension in litert-torch hardcodes stock Qwen3 prompt token IDs (for example 151644), which are outside Rumble's vocabulary. rumble_asr_extension.py rebuilds the encoder prompt from the checkpoint's own chat template and tokenizer, and takes the stop token from it.

pip install -r litert-torch/requirements.txt && pip install -e ./litert-torch
python -m litert_torch.generative.export_hf.export_main \
    --model=Reson8/Rumble-2-Flash \
    --output_dir=out \
    --extension_modules=./rumble_asr_extension.py \
    --quantization_recipe=dynamic_wi8_afp32

Exported with litert-torch commit 9d531ea, transformers 5.23.0 and torch 2.14.0.

License

Apache 2.0, inherited from Reson8/Rumble-2-Flash. All credit for the model goes to Reson8; this repository only repackages it for LiteRT-LM.

Downloads last month
43
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for litert-community/Rumble-2-Flash-LiteRT

Quantized
(10)
this model