Automatic Speech Recognition
GGUF
ggml
quantized
llama.cpp
qwen3-asr
speech-recognition
on-device
conversational
Instructions to use javierbits/Rumble-2-Flash-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use javierbits/Rumble-2-Flash-GGUF with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M # Run inference directly in the terminal: llama cli -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M # Run inference directly in the terminal: llama cli -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama serve -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M # Run inference directly in the terminal: ./build/bin/llama cli -hf javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Use Docker
docker model run hf.co/javierbits/Rumble-2-Flash-GGUF:Q5_K_M
- Llama new
- LM Studio
- Jan
- Ollama
How to use javierbits/Rumble-2-Flash-GGUF with Ollama:
ollama run hf.co/javierbits/Rumble-2-Flash-GGUF:Q5_K_M
- Unsloth Desktop
- Docker Model Runner
How to use javierbits/Rumble-2-Flash-GGUF with Docker Model Runner:
docker model run hf.co/javierbits/Rumble-2-Flash-GGUF:Q5_K_M
- Lemonade
How to use javierbits/Rumble-2-Flash-GGUF with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull javierbits/Rumble-2-Flash-GGUF:Q5_K_M
Run and chat with the model
lemonade run user.Rumble-2-Flash-GGUF-Q5_K_M
List all available models
lemonade list
- Atomic Chat
Download llama-cpp-rumble.patch from javierbits/Rumble-2-Flash-GGUF: direct link, hf CLI and curl.
- Browser
- Download file 4.06 kB
-
https://huggingface.co/javierbits/Rumble-2-Flash-GGUF/resolve/main/llama-cpp-rumble.patch
- Command line
-
hf download hf://javierbits/Rumble-2-Flash-GGUF/llama-cpp-rumble.patch
-
curl -L -o llama-cpp-rumble.patch https://huggingface.co/javierbits/Rumble-2-Flash-GGUF/resolve/main/llama-cpp-rumble.patch
4.06 kB
| diff --git a/conversion/base.py b/conversion/base.py | |
| index 7cfbff3..e7a850a 100644 | |
| --- a/conversion/base.py | |
| +++ b/conversion/base.py | |
| class TextModel(ModelBase): | |
| # ref: https://huggingface.co/thinkingmachines/Inkling-2-Small | |
| res = "inkling2" | |
| + if chkhsh == "9b33e5d83df24061dcdb1f30e51583be2f0879fb36c2e13d3b03c037742561f3": | |
| + # ref: https://huggingface.co/Reson8/Rumble-2-Flash (pruned Qwen3-ASR tokenizer) | |
| + res = "qwen2" | |
| + | |
| if res is None: | |
| logger.warning("\n") | |
| logger.warning("**************************************************************************************") | |
| diff --git a/conversion/qwen3vl.py b/conversion/qwen3vl.py | |
| index 9458366..2693b5b 100644 | |
| --- a/conversion/qwen3vl.py | |
| +++ b/conversion/qwen3vl.py | |
| from __future__ import annotations | |
| import json | |
| +import os | |
| from typing import Any, Callable, Iterable, TYPE_CHECKING | |
| class Qwen3ASRMmprojModel(Qwen3OmniMmprojModel): | |
| has_audio_encoder = True | |
| has_vision_encoder = False | |
| + def get_audio_config(self) -> dict[str, Any] | None: | |
| + # The Transformers-5 checkpoint keeps audio_config at the top level (no thinker_config). | |
| + return self.global_config.get("audio_config") or self.global_config["thinker_config"].get("audio_config") | |
| + | |
| + @classmethod | |
| + def filter_tensors(cls, item): | |
| + name, gen = item | |
| + if name.startswith("model."): | |
| + name = name.replace("model.", "", 1) | |
| + if name.startswith("multi_modal_projector.linear_"): | |
| + # Transformers-5 layout: projector is multi_modal_projector.linear_1/2; the GGUF map | |
| + # (and llama.cpp's qwen3a graph) expects audio_tower.proj1/2. | |
| + name = "audio_tower.proj" + name.rsplit("_", 1)[1] | |
| + if "audio_tower." not in name: | |
| + return None | |
| + return MmprojModel.filter_tensors((name, gen)) | |
| + | |
| @ModelBase.register("Glm4vForConditionalGeneration", "Glm4vMoeForConditionalGeneration", "GlmOcrForConditionalGeneration") | |
| @ModelBase.example("zai-org/GLM-4.1V-9B-Thinking", "zai-org/GLM-4.5V") | |
| class Qwen3ASRTextModel(Qwen3VLTextModel): | |
| def set_gguf_parameters(self): | |
| super().set_gguf_parameters() | |
| self.gguf_writer.add_num_deepstack_layers(0) | |
| + # Rumble's config drops mrope_section; Qwen3-ASR-0.6B uses [24, 20, 20, 0] and the | |
| + # audio/text positions are identical in every section, so this is plain RoPE. | |
| + if "mrope_section" not in self.rope_parameters: | |
| + self.gguf_writer.add_rope_dimension_sections([24, 20, 20, 0]) | |
| def set_vocab(self): | |
| super().set_vocab() | |
| - # fix chat template, use correct chatml format | |
| - self.gguf_writer.add_chat_template("{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}") | |
| + # llama-server runs the model with a media marker inside the user text, and the vendor | |
| + # template is written for typed content plus {% generation %}. RUMBLE_CHAT_TEMPLATE points | |
| + # at a llama.cpp-compatible adaptation (see the model repo); fall back to plain chatml. | |
| + template = None | |
| + template_path = os.environ.get("RUMBLE_CHAT_TEMPLATE") | |
| + if template_path and os.path.isfile(template_path): | |
| + with open(template_path, "r", encoding="utf-8") as f: | |
| + template = f.read() | |
| + if template is None: | |
| + template = "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}" | |
| + self.gguf_writer.add_chat_template(template) | |
| # correct BOS/EOS tokens | |
| with open(self.dir_model / "tokenizer_config.json", "r", encoding="utf-8") as f: | |
| tokenizer_config = json.load(f) | |