SenseNova-Loop-9B EXL3
EXL3 quantizations of sensenova/SenseNova-Loop-9B (revision 329a65f7) for ExLlamaV3.
Every branch includes the original vision tower in BF16 and the model's MTP head (4-bit) for speculative decoding: about 2.4x faster
generation with the same output.
Requires ExLlamaV3 with SenseNova Loop support, in review upstream (turboderp-org/exllamav3#609). Until it is released, install the fork (see How to run). Stock ExLlamaV3 and TabbyAPI do not load this model yet.
Pick your card
| GPU memory | Branch | Context, GPU only for the model | Context, GPU also runs your desktop |
|---|---|---|---|
| 12 GB | 4.0bpw |
136k (config) | 104k (config) |
| 16 GB | 5.0bpw |
232k (config) | 204k (config) |
| 24 GB | 6.0bpw |
456k (config) | 424k (config) |
| 32 GB | 8.0bpw |
512k, native max (config) | 512k (config) |
TabbyAPI configs with MTP drafting, Q8 KV cache and the largest KV pool that fits, leaving about 0.75 GB free (plus 1 GB for the display in the desktop configs) and room to encode one image of up to 4 MP. Variants for 4 parallel requests, a calculator for your exact card (desktop, text-only, other branches) and all measurements: Maxrio9/sensenova-loop-9b-exl3-recipes. The 16 GB desktop config ran unchanged in TabbyAPI, memory capped to the card, with a 4 MP image and a 205k-token prompt that fills its context: all five planted facts found, using 14.0 of the 14.2 GB allowed (the 12 and 16 GB configs without MTP passed the same check earlier). Q8 KV cache costs no measurable quality (held-out KL 0.0015 with Q8 and with FP16, on 6.0 bpw).
Quants
Each quant lives on its own branch; main holds only this card.
| Branch | Bits per weight | Output head | Size | KL vs BF16 | Top-1 agreement |
|---|---|---|---|---|---|
8.0bpw |
8.0 | 16-bit | 12.1 GB | 0.00016 | 99.7% |
6.0bpw |
6.0 | 8-bit | 9.3 GB | 0.0015 | 99.0% |
5.0bpw |
5.0 | 8-bit | 8.5 GB | 0.0030 | 98.6% |
4.0bpw |
4.0 | 8-bit | 7.6 GB | 0.0114 | 97.3% |
Sizes include the 0.9 GB vision tower and the 0.12 GB MTP head. KL is the mean divergence of each quant's next-token distribution from the unquantized model's, on 240 held-out conversations generated by the model (152,145 scored tokens: code, chat, German, long context, math, reasoning, tool calls). For scale: two BF16 runs differ by 0.00032, a community FP8 checkpoint by 0.0036, the official 4-bit AWQ by 0.044. 8.0 bpw is indistinguishable from BF16; 6.0 and 5.0 bpw are closer to BF16 than FP8; 4.0 bpw has about a quarter of the AWQ's KL. Math, code and tool calls lose least; German and open-ended chat lose most. Image input shows the same ranking: KL against BF16 on 200 visual questions is 0.0003 / 0.0025 / 0.0043 / 0.0145 for 8.0 / 6.0 / 5.0 / 4.0 bpw.
Benchmarks, BF16 (vLLM) against the quants (ExLlamaV3 through TabbyAPI), greedy, thinking off:
| Benchmark | BF16 | 8.0bpw | 6.0bpw | 4.0bpw |
|---|---|---|---|---|
| IFEval (prompt-level strict) | 73.8% | 73.2% | 73.6% | 71.9% |
| MATH-500 (100-problem subset) | 94.0% | 95.0% | 92.0% | 94.0% |
| MGSM, German | 86.4% | 86.8% | 86.4% | 86.4% |
| Tool calling (46 tasks) | 45/46 | 45/46 | 45/46 | 45/46 |
| Structured extraction (40 tasks) | 35/40 | 35/40 | 33/40 | 36/40 |
All differences are within the 95% confidence intervals of these sample sizes. 5.0 bpw was not benchmarked; its KL lies between 6.0 and 4.0.
Speed
RTX PRO 4500 Blackwell (32 GB), ExLlamaV3, one request:
| Branch | Context | Prefill tok/s | Decode tok/s |
|---|---|---|---|
| 8.0bpw | 8k | 1,983 | 38.2 |
| 8.0bpw | 128k | 1,428 | 28.2 |
| 6.0bpw | 8k | 2,037 | 47.6 |
| 6.0bpw | 128k | 1,473 | 33.6 |
| 4.0bpw | 8k | 2,014 | 50.3 |
| 4.0bpw | 128k | 1,497 | 38.9 |
With 8 parallel requests at 8k, total decode reaches 216 tok/s (8.0 bpw) and 261 tok/s (4.0 bpw). These numbers are without drafting.
With MTP drafting (the recipe settings, 5.0 bpw on a DGX Spark, greedy): 22 tok/s plain becomes 64 tok/s on code, 57 on math with thinking, 58 when editing a code block and 34 on prose and chat, 2.38x on average over ten tasks, with token-identical output. The RTX PRO 4500 was not measured with MTP yet. Only 8 of the 32 layers use full attention (the rest are gated delta-net), so the KV cache stays small at long context.
How to run
Install the fork (builds the CUDA extension, so it needs a CUDA toolkit matching your PyTorch), then download one branch:
pip install ninja wheel
pip install --no-build-isolation "git+https://github.com/Maxrio9/exllamav3.git@sensenova-loop-mtp" # builds against your installed PyTorch
hf download Maxrio/SenseNova-Loop-9B-exl3 --revision 6.0bpw --local-dir SenseNova-Loop-9B-exl3-6.0bpw
TabbyAPI (tested at commit 4728158): install TabbyAPI as usual, replace its ExLlamaV3 with the fork inside its environment
(pip install --no-deps --no-build-isolation --force-reinstall "git+https://github.com/Maxrio9/exllamav3.git@sensenova-loop-mtp"; with a uv-made environment use uv pip install and --reinstall), and use a config from
Pick your card (add model_dir for your folder). Send chat_template_kwargs: {"enable_thinking": true} to switch thinking on; the reasoning comes back in reasoning_content.
Python:
from exllamav3 import Config, Model, Cache, Tokenizer, Generator
from transformers import AutoTokenizer
from PIL import Image
path = "SenseNova-Loop-9B-exl3-6.0bpw"
config = Config.from_directory(path)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens = 32768, max_history = 7) # max_history: room to roll back rejected drafts
model.load()
mtp = Model.from_config(config, component = "mtp") # optional, speculative decoding
mtp_cache = Cache(mtp, max_num_tokens = 32768, max_history = 7)
mtp.load()
vision = Model.from_config(config, component = "vision") # optional, for image input
vision.load()
tokenizer = Tokenizer.from_config(config)
generator = Generator(model = model, cache = cache, tokenizer = tokenizer, draft_model = mtp, draft_cache = mtp_cache,
num_draft_tokens = 4, hybrid_draft_tokens = 7, sparse_draft_head = 16384)
hf_tok = AutoTokenizer.from_pretrained(path)
image = vision.get_image_embeddings(tokenizer = tokenizer, image = Image.open("cat.png"))
prompt = hf_tok.apply_chat_template(
[{"role": "user", "content": image.text_alias + "\nWhat is shown in this image?"}],
tokenize = False, add_generation_prompt = True, enable_thinking = False,
)
print(generator.generate(prompt = prompt, embeddings = [image], max_new_tokens = 256, encode_special_tokens = True,
stop_conditions = [248044, 248046], completion_only = True))
For text only, skip the vision lines and the embeddings argument; without drafting, skip the mtp lines and the draft arguments. For quantized KV cache in Python, pass
layer_type = CacheLayer_quant, k_bits = 8, v_bits = 8 to Cache. The base model's generation config uses temperature 1.0 and top-p 0.95.
What was checked
- All four branches: thinking on and off, German, tool calls and the answer after a tool result, multi-turn recall, retrieval at 28k and 93k, deterministic greedy output, 4 simultaneous requests, and image input (Python API and TabbyAPI).
- Text output is bit-identical with and without the vision code.
- The same held-out results on an NVIDIA GB10 (aarch64) and an RTX PRO 4500 (x86_64).
Limitations
- MTP speed was measured on a DGX Spark only; on GPUs with faster memory the speedup can differ. Bit-identical greedy output with
drafting needs
EXL3_INT8_GEMV=0and an FP32 delta-net state (the recipes set the latter); otherwise rare near-ties can resolve differently. - Long-context retrieval was checked with one five-fact prompt per recipe, not a full benchmark. Visual quality was compared against BF16 on 200 questions, not benchmarked against other models.
- At 4.0 bpw, German and open-ended chat drift most; 6.0 bpw is the safer choice for those.
License
Apache-2.0, the same as the base model. All credit for the model goes to the SenseNova team. Quantization and ExLlamaV3 port by Maxrio9.
Model tree for Maxrio/SenseNova-Loop-9B-exl3
Base model
sensenova/SenseNova-Loop-9B