SenseNova-Loop-9B EXL3

EXL3 quantizations of sensenova/SenseNova-Loop-9B (revision 329a65f7) for ExLlamaV3. Every branch includes the original vision tower in BF16 and the model's MTP head (4-bit) for speculative decoding: about 2.4x faster generation with the same output.

Requires ExLlamaV3 with SenseNova Loop support, in review upstream (turboderp-org/exllamav3#609). Until it is released, install the fork (see How to run). Stock ExLlamaV3 and TabbyAPI do not load this model yet.

Pick your card

GPU memory Branch Context, GPU only for the model Context, GPU also runs your desktop
12 GB 4.0bpw 136k (config) 104k (config)
16 GB 5.0bpw 232k (config) 204k (config)
24 GB 6.0bpw 456k (config) 424k (config)
32 GB 8.0bpw 512k, native max (config) 512k (config)

TabbyAPI configs with MTP drafting, Q8 KV cache and the largest KV pool that fits, leaving about 0.75 GB free (plus 1 GB for the display in the desktop configs) and room to encode one image of up to 4 MP. Variants for 4 parallel requests, a calculator for your exact card (desktop, text-only, other branches) and all measurements: Maxrio9/sensenova-loop-9b-exl3-recipes. The 16 GB desktop config ran unchanged in TabbyAPI, memory capped to the card, with a 4 MP image and a 205k-token prompt that fills its context: all five planted facts found, using 14.0 of the 14.2 GB allowed (the 12 and 16 GB configs without MTP passed the same check earlier). Q8 KV cache costs no measurable quality (held-out KL 0.0015 with Q8 and with FP16, on 6.0 bpw).

Quants

Each quant lives on its own branch; main holds only this card.

Branch Bits per weight Output head Size KL vs BF16 Top-1 agreement
8.0bpw 8.0 16-bit 12.1 GB 0.00016 99.7%
6.0bpw 6.0 8-bit 9.3 GB 0.0015 99.0%
5.0bpw 5.0 8-bit 8.5 GB 0.0030 98.6%
4.0bpw 4.0 8-bit 7.6 GB 0.0114 97.3%

Sizes include the 0.9 GB vision tower and the 0.12 GB MTP head. KL is the mean divergence of each quant's next-token distribution from the unquantized model's, on 240 held-out conversations generated by the model (152,145 scored tokens: code, chat, German, long context, math, reasoning, tool calls). For scale: two BF16 runs differ by 0.00032, a community FP8 checkpoint by 0.0036, the official 4-bit AWQ by 0.044. 8.0 bpw is indistinguishable from BF16; 6.0 and 5.0 bpw are closer to BF16 than FP8; 4.0 bpw has about a quarter of the AWQ's KL. Math, code and tool calls lose least; German and open-ended chat lose most. Image input shows the same ranking: KL against BF16 on 200 visual questions is 0.0003 / 0.0025 / 0.0043 / 0.0145 for 8.0 / 6.0 / 5.0 / 4.0 bpw.

Benchmarks, BF16 (vLLM) against the quants (ExLlamaV3 through TabbyAPI), greedy, thinking off:

Benchmark BF16 8.0bpw 6.0bpw 4.0bpw
IFEval (prompt-level strict) 73.8% 73.2% 73.6% 71.9%
MATH-500 (100-problem subset) 94.0% 95.0% 92.0% 94.0%
MGSM, German 86.4% 86.8% 86.4% 86.4%
Tool calling (46 tasks) 45/46 45/46 45/46 45/46
Structured extraction (40 tasks) 35/40 35/40 33/40 36/40

All differences are within the 95% confidence intervals of these sample sizes. 5.0 bpw was not benchmarked; its KL lies between 6.0 and 4.0.

Speed

RTX PRO 4500 Blackwell (32 GB), ExLlamaV3, one request:

Branch Context Prefill tok/s Decode tok/s
8.0bpw 8k 1,983 38.2
8.0bpw 128k 1,428 28.2
6.0bpw 8k 2,037 47.6
6.0bpw 128k 1,473 33.6
4.0bpw 8k 2,014 50.3
4.0bpw 128k 1,497 38.9

With 8 parallel requests at 8k, total decode reaches 216 tok/s (8.0 bpw) and 261 tok/s (4.0 bpw). These numbers are without drafting.

With MTP drafting (the recipe settings, 5.0 bpw on a DGX Spark, greedy): 22 tok/s plain becomes 64 tok/s on code, 57 on math with thinking, 58 when editing a code block and 34 on prose and chat, 2.38x on average over ten tasks, with token-identical output. The RTX PRO 4500 was not measured with MTP yet. Only 8 of the 32 layers use full attention (the rest are gated delta-net), so the KV cache stays small at long context.

How to run

Install the fork (builds the CUDA extension, so it needs a CUDA toolkit matching your PyTorch), then download one branch:

pip install ninja wheel
pip install --no-build-isolation "git+https://github.com/Maxrio9/exllamav3.git@sensenova-loop-mtp"   # builds against your installed PyTorch
hf download Maxrio/SenseNova-Loop-9B-exl3 --revision 6.0bpw --local-dir SenseNova-Loop-9B-exl3-6.0bpw

TabbyAPI (tested at commit 4728158): install TabbyAPI as usual, replace its ExLlamaV3 with the fork inside its environment (pip install --no-deps --no-build-isolation --force-reinstall "git+https://github.com/Maxrio9/exllamav3.git@sensenova-loop-mtp"; with a uv-made environment use uv pip install and --reinstall), and use a config from Pick your card (add model_dir for your folder). Send chat_template_kwargs: {"enable_thinking": true} to switch thinking on; the reasoning comes back in reasoning_content.

Python:

from exllamav3 import Config, Model, Cache, Tokenizer, Generator
from transformers import AutoTokenizer
from PIL import Image

path = "SenseNova-Loop-9B-exl3-6.0bpw"
config = Config.from_directory(path)
model = Model.from_config(config)
cache = Cache(model, max_num_tokens = 32768, max_history = 7)    # max_history: room to roll back rejected drafts
model.load()
mtp = Model.from_config(config, component = "mtp")                # optional, speculative decoding
mtp_cache = Cache(mtp, max_num_tokens = 32768, max_history = 7)
mtp.load()
vision = Model.from_config(config, component = "vision")   # optional, for image input
vision.load()
tokenizer = Tokenizer.from_config(config)
generator = Generator(model = model, cache = cache, tokenizer = tokenizer, draft_model = mtp, draft_cache = mtp_cache,
                      num_draft_tokens = 4, hybrid_draft_tokens = 7, sparse_draft_head = 16384)
hf_tok = AutoTokenizer.from_pretrained(path)

image = vision.get_image_embeddings(tokenizer = tokenizer, image = Image.open("cat.png"))
prompt = hf_tok.apply_chat_template(
    [{"role": "user", "content": image.text_alias + "\nWhat is shown in this image?"}],
    tokenize = False, add_generation_prompt = True, enable_thinking = False,
)
print(generator.generate(prompt = prompt, embeddings = [image], max_new_tokens = 256, encode_special_tokens = True,
                         stop_conditions = [248044, 248046], completion_only = True))

For text only, skip the vision lines and the embeddings argument; without drafting, skip the mtp lines and the draft arguments. For quantized KV cache in Python, pass layer_type = CacheLayer_quant, k_bits = 8, v_bits = 8 to Cache. The base model's generation config uses temperature 1.0 and top-p 0.95.

What was checked

  • All four branches: thinking on and off, German, tool calls and the answer after a tool result, multi-turn recall, retrieval at 28k and 93k, deterministic greedy output, 4 simultaneous requests, and image input (Python API and TabbyAPI).
  • Text output is bit-identical with and without the vision code.
  • The same held-out results on an NVIDIA GB10 (aarch64) and an RTX PRO 4500 (x86_64).

Limitations

  • MTP speed was measured on a DGX Spark only; on GPUs with faster memory the speedup can differ. Bit-identical greedy output with drafting needs EXL3_INT8_GEMV=0 and an FP32 delta-net state (the recipes set the latter); otherwise rare near-ties can resolve differently.
  • Long-context retrieval was checked with one five-fact prompt per recipe, not a full benchmark. Visual quality was compared against BF16 on 200 questions, not benchmarked against other models.
  • At 4.0 bpw, German and open-ended chat drift most; 6.0 bpw is the safer choice for those.

License

Apache-2.0, the same as the base model. All credit for the model goes to the SenseNova team. Quantization and ExLlamaV3 port by Maxrio9.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for Maxrio/SenseNova-Loop-9B-exl3

Quantized
(44)
this model