multimodalart/laguna-s-2.1-DFlash

Laguna S 2.1 DFlash drafter — Q8_0 GGUF (+ Apple Silicon recipe)

Q8_0 quantization of the Laguna S 2.1 DFlash speculative-decoding drafter from poolside/Laguna-S-2.1-GGUF (post the metadata fix in discussion #12), for llama.cpp (poolside fork, branch laguna).

• laguna-s-2.1-DFlash-Q8_0.gguf — 1.1 GB, half the weight reads per draft round vs BF16. Drafter quantization cannot affect output quality (all drafts are verified by the target model); measured acceptance is identical to BF16 (77.8% on a 7K-context coding A/B).

History

The originally shipped drafter GGUF was missing dflash.attention.sliding_window — the drafter ran full attention while trained with a 512 sliding window, so draft acceptance collapsed with context length (27% at 767 tokens → 1.2% at 7K), making DFlash a net slowdown on long prompts. Upstream fixed this in discussion #12; this repo's Q8_0 is quantized from the fixed file. (Note: the GGUF's rope.freq_base = 500000 differs from Laguna-S-2.1-DFlash-NVFP4's rope_theta = 10000, but these are different drafter checkpoints — the weights don't match — each quantization-matched to its target and internally consistent. A/B on the GGUF pairing confirms 500000 is its trained value: 77.8% vs 71.6% acceptance.)

Usage (Apple Silicon recipe)

Verify batches on this fine-grained MoE are expensive on memory-bound hardware, so shallow adaptive drafting wins — measured ~1.25× over base decoding on an M4 Max 128GB at ~10K context:

./build/bin/llama-server -m laguna-s-2.1-Q4_K_M.gguf \
  -md laguna-s-2.1-DFlash-Q8_0.gguf \
  --spec-type draft-dflash --spec-draft-n-max 2 --spec-draft-p-min 0.5 \
  -fa on --jinja -c 262144 -ub 4096 --cache-reuse 256 --port 8000

On CUDA-class hardware use the model card's --spec-draft-n-max 15 instead.

Related llama.cpp work (branch apolinario/llama.cpp@laguna, partially upstreamed via poolsideai/llama.cpp#2): Metal f16-overflow sanitization of DFlash target features and two Metal decode optimizations (~+10%).