a dspark model would be wonderful, even if silly
The model already runs very quickly, but its weird seeing how all lfm 2.5 models seem to have dspark drafters, including the small 2.6B, but this new one doesn't.
of course this is a little silly, since its already sich a small model, but higher speeds are still nice to have.
i agree.
LFM3-1.7B-DSpark: ⚡Up to 3.1× faster LFM3-1.7B inference with a DSpark draft model (vLLM + llama.cpp)
DSpark Model: https://huggingface.co/rasyosef/LFM3-1.7B-DSpark
GGUFs: https://huggingface.co/rasyosef/LFM3-1.7B-DSpark-GGUF
A lossless DSpark draft model for LiquidAI/LFM3-1.7B, trained with speculators (training code).
- Up to 3.14× faster on math and 2.63× on code, 2.88× on average (default generation params)
- Mean acceptance length 4.10 on math and 3.24 on code, 2.83 overall (block size 8)
Usage
vLLM
Requires vLLM from PR #66323 (Python-only changes, so precompiled kernels can be reused):
VLLM_USE_PRECOMPILED=1 uv pip install \
"vllm @ git+https://github.com/vllm-project/vllm.git@refs/pull/66323/head"
Then serve:
vllm serve LiquidAI/LFM3-1.7B --port 8000 \
--speculative-config '{"model": "rasyosef/LFM3-1.7B-DSpark", "num_speculative_tokens": 8}'
llama.cpp (GGUF)
A GGUF build of this drafter is available at rasyosef/LFM3-1.7B-DSpark-GGUF. Pair it with a LFM3-1.7B GGUF target:
llama-server -hf LiquidAI/LFM3-1.7B-GGUF:Q8_0 \
-hfd rasyosef/LFM3-1.7B-DSpark-GGUF:Q8_0 \
--spec-type draft-dspark --spec-draft-n-max 8 --spec-draft-n-min 0 \
-fa on -ngl 99
Details
- ~0.3B params, bfloat16, block size 8
- Draft vocab 32,000; aux hidden states from verifier layers 3/11/19/23
- 100,000 Open PerfectBlend prompts (filtered) with verifier-generated responses, sample length 2560, packed to 8192
- 4 epochs, lr 4e-4, cosine (4% warmup), 98/2 train/val split, loss
{"ce": 0.1, "tv": 0.9}
Evaluation
RedHatAI/speculator_benchmarks, default generation params (temperature 0.1, top_k 50, repetition_penalty 1.1 from the model's generation_config.json), single A100 40GB, concurrency 1.
Throughput (tokens/s)
Mean output throughput. Baseline is LFM3-1.7B without speculative decoding.
| subset | baseline (no drafter) | DSpark (this model) | speedup |
|---|---|---|---|
| math_reasoning | 220.8 | 692.9 | 3.14× |
| HumanEval | 221.0 | 580.2 | 2.63× |
| average | 220.9 | 636.6 | 2.88× |
Acceptance at default params
acceptance_length is mean tokens committed per verification step, including the bonus token (max 9.0 at block size 8).
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 4.101 | 78.5% | 61.1% | 47.7% | 37.4% | 29.3% | 23.3% | 18.4% | 14.5% |
| HumanEval | 3.243 | 71.4% | 49.1% | 34.4% | 24.3% | 17.4% | 12.4% | 8.9% | 6.3% |
| rag | 2.971 | 68.7% | 46.4% | 31.1% | 20.7% | 13.4% | 8.4% | 5.2% | 3.1% |
| tool_call | 2.684 | 64.4% | 39.7% | 24.9% | 15.8% | 10.2% | 6.5% | 4.1% | 2.7% |
| translation | 2.630 | 62.6% | 39.4% | 24.4% | 15.1% | 9.4% | 6.0% | 3.7% | 2.3% |
| writing | 2.586 | 61.4% | 36.7% | 22.7% | 14.7% | 9.7% | 6.3% | 4.2% | 2.9% |
| question | 2.573 | 61.6% | 36.7% | 22.4% | 14.2% | 9.4% | 6.1% | 4.2% | 2.8% |
| summarization | 2.467 | 63.1% | 37.4% | 21.4% | 12.4% | 6.7% | 3.4% | 1.6% | 0.7% |
| qa | 2.446 | 61.6% | 36.2% | 21.0% | 12.0% | 6.8% | 3.8% | 2.1% | 1.2% |
Weighted across all subsets: 2.832 over 485,518 verification steps.
Limitations
Only works with LFM3-1.7B. Throughput was measured only on math_reasoning and HumanEval. Temperature 0, concurrent load, and contexts beyond the 2,560-token training samples were not evaluated.
License
Apache 2.0