| --- |
| library_name: speculators |
| tags: |
| - speculative-decoding |
| - dspark |
| - draft-model |
| - vllm |
| base_model: thinkingmachines/Inkling-Small-NVFP4 |
| datasets: |
| - orestis-z/Inkling-Small-NVFP4-Regenerated-Collection |
| --- |
| |
| # DSpark drafter for Inkling-Small-NVFP4 |
|
|
| A [DSpark](https://github.com/neuralmagic/speculators) speculative-decoding drafter |
| trained for the verifier |
| [`thinkingmachines/Inkling-Small-NVFP4`](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4) |
| (multimodal MoE, NVFP4). DSpark drafts a whole block in one parallel pass |
| (DFlash-style backbone) and injects intra-block dependency with a lightweight |
| sequential Markov head, plus a per-position confidence head. |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Algorithm | DSpark (greedy proposal) | |
| | Verifier | `thinkingmachines/Inkling-Small-NVFP4` | |
| | Backbone layers | 6 | |
| | Hidden size | 4096 | |
| | Attention heads / KV heads | 32 / 8 | |
| | Draft vocab size | 201024 (full padded vocab) | |
| | Block size | 16 | |
| | Markov rank | 256 (vanilla) | |
| | Confidence head | enabled (with Markov) | |
| | Aux hidden-state layers | 2, 10, 18, 26, 34, 39 | |
| | dtype | bfloat16 | |
|
|
| ## Acceptance evaluation |
|
|
| Inference acceptance measured with vLLM spec-decode over 9 task subsets |
| (~300k drafts total). **Mean acceptance length (AL)** = expected number of |
| tokens accepted per draft step (higher is better; block size 16). |
|
|
| | Subset | Acceptance length | pos-0 | pos-1 | pos-2 | pos-3 | pos-4 | pos-5 | pos-6 | pos-7 | |
| |---|---|---|---|---|---|---|---|---|---| |
| | math_reasoning | 4.55 | 0.776 | 0.611 | 0.476 | 0.376 | 0.297 | 0.232 | 0.182 | 0.141 | |
| | HumanEval | 3.90 | 0.740 | 0.533 | 0.389 | 0.288 | 0.215 | 0.165 | 0.129 | 0.101 | |
| | rag | 3.73 | 0.736 | 0.537 | 0.394 | 0.288 | 0.212 | 0.156 | 0.113 | 0.083 | |
| | translation | 3.56 | 0.736 | 0.535 | 0.382 | 0.269 | 0.188 | 0.132 | 0.094 | 0.067 | |
| | summarization | 3.40 | 0.731 | 0.530 | 0.375 | 0.265 | 0.178 | 0.119 | 0.077 | 0.050 | |
| | tool_call | 3.20 | 0.707 | 0.484 | 0.328 | 0.222 | 0.148 | 0.100 | 0.067 | 0.045 | |
| | writing | 3.11 | 0.679 | 0.447 | 0.296 | 0.198 | 0.136 | 0.095 | 0.068 | 0.050 | |
| | qa | 3.10 | 0.684 | 0.464 | 0.318 | 0.216 | 0.146 | 0.098 | 0.065 | 0.043 | |
| | question | 3.09 | 0.679 | 0.446 | 0.294 | 0.197 | 0.136 | 0.093 | 0.067 | 0.048 | |
| | Average | 3.40 | 0.709 | 0.493 | 0.344 | 0.241 | 0.170 | 0.121 | 0.087 | 0.063 | |
|
|
| Draft-weighted average across all subsets shown in the last row. |
|
|
| Per-position acceptance rate, draft-weighted mean over all subsets: |
|
|
|  |
|
|
| Full per-position numbers are in [`acceptance_eval.csv`](./acceptance_eval.csv). |
|
|
| ## Usage (vLLM) |
|
|
| Serve the verifier with this drafter as the speculative model: |
|
|
| ```bash |
| vllm serve thinkingmachines/Inkling-Small-NVFP4 \ |
| --speculative-config '{"model": "<this-repo>", "num_speculative_tokens": 16}' \ |
| --enforce-eager \ |
| --trust-remote-code |
| ``` |
|
|
| > **Required vLLM patches** (both open at time of writing): |
| > - [vllm-project/vllm#52184](https://github.com/vllm-project/vllm/pull/52184) — |
| > adds Eagle3 hidden-state extraction support for Inkling (the verifier must |
| > expose the aux hidden states this drafter consumes). |
| > - [vllm-project/vllm#55133](https://github.com/vllm-project/vllm/pull/55133) — |
| > treats a full/padded-vocab draft (draft `vocab_size` 201024 vs. the verifier's |
| > logical 200058) as an identity map, so no `d2t` mapping is required to load it. |
| > |
| > Also note: Inkling currently requires `--enforce-eager` (CUDA-graph capture of |
| > the custom ops crashes). |
| |
| ## Training |
| |
| Trained with [speculators](https://github.com/neuralmagic/speculators) for 3 |
| epochs (`checkpoint_best` = best validation epoch). Validation at the selected |
| checkpoint: accept_len 3.67, accept_rate 0.40, pos-0 acc 0.776. Loss |
| `{"ce": 0.1, "tv": 0.9}`, lr 1e-4, seq len 8192, block size 16. |
|
|
| **Training data:** |
| [`orestis-z/Inkling-Small-NVFP4-Regenerated-Collection`](https://huggingface.co/datasets/orestis-z/Inkling-Small-NVFP4-Regenerated-Collection) |
| — on-policy data where Inkling-Small-NVFP4 regenerates the assistant responses |
| (turn-by-turn, thinking effort 0.9) over Magpie + UltraChat prompts (~500k |
| conversations). General chat/instruct mix. |
|
|
| Training curves (run `dspark_inkling_small_v2`, logged with trackio): |
|
|
|  |
|
|
| Provenance / reproducibility: |
| - [`train_command.txt`](./train_command.txt) — exact training command, git SHA, and |
| package versions (speculators 0.7.0.dev155, vllm 0.27.1, transformers 5.15.0, |
| torch 2.13.0+cu130, compressed-tensors 0.17.0). |
| - [`training_state.json`](./training_state.json) — epoch / global step of this checkpoint. |
| - [`val_metrics.json`](./val_metrics.json) — full validation metrics. |
|
|
