---
library_name: speculators
tags:
- speculative-decoding
- dspark
- draft-model
- vllm
base_model: thinkingmachines/Inkling-Small-NVFP4
datasets:
- orestis-z/Inkling-Small-NVFP4-Regenerated-Collection
---

# DSpark drafter for Inkling-Small-NVFP4

A [DSpark](https://github.com/neuralmagic/speculators) speculative-decoding drafter
trained for the verifier
[`thinkingmachines/Inkling-Small-NVFP4`](https://huggingface.co/thinkingmachines/Inkling-Small-NVFP4)
(multimodal MoE, NVFP4). DSpark drafts a whole block in one parallel pass
(DFlash-style backbone) and injects intra-block dependency with a lightweight
sequential Markov head, plus a per-position confidence head.

## Model details

| | |
|---|---|
| Algorithm | DSpark (greedy proposal) |
| Verifier | `thinkingmachines/Inkling-Small-NVFP4` |
| Backbone layers | 6 |
| Hidden size | 4096 |
| Attention heads / KV heads | 32 / 8 |
| Draft vocab size | 201024 (full padded vocab) |
| Block size | 16 |
| Markov rank | 256 (vanilla) |
| Confidence head | enabled (with Markov) |
| Aux hidden-state layers | 2, 10, 18, 26, 34, 39 |
| dtype | bfloat16 |

## Acceptance evaluation

Inference acceptance measured with vLLM spec-decode over 9 task subsets
(~300k drafts total). **Mean acceptance length (AL)** = expected number of
tokens accepted per draft step (higher is better; block size 16).

| Subset | Acceptance length | pos-0 | pos-1 | pos-2 | pos-3 | pos-4 | pos-5 | pos-6 | pos-7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 4.55 | 0.776 | 0.611 | 0.476 | 0.376 | 0.297 | 0.232 | 0.182 | 0.141 |
| HumanEval | 3.90 | 0.740 | 0.533 | 0.389 | 0.288 | 0.215 | 0.165 | 0.129 | 0.101 |
| rag | 3.73 | 0.736 | 0.537 | 0.394 | 0.288 | 0.212 | 0.156 | 0.113 | 0.083 |
| translation | 3.56 | 0.736 | 0.535 | 0.382 | 0.269 | 0.188 | 0.132 | 0.094 | 0.067 |
| summarization | 3.40 | 0.731 | 0.530 | 0.375 | 0.265 | 0.178 | 0.119 | 0.077 | 0.050 |
| tool_call | 3.20 | 0.707 | 0.484 | 0.328 | 0.222 | 0.148 | 0.100 | 0.067 | 0.045 |
| writing | 3.11 | 0.679 | 0.447 | 0.296 | 0.198 | 0.136 | 0.095 | 0.068 | 0.050 |
| qa | 3.10 | 0.684 | 0.464 | 0.318 | 0.216 | 0.146 | 0.098 | 0.065 | 0.043 |
| question | 3.09 | 0.679 | 0.446 | 0.294 | 0.197 | 0.136 | 0.093 | 0.067 | 0.048 |
| Average | 3.40 | 0.709 | 0.493 | 0.344 | 0.241 | 0.170 | 0.121 | 0.087 | 0.063 |

Draft-weighted average across all subsets shown in the last row.

Per-position acceptance rate, draft-weighted mean over all subsets:

![Per-position acceptance rate](./acceptance_by_position.png)

Full per-position numbers are in [`acceptance_eval.csv`](./acceptance_eval.csv).

## Usage (vLLM)

Serve the verifier with this drafter as the speculative model:

```bash
vllm serve thinkingmachines/Inkling-Small-NVFP4 \
    --speculative-config '{"model": "<this-repo>", "num_speculative_tokens": 16}' \
    --enforce-eager \
    --trust-remote-code
```

> **Required vLLM patches** (both open at time of writing):
> - [vllm-project/vllm#52184](https://github.com/vllm-project/vllm/pull/52184) —
>   adds Eagle3 hidden-state extraction support for Inkling (the verifier must
>   expose the aux hidden states this drafter consumes).
> - [vllm-project/vllm#55133](https://github.com/vllm-project/vllm/pull/55133) —
>   treats a full/padded-vocab draft (draft `vocab_size` 201024 vs. the verifier's
>   logical 200058) as an identity map, so no `d2t` mapping is required to load it.
>
> Also note: Inkling currently requires `--enforce-eager` (CUDA-graph capture of
> the custom ops crashes).

## Training

Trained with [speculators](https://github.com/neuralmagic/speculators) for 3
epochs (`checkpoint_best` = best validation epoch). Validation at the selected
checkpoint: accept_len 3.67, accept_rate 0.40, pos-0 acc 0.776. Loss
`{"ce": 0.1, "tv": 0.9}`, lr 1e-4, seq len 8192, block size 16.

**Training data:**
[`orestis-z/Inkling-Small-NVFP4-Regenerated-Collection`](https://huggingface.co/datasets/orestis-z/Inkling-Small-NVFP4-Regenerated-Collection)
— on-policy data where Inkling-Small-NVFP4 regenerates the assistant responses
(turn-by-turn, thinking effort 0.9) over Magpie + UltraChat prompts (~500k
conversations). General chat/instruct mix.

Training curves (run `dspark_inkling_small_v2`, logged with trackio):

![Training curves](./training_curves.png)

Provenance / reproducibility:
- [`train_command.txt`](./train_command.txt) — exact training command, git SHA, and
  package versions (speculators 0.7.0.dev155, vllm 0.27.1, transformers 5.15.0,
  torch 2.13.0+cu130, compressed-tensors 0.17.0).
- [`training_state.json`](./training_state.json) — epoch / global step of this checkpoint.
- [`val_metrics.json`](./val_metrics.json) — full validation metrics.