README.md

---
license: apache-2.0
library_name: vllm
inference: false
base_model: CohereLabs/North-Mini-Code-1.0-w4a16
tags:
- north-mini-code
- speculative-decoding
- draft-model
- dflash
- code
---

# North Mini Code 1.0 DFlash

This is a research-preview **auxiliary draft checkpoint** for speculative decoding with `CohereLabs/North-Mini-Code-1.0-w4a16`.
It is **not** a standalone language model and it cannot generate text without the North verifier/target model.
It will likely also work with `CohereLabs/North-Mini-Code-1.0-fp8` with slightly lower acceptance rates.

## Training

This DFlash draft model was trained on 10,000 Magicoder prompts with on-policy supervision from North Mini Code.

## Important Notes

- not a base model
- not a fine-tuning target for standalone inference
- not a correctness guarantee
- not byte-parity with target-only greedy decoding

## Architecture

- speculator type: `DFlashDraftModel`
- draft layers: 5 Qwen3 sliding-window layers
- hidden size: 2048
- attention heads: 32
- key/value heads: 4
- head dim: 128
- MLP intermediate size: 768
- sliding window: 2048
- draft vocabulary: 32,000
- target vocabulary: 262,144
- block size: 8
- `sample_from_anchor`: `false`
- auxiliary target hidden-state taps: layers `[2, 24, 46]`
- tensors: 62
- serialized tensor elements: 733,274,624
- BF16 parameters: 732,980,480
- selected weight hash: `0ac5964c48003f2eab08c4691357dedc11227d88302ea7c76fc47aaddab6d5b9`

Largest tensors include the target embedding matrix, `lm_head.weight`, and the draft `fc.weight` projection.
The tensor inventory is recorded in `tensor_info.json`.

## Training data and supervision

- prompt dataset: `ise-uiuc/Magicoder-Evol-Instruct-110K`
- license: Apache-2.0
- training rows: 10,000
- selected rows: 0-499, 600-5099, and 6100-11099
- locked holdout: rows 500-599, never trained
- supervision: on-policy responses and target hidden states generated once from `CohereLabs/North-Mini-Code-1.0-w4a16`
- response cap: 2,048 tokens
- target layers used for supervision: `[2, 24, 46]`
- later acceptance-mined continuation data: not included in this release checkpoint

Training metadata is recorded in `training_metadata.json`.

## Matched GB10 benchmarks

These numbers come from the same matched serving contract on a GB10 (DGX Spark) machine running Marlin (which is suboptimal but predictable).
Compare the rows **within the same host and contract only**.
These absolute token/s values are not representative of maximum performance, simply a demonstration that the draft model works. :)

| Contract | Target-only | DFlash K3 | DFlash / target | DSpark K4 | DSpark / target |
| --- | ---: | ---: | ---: | ---: | ---: |
| C1 | 39.9129 tok/s | 75.5106 tok/s | 1.8919x | 79.7341 tok/s | 1.9977x |
| C2 | 78.9486 tok/s | 121.5232 tok/s | 1.5393x | 127.6174 tok/s | 1.6165x |

| Model | C1 EAL | C1 draft acceptance | C2 EAL | C2 draft acceptance |
| --- | ---: | ---: | ---: | ---: |
| DFlash K3 | 2.2947 | 43.16% | 2.2856 | 42.85% |
| DSpark K4 | 2.4800 | 37.00% | 2.4779 | 36.95% |

Expected acceptance length (EAL) is the mean emitted tokens per speculative verification step, including the target bonus token.

## Additional result: NVIDIA RTX 3090

DFlash was also tested on an NVIDIA RTX 3090 under the same family of matched conditions.
That result is specific to DFlash; do not infer DSpark RTX 3090 portability from it.

- target-only: 110.2714 tok/s
- DFlash K3: 153.7188 tok/s
- official Eagle K3: 77.9583 tok/s

Also being realistic, you wouldn't try to serve this on a single 3090 anyway the model weights leave no room for context.

## Default proposal depth

Default proposal depth is **K3**.
That is the selected deployment setting in `config.json` and the configuration used for the release evidence.
More speculative tokens hit diminishing returns past position 3 for this DFlash drafter. You're likely to see the best speed here.

## Runtime boundary

This checkpoint is served as the **top-level model** because `config.json` already embeds the verifier reference.
Use it only with the companion North runtime, published at https://github.com/sdougbrown/north-mini-code-draft-runtime; this card does not describe a separate target-model `--speculative_config` deployment. The companion is a thin overlay on official vLLM 0.27.1 carrying PRs #49819 and #50937 (Cohere2MoE auxiliary hidden states, and an expert-bias loading fix), published as `ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937`.

```bash
export DRAFT_MODEL="${DRAFT_MODEL:?set this to the downloaded DFlash directory or Hub model ID}"
export VLLM_USE_V2_MODEL_RUNNER=1
vllm serve "${DRAFT_MODEL}" \
  --tensor-parallel-size 1 \
  --max-model-len 320000 \
  --tool-call-parser cohere_command4 \
  --tokenizer-mode cohere \
  --cohere-format cmd4 \
  --reasoning-config '{"reasoning_start_str":"<|START_THINKING|>","reasoning_end_str":"<|END_THINKING|>"}' \
  --enable-auto-tool-choice
```

## Limitations

- 32K draft vocabulary, not the 262,144-token target vocabulary
- target-dependent: cannot run standalone
- quantized greedy outputs are not guaranteed to match target-only greedy byte-for-byte
- throughput and acceptance do not establish correctness or response quality
- absolute throughput is host-specific

## References

- North base model: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-w4a16
- Magicoder dataset: https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K
- Companion North runtime: https://github.com/sdougbrown/north-mini-code-draft-runtime
- Companion image: `ghcr.io/sdougbrown/north-mini-code-runtime:v0.27.1-49819-50937`
- Official Eagle card: https://huggingface.co/CohereLabs/North-Mini-Code-1.0-eagle
- Speculators: pinned commit `f7ec34182826bc89934ce710283421778022b74d`, version `0.6.0`

## Files in this repo

- `README.md`
- `config.json`
- `config.py`
- `model.safetensors`
- `tensor_info.json`
- `training_metadata.json`
- `SHA256SUMS`
- `LICENSE`
- `.gitattributes`