darkmaniac7/TokForge-AccelerationPack-Draft

TokForge Acceleration Pack Qwen3 0.6B draft (MNN)

A small Qwen3 0.6B draft model, KL-distilled from Qwen3 8B, that TokForge runs next to a Qwen3 chat model for speculative decoding.

Published by TokForge for on-device inference.

What it is	Draft model for speculative decoding (KL-distilled)
Format	MNN bundle, 4-bit weights (block 128)
Runs on	Android phones (CPU), next to a Qwen3 MNN chat model
Phone class	4 GB RAM or more (TokForge helper module)
Download	339 MB (10 files)
Base model	Qwen/Qwen3-0.6B
License	Apache 2.0

TokForge

• Website: https://tokforge.ai
• Discord: https://discord.gg/Acv3CBtfVm
• Google Play: https://play.google.com/store/apps/details?id=dev.tokforge
• iOS TestFlight: https://testflight.apple.com/join/jnufjzRr

Runs on-device in the TokForge app.

Use it in TokForge

1. Install TokForge from Google Play.
2. On Android, open Settings > Speed & engine > Models & Modules and download Acceleration Pack: Qwen3-0.6B 40k (the app's name for it; this repo holds the 10,000-sample v3 draft).
3. TokForge uses it as the draft for speculative decoding with the Qwen3 chat models below when speculative decoding is on for that model and phone. The draft runs on the CPU and guesses a few tokens ahead (3 by default; TokForge tunes this per phone).
4. Load a Qwen3 MNN chat model (see the list below) and chat as usual. The draft guesses the next tokens and the chat model checks every guess.

Collections: TokForge Draft Models.

Files

File	Size	What it is
config.json	211 B	MNN runtime settings
config_cpu.json	211 B	Draft runtime settings (CPU, 2 threads, greedy)
config_opencl.json	172 B	Draft runtime settings for OpenCL (reference)
draft_config_cpu.json	211 B	Draft runtime settings (same as config_cpu.json)
llm.mnn	504 KB	Model graph
llm.mnn.json	1.01 MB	Model graph as JSON (reference copy)
llm.mnn.weight	336 MB	Model weights
llm_config.json	4.66 KB	Model settings and chat template
runtime_config.json	1.36 KB	Older recommended speculative decoding settings (reference)
tokenizer.txt	1.61 MB	Tokenizer

TokForge downloads llm.mnn, llm.mnn.weight, llm_config.json, config.json, config_cpu.json, draft_config_cpu.json and tokenizer.txt; the other files are for reference.

Which models it pairs with

Chat model in TokForge	Uses this draft
Qwen3 4B and Qwen3 8B (MNN)	Yes
Qwen3 14B (MNN)	Yes, unless Acceleration Pack: Qwen3-0.6B 14B-paired is downloaded; then the 14B uses that one
Qwen3 1.7B (MNN)	On some chips only
Qwen3 30B A3B (MNN)	No, TokForge keeps speculative decoding off for it
Qwen3.5 models	No, they use a different architecture (linear attention) and vocabulary

This bundle is for TokForge on Android. TokForge for iOS does not download it.

About the model

A 0.6B draft model for speculative decoding. The draft runs on the CPU next to your main model and predicts a few tokens ahead; the main model checks them in one batch and keeps the ones it agrees with. KL distillation trains the draft on the teacher's full logit distribution, not only its top-1 token, so it learns which tokens the teacher finds likely. MNN's greedy sampler needs the draft's top-1 token to match the teacher's top-1, and KL training aims at exactly that.

Architecture	Qwen3 (dense, full attention)
Parameters	0.6B (4-bit quantized)
Quantization	W4A16 (4-bit weights, block size 128)
Teacher	Qwen/Qwen3-8B

Recommended settings

Best with: target OpenCL, draft CPU, d=3, thread_num=2, power=high.

Setting	Value
Target backend	OpenCL
Draft backend	CPU
Draft threads	2
Draft predict length (d)	3
Sampler	Greedy
Precision / memory	Low / low

config_cpu.json:

{
    "backend_type": "cpu",
    "thread_num": 2,
    "precision": "low",
    "memory": "low",
    "sampler_type": "greedy",
    "power": "high"
}

Why thread_num: 2? On Android's WALT CPU governor, using too many threads (4+) can cause the scheduler to spread work across efficiency cores at low frequency. 2 threads stay on performance cores at high clock speeds.

How it was made

KL-distilled from Qwen/Qwen3-0.6B using Qwen/Qwen3-8B as teacher.

• Teacher: Qwen3-8B-HF (base)
• Student: Qwen3-0.6B-HF (base, LoRA r=16, alpha=32)
• Data: 10,000 teacher-generated samples (prose, code, Q&A)
• Loss: 80% KL divergence + 20% cross-entropy, temperature=1.5
• Training: 3 epochs, batch=4, grad_accum=4 (1,875 optimizer steps)
• Final KL: 0.339
• Hardware: 2x NVIDIA RTX PRO 6000 Blackwell (teacher GPU 0, student GPU 1)
• Export: MNN Q4 quantization (quant_bit=4, quant_block=128)

These are the v3 weights (KL distillation on 10,000 samples), uploaded on 2026-03-22.

Limitations

• Results are per device and per workload: backend routing, prompt length, and thermal state all change the outcome, and some device and target pairings do not benefit.
• Best current evidence is strongest on Qwen3-8B and Qwen3-14B.
• Works only with Qwen3 chat models, which share its tokenizer.
• A draft model is a helper for a bigger model, not a chat model on its own.
• This is a TokForge-specific runtime bundle, not a standard Transformers checkpoint.

License

License: Apache 2.0, the license of Qwen3 0.6B and of the Qwen3 8B teacher (Apache License 2.0).

Credits

• Qwen3-0.6B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license.
• Qwen3-8B by the Qwen team (Alibaba Cloud) (Hugging Face). Used under the Apache-2.0 license. Used as the distillation teacher.
• Converted with llmexport from MNN by Alibaba (Apache-2.0).