license: apache-2.0
base_model: Qwen/Qwen3-8B
tags:
- speculative-decoding
- dspark
- specforge
- draft-model
dense-1/8: a small Qwen3-8B DSpark dense drafter (2 layers, 1/4 attention width, 1/8 MLP)
Repo yushengsu/Qwen3-8B-2layer-Dense-v1 (renamed on 2026-11-04 from yushengsu/dense-1-8; naming scheme <base model>-<layers>layer-<Dense|MoE>[-<E>E<A>A], -vN = N-th run of that architecture by repo creation date).
Per-epoch checkpoints of one SpecForge run: the Qwen3-8B dense DSpark drafter
(causal draft block, block 16, no confidence head) shrunk in structure only:
2 layers instead of 5, attention GQA 8/2 heads x 128 (1/4 of the dense arm's
width), MLP intermediate_size 1536 (1/8 of 12288); residual hidden stays
4096. About 0.22B parameters. Trained on the Open Perfect Blend corpus
regenerated by Qwen3-8B (yaro1214/open_data), rendered with the
qwen3-instruct template and no system prompt.
Epochs 1-12: lr 6e-4, 4% warmup, cosine to 0 over 12 epochs.
Epochs 13-18: weights-only warm start from epoch 12, 1% warmup to 2e-4, cosine to 0 over 6 epochs, new prompt order.
Code, config and the per-epoch results table: https://github.com/yushengsu-thu/SpecForge/blob/dense-1/8/exp_read.me
Training curve (one W&B line for all 18 epochs): https://wandb.ai/eigent_radixark_training/specforge-qwen3-8b-dspark-dense/runs/dense-1-8-r8
Layout: epoch-01/ ... epoch-18/, each a specforge exported servable
directory (epoch k = global step 10764 x k; global batch 132 on 6 DDP ranks).
Folders appear as epochs finish.