Qwen3-8B-2layer-Dense-v2: DSpark dense-1/8 draft for Qwen3-8B, 5-epoch schedule

Repo yushengsu/Qwen3-8B-2layer-Dense-v2 (renamed on 2026-11-04 from yushengsu/dense-1-8-only-5-epoch (briefly yushengsu/Qwen3-8B-2layer-Dense); naming scheme <base model>-<layers>layer-<Dense|MoE>[-<E>E<A>A], -vN = N-th run of that architecture by repo creation date).

A DSpark draft model for Qwen/Qwen3-8B, trained with the code of SpecForge PR #1003 (commit cde2f059) with the hyperparameters of yushengsu/dense-1-8 on a 5-epoch cosine schedule (run dense-1-8-only-5-epoch). One folder per epoch, each directly servable by SGLang 0.5.18 (--speculative-algorithm DSPARK --speculative-draft-model-path <folder>).

Folder	Optimizer step	Last logged training metrics
1ep/	10,764	step 10760: acc 0.6032, ce_loss 1.7603, loss 0.6947
2ep/	21,528	step 21520: acc 0.6441, ce_loss 1.5688, loss 0.6937
3ep/	32,292	step 32290: acc 0.6582, ce_loss 1.4500, loss 0.7237
4ep/	43,056	step 43050: acc 0.6714, ce_loss 1.3807, loss 0.5948
5ep/	53,820	step 53820: acc 0.6624, ce_loss 1.4405, loss 0.5785

Architecture

2 decoder layers (all full attention, causal inside the draft block), residual hidden 4096, GQA 8 heads / 2 KV heads (head_dim 128), one SwiGLU MLP of width 1536 per layer, DSpark projector over target layers 1/9/17/25/33, vanilla Markov head (rank 256), block size 16, no confidence head, target embedding and lm_head.

Training

• Data: Open Perfect Blend prompts with Qwen3-8B responses (temperature 0, thinking off), 1,420,909 rows, qwen3-instruct template without a system prompt, max length 8192.
• Online DSpark training (2 SGLang capture servers + 6 DDP trainer ranks on one 8x H200 node), global batch 132 (2 x 11 x 6), AdamW, lr 6e-4, cosine over 5 epochs (53,820 steps) with 4% warmup, max_grad_norm 1.0, loss 0.1 CE + 0.9 L1, 512 anchors, loss_decay_gamma 7.0, bf16 without Liger kernels, seed / prompt_seed 42.
• Exported with specforge export --to hf (frozen Qwen3-8B embedding) and normalized for SGLang (scripts/gates/normalize_dflash_export.py --block-size 16).