dense-same: Qwen3-8B DSpark dense drafter, up to 24 epochs
Repo yushengsu/Qwen3-8B-5layer-Dense (renamed on 2026-11-04 from yushengsu/dense-same; naming scheme <base model>-<layers>layer-<Dense|MoE>[-<E>E<A>A], -vN = N-th run of that architecture by repo creation date).
Draft model for Qwen/Qwen3-8B trained with SpecForge
(branch dense-same): DSpark dense drafter, block size 16, no confidence head, 5 layers, hidden 4096, FFN 12288,
hyperparameters of the published Qwen3-8B dense DSpark run (global batch 128, AdamW lr 6e-4 cosine, 4% warm-up,
0.1 CE + 0.9 L1), 12 epochs on Open Perfect Blend regenerated by Qwen3-8B at temperature 0 (1,420,909 rows),
then weights-only continuations logged on the same W&B curve (AdamW reset each time): epoch 13 from the epoch-12
checkpoint (1% warm-up to 2e-4, 6-epoch cosine, prompt seed 54; that segment crashed in epoch 14), epochs 14-18 from
the epoch-13 checkpoint (1% warm-up to 1.88e-4 = the planned cosine value there, cosine to 0 over 5 epochs, prompt
seed 55), and epochs 19-24 from the epoch-18 checkpoint (1% warm-up to 2e-4, cosine to 0 over 6 epochs, prompt seed 60).
Each folder <k>ep/ is the servable export of the checkpoint at the end of epoch k. Full parameters and log:
exp_read.me.
| epoch | end step | train/acc (last 100 steps) | acc peak | train/ce_loss (last 100 steps) | HF checkpoint | uploaded (UTC) |
|---|---|---|---|---|---|---|
| 1 | 11100 | 0.6989 | 0.7342 | 1.284 | 1ep | 2026-10-16 08:29 |
| 2 | 22200 | 0.7336 | 0.7788 | 1.092 | 2ep | 2026-10-16 17:05 |
| 3 | 33300 | 0.7616 | 0.806 | 0.956 | 3ep | 2026-10-17 02:13 |
| 4 | 44400 | 0.7718 | 0.827 | 0.919 | 4ep | 2026-10-17 11:19 |
| 5 | 55500 | 0.774 | 0.832 | 0.899 | 5ep | 2026-10-17 20:26 |
| 6 | 66600 | 0.8035 | 0.8587 | 0.778 | 6ep | 2026-10-18 05:34 |
| 7 | 77700 | 0.803 | 0.8537 | 0.779 | 7ep | 2026-10-18 14:40 |
| 8 | 88800 | 0.8133 | 0.8738 | 0.746 | 8ep | 2026-10-18 23:49 |
| 9 | 99900 | 0.8271 | 0.8681 | 0.692 | 9ep | 2026-10-19 08:56 |
| 10 | 111000 | 0.8301 | 0.8847 | 0.685 | 10ep | 2026-10-19 18:04 |
| 11 | 122100 | 0.84 | 0.8863 | 0.651 | 11ep | 2026-10-20 03:12 |
| 12 | 133209 | 0.8313 | 0.8832 | 0.687 | 12ep | 2026-10-20 12:42 |
| 13 | 144300 | 0.8264 | 0.8802 | 0.691 | 13ep | 2026-10-20 22:01 |
| 14 | 155400 | 0.8216 | 0.8803 | 0.716 | 14ep | 2026-10-21 14:08 |
| 15 | 166500 | 0.8383 | 0.8828 | 0.649 | 15ep | 2026-10-21 23:14 |
| 16 | - | - | - | - | - | - |
| 17 | - | - | - | - | - | - |
| 18 | - | - | - | - | - | - |
| 19 | - | - | - | - | - | - |
| 20 | - | - | - | - | - | - |
| 21 | - | - | - | - | - | - |
| 22 | - | - | - | - | - | - |
| 23 | - | - | - | - | - | - |
| 24 | - | - | - | - | - | - |
Highest train/acc so far: 0.8863 (epoch 11, peak value).