README.md

# MoS-27B-Checkpoints

Epoch checkpoints of the Qwen3.8-27B DFlash2 MoS line (project MoS, branch `ryan/mos-improve`,
`experiments/dflash2/qwen3.8-27b-100k/`). Started 2026-09-22, when the general archive repo
`ryan-0608/MoS-Aurora-Experiment-Archive` reached the Hugging Face 20,000-file limit.

## Layout

```
dflash2_27b_traj_20260915/main-800k/<arm>/epoch<N>/
    model.safetensors            drafter weights (speculators DFlash2MoSDraftModel)
    optimizer_state_dict.pt      full AdamW state -> exact resume
    scheduler_state_dict.pt, training_state.json, config.json, config.py, train_command.txt
    val_metrics.json             trainer held-out 10% validation at the end of the epoch
    serving/                     fixed-500 serving result for this checkpoint
        acceptance_summary.json  pooled AL = sum completion tokens / sum verify calls
        acceptance_trace.jsonl   per-request counts
        client.log, server.log
```

Since 2026-09-30 this repo holds the only copies of the `main-800k` epoch weights (the PVC copies of x4rand E3/E4 and
dense4 E1/E3/E4 were deleted after presence + byte-size checks here).

Also in this repo (2026-09-30, PVC cleanup; the PVC copies of the large files were deleted after presence + byte-size
checks here):

```
dflash2_27b_traj_20260915/inputs/<init>/          config.json, model.safetensors, build-*.log
dflash2_27b_traj_20260915/data/current-800k-self27b-traj/
    train.jsonl        cleaned 800K self-distilled corpus, 799,395 rows (8,926,961,996 B)
    traj.jsonl         the same rows as pre-built input_ids / loss_mask (19,323,325,279 B); the 800K runs were
                       prepared from it (speculators prepare_data, 80 Arrow shards)
    manifest.json, public-overlap-27b-800k.json, evidence-archived-20260915/
```

| init | what | used by |
|---|---|---|
| `dflash2-27b-zlab-x8rand-b16` | z-lab import + K=8 random experts (gate/up N(0,0.02), down N(0,1e-3), seed 20260917) | 300-step K8 memory/speed probe 2026-09-29; the K8 4-epoch run was dropped |
| `dflash2-27b-zlab-x4rand-sel1024-b16` | x4rand init with the candidate selector widened 256 -> 1024 (new successor columns 0, so step-0 scores are unchanged; seed 20260929) | selector-rank smoke 2026-09-29/30: path AL +0.09% vs x4rand at steps 13k-18k, not pursued |

## Arms

| arm (path) | Ryan's name | recipe |
|---|---|---|
| `dense4` | 27B dense (4-epoch baseline) | dense drafter warm-started from `z-lab/Qwen3.8-27B-DFlash2` @50307d4c; base LR 1e-4; cosine over 4 epochs, warmup 0.005; same corpus, 3 verifier + 5 trainer layout, global batch and 73,670 steps per epoch as `x4rand`; finished 2026-09-29 |
| `x4rand` | 27B 4expert rand LR6 | K=4 full-width expert MLPs per draft layer, random init (gate/up N(0,0.02), down N(0,1e-3)); shared MLP, attention and the rest warm-started from `z-lab/Qwen3.8-27B-DFlash2` @50307d4c; base LR 1e-4, expert LR 6e-4, router LR 5e-4; top-2, tau 0.9, balance 0.01; cosine over 4 epochs, warmup 0.005; corpus Current-800K-Self-27B-Traj (719,455 train rows); 73,670 steps per epoch |

## Results (target Qwen/Qwen3.8-27B @1d4bf0f2)

Serving protocol: sglang main f5866545, TP2, DFLASH 16 draft tokens, fixed-500 prompts, max 128 new tokens,
greedy, concurrency 1, 500/500 completed. Same protocol for every arm.

| arm | epoch | global_step | train-side val AL | serving pooled AL | vs dense4 same epoch |
|---|---|---|---|---|---|
| dense4 | 1 | 73,670 | 5.0262 | 5.0571 | — |
| dense4 | 2 | 147,335 | 5.0261 | 5.0510 | — |
| dense4 | 3 | 221,001 | 5.026 (from train.log) | 5.0558 | — |
| dense4 | 4 | 294,680 | 5.024 (from train.log) | 5.0538 | — |
| x4rand | 1 | 73,670 | 5.1485 | 5.2314 | +3.45% (dense4 5.0571) |
| x4rand | 2 | 147,335 | 5.2056 | 5.2922 | +4.78% (dense4 5.0510) |
| x4rand | 3 | 221,001 | 5.2202 | 5.3261 | +5.34% (dense4 5.0558) |
| x4rand | 4 | 294,682 | 5.2201 | 5.3113 | +5.09% (dense4 5.0538); best epoch = 3 |

The K4 jitter arm (shared MLP copies + 1% jitter, expert LR 1e-4; E1 only, serving 5.1501, +1.84% vs dense4 E1)
is model-only in `ryan-0608/MoS-Aurora-Experiment-Archive` under `dflash2_27b_traj_20260915/main-800k/mos/epoch1`.

## Public sets (max tokens 512, c=1, TP2, pooled AL)

x4rand E3 against dense4 E1 (each arm's best fixed-500 @128 epoch); results and traces under
`main-800k/{x4rand/epoch3,dense4/epoch1}/serving-public-512/<set>/`.

| set (rows) | dense4 E1 | x4rand E3 | x4rand vs dense |
|---|---|---|---|
| gsm8k (1,319) | 6.5957 | 6.8636 | +4.06% |
| humaneval (164) | 5.1535 | 5.2613 | +2.09% |
| math500 (500) | 6.0312 | 6.1622 | +2.17% |
| mbpp_full (500) | 5.6934 | 5.8311 | +2.42% |
| mt_bench_turn1 (80) | 4.0139 | 4.0908 | +1.91% |

## Use

Resume training: pass the epoch directory as the checkpoint dir to speculators `train.py` (same `train_command.txt`,
`--epochs 4`). Serve: `export_d2_for_sglang.py <epoch dir> <out>` then sglang `--speculative-algorithm DFLASH
--speculative-draft-model-path <out>`; the MoS architecture needs `apply_serving_dflash2_mos.py` on sglang main.