Mixture of Layers with Hybrid Attention β checkpoints
Anonymous checkpoint release accompanying a double-blind submission. The model code, training and
evaluation scripts, and scripts/fetch_checkpoints.py (which places these files where the scripts
read them) are in the paper's supplementary material.
File layout. The billion-scale files (085, 087, 089) are torch.save dicts with model
(state dict, stored in bf16), cfg (architecture and training config) and step; they hold weights
only. All other .pt files are dicts with model_state_dict, step and val_ppl (some also condition); their
architecture comes from the corresponding run script in the supplementary material. *_results.json
holds each run's logged validation-PPL trajectory.
| Folder | Dataset | Contents | Paper |
|---|---|---|---|
060_sparse_mol, 061_dense_mol, 062_attention_modes, 063_sparse_dispatch |
WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 |
074_dthin_ablation |
WikiText-103 | d_thin ablation | Sec 3 |
076_moe_ffn_baseline |
WikiText-103 | MoE-FFN control | Sec 5.1 |
075_cosmopedia_mol, 077_cosmopedia_moe_ffn, 078_cosmopedia_dense_deltanet |
Cosmopedia v2 | MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines | Tab 3 |
079_fineweb_hybrid_sweep |
FineWeb-Edu | 80M hybrid topology sweep 1+3of{5,10,15}, k_active rows, two dense baselines (extended schedule) | Tab 4, App |
093_fineweb_80m_controls |
FineWeb-Edu | 80M clean-schedule runs: Dense 95M / 98M / 152M, MoE-FFN top-2, MoL 1+3of5 and 1+3of15, extra seeds | Tab 4 |
085_dense_softmax_1p3b |
FineWeb-Edu 20B | Dense Softmax 1.3B, step 20000 | Tab 5 |
087_mol_hybrid_1p3b |
FineWeb-Edu 20B | MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000 | Tab 5 |
089_dense_reduced_ffn_1p3b |
FineWeb-Edu 20B | Dense Softmax 0.7B (iso-active), step 20000 | Tab 5 |
The WikiText-103 dense baselines are not hosted. Matched validation PPL at 20B tokens: Dense 1.3B 14.72, MoL Hybrid 17.92, Dense 0.7B 18.09.
from huggingface_hub import hf_hub_download
path = hf_hub_download("iclr-mol/mol-checkpoints", "087_mol_hybrid_1p3b/ckpt_step_20000.pt")
Notes on the files
Three *_results.json files (061_dense_mol/dense_k5_top3, 062_attention_modes/restricted,
079_fineweb_hybrid_sweep/dense_softmax) had been overwritten by later short debug runs and were
rebuilt from the corresponding training logs, which are included in the supplementary material.
The checkpoints in this repository are the trained ones.