mol-checkpoints / README.md
iclr-mol's picture
Model card: checkpoint fields
60e9991 verified
|
Raw History Blame Contribute Delete
2.76 kB
---
license: mit
tags:
- mixture-of-experts
- language-model
- pytorch
---
# Mixture of Layers with Hybrid Attention — checkpoints
Anonymous checkpoint release accompanying a double-blind submission. The model code, training and
evaluation scripts, and `scripts/fetch_checkpoints.py` (which places these files where the scripts
read them) are in the paper's supplementary material.
**File layout.** The billion-scale files (`085`, `087`, `089`) are `torch.save` dicts with `model`
(state dict, stored in bf16), `cfg` (architecture and training config) and `step`; they hold weights
only. All other `.pt` files are dicts with `model_state_dict`, `step` and `val_ppl` (some also `condition`); their
architecture comes from the corresponding run script in the supplementary material. `*_results.json`
holds each run's logged validation-PPL trajectory.
| Folder | Dataset | Contents | Paper |
|---|---|---|---|
| `060_sparse_mol`, `061_dense_mol`, `062_attention_modes`, `063_sparse_dispatch` | WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 |
| `074_dthin_ablation` | WikiText-103 | d_thin ablation | Sec 3 |
| `076_moe_ffn_baseline` | WikiText-103 | MoE-FFN control | Sec 5.1 |
| `075_cosmopedia_mol`, `077_cosmopedia_moe_ffn`, `078_cosmopedia_dense_deltanet` | Cosmopedia v2 | MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines | Tab 3 |
| `079_fineweb_hybrid_sweep` | FineWeb-Edu | 80M hybrid topology sweep 1+3of{5,10,15}, k_active rows, two dense baselines (extended schedule) | Tab 4, App |
| `093_fineweb_80m_controls` | FineWeb-Edu | 80M clean-schedule runs: Dense 95M / 98M / 152M, MoE-FFN top-2, MoL 1+3of5 and 1+3of15, extra seeds | Tab 4 |
| `085_dense_softmax_1p3b` | FineWeb-Edu 20B | Dense Softmax 1.3B, step 20000 | Tab 5 |
| `087_mol_hybrid_1p3b` | FineWeb-Edu 20B | MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000 | Tab 5 |
| `089_dense_reduced_ffn_1p3b` | FineWeb-Edu 20B | Dense Softmax 0.7B (iso-active), step 20000 | Tab 5 |
The WikiText-103 dense baselines are not hosted. Matched validation PPL at 20B tokens: Dense 1.3B 14.72,
MoL Hybrid 17.92, Dense 0.7B 18.09.
```python
from huggingface_hub import hf_hub_download
path = hf_hub_download("iclr-mol/mol-checkpoints", "087_mol_hybrid_1p3b/ckpt_step_20000.pt")
```
## Notes on the files
Three `*_results.json` files (`061_dense_mol/dense_k5_top3`, `062_attention_modes/restricted`,
`079_fineweb_hybrid_sweep/dense_softmax`) had been overwritten by later short debug runs and were
rebuilt from the corresponding training logs, which are included in the supplementary material.
The checkpoints in this repository are the trained ones.