Model card: file layout per folder, 093 controls, 076 mapping
Browse files
README.md
CHANGED
|
@@ -8,25 +8,30 @@ tags:
|
|
| 8 |
|
| 9 |
# Mixture of Layers with Hybrid Attention — checkpoints
|
| 10 |
|
| 11 |
-
Anonymous checkpoint release accompanying
|
| 12 |
-
|
| 13 |
-
|
| 14 |
|
| 15 |
-
|
| 16 |
-
and training config) and `step`
|
| 17 |
-
|
|
|
|
|
|
|
| 18 |
|
| 19 |
| Folder | Dataset | Contents | Paper |
|
| 20 |
|---|---|---|---|
|
| 21 |
| `060_sparse_mol`, `061_dense_mol`, `062_attention_modes`, `063_sparse_dispatch` | WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 |
|
| 22 |
| `074_dthin_ablation` | WikiText-103 | d_thin ablation | Sec 3 |
|
| 23 |
-
| `
|
| 24 |
-
| `
|
| 25 |
-
| `
|
| 26 |
-
| `
|
| 27 |
-
| `
|
|
|
|
|
|
|
| 28 |
|
| 29 |
-
|
|
|
|
| 30 |
|
| 31 |
```python
|
| 32 |
from huggingface_hub import hf_hub_download
|
|
@@ -35,8 +40,7 @@ path = hf_hub_download("iclr-mol/mol-checkpoints", "087_mol_hybrid_1p3b/ckpt_ste
|
|
| 35 |
|
| 36 |
## Notes on the files
|
| 37 |
|
| 38 |
-
`*_results.json`
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
are the ones reported in the paper.
|
|
|
|
| 8 |
|
| 9 |
# Mixture of Layers with Hybrid Attention — checkpoints
|
| 10 |
|
| 11 |
+
Anonymous checkpoint release accompanying a double-blind submission. The model code, training and
|
| 12 |
+
evaluation scripts, and `scripts/fetch_checkpoints.py` (which places these files where the scripts
|
| 13 |
+
read them) are in the paper's supplementary material.
|
| 14 |
|
| 15 |
+
**File layout.** The billion-scale files (`085`, `087`, `089`) are `torch.save` dicts with `model`
|
| 16 |
+
(state dict, stored in bf16), `cfg` (architecture and training config) and `step`; they hold weights
|
| 17 |
+
only. All other `.pt` files are dicts with `model_state_dict`, `step`, `val_ppl` and `condition`; their
|
| 18 |
+
architecture comes from the corresponding run script in the supplementary material. `*_results.json`
|
| 19 |
+
holds each run's logged validation-PPL trajectory.
|
| 20 |
|
| 21 |
| Folder | Dataset | Contents | Paper |
|
| 22 |
|---|---|---|---|
|
| 23 |
| `060_sparse_mol`, `061_dense_mol`, `062_attention_modes`, `063_sparse_dispatch` | WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 |
|
| 24 |
| `074_dthin_ablation` | WikiText-103 | d_thin ablation | Sec 3 |
|
| 25 |
+
| `076_moe_ffn_baseline` | WikiText-103 | MoE-FFN control | Sec 5.1 |
|
| 26 |
+
| `075_cosmopedia_mol`, `077_cosmopedia_moe_ffn`, `078_cosmopedia_dense_deltanet` | Cosmopedia v2 | MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines | Tab 3 |
|
| 27 |
+
| `079_fineweb_hybrid_sweep` | FineWeb-Edu | 80M hybrid topology sweep 1+3of{5,10,15}, k_active rows, two dense baselines (extended schedule) | Tab 4, App |
|
| 28 |
+
| `093_fineweb_80m_controls` | FineWeb-Edu | 80M clean-schedule runs: Dense 95M / 98M / 152M, MoE-FFN top-2, MoL 1+3of5 and 1+3of15, extra seeds | Tab 4 |
|
| 29 |
+
| `085_dense_softmax_1p3b` | FineWeb-Edu 20B | Dense Softmax 1.3B, step 20000 | Tab 5 |
|
| 30 |
+
| `087_mol_hybrid_1p3b` | FineWeb-Edu 20B | MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000 | Tab 5 |
|
| 31 |
+
| `089_dense_reduced_ffn_1p3b` | FineWeb-Edu 20B | Dense Softmax 0.7B (iso-active), step 20000 | Tab 5 |
|
| 32 |
|
| 33 |
+
The WikiText-103 dense baselines are not hosted. Matched validation PPL at 20B tokens: Dense 1.3B 14.72,
|
| 34 |
+
MoL Hybrid 17.92, Dense 0.7B 18.09.
|
| 35 |
|
| 36 |
```python
|
| 37 |
from huggingface_hub import hf_hub_download
|
|
|
|
| 40 |
|
| 41 |
## Notes on the files
|
| 42 |
|
| 43 |
+
Three `*_results.json` files (`061_dense_mol/dense_k5_top3`, `062_attention_modes/restricted`,
|
| 44 |
+
`079_fineweb_hybrid_sweep/dense_softmax`) had been overwritten by later short debug runs and were
|
| 45 |
+
rebuilt from the corresponding training logs, which are included in the supplementary material.
|
| 46 |
+
The checkpoints in this repository are the trained ones.
|
|
|