|
Download README.md from iclr-mol/mol-checkpoints: direct link, hf CLI and curl.
- Browser
- Download file 2.76 kB
-
https://huggingface.co/iclr-mol/mol-checkpoints/resolve/main/README.md
- Command line
-
hf download hf://iclr-mol/mol-checkpoints/README.md
-
curl -L -o README.md https://huggingface.co/iclr-mol/mol-checkpoints/resolve/main/README.md
2.76 kB
| license: mit | |
| tags: | |
| - mixture-of-experts | |
| - language-model | |
| - pytorch | |
| # Mixture of Layers with Hybrid Attention — checkpoints | |
| Anonymous checkpoint release accompanying a double-blind submission. The model code, training and | |
| evaluation scripts, and `scripts/fetch_checkpoints.py` (which places these files where the scripts | |
| read them) are in the paper's supplementary material. | |
| **File layout.** The billion-scale files (`085`, `087`, `089`) are `torch.save` dicts with `model` | |
| (state dict, stored in bf16), `cfg` (architecture and training config) and `step`; they hold weights | |
| only. All other `.pt` files are dicts with `model_state_dict`, `step` and `val_ppl` (some also `condition`); their | |
| architecture comes from the corresponding run script in the supplementary material. `*_results.json` | |
| holds each run's logged validation-PPL trajectory. | |
| | Folder | Dataset | Contents | Paper | | |
| |---|---|---|---| | |
| | `060_sparse_mol`, `061_dense_mol`, `062_attention_modes`, `063_sparse_dispatch` | WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 | | |
| | `074_dthin_ablation` | WikiText-103 | d_thin ablation | Sec 3 | | |
| | `076_moe_ffn_baseline` | WikiText-103 | MoE-FFN control | Sec 5.1 | | |
| | `075_cosmopedia_mol`, `077_cosmopedia_moe_ffn`, `078_cosmopedia_dense_deltanet` | Cosmopedia v2 | MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines | Tab 3 | | |
| | `079_fineweb_hybrid_sweep` | FineWeb-Edu | 80M hybrid topology sweep 1+3of{5,10,15}, k_active rows, two dense baselines (extended schedule) | Tab 4, App | | |
| | `093_fineweb_80m_controls` | FineWeb-Edu | 80M clean-schedule runs: Dense 95M / 98M / 152M, MoE-FFN top-2, MoL 1+3of5 and 1+3of15, extra seeds | Tab 4 | | |
| | `085_dense_softmax_1p3b` | FineWeb-Edu 20B | Dense Softmax 1.3B, step 20000 | Tab 5 | | |
| | `087_mol_hybrid_1p3b` | FineWeb-Edu 20B | MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000 | Tab 5 | | |
| | `089_dense_reduced_ffn_1p3b` | FineWeb-Edu 20B | Dense Softmax 0.7B (iso-active), step 20000 | Tab 5 | | |
| The WikiText-103 dense baselines are not hosted. Matched validation PPL at 20B tokens: Dense 1.3B 14.72, | |
| MoL Hybrid 17.92, Dense 0.7B 18.09. | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| path = hf_hub_download("iclr-mol/mol-checkpoints", "087_mol_hybrid_1p3b/ckpt_step_20000.pt") | |
| ``` | |
| ## Notes on the files | |
| Three `*_results.json` files (`061_dense_mol/dense_k5_top3`, `062_attention_modes/restricted`, | |
| `079_fineweb_hybrid_sweep/dense_softmax`) had been overwritten by later short debug runs and were | |
| rebuilt from the corresponding training logs, which are included in the supplementary material. | |
| The checkpoints in this repository are the trained ones. | |