Mixture of Layers with Hybrid Attention β€” checkpoints

Anonymous checkpoint release accompanying a double-blind submission. The model code, training and evaluation scripts, and scripts/fetch_checkpoints.py (which places these files where the scripts read them) are in the paper's supplementary material.

File layout. The billion-scale files (085, 087, 089) are torch.save dicts with model (state dict, stored in bf16), cfg (architecture and training config) and step; they hold weights only. All other .pt files are dicts with model_state_dict, step and val_ppl (some also condition); their architecture comes from the corresponding run script in the supplementary material. *_results.json holds each run's logged validation-PPL trajectory.

Folder Dataset Contents Paper
060_sparse_mol, 061_dense_mol, 062_attention_modes, 063_sparse_dispatch WikiText-103 MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) Tab 1, 2
074_dthin_ablation WikiText-103 d_thin ablation Sec 3
076_moe_ffn_baseline WikiText-103 MoE-FFN control Sec 5.1
075_cosmopedia_mol, 077_cosmopedia_moe_ffn, 078_cosmopedia_dense_deltanet Cosmopedia v2 MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines Tab 3
079_fineweb_hybrid_sweep FineWeb-Edu 80M hybrid topology sweep 1+3of{5,10,15}, k_active rows, two dense baselines (extended schedule) Tab 4, App
093_fineweb_80m_controls FineWeb-Edu 80M clean-schedule runs: Dense 95M / 98M / 152M, MoE-FFN top-2, MoL 1+3of5 and 1+3of15, extra seeds Tab 4
085_dense_softmax_1p3b FineWeb-Edu 20B Dense Softmax 1.3B, step 20000 Tab 5
087_mol_hybrid_1p3b FineWeb-Edu 20B MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000 Tab 5
089_dense_reduced_ffn_1p3b FineWeb-Edu 20B Dense Softmax 0.7B (iso-active), step 20000 Tab 5

The WikiText-103 dense baselines are not hosted. Matched validation PPL at 20B tokens: Dense 1.3B 14.72, MoL Hybrid 17.92, Dense 0.7B 18.09.

from huggingface_hub import hf_hub_download
path = hf_hub_download("iclr-mol/mol-checkpoints", "087_mol_hybrid_1p3b/ckpt_step_20000.pt")

Notes on the files

Three *_results.json files (061_dense_mol/dense_k5_top3, 062_attention_modes/restricted, 079_fineweb_hybrid_sweep/dense_softmax) had been overwritten by later short debug runs and were rebuilt from the corresponding training logs, which are included in the supplementary material. The checkpoints in this repository are the trained ones.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support