iclr-mol commited on
Commit
87e72fd
·
verified ·
1 Parent(s): bcd54e3

Model card: file layout per folder, 093 controls, 076 mapping

Browse files
Files changed (1) hide show
  1. README.md +21 -17
README.md CHANGED
@@ -8,25 +8,30 @@ tags:
8
 
9
  # Mixture of Layers with Hybrid Attention — checkpoints
10
 
11
- Anonymous checkpoint release accompanying the double-blind submission
12
- *"Mixture of Layers with Hybrid Attention: Parallel Thin Blocks for Sparse Transformer Compute"*.
13
- The model code and training scripts are in the paper's supplementary material.
14
 
15
- Each `.pt` file is a `torch.save` dict with `model` (state dict), `cfg` (architecture
16
- and training config) and `step`. `*_results.json` files hold the logged validation
17
- PPL trajectory of the run.
 
 
18
 
19
  | Folder | Dataset | Contents | Paper |
20
  |---|---|---|---|
21
  | `060_sparse_mol`, `061_dense_mol`, `062_attention_modes`, `063_sparse_dispatch` | WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 |
22
  | `074_dthin_ablation` | WikiText-103 | d_thin ablation | Sec 3 |
23
- | `075_cosmopedia_mol`, `076_moe_ffn_baseline`, `077_cosmopedia_moe_ffn`, `078_cosmopedia_dense_deltanet` | Cosmopedia v2 | MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines | Tab 3 |
24
- | `079_fineweb_hybrid_sweep` | FineWeb-Edu | 80M hybrid topology sweep 1+3of{5,10,15} and two dense baselines | Tab 4 |
25
- | `085_dense_softmax_1p3b` | FineWeb-Edu 20B | Dense Softmax 1.3B, step 20000, weights only | Tab 5, 6 |
26
- | `087_mol_hybrid_1p3b` | FineWeb-Edu 20B | MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000, weights only | Tab 5, 6 |
27
- | `089_dense_reduced_ffn_1p3b` | FineWeb-Edu 20B | Dense Softmax 0.7B (iso-active), step 20000, weights only | Tab 5 |
 
 
28
 
29
- Headline validation PPL at 20B tokens: Dense 1.3B 15.03, MoL Hybrid 18.04, Dense 0.7B 18.53.
 
30
 
31
  ```python
32
  from huggingface_hub import hf_hub_download
@@ -35,8 +40,7 @@ path = hf_hub_download("iclr-mol/mol-checkpoints", "087_mol_hybrid_1p3b/ckpt_ste
35
 
36
  ## Notes on the files
37
 
38
- `*_results.json` holds the validation-PPL trajectory logged during training. Three of them
39
- (`061_dense_mol/dense_k5_top3`, `062_attention_modes/restricted`, `079_fineweb_hybrid_sweep/dense_softmax`)
40
- had been overwritten by later short debug runs and were rebuilt from the corresponding training logs,
41
- which are included in the paper's supplementary material; their final values (30.85, 32.31, 24.97)
42
- are the ones reported in the paper.
 
8
 
9
  # Mixture of Layers with Hybrid Attention — checkpoints
10
 
11
+ Anonymous checkpoint release accompanying a double-blind submission. The model code, training and
12
+ evaluation scripts, and `scripts/fetch_checkpoints.py` (which places these files where the scripts
13
+ read them) are in the paper's supplementary material.
14
 
15
+ **File layout.** The billion-scale files (`085`, `087`, `089`) are `torch.save` dicts with `model`
16
+ (state dict, stored in bf16), `cfg` (architecture and training config) and `step`; they hold weights
17
+ only. All other `.pt` files are dicts with `model_state_dict`, `step`, `val_ppl` and `condition`; their
18
+ architecture comes from the corresponding run script in the supplementary material. `*_results.json`
19
+ holds each run's logged validation-PPL trajectory.
20
 
21
  | Folder | Dataset | Contents | Paper |
22
  |---|---|---|---|
23
  | `060_sparse_mol`, `061_dense_mol`, `062_attention_modes`, `063_sparse_dispatch` | WikiText-103 | MoL ablations (dense FFN thin blocks, attention modes, sparse dispatch, hybrid shared+routed, 3 seeds for headline rows) | Tab 1, 2 |
24
  | `074_dthin_ablation` | WikiText-103 | d_thin ablation | Sec 3 |
25
+ | `076_moe_ffn_baseline` | WikiText-103 | MoE-FFN control | Sec 5.1 |
26
+ | `075_cosmopedia_mol`, `077_cosmopedia_moe_ffn`, `078_cosmopedia_dense_deltanet` | Cosmopedia v2 | MoL K=5, sparse 1+2of5, dense softmax / dense DeltaNet / MoE-FFN baselines | Tab 3 |
27
+ | `079_fineweb_hybrid_sweep` | FineWeb-Edu | 80M hybrid topology sweep 1+3of{5,10,15}, k_active rows, two dense baselines (extended schedule) | Tab 4, App |
28
+ | `093_fineweb_80m_controls` | FineWeb-Edu | 80M clean-schedule runs: Dense 95M / 98M / 152M, MoE-FFN top-2, MoL 1+3of5 and 1+3of15, extra seeds | Tab 4 |
29
+ | `085_dense_softmax_1p3b` | FineWeb-Edu 20B | Dense Softmax 1.3B, step 20000 | Tab 5 |
30
+ | `087_mol_hybrid_1p3b` | FineWeb-Edu 20B | MoL Hybrid 1+3of15, 2.08B total / 0.63B active, step 20000 | Tab 5 |
31
+ | `089_dense_reduced_ffn_1p3b` | FineWeb-Edu 20B | Dense Softmax 0.7B (iso-active), step 20000 | Tab 5 |
32
 
33
+ The WikiText-103 dense baselines are not hosted. Matched validation PPL at 20B tokens: Dense 1.3B 14.72,
34
+ MoL Hybrid 17.92, Dense 0.7B 18.09.
35
 
36
  ```python
37
  from huggingface_hub import hf_hub_download
 
40
 
41
  ## Notes on the files
42
 
43
+ Three `*_results.json` files (`061_dense_mol/dense_k5_top3`, `062_attention_modes/restricted`,
44
+ `079_fineweb_hybrid_sweep/dense_softmax`) had been overwritten by later short debug runs and were
45
+ rebuilt from the corresponding training logs, which are included in the supplementary material.
46
+ The checkpoints in this repository are the trained ones.