| --- |
| datasets: |
| - mlfoundations/dclm-baseline-1.0 |
| library_name: megatron-lm |
| license: mit |
| tags: |
| - mixture-of-experts |
| - moe |
| - distributionally-robust-optimization |
| - megatron |
| pipeline_tag: text-generation |
| --- |
| |
| # DRMoET checkpoints |
|
|
| **Paper:** [Distributionally Robust Mixture-of-Experts Training](https://huggingface.co/papers/2610.07207) |
|
|
| **Project page:** <https://drmoet.github.io> **Code:** <https://github.com/MAPS-research/DRMoET> |
|
|
| The weights in this repository are the checkpoints from the DRMoET paper: two DRMoET models (activation credit), each paired with the FLAME-MoE baseline trained with the same data, seed, and schedule. The folders below describe each checkpoint. |
|
|
| ## Checkpoints |
|
|
| | Folder | Method | Experts (top-k) | Layers / hidden | Train iters | DRO (β, η, α, credit) | |
| |---|---|---|---|---|---| |
| | `DRMoET-1.7B` | DRMoET, activation credit | 64 (top-6) + shared | 18 / 2048 | 32,000 | 0.999, 0.1, 1.0, L2 activation norm | |
| | `FLAME-MoE-1.7B` | FLAME-MoE baseline | 64 (top-6) + shared | 18 / 2048 | 32,000 | — | |
| | `DRMoET-290M-32E` | DRMoET, activation credit | 32 (top-6) + shared | 9 / 1024 | 16,000 | 0.999, 0.001, 1.0, L2 activation norm | |
| | `FLAME-MoE-290M-32E` | FLAME-MoE baseline | 32 (top-6) + shared | 9 / 1024 | 16,000 | — | |
|
|
| All models use sequence length 2048, global batch size 1024, learning rate 3e-4, auxiliary load-balancing loss coefficient 0.01, and the `EleutherAI/pythia-12b` tokenizer, trained on DCLM. Each DRMoET checkpoint and its baseline share the same seed (1,234 for 1.7B and 3,407 for 290M-32E). |
