File size: 1,552 Bytes
9590620 7cd3987 1bdf877 7cd3987 1bdf877 7cd3987 9590620 1bdf877 7cd3987 1bdf877 a3de07f 1bdf877 a3de07f 1bdf877 f974884 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | ---
datasets:
- mlfoundations/dclm-baseline-1.0
library_name: megatron-lm
license: mit
tags:
- mixture-of-experts
- moe
- distributionally-robust-optimization
- megatron
pipeline_tag: text-generation
---
# DRMoET checkpoints
**Paper:** [Distributionally Robust Mixture-of-Experts Training](https://huggingface.co/papers/2610.07207)
**Project page:** <https://drmoet.github.io> **Code:** <https://github.com/MAPS-research/DRMoET>
The weights in this repository are the checkpoints from the DRMoET paper: two DRMoET models (activation credit), each paired with the FLAME-MoE baseline trained with the same data, seed, and schedule. The folders below describe each checkpoint.
## Checkpoints
| Folder | Method | Experts (top-k) | Layers / hidden | Train iters | DRO (β, η, α, credit) |
|---|---|---|---|---|---|
| `DRMoET-1.7B` | DRMoET, activation credit | 64 (top-6) + shared | 18 / 2048 | 32,000 | 0.999, 0.1, 1.0, L2 activation norm |
| `FLAME-MoE-1.7B` | FLAME-MoE baseline | 64 (top-6) + shared | 18 / 2048 | 32,000 | — |
| `DRMoET-290M-32E` | DRMoET, activation credit | 32 (top-6) + shared | 9 / 1024 | 16,000 | 0.999, 0.001, 1.0, L2 activation norm |
| `FLAME-MoE-290M-32E` | FLAME-MoE baseline | 32 (top-6) + shared | 9 / 1024 | 16,000 | — |
All models use sequence length 2048, global batch size 1024, learning rate 3e-4, auxiliary load-balancing loss coefficient 0.01, and the `EleutherAI/pythia-12b` tokenizer, trained on DCLM. Each DRMoET checkpoint and its baseline share the same seed (1,234 for 1.7B and 3,407 for 290M-32E). |