File size: 2,132 Bytes
99aecf8 c515c7e 99aecf8 c515c7e 60fccd3 c515c7e 60fccd3 c515c7e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 | ---
language:
- en
tags:
- mixture-of-experts
- moe
- gpt
- language-model
datasets:
- HuggingFaceFW/fineweb-edu
---
# miniMoE
miniMoE is a sparse GPT-style language model I trained from scratch on 10B FineWeb-Edu tokens across 8x A100 GPUs with DDP.
This repo holds the trained checkpoint.
The code, training pipeline, and full write-up are on GitHub: https://github.com/maokner/miniMoE
## What's in here
`minimoe_step_0019073.pt` is the final checkpoint, taken at step 19,073 after ~10B tokens.
It is a full training checkpoint, so alongside the model weights it keeps the optimizer and RNG state.
That means you can either sample from it or resume training where it left off.
## Model
| Item | Value |
|---|---:|
| Context length | 1,024 |
| Vocabulary | 50,304 GPT-2 BPE tokens |
| Transformer blocks | 6 |
| Hidden size | 768 |
| Experts per block | 8 |
| Active experts per token | 2 |
| Total parameters | 280.4M |
| Active parameters per token | 110.4M |
Every dense feed-forward block is replaced with an 8-expert top-2 MoE layer.
Attention, embeddings, layer norms, the router, and the tied output head are shared across tokens.
## Results
These come from the completed training log.
| Metric | miniMoE |
|---|---:|
| Train loss | 3.0869 |
| Validation loss | 3.0409 |
| Test loss | 3.0725 |
On the full HellaSwag benchmark (10,042 examples, completion-style scoring), this checkpoint edges out GPT-2 124M while activating fewer parameters per token.
| Model | Active params per token | HellaSwag |
|---|---:|---:|
| miniMoE (this checkpoint) | 110.4M | 31.2% |
| GPT-2 124M | 124M | 29.6% |
The instruction-tuned variant holds HellaSwag steady at 30.6%; see the GitHub repo for the full evaluation and routing ablations.
## Usage
Clone the GitHub repo, drop the checkpoint next to the code, and sample from it:
```bash
pip install -r requirements.txt
python sample.py -c minimoe_step_0019073.pt -p "Sparse expert language models"
```
To pull just the checkpoint from here:
```python
from huggingface_hub import hf_hub_download
path = hf_hub_download("mokner123/miniMoE", "minimoe_step_0019073.pt")
```
|