File size: 2,132 Bytes
99aecf8
c515c7e
 
 
 
 
 
 
 
 
99aecf8
c515c7e
 
 
60fccd3
c515c7e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
60fccd3
 
 
 
 
 
 
 
 
c515c7e
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
---
language:
- en
tags:
- mixture-of-experts
- moe
- gpt
- language-model
datasets:
- HuggingFaceFW/fineweb-edu
---

# miniMoE

miniMoE is a sparse GPT-style language model I trained from scratch on 10B FineWeb-Edu tokens across 8x A100 GPUs with DDP.
This repo holds the trained checkpoint.
The code, training pipeline, and full write-up are on GitHub: https://github.com/maokner/miniMoE

## What's in here

`minimoe_step_0019073.pt` is the final checkpoint, taken at step 19,073 after ~10B tokens.
It is a full training checkpoint, so alongside the model weights it keeps the optimizer and RNG state.
That means you can either sample from it or resume training where it left off.

## Model

| Item | Value |
|---|---:|
| Context length | 1,024 |
| Vocabulary | 50,304 GPT-2 BPE tokens |
| Transformer blocks | 6 |
| Hidden size | 768 |
| Experts per block | 8 |
| Active experts per token | 2 |
| Total parameters | 280.4M |
| Active parameters per token | 110.4M |

Every dense feed-forward block is replaced with an 8-expert top-2 MoE layer.
Attention, embeddings, layer norms, the router, and the tied output head are shared across tokens.

## Results

These come from the completed training log.

| Metric | miniMoE |
|---|---:|
| Train loss | 3.0869 |
| Validation loss | 3.0409 |
| Test loss | 3.0725 |

On the full HellaSwag benchmark (10,042 examples, completion-style scoring), this checkpoint edges out GPT-2 124M while activating fewer parameters per token.

| Model | Active params per token | HellaSwag |
|---|---:|---:|
| miniMoE (this checkpoint) | 110.4M | 31.2% |
| GPT-2 124M | 124M | 29.6% |

The instruction-tuned variant holds HellaSwag steady at 30.6%; see the GitHub repo for the full evaluation and routing ablations.

## Usage

Clone the GitHub repo, drop the checkpoint next to the code, and sample from it:

```bash
pip install -r requirements.txt
python sample.py -c minimoe_step_0019073.pt -p "Sparse expert language models"
```

To pull just the checkpoint from here:

```python
from huggingface_hub import hf_hub_download

path = hf_hub_download("mokner123/miniMoE", "minimoe_step_0019073.pt")
```