|
Download README.md from mokner123/miniMoE: direct link, hf CLI and curl.
- Browser
- Download file 2.13 kB
-
https://huggingface.co/mokner123/miniMoE/resolve/main/README.md
- Command line
-
hf download hf://mokner123/miniMoE/README.md
-
curl -L -o README.md https://huggingface.co/mokner123/miniMoE/resolve/main/README.md
2.13 kB
| language: | |
| - en | |
| tags: | |
| - mixture-of-experts | |
| - moe | |
| - gpt | |
| - language-model | |
| datasets: | |
| - HuggingFaceFW/fineweb-edu | |
| # miniMoE | |
| miniMoE is a sparse GPT-style language model I trained from scratch on 10B FineWeb-Edu tokens across 8x A100 GPUs with DDP. | |
| This repo holds the trained checkpoint. | |
| The code, training pipeline, and full write-up are on GitHub: https://github.com/maokner/miniMoE | |
| ## What's in here | |
| `minimoe_step_0019073.pt` is the final checkpoint, taken at step 19,073 after ~10B tokens. | |
| It is a full training checkpoint, so alongside the model weights it keeps the optimizer and RNG state. | |
| That means you can either sample from it or resume training where it left off. | |
| ## Model | |
| | Item | Value | | |
| |---|---:| | |
| | Context length | 1,024 | | |
| | Vocabulary | 50,304 GPT-2 BPE tokens | | |
| | Transformer blocks | 6 | | |
| | Hidden size | 768 | | |
| | Experts per block | 8 | | |
| | Active experts per token | 2 | | |
| | Total parameters | 280.4M | | |
| | Active parameters per token | 110.4M | | |
| Every dense feed-forward block is replaced with an 8-expert top-2 MoE layer. | |
| Attention, embeddings, layer norms, the router, and the tied output head are shared across tokens. | |
| ## Results | |
| These come from the completed training log. | |
| | Metric | miniMoE | | |
| |---|---:| | |
| | Train loss | 3.0869 | | |
| | Validation loss | 3.0409 | | |
| | Test loss | 3.0725 | | |
| On the full HellaSwag benchmark (10,042 examples, completion-style scoring), this checkpoint edges out GPT-2 124M while activating fewer parameters per token. | |
| | Model | Active params per token | HellaSwag | | |
| |---|---:|---:| | |
| | miniMoE (this checkpoint) | 110.4M | 31.2% | | |
| | GPT-2 124M | 124M | 29.6% | | |
| The instruction-tuned variant holds HellaSwag steady at 30.6%; see the GitHub repo for the full evaluation and routing ablations. | |
| ## Usage | |
| Clone the GitHub repo, drop the checkpoint next to the code, and sample from it: | |
| ```bash | |
| pip install -r requirements.txt | |
| python sample.py -c minimoe_step_0019073.pt -p "Sparse expert language models" | |
| ``` | |
| To pull just the checkpoint from here: | |
| ```python | |
| from huggingface_hub import hf_hub_download | |
| path = hf_hub_download("mokner123/miniMoE", "minimoe_step_0019073.pt") | |
| ``` | |