Update README.md

README.md

@@ -8,6 +8,8 @@ tags:
 - moe
 - gqa
 - rope
+- pretrain
+- undertrained
 datasets:
 - HuggingFaceFW/fineweb-edu
 - mlfoundations/dclm-baseline-1.0
@@ -16,13 +18,25 @@ pipeline_tag: text-generation
 
 # linnet-497M
 
-A 497M parameter Mixture of Experts language model with 8 experts and 2 active experts per token and 157M active parameters. Trained from scratch using [rudyon/pipeline](https://github.com/rudyon/pipeline) on the [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and [mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) datasets.
+A 497M parameter Mixture of Experts base language model with 8 experts and 2 active experts per token and 157M active parameters. Trained from scratch using [rudyon/pipeline](https://github.com/rudyon/pipeline) on the [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) and [mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) datasets.
 
 Training was done on a single H100 GPU rented on [Prime Intellect](https://www.primeintellect.ai/) for about $17.
 
+## training status
+⚠️ This model is **undertrained**. Chinchilla-optimal training would require ~19000 steps 
+on ~10B tokens. This checkpoint was saved at step ~5000 (~26% of optimal), due to 
+compute budget constraints. The loss curve was still descending at the time of stopping.
+
+| Metric | Value |
+|--------|-------|
+| Steps completed | 5281 / 18965 |
+| Tokens seen | ~2.9B / 10B |
+| Final val bpb | ~1.21 |
+| HellaSwag (0-shot) | ~38% (random = 25%) |
+
 ## architecture
 
-The model is a 12-layer causal transformer with the following modifications:
+The model is a 12-layer causal transformer with the following architecture:
 
 | Component | Implementation |
 |-----------|---------------|