Lemma
Collection
Parameter-efficient by design. Lemma-2.2M is best under 3M on BETU (13.78, #73/173). Lemma-1.1-250K is best under 1M on BETU (#126/173). โข 5 items โข Updated
Test model. Soup of EMA and post-trained models. No benchmarks were used as training data to avoid contamination.
| Model | Parameters | HellaSwag | ARC-Easy | ARC-Challenge | PIQA | ArithMark-3.0 |
|---|---|---|---|---|---|---|
| oscar128372/test-250k-soup | 250K | 28.30% | 27.56% | 19.45% | 52.18% | 32.10% |
| BananaMind/MicroBananaMind-v1 | 902K | 26.53% | 31.40% | 20.99% | 53.86% | 30.00% |
| fromziro/Syn-2.6M | 2.6M | 26.96% | 32.03% | 20.39% | 53.65% | 27.60% |
| fromziro/ZeroS-Pico-v1.1 | 60K | 26.19% | 26.30% | 24.57% | 51.14% | 30.50% |
| SLM-Archive/LowOnMind-300k | 297K | 25.07% | 29.00% | 23.12% | 50.22% | 29.40% |
This was made as a test to see how far I could push my architecture and training pipeline. This means no code to run inference with it is available in this repository.
Noticeably, it outperforms models similar in size.
The selected checkpoint (step 20,000/35,000) was trained for 81.92M tokens, with a sequence length of 512. It was then further post-trained for 500 steps and then souped 85:15 with the original EMA model, yielding this.
Apple MacBook M5 Air, 16 GB RAM, 10-core CPU, 8-core GPU MPS used.
Training: 45 minutes (pre-train) + 2 minutes (post-train + soup)
Evaluating: 4 minutes
See modeling.py.
This model is licensed under MIT.