Test model. Soup of EMA and post-trained models. No benchmarks were used as training data to avoid contamination.

Model Parameters HellaSwag ARC-Easy ARC-Challenge PIQA ArithMark-3.0
oscar128372/test-250k-soup 250K 28.30% 27.56% 19.45% 52.18% 32.10%
BananaMind/MicroBananaMind-v1 902K 26.53% 31.40% 20.99% 53.86% 30.00%
fromziro/Syn-2.6M 2.6M 26.96% 32.03% 20.39% 53.65% 27.60%
fromziro/ZeroS-Pico-v1.1 60K 26.19% 26.30% 24.57% 51.14% 30.50%
SLM-Archive/LowOnMind-300k 297K 25.07% 29.00% 23.12% 50.22% 29.40%

This was made as a test to see how far I could push my architecture and training pipeline. This means no code to run inference with it is available in this repository.

Noticeably, it outperforms models similar in size.

Training

The selected checkpoint (step 20,000/35,000) was trained for 81.92M tokens, with a sequence length of 512. It was then further post-trained for 500 steps and then souped 85:15 with the original EMA model, yielding this.

Hardware

Apple MacBook M5 Air, 16 GB RAM, 10-core CPU, 8-core GPU MPS used.

Time

Training: 45 minutes (pre-train) + 2 minutes (post-train + soup)

Evaluating: 4 minutes

Use

See modeling.py.

License

This model is licensed under MIT.

Downloads last month
48
Safetensors
Model size
250k params
Tensor type
F16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including oscar128372/test-250k-soup