The speedrun track is a perf flex, and I understand that sort of thing. But what I find sad is that not only do people even look at the inference capability of this tiny 11 layer model with a loss of 3.28 but also the same sort of efforts don't seem to have been attempted on the train_gpt_medium track. Yes, there are some but a lot of things like the combined NorMuonAndAdam optimizer isn't used in medium and all the stuff one finds in triton_kernels.py were written for the speed track.
The medium model isn't that much larger but to train it takes 17 minutes to hit the medium loss target of 2.9. Is there room for more improvement? The medium model is tickling the bottom end of models which exhibit useful behavior like as with the QWen3.5 0.8B or the OLMo-1B model. Having a quick turn around time opens up a different approach to experimentation.
