Just for performing some experiments. Do not use.
Since the loss seem to start going up, I did have to restore this from 9e945cb0636bde60bec30bd7df5db30f80401cc7 (2 step 600k/200). I am then restarting with warmup decaying from 1e-4.
That did failed. Checked out c94b5bb43b05fc798f9db013d940b05b3b47cd98 instead and restarted step 3 from here.