Joint training with a tokenizer-hardness predictor
Fresh 20,000-step training from the original SmolLM2-1.7B and identity-warmed input hypernetwork. The complete original 1,000-tokenizer pool and 23-bucket corpus are reused.
The shared settings network predicts the paired BPB gap. Each step measures the actual pre-update transferred BPB minus the current canonical BPB and makes one squared-error update. Every fourth step checks the next tokenizer from a shuffled complete-pool cycle; other steps choose the largest prediction. This is online regression with coverage exploration, without REINFORCE or an entropy bonus.
Training preserves corrected paired boundaries, variable continuations and 128 total positions including BOS; batch 128, microbatch 32; original LM/HN objectives and original 190k learning-rate schedule. One H200 on Tillicum uses the validated GPU memory implementation.
Weights are saved every 500 steps and full restarts every 2,000 plus graceful-stop/final checkpoints. Payload hashes are verified before publishing the latest restart pointer. A real three-update uninterrupted-versus-resumed GPU preflight must pass before full training starts.
See metadata/plan.json for provenance, source/ for pinned code, and checkpoints/ for saved weights and restarts. Local live monitoring: http://100.84.104.59:1337/?run=anytok-hardness-training