Commit History
-
Restore canonical tokenizer to 32K vocabulary
Verified signatureRestore the exact 32,000-token tokenizer bundle from revision 3d43ff0569f01c490402ba3c3b028240675dfb3c. Remove the four 32,004 reasoning-boundary tokens and restore the original tokenizer metadata and card.6423b5b
-
Promote canonical 32,004-token reasoning-boundary tokenizer
Verified signaturePreserve IDs 0-31999 exactly and append think/final boundary tokens at IDs 32000-32003. Keep BOS/EOS/PAD/UNK metadata and offline tokenizer sidecars.f9e5969
-
Update config.json
Verified signature3d43ff0 -
Update config.json
Verified signature8b98ce9 -
Upload config.json
Verified signature735c4e4
-
Add reproducible r50k_base tokenizer comparison
Verified signature9a97c2c -
Publish TR-HASH 32K tokenizer
Verified signaturef1610d9 -
initial commit
Verified signature7fd06fc