Commit History
-
Restore canonical tokenizer to 32K vocabulary
Verified signatureRestore the exact 32,000-token tokenizer bundle from revision 3d43ff0569f01c490402ba3c3b028240675dfb3c. Remove the four 32,004 reasoning-boundary tokens and restore the original tokenizer metadata and card.6423b5b
-
Promote canonical 32,004-token reasoning-boundary tokenizer
Verified signaturePreserve IDs 0-31999 exactly and append think/final boundary tokens at IDs 32000-32003. Keep BOS/EOS/PAD/UNK metadata and offline tokenizer sidecars.f9e5969
-
Publish TR-HASH 32K tokenizer
Verified signaturef1610d9