Commit History
7 days ago
-
Match upstream tokenizer.json formatting
Verified signatureFormatting-only change: re-serialize the patched tokenizer.json with the same pretty-printed layout as upstream (indent=2, identical key order). Parsed content is unchanged versus the previous commit: same vocab (size preserved), same merge list (original order), same normalizer/pre-tokenizer/post-processor/decoder. Why: tokenizer.json is Git-LFS tracked, so the Hub UI only shows the LFS pointer diff. With matching formatting, `diff <upstream tokenizer.json> <this tokenizer.json>` now shows ONLY the removed BPE merge entries (and the single placeholder vocab rename where applicable), instead of a whole-file reformat, and the LFS size stays ~32 MB instead of dropping to ~15 MB.720fffe
30 days ago
-
fix: Remediate 219 glitch tokens in tokenizer.json
Verified signature## Glitch Token Remediation — Tokenizer Patch ### Summary This PR patches `tokenizer.json` to remediate **219 glitch tokens** identified by embedding vector analysis. Glitch tokens are vocabulary entries whose embeddings have collapsed to near-identical representations, causing unpredictable model behavior when they appear in input. **No model weights are modified** — this is a tokenizer-only fix. ### Technique: Hybrid (merge pruning + placeholder rename) The patch uses two complementary strategies: **1. Merge pruning** (219 tokens): Removes the BPE merge rules that produce multi-character glitch tokens. The vocab entry stays (ID preserved) but becomes unreachable — text that would have matched is now tokenized as trained constituent subwords instead. **2. Placeholder rename** (0 tokens): For single-character glitch tokens (rare Unicode characters) that have no producing merge rule, the vocab string is renamed to `<glitch_pruned_ID>`. The original character now falls through to …f6a5db3
-
Upload tokenizer.json
Verified signatureAdd BOS by default for base models7aa32e6 -
Preparing for release!
Co-authored-by: RyanMullins <RyanMullins@users.noreply.huggingface.co> Co-authored-by: pcuenq <pcuenq@users.noreply.huggingface.co> Co-authored-by: bebechien <bebechien@users.noreply.huggingface.co> Co-authored-by: osanseviero <osanseviero@users.noreply.huggingface.co>9f9f0f2
