Commit History
11 days ago
-
fix: Remediate 209 glitch tokens in tokenizer.json
Verified signature## Glitch Token Remediation — Tokenizer Patch ### Summary This PR patches `tokenizer.json` to remediate **209 glitch tokens** identified by embedding vector analysis. Glitch tokens are vocabulary entries whose embeddings have collapsed to near-identical representations, causing unpredictable model behavior when they appear in input. **No model weights are modified** — this is a tokenizer-only fix. ### Technique: Hybrid (merge pruning + placeholder rename) The patch uses two complementary strategies: **1. Merge pruning** (209 tokens): Removes the BPE merge rules that produce multi-character glitch tokens. The vocab entry stays (ID preserved) but becomes unreachable — text that would have matched is now tokenized as trained constituent subwords instead. **2. Placeholder rename** (0 tokens): For single-character glitch tokens (rare Unicode characters) that have no producing merge rule, the vocab string is renamed to `<glitch_pruned_ID>`. The original character now falls through to …7a62392
-
Preparing for release!
Co-authored-by: osanseviero <osanseviero@users.noreply.huggingface.co> Co-authored-by: douglas reid <douglas reid@users.noreply.huggingface.co> Co-authored-by: dougreid <dougreid@users.noreply.huggingface.co>be8e26a
