dougreid gkielian commited on
Commit
11b8250
·
1 Parent(s): 411aa17

fix: Remediate 219 glitch tokens in tokenizer.json (#7)

Browse files

- fix: Remediate 219 glitch tokens in tokenizer.json (f6a5db310ec95329671e57d28261253c4d7cb60a)
- Match upstream tokenizer.json formatting (720fffecc76c2247272372550d2e93a60e69e1a8)


Co-authored-by: Gregory Kielian <gkielian@users.noreply.huggingface.co>

Files changed (1) hide show
  1. tokenizer.json +2 -2
tokenizer.json CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:12bac982b793c44b03d52a250a9f0d0b666813da566b910c24a6da0695fd11e6
3
- size 32170070
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:7b512484de7440c449024ae8686cb8dbab12f2cf59bb53aaa89760cd6d86320a
3
+ size 32156872