picur-tokenizer / README.md

Commit History

  1. a2C vocabulary: refit on the cleaned, budget-proportional corpus⁠

    Verified signature
    a858237

  1. docs: drop an unsupported claim

    Verified signature
    92c53f9
  2. docs: correct the fertility delta and spell out the eviction example

    Verified signature
    eb271df
  3. a2B: reproducible seed, public benchmarks, benchmark.py

    Verified signature
    4079ffe
  4. a2u: 28,000-stem seed, measured on text withheld from the fit

    Verified signature
    a34a31c

  1. docs: the diminutive joke, as the counterexample for over-splitting

    Verified signature
    f26b725
  2. picur-tokenizer a2x: 49,152-piece Hungarian Unigram vocabulary

    Verified signature
    a7cf8a7