Commit History

  1. Clarify special tokens purpose for file boundaries

    Verified signature
    3f7388f
  2. Update README with accurate usage examples

    Verified signature
    90028a3
  3. Upload binary-tokenizer-005

    Verified signature
    8ccf614
  4. initial commit

    Verified signature
    f9ce65e