Commit History

  1. Register tokenizer as HF SentencePiece tokenizer

    Verified signature
    358fd65
  2. Add tokenizer fineweb_fr

    Verified signature
    0daa6d8
  3. initial commit

    Verified signature
    f7aa0a9