somali-tokenizer / README.md

Commit History

3 days ago

  1. Update model card: new id, Python package, benchmark against general tokenizers

    Verified signature
    7420ad3

  1. Add Somali ByteLevel BPE v2

    Verified signature
    e7d6827