rootformer-v18-basran-transmute / tokenization_rootformer.py

Commit History

  1. Add classical Arabic morphological analyzer integration (tokenization_rootformer.py)

    Verified signature
    a952d1d
  2. Add official Rootformer Tokenizer & AutoClass integration (tokenization_rootformer.py)

    Verified signature
    8bde5d0