Commit History

  1. add: 4k simplestories

    Verified signature
    1d4824a

  1. add: mdm mask spec token

    Verified signature
    5a776e2

  1. remove: mdm mask

    Verified signature
    b52ca83

  1. add: mdm mask in special tokens

    Verified signature
    4355810

  1. add: simplestories 8k tokenizer config

    Verified signature
    64d565f

  1. add: 8k vocab

    Verified signature
    1eefe41

  1. remove: pad token

    Verified signature
    c6032b5

  1. add: pad token

    Verified signature
    129e9a9

  1. remove: pad token

    Verified signature
    6793895

  1. add: pad token

    Verified signature
    26dd6df

  1. refactor: remove pad token

    Verified signature
    21b6e51
  2. add: pad special token

    Verified signature
    353af19

  1. exp: remove bos token

    Verified signature
    b31ccbe

  1. clean: extra comma in json

    Verified signature
    31233c0
  2. fix: remove eos from vocab

    Verified signature
    da996f2
  3. add: tokenizer config

    Verified signature
    08fb3c4
  4. initial commit

    Verified signature
    aeb4ceb