Commit History

  1. move json files to datasets/

    67c1004

  1. reorganise files and fix import bugs

    aec018b
  2. all logic exposed in frontend

    0386dad
  3. initial frontend work

    b500a16
  4. tokenize after removing word

    fa33ecb
  5. countLogLoss

    3bd66ee
  6. move most expected values into json files

    b297208
  7. move vocabulary into json file

    8b6c9f1

  1. tokenizeTextLM loop and cache tokenizeLM

    37fc247
  2. tokenizeTextLM

    efb68ae
  3. tokenizeLM (no cache)

    8b43a5b
  4. also return vocabulary from BPE

    4ebb38a
  5. unigramLM

    19d4dca
  6. tokenizeBPE

    5a9f18c
  7. fix preprocess

    cdbb891
  8. BPE

    7e96b6d
  9. mostFrequentPair

    2c0ab92
  10. mergeBigrams

    dd9e999
  11. pairCount

    ff51fd5
  12. preprocess and initialVocabulary

    2191a1b