imdb-tokenizers / README.md
stevenmaschan's picture
Add IMDb word-level and BPE tokenisers
148bce2 verified
|
Raw History Blame Contribute Delete
1.82 kB
metadata
language: en
tags:
  - tokenizers
  - sentiment-analysis
datasets:
  - stanfordnlp/imdb

IMDb tokenisers

Word-level and BPE tokenisers for English movie reviews, trained with the Hugging Face tokenizers library on the 50,000 unlabelled reviews (the unsupervised split) of stanfordnlp/imdb. They were made for a university NLP project comparing sentiment classifiers on IMDb.

file model vocabulary
word_vs_32000_mf_2.json whole words; other words become [UNK] 32,000
bpe_vs_8000_mf_2.json BPE (pieces inside a word start with ##) 8,000
bpe_vs_16000_mf_2.json BPE 16,000
bpe_vs_32000_mf_2.json BPE 32,000
bpe_vs_64000_mf_2.json BPE 64,000

mf_2: tokens (word-level) or merges (BPE) seen fewer than 2 times were not added to the vocabulary.

Usage

from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file(hf_hub_download("stevenmaschan/imdb-tokenizers", "bpe_vs_32000_mf_2.json"))
tokenizer.encode("I didn't like it!!!!!").tokens
# ['i', 'did', "n't", 'like', 'it', '!', '!']

Text cleaning

The cleaning is part of each tokeniser, so raw reviews can be passed in directly:

  1. HTML line breaks (<br />) become spaces.
  2. Characters broken by a Windows-1252 decoding error (\x96, \x97, \x85, \x91) become -, ... and '; runs of two or more hyphens become -; curly (’) and acute (´) apostrophes become '.
  3. Unicode NFKC normalisation; runs of !, ? and . are capped at !!, ?? and ....
  4. Lowercasing.
  5. Splitting into words and punctuation, with clitics split off: didn't → did n't, kubrick's → kubrick 's.