--- language: en tags: - tokenizers - sentiment-analysis datasets: - stanfordnlp/imdb --- # IMDb tokenisers Word-level and BPE tokenisers for English movie reviews, trained with the Hugging Face [`tokenizers`](https://github.com/huggingface/tokenizers) library on the 50,000 unlabelled reviews (the `unsupervised` split) of [stanfordnlp/imdb](https://huggingface.co/datasets/stanfordnlp/imdb). They were made for a university NLP project comparing sentiment classifiers on IMDb. | file | model | vocabulary | |---|---|---| | `word_vs_32000_mf_2.json` | whole words; other words become `[UNK]` | 32,000 | | `bpe_vs_8000_mf_2.json` | BPE (pieces inside a word start with `##`) | 8,000 | | `bpe_vs_16000_mf_2.json` | BPE | 16,000 | | `bpe_vs_32000_mf_2.json` | BPE | 32,000 | | `bpe_vs_64000_mf_2.json` | BPE | 64,000 | `mf_2`: tokens (word-level) or merges (BPE) seen fewer than 2 times were not added to the vocabulary. ## Usage ```python from huggingface_hub import hf_hub_download from tokenizers import Tokenizer tokenizer = Tokenizer.from_file(hf_hub_download("stevenmaschan/imdb-tokenizers", "bpe_vs_32000_mf_2.json")) tokenizer.encode("I didn't like it!!!!!").tokens # ['i', 'did', "n't", 'like', 'it', '!', '!'] ``` ## Text cleaning The cleaning is part of each tokeniser, so raw reviews can be passed in directly: 1. HTML line breaks (`
`) become spaces. 2. Characters broken by a Windows-1252 decoding error (`\x96`, `\x97`, `\x85`, `\x91`) become `-`, `...` and `'`; runs of two or more hyphens become ` - `; curly (โ€™) and acute (ยด) apostrophes become `'`. 3. Unicode NFKC normalisation; runs of `!`, `?` and `.` are capped at `!!`, `??` and `...`. 4. Lowercasing. 5. Splitting into words and punctuation, with clitics split off: `didn't` โ†’ `did` `n't`, `kubrick's` โ†’ `kubrick` `'s`.