|
Download README.md from stevenmaschan/imdb-tokenizers: direct link, hf CLI and curl.
- Browser
- Download file 1.82 kB
-
https://huggingface.co/stevenmaschan/imdb-tokenizers/resolve/main/README.md
- Command line
-
hf download hf://stevenmaschan/imdb-tokenizers/README.md
-
curl -L -o README.md https://huggingface.co/stevenmaschan/imdb-tokenizers/resolve/main/README.md
1.82 kB
metadata
language: en
tags:
- tokenizers
- sentiment-analysis
datasets:
- stanfordnlp/imdb
IMDb tokenisers
Word-level and BPE tokenisers for English movie reviews, trained with the Hugging Face
tokenizers library on the 50,000 unlabelled reviews
(the unsupervised split) of stanfordnlp/imdb.
They were made for a university NLP project comparing sentiment classifiers on IMDb.
| file | model | vocabulary |
|---|---|---|
word_vs_32000_mf_2.json |
whole words; other words become [UNK] |
32,000 |
bpe_vs_8000_mf_2.json |
BPE (pieces inside a word start with ##) |
8,000 |
bpe_vs_16000_mf_2.json |
BPE | 16,000 |
bpe_vs_32000_mf_2.json |
BPE | 32,000 |
bpe_vs_64000_mf_2.json |
BPE | 64,000 |
mf_2: tokens (word-level) or merges (BPE) seen fewer than 2 times were not added to the vocabulary.
Usage
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file(hf_hub_download("stevenmaschan/imdb-tokenizers", "bpe_vs_32000_mf_2.json"))
tokenizer.encode("I didn't like it!!!!!").tokens
# ['i', 'did', "n't", 'like', 'it', '!', '!']
Text cleaning
The cleaning is part of each tokeniser, so raw reviews can be passed in directly:
- HTML line breaks (
<br />) become spaces. - Characters broken by a Windows-1252 decoding error (
\x96,\x97,\x85,\x91) become-,...and'; runs of two or more hyphens become-; curly (’) and acute (´) apostrophes become'. - Unicode NFKC normalisation; runs of
!,?and.are capped at!!,??and.... - Lowercasing.
- Splitting into words and punctuation, with clitics split off:
didn't→didn't,kubrick's→kubrick's.