Download tokenization.py from hf-internal-testing/test_dynamic_processor: direct link, hf CLI and curl.
- Browser
- Download file 113 Bytes
-
https://huggingface.co/hf-internal-testing/test_dynamic_processor/resolve/24b4f962cbc2c1df3750c77e0eee7890e4ced959/tokenization.py
- Command line
-
hf download hf://hf-internal-testing/test_dynamic_processor@24b4f962cbc2c1df3750c77e0eee7890e4ced959/tokenization.py
-
curl -L -o tokenization.py https://huggingface.co/hf-internal-testing/test_dynamic_processor/resolve/24b4f962cbc2c1df3750c77e0eee7890e4ced959/tokenization.py
113 Bytes
| from transformers import BertTokenizer | |
| class NewTokenizer(BertTokenizer): | |
| special_attribute_present = True | |