Instructions to use kuramot6f/classical-ja-tokenizer with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use kuramot6f/classical-ja-tokenizer with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("kuramot6f/classical-ja-tokenizer", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download config.json from kuramot6f/classical-ja-tokenizer: direct link, hf CLI and curl.
- Browser
- Download file 934 Bytes
-
https://huggingface.co/kuramot6f/classical-ja-tokenizer/resolve/main/config.json
- Command line
-
hf download hf://kuramot6f/classical-ja-tokenizer/config.json
-
curl -L -o config.json https://huggingface.co/kuramot6f/classical-ja-tokenizer/resolve/main/config.json
934 Bytes
| { | |
| "trainer": { | |
| "model_type": "unigram", | |
| "vocab_size": 32000, | |
| "character_coverage": 0.9995, | |
| "byte_fallback": true, | |
| "normalization_rule_name": "identity", | |
| "add_dummy_prefix": false, | |
| "remove_extra_whitespaces": false, | |
| "unk_id": 0, | |
| "bos_id": 1, | |
| "eos_id": 2, | |
| "pad_id": 3, | |
| "hard_vocab_limit": true, | |
| "input_sentence_size": 0, | |
| "shuffle_input_sentence": false, | |
| "max_sentence_length": 8192, | |
| "num_threads": 16, | |
| "seed_sentencepiece_size": 1000000 | |
| }, | |
| "sampling": { | |
| "method": "uniform reservoir over nonblank line chunks; no source reweighting", | |
| "seed": 20261020, | |
| "max_sentences": 2000000, | |
| "chunk_chars": 1024, | |
| "evaluation_per_source": 128, | |
| "evaluation": "uniform document reservoir, up to 2048 characters at document midpoint; overlaps training; not held-out", | |
| "literal_u2581": "training boundary; encode wrapper forces UTF-8 byte pieces" | |
| } | |
| } | |