kuramot6f/classical-ja-tokenizer

Classical Japanese Tokenizer (32k SentencePiece)

A SentencePiece unigram tokenizer with 32,000 pieces, trained only on texts written before 1868: classical Japanese, early-modern Japanese and Literary Chinese (漢文). Tokenizers built for modern Japanese split classical text poorly. This one is fitted to historical kana usage, kanbun and the rare characters common in premodern books. Byte fallback means it never produces <unk>.

It is the tokenizer of classical-smollm2-trunk and classical-smollm2-sft.

Usage

With 🤗 Transformers (the small custom wrapper keeps the exact ID mapping):

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("kuramot6f/classical-ja-tokenizer", trust_remote_code=True)
enc = tok("春はあけぼの。やうやう白くなりゆく山ぎは")
print(enc["input_ids"])                      # starts with BOS (=1)
print(tok.decode(enc["input_ids"], skip_special_tokens=True))

With plain SentencePiece:

import sentencepiece as spm
from huggingface_hub import hf_hub_download
sp = spm.SentencePieceProcessor(model_file=hf_hub_download("kuramot6f/classical-ja-tokenizer", "tokenizer.model"))
print(sp.encode("子曰學而時習之", out_type=str))

The wrapper differs from plain SentencePiece in one way: a literal ▁ (U+2581) in the input is encoded as UTF-8 bytes, so decoding always gives back the original text exactly.

Specs

• Vocabulary 32,000: <unk>=0, <s>=1, </s>=2, <pad>=3, plus 256 byte-fallback pieces.
• Unigram model, identity normalization (no NFKC), character_coverage=0.9995, byte_fallback=true, SentencePiece 0.2.1.
• Trained on 2M line fragments (70M characters) sampled from 121k documents and 1.33B characters of the pre-1868 corpus. This is a slightly larger pre-filter version of classical-ja-pretrain.

source	characters per token	byte-token rate	exact round-trip
Kanripo (Chinese classics)	1.24	6.8%	128/128
Minna de Honkoku (Edo manuscripts)	1.44	11.2%	128/128
NDL (Edo printed books, OCR)	1.37	0.4%	128/128
Wikisource (classical literature)	1.41	2.5%	128/128

These are diagnostic samples drawn from the training distribution, not a held-out benchmark.

Files

tokenizer.model / tokenizer.vocab (SentencePiece), classical_tokenizer.py + tokenizer_config.json (Transformers wrapper), config.json / sampling.json (training settings), REPORT.md (training report, Japanese), evaluation.jsonl / examples.json (diagnostics), manifest.json (hashes), train.log.

License

Code and tokenizer model: Apache-2.0. The tokenizer was trained on texts under several licenses: public domain, CC BY-SA 4.0, and CC BY-NC-SA 3.0 TW for CBETA Buddhist texts. See classical-ja-pretrain/LICENSES. The short text excerpts in examples.json / evaluation.jsonl keep their source licenses.