cadazar/han2han-pt

Han2Han PT

The pre-trained (PT) checkpoint of Han2Han, a 169M-parameter encoder-decoder model that learns script-invariant representations of Korean text. A document written in Hanja and its Hangul transcription land at the same point in embedding space. The recipe (jamo and character-level embedding fusion, morpheme-aware denoising, bidirectional Hanja-Hangul transcription) is described in the paper, published in Findings of EMNLP 2026.

This is the starting point for fine-tuning, and the first of three checkpoints:

Repo	Stage
cadazar/han2han-pt (this one)	pre-training, 35B tokens
cadazar/han2han-it	instruction tuning from these weights
cadazar/han2han-rl	reinforcement learning on transcription in both directions (Hangul to Hanja and Hanja to Hangul), from han2han-it

The orbax (Flax) checkpoints behind the last two stages are the flax revision of cadazar/han2han-it and of cadazar/han2han-rl (hf download cadazar/han2han-it --revision flax). The reinforcement-learning stage, its rewards and its data are in the GitHub repo (rl/) and the dataset cadazar/han2han-rl-data.

These weights are an average

The weights are the uniform average of the last five checkpoints of the pre-training run (steps 6347, 6441, 6536, 6630 and 6725, saved every 0.5B tokens from 33B to 35B), not the final checkpoint alone. The average is what the instruction tuning started from, chosen after comparing starting points, so it is the pre-trained model that han2han-it and han2han-rl descend from. The learning rate had nearly finished its cooldown over those steps. No weight differs from the final checkpoint by more than 0.0015.

Pre-training data

The pre-training mix drew on the Open Korean Historical Corpus (Song et al. 2025, arXiv 2510.24541, CC BY-NC 4.0) among its sources, and reproducibility/preprocess_hist_korea_corpus.ipynb in the GitHub repo shows how its copy was made from the Hub files. The paper describes the whole mix.

Intended use

Fine-tuning on downstream tasks, and sentence embeddings that treat Hanja and Hangul spellings of the same text alike.

It is not a chat model. The tokenizer and the chat template are the same as in the other two repos, but pre-training never used the chat tokens (<|system|>, <|user|>, <|assistant|>, <|think|>, <|end_of_turn|>). Their embedding rows are still nearly one shared vector here, as for every other token pre-training never saw. For generation from a prompt use han2han-it or han2han-rl.

Usage

Running the model needs torch and transformers (tested with torch 2.14.1 and transformers 5.18.0 on CPU). The model class ships in this repo, so loading needs trust_remote_code=True. The tokenizer itself runs no custom code, but without the flag AutoTokenizer stops to ask.

output_sentence_embeddings=True returns the mean-pooled encoder states, one 640-dimensional vector per input.

import torch
import torch.nn.functional as F
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

repo = "cadazar/han2han-pt"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True).eval()

texts = [
    "會場을 一巡하고 돌아올 때까지도 日本人側 畵家까지도 一人도 發見할 수가 없었다.",
    "회장을 일순하고 돌아올 때까지도 일본인측 화가까지도 일인도 발견할 수가 없었다.",
    "南畵나 四君子에서는 破墨의 妙法으로 白雪을 象徵할 수 있다.",
]
inputs = tokenizer(texts, return_tensors="pt", padding=True)
with torch.no_grad():
    embeddings = model(**inputs, output_sentence_embeddings=True)[0]
embeddings = F.normalize(embeddings, dim=-1)
print(embeddings @ embeddings.T)
# tensor([[1.0000, 0.9315, 0.8253],
#         [0.9315, 1.0000, 0.7910],
#         [0.8253, 0.7910, 1.0000]])

The first two rows are the same 1920s newspaper sentence in mixed script and in Hangul, and the third is a different sentence.

Tokenizer

tokenizer.json is a tokenizers-library build of the SentencePiece model (spiece.model, kept here as the source), made by scripts/build_hf_tokenizer.py in the GitHub repo. Calling the tokenizer does not add BOS or EOS tokens, and special tokens written into the text are mapped to their ids. It encodes like the SentencePiece wrapper the training code uses, with one known difference. Where two segmentations of a span have exactly the same score (runs of digits, mostly), the same pieces can come out in a different order.

Files

File	Contents
model.safetensors	fp32 weights, 169.2M parameters, plus the jbu / cbu subword bucket tables
config.json, generation_config.json	model and generation config, with auto_map entries for the Auto classes
tokenizer.json, tokenizer_config.json, chat_template.jinja	fast tokenizer (38400 pieces) and chat template, the same as in han2han-it
spiece.model	the SentencePiece model tokenizer.json was built from
modeling_han2han.py, han2han_config.py	modeling code from the GitHub repo at commit af1330e, the same files as in han2han-it

Acknowledgements

Research supported with Cloud TPUs from Google's TPU Research Cloud (TRC). The pre-training run (35B tokens on a TPU v5e-64 slice) used TRC quota.

Citation

Published in Findings of the Association for Computational Linguistics: EMNLP 2026 (Budapest), pages 45857-45876: https://aclanthology.org/2026.findings-emnlp.2146/

@inproceedings{adams-etal-2026-han2han,
    title = "{H}an2{H}an: Efficient Language-Specific Character Representation through Script-Aware Pre-Training for Historical Text Analysis",
    author = "Adams, Cellik  and
      Jo, EunKyoung  and
      Kim, Juae",
    editor = "Lee, Hung-Yi  and
      Dev, Sunipa  and
      Li, Jessy  and
      Elliott, Desmond",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {EMNLP} 2026",
    month = oct,
    year = "2026",
    address = "Budapest, Hungary",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-emnlp.2146/",
    pages = "45857--45876",
    ISBN = "979-8-89176-462-0"
}

License

Apache License 2.0, the same as the GitHub repo.