zxc0zxc0zxc/gigachat-nanai-extended-tokenizer

GigaChat Nanai Extended Tokenizer

This repository contains an extended tokenizer built on top of ai-sage/GigaChat3-10B-A1.8B-base using the zxc0zxc0zxc/nanai-language dataset.

The goal of this tokenizer is to improve tokenization quality for Nanai while staying close to the original GigaChat tokenizer setup.

Base model

This tokenizer is based on ai-sage/GigaChat3-10B-A1.8B-base, which is the * base/pretrain* GigaChat 3 model. According to the model card, it is a Mixture-of-Experts model with 10B total parameters and 1.8B active parameters, and the instruct version is published separately.

Dataset

Tokenizer adaptation experiments were run using the dataset zxc0zxc0zxc/nanai-language, which is labeled on Hugging Face as a Russian + Nanai dataset for low-resource language work, dictionary data, bilingual lexicon extraction, tokenizer training, and pretraining-oriented corpus preparation.

What “extended” means

This repository contains an extended tokenizer, not a fully retrained tokenizer.

That means:

• the original GigaChat tokenizer was kept as the base
• additional vocabulary items were added for better Nanai coverage
• this variant is intended to be the more practical option for downstream adaptation of the original model

Files

Tokenizer files

• special_tokens_map.json
• tokenizer.json
• tokenizer_config.json

Artifacts

The artifacts/ directory contains evaluation files used during tokenizer comparison:

• baseline_tokenizer_metrics.csv
• final_tokenizer_metrics.csv
• focus_tokenization_examples.csv
• tokenization_examples.csv
• tokenizer_experiment_summary.json

Images

The img/ directory contains before/after comparison plots for tokenization quality.

Comparison plots

Average tokenization metrics

Base	Final

Single-token-rate comparison

Base	Final

Intended use

This tokenizer is intended for:

• continued pretraining of GigaChat on Nanai-enriched corpora
• tokenizer experiments for low-resource language adaptation
• comparison against the original GigaChat tokenizer
• lexical and orthographic coverage improvements for Nanai

Notes

This repository should be treated as an experimental tokenizer adaptation artifact.

It is most useful in workflows where:

• you want to preserve compatibility with the original tokenizer structure as much as possible
• you want better segmentation for Nanai words and dictionary-derived text
• you plan to continue pretraining or adapt the original model with an extended vocabulary

Example

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("zxc0zxc0zxc/gigachat-nanai-extended-tokenizer")

text = "дёан биа"
print(tokenizer.tokenize(text))

Acknowledgements

Base tokenizer/model: ai-sage/GigaChat3-10B-A1.8B-base