Blame for README.md

---
language:
- ru
- en
- zh
license: apache-2.0
base_model: taobao-mnn/MNN-Pocket-0.3B-Base
library_name: transformers
pipeline_tag: text-generation
datasets:
- HuggingFaceFW/fineweb-2
- HuggingFaceFW/fineweb
tags:
- russian
- on-device
- edge
- mnn
- qwen3
- gguf
- keyboard
- work-in-progress
---

![MNN-Pocket-0.3B-RU — a Russian continuation model on a 0.3B base](assets/cover.png)

![params](https://img.shields.io/badge/params-361M-E2543F?style=flat-square)
![tokens](https://img.shields.io/badge/trained%20on-1.2B%20tokens-5FD3B4?style=flat-square)
![context](https://img.shields.io/badge/context-32k%20(base)-6FA8FF?style=flat-square)
![gguf](https://img.shields.io/badge/gguf-Q8__0%20%C2%B7%20Q4__K__M-E2543F?style=flat-square)
![license](https://img.shields.io/badge/license-apache--2.0-8B95A4?style=flat-square)
![status](https://img.shields.io/badge/status-v1%20released-5FD3B4?style=flat-square)

> **Status: v1 released.** 1.2 B tokens, one epoch, weights in this repository, GGUF in `gguf/`.
> Training log, checkpoints and the evaluation script live in the private repo
> [`Rahspide/mnn-pocket-ru-run`](https://huggingface.co/Rahspide/mnn-pocket-ru-run).

A continued-pretraining run of [`taobao-mnn/MNN-Pocket-0.3B-Base`](https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B-Base)
that teaches it Russian while keeping English and Chinese readable. The keyboard is the use case that set the
size and latency budget.

It is trained as a plain causal LM on raw text, so it **continues** a typed prefix instead of answering it.
For dialogue there is a separate checkpoint — [`Rahspide/MNN-Pocket-0.3B-RU-Chat`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU-Chat).

## Two models, two jobs

![One base, two jobs](assets/two_jobs.png)

| | This model — `MNN-Pocket-0.3B-RU` | Sister — `MNN-Pocket-0.3B-RU-Chat` |
|---|---|---|
| Job | continue text | follow instructions |
| Input | a prefix, often mid-sentence | a system + user message |
| Output | the plausible continuation | an answer |
| Trained with | continued pretraining, 1.2 B tokens of raw text | SFT on top of this model, 50 M tokens |
| Choose it for | keyboard suggestions, autocomplete, prefix completion | chat, Q&A, short rewrites |
| Do not choose it for | assistant behaviour: it continues your question instead of answering | autocomplete: it answers instead of continuing |

The two tasks reward opposite behaviour. Trained on both jobs, one checkpoint does each badly: it answers when
it should continue, and continues when it should answer.

## Specs

| | |
|---|---|
| Base | `taobao-mnn/MNN-Pocket-0.3B-Base` (Qwen3 architecture, 303 M non-embedding / 361 M total) |
| Tokenizer | 65,536 tokens, byte-level BPE, `<|endoftext|>` = 65510 |
| Training tokens | 1.2 B — 0.9 B ru + 0.204 B en + 0.096 B zh |
| Method | full fine-tune, 1 epoch, no LoRA |
| Steps | 36,621 × 32,768 tokens |
| Hardware | one Colab T4 16 GB, 4.8 k tok/s, ~69 h |
| Precision | fp32 master weights, fp16 autocast (T4 has no usable bf16) |

## Results

![Bits per character: base vs v1](assets/eval.png)

100 k held-out tokens per language, identical settings for both checkpoints:

| Lang | ppl base → v1 | bits/char base → v1 | top-1 base → v1 | top-5 base → v1 |
|---|---|---|---|---|
| **ru** | 10.42 → **5.05** | 1.625 → **1.123** (−31%) | 46.2% → **61.0%** | 69.3% → **80.6%** |
| en | 21.85 → 41.16 | 1.016 → 1.225 (+21%) | 39.9% → 35.5% | 64.1% → 58.7% |
| zh | 23.95 → 51.86 | 3.274 → 4.071 (+24%) | 41.9% → 37.7% | 62.0% → 55.6% |

The published weights were re-downloaded and re-evaluated independently (CPU, fp32, 20 k tokens per language)
and reproduce the same ordering: ru 1.022 against base 1.519, en 1.337 against 1.124, zh 4.053 against 3.393.

Russian landed past the target (1.123 against a 1.40 gate). English and Chinese regressed 21–24% instead of
the ≤10% the recipe aimed for. a 25% replay mix is too
thin at 0.36 B when the target language is tokenised this badly. A second pass with a 50/25/25 mix at lr 5e-5
was started, got through ~50 M of a planned 200 M tokens, and changed nothing measurable on English
(1.337 → 1.337 on the same eval), so it is **not** published. Treat v1 as Russian-first, with working but
measurably weaker English and Chinese.

## Why the tokenizer shapes the recipe

![Russian characters per token, base vs extended vocabulary](assets/tokenizer.png)

The base vocabulary is a distilled 65 k set with almost no Cyrillic merges, and it is byte-level: the
vocabulary stores UTF-8 bytes remapped to latin-1, so `convert_ids_to_tokens` prints `С` where the text says
`С`. That is a display artifact; `decode` returns the word. The count is what matters:

| Word | Characters | Tokens | Pieces |
|---|---|---|---|
| `Спасибо` | 7 | 6 | `С · п · ас · и · б · о` |
| `привет` | 6 | 3 | `пр · ив · ет` |
| `покупать` | 8 | 4 | `п · ок · уп · ать` |
| `Ёжик` | 4 | 4 | `Ё` lands in two separate byte tokens |

Because the vocabulary is byte-level, a token can end in the middle of a character: in `Ёжик` and `ЩУКА` the
uppercase letters are cut into single bytes. Decoding the whole sequence is still exact, but a streaming client
must hold an incomplete UTF-8 sequence back instead of printing it. `llama.cpp` and `transformers` buffer it for
you; a hand-rolled decoder has to do the same or the output shows replacement characters.

| Tokenizer | RU chars/token | RU tokens per word | EN chars/token |
|---|---|---|---|
| MNN-Pocket / Qwen3-distilled (65,536) | **1.75–1.85** (2.08 on dialogue text) | **≈3.3** | 3.78 |
| +8,192 Cyrillic merges (73,728) | **3.10** | ≈1.8 | — |
| +16,384 Cyrillic merges (81,920) | **3.39** | ≈1.6 | — |
| ruGPT3-small (50 k, Russian BPE) | 4.2 | ≈1.3 | — |

Consequences that shaped the plan:

- the same 1.2 B-token budget buys ~2.2 B Russian characters (~340 M words), roughly 2.2× less text than a
  Russian-native tokenizer would fit;
- on a phone one suggested Russian word costs ~3 decode steps instead of ~1, which matters for a keyboard
  latency budget;
- vocabulary surgery was measured but kept out of v1, so v1 stays tokenizer-compatible with the base and drops
  into the existing MNN / GGUF toolchain. An extended-vocabulary artefact (`model_ext`, 73,728 rows, embeddings
  initialised as the mean of their child tokens) exists in the private run repo but is not part of this release.

## Recipe

- Full fine-tune from `MNN-Pocket-0.3B-Base`, fp32 master weights with fp16 autocast.
- Sequence length 1024, random windows over a packed `uint16` token stream, `<|endoftext|>` between documents.
- 32,768 tokens per optimizer step (micro-batch 4 × 1024, gradient accumulation 8).
- AdamW, β = (0.9, 0.95), weight decay 0.1, gradient clip 1.0, lr 2e-4 with 1% linear warmup and cosine decay.
- One epoch over 1.2 B tokens = 36,621 steps.
- Checkpoint every 500 steps to a private HF repo; training auto-resumes from the last checkpoint after a
  session restart.

## Data

All sources are public Hugging Face datasets. `*` marks the validation split: the first 100 k tokens of each
stream are held out before training starts, so validation text is never seen.

| Language | Dataset | Config | Tokens | Share |
|---|---|---|---|---|
| ru | [`HuggingFaceFW/fineweb-2`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) | `rus_Cyrl` | 0.90 B (100 k held out*) | 75% |
| en | [`HuggingFaceFW/fineweb`](https://huggingface.co/datasets/HuggingFaceFW/fineweb) | `sample-100BT` | 0.204 B (100 k held out*) | 17% |
| zh | [`HuggingFaceFW/fineweb-2`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) | `cmn_Hani` | 0.096 B (100 k held out*) | 8% |

Licences: FineWeb / FineWeb-2 are ODC-BY 1.0. No scraped Telegram dumps or other unlicensed chat corpora were
used, so the resulting weights stay publishable.

## Builds

![GGUF sizes](assets/sizes.png)

| Build | Size | Download | Note |
|---|---|---|---|
| Q8_0 | 310 MB | [`MNN-Pocket-0.3B-RU-Q8_0.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU/blob/main/gguf/MNN-Pocket-0.3B-RU-Q8_0.gguf) | near-lossless, for desktop |
| Q4_K_M | 223 MB | [`MNN-Pocket-0.3B-RU-Q4_K_M.gguf`](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU/blob/main/gguf/MNN-Pocket-0.3B-RU-Q4_K_M.gguf) | the size to ship on a phone |
| MNN INT4 | 165 MB | [mnn/MNN-Pocket-0.3B-RU-INT4](https://huggingface.co/Rahspide/MNN-Pocket-0.3B-RU/tree/main/mnn/MNN-Pocket-0.3B-RU-INT4) | MNN runtime, Android |

Both files sit under `gguf/` with the quant tag in the name, which is what llama.cpp matches on:

```bash
llama-cli -hf Rahspide/MNN-Pocket-0.3B-RU:Q4_K_M -p "Сегодня я" -n 40
```

Converted with llama.cpp `convert_hf_to_gguf.py` (f16) and quantised with `llama-quantize`. Measured on a
16-thread x86 CPU, no GPU involved:

| Build | Prompt processing | Generation |
|---|---|---|
| Q4_K_M | ~410 tok/s | **~130 tok/s** |
| Q8_0 | ~525 tok/s | ~99 tok/s |

Phone figures will be several times lower; these are desktop-CPU numbers.

The MNN build runs through MNN's own runtime. Build `llm_demo` from the MNN repository with
`-DMNN_BUILD_LLM=ON`, then point it at the folder:

```bash
cd mnn/MNN-Pocket-0.3B-RU-INT4 && llm_demo config.json prompt.txt
```

165 MB on disk, ~354 tok/s prefill and ~77 tok/s decode on 12 CPU threads here. `llm_config.json` carries no
chat template on purpose: this checkpoint continues the prompt, so the runtime should not wrap it in a dialogue
format.

Two conversion quirks, both already fixed in this repository's `tokenizer_config.json`.

- the base pre-tokenizer does not hash-match in current llama.cpp, so the Qwen pre-tokenizer has to be pinned
  by hand (`qwen2`);
- `extra_special_tokens` must not be a list, or transformers v4 refuses to load the tokenizer.

## Reproduce

Everything below runs on CPU. The numbers in this card came from these paths.

```bash
# bits/char, ppl and top-k against the base, 20k tokens per language
python bench/eval_bpc.py --model Rahspide/MNN-Pocket-0.3B-RU --base taobao-mnn/MNN-Pocket-0.3B-Base \
    --dataset HuggingFaceFW/fineweb-2 --config rus_Cyrl --tokens 20000

# throughput and a smoke generation on the quantised build
LLAMA_CLI=/path/to/llama-cli bash bench/bench_cpu.sh gguf/MNN-Pocket-0.3B-RU-Q4_K_M.gguf
```

`bench/eval_bpc.py` reports the same three metrics as the tables above (ppl, bits/char, top-1/top-5) and exits
non-zero when the model loses more than a given `--max-bpc-regression` against the base, so it can run as a
regression gate in CI. `bench/bench_cpu.sh` prints prompt and generation tok/s and checks that generation stops
at `<|endoftext|>`.

## Intended use

- On-device next-word and short-phrase suggestion in a keyboard or IME, Russian first, English and Chinese from
  the same checkpoint (a keyboard can pick the language from the current word or the layout).
- Text completion, prefix continuation and autocomplete for Russian anywhere a ~220 MB quantised model has to
  run locally: chat composers, note apps, search fields, terminal prompts.
- Any pipeline that needs a small causal LM fluent in Russian that can still read English and Chinese, for
  example tagging, short rewriting suggestions, or generating Russian seed text.

Not intended for: instruction following, factual question answering, code, long-form reasoning, or translation.
This is a 0.36 B model trained for 1.2 B tokens; its knowledge is narrow by construction. The language mix is a
deliberate trade: Russian is the target, English and Chinese are kept from degrading, neither is improved.

## Limitations

- Russian is tokenised near character level (see above): slower decoding and a larger effective cost per Russian
  word than the same model pays for English.
- Token boundaries do not follow character boundaries. Rare and uppercase Cyrillic letters are split into
  single bytes, so a streaming decoder must buffer partial UTF-8 sequences; `llama.cpp` and `transformers`
  already do this.
- Web text only. Register is mostly written/informal blog Russian; no dialogue corpora, no post-training.
- Capacity is 0.36 B. Expect weak facts, weak arithmetic, and drift on long generations.
- English and Chinese regressed 21–24% against the base. The 25% replay mix limits the damage; it does not
  remove it.
- The ChatML template from the base is inherited but untrained here. Do not use this checkpoint in chat mode.

## Releases

| Date | Version | What changed |
|---|---|---|
| 2026-11-10 | **v1.0** | 1.2 B-token continued pretraining released: bf16 weights, GGUF Q8_0 and Q4_K_M, measured ru/en/zh bits-per-char |
| 2026-11-11 | v1.1 | MNN INT4 build for the MNN runtime: 165 MB, ~77 tok/s decode on 12 CPU threads |
| 2026-11-10 | v2.0 (sister) | Assistant checkpoint released: 50 M-token SFT on top of v1, weights and GGUF |

## Licence and attribution

Weights: Apache-2.0, inherited from the base model. Data: ODC-BY 1.0 (FineWeb, FineWeb-2).
If you use this model, credit the base release:

```bibtex
@misc{mnn_pocket_03b,
  title  = {MNN-Pocket-0.3B},
  author = {Taobao MNN team},
  year   = {2026},
  url    = {https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B}
}
```