docs: model card (work in progress)

README.md

@@ -0,0 +1,142 @@
+---
+language:
+- ru
+- en
+- zh
+license: apache-2.0
+base_model: taobao-mnn/MNN-Pocket-0.3B-Base
+library_name: transformers
+pipeline_tag: text-generation
+datasets:
+- HuggingFaceFW/fineweb-2
+- HuggingFaceFW/fineweb
+tags:
+- russian
+- on-device
+- edge
+- mnn
+- qwen3
+- gguf
+- keyboard
+- work-in-progress
+---
+
+# MNN-Pocket-0.3B-RU (work in progress)
+
+> **Status: still training.** No weights in this repository yet. This card documents the recipe and the
+> measurements taken so far; checkpoints live in the private repo `Rahspide/mnn-pocket-ru-run`
+> (`ckpt/state.pt`). Sections marked *(pending)* will be filled when training finishes.
+
+A continued-pretraining run of [`taobao-mnn/MNN-Pocket-0.3B-Base`](https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B-Base)
+on Russian, aimed at on-device next-word and short-phrase suggestions for an Android keyboard.
+
+The goal is not instruction following. The model is trained as a plain causal LM on raw Russian text, so it
+continues a typed prefix instead of answering it. English and Chinese are kept alive with a 25% replay mix.
+
+| | |
+|---|---|
+| Base | `taobao-mnn/MNN-Pocket-0.3B-Base` (Qwen3 architecture, 303M non-embedding / 361M total params) |
+| Tokenizer | 65,536 tokens, byte-level BPE, `<|endoftext|>` = 65510 |
+| Training tokens | 1.2B (0.9B ru + 0.204B en + 0.096B zh) |
+| Method | full fine-tune, 1 epoch, no LoRA |
+| Hardware | single Colab T4 16 GB, 4.8k tok/s, ~69 h total |
+| Progress (2026-11-09) | step ~1,100 / 36,621, ~36M tokens |
+
+## Why the recipe looks like this: Russian is expensive in this tokenizer
+
+The base tokenizer is a distilled 65k vocabulary with almost no Cyrillic merges. Russian comes out
+close to character level — `Спасибо` becomes 6 tokens:
+
+```
+"Спасибо" -> ['С', 'п', 'аÑģ', 'и', 'б', 'о']
+```
+
+Measured on prose samples with the real tokenizer:
+
+| Tokenizer | RU chars/token | RU tokens per word | EN chars/token |
+|---|---|---|---|
+| MNN-Pocket / Qwen3-distilled (65,536) | **1.75–1.85** | **≈3.3** | 3.78 |
+| ruGPT3-small (50k, Russian BPE) | 4.2 | ≈1.3 | — |
+
+Consequences that shaped the plan:
+
+- the same 1.2B-token budget buys ~2.2B Russian characters (~340M words), roughly 2.2× less text than
+  a Russian-native tokenizer would fit;
+- on a phone, one suggested Russian word costs ~3 decode steps instead of ~1, which matters for a
+  keyboard latency budget;
+- vocabulary surgery (adding a few thousand Cyrillic merges plus an embedding resize) is a separate
+  experiment, not part of v1. v1 stays tokenizer-compatible with the base model so it drops into the
+  existing MNN / GGUF toolchain.
+
+## Training recipe
+
+- Full fine-tune from `MNN-Pocket-0.3B-Base`, fp32 master weights with fp16 autocast (T4 has no usable bf16).
+- Sequence length 1024, random windows over a packed `uint16` token stream, `<|endoftext|>` between documents.
+- 32,768 tokens per optimizer step (micro-batch 4 × 1024, gradient accumulation 8).
+- AdamW, β = (0.9, 0.95), weight decay 0.1, gradient clip 1.0, lr 2e-4 with 1% linear warmup and cosine decay.
+- One epoch over 1.2B tokens = 36,621 steps.
+- Checkpoint every 500 steps, uploaded to a private HF repo; training auto-resumes from the last checkpoint
+  after a session restart.
+
+## Data
+
+All sources are public Hugging Face datasets. `*` marks the validation split: the first 100k tokens of each
+stream are held out before training starts, so validation text is never seen.
+
+| Language | Dataset | Config | Tokens | Share |
+|---|---|---|---|---|
+| ru | [`HuggingFaceFW/fineweb-2`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) | `rus_Cyrl` | 0.90B (100k held out*) | 75% |
+| en | [`HuggingFaceFW/fineweb`](https://huggingface.co/datasets/HuggingFaceFW/fineweb) | `sample-100BT` | 0.204B (100k held out*) | 17% |
+| zh | [`HuggingFaceFW/fineweb-2`](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) | `cmn_Hani` | 0.096B (100k held out*) | 8% |
+
+Licences: FineWeb / FineWeb-2 are ODC-BY 1.0. No scraped Telegram dumps or other unlicensed chat corpora
+were used, so the resulting weights stay publishable.
+
+## Evaluation
+
+Per language: perplexity, **bits per character**, and top-1 / top-5 next-token accuracy. Bits/char is the
+metric that stays comparable across tokenizers, which perplexity is not.
+
+Baseline, the untouched base model (measured 2026-11-09, 100k held-out tokens per language):
+
+| Lang | ppl | bits/char | top-1 | top-5 | chars/token |
+|---|---|---|---|---|---|
+| ru | 10.42 | 1.625 | 46.2% | 69.3% | 2.08 |
+| en | 21.85 | 1.016 | 39.9% | 64.1% | 4.40 |
+| zh | 23.95 | 3.274 | 41.9% | 62.0% | 1.42 |
+
+Fine-tuned numbers: *(pending)*. The acceptance gate is a clear drop in Russian bits/char with at most a
+~10% regression on English and Chinese bits/char.
+
+## Intended use
+
+- On-device next-word / short-phrase suggestion in a Russian keyboard.
+- Any task that needs a ~200 MB quantised causal LM that continues Russian text.
+
+Not intended for: instruction following, factual question answering, code, long-form reasoning. This is a
+0.36B model trained for 1.2B tokens; its knowledge is narrow by construction.
+
+## Limitations
+
+- **Work in progress.** No quality claims until the run finishes and the table above is filled.
+- Russian is tokenised near character level (see above): slower decoding and a larger effective cost per
+  Russian word than the same model pays for English.
+- Web text only. Register is mostly written/informal blog Russian; no dialogue corpora, no post-training.
+- Capacity is 0.36B. Expect weak facts, weak arithmetic, and drift on long generations.
+- English and Chinese are expected to regress somewhat; the 25% replay mix limits but does not remove this.
+- The ChatML template from the base model is inherited but untrained here — do not use this checkpoint in
+  chat mode.
+
+## Licence and attribution
+
+Weights: Apache-2.0, inherited from the base model. Data: ODC-BY 1.0 (FineWeb, FineWeb-2).
+If you use this model, credit the base release:
+
+```bibtex
+@misc{mnn_pocket_03b,
+  title  = {MNN-Pocket-0.3B},
+  author = {Taobao MNN team},
+  year   = {2026},
+  url    = {https://huggingface.co/taobao-mnn/MNN-Pocket-0.3B}
+}
+```