genmai-90M / README.md
subdiox's picture
v1.0.1: layer norms safe in fp16 (macOS 26 computed their variance in fp16: garbage output)
fd4f82e verified
|
Raw History Blame Contribute Delete
7.36 kB
---
license: cc-by-sa-4.0
language: ja
base_model: Miwa-Keita/zenz-v3.2-small-gguf
pipeline_tag: text-generation
tags:
- japanese
- input-method
- kana-kanji-conversion
---
# genmai-90M
A small Japanese **kana-kanji conversion** model: it converts a reading (katakana) into Japanese text, optionally
conditioned on the text before the cursor. It is the light-weight variant of
[subdiox/genmai-310M](https://huggingface.co/subdiox/genmai-310M), distilled from it, meant for devices with less
memory (e.g. iPhone). It uses the same prompt format, tokenizer and file layout as genmai-310M.
- Architecture: GPT-2, character-level vocabulary (6,000), 12 layers, 768 hidden, 90M parameters
- Base model: zenz-v3.2-small ([Miwa-Keita/zenz-v3.2-small-gguf](https://huggingface.co/Miwa-Keita/zenz-v3.2-small-gguf),
Apache-2.0), dequantized from the Q5_K_M GGUF
- Teacher: genmai-310M
## Files
Releases are git tags (`v1.0.1`); use a tag to pin a version. `main` is the latest release.
| file | contents |
|---|---|
| `model.safetensors` | fp32 weights in the Hugging Face GPT-2 layout (362 MB), for transformers |
| `config.json` | GPT-2 config; the prompt format is recorded as `"genmai_prompt_format": "zenzlc"` |
| `tokenizer.json`, `tokenizer_config.json` | the tokenizer |
| `model-coreml.zip` | what the genmai input method runs: the model as a stateful Core ML package for the Apple Neural Engine (82 MB; see below): `genmai.mlpackage` (int8 weights) and `genmai_coreml.json` (its static shapes: `format` `genmai-coreml-1`, `prompt_slots` 256, `output_steps` 64, `rows` 5, `prefill_chunk` 32) |
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("subdiox/genmai-90M", revision="v1.0.1")
tok = AutoTokenizer.from_pretrained("subdiox/genmai-90M", revision="v1.0.1")
x = tok("<s>\uee00" + "ใƒใƒใƒใƒใ‚ฌใ‚คใ‚ฟใ‚คใƒฉใ‚ทใ‚ค" + "\uee01", return_tensors="pt", add_special_tokens=False)
y = m.generate(**x, num_beams=5, num_return_sequences=5, length_penalty=0.0, max_new_tokens=64)
print([tok.decode(s[x.input_ids.shape[1]:], skip_special_tokens=True) for s in y]) # ๆฏใฏๆญฏใŒ็—›ใ„ใ‚‰ใ—ใ„, ...
```
Rank beams by the plain sum of log-probabilities (`length_penalty=0.0`): with transformers' default length
normalisation, long degenerate outputs such as ใƒใƒใƒใƒใ‚ฌใ‚คใ‚ฟใ‚คใ‚‰ใ—ใ„ can win.
## Prompt format (`zenzlc`)
Left context first (up to the last 500 characters), then the reading, then the output:
```
<s>[U+EE02]{left context}[U+EE00]{reading in katakana}[U+EE01]{output}</s>
```
`[U+EE0x]` are the private-use marker characters U+EE02 (context), U+EE00 (input) and U+EE01 (output).
Without context the `[U+EE02]{left context}` part is omitted. Spaces are written as U+3000 and newlines are
removed. Decode with beam search (5 beams) and stop at EOS; long readings are split at ใ€ใ€‚ into chunks of at most
40 characters, each decoded after the previous output.
## Core ML (Apple Neural Engine)
`genmai.mlpackage` in `model-coreml.zip` is the same model converted with coremltools 9 (macOS 15 / iOS 18 or later) for the
genmai input method's Core ML decoder. It is one multifunction package with two functions that share one set of
states (per layer, the keys and values in fp16):
- `prefill`: `ids`, `pos` (1, 32) int32, `start` (1,) int32, `mask` (32, 256) fp16 additive โ†’ `logp` (32, 6000):
writes 32 prompt tokens to prompt slots `start ..< start + 32` and returns the log-probabilities after each.
- `decode`: `tok` (5, 1) int32, `pos` (1,) int32, `wsel` (320, 5) / `keep` (320, 1) fp16, `pmask` (1, 256) /
`smask` (5, 320) fp16 additive โ†’ `logp` (5, 6000): one token for each of 5 beams. Beam r's token of step t is
stored at output slot t ยท 5 + r (`wsel` is the one-hot placement, `keep` = 1 โˆ’ its row sums); each beam attends to
the prompt slots in `pmask` and to its ancestors' output slots in `smask`, so reordering beams copies nothing.
Linear weights are int8 (symmetric, per channel), the rest fp16. Prompts are limited to 256 tokens and outputs to
64 tokens per chunk. The conversion script is `Tools/coreml/genmai_coreml.py` of
[genmai-ime](https://github.com/subdiox/genmai-ime).
Layer norms divide their input by its maximum magnitude first (epsilon divided by its square; exact): the first
token carries a residual value of about 5000, whose square exceeds fp16, and macOS 26 computes a layer norm's
variance in fp16 (v1.0.0 gave garbage there; fixed in v1.0.1).
On an M4 Pro (5 beams): the same top-1 as the weights in 8 bits on the GPU for 101 / 101 everyday sentences,
199 / 200 AJIMEE items and 27 / 28 hard sentences; about 4 ms per decode step on the Neural Engine; process footprint
about 50โ€“90 MB. The first load compiles the model for
the Neural Engine (about 20 s; macOS caches the result).
## Training
Knowledge distillation from genmai-310M: at every output position the loss is
0.5 ยท cross-entropy to the reference + 0.5 ยท KL divergence to the teacher's next-character distribution (both models
share the vocabulary), plus an auxiliary next-character language-model loss (weight 0.3) on the text before the
cursor. Stages:
1. ~3M automatically read sentences (Wikipedia, FineWeb-2 Japanese, Aozora Bunko, Diet minutes) and sentences
containing rare Mozc dictionary words;
2. ~3.2M further Wikipedia / FineWeb-2 sentences;
3. final stage on curated data (hand-checked items, synthetic and homophone sentences generated with Claude,
LLM-judged corrections) with replay of the earlier data.
## Evaluation
Top-1 accuracy, beam size 5; candidates whose reading does not match the input are dropped. AJIMEE, everyday
sentences and ime_v1 dev are judged leniently (orthographic variants accepted); homophones = word-level accuracy
on held-out homophone sentences; hard = 28 sentences written to stress an IME (exact match up to orthography).
| set | genmai-90M (bf16) | genmai-310M (bf16) | Zenzai (zenz-v3.2-small, azooKey settings) |
|---|---|---|---|
| AJIMEE-Bench (200) | 91.0 % | 95.0 % | 85.5 % |
| everyday sentences (101) | 98.0 % | 98.0 % | 97.0 % |
| ime_v1 dev (544) | 94.3 % | 93.6 % | โ€“ |
| homophone sentences (1,607) | 89.6 % | 93.1 % | โ€“ |
| average of the four | 93.2 % | 94.9 % | โ€“ |
| hard IME sentences (28) | 22 / 28 | 22 / 28 | โ€“ |
Speed (RTX 3090, bf16, 5 beams, everyday sentences): median 18 ms per conversion vs. 36 ms for genmai-310M. In the
input method (Core ML, M4 Pro), the conversion key answers in a median 16 / 22 ms (everyday sentences / AJIMEE).
## Limitations
- Compared with genmai-310M it is weaker on knowledge-heavy text (rare proper nouns, technical terms) and on fine
homophone distinctions (about 3โ€“4 points lower on AJIMEE and the homophone set).
- Readings that repeat one sound (ใใ—ใ‚ƒใฎใใ—ใ‚ƒใŒใใ—ใ‚ƒใงใใ—ใ‚ƒใ—ใŸ) tend to map every occurrence to the same word.
- The model copies the left context strongly: a wrong conversion earlier in the context can be repeated.
## License and attribution
CC BY-SA 4.0. Base model zenz-v3.2-small by Miwa-Keita (Apache-2.0). The training data include Wikipedia
(CC BY-SA), FineWeb-2 (ODC-By), Aozora Bunko, the National Diet minutes, and the Mozc dictionary (BSD-3-Clause,
Copyright Google Inc.; contains data derived from IPAdic, Copyright Nara Institute of Science and Technology).