genmai 1.0.0

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,127 @@
+---
+license: cc-by-sa-4.0
+language: ja
+base_model: Miwa-Keita/zenz-v2.5-medium
+pipeline_tag: text-generation
+tags:
+  - japanese
+  - input-method
+  - kana-kanji-conversion
+---
+# genmai-310M
+
+A Japanese **kana-kanji conversion** model: it converts a reading (katakana) into Japanese text, optionally
+conditioned on the text before the cursor. It is the AI converter of the macOS input method
+[genmai](https://github.com/subdiox/genmai-ime), which runs it on device with Core ML (`model-coreml.zip`).
+
+- Architecture: GPT-2, character-level vocabulary, 24 layers, 1024 hidden, 310M parameters
+- Smaller variant (90M, distilled from this model, about twice as fast): [subdiox/genmai-90M](https://huggingface.co/subdiox/genmai-90M)
+- Base model: [Miwa-Keita/zenz-v2.5-medium](https://huggingface.co/Miwa-Keita/zenz-v2.5-medium) (CC BY-SA 4.0)
+
+## Files
+
+Releases are git tags (`v1.0.0`); use a tag to pin a version. `main` is the latest release.
+
+| file | contents |
+|---|---|
+| `model.safetensors` | fp32 weights in the Hugging Face GPT-2 layout (1.2 GB), for transformers |
+| `config.json` | GPT-2 config; the prompt format is recorded as `"genmai_prompt_format": "zenzlc"` |
+| `tokenizer.json`, `tokenizer_config.json` | the tokenizer |
+| `model-coreml.zip` | what the genmai input method runs: the model as stateful Core ML packages for the Apple Neural Engine (283 MB; see below): `genmai_0.mlpackage` (layers 0–11) and `genmai_1.mlpackage` (layers 12–23 and the head), int8 weights, and `genmai_coreml.json` (static shapes: `format` `genmai-coreml-1`, `prompt_slots` 256, `output_steps` 64, `rows` 5, `prefill_chunk` 32, `chunks` 2) |
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+m = AutoModelForCausalLM.from_pretrained("subdiox/genmai-310M", revision="v1.0.0")
+tok = AutoTokenizer.from_pretrained("subdiox/genmai-310M", revision="v1.0.0")
+x = tok("<s>\uee00" + "ハハハハガイタイラシイ" + "\uee01", return_tensors="pt", add_special_tokens=False)
+y = m.generate(**x, num_beams=5, num_return_sequences=5, length_penalty=0.0, max_new_tokens=64)
+print([tok.decode(s[x.input_ids.shape[1]:], skip_special_tokens=True) for s in y])  # 母は歯が痛いらしい, ...
+```
+
+Rank beams by the plain sum of log-probabilities (`length_penalty=0.0`): with transformers' default length
+normalisation, long degenerate outputs such as ハハハハガイタイらしい can win.
+
+## Prompt format (`zenzlc`)
+
+Left context first (up to the last 500 characters), then the reading, then the output:
+
+```
+<s>[U+EE02]{left context}[U+EE00]{reading in katakana}[U+EE01]{output}</s>
+```
+
+`[U+EE0x]` are the private-use marker characters U+EE02 (context), U+EE00 (input) and U+EE01 (output).
+Without context the `[U+EE02]{left context}` part is omitted. Spaces are written as U+3000 and newlines are
+removed. `config.json` records the format name (`genmai_prompt_format`). Decode with beam search (genmai uses 5 beams) and stop at EOS;
+long readings are split at 、。 into chunks of at most 40 characters, each decoded after the previous output.
+
+## Core ML (Apple Neural Engine)
+
+`model-coreml.zip` holds the same model converted with coremltools 9 (macOS 15 / iOS 18 or later) for the genmai
+input method's Core ML decoder. A package with all 24 layers does not load on the Neural Engine, so the model is
+split into two packages; each has two functions that share the package's states (per layer, keys and values in
+fp16):
+
+- `prefill`: 32 prompt tokens written to prompt slots `start ..< start + 32` (`mask` (32, 256) additive). Inputs of
+  `genmai_0`: `ids`, `pos` (1, 32) int32, `start` (1,) int32, `mask`; of `genmai_1`: `hidden` (1, 32, 1024) fp16 (the
+  output of `genmai_0`), `start`, `mask`. Outputs: `hidden` (`genmai_0`), `logp` (32, 6000) (`genmai_1`).
+- `decode`: one token for each of 5 beams. Inputs of `genmai_0`: `tok` (5, 1) int32, `pos` (1,) int32; of
+  `genmai_1`: `hidden` (5, 1, 1024) fp16; both: `wsel` (320, 5) / `keep` (320, 1), `pmask` (1, 256) / `smask` (5, 320)
+  fp16 additive. Beam r's token of step t is stored at output slot t · 5 + r (`wsel` is the one-hot placement, `keep`
+  = 1 − its row sums); each beam attends to the prompt slots in `pmask` and to its ancestors' slots in `smask`.
+
+The Neural Engine computes in fp16 (at most 65504), which this model exceeds: its residual stream reaches about 81k
+and one attention head of the second layer has logits up to about 500k. The packages are reparametrized exactly to stay in
+range: the residual stream is scaled by 0.25 (embeddings and the attention / MLP output projections × 0.25, layer
+norm epsilon × 1/16; the head uses the unscaled embeddings), and attention logits are computed / 256 (folded into the
+query weights), the row maximum subtracted and the result × 256 before the softmax. Linear weights are int8
+(symmetric, per channel), the rest fp16. The conversion script is `Tools/coreml/genmai_coreml.py` of
+[genmai-ime](https://github.com/subdiox/genmai-ime) (`--chunks 2 --residual-scale 0.25 --score-scale 256`).
+
+On an M4 Pro (5 beams): top-1 accuracy (exact) 94.1 / 93.0 / 78.6 % on everyday sentences / AJIMEE / hard sentences
+(the weights in 8 bits on the GPU: 93.1 / 92.5 / 78.6 %, top-1 identical on 96–98.5 % of the items); process
+footprint 80–140 MB; a whole everyday sentence converts in a median 113 ms, the input method's conversion key (which
+converts while you type) answers in a median 30 / 39 ms (everyday sentences / AJIMEE). The first load compiles the model for the Neural Engine
+(about 20 s; macOS caches the result).
+
+## Training
+
+Fine-tuned from zenz-v2.5-medium in several stages on (reading, text) pairs:
+
+1. ~6M automatically read sentences from Japanese Wikipedia, FineWeb-2 (Japanese), Aozora Bunko and Diet
+   minutes (readings by UniDic with corrections learned from reviewed samples);
+2. long left context (up to 500 characters);
+3. reading/surface pairs of the Mozc dictionary (dictionary_oss) and sentences containing rare dictionary words;
+4. curated data: hand-checked items, synthetic sentences and homophone sets generated with Claude, and
+   LLM-judged corrections, with replay of the earlier data;
+5. training with an auxiliary next-character language-model loss on the real text before the cursor (so the
+   model learns how words are used in running text, not only the conversions).
+
+The release is a single model after one more round of stage 5 on ~1M further pairs.
+
+## Evaluation
+
+Top-1 accuracy, beam size 5, bf16; candidates whose reading does not match the input are dropped. AJIMEE, everyday
+sentences and ime_v1 dev are judged leniently (orthographic variants accepted); homophones = word-level accuracy
+on held-out homophone sentences; hard = 28 sentences written to stress an IME (exact match up to orthography).
+
+| set | genmai-310M (bf16) | Zenzai (zenz-v3.2-small, azooKey settings) |
+|---|---|---|
+| AJIMEE-Bench (200) | 95.0 % | 85.5 % |
+| everyday sentences (101) | 98.0 % | 97.0 % |
+| ime_v1 dev (544) | 93.6 % | – |
+| homophone sentences (1,607) | 93.1 % | – |
+| average of the four | 94.9 % | – |
+| hard IME sentences (28) | 22 / 28 | – |
+
+## Limitations
+
+- Readings that repeat one sound (きしゃのきしゃがきしゃできしゃした) tend to map every occurrence to the same word
+  or drop part of the reading.
+- Rare proper nouns and fine homophone distinctions (特徴/特長, 追及/追求) are still error-prone.
+- The model copies the left context strongly: a wrong conversion earlier in the context can be repeated.
+
+## License and attribution
+
+CC BY-SA 4.0. The training data include Wikipedia (CC BY-SA), FineWeb-2 (ODC-By), Aozora Bunko, the National
+Diet minutes, and the Mozc dictionary (BSD-3-Clause, Copyright Google Inc.; contains data derived from IPAdic,
+Copyright Nara Institute of Science and Technology).

config.json

@@ -0,0 +1,36 @@
+{
+  "activation_function": "gelu_pytorch_tanh",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 3,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 1024,
+  "n_head": 16,
+  "n_inner": null,
+  "n_layer": 24,
+  "n_positions": 1024,
+  "pad_token_id": 1,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.20.0",
+  "use_cache": false,
+  "vocab_size": 6000,
+  "genmai_prompt_format": "zenzlc"
+}

model-coreml.zip

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:da2a6e1a9fbf0729c84b8497462ae4ffaf8b2dd2b74c808f3de971b5643ef3ff
+size 282924021

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d44679bed94faf1e8baa4edf7a4d0bc57c5664d8094afb7688feed98fc0f279b
+size 1238046208

tokenizer_config.json

@@ -0,0 +1,12 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "</s>",
+  "errors": "replace",
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "[UNK]"
+}