keylime-m3-preview / FORMAT.md
MostLime's picture
FORMAT.md: accuracy_level 4 set at load
acf062e verified
|
Raw History Blame Contribute Delete
8 kB
# m3 app files
This folder holds the files that the Keylime apps download for the m3 model. The algorithm that uses them is in
`m3dict_cpt/PORT_SPEC.md` (the m3dict decoder). The reference code is `m3dict_cpt/onnx_backend.py` (`M3Dict`).
The test vectors are in `m3dict_cpt/test_vectors.json`.
All binary files are little-endian. All offsets are in bytes.
## Files
| File | Content |
|---|---|
| `m3.encoder.int4.onnx` | Encoder. Inputs: `src_ids` int64 [1,S], `taps` float32 [1,S,2] (zeros). Outputs: `memory`, `cross_k0..2`, `cross_v0..2` float32 [1,2,S,64]. |
| `m3.decoder.int4.onnx` | Decoder with KV cache. See "Decoder" below. |
| `m3.manifest.json` | Export metadata: config, style ids (casual 256, standard 257, formal 258), EOS 2, `max_src` 384, `max_tgt` 112. |
| `vocab_m3.json` | SentencePiece BPE vocabulary (32,768 pieces, byte fallback) in the `web/export_vocab.py` format. Byte-identical to `vocab_m2.json`. |
| `m3_lexicon.tsv` | The 100,000-word shipping lexicon. One `word<TAB>count` per line, sorted by count (high first). Line number = lexicon id. |
| `m3_symspell.bin` | Delete index for candidate lookup. |
| `m3_ngram.bin` | Pruned and quantized KN word trigram (the `ng` feature). |
The apps derive the other tables from `m3_lexicon.tsv` when they load it:
- **Log prior**: `log(count / sum of counts)` (float64).
- **Letter bag**: 26 letter counts per word (apostrophes not counted), and the letter count.
- **Prefix table**: for each prefix of a lexicon word, the shortest letter count (apostrophes not counted) of a word
with that prefix. A sorted word list plus a range-minimum query gives the same values as a full prefix map.
- **Apostrophe expansions**: the word without apostrophes -> the lexicon words with apostrophes, in lexicon order.
## Accuracy level (set at load)
Both graphs are stored with `accuracy_level` 0 on every MatMulNBits node (fp32 dequantization: slow). The apps run
them with `accuracy_level` 4 (int8 activations against the same int4 weights: about 9 times faster per decoder step).
The change is made in memory before the session is created: in the graph bytes, replace every occurrence of
```
0a 0e "accuracy_level" 18 00 (attribute name, then the int value 0)
```
with the same bytes but a last byte of 04. The protobuf lengths do not change. The encoder has 65 such nodes and the
decoder has 28. ONNX Runtime has no session option for this (`session.qdq_matmulnbits_accuracy_level` only applies
to QDQ fusion). The test vectors are made with level 4 (`onnx_backend.with_accuracy_level`). Keyboard word-end path
on Typing37-Clean held-out (30,177 messages): level 0 fix 64.487 / FC 0.989, level 4 fix 64.426 / FC 0.992.
## Input to the encoder
`src_ids` = `[style id]` + the UTF-8 bytes of the text, cut to 384 ids in total. `taps` = zeros [1,S,2].
## Decoder
Inputs: `ids` int64 [1,N], `next_ids` int64 [1,N] (zeros when not needed), `pos` int64 [N] (absolute positions),
`past_mask` bool [P], `src_mask` bool [S] (all true), `past_k0`/`past_v0` float32 [1,2,P,64], `past_c0`/`past_c1`
float32 [1,6,512], `cross_*` (from the encoder).
Outputs: `last_lp` float32 [1,32768] (log-probs after the last id), `tok_lp` float32 [1,N] (log-prob of `next_ids[i]`
after `ids[..i]`), `present_k0`, `present_v0`, `present_c0`, `present_c1`.
The start state: `past_k0`/`past_v0` zeros [1,2,1,64], `past_c*` zeros [1,6,512], `past_mask` = `[false]`, position 0.
The first id is EOS (2). After a call, the new mask is the old mask + N times `true`, and the position moves by N.
## m3_symspell.bin
```
offset 0 char[4] "KLSS"
offset 4 u32 version (1)
offset 8 u32 n_words (100000)
offset 12 u32 n_entries
offset 16 u32 maxd (2)
offset 20 u32 short (2)
offset 24 u32 key[n_entries]
u32 word_id[n_entries]
```
For each lexicon word w, let s = w without apostrophes. The index has one entry for each string in deletes(s, d)
(s itself and all strings made by deleting up to d letters), with d = maxd if len(s) > short, else d = 1.
`key` = the high 32 bits of FNV-1a 64 of the UTF-8 bytes of the delete string. Entries are sorted by (key, word_id).
Lookup for a typed word t (lowercase, apostrophes removed, length n): make deletes(t, dq) with dq = 3 if n >= 7,
dq = 2 if n > 2, else dq = 1. For each delete q, find the entries with key = hash32(q). Accept a word id only if
q is a subsequence of the word's s and 0 <= len(s) - len(q) <= d (d of that word, as above). This check removes all
32-bit key collisions, so the result is exactly the set of words that share a delete with t. Return the sorted
unique ids.
## m3_ngram.bin
The KN word trigram of the m2 data (`data/stats/ngram_kn.npz`, 508 MB), pruned to trigram count >= 12 and bigram
continuation count >= 12 (`m3dict_cpt/qkn.py`). The backoff weights are computed again from the full totals, so the
pruned model is still an interpolated KN model. It is stored in backoff form:
```
log P(w | u v) = T(u,v,w) if the trigram (u,v,w) is stored
= G3(u,v) + B(v,w) else, if the history (u,v) is stored
= B(v,w) else
B(v,w) = Q2(v,w) if the bigram (v,w) is stored
= G2(v) + P1(w) else
```
Each value x is stored as q = round(-x * 1024), and the value is read back as -q / 1024. All sums of these values are
exact in float64, so all ports get bit-identical results.
```
offset 0 char[4] "KLQN"
offset 4 u32 version (1)
u32 V (vocabulary size)
u32 n_bi, n_hist, n_tri
u32 vocab_bytes
u8 vocab[vocab_bytes] UTF-8, words joined by '\n'; id 0 = <unk>, 1 = <s>, 2 = <num>
pad to a multiple of 4
u16 P1[V] q of log P1(w)
u16 G2[V] q of log G2(v) (0 when v has no bigram statistics)
u32 bi_off[V+1] bigrams of v: entries bi_off[v] .. bi_off[v+1]-1
u32 bi[n_bi] (w << 15) | q(Q2(v,w)), w ascending in each v
u32 hist_off[V+1] histories (u, *): entries hist_off[u] .. hist_off[u+1]-1
u32 hist[n_hist] (v << 15) | q(G3(u,v)), v ascending in each u
u32 tri_off[n_hist+1] trigrams of history h: entries tri_off[h] .. tri_off[h+1]-1
u32 tri[n_tri] (w << 15) | q(T(u,v,w)), w ascending in each history
```
Find an entry with a binary search on `entry >> 15` in its range. The 15-bit fields hold q up to 32767 (log value
down to -32). The u16 fields hold q up to 65535.
Sentence log-prob (`sent_lp`): words = the lowercase SEG words (`[A-Za-z]+(?:'[A-Za-z]+)*`) of the text. The id of a
word is its vocabulary index, or 0 if it is not in the vocabulary (a word that starts with a digit gets 2). History
starts as (1, 1). For each word: add -16.0 if its id is 0, else log P(w | u v). Then shift the history (an OOV word
enters the history as id 0). Add the values from left to right in float64.
`ng` of an alternative = sent_lp(the greedy text with the slot replaced) - sent_lp(the greedy text).
## Decoder constants
Shipping parameters: `m3dict_cpt/params_ship.json` (also in the manifest entry). Constants that are not in that file:
| Name | Value |
|---|---|
| top-K tokens per greedy step | 8 (ties: log-prob desc, then token id asc) |
| tau_split | 4.0 |
| greedy steps | min(112, len(src_ids) // 2 + 24) |
| candidates per slot (slot plan, injection) | 8 |
| RARE rank | 10000 (a slot plan uses min(10000, params.rare)) |
| generator | prior weight 0.5, max hits 4000, qd3 7, short 2, bag top 300 for words of >= 6 letters (ties: score desc, then lexicon id asc) |
| spatial model | keyboard.SpatialModel defaults (sigma 0.3, key-centre touches), QWERTY eval geometry; splits use space bar x 3.5 .. 8.5 |
| feature rounding | dm3, sp, pr, ng: Python round(x, 4) |
| LP_OOV | -16.0 |