|
Download FORMAT.md from MostLime/keylime-m3-preview: direct link, hf CLI and curl.
- Browser
- Download file 8 kB
-
https://huggingface.co/MostLime/keylime-m3-preview/resolve/main/FORMAT.md
- Command line
-
hf download hf://MostLime/keylime-m3-preview/FORMAT.md
-
curl -L -o FORMAT.md https://huggingface.co/MostLime/keylime-m3-preview/resolve/main/FORMAT.md
8 kB
| # m3 app files | |
| This folder holds the files that the Keylime apps download for the m3 model. The algorithm that uses them is in | |
| `m3dict_cpt/PORT_SPEC.md` (the m3dict decoder). The reference code is `m3dict_cpt/onnx_backend.py` (`M3Dict`). | |
| The test vectors are in `m3dict_cpt/test_vectors.json`. | |
| All binary files are little-endian. All offsets are in bytes. | |
| ## Files | |
| | File | Content | | |
| |---|---| | |
| | `m3.encoder.int4.onnx` | Encoder. Inputs: `src_ids` int64 [1,S], `taps` float32 [1,S,2] (zeros). Outputs: `memory`, `cross_k0..2`, `cross_v0..2` float32 [1,2,S,64]. | | |
| | `m3.decoder.int4.onnx` | Decoder with KV cache. See "Decoder" below. | | |
| | `m3.manifest.json` | Export metadata: config, style ids (casual 256, standard 257, formal 258), EOS 2, `max_src` 384, `max_tgt` 112. | | |
| | `vocab_m3.json` | SentencePiece BPE vocabulary (32,768 pieces, byte fallback) in the `web/export_vocab.py` format. Byte-identical to `vocab_m2.json`. | | |
| | `m3_lexicon.tsv` | The 100,000-word shipping lexicon. One `word<TAB>count` per line, sorted by count (high first). Line number = lexicon id. | | |
| | `m3_symspell.bin` | Delete index for candidate lookup. | | |
| | `m3_ngram.bin` | Pruned and quantized KN word trigram (the `ng` feature). | | |
| The apps derive the other tables from `m3_lexicon.tsv` when they load it: | |
| - **Log prior**: `log(count / sum of counts)` (float64). | |
| - **Letter bag**: 26 letter counts per word (apostrophes not counted), and the letter count. | |
| - **Prefix table**: for each prefix of a lexicon word, the shortest letter count (apostrophes not counted) of a word | |
| with that prefix. A sorted word list plus a range-minimum query gives the same values as a full prefix map. | |
| - **Apostrophe expansions**: the word without apostrophes -> the lexicon words with apostrophes, in lexicon order. | |
| ## Accuracy level (set at load) | |
| Both graphs are stored with `accuracy_level` 0 on every MatMulNBits node (fp32 dequantization: slow). The apps run | |
| them with `accuracy_level` 4 (int8 activations against the same int4 weights: about 9 times faster per decoder step). | |
| The change is made in memory before the session is created: in the graph bytes, replace every occurrence of | |
| ``` | |
| 0a 0e "accuracy_level" 18 00 (attribute name, then the int value 0) | |
| ``` | |
| with the same bytes but a last byte of 04. The protobuf lengths do not change. The encoder has 65 such nodes and the | |
| decoder has 28. ONNX Runtime has no session option for this (`session.qdq_matmulnbits_accuracy_level` only applies | |
| to QDQ fusion). The test vectors are made with level 4 (`onnx_backend.with_accuracy_level`). Keyboard word-end path | |
| on Typing37-Clean held-out (30,177 messages): level 0 fix 64.487 / FC 0.989, level 4 fix 64.426 / FC 0.992. | |
| ## Input to the encoder | |
| `src_ids` = `[style id]` + the UTF-8 bytes of the text, cut to 384 ids in total. `taps` = zeros [1,S,2]. | |
| ## Decoder | |
| Inputs: `ids` int64 [1,N], `next_ids` int64 [1,N] (zeros when not needed), `pos` int64 [N] (absolute positions), | |
| `past_mask` bool [P], `src_mask` bool [S] (all true), `past_k0`/`past_v0` float32 [1,2,P,64], `past_c0`/`past_c1` | |
| float32 [1,6,512], `cross_*` (from the encoder). | |
| Outputs: `last_lp` float32 [1,32768] (log-probs after the last id), `tok_lp` float32 [1,N] (log-prob of `next_ids[i]` | |
| after `ids[..i]`), `present_k0`, `present_v0`, `present_c0`, `present_c1`. | |
| The start state: `past_k0`/`past_v0` zeros [1,2,1,64], `past_c*` zeros [1,6,512], `past_mask` = `[false]`, position 0. | |
| The first id is EOS (2). After a call, the new mask is the old mask + N times `true`, and the position moves by N. | |
| ## m3_symspell.bin | |
| ``` | |
| offset 0 char[4] "KLSS" | |
| offset 4 u32 version (1) | |
| offset 8 u32 n_words (100000) | |
| offset 12 u32 n_entries | |
| offset 16 u32 maxd (2) | |
| offset 20 u32 short (2) | |
| offset 24 u32 key[n_entries] | |
| u32 word_id[n_entries] | |
| ``` | |
| For each lexicon word w, let s = w without apostrophes. The index has one entry for each string in deletes(s, d) | |
| (s itself and all strings made by deleting up to d letters), with d = maxd if len(s) > short, else d = 1. | |
| `key` = the high 32 bits of FNV-1a 64 of the UTF-8 bytes of the delete string. Entries are sorted by (key, word_id). | |
| Lookup for a typed word t (lowercase, apostrophes removed, length n): make deletes(t, dq) with dq = 3 if n >= 7, | |
| dq = 2 if n > 2, else dq = 1. For each delete q, find the entries with key = hash32(q). Accept a word id only if | |
| q is a subsequence of the word's s and 0 <= len(s) - len(q) <= d (d of that word, as above). This check removes all | |
| 32-bit key collisions, so the result is exactly the set of words that share a delete with t. Return the sorted | |
| unique ids. | |
| ## m3_ngram.bin | |
| The KN word trigram of the m2 data (`data/stats/ngram_kn.npz`, 508 MB), pruned to trigram count >= 12 and bigram | |
| continuation count >= 12 (`m3dict_cpt/qkn.py`). The backoff weights are computed again from the full totals, so the | |
| pruned model is still an interpolated KN model. It is stored in backoff form: | |
| ``` | |
| log P(w | u v) = T(u,v,w) if the trigram (u,v,w) is stored | |
| = G3(u,v) + B(v,w) else, if the history (u,v) is stored | |
| = B(v,w) else | |
| B(v,w) = Q2(v,w) if the bigram (v,w) is stored | |
| = G2(v) + P1(w) else | |
| ``` | |
| Each value x is stored as q = round(-x * 1024), and the value is read back as -q / 1024. All sums of these values are | |
| exact in float64, so all ports get bit-identical results. | |
| ``` | |
| offset 0 char[4] "KLQN" | |
| offset 4 u32 version (1) | |
| u32 V (vocabulary size) | |
| u32 n_bi, n_hist, n_tri | |
| u32 vocab_bytes | |
| u8 vocab[vocab_bytes] UTF-8, words joined by '\n'; id 0 = <unk>, 1 = <s>, 2 = <num> | |
| pad to a multiple of 4 | |
| u16 P1[V] q of log P1(w) | |
| u16 G2[V] q of log G2(v) (0 when v has no bigram statistics) | |
| u32 bi_off[V+1] bigrams of v: entries bi_off[v] .. bi_off[v+1]-1 | |
| u32 bi[n_bi] (w << 15) | q(Q2(v,w)), w ascending in each v | |
| u32 hist_off[V+1] histories (u, *): entries hist_off[u] .. hist_off[u+1]-1 | |
| u32 hist[n_hist] (v << 15) | q(G3(u,v)), v ascending in each u | |
| u32 tri_off[n_hist+1] trigrams of history h: entries tri_off[h] .. tri_off[h+1]-1 | |
| u32 tri[n_tri] (w << 15) | q(T(u,v,w)), w ascending in each history | |
| ``` | |
| Find an entry with a binary search on `entry >> 15` in its range. The 15-bit fields hold q up to 32767 (log value | |
| down to -32). The u16 fields hold q up to 65535. | |
| Sentence log-prob (`sent_lp`): words = the lowercase SEG words (`[A-Za-z]+(?:'[A-Za-z]+)*`) of the text. The id of a | |
| word is its vocabulary index, or 0 if it is not in the vocabulary (a word that starts with a digit gets 2). History | |
| starts as (1, 1). For each word: add -16.0 if its id is 0, else log P(w | u v). Then shift the history (an OOV word | |
| enters the history as id 0). Add the values from left to right in float64. | |
| `ng` of an alternative = sent_lp(the greedy text with the slot replaced) - sent_lp(the greedy text). | |
| ## Decoder constants | |
| Shipping parameters: `m3dict_cpt/params_ship.json` (also in the manifest entry). Constants that are not in that file: | |
| | Name | Value | | |
| |---|---| | |
| | top-K tokens per greedy step | 8 (ties: log-prob desc, then token id asc) | | |
| | tau_split | 4.0 | | |
| | greedy steps | min(112, len(src_ids) // 2 + 24) | | |
| | candidates per slot (slot plan, injection) | 8 | | |
| | RARE rank | 10000 (a slot plan uses min(10000, params.rare)) | | |
| | generator | prior weight 0.5, max hits 4000, qd3 7, short 2, bag top 300 for words of >= 6 letters (ties: score desc, then lexicon id asc) | | |
| | spatial model | keyboard.SpatialModel defaults (sigma 0.3, key-centre touches), QWERTY eval geometry; splits use space bar x 3.5 .. 8.5 | | |
| | feature rounding | dm3, sp, pr, ng: Python round(x, 4) | | |
| | LP_OOV | -16.0 | | |