keylime-m3-preview / FORMAT.md
MostLime's picture
FORMAT.md: accuracy_level 4 set at load
acf062e verified
|
Raw History Blame Contribute Delete
8 kB

m3 app files

This folder holds the files that the Keylime apps download for the m3 model. The algorithm that uses them is in m3dict_cpt/PORT_SPEC.md (the m3dict decoder). The reference code is m3dict_cpt/onnx_backend.py (M3Dict). The test vectors are in m3dict_cpt/test_vectors.json.

All binary files are little-endian. All offsets are in bytes.

Files

File Content
m3.encoder.int4.onnx Encoder. Inputs: src_ids int64 [1,S], taps float32 [1,S,2] (zeros). Outputs: memory, cross_k0..2, cross_v0..2 float32 [1,2,S,64].
m3.decoder.int4.onnx Decoder with KV cache. See "Decoder" below.
m3.manifest.json Export metadata: config, style ids (casual 256, standard 257, formal 258), EOS 2, max_src 384, max_tgt 112.
vocab_m3.json SentencePiece BPE vocabulary (32,768 pieces, byte fallback) in the web/export_vocab.py format. Byte-identical to vocab_m2.json.
m3_lexicon.tsv The 100,000-word shipping lexicon. One word<TAB>count per line, sorted by count (high first). Line number = lexicon id.
m3_symspell.bin Delete index for candidate lookup.
m3_ngram.bin Pruned and quantized KN word trigram (the ng feature).

The apps derive the other tables from m3_lexicon.tsv when they load it:

  • Log prior: log(count / sum of counts) (float64).
  • Letter bag: 26 letter counts per word (apostrophes not counted), and the letter count.
  • Prefix table: for each prefix of a lexicon word, the shortest letter count (apostrophes not counted) of a word with that prefix. A sorted word list plus a range-minimum query gives the same values as a full prefix map.
  • Apostrophe expansions: the word without apostrophes -> the lexicon words with apostrophes, in lexicon order.

Accuracy level (set at load)

Both graphs are stored with accuracy_level 0 on every MatMulNBits node (fp32 dequantization: slow). The apps run them with accuracy_level 4 (int8 activations against the same int4 weights: about 9 times faster per decoder step). The change is made in memory before the session is created: in the graph bytes, replace every occurrence of

0a 0e "accuracy_level" 18 00      (attribute name, then the int value 0)

with the same bytes but a last byte of 04. The protobuf lengths do not change. The encoder has 65 such nodes and the decoder has 28. ONNX Runtime has no session option for this (session.qdq_matmulnbits_accuracy_level only applies to QDQ fusion). The test vectors are made with level 4 (onnx_backend.with_accuracy_level). Keyboard word-end path on Typing37-Clean held-out (30,177 messages): level 0 fix 64.487 / FC 0.989, level 4 fix 64.426 / FC 0.992.

Input to the encoder

src_ids = [style id] + the UTF-8 bytes of the text, cut to 384 ids in total. taps = zeros [1,S,2].

Decoder

Inputs: ids int64 [1,N], next_ids int64 [1,N] (zeros when not needed), pos int64 [N] (absolute positions), past_mask bool [P], src_mask bool [S] (all true), past_k0/past_v0 float32 [1,2,P,64], past_c0/past_c1 float32 [1,6,512], cross_* (from the encoder).

Outputs: last_lp float32 [1,32768] (log-probs after the last id), tok_lp float32 [1,N] (log-prob of next_ids[i] after ids[..i]), present_k0, present_v0, present_c0, present_c1.

The start state: past_k0/past_v0 zeros [1,2,1,64], past_c* zeros [1,6,512], past_mask = [false], position 0. The first id is EOS (2). After a call, the new mask is the old mask + N times true, and the position moves by N.

m3_symspell.bin

offset 0   char[4]  "KLSS"
offset 4   u32      version (1)
offset 8   u32      n_words (100000)
offset 12  u32      n_entries
offset 16  u32      maxd (2)
offset 20  u32      short (2)
offset 24  u32      key[n_entries]
           u32      word_id[n_entries]

For each lexicon word w, let s = w without apostrophes. The index has one entry for each string in deletes(s, d) (s itself and all strings made by deleting up to d letters), with d = maxd if len(s) > short, else d = 1. key = the high 32 bits of FNV-1a 64 of the UTF-8 bytes of the delete string. Entries are sorted by (key, word_id).

Lookup for a typed word t (lowercase, apostrophes removed, length n): make deletes(t, dq) with dq = 3 if n >= 7, dq = 2 if n > 2, else dq = 1. For each delete q, find the entries with key = hash32(q). Accept a word id only if q is a subsequence of the word's s and 0 <= len(s) - len(q) <= d (d of that word, as above). This check removes all 32-bit key collisions, so the result is exactly the set of words that share a delete with t. Return the sorted unique ids.

m3_ngram.bin

The KN word trigram of the m2 data (data/stats/ngram_kn.npz, 508 MB), pruned to trigram count >= 12 and bigram continuation count >= 12 (m3dict_cpt/qkn.py). The backoff weights are computed again from the full totals, so the pruned model is still an interpolated KN model. It is stored in backoff form:

log P(w | u v) = T(u,v,w)              if the trigram (u,v,w) is stored
               = G3(u,v) + B(v,w)      else, if the history (u,v) is stored
               = B(v,w)                else
B(v,w)         = Q2(v,w)               if the bigram (v,w) is stored
               = G2(v) + P1(w)         else

Each value x is stored as q = round(-x * 1024), and the value is read back as -q / 1024. All sums of these values are exact in float64, so all ports get bit-identical results.

offset 0   char[4]  "KLQN"
offset 4   u32      version (1)
           u32      V (vocabulary size)
           u32      n_bi, n_hist, n_tri
           u32      vocab_bytes
           u8       vocab[vocab_bytes]       UTF-8, words joined by '\n'; id 0 = <unk>, 1 = <s>, 2 = <num>
           pad to a multiple of 4
           u16      P1[V]                    q of log P1(w)
           u16      G2[V]                    q of log G2(v)  (0 when v has no bigram statistics)
           u32      bi_off[V+1]              bigrams of v: entries bi_off[v] .. bi_off[v+1]-1
           u32      bi[n_bi]                 (w << 15) | q(Q2(v,w)), w ascending in each v
           u32      hist_off[V+1]            histories (u, *): entries hist_off[u] .. hist_off[u+1]-1
           u32      hist[n_hist]             (v << 15) | q(G3(u,v)), v ascending in each u
           u32      tri_off[n_hist+1]        trigrams of history h: entries tri_off[h] .. tri_off[h+1]-1
           u32      tri[n_tri]               (w << 15) | q(T(u,v,w)), w ascending in each history

Find an entry with a binary search on entry >> 15 in its range. The 15-bit fields hold q up to 32767 (log value down to -32). The u16 fields hold q up to 65535.

Sentence log-prob (sent_lp): words = the lowercase SEG words ([A-Za-z]+(?:'[A-Za-z]+)*) of the text. The id of a word is its vocabulary index, or 0 if it is not in the vocabulary (a word that starts with a digit gets 2). History starts as (1, 1). For each word: add -16.0 if its id is 0, else log P(w | u v). Then shift the history (an OOV word enters the history as id 0). Add the values from left to right in float64.

ng of an alternative = sent_lp(the greedy text with the slot replaced) - sent_lp(the greedy text).

Decoder constants

Shipping parameters: m3dict_cpt/params_ship.json (also in the manifest entry). Constants that are not in that file:

Name Value
top-K tokens per greedy step 8 (ties: log-prob desc, then token id asc)
tau_split 4.0
greedy steps min(112, len(src_ids) // 2 + 24)
candidates per slot (slot plan, injection) 8
RARE rank 10000 (a slot plan uses min(10000, params.rare))
generator prior weight 0.5, max hits 4000, qd3 7, short 2, bag top 300 for words of >= 6 letters (ties: score desc, then lexicon id asc)
spatial model keyboard.SpatialModel defaults (sigma 0.3, key-centre touches), QWERTY eval geometry; splits use space bar x 3.5 .. 8.5
feature rounding dm3, sp, pr, ng: Python round(x, 4)
LP_OOV -16.0