urt-rs / docs /research /HUFFMAN_ENCODING.md
nmarshall's picture
Recover native Huffman frequencies and initialization
37a2f91
|
Raw History Blame Contribute Delete
6.2 kB

Bounded Huffman send and supplied-tree symbol streams

Reproduce with python3 scripts/analyze_huffman_encoding.py; validate with python3 scripts/test_huffman_encoding.py. huffman-encoding.json retains three hash-verified graphs per Linux ELF64 engine: send, Huff_offsetTransmit and Huff_offsetReceive. This follows helper and lookup probes.

These compositions evaluate native instructions over supplied trees, symbol tables and buffers. They do not run original processes or native callees. The send frame starts after register saves/stack allocation and stops before restores; its output-buffer register and stack pointer are supplied explicitly. Recursive calls are stopped, evaluated with separate virtual stack frames spaced by 0x1000, then resumed with all caller registers explicitly restored and child memory effects retained. Wrapper PUSH is omitted and execution stops before POP. This is a bounded composition model, not proven ABI behavior or native stack usage.

ENC-001 — Root-to-leaf bits, unrolled lanes and recursion (RE-349)

Seven hundred twenty paired send compositions cover selected path lengths 0–10/16/17/24/32; all-zero, all-one and both alternating branch patterns; initial cursors 0/7/8; and buffer sentinels zero/0xff. Each supplied comb tree has a distinct sibling leaf at every path level, consistent parent links and one selected symbol leaf. Tree construction is synthetic and does not recover game codebooks or balancing.

The composed sender walks parent pointers toward the root, then emits branches in root-to-leaf order. A child equal to the parent's right pointer at +8 produces one; other supplied consistent children produce zero. Parent links are at +16. It writes bits least-significant-first within bytes and updates global bloc. At byte boundaries the output byte clears before either branch; within a byte zero retains prior bits and one ORs its bit. Dirty starting partial bytes therefore retain prior set bits, matching the earlier raw-helper finding.

Nine edges are unrolled before another recursive composition is needed. Depth nine uses one virtual frame, ten uses two, twenty-four uses three and thirty-two uses four; the sampled frame count is ceil(path_length/9). The collector stops the native recursive CALL and composes its instruction body explicitly. No original CALL or RET executes. The zero-depth entries record the wrapper's no-send case, with no send instructions or output mutation; they do not exercise send on a null current-node argument.

Independent root-path bit packing verifies all sixteen retained output bytes, global cursor, outside canaries and recursion thresholds. Cyclic or inconsistent parent/child trees, null current nodes, arbitrary depths and buffer capacity are untested.

ENC-002 — Transmit and receive symbol composition (RE-350)

One hundred ninety-two paired symbol round trips cover depths 0/1/2/8/9/10/17/32, four branch patterns and cursors 0/7/8. Huff_offsetTransmit selects symbol 42 through the supplied table, then its stopped send request is composed and the wrapper resumes to publish global bloc into the caller's cursor. The receive cursor is reset to the same starting position; Huff_offsetReceive traverses the same supplied tree and returns symbol 42.

Both directions consume exactly the supplied path length and agree on the final cursor. Encoded bytes match independent packing; outside canaries remain intact. A zero-depth tree consists of a root leaf and yields symbol 42 with no bit consumption. This degenerate synthetic tree is useful for local wrapper behavior but is not evidence that the game's actual codebook permits zero-length symbols.

These fixtures replace supplied decoder symbols with actual supplied-tree instruction traversal and replace stopped encoder requests with composed send instructions. They still do not establish the initialized adaptive tree, frequency tables, code lengths, packet compatibility or complete MSG_WriteBits/MSG_ReadBits behavior.

ENC-003 — Shared-tree multi-symbol streams (RE-351)

Thirty paired sequences use one supplied complete depth-three tree, with symbols 40–47 at its eight leaves. Sequences are empty, one symbol, two ascending cycles, sixteen repeated final symbols and alternating first/final symbols, each at cursor 0/7/8. The tree remains shared throughout transmission and reception; lookup, send and receive instructions compose for every symbol.

Each code is the three-bit root-to-leaf representation of symbol-minus-40, emitted root bit first but packed LSB-first within the output bytes. Every symbol advances the cursor by three. Independent stream packing verifies bytes across boundaries, both per-symbol cursor lists and decoded sequences. Repeated symbols do not trigger tree updates inside these three sampled helper functions; adaptive initialization/update ownership outside them remains open.

Buffers begin at zero, while the separate dirty-buffer probes characterize retained partial-byte bits. These are supplied-codebook round trips, not real game symbol streams, network messages or corpus conformance.

Validation and next work

Eight independent tests check dirty-buffer/root-path packing, nine-edge recursive thresholds, encoded bytes/symbols/cursors/canaries, supplied parent/child consistency, shared stream packing/repetition, zero-depth/empty streams, recomposed traces excluding native CALLs and paired outputs. Reports retain trace lengths/hashes and verified full graphs. No evaluator extension was needed.

RE-352: recover Huffman initialization, node allocation/swapping/rank/frequency updates and actual game codebooks; compose full message writers/readers with real code lengths and packets; recover capacity/overflow/truncation and legal symbol/tree/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration before Rust implementation.

Initialization follow-up: native frequencies and empty-tree probes recovers actual seed counts and bounded initialization/training requests (RE-353–355). Adaptive tree effects and final native codewords remain open.