Download docs/research/HUFFMAN_INITIALIZATION.md from nmarshall/urt-rs: direct link, hf CLI and curl.
- Browser
- Download file 6.13 kB
-
https://huggingface.co/nmarshall/urt-rs/resolve/main/docs/research/HUFFMAN_INITIALIZATION.md
- Command line
-
hf download hf://nmarshall/urt-rs/docs/research/HUFFMAN_INITIALIZATION.md
-
curl -L -o HUFFMAN_INITIALIZATION.md https://huggingface.co/nmarshall/urt-rs/resolve/main/docs/research/HUFFMAN_INITIALIZATION.md
Huffman frequencies, empty-tree initialization and training requests
Reproduce with python3 scripts/analyze_huffman_initialization.py; validate with python3 scripts/test_huffman_initialization.py. huffman-initialization.json retains four hash-verified graphs per Linux ELF64 engine: MSG_initHuffman, Huff_Init, Huff_addRef and increment. This follows supplied-tree encoder compositions.
The actual frequency table is extracted from verified file-backed native images. Initialization and scheduling are bounded synthetic instruction evaluations: PUSH instructions are explicitly omitted, POP/RET and all callees are stopped, and memset effects are supplied. Small scheduling fixtures supply count tables and retain caller registers across stopped addRef requests. Huff_addRef and increment graphs are static next-step evidence; neither is evaluated here. Original processes and native callees never execute, and no trained tree/codebook or packet conformance is established.
INIT-001 — Exact native seed frequencies (RE-353)
The MSG_initHuffman comparison operands identify a table of 256 little-endian 32-bit counts. Both verified Linux images contain identical table bytes, with SHA-256:
1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf
The report retains every count, virtual address, file offset and source/disassembly provenance. All counts are positive and fit signed 32-bit comparison: minimum 740, maximum 250315, total 1,053,340. Symbol zero has count 250315; the first sixteen counts are [250315,41193,6292,7106,3730,3750,6110,23283,33317,6950,7838,9714,9257,17259,3949,1778].
The loop implies 1,053,340 repetitions per tree, or 2,106,680 paired addRef requests. A conceptual schedule digest is generated from the extracted counts: for each symbol ascending, repeat bytes(symbol, tree0, symbol, tree1) count times. This generated digest records the implied request stream independently of native addresses. It is not a trace of all native training iterations and does not establish the resulting tree. Reconstructing a generic frequency-optimal tree without recovering native tie/rank/swap behavior could produce different codewords.
INIT-002 — Two cleared objects and initial NYT nodes (RE-354)
Four paired initialization probes use two virtual base addresses and dirty sentinel buffers. Huff_Init requests two adjacent memset operations, each of size 0xc830 (51,248 bytes), value zero: base and base +0xc830. The collector supplies those effects; native memset never executes. Both outside canaries retain their sentinel.
After the clears, each object's node count at +0 becomes one. Its initial node is at object +0x830 (2,096); native allocation address arithmetic computes 64count minus 8count, a 56-byte node stride. The root pointer +8, pointer slot +16 and symbol-256 lookup slot +0x820 all refer to this initial node. The second object's pointer slot +24 also refers to it; the first object's +24 remains zero. This asymmetry is retained rather than normalized. Full list-slot semantics require addRef/increment analysis.
The initial 56-byte node has symbol word 256 at +52 and weight word zero at +48. Child/parent pointers +0/+8/+16 and list-related slots +24/+32 are zero. The complete objects contain only the expected nonzero count/pointer/symbol bytes; the report records the sparse nonzero-byte image, not just selected fields. Slot +0x820 agrees with the transmit lookup formula object +32 +8*symbol for symbol 256. This supports identifying the initial not-yet-transmitted (NYT) node, while actual NYT insertion/escape behavior remains open.
These native layout offsets are evidence for reconstruction, not a required Rust memory layout. Capacities, node/pointer-pool ownership, alias invariants and allocator/error paths remain unresolved.
INIT-003 — Flag, ordering and bounded seed loop (RE-355)
The entry slice sets msgInit to one before requesting Huff_Init at the first object pointer. It resets local symbol/repetition registers to zero. The caller guard, thread/reentrancy behavior and initialization failure semantics are outside this slice.
Eight paired loop compositions supply four count tables: all zero; symbols zero twice and 255 three times; sparse symbols 1/127/128/254; and every symbol once. Symbol indices advance from zero through 255 in ascending order, and the loop terminates with local symbol 256. Each repetition requests Huff_addRef(first_object,symbol), then Huff_addRef(first_object +0xc830,symbol), before testing the next repetition. Zero-count symbols produce no requests. Supplied counts and all stopped request arguments are retained, with trace hashes/counts.
The native helper requests total 1,068 across these bounded fixtures: 534 per engine. Actual seed frequencies are extracted separately and are not substituted into an evaluated million-iteration loop. The small fixtures prove local loop ordering/skip/termination and paired object selection; they do not recover addRef effects, final weights or trained codewords.
Validation and next work
Eight independent tests verify exact frequency hash/summary, a separately generated per-request conceptual digest, entry flag/init arguments, complete bounded ascending/paired request order, exact clear spans/canaries, complete sparse object/NYT layout and asymmetry, stopped CALL traces and paired frequency/virtual-layout results. No evaluator extension was required.
RE-356: recover Huff_addRef insertion/existing-symbol branches, increment rank/block/head/freelist changes, node/list swaps and recursion; reconstruct initialized trees with the exact seed schedule and derive verified codewords; compose actual message encoders/decoders and packet fixtures, with capacity/truncation/overflow, legal invariants and original-runtime/all-platform validation.
Update follow-up: bounded adaptive compositions recovers insertion and increment effects with independent tree/list/block invariants (RE-357–359). Full seed-trained trees and actual codewords remain open.