nmarshall commited on
Commit
37a2f91
·
1 Parent(s): afac81f

Recover native Huffman frequencies and initialization

Browse files
docs/ASSET_PIPELINE_TODO.md CHANGED
@@ -218,3 +218,5 @@ Message-bit follow-up: [read modes and assembly](research/MESSAGE_BITS.md) estab
218
  Huffman follow-up: [helper probes](research/HUFFMAN_HELPERS.md) establishes raw-bit round trips and supplied-tree decoding, with explicit null-path cursor differences. Real initialized codebooks and message fixtures remain prerequisites for network/map/renderer integration (RE-348).
219
 
220
  Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md) validates bounded transmit/receive compositions with synthetic codebooks. Actual game codebooks and full packet contracts remain prerequisites for network/map/renderer integration (RE-352).
 
 
 
218
  Huffman follow-up: [helper probes](research/HUFFMAN_HELPERS.md) establishes raw-bit round trips and supplied-tree decoding, with explicit null-path cursor differences. Real initialized codebooks and message fixtures remain prerequisites for network/map/renderer integration (RE-348).
219
 
220
  Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md) validates bounded transmit/receive compositions with synthetic codebooks. Actual game codebooks and full packet contracts remain prerequisites for network/map/renderer integration (RE-352).
221
+
222
+ Initialization follow-up: [native frequency tables](research/HUFFMAN_INITIALIZATION.md) preserves exact codec seeds and initial layouts. Recover adaptive training and actual codewords before real network/map/renderer integration; supplied-tree round trips alone remain insufficient (RE-356).
docs/FINDINGS.md CHANGED
@@ -384,3 +384,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
384
  **F-122 — Observed supplied-codebook round trips:** composed transmit/send/receive returns symbol 42 with matching bytes and cursors for sampled paths up to 32 edges, including a synthetic root-leaf zero-length code. Actual initialized trees and game packet compatibility remain open. [ENC-002](research/HUFFMAN_ENCODING.md).
385
 
386
  **F-123 — Observed shared symbol streams:** supplied depth-three codebooks encode/decode repeated and mixed symbols across byte boundaries with three-bit cursor increments. No tree updates occur inside these sampled helpers; actual adaptive initialization/update ownership and complete message codecs remain open. [ENC-003](research/HUFFMAN_ENCODING.md).
 
 
 
 
 
 
 
384
  **F-122 — Observed supplied-codebook round trips:** composed transmit/send/receive returns symbol 42 with matching bytes and cursors for sampled paths up to 32 edges, including a synthetic root-leaf zero-length code. Actual initialized trees and game packet compatibility remain open. [ENC-002](research/HUFFMAN_ENCODING.md).
385
 
386
  **F-123 — Observed shared symbol streams:** supplied depth-three codebooks encode/decode repeated and mixed symbols across byte boundaries with three-bit cursor increments. No tree updates occur inside these sampled helpers; actual adaptive initialization/update ownership and complete message codecs remain open. [ENC-003](research/HUFFMAN_ENCODING.md).
387
+
388
+ **F-124 — Observed native frequencies:** both Linux images contain the same verified 256-count table, totaling 1,053,340 repetitions per tree. A generated conceptual paired schedule is retained separately from native execution; trained trees and codewords remain open. [INIT-001](research/HUFFMAN_INITIALIZATION.md).
389
+
390
+ **F-125 — Observed empty-tree initialization:** two 51,248-byte clear requests precede initial symbol-256 nodes at object +2,096, count one and lookup/root aliases. Pointer slot +24 differs between objects. Supplied clear effects and complete sparse layouts do not establish later allocation/list/adaptive behavior. [INIT-002](research/HUFFMAN_INITIALIZATION.md).
391
+
392
+ **F-126 — Observed training requests:** entry sets msgInit before requesting Huff_Init; bounded count fixtures schedule symbols ascending with first-object then second-object addRef requests per repetition, ending at symbol 256. addRef effects and actual full seeded training remain unexecuted. [INIT-003](research/HUFFMAN_INITIALIZATION.md).
docs/PROVISIONAL_TODO.md CHANGED
@@ -530,3 +530,5 @@ Research progress: [message read modes and assembly](research/MESSAGE_BITS.md) c
530
  Research progress: [Huffman helper probes](research/HUFFMAN_HELPERS.md) completes bounded RE-345–347. Broad protocol/runtime milestones remain open; RE-348 tracks full send, initialized trees/codebooks, actual packet round trips and platform coverage.
531
 
532
  Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) completes RE-349–351. Broad network/runtime milestones remain open; RE-352 tracks adaptive initialization/codebooks, real message packets and platform fixtures.
 
 
 
530
  Research progress: [Huffman helper probes](research/HUFFMAN_HELPERS.md) completes bounded RE-345–347. Broad protocol/runtime milestones remain open; RE-348 tracks full send, initialized trees/codebooks, actual packet round trips and platform coverage.
531
 
532
  Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) completes RE-349–351. Broad network/runtime milestones remain open; RE-352 tracks adaptive initialization/codebooks, real message packets and platform fixtures.
533
+
534
+ Research progress: [Huffman initialization](research/HUFFMAN_INITIALIZATION.md) completes bounded RE-353–355. Broad network/runtime milestones remain open; RE-356 tracks adaptive updates, initialized codewords and actual message/packet/platform fixtures.
docs/RESEARCH_LEDGER.md CHANGED
@@ -213,3 +213,7 @@ Completed bounded RE-345–347 with `analyze_huffman_helpers.py`: ten verified g
213
  ## 2026-10-11 — Bounded Huffman send and symbol streams
214
 
215
  Completed bounded RE-349–351 with `analyze_huffman_encoding.py`: six verified graphs across the Linux ELF64 engines, 720 supplied-path send compositions, 192 symbol round trips and thirty shared-tree streams. ENC-001–003/F-121–123 distinguish root-order code bits from LSB byte packing, dirty partial-byte retention from byte-start clearing, nine-edge unrolling from explicit recursive virtual frames, supplied codebooks from actual initialized game trees, and symbol round trips from full message/packet conformance. Frame prologues/epilogues and wrapper PUSH/POP are excluded explicitly; full caller-register restoration and disjoint virtual stacks preserve child memory effects across stopped CALL boundaries. No original process or native callee executes. Eight independent tests verify packing/canaries/cursors, recursion thresholds, parent-child tree consistency, transmit/receive symbols, repeated streams, zero-depth/empty cases, recomposed call-free traces and paired results. All 310 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-348 remains open; RE-352 tracks adaptive initialization/node allocation/rank/swaps/frequencies/codebooks, actual message/packet composition, legal inputs/capacity/truncation and ABI/runtime/all-platform/snapshot/renderer coverage.
 
 
 
 
 
213
  ## 2026-10-11 — Bounded Huffman send and symbol streams
214
 
215
  Completed bounded RE-349–351 with `analyze_huffman_encoding.py`: six verified graphs across the Linux ELF64 engines, 720 supplied-path send compositions, 192 symbol round trips and thirty shared-tree streams. ENC-001–003/F-121–123 distinguish root-order code bits from LSB byte packing, dirty partial-byte retention from byte-start clearing, nine-edge unrolling from explicit recursive virtual frames, supplied codebooks from actual initialized game trees, and symbol round trips from full message/packet conformance. Frame prologues/epilogues and wrapper PUSH/POP are excluded explicitly; full caller-register restoration and disjoint virtual stacks preserve child memory effects across stopped CALL boundaries. No original process or native callee executes. Eight independent tests verify packing/canaries/cursors, recursion thresholds, parent-child tree consistency, transmit/receive symbols, repeated streams, zero-depth/empty cases, recomposed call-free traces and paired results. All 310 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-348 remains open; RE-352 tracks adaptive initialization/node allocation/rank/swaps/frequencies/codebooks, actual message/packet composition, legal inputs/capacity/truncation and ABI/runtime/all-platform/snapshot/renderer coverage.
216
+
217
+ ## 2026-10-11 — Native Huffman frequencies and initialization
218
+
219
+ Completed bounded RE-353–355 with `analyze_huffman_initialization.py`: eight verified graphs across the Linux ELF64 engines, two identical file-backed 256-count frequency tables, four empty-tree initialization probes and eight seed-loop compositions. INIT-001–003/F-124–126 distinguish actual table extraction from generated conceptual schedule digests, supplied memset effects from native clear execution, exact sparse initial/NYT layouts from trained trees, pointer-slot asymmetry from inferred list semantics, and bounded paired addRef requests from adaptive updates. The tables total 1,053,340 repetitions per tree; their implied 2,106,680-request schedule is not evaluated as a full native training trace. The bounded schedules retain 1,068 stopped requests with explicit caller-register retention. PUSH omission and POP/RET/callee boundaries are documented; no original process or native callee executes. Eight independent tests verify frequencies/hash/summary, separately generated conceptual request digest, entry flag/init arguments, ascending paired schedules, clear spans/canaries, full sparse layouts, trace boundaries and paired results. All 318 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-352 remains open; RE-356 tracks addRef/increment/swaps/block/freelist effects, actual trained trees/codewords, full packet/message codecs and legal/runtime/all-platform invariants.
docs/REVERSE_ENGINEERING_TODO.md CHANGED
@@ -422,3 +422,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
422
  - [x] RE-350 Compose transmit/send/receive against supplied comb trees with exact symbol/cursor/byte round trips ([ENC-002](research/HUFFMAN_ENCODING.md)).
423
  - [x] RE-351 Compose shared-tree repeated/multi-symbol streams with independent byte and per-symbol cursor oracles ([ENC-003](research/HUFFMAN_ENCODING.md)).
424
  - [ ] RE-352 Recover adaptive Huffman initialization/node allocation/rank/swap/frequency updates and actual codebooks, full message writers/readers/real-packet compositions, overflow/truncation and legal tree/symbol/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration.
 
 
 
 
 
 
422
  - [x] RE-350 Compose transmit/send/receive against supplied comb trees with exact symbol/cursor/byte round trips ([ENC-002](research/HUFFMAN_ENCODING.md)).
423
  - [x] RE-351 Compose shared-tree repeated/multi-symbol streams with independent byte and per-symbol cursor oracles ([ENC-003](research/HUFFMAN_ENCODING.md)).
424
  - [ ] RE-352 Recover adaptive Huffman initialization/node allocation/rank/swap/frequency updates and actual codebooks, full message writers/readers/real-packet compositions, overflow/truncation and legal tree/symbol/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration.
425
+
426
+ - [x] RE-353 Extract and verify paired native frequency tables, totals and conceptual training-schedule provenance ([INIT-001](research/HUFFMAN_INITIALIZATION.md)).
427
+ - [x] RE-354 Recover exact two-object clear requests, initial node/lookup pointers and sparse NYT layout/asymmetry ([INIT-002](research/HUFFMAN_INITIALIZATION.md)).
428
+ - [x] RE-355 Recover initialization flag/request and bounded ascending paired addRef scheduling/skip/termination ([INIT-003](research/HUFFMAN_INITIALIZATION.md)).
429
+ - [ ] RE-356 Recover addRef insertion/existing-symbol effects and increment/block/head/freelist/node/list swap recursion; reconstruct actual initialized trees/codewords from exact native seeds, compose real message/packet codecs and validate capacity/overflow/truncation, legal invariants and original-runtime/all-platform contracts.
docs/research/HUFFMAN_ENCODING.md CHANGED
@@ -35,3 +35,5 @@ Buffers begin at zero, while the separate dirty-buffer probes characterize retai
35
  Eight independent tests check dirty-buffer/root-path packing, nine-edge recursive thresholds, encoded bytes/symbols/cursors/canaries, supplied parent/child consistency, shared stream packing/repetition, zero-depth/empty streams, recomposed traces excluding native CALLs and paired outputs. Reports retain trace lengths/hashes and verified full graphs. No evaluator extension was needed.
36
 
37
  RE-352: recover Huffman initialization, node allocation/swapping/rank/frequency updates and actual game codebooks; compose full message writers/readers with real code lengths and packets; recover capacity/overflow/truncation and legal symbol/tree/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration before Rust implementation.
 
 
 
35
  Eight independent tests check dirty-buffer/root-path packing, nine-edge recursive thresholds, encoded bytes/symbols/cursors/canaries, supplied parent/child consistency, shared stream packing/repetition, zero-depth/empty streams, recomposed traces excluding native CALLs and paired outputs. Reports retain trace lengths/hashes and verified full graphs. No evaluator extension was needed.
36
 
37
  RE-352: recover Huffman initialization, node allocation/swapping/rank/frequency updates and actual game codebooks; compose full message writers/readers with real code lengths and packets; recover capacity/overflow/truncation and legal symbol/tree/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration before Rust implementation.
38
+
39
+ Initialization follow-up: [native frequencies and empty-tree probes](HUFFMAN_INITIALIZATION.md) recovers actual seed counts and bounded initialization/training requests (RE-353–355). Adaptive tree effects and final native codewords remain open.
docs/research/HUFFMAN_INITIALIZATION.md ADDED
@@ -0,0 +1,39 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Huffman frequencies, empty-tree initialization and training requests
2
+
3
+ Reproduce with `python3 scripts/analyze_huffman_initialization.py`; validate with `python3 scripts/test_huffman_initialization.py`. [huffman-initialization.json](huffman-initialization.json) retains four hash-verified graphs per Linux ELF64 engine: MSG_initHuffman, Huff_Init, Huff_addRef and increment. This follows [supplied-tree encoder compositions](HUFFMAN_ENCODING.md).
4
+
5
+ The actual frequency table is extracted from verified file-backed native images. Initialization and scheduling are bounded synthetic instruction evaluations: PUSH instructions are explicitly omitted, POP/RET and all callees are stopped, and memset effects are supplied. Small scheduling fixtures supply count tables and retain caller registers across stopped addRef requests. Huff_addRef and increment graphs are static next-step evidence; neither is evaluated here. Original processes and native callees never execute, and no trained tree/codebook or packet conformance is established.
6
+
7
+ ## INIT-001 — Exact native seed frequencies (RE-353)
8
+
9
+ The MSG_initHuffman comparison operands identify a table of 256 little-endian 32-bit counts. Both verified Linux images contain identical table bytes, with SHA-256:
10
+
11
+ `1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf`
12
+
13
+ The report retains every count, virtual address, file offset and source/disassembly provenance. All counts are positive and fit signed 32-bit comparison: minimum 740, maximum 250315, total 1,053,340. Symbol zero has count 250315; the first sixteen counts are `[250315,41193,6292,7106,3730,3750,6110,23283,33317,6950,7838,9714,9257,17259,3949,1778]`.
14
+
15
+ The loop implies 1,053,340 repetitions per tree, or 2,106,680 paired addRef requests. A conceptual schedule digest is generated from the extracted counts: for each symbol ascending, repeat `bytes(symbol, tree0, symbol, tree1)` count times. This generated digest records the implied request stream independently of native addresses. It is **not** a trace of all native training iterations and does not establish the resulting tree. Reconstructing a generic frequency-optimal tree without recovering native tie/rank/swap behavior could produce different codewords.
16
+
17
+ ## INIT-002 — Two cleared objects and initial NYT nodes (RE-354)
18
+
19
+ Four paired initialization probes use two virtual base addresses and dirty sentinel buffers. Huff_Init requests two adjacent memset operations, each of size 0xc830 (51,248 bytes), value zero: base and base +0xc830. The collector supplies those effects; native memset never executes. Both outside canaries retain their sentinel.
20
+
21
+ After the clears, each object's node count at +0 becomes one. Its initial node is at object +0x830 (2,096); native allocation address arithmetic computes 64*count minus 8*count, a 56-byte node stride. The root pointer +8, pointer slot +16 and symbol-256 lookup slot +0x820 all refer to this initial node. The second object's pointer slot +24 also refers to it; the first object's +24 remains zero. This asymmetry is retained rather than normalized. Full list-slot semantics require addRef/increment analysis.
22
+
23
+ The initial 56-byte node has symbol word 256 at +52 and weight word zero at +48. Child/parent pointers +0/+8/+16 and list-related slots +24/+32 are zero. The complete objects contain only the expected nonzero count/pointer/symbol bytes; the report records the sparse nonzero-byte image, not just selected fields. Slot +0x820 agrees with the transmit lookup formula `object +32 +8*symbol` for symbol 256. This supports identifying the initial not-yet-transmitted (NYT) node, while actual NYT insertion/escape behavior remains open.
24
+
25
+ These native layout offsets are evidence for reconstruction, not a required Rust memory layout. Capacities, node/pointer-pool ownership, alias invariants and allocator/error paths remain unresolved.
26
+
27
+ ## INIT-003 — Flag, ordering and bounded seed loop (RE-355)
28
+
29
+ The entry slice sets msgInit to one before requesting Huff_Init at the first object pointer. It resets local symbol/repetition registers to zero. The caller guard, thread/reentrancy behavior and initialization failure semantics are outside this slice.
30
+
31
+ Eight paired loop compositions supply four count tables: all zero; symbols zero twice and 255 three times; sparse symbols 1/127/128/254; and every symbol once. Symbol indices advance from zero through 255 in ascending order, and the loop terminates with local symbol 256. Each repetition requests Huff_addRef(first_object,symbol), then Huff_addRef(first_object +0xc830,symbol), before testing the next repetition. Zero-count symbols produce no requests. Supplied counts and all stopped request arguments are retained, with trace hashes/counts.
32
+
33
+ The native helper requests total 1,068 across these bounded fixtures: 534 per engine. Actual seed frequencies are extracted separately and are not substituted into an evaluated million-iteration loop. The small fixtures prove local loop ordering/skip/termination and paired object selection; they do not recover addRef effects, final weights or trained codewords.
34
+
35
+ ## Validation and next work
36
+
37
+ Eight independent tests verify exact frequency hash/summary, a separately generated per-request conceptual digest, entry flag/init arguments, complete bounded ascending/paired request order, exact clear spans/canaries, complete sparse object/NYT layout and asymmetry, stopped CALL traces and paired frequency/virtual-layout results. No evaluator extension was required.
38
+
39
+ RE-356: recover Huff_addRef insertion/existing-symbol branches, increment rank/block/head/freelist changes, node/list swaps and recursion; reconstruct initialized trees with the exact seed schedule and derive verified codewords; compose actual message encoders/decoders and packet fixtures, with capacity/truncation/overflow, legal invariants and original-runtime/all-platform validation.
docs/research/README.md CHANGED
@@ -131,3 +131,5 @@ The extension collector adds five focused tests for native tail calls, command-r
131
  [Raw-bit helpers and supplied Huffman trees](HUFFMAN_HELPERS.md) adds RE-345–347: byte clear/OR, raw round trips, supplied-tree decoding/null cursor paths and transmit lookup/request boundaries. Reproduce with `python3 scripts/analyze_huffman_helpers.py`; check with `python3 scripts/test_huffman_helpers.py`. Full send, real codebooks/messages and runtime/platform contracts remain open.
132
 
133
  [Bounded Huffman encoding and symbol streams](HUFFMAN_ENCODING.md) adds RE-349–351: unrolled/recursive send, supplied-tree symbol round trips and shared multi-symbol streams. Reproduce with `python3 scripts/analyze_huffman_encoding.py`; check with `python3 scripts/test_huffman_encoding.py`. Actual adaptive initialization/codebooks, message packets and runtime/platform contracts remain open.
 
 
 
131
  [Raw-bit helpers and supplied Huffman trees](HUFFMAN_HELPERS.md) adds RE-345–347: byte clear/OR, raw round trips, supplied-tree decoding/null cursor paths and transmit lookup/request boundaries. Reproduce with `python3 scripts/analyze_huffman_helpers.py`; check with `python3 scripts/test_huffman_helpers.py`. Full send, real codebooks/messages and runtime/platform contracts remain open.
132
 
133
  [Bounded Huffman encoding and symbol streams](HUFFMAN_ENCODING.md) adds RE-349–351: unrolled/recursive send, supplied-tree symbol round trips and shared multi-symbol streams. Reproduce with `python3 scripts/analyze_huffman_encoding.py`; check with `python3 scripts/test_huffman_encoding.py`. Actual adaptive initialization/codebooks, message packets and runtime/platform contracts remain open.
134
+
135
+ [Huffman frequencies and initialization](HUFFMAN_INITIALIZATION.md) adds RE-353–355: verified native frequencies, empty-tree layout and bounded paired seed requests. Reproduce with `python3 scripts/analyze_huffman_initialization.py`; check with `python3 scripts/test_huffman_initialization.py`. Adaptive updates, trained codewords and actual message/runtime/platform contracts remain open.
docs/research/huffman-initialization.json ADDED
The diff for this file is too large to render. See raw diff
 
scripts/analyze_huffman_initialization.py ADDED
@@ -0,0 +1,76 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Verified native frequencies, supplied memset initialization and seed-loop requests."""
3
+ import copy
4
+ import gzip
5
+ import hashlib
6
+ import json
7
+ from pathlib import Path
8
+ import re
9
+ import struct
10
+ from analyze_area_queries import site
11
+ from analyze_bsp_references import global_address,trace_fields
12
+ from analyze_collision_composition import evaluate,get_word,put_word
13
+ from analyze_engine_control import control_flow
14
+ from analyze_frame_timing import read_address
15
+ from analyze_native import elf
16
+ from analyze_qvm import write_json
17
+ from analyze_vm_extensions import native_functions,verified
18
+
19
+
20
+ SIZE=0xc830
21
+
22
+
23
+ def run(g,start,stops,m,regs):return evaluate(g,start,stops,m,regs,limit=10000,capture_registers=True)
24
+
25
+
26
+ def omit_push(g):
27
+ out=copy.deepcopy(g);addresses=[]
28
+ for x in out['instructions']:
29
+ if x['opcode']=='pushq':addresses.append(x['address']);x['opcode']='nop';x['operands']=''
30
+ return out,addresses
31
+
32
+
33
+ def main():
34
+ out={'native':{},'limitations':'Paired Linux ELF64 bounded synthetic initialization/schedule slices. Original processes/native callees never execute. PUSH omitted, POP/RET stopped; memset effects supplied. Real frequency table extracted from verified file-backed image, but conceptual full schedule digest is generated from counts, not evaluated through all native training iterations. Huff_addRef requests stop before calls; no trained/adaptive tree or actual codebook/runtime/platform conformance is established.'}
35
+ for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
36
+ r=json.loads(Path(f'docs/research/native/{label}.json').read_text());data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);meta,_=elf(data);a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());graphs={n:control_flow(fs[n]) for n in ['MSG_initHuffman','Huff_Init','Huff_addRef','increment']}
37
+ g,omitted=omit_push(graphs['MSG_initHuffman']);rows=g['instructions'];table_ins=next(x for x in rows if x['opcode']=='cmpl' and '(,%r13,4)' in x['operands']);address=int(re.search(r'0x([0-9a-f]+)\(',table_ins['operands'])[1],16);raw,offset=read_address(data,meta['sections'],address,1024);frequencies=list(struct.unpack('<256I',raw))
38
+ digest=hashlib.sha256()
39
+ # Relative tree ordinals keep this conceptual request digest independent of addresses.
40
+ for symbol,count in enumerate(frequencies):digest.update(bytes([symbol,0,symbol,1])*count)
41
+ init_idx=next(i for i,x in enumerate(rows) if '<Huff_Init>' in x['operands']);flag=global_address(next(x for x in rows if x['opcode']=='movl' and '(%rip)' in x['operands']))
42
+ entry=run(g,rows[0]['address'],[rows[init_idx]['address']],{}, {'%rsp':0x90000})
43
+ add_calls={x['address']:i for i,x in enumerate(rows) if '<Huff_addRef>' in x['operands']};end=site(g,'addq','$0x8, %rsp');schedule=[]
44
+ for name in ['zero','edges','sparse','ones']:
45
+ counts=[0]*256
46
+ if name=='edges':counts[0]=2;counts[255]=3
47
+ elif name=='sparse':counts[1]=1;counts[127]=2;counts[128]=1;counts[254]=2
48
+ elif name=='ones':counts=[1]*256
49
+ m=dict(entry['memory'])
50
+ for i,count in enumerate(counts):put_word(m,address+4*i,count)
51
+ regs=dict(entry['registers']);start=rows[init_idx+1]['address'];requests=[];trace=[]
52
+ while True:
53
+ t=run(g,start,list(add_calls)+[end],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
54
+ if t['stop']==end:break
55
+ requests.append([regs['rdi'],regs['rsi']]);start=rows[add_calls[t['stop']]+1]['address']
56
+ if len(requests)>512:raise ValueError('fixture request bound')
57
+ schedule.append({'fixture':name,'supplied_counts':counts,'requests':requests,'final_symbol':regs['rbp'],'flag_word':get_word(m,flag),**trace_fields({'trace':trace},False)})
58
+ init,init_omitted=omit_push(graphs['Huff_Init']);ir=init['instructions'];calls={x['address']:i for i,x in enumerate(ir) if x['opcode']=='callq'};end_init=next(x['address'] for x in ir if x['opcode']=='popq');probes=[]
59
+ for base,sentinel in [(0x10000,0xa5),(0x40000,255)]:
60
+ m={base+i:sentinel for i in range(-1,2*SIZE+1)};regs={'%rdi':base};start=ir[0]['address'];requests=[];trace=[]
61
+ while True:
62
+ t=run(init,start,list(calls)+[end_init],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
63
+ if t['stop']==end_init:break
64
+ requests.append([regs[k] for k in ['rdi','rsi','rdx']])
65
+ if 'memset@plt' not in ir[calls[t['stop']]]['operands']:raise ValueError('unexpected initialization callee')
66
+ for i in range(regs['rdx']):m[regs['rdi']+i]=regs['rsi']&255
67
+ start=ir[calls[t['stop']]+1]['address']
68
+ objects=[]
69
+ for obj in [base,base+SIZE]:
70
+ node=get_word(m,obj+8,8);objects.append({'offset':obj-base,'node_count':get_word(m,obj),'root_pointer':node,'list_pointers':[get_word(m,obj+x,8) for x in [16,24]],'nyt_lookup_pointer':get_word(m,obj+0x820,8),'node_pool_offset':node-obj,'node_bytes':[m[node+i] for i in range(56)],'nonzero_bytes':[[i,m[obj+i]] for i in range(SIZE) if m[obj+i]]})
71
+ probes.append({'base':base,'sentinel':sentinel,'memset_requests':requests,'objects':objects,'canaries':[m[base-1],m[base+2*SIZE]],**trace_fields({'trace':trace},True)})
72
+ out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,'frequency_table':{'address':address,'file_offset':offset,'sha256':hashlib.sha256(raw).hexdigest(),'counts':frequencies,'total':sum(frequencies),'paired_request_count':2*sum(frequencies),'conceptual_schedule_sha256':digest.hexdigest(),'digest_encoding':'for each symbol ascending, repeated count times: bytes(symbol,tree0,symbol,tree1)'},'entry_request':{'huff_init_pointer':entry['registers']['rdi'],'flag_word':get_word(entry['memory'],flag),'omitted_push_addresses':omitted,**trace_fields(entry,True)},'schedule_probes':schedule,'initialization_probes':probes,'init_omitted_push_addresses':init_omitted}
73
+ write_json(Path('docs/research/huffman-initialization.json'),out);print('Recovered paired 256-entry frequencies, eight schedule and four empty-tree initialization probes')
74
+
75
+
76
+ if __name__=='__main__':main()
scripts/test_huffman_initialization.py ADDED
@@ -0,0 +1,71 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Independent frequency, paired scheduling and empty-object layout oracles."""
3
+ import hashlib
4
+ import json
5
+ from pathlib import Path
6
+ import struct
7
+ import unittest
8
+
9
+
10
+ class HuffmanInitializationTests(unittest.TestCase):
11
+ @classmethod
12
+ def setUpClass(cls):cls.r=json.loads(Path('docs/research/huffman-initialization.json').read_text())
13
+
14
+ def test_verified_frequency_words_hash_and_summary(self):
15
+ for n in self.r['native'].values():
16
+ f=n['frequency_table'];counts=f['counts'];self.assertEqual(len(counts),256)
17
+ self.assertEqual(hashlib.sha256(struct.pack('<256I',*counts)).hexdigest(),'1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf')
18
+ self.assertEqual(f['sha256'],'1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf');self.assertEqual(sum(counts),1053340);self.assertEqual(f['total'],1053340);self.assertEqual(f['paired_request_count'],2106680)
19
+ self.assertEqual(min(counts),740);self.assertEqual(max(counts),250315);self.assertTrue(all(0<v<2**31 for v in counts))
20
+
21
+ def test_conceptual_schedule_digest_from_individual_requests(self):
22
+ n=next(iter(self.r['native'].values()));f=n['frequency_table'];digest=hashlib.sha256()
23
+ for symbol,count in enumerate(f['counts']):
24
+ for _ in range(count):
25
+ digest.update(bytes([symbol,0]));digest.update(bytes([symbol,1]))
26
+ for n in self.r['native'].values():self.assertEqual(n['frequency_table']['conceptual_schedule_sha256'],digest.hexdigest())
27
+
28
+ def test_seed_flag_and_init_request(self):
29
+ for n in self.r['native'].values():
30
+ e=n['entry_request'];self.assertEqual(e['flag_word'],1);self.assertEqual(len(e['omitted_push_addresses']),4)
31
+ first=n['schedule_probes'][1]['requests'][0][0];self.assertEqual(e['huff_init_pointer'],first)
32
+
33
+ def test_native_loop_ascending_symbols_and_paired_request_order(self):
34
+ for n in self.r['native'].values():
35
+ self.assertEqual(len(n['schedule_probes']),4);base=n['entry_request']['huff_init_pointer']
36
+ for p in n['schedule_probes']:
37
+ expected=[[base+tree*51248,symbol] for symbol,count in enumerate(p['supplied_counts']) for _ in range(count) for tree in [0,1]]
38
+ self.assertEqual(p['requests'],expected);self.assertEqual(p['final_symbol'],256);self.assertEqual(p['flag_word'],1)
39
+
40
+ def test_two_exact_clear_requests_and_outside_canaries(self):
41
+ for n in self.r['native'].values():
42
+ self.assertEqual(len(n['init_omitted_push_addresses']),1)
43
+ for p in n['initialization_probes']:
44
+ b=p['base'];self.assertEqual(p['memset_requests'],[[b,0,51248],[b+51248,0,51248]]);self.assertEqual(p['canaries'],[p['sentinel']]*2)
45
+
46
+ def test_complete_sparse_initialization_layout_and_nyt_node(self):
47
+ for n in self.r['native'].values():
48
+ for p in n['initialization_probes']:
49
+ for i,o in enumerate(p['objects']):
50
+ base=p['base']+i*51248;node=base+2096;expected=bytearray(51248)
51
+ expected[0:4]=(1).to_bytes(4,'little')
52
+ for slot in [8,16,2080]+([24] if i else []):expected[slot:slot+8]=node.to_bytes(8,'little')
53
+ expected[2096+52:2096+56]=(256).to_bytes(4,'little')
54
+ self.assertEqual(o['nonzero_bytes'],[[j,v] for j,v in enumerate(expected) if v]);self.assertEqual(o['root_pointer'],node);self.assertEqual(o['node_count'],1);self.assertEqual(o['node_pool_offset'],2096);self.assertEqual(o['nyt_lookup_pointer'],node)
55
+ self.assertEqual(o['node_bytes'],list(expected[2096:2096+56]));self.assertEqual(o['list_pointers'],[node,node if i else 0])
56
+
57
+ def test_explicit_traces_stop_before_calls(self):
58
+ for n in self.r['native'].values():
59
+ calls={x['address'] for g in n['graphs'].values() for x in g['instructions'] if x['opcode']=='callq'}
60
+ for p in [n['entry_request']]+n['initialization_probes']:
61
+ self.assertTrue(calls.isdisjoint(p['trace']));self.assertEqual(len(p['trace']),p['trace_length'])
62
+
63
+ def test_paired_frequencies_and_virtual_initialization_match(self):
64
+ a,b=self.r['native'].values()
65
+ for key in ['counts','sha256','total','paired_request_count','conceptual_schedule_sha256']:self.assertEqual(a['frequency_table'][key],b['frequency_table'][key])
66
+ def clean(rows):return [{k:v for k,v in p.items() if k not in ['trace','trace_sha256']} for p in rows]
67
+ self.assertEqual(clean(a['initialization_probes']),clean(b['initialization_probes']))
68
+ for n in [a,b]:self.assertEqual(len(n['graphs']),4)
69
+
70
+
71
+ if __name__=='__main__':unittest.main()