nmarshall commited on
Commit
d14d999
·
1 Parent(s): 37a2f91

Compose adaptive Huffman insertion and weight updates

Browse files
docs/ASSET_PIPELINE_TODO.md CHANGED
@@ -220,3 +220,5 @@ Huffman follow-up: [helper probes](research/HUFFMAN_HELPERS.md) establishes raw-
220
  Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md) validates bounded transmit/receive compositions with synthetic codebooks. Actual game codebooks and full packet contracts remain prerequisites for network/map/renderer integration (RE-352).
221
 
222
  Initialization follow-up: [native frequency tables](research/HUFFMAN_INITIALIZATION.md) preserves exact codec seeds and initial layouts. Recover adaptive training and actual codewords before real network/map/renderer integration; supplied-tree round trips alone remain insufficient (RE-356).
 
 
 
220
  Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md) validates bounded transmit/receive compositions with synthetic codebooks. Actual game codebooks and full packet contracts remain prerequisites for network/map/renderer integration (RE-352).
221
 
222
  Initialization follow-up: [native frequency tables](research/HUFFMAN_INITIALIZATION.md) preserves exact codec seeds and initial layouts. Recover adaptive training and actual codewords before real network/map/renderer integration; supplied-tree round trips alone remain insufficient (RE-356).
223
+
224
+ Adaptive-codec follow-up: [update compositions](research/HUFFMAN_UPDATES.md) validates sampled supplied-tree insertion/count/list/block behavior. Actual seed-trained codebooks, real packets and runtime/platform validation remain required before network/map/renderer integration (RE-360).
docs/FINDINGS.md CHANGED
@@ -390,3 +390,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
390
  **F-125 — Observed empty-tree initialization:** two 51,248-byte clear requests precede initial symbol-256 nodes at object +2,096, count one and lookup/root aliases. Pointer slot +24 differs between objects. Supplied clear effects and complete sparse layouts do not establish later allocation/list/adaptive behavior. [INIT-002](research/HUFFMAN_INITIALIZATION.md).
391
 
392
  **F-126 — Observed training requests:** entry sets msgInit before requesting Huff_Init; bounded count fixtures schedule symbols ascending with first-object then second-object addRef requests per repetition, ending at symbol 256. addRef effects and actual full seeded training remain unexecuted. [INIT-003](research/HUFFMAN_INITIALIZATION.md).
 
 
 
 
 
 
 
390
  **F-125 — Observed empty-tree initialization:** two 51,248-byte clear requests precede initial symbol-256 nodes at object +2,096, count one and lookup/root aliases. Pointer slot +24 differs between objects. Supplied clear effects and complete sparse layouts do not establish later allocation/list/adaptive behavior. [INIT-002](research/HUFFMAN_INITIALIZATION.md).
391
 
392
  **F-126 — Observed training requests:** entry sets msgInit before requesting Huff_Init; bounded count fixtures schedule symbols ascending with first-object then second-object addRef requests per repetition, ending at symbol 256. addRef effects and actual full seeded training remain unexecuted. [INIT-003](research/HUFFMAN_INITIALIZATION.md).
393
+
394
+ **F-127 — Observed addRef insertion:** input normalizes to its low byte; existing lookup forwards directly to increment. New symbols add a stride-56 leaf/internal pair around NYT and repair lookup/parent/list links. Sampled all-byte insertion reaches 513 nodes. Legal aliases/capacity and every insertion/tie path remain open. [UPDATE-001](research/HUFFMAN_UPDATES.md).
395
+
396
+ **F-128 — Observed adaptive count compositions:** repeated/new sequences retain occurrence-count leaf weights, root total and child-weight sums across recursive increment and reorder instructions. Null updates preserve memory; sampled recursion reaches depth eight. Real seeded training/codewords and complete runtime/ABI/platform behavior remain open. [UPDATE-002](research/HUFFMAN_UPDATES.md).
397
+
398
+ **F-129 — Observed rank/block reuse:** sampled rank lists cover all nodes with nondecreasing weights and reverse-link consistency; weight blocks share last-rank leader cells. Allocated cells partition into live heads and an acyclic free chain, with fifteen cells in the full-alphabet fixture. Complete swaps, second-object behavior and capacities remain open. [UPDATE-003](research/HUFFMAN_UPDATES.md).
docs/PROVISIONAL_TODO.md CHANGED
@@ -532,3 +532,5 @@ Research progress: [Huffman helper probes](research/HUFFMAN_HELPERS.md) complete
532
  Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) completes RE-349–351. Broad network/runtime milestones remain open; RE-352 tracks adaptive initialization/codebooks, real message packets and platform fixtures.
533
 
534
  Research progress: [Huffman initialization](research/HUFFMAN_INITIALIZATION.md) completes bounded RE-353–355. Broad network/runtime milestones remain open; RE-356 tracks adaptive updates, initialized codewords and actual message/packet/platform fixtures.
 
 
 
532
  Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) completes RE-349–351. Broad network/runtime milestones remain open; RE-352 tracks adaptive initialization/codebooks, real message packets and platform fixtures.
533
 
534
  Research progress: [Huffman initialization](research/HUFFMAN_INITIALIZATION.md) completes bounded RE-353–355. Broad network/runtime milestones remain open; RE-356 tracks adaptive updates, initialized codewords and actual message/packet/platform fixtures.
535
+
536
+ Research progress: [adaptive Huffman updates](research/HUFFMAN_UPDATES.md) completes bounded RE-357–359. Broad network/runtime milestones remain open; RE-360 tracks complete swap/tie paths, second-object equivalence, actual trained codebooks and real packet/platform fixtures.
docs/RESEARCH_LEDGER.md CHANGED
@@ -217,3 +217,7 @@ Completed bounded RE-349–351 with `analyze_huffman_encoding.py`: six verified
217
  ## 2026-10-11 — Native Huffman frequencies and initialization
218
 
219
  Completed bounded RE-353–355 with `analyze_huffman_initialization.py`: eight verified graphs across the Linux ELF64 engines, two identical file-backed 256-count frequency tables, four empty-tree initialization probes and eight seed-loop compositions. INIT-001–003/F-124–126 distinguish actual table extraction from generated conceptual schedule digests, supplied memset effects from native clear execution, exact sparse initial/NYT layouts from trained trees, pointer-slot asymmetry from inferred list semantics, and bounded paired addRef requests from adaptive updates. The tables total 1,053,340 repetitions per tree; their implied 2,106,680-request schedule is not evaluated as a full native training trace. The bounded schedules retain 1,068 stopped requests with explicit caller-register retention. PUSH omission and POP/RET/callee boundaries are documented; no original process or native callee executes. Eight independent tests verify frequencies/hash/summary, separately generated conceptual request digest, entry flag/init arguments, ascending paired schedules, clear spans/canaries, full sparse layouts, trace boundaries and paired results. All 318 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-352 remains open; RE-356 tracks addRef/increment/swaps/block/freelist effects, actual trained trees/codewords, full packet/message codecs and legal/runtime/all-platform invariants.
 
 
 
 
 
217
  ## 2026-10-11 — Native Huffman frequencies and initialization
218
 
219
  Completed bounded RE-353–355 with `analyze_huffman_initialization.py`: eight verified graphs across the Linux ELF64 engines, two identical file-backed 256-count frequency tables, four empty-tree initialization probes and eight seed-loop compositions. INIT-001–003/F-124–126 distinguish actual table extraction from generated conceptual schedule digests, supplied memset effects from native clear execution, exact sparse initial/NYT layouts from trained trees, pointer-slot asymmetry from inferred list semantics, and bounded paired addRef requests from adaptive updates. The tables total 1,053,340 repetitions per tree; their implied 2,106,680-request schedule is not evaluated as a full native training trace. The bounded schedules retain 1,068 stopped requests with explicit caller-register retention. PUSH omission and POP/RET/callee boundaries are documented; no original process or native callee executes. Eight independent tests verify frequencies/hash/summary, separately generated conceptual request digest, entry flag/init arguments, ascending paired schedules, clear spans/canaries, full sparse layouts, trace boundaries and paired results. All 318 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-352 remains open; RE-356 tracks addRef/increment/swaps/block/freelist effects, actual trained trees/codewords, full packet/message codecs and legal/runtime/all-platform invariants.
220
+
221
+ ## 2026-10-11 — Adaptive Huffman insertion and recursive updates
222
+
223
+ Completed bounded RE-357–359 with `analyze_huffman_updates.py`: four verified graphs across the Linux ELF64 engines, 22 adaptive sequences, twelve low-byte existing-entry dispatch probes and two null-increment probes. UPDATE-001–003/F-127–129 distinguish byte normalization from transmit's signed lookup, new leaf/internal insertion from reuse, sampled Counter/subtree/rank/head/freelist invariants from exhaustive tie/swap coverage, and actual source-matched initial bytes from full game-seed training. Composed sequences include every byte once, yielding 513 reachable nodes and fifteen allocated pointer cells; recursion reaches depth eight. Prefix states are retained for sequences up to 24 steps, with complete final states for longer cases. increment PUSH omission, stopped POP/RET/CALL boundaries and explicit recursive caller-register restoration are documented; no original process or native callee executes. Eight independent tests verify allocation/lookup/counts, tree/parent/subtree invariants, rank/reverse links, shared leaders, live/free cell partition/reuse, aliases/dispatch, null/recursive traces and paired results. All 326 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-356 remains open; RE-360 tracks branch-complete tie/swaps, second-object equivalence, malformed/alias/capacity/wrap states, efficient exact seed-trained codebook reconstruction and actual message/packet/runtime/all-platform contracts.
docs/REVERSE_ENGINEERING_TODO.md CHANGED
@@ -427,3 +427,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
427
  - [x] RE-354 Recover exact two-object clear requests, initial node/lookup pointers and sparse NYT layout/asymmetry ([INIT-002](research/HUFFMAN_INITIALIZATION.md)).
428
  - [x] RE-355 Recover initialization flag/request and bounded ascending paired addRef scheduling/skip/termination ([INIT-003](research/HUFFMAN_INITIALIZATION.md)).
429
  - [ ] RE-356 Recover addRef insertion/existing-symbol effects and increment/block/head/freelist/node/list swap recursion; reconstruct actual initialized trees/codewords from exact native seeds, compose real message/packet codecs and validate capacity/overflow/truncation, legal invariants and original-runtime/all-platform contracts.
 
 
 
 
 
 
427
  - [x] RE-354 Recover exact two-object clear requests, initial node/lookup pointers and sparse NYT layout/asymmetry ([INIT-002](research/HUFFMAN_INITIALIZATION.md)).
428
  - [x] RE-355 Recover initialization flag/request and bounded ascending paired addRef scheduling/skip/termination ([INIT-003](research/HUFFMAN_INITIALIZATION.md)).
429
  - [ ] RE-356 Recover addRef insertion/existing-symbol effects and increment/block/head/freelist/node/list swap recursion; reconstruct actual initialized trees/codewords from exact native seeds, compose real message/packet codecs and validate capacity/overflow/truncation, legal invariants and original-runtime/all-platform contracts.
430
+
431
+ - [x] RE-357 Recover low-byte existing/new dispatch and bounded NYT/node/list/lookup insertion compositions ([UPDATE-001](research/HUFFMAN_UPDATES.md)).
432
+ - [x] RE-358 Compose repeated/new symbol and recursive increment sequences with independent count/tree/subtree-weight invariants ([UPDATE-002](research/HUFFMAN_UPDATES.md)).
433
+ - [x] RE-359 Recover sampled rank links, shared block leaders and live/free pointer-cell partition/reuse ([UPDATE-003](research/HUFFMAN_UPDATES.md)).
434
+ - [ ] RE-360 Recover branch-complete swap/tie/parent/leader behavior and second-object update equivalence; validate malformed/alias/capacity/wrap states and legal callers, efficiently reconstruct actual seed-trained codebooks with equivalence checks, and compose real message/packet/original-runtime/all-platform fixtures.
docs/research/HUFFMAN_INITIALIZATION.md CHANGED
@@ -37,3 +37,5 @@ The native helper requests total 1,068 across these bounded fixtures: 534 per en
37
  Eight independent tests verify exact frequency hash/summary, a separately generated per-request conceptual digest, entry flag/init arguments, complete bounded ascending/paired request order, exact clear spans/canaries, complete sparse object/NYT layout and asymmetry, stopped CALL traces and paired frequency/virtual-layout results. No evaluator extension was required.
38
 
39
  RE-356: recover Huff_addRef insertion/existing-symbol branches, increment rank/block/head/freelist changes, node/list swaps and recursion; reconstruct initialized trees with the exact seed schedule and derive verified codewords; compose actual message encoders/decoders and packet fixtures, with capacity/truncation/overflow, legal invariants and original-runtime/all-platform validation.
 
 
 
37
  Eight independent tests verify exact frequency hash/summary, a separately generated per-request conceptual digest, entry flag/init arguments, complete bounded ascending/paired request order, exact clear spans/canaries, complete sparse object/NYT layout and asymmetry, stopped CALL traces and paired frequency/virtual-layout results. No evaluator extension was required.
38
 
39
  RE-356: recover Huff_addRef insertion/existing-symbol branches, increment rank/block/head/freelist changes, node/list swaps and recursion; reconstruct initialized trees with the exact seed schedule and derive verified codewords; compose actual message encoders/decoders and packet fixtures, with capacity/truncation/overflow, legal invariants and original-runtime/all-platform validation.
40
+
41
+ Update follow-up: [bounded adaptive compositions](HUFFMAN_UPDATES.md) recovers insertion and increment effects with independent tree/list/block invariants (RE-357–359). Full seed-trained trees and actual codewords remain open.
docs/research/HUFFMAN_UPDATES.md ADDED
@@ -0,0 +1,37 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Bounded adaptive Huffman insertion and increment
2
+
3
+ Reproduce with `python3 scripts/analyze_huffman_updates.py`; validate with `python3 scripts/test_huffman_updates.py`. [huffman-updates.json](huffman-updates.json) retains two hash-verified graphs per Linux ELF64 engine: Huff_addRef and increment. Initial object bytes are supplied from the source/disassembly-matched [initialization report](HUFFMAN_INITIALIZATION.md), whose full report hash is recorded.
4
+
5
+ These compositions evaluate native instructions over virtual objects. The increment PUSH is changed to NOP and evaluation stops before POP/RET; recursive CALLs are stopped and their bodies evaluated explicitly, restoring every caller register while retaining child memory effects. No original process, native callee, memset, or actual full seed-training execution occurs. Validity checks apply only to the sampled finite trees and sequences; they do not prove ABI, all adaptive paths, original-runtime or platform conformance.
6
+
7
+ ## UPDATE-001 — Low-byte dispatch and NYT insertion (RE-357)
8
+
9
+ Huff_addRef first zero-extends the input's low byte, selecting lookup `object +32 +8*byte`. An existing entry forwards that node into increment without prior memory writes. Twelve paired isolated probes cover input words 0/1/255/256/257/0xffffffff with explicitly supplied existing entries. The supplied targets use the initial NYT address solely to characterize dispatch; these aliases are not legal symbol/tree fixtures. Unlike the previously observed signed index in Huff_offsetTransmit, addRef locally normalizes its input to a byte. Valid callers still require audit.
10
+
11
+ For a new byte, the object node count increases by two. A leaf uses the old count's slot and an internal node the following slot, with addresses `object +0x830 +56*index`. The leaf's symbol is the normalized byte; the internal symbol is 257. Both start at weight one. The old NYT becomes the internal node's left child and the new symbol its right child. The internal node replaces NYT at its former parent or becomes root, and parent/list/lookup pointers are repaired. The subsequent increment request starts at the new internal node's parent; when the first symbol creates a root, that request receives null.
12
+
13
+ The first insertion into the supplied empty object yields node count three: NYT at index zero with weight zero; byte leaf at index one with weight one; root internal at index two with weight one. NYT and leaf share the root as parent, and the two positive-weight nodes share a leader cell. Every new byte creates exactly one additional leaf/internal pair in sampled sequences. Visiting all 256 byte symbols once yields 513 reachable nodes, with lookup entries for every byte and NYT 256.
14
+
15
+ ## UPDATE-002 — Repeated symbols and recursive weights (RE-358)
16
+
17
+ Twenty-two paired sequence compositions cover empty input, first bytes 0/255, sixteen repeats of zero, alternating 0/1, ascending/reversed eight-byte orders, two sixteen-byte cycles, repeated boundary-byte mixtures and the complete ascending byte alphabet. Updates retain complete prefix states for sequences of length at most 24; longer sequences retain final states and all dispatch/trace summaries.
18
+
19
+ Existing leaves are reused. increment updates weight and may reorder tree/list nodes before recursively updating parents; the caller's post-recursion adjustments also execute in these compositions. Independent Counter oracles verify every leaf weight equals its occurrence count. A separate tree traversal verifies root weight equals sequence length, every internal weight equals its two child weights, child/parent pointers agree, the graph is acyclic and every allocated node is reachable. NYT stays at index zero with weight zero.
20
+
21
+ Null increment retains all supplied memory. Recursive compositions reach depth eight in the full byte-alphabet fixture. This is bounded recursion through current mutated parent links, not a substitute for proving every tie/swap case or ABI register preservation. Trace summaries describe synthetic compositions; no native CALL executes implicitly. Exact trained codewords require the real 1,053,340-repetition seed sequence and all relevant tie behavior.
22
+
23
+ ## UPDATE-003 — Rank links, block leaders and pointer-cell reuse (RE-359)
24
+
25
+ The sampled node fields are left +0, right +8, parent +16, next +24, previous +32, leader-cell pointer +40, weight +48 and symbol +52. Node layout is 56 bytes. The rank list begins at the object's +16 pointer, which retains NYT as its start. Object +24 remains zero in these first-object compositions; the asymmetric second-object initialization was not composed here.
26
+
27
+ Independent list traversal verifies every node appears exactly once, next/previous links agree and weights are nondecreasing. Positive-weight nodes of one weight share one cell, and each cell points to the last node of that weight in rank order. NYT has no leader cell. These observed invariants provide an independent check of the native swap/head-maintenance paths exercised by the sequence corpus, not exhaustive coverage of arbitrary malformed or tied trees.
28
+
29
+ Pointer cells allocate at `object +0xb030 +8*index`, using the counter at +4. The free-cell chain begins at object +0x828 and stores its next link in the freed cell itself. The native routines reuse those cells when creating new weight blocks. Tests partition every allocated cell into exactly one live leader cell or free-chain member, with no duplicate/cyclic free entries in the sampled states. The full alphabet requires only fifteen allocated cells for 513 nodes; thus cells are shared/reused rather than allocated per node. Exact capacity bounds and overflow/error behavior remain open.
30
+
31
+ The object-wide outside canaries are preserved for all sequences. This does not prove arbitrary writes remain inside pool capacity or validate aliased/corrupted objects.
32
+
33
+ ## Validation and next work
34
+
35
+ Eight independent tests verify node allocation/lookup/Counter counts, complete tree reachability/parent/subtree weights, rank ordering/reverse links, leader cells, live/free pool partition and reuse, dispatch/low-byte aliases, null/recursive stopped-call behavior and paired results. A fresh isolated root increment additionally verifies its weight change and call-free evaluated trace. No evaluator extension was needed.
36
+
37
+ RE-360: audit individual tree/list swaps and parent/leader tie gates with branch-complete fixtures, second-object initialization/update equivalence, malformed/aliased/capacity/wrap states and legal callers; reconstruct actual initialized codebooks efficiently using the exact seed schedule with native-equivalence checks; compose message encoders/decoders and real packets and validate original-runtime/all-platform contracts.
docs/research/README.md CHANGED
@@ -133,3 +133,5 @@ The extension collector adds five focused tests for native tail calls, command-r
133
  [Bounded Huffman encoding and symbol streams](HUFFMAN_ENCODING.md) adds RE-349–351: unrolled/recursive send, supplied-tree symbol round trips and shared multi-symbol streams. Reproduce with `python3 scripts/analyze_huffman_encoding.py`; check with `python3 scripts/test_huffman_encoding.py`. Actual adaptive initialization/codebooks, message packets and runtime/platform contracts remain open.
134
 
135
  [Huffman frequencies and initialization](HUFFMAN_INITIALIZATION.md) adds RE-353–355: verified native frequencies, empty-tree layout and bounded paired seed requests. Reproduce with `python3 scripts/analyze_huffman_initialization.py`; check with `python3 scripts/test_huffman_initialization.py`. Adaptive updates, trained codewords and actual message/runtime/platform contracts remain open.
 
 
 
133
  [Bounded Huffman encoding and symbol streams](HUFFMAN_ENCODING.md) adds RE-349–351: unrolled/recursive send, supplied-tree symbol round trips and shared multi-symbol streams. Reproduce with `python3 scripts/analyze_huffman_encoding.py`; check with `python3 scripts/test_huffman_encoding.py`. Actual adaptive initialization/codebooks, message packets and runtime/platform contracts remain open.
134
 
135
  [Huffman frequencies and initialization](HUFFMAN_INITIALIZATION.md) adds RE-353–355: verified native frequencies, empty-tree layout and bounded paired seed requests. Reproduce with `python3 scripts/analyze_huffman_initialization.py`; check with `python3 scripts/test_huffman_initialization.py`. Adaptive updates, trained codewords and actual message/runtime/platform contracts remain open.
136
+
137
+ [Adaptive Huffman update compositions](HUFFMAN_UPDATES.md) adds RE-357–359: insertion/dispatch, recursive count/tree invariants and rank/block/freelist reuse. Reproduce with `python3 scripts/analyze_huffman_updates.py`; check with `python3 scripts/test_huffman_updates.py`. Full branch/tie/second-object/seeded-codebook/message/runtime/platform contracts remain open.
docs/research/huffman-updates.json ADDED
The diff for this file is too large to render. See raw diff
 
scripts/analyze_huffman_updates.py ADDED
@@ -0,0 +1,69 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Bounded adaptive Huffman insertion and recursive increment compositions."""
3
+ import copy
4
+ import gzip
5
+ import hashlib
6
+ import json
7
+ from pathlib import Path
8
+ from analyze_bsp_references import trace_fields
9
+ from analyze_collision_composition import evaluate,get_word,put_word
10
+ from analyze_engine_control import control_flow
11
+ from analyze_qvm import write_json
12
+ from analyze_vm_extensions import native_functions,verified
13
+
14
+ BASE=0x10000
15
+
16
+
17
+ def run(g,start,stops,m,regs):return evaluate(g,start,stops,m,regs,limit=10000,capture_registers=True)
18
+
19
+
20
+ def increment(g,m,obj,node,depth=1):
21
+ if depth>64:raise ValueError('synthetic increment recursion bound')
22
+ graph=copy.deepcopy(g)
23
+ for x in graph['instructions']:
24
+ if x['opcode']=='pushq':x['opcode']='nop';x['operands']=''
25
+ rows=graph['instructions'];ends=[x['address'] for x in rows if x['opcode']=='popq'];idx=next(i for i,x in enumerate(rows) if x['opcode']=='callq');call=rows[idx]['address'];start=rows[0]['address'];regs={'%rdi':obj,'%rsi':node};trace=[];maximum=depth;frames=1
26
+ while True:
27
+ t=run(graph,start,ends+[call],m,regs);m=t['memory'];regs=t['registers'];trace+=t['trace']
28
+ if t['stop'] in ends:break
29
+ child=increment(g,m,regs['rdi'],regs['rsi'],depth+1);m=child['memory'];trace+=child['trace'];frames+=child['frames'];maximum=max(maximum,child['maximum_depth']);start=rows[idx+1]['address']
30
+ return {'memory':m,'trace':trace,'frames':frames,'maximum_depth':maximum}
31
+
32
+
33
+ def state(m,obj=BASE):
34
+ count=get_word(m,obj);nodes=[]
35
+ if count>513:raise ValueError('fixture node capacity')
36
+ for i in range(count):
37
+ addr=obj+0x830+56*i;nodes.append({'index':i,'address':addr,'left':get_word(m,addr,8),'right':get_word(m,addr+8,8),'parent':get_word(m,addr+16,8),'next':get_word(m,addr+24,8),'previous':get_word(m,addr+32,8),'head_cell':get_word(m,addr+40,8),'weight':get_word(m,addr+48),'symbol':get_word(m,addr+52)})
38
+ heads=sorted({n['head_cell'] for n in nodes if n['head_cell']});pool=get_word(m,obj+4)
39
+ return {'node_count':count,'pointer_cell_count':pool,'root':get_word(m,obj+8,8),'list_start':get_word(m,obj+16,8),'tail_slot':get_word(m,obj+24,8),'free_head':get_word(m,obj+0x828,8),'nodes':nodes,'heads':[[p,get_word(m,p,8)] for p in heads],'pointer_cells':[get_word(m,obj+0xb030+8*i,8) for i in range(pool)],'lookup':[[i,get_word(m,obj+32+8*i,8)] for i in range(257) if get_word(m,obj+32+8*i,8)]}
40
+
41
+
42
+ def main():
43
+ seeds=json.loads(Path('docs/research/huffman-initialization.json').read_text());out={'native':{},'limitations':'Two Linux ELF64 engines, bounded synthetic adaptive instruction compositions. Initial object bytes supplied from hash-matched prior initialization report; original memset/process/native callees never execute. increment PUSH omitted, POP/RET stopped; recursive callers restored explicitly. Full real million-repetition seed training, actual game codewords/messages, malformed/cyclic/overflow/all-platform/ABI/runtime invariants remain open.'}
44
+ sequences=[[],[0],[255],[0]*16,[0,1],[0,1]*8,list(range(8)),list(reversed(range(8))),list(range(16))+list(range(16)),[255,0,128,127,1,254]*4,list(range(256))]
45
+ for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
46
+ r=json.loads(Path(f'docs/research/native/{label}.json').read_text());verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());graphs={n:control_flow(fs[n]) for n in ['Huff_addRef','increment']};g=graphs['Huff_addRef'];end=graphs['increment']['instructions'][0]['address']
47
+ prior=seeds['native'][label]
48
+ if prior['source']!=r['source'] or prior['disassembly_sha256']!=a['uncompressed_sha256']:raise ValueError('initialization provenance mismatch')
49
+ seed=prior['initialization_probes'][0]['objects'][0];memory={BASE+i:0 for i in range(0xc830)}
50
+ for offset,value in seed['nonzero_bytes']:memory[BASE+offset]=value
51
+ memory[BASE-1]=memory[BASE+0xc830]=0x7b;records=[]
52
+ for sequence in sequences:
53
+ m=dict(memory);trace=[];steps=[];maximum=0;frames=0;dispatch=[]
54
+ for symbol in sequence:
55
+ before=get_word(m,BASE+32+8*(symbol&255),8);t=run(g,g['instructions'][0]['address'],[end],m,{'%rdi':BASE,'%rsi':symbol});m=t['memory'];trace+=t['trace'];regs=t['registers'];dispatch.append([symbol,'existing' if before else 'new',regs['rsi']])
56
+ c=increment(graphs['increment'],m,regs['rdi'],regs['rsi']);m=c['memory'];trace+=c['trace'];maximum=max(maximum,c['maximum_depth']);frames+=c['frames']
57
+ if len(sequence)<=24:steps.append(state(m))
58
+ records.append({'symbols':sequence,'dispatch':dispatch,'steps':steps,'final_state':state(m),'increment_frames':frames,'maximum_increment_depth':maximum,'canaries':[m[BASE-1],m[BASE+0xc830]],**trace_fields({'trace':trace},False)})
59
+ # Existing-entry normalization is isolated from increment effects.
60
+ aliases=[]
61
+ for word in [0,1,255,256,257,0xffffffff]:
62
+ m=dict(memory);target=BASE+0x830;put_word(m,BASE+32+8*(word&255),target,8);before=dict(m);t=run(g,g['instructions'][0]['address'],[end],m,{'%rdi':BASE,'%rsi':word})
63
+ aliases.append({'input_word':word,'normalized_byte':word&255,'increment_node':t['registers']['rsi'],'memory_unchanged':t['memory']==before,**trace_fields(t,True)})
64
+ null=increment(graphs['increment'],memory,BASE,0)
65
+ out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'seed_report_sha256':hashlib.sha256(Path('docs/research/huffman-initialization.json').read_bytes()).hexdigest(),'graphs':graphs,'sequences':records,'existing_alias_probes':aliases,'null_increment_probe':{'memory_unchanged':null['memory']==memory,**trace_fields(null,True)}}
66
+ write_json(Path('docs/research/huffman-updates.json'),out);print('Recovered 22 adaptive sequences, twelve byte-normalized dispatch and two null-increment probes')
67
+
68
+
69
+ if __name__=='__main__':main()
scripts/test_huffman_updates.py ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Independent count/tree/rank/block/freelist invariants for bounded updates."""
3
+ from collections import Counter
4
+ import json
5
+ from pathlib import Path
6
+ import unittest
7
+ from analyze_huffman_updates import BASE,increment
8
+ from analyze_collision_composition import get_word,put_word
9
+
10
+
11
+ def snapshots(p):
12
+ return [(s,p['symbols'][:i+1]) for i,s in enumerate(p['steps'])]+[(p['final_state'],p['symbols'])]
13
+
14
+
15
+ class HuffmanUpdateTests(unittest.TestCase):
16
+ @classmethod
17
+ def setUpClass(cls):cls.r=json.loads(Path('docs/research/huffman-updates.json').read_text())
18
+
19
+ def test_node_allocation_lookup_and_independent_frequency_counts(self):
20
+ for n in self.r['native'].values():
21
+ self.assertEqual(len(n['sequences']),11)
22
+ for p in n['sequences']:
23
+ self.assertEqual(p['canaries'],[0x7b]*2)
24
+ for s,sequence in snapshots(p):
25
+ counts=Counter(sequence);nodes={x['address']:x for x in s['nodes']};lookup=dict(s['lookup'])
26
+ self.assertEqual(s['node_count'],1+2*len(counts));self.assertEqual(set(lookup),set(counts)|{256})
27
+ for symbol,count in counts.items():self.assertEqual(nodes[lookup[symbol]]['weight'],count);self.assertEqual(nodes[lookup[symbol]]['symbol'],symbol)
28
+ self.assertEqual(lookup[256],BASE+0x830);self.assertEqual(nodes[lookup[256]]['weight'],0)
29
+ for i,node in enumerate(s['nodes']):self.assertEqual(node['address'],BASE+0x830+56*i)
30
+
31
+ def test_independent_tree_reachability_parent_links_and_subtree_weights(self):
32
+ for n in self.r['native'].values():
33
+ for p in n['sequences']:
34
+ for s,sequence in snapshots(p):
35
+ nodes={x['address']:x for x in s['nodes']};visited=set()
36
+ def walk(addr,parent):
37
+ self.assertNotIn(addr,visited);visited.add(addr);node=nodes[addr];self.assertEqual(node['parent'],parent)
38
+ if node['symbol']!=257:self.assertEqual([node['left'],node['right']],[0,0]);return node['weight']
39
+ weight=walk(node['left'],addr)+walk(node['right'],addr);self.assertEqual(node['weight'],weight);return weight
40
+ self.assertEqual(walk(s['root'],0),len(sequence));self.assertEqual(visited,set(nodes))
41
+
42
+ def test_rank_list_bidirectional_coverage_and_non_decreasing_weights(self):
43
+ for n in self.r['native'].values():
44
+ for p in n['sequences']:
45
+ for s,_ in snapshots(p):
46
+ nodes={x['address']:x for x in s['nodes']};prev=0;cur=s['list_start'];chain=[]
47
+ while cur:
48
+ self.assertNotIn(cur,chain);node=nodes[cur];self.assertEqual(node['previous'],prev);chain.append(cur);prev=cur;cur=node['next']
49
+ self.assertEqual(set(chain),set(nodes));self.assertEqual([nodes[a]['weight'] for a in chain],sorted(nodes[a]['weight'] for a in chain));self.assertEqual(s['tail_slot'],0)
50
+
51
+ def test_weight_blocks_share_last_rank_leader_cells(self):
52
+ for n in self.r['native'].values():
53
+ for p in n['sequences']:
54
+ for s,_ in snapshots(p):
55
+ nodes={x['address']:x for x in s['nodes']};heads=dict(s['heads']);chain=[];cur=s['list_start']
56
+ while cur:chain.append(cur);cur=nodes[cur]['next']
57
+ cells={}
58
+ for addr in chain:
59
+ node=nodes[addr]
60
+ if not node['weight']:self.assertEqual(node['head_cell'],0);continue
61
+ cell=node['head_cell'];self.assertNotEqual(cell,0);cells.setdefault(node['weight'],cell);self.assertEqual(cells[node['weight']],cell)
62
+ leader=next(a for a in reversed(chain) if nodes[a]['weight']==node['weight']);self.assertEqual(heads[cell],leader)
63
+ self.assertEqual(len(set(cells.values())),len(cells))
64
+
65
+ def test_pointer_pool_live_and_free_partition_and_reuse(self):
66
+ for n in self.r['native'].values():
67
+ saw_free=False
68
+ for p in n['sequences']:
69
+ for s,_ in snapshots(p):
70
+ pool={BASE+0xb030+8*i:v for i,v in enumerate(s['pointer_cells'])};live={a for a,_ in s['heads']};free=set();cur=s['free_head']
71
+ while cur:self.assertIn(cur,pool);self.assertNotIn(cur,free);free.add(cur);cur=pool[cur]
72
+ self.assertFalse(live&free);self.assertEqual(live|free,set(pool));saw_free|=bool(free)
73
+ self.assertTrue(saw_free)
74
+ full=next(p for p in n['sequences'] if len(p['symbols'])==256)['final_state'];self.assertEqual(full['node_count'],513);self.assertLess(full['pointer_cell_count'],full['node_count'])
75
+
76
+ def test_new_and_existing_dispatch_and_byte_aliases(self):
77
+ for n in self.r['native'].values():
78
+ for p in n['sequences']:
79
+ seen=set()
80
+ for symbol,branch,node in p['dispatch']:
81
+ self.assertEqual(branch,'existing' if symbol in seen else 'new')
82
+ if branch=='new' and not seen:self.assertEqual(node,0)
83
+ seen.add(symbol)
84
+ for p in n['existing_alias_probes']:self.assertEqual(p['normalized_byte'],p['input_word']&255);self.assertEqual(p['increment_node'],BASE+0x830);self.assertTrue(p['memory_unchanged'])
85
+
86
+ def test_null_and_recursive_increment_stops_before_native_calls(self):
87
+ for n in self.r['native'].values():
88
+ self.assertTrue(n['null_increment_probe']['memory_unchanged']);self.assertTrue(any(p['maximum_increment_depth']>=8 for p in n['sequences']))
89
+ p=next(p for p in n['sequences'] if p['symbols']==[0]);s=p['final_state'];m={BASE+i:0 for i in range(0xc830)}
90
+ # Recompose an isolated non-null root using recorded first-insertion state.
91
+ for node in s['nodes']:
92
+ for offset,key in [(0,'left'),(8,'right'),(16,'parent'),(24,'next'),(32,'previous'),(40,'head_cell')]:put_word(m,node['address']+offset,node[key],8)
93
+ put_word(m,node['address']+48,node['weight']);put_word(m,node['address']+52,node['symbol'])
94
+ put_word(m,BASE+0x828,s['free_head'],8);put_word(m,BASE+4,s['pointer_cell_count'])
95
+ for cell,value in s['heads']:put_word(m,cell,value,8)
96
+ t=increment(n['graphs']['increment'],m,BASE,s['root']);calls={x['address'] for x in n['graphs']['increment']['instructions'] if x['opcode']=='callq'};self.assertTrue(calls.isdisjoint(t['trace']));self.assertEqual(get_word(t['memory'],s['root']+48),2)
97
+
98
+ def test_paired_results_without_instruction_hashes(self):
99
+ a,b=self.r['native'].values()
100
+ for key in ['sequences','existing_alias_probes']:
101
+ def clean(rows):return [{k:v for k,v in p.items() if k not in ['trace','trace_sha256']} for p in rows]
102
+ self.assertEqual(clean(a[key]),clean(b[key]))
103
+
104
+
105
+ if __name__=='__main__':unittest.main()