Recover native Huffman frequencies and initialization
Browse files- docs/ASSET_PIPELINE_TODO.md +2 -0
- docs/FINDINGS.md +6 -0
- docs/PROVISIONAL_TODO.md +2 -0
- docs/RESEARCH_LEDGER.md +4 -0
- docs/REVERSE_ENGINEERING_TODO.md +5 -0
- docs/research/HUFFMAN_ENCODING.md +2 -0
- docs/research/HUFFMAN_INITIALIZATION.md +39 -0
- docs/research/README.md +2 -0
- docs/research/huffman-initialization.json +0 -0
- scripts/analyze_huffman_initialization.py +76 -0
- scripts/test_huffman_initialization.py +71 -0
docs/ASSET_PIPELINE_TODO.md
CHANGED
|
@@ -218,3 +218,5 @@ Message-bit follow-up: [read modes and assembly](research/MESSAGE_BITS.md) estab
|
|
| 218 |
Huffman follow-up: [helper probes](research/HUFFMAN_HELPERS.md) establishes raw-bit round trips and supplied-tree decoding, with explicit null-path cursor differences. Real initialized codebooks and message fixtures remain prerequisites for network/map/renderer integration (RE-348).
|
| 219 |
|
| 220 |
Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md) validates bounded transmit/receive compositions with synthetic codebooks. Actual game codebooks and full packet contracts remain prerequisites for network/map/renderer integration (RE-352).
|
|
|
|
|
|
|
|
|
| 218 |
Huffman follow-up: [helper probes](research/HUFFMAN_HELPERS.md) establishes raw-bit round trips and supplied-tree decoding, with explicit null-path cursor differences. Real initialized codebooks and message fixtures remain prerequisites for network/map/renderer integration (RE-348).
|
| 219 |
|
| 220 |
Encoder follow-up: [supplied-tree symbol streams](research/HUFFMAN_ENCODING.md) validates bounded transmit/receive compositions with synthetic codebooks. Actual game codebooks and full packet contracts remain prerequisites for network/map/renderer integration (RE-352).
|
| 221 |
+
|
| 222 |
+
Initialization follow-up: [native frequency tables](research/HUFFMAN_INITIALIZATION.md) preserves exact codec seeds and initial layouts. Recover adaptive training and actual codewords before real network/map/renderer integration; supplied-tree round trips alone remain insufficient (RE-356).
|
docs/FINDINGS.md
CHANGED
|
@@ -384,3 +384,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
|
|
| 384 |
**F-122 — Observed supplied-codebook round trips:** composed transmit/send/receive returns symbol 42 with matching bytes and cursors for sampled paths up to 32 edges, including a synthetic root-leaf zero-length code. Actual initialized trees and game packet compatibility remain open. [ENC-002](research/HUFFMAN_ENCODING.md).
|
| 385 |
|
| 386 |
**F-123 — Observed shared symbol streams:** supplied depth-three codebooks encode/decode repeated and mixed symbols across byte boundaries with three-bit cursor increments. No tree updates occur inside these sampled helpers; actual adaptive initialization/update ownership and complete message codecs remain open. [ENC-003](research/HUFFMAN_ENCODING.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 384 |
**F-122 — Observed supplied-codebook round trips:** composed transmit/send/receive returns symbol 42 with matching bytes and cursors for sampled paths up to 32 edges, including a synthetic root-leaf zero-length code. Actual initialized trees and game packet compatibility remain open. [ENC-002](research/HUFFMAN_ENCODING.md).
|
| 385 |
|
| 386 |
**F-123 — Observed shared symbol streams:** supplied depth-three codebooks encode/decode repeated and mixed symbols across byte boundaries with three-bit cursor increments. No tree updates occur inside these sampled helpers; actual adaptive initialization/update ownership and complete message codecs remain open. [ENC-003](research/HUFFMAN_ENCODING.md).
|
| 387 |
+
|
| 388 |
+
**F-124 — Observed native frequencies:** both Linux images contain the same verified 256-count table, totaling 1,053,340 repetitions per tree. A generated conceptual paired schedule is retained separately from native execution; trained trees and codewords remain open. [INIT-001](research/HUFFMAN_INITIALIZATION.md).
|
| 389 |
+
|
| 390 |
+
**F-125 — Observed empty-tree initialization:** two 51,248-byte clear requests precede initial symbol-256 nodes at object +2,096, count one and lookup/root aliases. Pointer slot +24 differs between objects. Supplied clear effects and complete sparse layouts do not establish later allocation/list/adaptive behavior. [INIT-002](research/HUFFMAN_INITIALIZATION.md).
|
| 391 |
+
|
| 392 |
+
**F-126 — Observed training requests:** entry sets msgInit before requesting Huff_Init; bounded count fixtures schedule symbols ascending with first-object then second-object addRef requests per repetition, ending at symbol 256. addRef effects and actual full seeded training remain unexecuted. [INIT-003](research/HUFFMAN_INITIALIZATION.md).
|
docs/PROVISIONAL_TODO.md
CHANGED
|
@@ -530,3 +530,5 @@ Research progress: [message read modes and assembly](research/MESSAGE_BITS.md) c
|
|
| 530 |
Research progress: [Huffman helper probes](research/HUFFMAN_HELPERS.md) completes bounded RE-345–347. Broad protocol/runtime milestones remain open; RE-348 tracks full send, initialized trees/codebooks, actual packet round trips and platform coverage.
|
| 531 |
|
| 532 |
Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) completes RE-349–351. Broad network/runtime milestones remain open; RE-352 tracks adaptive initialization/codebooks, real message packets and platform fixtures.
|
|
|
|
|
|
|
|
|
| 530 |
Research progress: [Huffman helper probes](research/HUFFMAN_HELPERS.md) completes bounded RE-345–347. Broad protocol/runtime milestones remain open; RE-348 tracks full send, initialized trees/codebooks, actual packet round trips and platform coverage.
|
| 531 |
|
| 532 |
Research progress: [bounded Huffman encoding](research/HUFFMAN_ENCODING.md) completes RE-349–351. Broad network/runtime milestones remain open; RE-352 tracks adaptive initialization/codebooks, real message packets and platform fixtures.
|
| 533 |
+
|
| 534 |
+
Research progress: [Huffman initialization](research/HUFFMAN_INITIALIZATION.md) completes bounded RE-353–355. Broad network/runtime milestones remain open; RE-356 tracks adaptive updates, initialized codewords and actual message/packet/platform fixtures.
|
docs/RESEARCH_LEDGER.md
CHANGED
|
@@ -213,3 +213,7 @@ Completed bounded RE-345–347 with `analyze_huffman_helpers.py`: ten verified g
|
|
| 213 |
## 2026-10-11 — Bounded Huffman send and symbol streams
|
| 214 |
|
| 215 |
Completed bounded RE-349–351 with `analyze_huffman_encoding.py`: six verified graphs across the Linux ELF64 engines, 720 supplied-path send compositions, 192 symbol round trips and thirty shared-tree streams. ENC-001–003/F-121–123 distinguish root-order code bits from LSB byte packing, dirty partial-byte retention from byte-start clearing, nine-edge unrolling from explicit recursive virtual frames, supplied codebooks from actual initialized game trees, and symbol round trips from full message/packet conformance. Frame prologues/epilogues and wrapper PUSH/POP are excluded explicitly; full caller-register restoration and disjoint virtual stacks preserve child memory effects across stopped CALL boundaries. No original process or native callee executes. Eight independent tests verify packing/canaries/cursors, recursion thresholds, parent-child tree consistency, transmit/receive symbols, repeated streams, zero-depth/empty cases, recomposed call-free traces and paired results. All 310 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-348 remains open; RE-352 tracks adaptive initialization/node allocation/rank/swaps/frequencies/codebooks, actual message/packet composition, legal inputs/capacity/truncation and ABI/runtime/all-platform/snapshot/renderer coverage.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 213 |
## 2026-10-11 — Bounded Huffman send and symbol streams
|
| 214 |
|
| 215 |
Completed bounded RE-349–351 with `analyze_huffman_encoding.py`: six verified graphs across the Linux ELF64 engines, 720 supplied-path send compositions, 192 symbol round trips and thirty shared-tree streams. ENC-001–003/F-121–123 distinguish root-order code bits from LSB byte packing, dirty partial-byte retention from byte-start clearing, nine-edge unrolling from explicit recursive virtual frames, supplied codebooks from actual initialized game trees, and symbol round trips from full message/packet conformance. Frame prologues/epilogues and wrapper PUSH/POP are excluded explicitly; full caller-register restoration and disjoint virtual stacks preserve child memory effects across stopped CALL boundaries. No original process or native callee executes. Eight independent tests verify packing/canaries/cursors, recursion thresholds, parent-child tree consistency, transmit/receive symbols, repeated streams, zero-depth/empty cases, recomposed call-free traces and paired results. All 310 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-348 remains open; RE-352 tracks adaptive initialization/node allocation/rank/swaps/frequencies/codebooks, actual message/packet composition, legal inputs/capacity/truncation and ABI/runtime/all-platform/snapshot/renderer coverage.
|
| 216 |
+
|
| 217 |
+
## 2026-10-11 — Native Huffman frequencies and initialization
|
| 218 |
+
|
| 219 |
+
Completed bounded RE-353–355 with `analyze_huffman_initialization.py`: eight verified graphs across the Linux ELF64 engines, two identical file-backed 256-count frequency tables, four empty-tree initialization probes and eight seed-loop compositions. INIT-001–003/F-124–126 distinguish actual table extraction from generated conceptual schedule digests, supplied memset effects from native clear execution, exact sparse initial/NYT layouts from trained trees, pointer-slot asymmetry from inferred list semantics, and bounded paired addRef requests from adaptive updates. The tables total 1,053,340 repetitions per tree; their implied 2,106,680-request schedule is not evaluated as a full native training trace. The bounded schedules retain 1,068 stopped requests with explicit caller-register retention. PUSH omission and POP/RET/callee boundaries are documented; no original process or native callee executes. Eight independent tests verify frequencies/hash/summary, separately generated conceptual request digest, entry flag/init arguments, ascending paired schedules, clear spans/canaries, full sparse layouts, trace boundaries and paired results. All 318 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-352 remains open; RE-356 tracks addRef/increment/swaps/block/freelist effects, actual trained trees/codewords, full packet/message codecs and legal/runtime/all-platform invariants.
|
docs/REVERSE_ENGINEERING_TODO.md
CHANGED
|
@@ -422,3 +422,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
|
|
| 422 |
- [x] RE-350 Compose transmit/send/receive against supplied comb trees with exact symbol/cursor/byte round trips ([ENC-002](research/HUFFMAN_ENCODING.md)).
|
| 423 |
- [x] RE-351 Compose shared-tree repeated/multi-symbol streams with independent byte and per-symbol cursor oracles ([ENC-003](research/HUFFMAN_ENCODING.md)).
|
| 424 |
- [ ] RE-352 Recover adaptive Huffman initialization/node allocation/rank/swap/frequency updates and actual codebooks, full message writers/readers/real-packet compositions, overflow/truncation and legal tree/symbol/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 422 |
- [x] RE-350 Compose transmit/send/receive against supplied comb trees with exact symbol/cursor/byte round trips ([ENC-002](research/HUFFMAN_ENCODING.md)).
|
| 423 |
- [x] RE-351 Compose shared-tree repeated/multi-symbol streams with independent byte and per-symbol cursor oracles ([ENC-003](research/HUFFMAN_ENCODING.md)).
|
| 424 |
- [ ] RE-352 Recover adaptive Huffman initialization/node allocation/rank/swap/frequency updates and actual codebooks, full message writers/readers/real-packet compositions, overflow/truncation and legal tree/symbol/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration.
|
| 425 |
+
|
| 426 |
+
- [x] RE-353 Extract and verify paired native frequency tables, totals and conceptual training-schedule provenance ([INIT-001](research/HUFFMAN_INITIALIZATION.md)).
|
| 427 |
+
- [x] RE-354 Recover exact two-object clear requests, initial node/lookup pointers and sparse NYT layout/asymmetry ([INIT-002](research/HUFFMAN_INITIALIZATION.md)).
|
| 428 |
+
- [x] RE-355 Recover initialization flag/request and bounded ascending paired addRef scheduling/skip/termination ([INIT-003](research/HUFFMAN_INITIALIZATION.md)).
|
| 429 |
+
- [ ] RE-356 Recover addRef insertion/existing-symbol effects and increment/block/head/freelist/node/list swap recursion; reconstruct actual initialized trees/codewords from exact native seeds, compose real message/packet codecs and validate capacity/overflow/truncation, legal invariants and original-runtime/all-platform contracts.
|
docs/research/HUFFMAN_ENCODING.md
CHANGED
|
@@ -35,3 +35,5 @@ Buffers begin at zero, while the separate dirty-buffer probes characterize retai
|
|
| 35 |
Eight independent tests check dirty-buffer/root-path packing, nine-edge recursive thresholds, encoded bytes/symbols/cursors/canaries, supplied parent/child consistency, shared stream packing/repetition, zero-depth/empty streams, recomposed traces excluding native CALLs and paired outputs. Reports retain trace lengths/hashes and verified full graphs. No evaluator extension was needed.
|
| 36 |
|
| 37 |
RE-352: recover Huffman initialization, node allocation/swapping/rank/frequency updates and actual game codebooks; compose full message writers/readers with real code lengths and packets; recover capacity/overflow/truncation and legal symbol/tree/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration before Rust implementation.
|
|
|
|
|
|
|
|
|
| 35 |
Eight independent tests check dirty-buffer/root-path packing, nine-edge recursive thresholds, encoded bytes/symbols/cursors/canaries, supplied parent/child consistency, shared stream packing/repetition, zero-depth/empty streams, recomposed traces excluding native CALLs and paired outputs. Reports retain trace lengths/hashes and verified full graphs. No evaluator extension was needed.
|
| 36 |
|
| 37 |
RE-352: recover Huffman initialization, node allocation/swapping/rank/frequency updates and actual game codebooks; compose full message writers/readers with real code lengths and packets; recover capacity/overflow/truncation and legal symbol/tree/cursor invariants; validate ABI, original-runtime/all-platform and snapshot/renderer integration before Rust implementation.
|
| 38 |
+
|
| 39 |
+
Initialization follow-up: [native frequencies and empty-tree probes](HUFFMAN_INITIALIZATION.md) recovers actual seed counts and bounded initialization/training requests (RE-353–355). Adaptive tree effects and final native codewords remain open.
|
docs/research/HUFFMAN_INITIALIZATION.md
ADDED
|
@@ -0,0 +1,39 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Huffman frequencies, empty-tree initialization and training requests
|
| 2 |
+
|
| 3 |
+
Reproduce with `python3 scripts/analyze_huffman_initialization.py`; validate with `python3 scripts/test_huffman_initialization.py`. [huffman-initialization.json](huffman-initialization.json) retains four hash-verified graphs per Linux ELF64 engine: MSG_initHuffman, Huff_Init, Huff_addRef and increment. This follows [supplied-tree encoder compositions](HUFFMAN_ENCODING.md).
|
| 4 |
+
|
| 5 |
+
The actual frequency table is extracted from verified file-backed native images. Initialization and scheduling are bounded synthetic instruction evaluations: PUSH instructions are explicitly omitted, POP/RET and all callees are stopped, and memset effects are supplied. Small scheduling fixtures supply count tables and retain caller registers across stopped addRef requests. Huff_addRef and increment graphs are static next-step evidence; neither is evaluated here. Original processes and native callees never execute, and no trained tree/codebook or packet conformance is established.
|
| 6 |
+
|
| 7 |
+
## INIT-001 — Exact native seed frequencies (RE-353)
|
| 8 |
+
|
| 9 |
+
The MSG_initHuffman comparison operands identify a table of 256 little-endian 32-bit counts. Both verified Linux images contain identical table bytes, with SHA-256:
|
| 10 |
+
|
| 11 |
+
`1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf`
|
| 12 |
+
|
| 13 |
+
The report retains every count, virtual address, file offset and source/disassembly provenance. All counts are positive and fit signed 32-bit comparison: minimum 740, maximum 250315, total 1,053,340. Symbol zero has count 250315; the first sixteen counts are `[250315,41193,6292,7106,3730,3750,6110,23283,33317,6950,7838,9714,9257,17259,3949,1778]`.
|
| 14 |
+
|
| 15 |
+
The loop implies 1,053,340 repetitions per tree, or 2,106,680 paired addRef requests. A conceptual schedule digest is generated from the extracted counts: for each symbol ascending, repeat `bytes(symbol, tree0, symbol, tree1)` count times. This generated digest records the implied request stream independently of native addresses. It is **not** a trace of all native training iterations and does not establish the resulting tree. Reconstructing a generic frequency-optimal tree without recovering native tie/rank/swap behavior could produce different codewords.
|
| 16 |
+
|
| 17 |
+
## INIT-002 — Two cleared objects and initial NYT nodes (RE-354)
|
| 18 |
+
|
| 19 |
+
Four paired initialization probes use two virtual base addresses and dirty sentinel buffers. Huff_Init requests two adjacent memset operations, each of size 0xc830 (51,248 bytes), value zero: base and base +0xc830. The collector supplies those effects; native memset never executes. Both outside canaries retain their sentinel.
|
| 20 |
+
|
| 21 |
+
After the clears, each object's node count at +0 becomes one. Its initial node is at object +0x830 (2,096); native allocation address arithmetic computes 64*count minus 8*count, a 56-byte node stride. The root pointer +8, pointer slot +16 and symbol-256 lookup slot +0x820 all refer to this initial node. The second object's pointer slot +24 also refers to it; the first object's +24 remains zero. This asymmetry is retained rather than normalized. Full list-slot semantics require addRef/increment analysis.
|
| 22 |
+
|
| 23 |
+
The initial 56-byte node has symbol word 256 at +52 and weight word zero at +48. Child/parent pointers +0/+8/+16 and list-related slots +24/+32 are zero. The complete objects contain only the expected nonzero count/pointer/symbol bytes; the report records the sparse nonzero-byte image, not just selected fields. Slot +0x820 agrees with the transmit lookup formula `object +32 +8*symbol` for symbol 256. This supports identifying the initial not-yet-transmitted (NYT) node, while actual NYT insertion/escape behavior remains open.
|
| 24 |
+
|
| 25 |
+
These native layout offsets are evidence for reconstruction, not a required Rust memory layout. Capacities, node/pointer-pool ownership, alias invariants and allocator/error paths remain unresolved.
|
| 26 |
+
|
| 27 |
+
## INIT-003 — Flag, ordering and bounded seed loop (RE-355)
|
| 28 |
+
|
| 29 |
+
The entry slice sets msgInit to one before requesting Huff_Init at the first object pointer. It resets local symbol/repetition registers to zero. The caller guard, thread/reentrancy behavior and initialization failure semantics are outside this slice.
|
| 30 |
+
|
| 31 |
+
Eight paired loop compositions supply four count tables: all zero; symbols zero twice and 255 three times; sparse symbols 1/127/128/254; and every symbol once. Symbol indices advance from zero through 255 in ascending order, and the loop terminates with local symbol 256. Each repetition requests Huff_addRef(first_object,symbol), then Huff_addRef(first_object +0xc830,symbol), before testing the next repetition. Zero-count symbols produce no requests. Supplied counts and all stopped request arguments are retained, with trace hashes/counts.
|
| 32 |
+
|
| 33 |
+
The native helper requests total 1,068 across these bounded fixtures: 534 per engine. Actual seed frequencies are extracted separately and are not substituted into an evaluated million-iteration loop. The small fixtures prove local loop ordering/skip/termination and paired object selection; they do not recover addRef effects, final weights or trained codewords.
|
| 34 |
+
|
| 35 |
+
## Validation and next work
|
| 36 |
+
|
| 37 |
+
Eight independent tests verify exact frequency hash/summary, a separately generated per-request conceptual digest, entry flag/init arguments, complete bounded ascending/paired request order, exact clear spans/canaries, complete sparse object/NYT layout and asymmetry, stopped CALL traces and paired frequency/virtual-layout results. No evaluator extension was required.
|
| 38 |
+
|
| 39 |
+
RE-356: recover Huff_addRef insertion/existing-symbol branches, increment rank/block/head/freelist changes, node/list swaps and recursion; reconstruct initialized trees with the exact seed schedule and derive verified codewords; compose actual message encoders/decoders and packet fixtures, with capacity/truncation/overflow, legal invariants and original-runtime/all-platform validation.
|
docs/research/README.md
CHANGED
|
@@ -131,3 +131,5 @@ The extension collector adds five focused tests for native tail calls, command-r
|
|
| 131 |
[Raw-bit helpers and supplied Huffman trees](HUFFMAN_HELPERS.md) adds RE-345–347: byte clear/OR, raw round trips, supplied-tree decoding/null cursor paths and transmit lookup/request boundaries. Reproduce with `python3 scripts/analyze_huffman_helpers.py`; check with `python3 scripts/test_huffman_helpers.py`. Full send, real codebooks/messages and runtime/platform contracts remain open.
|
| 132 |
|
| 133 |
[Bounded Huffman encoding and symbol streams](HUFFMAN_ENCODING.md) adds RE-349–351: unrolled/recursive send, supplied-tree symbol round trips and shared multi-symbol streams. Reproduce with `python3 scripts/analyze_huffman_encoding.py`; check with `python3 scripts/test_huffman_encoding.py`. Actual adaptive initialization/codebooks, message packets and runtime/platform contracts remain open.
|
|
|
|
|
|
|
|
|
| 131 |
[Raw-bit helpers and supplied Huffman trees](HUFFMAN_HELPERS.md) adds RE-345–347: byte clear/OR, raw round trips, supplied-tree decoding/null cursor paths and transmit lookup/request boundaries. Reproduce with `python3 scripts/analyze_huffman_helpers.py`; check with `python3 scripts/test_huffman_helpers.py`. Full send, real codebooks/messages and runtime/platform contracts remain open.
|
| 132 |
|
| 133 |
[Bounded Huffman encoding and symbol streams](HUFFMAN_ENCODING.md) adds RE-349–351: unrolled/recursive send, supplied-tree symbol round trips and shared multi-symbol streams. Reproduce with `python3 scripts/analyze_huffman_encoding.py`; check with `python3 scripts/test_huffman_encoding.py`. Actual adaptive initialization/codebooks, message packets and runtime/platform contracts remain open.
|
| 134 |
+
|
| 135 |
+
[Huffman frequencies and initialization](HUFFMAN_INITIALIZATION.md) adds RE-353–355: verified native frequencies, empty-tree layout and bounded paired seed requests. Reproduce with `python3 scripts/analyze_huffman_initialization.py`; check with `python3 scripts/test_huffman_initialization.py`. Adaptive updates, trained codewords and actual message/runtime/platform contracts remain open.
|
docs/research/huffman-initialization.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
scripts/analyze_huffman_initialization.py
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Verified native frequencies, supplied memset initialization and seed-loop requests."""
|
| 3 |
+
import copy
|
| 4 |
+
import gzip
|
| 5 |
+
import hashlib
|
| 6 |
+
import json
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
import re
|
| 9 |
+
import struct
|
| 10 |
+
from analyze_area_queries import site
|
| 11 |
+
from analyze_bsp_references import global_address,trace_fields
|
| 12 |
+
from analyze_collision_composition import evaluate,get_word,put_word
|
| 13 |
+
from analyze_engine_control import control_flow
|
| 14 |
+
from analyze_frame_timing import read_address
|
| 15 |
+
from analyze_native import elf
|
| 16 |
+
from analyze_qvm import write_json
|
| 17 |
+
from analyze_vm_extensions import native_functions,verified
|
| 18 |
+
|
| 19 |
+
|
| 20 |
+
SIZE=0xc830
|
| 21 |
+
|
| 22 |
+
|
| 23 |
+
def run(g,start,stops,m,regs):return evaluate(g,start,stops,m,regs,limit=10000,capture_registers=True)
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def omit_push(g):
|
| 27 |
+
out=copy.deepcopy(g);addresses=[]
|
| 28 |
+
for x in out['instructions']:
|
| 29 |
+
if x['opcode']=='pushq':addresses.append(x['address']);x['opcode']='nop';x['operands']=''
|
| 30 |
+
return out,addresses
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def main():
|
| 34 |
+
out={'native':{},'limitations':'Paired Linux ELF64 bounded synthetic initialization/schedule slices. Original processes/native callees never execute. PUSH omitted, POP/RET stopped; memset effects supplied. Real frequency table extracted from verified file-backed image, but conceptual full schedule digest is generated from counts, not evaluated through all native training iterations. Huff_addRef requests stop before calls; no trained/adaptive tree or actual codebook/runtime/platform conformance is established.'}
|
| 35 |
+
for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
|
| 36 |
+
r=json.loads(Path(f'docs/research/native/{label}.json').read_text());data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);meta,_=elf(data);a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());graphs={n:control_flow(fs[n]) for n in ['MSG_initHuffman','Huff_Init','Huff_addRef','increment']}
|
| 37 |
+
g,omitted=omit_push(graphs['MSG_initHuffman']);rows=g['instructions'];table_ins=next(x for x in rows if x['opcode']=='cmpl' and '(,%r13,4)' in x['operands']);address=int(re.search(r'0x([0-9a-f]+)\(',table_ins['operands'])[1],16);raw,offset=read_address(data,meta['sections'],address,1024);frequencies=list(struct.unpack('<256I',raw))
|
| 38 |
+
digest=hashlib.sha256()
|
| 39 |
+
# Relative tree ordinals keep this conceptual request digest independent of addresses.
|
| 40 |
+
for symbol,count in enumerate(frequencies):digest.update(bytes([symbol,0,symbol,1])*count)
|
| 41 |
+
init_idx=next(i for i,x in enumerate(rows) if '<Huff_Init>' in x['operands']);flag=global_address(next(x for x in rows if x['opcode']=='movl' and '(%rip)' in x['operands']))
|
| 42 |
+
entry=run(g,rows[0]['address'],[rows[init_idx]['address']],{}, {'%rsp':0x90000})
|
| 43 |
+
add_calls={x['address']:i for i,x in enumerate(rows) if '<Huff_addRef>' in x['operands']};end=site(g,'addq','$0x8, %rsp');schedule=[]
|
| 44 |
+
for name in ['zero','edges','sparse','ones']:
|
| 45 |
+
counts=[0]*256
|
| 46 |
+
if name=='edges':counts[0]=2;counts[255]=3
|
| 47 |
+
elif name=='sparse':counts[1]=1;counts[127]=2;counts[128]=1;counts[254]=2
|
| 48 |
+
elif name=='ones':counts=[1]*256
|
| 49 |
+
m=dict(entry['memory'])
|
| 50 |
+
for i,count in enumerate(counts):put_word(m,address+4*i,count)
|
| 51 |
+
regs=dict(entry['registers']);start=rows[init_idx+1]['address'];requests=[];trace=[]
|
| 52 |
+
while True:
|
| 53 |
+
t=run(g,start,list(add_calls)+[end],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
|
| 54 |
+
if t['stop']==end:break
|
| 55 |
+
requests.append([regs['rdi'],regs['rsi']]);start=rows[add_calls[t['stop']]+1]['address']
|
| 56 |
+
if len(requests)>512:raise ValueError('fixture request bound')
|
| 57 |
+
schedule.append({'fixture':name,'supplied_counts':counts,'requests':requests,'final_symbol':regs['rbp'],'flag_word':get_word(m,flag),**trace_fields({'trace':trace},False)})
|
| 58 |
+
init,init_omitted=omit_push(graphs['Huff_Init']);ir=init['instructions'];calls={x['address']:i for i,x in enumerate(ir) if x['opcode']=='callq'};end_init=next(x['address'] for x in ir if x['opcode']=='popq');probes=[]
|
| 59 |
+
for base,sentinel in [(0x10000,0xa5),(0x40000,255)]:
|
| 60 |
+
m={base+i:sentinel for i in range(-1,2*SIZE+1)};regs={'%rdi':base};start=ir[0]['address'];requests=[];trace=[]
|
| 61 |
+
while True:
|
| 62 |
+
t=run(init,start,list(calls)+[end_init],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
|
| 63 |
+
if t['stop']==end_init:break
|
| 64 |
+
requests.append([regs[k] for k in ['rdi','rsi','rdx']])
|
| 65 |
+
if 'memset@plt' not in ir[calls[t['stop']]]['operands']:raise ValueError('unexpected initialization callee')
|
| 66 |
+
for i in range(regs['rdx']):m[regs['rdi']+i]=regs['rsi']&255
|
| 67 |
+
start=ir[calls[t['stop']]+1]['address']
|
| 68 |
+
objects=[]
|
| 69 |
+
for obj in [base,base+SIZE]:
|
| 70 |
+
node=get_word(m,obj+8,8);objects.append({'offset':obj-base,'node_count':get_word(m,obj),'root_pointer':node,'list_pointers':[get_word(m,obj+x,8) for x in [16,24]],'nyt_lookup_pointer':get_word(m,obj+0x820,8),'node_pool_offset':node-obj,'node_bytes':[m[node+i] for i in range(56)],'nonzero_bytes':[[i,m[obj+i]] for i in range(SIZE) if m[obj+i]]})
|
| 71 |
+
probes.append({'base':base,'sentinel':sentinel,'memset_requests':requests,'objects':objects,'canaries':[m[base-1],m[base+2*SIZE]],**trace_fields({'trace':trace},True)})
|
| 72 |
+
out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,'frequency_table':{'address':address,'file_offset':offset,'sha256':hashlib.sha256(raw).hexdigest(),'counts':frequencies,'total':sum(frequencies),'paired_request_count':2*sum(frequencies),'conceptual_schedule_sha256':digest.hexdigest(),'digest_encoding':'for each symbol ascending, repeated count times: bytes(symbol,tree0,symbol,tree1)'},'entry_request':{'huff_init_pointer':entry['registers']['rdi'],'flag_word':get_word(entry['memory'],flag),'omitted_push_addresses':omitted,**trace_fields(entry,True)},'schedule_probes':schedule,'initialization_probes':probes,'init_omitted_push_addresses':init_omitted}
|
| 73 |
+
write_json(Path('docs/research/huffman-initialization.json'),out);print('Recovered paired 256-entry frequencies, eight schedule and four empty-tree initialization probes')
|
| 74 |
+
|
| 75 |
+
|
| 76 |
+
if __name__=='__main__':main()
|
scripts/test_huffman_initialization.py
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Independent frequency, paired scheduling and empty-object layout oracles."""
|
| 3 |
+
import hashlib
|
| 4 |
+
import json
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
import struct
|
| 7 |
+
import unittest
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
class HuffmanInitializationTests(unittest.TestCase):
|
| 11 |
+
@classmethod
|
| 12 |
+
def setUpClass(cls):cls.r=json.loads(Path('docs/research/huffman-initialization.json').read_text())
|
| 13 |
+
|
| 14 |
+
def test_verified_frequency_words_hash_and_summary(self):
|
| 15 |
+
for n in self.r['native'].values():
|
| 16 |
+
f=n['frequency_table'];counts=f['counts'];self.assertEqual(len(counts),256)
|
| 17 |
+
self.assertEqual(hashlib.sha256(struct.pack('<256I',*counts)).hexdigest(),'1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf')
|
| 18 |
+
self.assertEqual(f['sha256'],'1cbdca8d6c9bb22247525a4807c5e18cec9b87336c56016424eecd879b505abf');self.assertEqual(sum(counts),1053340);self.assertEqual(f['total'],1053340);self.assertEqual(f['paired_request_count'],2106680)
|
| 19 |
+
self.assertEqual(min(counts),740);self.assertEqual(max(counts),250315);self.assertTrue(all(0<v<2**31 for v in counts))
|
| 20 |
+
|
| 21 |
+
def test_conceptual_schedule_digest_from_individual_requests(self):
|
| 22 |
+
n=next(iter(self.r['native'].values()));f=n['frequency_table'];digest=hashlib.sha256()
|
| 23 |
+
for symbol,count in enumerate(f['counts']):
|
| 24 |
+
for _ in range(count):
|
| 25 |
+
digest.update(bytes([symbol,0]));digest.update(bytes([symbol,1]))
|
| 26 |
+
for n in self.r['native'].values():self.assertEqual(n['frequency_table']['conceptual_schedule_sha256'],digest.hexdigest())
|
| 27 |
+
|
| 28 |
+
def test_seed_flag_and_init_request(self):
|
| 29 |
+
for n in self.r['native'].values():
|
| 30 |
+
e=n['entry_request'];self.assertEqual(e['flag_word'],1);self.assertEqual(len(e['omitted_push_addresses']),4)
|
| 31 |
+
first=n['schedule_probes'][1]['requests'][0][0];self.assertEqual(e['huff_init_pointer'],first)
|
| 32 |
+
|
| 33 |
+
def test_native_loop_ascending_symbols_and_paired_request_order(self):
|
| 34 |
+
for n in self.r['native'].values():
|
| 35 |
+
self.assertEqual(len(n['schedule_probes']),4);base=n['entry_request']['huff_init_pointer']
|
| 36 |
+
for p in n['schedule_probes']:
|
| 37 |
+
expected=[[base+tree*51248,symbol] for symbol,count in enumerate(p['supplied_counts']) for _ in range(count) for tree in [0,1]]
|
| 38 |
+
self.assertEqual(p['requests'],expected);self.assertEqual(p['final_symbol'],256);self.assertEqual(p['flag_word'],1)
|
| 39 |
+
|
| 40 |
+
def test_two_exact_clear_requests_and_outside_canaries(self):
|
| 41 |
+
for n in self.r['native'].values():
|
| 42 |
+
self.assertEqual(len(n['init_omitted_push_addresses']),1)
|
| 43 |
+
for p in n['initialization_probes']:
|
| 44 |
+
b=p['base'];self.assertEqual(p['memset_requests'],[[b,0,51248],[b+51248,0,51248]]);self.assertEqual(p['canaries'],[p['sentinel']]*2)
|
| 45 |
+
|
| 46 |
+
def test_complete_sparse_initialization_layout_and_nyt_node(self):
|
| 47 |
+
for n in self.r['native'].values():
|
| 48 |
+
for p in n['initialization_probes']:
|
| 49 |
+
for i,o in enumerate(p['objects']):
|
| 50 |
+
base=p['base']+i*51248;node=base+2096;expected=bytearray(51248)
|
| 51 |
+
expected[0:4]=(1).to_bytes(4,'little')
|
| 52 |
+
for slot in [8,16,2080]+([24] if i else []):expected[slot:slot+8]=node.to_bytes(8,'little')
|
| 53 |
+
expected[2096+52:2096+56]=(256).to_bytes(4,'little')
|
| 54 |
+
self.assertEqual(o['nonzero_bytes'],[[j,v] for j,v in enumerate(expected) if v]);self.assertEqual(o['root_pointer'],node);self.assertEqual(o['node_count'],1);self.assertEqual(o['node_pool_offset'],2096);self.assertEqual(o['nyt_lookup_pointer'],node)
|
| 55 |
+
self.assertEqual(o['node_bytes'],list(expected[2096:2096+56]));self.assertEqual(o['list_pointers'],[node,node if i else 0])
|
| 56 |
+
|
| 57 |
+
def test_explicit_traces_stop_before_calls(self):
|
| 58 |
+
for n in self.r['native'].values():
|
| 59 |
+
calls={x['address'] for g in n['graphs'].values() for x in g['instructions'] if x['opcode']=='callq'}
|
| 60 |
+
for p in [n['entry_request']]+n['initialization_probes']:
|
| 61 |
+
self.assertTrue(calls.isdisjoint(p['trace']));self.assertEqual(len(p['trace']),p['trace_length'])
|
| 62 |
+
|
| 63 |
+
def test_paired_frequencies_and_virtual_initialization_match(self):
|
| 64 |
+
a,b=self.r['native'].values()
|
| 65 |
+
for key in ['counts','sha256','total','paired_request_count','conceptual_schedule_sha256']:self.assertEqual(a['frequency_table'][key],b['frequency_table'][key])
|
| 66 |
+
def clean(rows):return [{k:v for k,v in p.items() if k not in ['trace','trace_sha256']} for p in rows]
|
| 67 |
+
self.assertEqual(clean(a['initialization_probes']),clean(b['initialization_probes']))
|
| 68 |
+
for n in [a,b]:self.assertEqual(len(n['graphs']),4)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
if __name__=='__main__':unittest.main()
|