Recover keyed delta masks and source-dependent decode boundaries

docs/ASSET_PIPELINE_TODO.md

@@ -236,3 +236,5 @@ Research progress: [message strings and data reads](research/MESSAGE_STRINGS.md)
 Research progress: [numeric message codecs](research/MESSAGE_NUMERIC.md) adds raw float and finite angle contracts (RE-381–383). RE-384 retains compressed/delta/caller/numeric-control/packet/runtime/platform validation; broad codec conversion milestones remain open.
 
 Research progress: [scalar message deltas](research/MESSAGE_DELTA.md) adds integer/finite-float and modeled-tree compressed contracts (RE-385–387). RE-388 retains keyed/higher-level/caller/actual-packet/runtime/platform work; broad network conversion remains open.
+
+Research progress: [keyed message deltas](research/MESSAGE_DELTA_KEY.md) adds integer mask/XOR and finite float compressed contracts (RE-389–391). RE-392 retains caller/index/runtime/higher-level/packet/platform work; broad networking conversion remains open.

docs/FINDINGS.md

@@ -438,3 +438,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
 **F-149 — Observed finite float delta:** finite numeric equality controls payload omission, including opposite signed zeros; unchanged reads retain baseline bits. Changed payloads forward raw 32-bit words. Nonfinite/control-state/caller/runtime behavior remains open. [DELTA-002](research/MESSAGE_DELTA.md).
 
 **F-150 — Observed compressed scalar delta composition:** native wrapper/bit/Huffman instruction slices agree with independent flag/remainder/modeled-codeword packing and cursor/accounting checks. Seed trees remain model-derived; actual packet/overflow/EOF/full-native-training/platform contracts remain open. [DELTA-003](research/MESSAGE_DELTA.md).
+
+**F-151 — Observed keyed integer mask indexing:** writer compares before full-word XOR; reader directly indexes a 32-word kbitmask by width. Sampled widths below 32 retain one extra key bit; width 32 reads the adjacent entityStateFields low word, differing between the Linux images. Caller width/key/narrowing/legal-index/runtime effects remain open. [KEY-001](research/MESSAGE_DELTA_KEY.md).
+
+**F-152 — Observed keyed finite float delta:** numeric equality omits payload and preserves baseline bits; changed words XOR the full key before/after 32-bit transport, without the integer mask table. Nonfinite/control-state/caller/platform behavior remains open. [KEY-002](research/MESSAGE_DELTA_KEY.md).
+
+**F-153 — Observed keyed compressed composition:** independent modeled-codeword packing agrees with native wrapper/bit/Huffman slices and paired bytes/cursors. Integer width-32 decode returns can differ for identical bytes due to actual adjacent file words; legal game-level reachability remains unverified. Native training/packet/overflow/EOF/runtime/platform contracts remain open. [KEY-003](research/MESSAGE_DELTA_KEY.md).

docs/PROVISIONAL_TODO.md

@@ -548,3 +548,5 @@ Research progress: [message strings and data reads](research/MESSAGE_STRINGS.md)
 Research progress: [numeric message codecs](research/MESSAGE_NUMERIC.md) completes bounded RE-381–383. RE-384 leaves delta/snapshot/angle callers and complete compressed/numeric-control/runtime/platform contracts open.
 
 Research progress: [scalar message deltas](research/MESSAGE_DELTA.md) completes bounded RE-385–387. RE-388 leaves keyed/usercmd/entity/player/snapshot and complete caller/overflow/EOF/floating/runtime/platform contracts open.
+
+Research progress: [keyed message deltas](research/MESSAGE_DELTA_KEY.md) completes bounded RE-389–391. RE-392 leaves keyed caller/narrowing/index reachability, higher-level codecs and actual runtime/packet/platform contracts open.

docs/RESEARCH_LEDGER.md

@@ -249,3 +249,7 @@ Completed bounded RE-381–383 with `analyze_message_numeric.py`: fourteen verif
 ## 2026-10-11 — Scalar deltas and modeled-tree compressed composition
 
 Completed bounded RE-385–387 with `analyze_message_delta.py`: 22 verified graphs across the Linux ELF64 engines, 296 compressed scalar delta streams and 48 supplied-reader control probes. DELTA-001–003/F-148–150 distinguish full-word integer equality/baseline retention from changed-width masking, finite float equality (including opposite signed zeros) from raw word equality, and local nonzero flag dispatch from legal one-bit decoder output. Changed float payloads retain raw words; unchanged reads retain baseline bits. Native delta/bit/raw/Huffman instruction slices compose exact bytes against MODEL-DERIVED seeded trees, with independent flag/remainder/codeword packing, decoded value, cursor, oldsize/range diagnostic and canary oracles. Source/disassembly identities and codebook/initialization report hashes are retained and verified. Frame saves/restores/PUSH/POP are omitted; stopped helper/tail boundaries, disjoint virtual frames and caller-register restoration are explicit. No original process or native callee executes, and finite comparison rejection does not specify native nonfinite behavior. Eight tests verify integer/float requests and equality, independent packing, changed signed tails, cursor/accounting/diagnostics/canaries, supplied nonzero flags, dependency hashes/paired outcomes and finite comparison model boundaries. All 382 tests pass; new and preceding reports regenerate byte-identically, with storage/link/whitespace checks passing. RE-384 remains open; RE-388 tracks keyed masks/XOR/table bounds, usercmd/entity/player/snapshot/caller contracts, remaining widths/aliases, compressed capacity/overflow/reset/truncation/EOF/padding/nonfinite control behavior, native seeded codewords and actual packet/runtime/all-platform/renderer conformance.
+
+## 2026-10-11 — Keyed deltas, direct mask indexing and adjacent data
+
+Completed bounded RE-389–391 with `analyze_message_delta_key.py`: 22 verified graphs across the Linux ELF64 engines, two symbol/file-backed 32-word mask tables plus adjacent entityStateFields words, 288 compressed keyed streams and 48 supplied-reader control probes. KEY-001–003/F-151–153 distinguish comparison before XOR from keyed payload encoding, direct width indexing from an inferred width-minus-one mask, extra returned key bits from caller-visible effects, and width-32 adjacent file reads from a valid mask entry. Identical encoded width-32 integer streams can return different words between images because the adjacent low words differ; legal caller/index/narrowing/runtime reachability remains unverified. Finite keyed float payloads XOR the complete word and recover changed values; numeric equality/signed zeros retain baseline bits without using the integer mask table. Independent flag/remainder/modeled-codeword packing, exact decode formula, cursor/accounting/XOR-range diagnostic and canary oracles validate the sampled instruction compositions. Source/disassembly and dependency identities/hashes are retained; compressed trees remain MODEL-DERIVED. Frame saves/restores/PUSH/POP are omitted, stopped helper/tail boundaries and caller-register restoration are explicit, and no original process or native callee executes. Eight tests verify table/symbol/adjacent file bytes, keyed decisions/requests, float behavior, independent packing/cursors, integer decode masks, supplied flags, accounting/diagnostics/dependencies and paired payloads with source-specific width-32 results. All 390 tests pass; new and preceding reports regenerate byte-identically, with storage/link/whitespace checks passing. RE-388 remains open; RE-392 tracks caller widths/keys/narrowing/index reachability/runtime adjacent effects, usercmd/entity/player/snapshot codecs, remaining scalar/overflow/EOF/nonfinite controls, native seeded codewords and actual packet/runtime/all-platform/renderer contracts before Rust mask compatibility decisions.

docs/REVERSE_ENGINEERING_TODO.md

@@ -467,3 +467,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
 - [x] RE-386 Recover finite float delta comparisons, signed-zero baseline retention and raw changed payloads ([DELTA-002](research/MESSAGE_DELTA.md)).
 - [x] RE-387 Compose bounded scalar delta compressed writes/reads against modeled trees with independent payload/cursor/accounting oracles ([DELTA-003](research/MESSAGE_DELTA.md)).
 - [ ] RE-388 Recover keyed integer/float delta masks/XOR/table bounds, usercmd/entity/player/snapshot codecs and callers; validate remaining widths/value aliases, compressed capacity/overflow/reset/truncation/EOF/padding/nonfinite-control contracts, native seeded codewords and actual packet/runtime/all-platform/renderer conformance.
+
+- [x] RE-389 Recover keyed integer XOR/direct-index masks, symbol-backed array bounds and sampled adjacent width-32 reads ([KEY-001](research/MESSAGE_DELTA_KEY.md)).
+- [x] RE-390 Recover keyed finite float comparisons, baseline retention and full-word XOR payloads ([KEY-002](research/MESSAGE_DELTA_KEY.md)).
+- [x] RE-391 Compose modeled-tree keyed streams with independent packing/cursors and source-dependent integer width-32 return checks ([KEY-003](research/MESSAGE_DELTA_KEY.md)).
+- [ ] RE-392 Recover keyed callers/widths/keys/narrowing and zero/negative/out-of-range mask-index reachability, runtime adjacent-data effects, usercmd/entity/player/snapshot codecs, remaining scalar/overflow/EOF/nonfinite/control-state behavior, native seeded codewords and actual packet/runtime/all-platform/renderer contracts; resolve caller evidence before Rust mask compatibility decisions.

docs/research/MESSAGE_DELTA.md

@@ -37,3 +37,5 @@ Twenty-four additional reader probes per engine supply flag words 0/1/2/0xffffff
 Eight independent tests verify integer full-word/request behavior, finite float comparisons/signed-zero retention, independent flag/remainder/codeword packing, changed integer masking/signed tails, cursor/accounting/diagnostic/canary effects, nonzero supplied reader flags, report dependencies/paired results and the finite comparison model boundary. No evaluator extension was needed.
 
 RE-388: recover integer/float keyed delta masks/XOR/table bounds, usercmd/entity/player/snapshot codecs and actual callers; validate remaining scalar widths/value aliases, compressed capacity/overflow/reset/truncation/EOF/padding, nonfinite floating control behavior, native seeded codewords and actual packet/runtime/all-platform/renderer contracts.
+
+Follow-up: [keyed message deltas](MESSAGE_DELTA_KEY.md) recovers bounded integer mask/XOR, finite float XOR and modeled-tree streams. Broader RE-388 remains open; RE-392 tracks caller/index/narrowing and complete packet/platform contracts.

docs/research/MESSAGE_DELTA_KEY.md

@@ -0,0 +1,56 @@
+# Keyed delta masks, XOR and compressed compositions
+
+Reproduce with `python3 scripts/analyze_message_delta_key.py`; validate with `python3 scripts/test_message_delta_key.py`. [message-delta-key.json](message-delta-key.json) retains 22 source/disassembly-hash-verified graphs across the Linux ELF64 engines, symbol-backed mask-table bytes and adjacent data, 288 compressed keyed streams and 48 supplied-reader control probes. It hashes and checks native identities for the [modeled codebook](HUFFMAN_CODEBOOK.md) and [initialization report](HUFFMAN_INITIALIZATION.md).
+
+All results are synthetic native instruction compositions; original processes and native callees never execute. Compressed trees remain MODEL-DERIVED. Frame saves/restores/PUSH/POP are omitted; helper CALL/tail boundaries stop and compose with explicit caller-register restoration and disjoint virtual frames. Supplied flag/payload probes do not decode actual bits/cursors. Float comparisons cover finite inputs only. Legal caller widths/keys/truncation, full native training, runtime/all-platform and actual packet conformance remain open.
+
+## KEY-001 — Integer XOR and direct mask-table indexing (RE-389)
+
+MSG_WriteDeltaKey compares complete baseline/new words before XOR. Equal words request only a zero flag of width one, regardless of the key. Changed words request a one flag, then forward `(new_word XOR key_word)` with the supplied width. The underlying writer masks/range-checks that XOR word; the delta wrapper does not validate the width or key locally.
+
+MSG_ReadDeltaKey requests width one, returning the full baseline when the flag is zero. For nonzero flags it reads the supplied payload width, indexes the file-backed kbitmask array directly by that signed width, masks the key with the indexed word, then XORs that masked key into the read result. It does not locally clamp the index or add a scalar-wrapper post-read EOF check.
+
+Both ELF symbol tables identify kbitmask as 128 bytes: exactly 32 little-endian words. Independent extraction/hash checks recover entry i as `(1 << (i+1)) - 1` for i zero through 31. Thus the directly selected mask is one bit wider than the requested payload width for sampled positive widths below 32:
+
+| Supplied width | Indexed mask |
+| --- | --- |
+| 1 | 0x00000003 |
+| 7 | 0x000000ff |
+| 8 | 0x000001ff |
+| 16 | 0x0001ffff |
+| 31 | 0xffffffff |
+
+For sampled changed positive widths W below 32, the composed returned word is `(new_word & ((1 << W)-1)) | (key_word & (1 << W))`. The extra returned bit comes from the key. This is a local return-word contract, not a proof of a game-level discrepancy: callers may restrict keys/widths or truncate to narrower fields. Those callers must be recovered before choosing Rust compatibility behavior.
+
+Width 32 indexes immediately beyond the symbol-backed array. The next ELF symbol is entityStateFields, starting exactly at kbitmask +128. The synthetic composition supplies its actual file-backed first four bytes, not an invented 33rd mask entry:
+
+| Image | Adjacent low word |
+| --- | --- |
+| Linux dedicated ELF64 | 0x004b8135 |
+| Linux client ELF64 | 0x0056fcb4 |
+
+For these supplied width-32 states the returned word is `(new XOR key) XOR (key & adjacent_word)`, and can differ between images for identical encoded bytes. The report retains array and neighboring symbol bounds, addresses, offsets and exact bytes. It does not establish that a width-32 integer-key call is legal/reachable, or that runtime relocation/lifetime/platform behavior matches file-backed memory. Negative/zero/out-of-range indexes and their callers remain open.
+
+## KEY-002 — Keyed finite float delta (RE-390)
+
+MSG_WriteDeltaKeyFloat uses finite scalar comparison to decide whether to emit a payload. Equal numeric values request only a zero flag; opposite signed zeros therefore retain the baseline's bits. Changed values request a one flag and then the raw new scalar word XOR the full key with width thirty-two.
+
+The float reader returns baseline bits for a zero flag. For any supplied nonzero flag it reads thirty-two bits, XORs the full key into the raw word and returns that scalar payload through moves. It does not use kbitmask. Encoded XOR words can look nonfinite, but the sampled arithmetic comparison happens on finite inputs before XOR; payload handling is integer/raw-bit transport.
+
+Forty-eight finite float streams per engine cover eight baseline/new pairs, keys zero/0x80000100/0xffffffff and initial bit positions zero/seven. Changed values recover the complete new word; unchanged values retain the baseline, including signed-zero cases. Native NaN/unordered/exception/MXCSR/FTZ/DAZ and caller sentinel behavior remain unmodeled.
+
+## KEY-003 — Compressed keyed streams and paired-image differences (RE-391)
+
+Each image also has 96 integer streams across widths 1/7/8/9/16/24/31/32, four baseline/new pairs and the same three keys, with initial bit zero. Every stream composes key-delta requests with WriteBits/raw-bit/transmit/send instructions, then resets the supplied read cursor and composes ReadBits/raw-bit/receive and key-delta instructions against the exact output bytes. The modeled tree determines Huffman symbols/code lengths; they are not supplied separately at helper boundaries.
+
+Independent packing verifies the change bit, XOR payload's raw remainder and modeled Huffman byte codes. Payload bytes/cursors agree between native images. Decoded integer results match independently extracted direct-index masks, including the width-32 adjacent-word difference. Keyed float results recover the expected raw word without that table effect.
+
+Oldsize counts one flag bit plus the requested payload width when changed. Range diagnostics inspect the signed XOR payload before width masking; they do not stop the sampled writes. Message overflow stays zero in explicitly oversized 256-byte, zero-initialized buffers with canaries. Size/readcount retain `(bit >> 3) + 1` for the sampled nonnegative cursors. Near-capacity expansion, dirty prefixes, padding, truncation/EOF and full networking recovery remain open.
+
+Twenty-four reader control probes per engine supply flags 0/1/2/0xffffffff, three keys, baseline 0x1234 and a raw payload word. Zero retains baseline; other words trigger integer width thirteen or float width thirty-two payload requests. Integer return XOR uses the direct table entry; float uses the full key. These supplied noncanonical flags are boundary probes, not legal one-bit decoder outputs.
+
+## Validation and next work
+
+Eight independent tests verify symbol/file-backed table and adjacent bytes, integer decision/request XOR, finite float/signed-zero behavior, independent compressed packing/cursors, exact integer decode masks, supplied flags, accounting/XOR diagnostics/dependency hashes, and paired payloads with explicitly source-dependent width-32 results. No evaluator extension was needed.
+
+RE-392: recover keyed delta caller widths/keys and narrowing, mask-index zero/negative/out-of-range reachability, runtime adjacent-data effects, usercmd/entity/player/snapshot codecs, remaining scalar/overflow/EOF/nonfinite/control-state behavior, native trained codewords and actual packet/runtime/all-platform/renderer contracts. Resolve caller evidence before normalizing or preserving the observed mask behavior in Rust.

docs/research/README.md

@@ -149,3 +149,5 @@ The extension collector adds five focused tests for native tail calls, command-r
 [Raw float transport and finite angle codecs](MESSAGE_NUMERIC.md) adds RE-381–383: raw float OOB/EOF behavior, finite angle encoding and signed angle16 reads. Reproduce with `python3 scripts/analyze_message_numeric.py`; check with `python3 scripts/test_message_numeric.py`. Delta/caller/control-state/compressed and actual packet/runtime/platform contracts remain open.
 
 [Scalar delta flags and compressed compositions](MESSAGE_DELTA.md) adds RE-385–387: integer changes, finite float equality/signed zeros and modeled-tree byte/cursor composition. Reproduce with `python3 scripts/analyze_message_delta.py`; check with `python3 scripts/test_message_delta.py`. Keyed/higher-level/overflow/EOF/nonfinite and actual packet/runtime/platform contracts remain open.
+
+[Keyed delta masks, XOR and compressed compositions](MESSAGE_DELTA_KEY.md) adds RE-389–391: actual integer table/bounds/adjacent reads, finite float XOR and modeled-tree payload/cursor checks. Reproduce with `python3 scripts/analyze_message_delta_key.py`; check with `python3 scripts/test_message_delta_key.py`. Caller/narrowing/index/higher-level and actual packet/runtime/platform contracts remain open.

scripts/analyze_message_delta_key.py

@@ -0,0 +1,100 @@
+#!/usr/bin/env python3
+"""Keyed delta masks/XOR and modeled-tree compressed compositions."""
+import gzip
+import hashlib
+import itertools
+import json
+import re
+import struct
+from pathlib import Path
+from analyze_bsp_loading import read_address
+from analyze_native import elf
+from analyze_bsp_references import global_address, trace_fields
+from analyze_collision_composition import get_word, put_word
+from analyze_engine_control import control_flow
+from analyze_message_byte_codec import MSG, DATA, install_tree
+from analyze_message_numeric import frame_graph
+from analyze_message_widths import reader, run, writer
+from analyze_message_wrappers import WRAPPER_STACK
+from analyze_qvm import write_json
+from analyze_vm_extensions import native_functions, verified
+
+
+INTEGER_PAIRS=[(0,0),(0,1),(1,0),(0x1234,0x1234),(0x1234,0xffffffff),(0xffffffff,0)]
+FLOAT_PAIRS=[(0,0),(0,0x80000000),(0x80000000,0),(0x3f800000,0x3f800000),(0x3f800000,0x3fc00000),(0xbf800000,0x3f800000),(0x3f800000,0),(0x7f7fffff,0x7f7fffff)]
+
+
+def write_delta(graphs,m,kind,baseline,value,width,key):
+    name='MSG_WriteDeltaKeyFloat' if kind=='float' else 'MSG_WriteDeltaKey';g,omitted=frame_graph(graphs[name]);rows=g['instructions'];calls={x['address']:i for i,x in enumerate(rows) if x['opcode']=='callq'};tails={x['address'] for x in rows if x['opcode']=='jmp' and '<MSG_WriteBits>' in x['operands']};ends={x['address'] for x in rows if x['opcode']=='retq'}
+    regs={'%rdi':MSG,'%rsi':key,'%rdx':baseline,'%rcx':value,'%r8':width&0xffffffff,'%xmm0':baseline,'%xmm1':value,'%rsp':WRAPPER_STACK};pc=rows[0]['address'];requests=[];trace=[]
+    while True:
+        t=run(g,pc,list(calls)+list(tails)+list(ends),m,regs);m=t['memory'];regs=t['registers'];trace+=t['trace']
+        if t['stop'] in ends:break
+        request=[regs[k] for k in ['rdi','rsi','rdx']];requests.append(request)
+        if request[0]!=MSG:raise ValueError('delta writer pointer')
+        word_width=request[2];signed_width=word_width if word_width<2**31 else word_width-2**32
+        m,_,error,child=writer(graphs,m,signed_width,request[1]);trace+=child
+        if error:raise ValueError('delta writer fixture error')
+        if t['stop'] in tails:break
+        pc=rows[calls[t['stop']]+1]['address']
+        if len(requests)>2:raise ValueError('delta writer bound')
+    return m,requests,trace,omitted
+
+
+def read_delta(graphs,m,kind,baseline,width,key,supplied=None):
+    name='MSG_ReadDeltaKeyFloat' if kind=='float' else 'MSG_ReadDeltaKey';g,omitted=frame_graph(graphs[name]);rows=g['instructions'];calls={x['address']:i for i,x in enumerate(rows) if x['opcode']=='callq'};tails={x['address'] for x in rows if x['opcode']=='jmp' and '<MSG_ReadBits>' in x['operands']};ends={x['address'] for x in rows if x['opcode']=='retq'}
+    regs={'%rdi':MSG,'%rsi':key,'%rdx':baseline,'%rcx':width&0xffffffff,'%xmm0':baseline,'%rsp':WRAPPER_STACK};pc=rows[0]['address'];requests=[];trace=[]
+    while True:
+        t=run(g,pc,list(calls)+list(tails)+list(ends),m,regs);m=t['memory'];regs=t['registers'];trace+=t['trace']
+        if t['stop'] in ends:break
+        request=[regs[k] for k in ['rdi','rsi']];requests.append(request)
+        if request[0]!=MSG:raise ValueError('delta reader pointer')
+        word_width=request[1];signed_width=word_width if word_width<2**31 else word_width-2**32
+        if supplied is None:
+            m,value,error,child=reader(graphs,m,signed_width);trace+=child
+            if error:raise ValueError('delta reader fixture error')
+        else:value=supplied[len(requests)-1]
+        regs['rax']=value
+        if t['stop'] in tails:break
+        pc=rows[calls[t['stop']]+1]['address']
+        if len(requests)>2:raise ValueError('delta reader bound')
+    return m,regs['xmm0' if kind=='float' else 'rax']&0xffffffff,requests,trace,omitted
+
+
+def main():
+    cp=Path('docs/research/huffman-codebook.json');ip=Path('docs/research/huffman-initialization.json');code=json.loads(cp.read_text());initial=json.loads(ip.read_text())
+    out={'native':{},'codebook_report_sha256':hashlib.sha256(cp.read_bytes()).hexdigest(),'initialization_report_sha256':hashlib.sha256(ip.read_bytes()).hexdigest(),'limitations':'Synthetic Linux ELF64 instruction compositions only; original processes/native callees never execute. Compressed tree is MODEL-DERIVED, not complete native training/runtime capture. Save/restore/PUSH/POP omitted; disjoint virtual frames and caller-register restoration explicit. Float writer comparisons are finite only; signed zeros compare equal. Separate supplied-reader flag/return probes do not compose actual bits or cursor effects. Ample zero-initialized buffers, sampled integer widths and pairs only. Integer mask table has 32 symbol-backed words; sampled width32 reads a file-backed adjacent entityStateFields word, not a valid mask entry. No legal-index/overflow/truncation/usercmd/entity/player/snapshot/real-packet/all-platform conformance.'}
+    for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
+        r=json.loads(Path(f'docs/research/native/{label}.json').read_text());data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);meta,symbols=elf(data);a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());names=['MSG_WriteDeltaKey','MSG_ReadDeltaKey','MSG_WriteDeltaKeyFloat','MSG_ReadDeltaKeyFloat','MSG_WriteBits','MSG_ReadBits','Huff_putBit','Huff_getBit','Huff_offsetTransmit','Huff_offsetReceive','send'];graphs={n:control_flow(fs[n]) for n in names}
+        if code['native'][label]['source']!=r['source'] or code['native'][label]['disassembly_sha256']!=a['uncompressed_sha256']:raise ValueError('codebook provenance')
+        if initial['native'][label]['source']!=r['source'] or initial['native'][label]['disassembly_sha256']!=a['uncompressed_sha256']:raise ValueError('initialization provenance')
+        mask_instruction=next(x for x in graphs['MSG_ReadDeltaKey']['instructions'] if x['opcode']=='andl' and '(,%rax,4)' in x['operands'])
+        mask_address=int(re.search(r'(0x[0-9a-f]+)\(',mask_instruction['operands'])[1],16);raw,offset=read_address(data,meta['sections'],mask_address,132)
+        mask_symbol=next(x for x in symbols if x['name']=='kbitmask' and x['address']==mask_address);adjacent=next(x for x in symbols if x['name']=='entityStateFields' and x['address']==mask_address+128)
+        if mask_symbol['bytes']!=128:raise ValueError('mask symbol size')
+        mask_words=list(struct.unpack('<33I',raw));mask_report={'address':mask_address,'file_offset':offset,'symbol':mask_symbol,'table_words':mask_words[:32],'table_sha256':hashlib.sha256(raw[:128]).hexdigest(),'adjacent_word':mask_words[32],'adjacent_symbol':adjacent,'adjacent_bytes_hex':raw[128:].hex()}
+        template={mask_address+i:v for i,v in enumerate(raw)};base=initial['native'][label]['entry_request']['huff_init_pointer'];state=code['model_derived_seed_training']['final_state'];install_tree(template,state,base);root=install_tree(template,state,base+0xc830)
+        for x in graphs['MSG_ReadBits']['instructions']:
+            if x['opcode']=='movq' and '(%rip)' in x['operands']:put_word(template,global_address(x),root,8)
+        g=graphs['MSG_WriteBits'];old=global_address(next(x for x in g['instructions'] if x['opcode']=='addl' and '(%rip)' in x['operands']));over=global_address(next(x for x in g['instructions'] if x['opcode']=='addl' and '$0x1,' in x['operands'] and '(%rip)' in x['operands']))
+        streams=[]
+        fixtures=[('integer',w,b,v,0,key) for w,(b,v),key in itertools.product([1,7,8,9,16,24,31,32],[(0,0),(0,1),(0x1234,0x1234),(0x1234,0xffffffff)],[0,0x80000100,0xffffffff])]+[('float',32,b,v,bit,key) for (b,v),bit,key in itertools.product(FLOAT_PAIRS,[0,7],[0,0x80000100,0xffffffff])]
+        for kind,width,baseline,value,bit,key in fixtures:
+            m=dict(template)
+            for i in range(256):m[DATA+i]=0
+            m[DATA-1]=m[DATA+256]=0x7b
+            for offset,v in [(4,0),(8,0),(16,DATA),(24,256),(28,bit//8+1),(32,0),(36,bit)]:put_word(m,MSG+offset,v,8 if offset==16 else 4)
+            put_word(m,old,0);put_word(m,over,0)
+            m,wrequests,trace,womitted=write_delta(graphs,m,kind,baseline,value,width,key);write_bit=get_word(m,MSG+36);size=get_word(m,MSG+28);payload=bytes(m[DATA+i] for i in range(size)).hex();account=get_word(m,old);counter=get_word(m,over)
+            put_word(m,MSG+36,bit);put_word(m,MSG+32,0);m,result,rrequests,child,romitted=read_delta(graphs,m,kind,baseline,width,key);trace+=child
+            streams.append({'kind':kind,'key_word':key,'width':width,'baseline_word':baseline,'value_word':value,'initial_bit':bit,'write_requests':wrequests,'read_requests':rrequests,'payload_hex':payload,'write_bit':write_bit,'write_size':size,'read_bit':get_word(m,MSG+36),'read_count':get_word(m,MSG+32),'result_word':result,'oldsize_word':account,'range_counter':counter,'overflow_flag':get_word(m,MSG+4),'canaries':[m[DATA-1],m[DATA+256]],'writer_omitted':womitted,'reader_omitted':romitted,**trace_fields({'trace':trace},False)})
+        probes=[]
+        for kind,flag,key in itertools.product(['integer','float'],[0,1,2,0xffffffff],[0,0x80000100,0xffffffff]):
+            baseline=0x1234;payload=0x8196a5ff
+            _,result,requests,trace,omitted=read_delta(graphs,template,kind,baseline,13,key,[flag,payload]);probes.append({'kind':kind,'key_word':key,'supplied_flag_word':flag,'baseline_word':baseline,'supplied_payload_word':payload,'width':13,'result_word':result,'requests':requests,'omitted':omitted,**trace_fields({'trace':trace},False)})
+        out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,'mask_table':mask_report,'compressed_streams':streams,'supplied_reader_probes':probes}
+    write_json(Path('docs/research/message-delta-key.json'),out)
+    print('Recovered keyed delta mask/XOR boundaries and modeled-tree streams')
+
+
+if __name__=='__main__':main()

scripts/test_message_delta_key.py

@@ -0,0 +1,81 @@
+#!/usr/bin/env python3
+"""Independent keyed masks/XOR, payload packing and source-boundary oracles."""
+import hashlib
+import json
+from pathlib import Path
+import struct
+import unittest
+
+
+def f32(word):return struct.unpack('<f',struct.pack('<I',word))[0]
+def changed(p):return f32(p['baseline_word'])!=f32(p['value_word']) if p['kind']=='float' else p['baseline_word']!=p['value_word']
+
+
+class MessageDeltaKeyTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):
+        cls.r=json.loads(Path('docs/research/message-delta-key.json').read_text());cls.words={x['symbol']:x['bits'] for x in json.loads(Path('docs/research/huffman-codebook.json').read_text())['model_derived_seed_training']['codebook']}
+
+    def test_symbol_backed_mask_table_and_adjacent_file_word(self):
+        for n in self.r['native'].values():
+            p=n['mask_table'];expected=[(1<<(i+1))-1 for i in range(32)];self.assertEqual(p['table_words'],expected);self.assertEqual(p['symbol']['bytes'],128);self.assertEqual(p['symbol']['name'],'kbitmask');self.assertEqual(p['symbol']['address'],p['address']);self.assertEqual(p['adjacent_symbol']['name'],'entityStateFields');self.assertEqual(p['adjacent_symbol']['address'],p['address']+128)
+            raw=struct.pack('<32I',*expected);self.assertEqual(p['table_sha256'],hashlib.sha256(raw).hexdigest());self.assertEqual(p['adjacent_bytes_hex'],p['adjacent_word'].to_bytes(4,'little').hex())
+            data=(Path('.cache/UrbanTerror43')/n['source']['path']).read_bytes();self.assertEqual(data[p['file_offset']:p['file_offset']+132],raw+bytes.fromhex(p['adjacent_bytes_hex']))
+
+    def test_integer_keyed_requests_compare_before_xor(self):
+        for n in self.r['native'].values():
+            rows=[p for p in n['compressed_streams'] if p['kind']=='integer'];self.assertEqual(len(rows),96)
+            for p in rows:
+                c=changed(p);self.assertEqual(p['write_requests'],[[0x90000,int(c),1]]+([[0x90000,p['value_word']^p['key_word'],p['width']]] if c else []));self.assertEqual(p['read_requests'],[[0x90000,1]]+([[0x90000,p['width']]] if c else []))
+
+    def test_float_full_word_xor_and_signed_zero_baseline(self):
+        for n in self.r['native'].values():
+            rows=[p for p in n['compressed_streams'] if p['kind']=='float'];self.assertEqual(len(rows),48)
+            for p in rows:
+                c=changed(p);self.assertEqual(p['write_requests'],[[0x90000,int(c),1]]+([[0x90000,p['value_word']^p['key_word'],32]] if c else []));self.assertEqual(p['result_word'],p['value_word'] if c else p['baseline_word'])
+                if {p['baseline_word'],p['value_word']}=={0,0x80000000}:self.assertEqual(len(p['write_requests']),1)
+
+    def test_independent_flag_raw_remainder_codeword_bytes_and_cursors(self):
+        for n in self.r['native'].values():
+            self.assertEqual(len(n['compressed_streams']),144)
+            for p in n['compressed_streams']:
+                c=changed(p);bits=[int(c)]
+                if c:
+                    width=p['width'];rem=width%8;word=(p['value_word']^p['key_word'])&((1<<width)-1);bits += [(word>>i)&1 for i in range(rem)]
+                    for i in range(width//8):bits+=self.words[(word>>(rem+8*i))&255]
+                final=p['initial_bit']+len(bits);payload=bytearray(final//8+1)
+                for i,bit in enumerate(bits):pos=p['initial_bit']+i;payload[pos//8]|=bit<<(pos%8)
+                self.assertEqual(p['payload_hex'],payload.hex());self.assertEqual(p['write_bit'],final);self.assertEqual(p['read_bit'],final);self.assertEqual(p['write_size'],final//8+1);self.assertEqual(p['read_count'],final//8+1)
+
+    def test_integer_decoder_direct_index_extra_key_bit_and_adjacent_word(self):
+        for n in self.r['native'].values():
+            masks=n['mask_table']['table_words']+[n['mask_table']['adjacent_word']]
+            for p in n['compressed_streams']:
+                if p['kind']!='integer':continue
+                expected=p['baseline_word']
+                if changed(p):expected=((p['value_word']^p['key_word'])&((1<<p['width'])-1))^(p['key_word']&masks[p['width']])
+                self.assertEqual(p['result_word'],expected)
+                if changed(p) and p['width']<32:self.assertEqual(p['result_word'],(p['value_word']&((1<<p['width'])-1))|(p['key_word']&(1<<p['width'])))
+
+    def test_supplied_nonzero_flags_key_effects(self):
+        for n in self.r['native'].values():
+            self.assertEqual(len(n['supplied_reader_probes']),24)
+            for p in n['supplied_reader_probes']:
+                c=p['supplied_flag_word']!=0;mask=0xffffffff if p['kind']=='float' else n['mask_table']['table_words'][13];expected=p['supplied_payload_word']^(p['key_word']&mask) if c else p['baseline_word'];self.assertEqual(p['result_word'],expected);width=32 if p['kind']=='float' else 13;self.assertEqual(p['requests'],[[0x90000,1]]+([[0x90000,width]] if c else []))
+
+    def test_requested_accounting_xored_range_diagnostic_and_dependencies(self):
+        for key,name in [('codebook_report_sha256','huffman-codebook.json'),('initialization_report_sha256','huffman-initialization.json')]:self.assertEqual(self.r[key],hashlib.sha256((Path('docs/research')/name).read_bytes()).hexdigest())
+        for n in self.r['native'].values():
+            for p in n['compressed_streams']:
+                word=p['value_word']^p['key_word'];value=word if word<2**31 else word-2**32;overflow=changed(p) and p['width']!=32 and (value<0 or value>(1<<p['width'])-1)
+                self.assertEqual(p['range_counter'],int(overflow));self.assertEqual(p['oldsize_word'],1+(p['width'] if changed(p) else 0));self.assertEqual(p['overflow_flag'],0);self.assertEqual(p['canaries'],[0x7b]*2)
+
+    def test_paired_payloads_and_source_specific_width32_decode(self):
+        a,b=self.r['native'].values();self.assertEqual(a['mask_table']['table_words'],b['mask_table']['table_words']);self.assertNotEqual(a['mask_table']['adjacent_word'],b['mask_table']['adjacent_word'])
+        def clean(rows):return [{k:v for k,v in p.items() if k not in ['trace','trace_sha256','writer_omitted','reader_omitted','omitted'] and not(k=='result_word' and p['kind']=='integer' and p['width']==32)} for p in rows]
+        self.assertEqual(clean(a['compressed_streams']),clean(b['compressed_streams']));self.assertEqual(clean(a['supplied_reader_probes']),clean(b['supplied_reader_probes']))
+        differences=[(x,y) for x,y in zip(a['compressed_streams'],b['compressed_streams']) if x['result_word']!=y['result_word']];self.assertTrue(differences)
+        for x,y in differences:self.assertEqual(x['kind'],'integer');self.assertEqual(x['width'],32);self.assertEqual(x['payload_hex'],y['payload_hex'])
+
+
+if __name__=='__main__':unittest.main()