Recover message read modes and bounded bit assembly

docs/ASSET_PIPELINE_TODO.md

@@ -212,3 +212,5 @@ Area-state follow-up: [flooding and bit output](research/AREA_FLOODING.md) compo
 Snapshot follow-up: [area-mask lifecycle](research/SNAPSHOT_AREAS.md) connects supplied area labels to bounded native frame masks and message requests. Establish actual map area limits/reference validity before runtime conversion and renderer/network integration (RE-336).
 
 Client-mask follow-up: [client parsing](research/CLIENT_MASKS.md) bounds accepted area-mask length to 32 in the sampled Linux parser. Map-derived area limits, real protocol decoding and renderer mask interpretation still require validation before asset/runtime integration (RE-340).
+
+Message-bit follow-up: [read modes and assembly](research/MESSAGE_BITS.md) establishes bounded cursor/assembly behavior with supplied Huffman symbols. Recover actual tree/codebooks and real packets before network/renderer integration with converted map assets (RE-344).

docs/FINDINGS.md

@@ -366,3 +366,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
 **F-113 — Observed byte-reader boundary:** signed post-readcount <= cursize returns the supplied result's zero-extended low byte; larger readcounts return 0xffffffff. Malformed signed fixtures are local comparison probes, not valid runtime messages; actual bit/cursor updates remain open. [CLIENT-002](research/CLIENT_MASKS.md).
 
 **F-114 — Observed data-copy behavior:** positive lengths request one eight-bit read per byte even after overflow, storing 0xff past supplied cursize; signed nonpositive lengths exit. Supplied snapshot reset leaves unused mask bytes zero. This is bounded native instruction composition, not encoded-packet or original-runtime conformance. [CLIENT-003](research/CLIENT_MASKS.md).
+
+**F-115 — Observed read-mode resets:** both Linux ELF64 engines clear readcount +32 and bit +36; ordinary/OOB entry sets mode +8 to zero/one, retaining other sampled bytes. Whole message initialization and mode lifecycle remain open. [BITS-001](research/MESSAGE_BITS.md).
+
+**F-116 — Observed OOB bit reads:** sampled nonzero mode reads only absolute widths 8/16/32 with little-endian loads and direct cursor increments, without consulting cursize locally; unsupported widths request an error. The -32 top-bit-set tail yields all ones through x86 shift masking. Actual legal caller/memory/runtime/platform behavior remains open. [BITS-002](research/MESSAGE_BITS.md).
+
+**F-117 — Observed compressed reader assembly:** raw low remainder bits precede supplied Huffman symbol bytes; readcount becomes signed bit/8 +1. Signed tails use the remaining byte-multiple width after remainder subtraction. Huff_getBit instructions update cursor and bloc; real tree decoding, packets and original-runtime conformance remain open. [BITS-003](research/MESSAGE_BITS.md).

docs/PROVISIONAL_TODO.md

@@ -524,3 +524,5 @@ Research progress: [area flooding and bit output](research/AREA_FLOODING.md) com
 Research progress: [snapshot area-mask lifecycle](research/SNAPSHOT_AREAS.md) completes bounded RE-333–335. Full networking/renderer/runtime milestones remain open; RE-336 tracks real viewpoint selection, codecs, client parsing and mask consumers.
 
 Research progress: [client area-mask parsing](research/CLIENT_MASKS.md) completes bounded RE-337–339. Broad protocol/renderer/runtime milestones remain open; RE-340 tracks actual codec/cursors, validation/publication and real packet/platform coverage.
+
+Research progress: [message read modes and assembly](research/MESSAGE_BITS.md) completes bounded RE-341–343. Broad networking/runtime milestones remain open; RE-344 tracks real Huffman codecs, caller invariants and packet/platform fixtures.

docs/RESEARCH_LEDGER.md

@@ -201,3 +201,7 @@ Completed bounded RE-333–335 with `analyze_snapshot_areas.py`: ten verified gr
 ## 2026-10-11 — Client snapshot mask parsing and reader boundaries
 
 Completed bounded RE-337–339 with `analyze_client_masks.py`: three verified Linux client graphs, one reset-argument slice, 22 unsigned guard probes, 210 signed byte-reader probes and 53 unrolled data-reader compositions. CLIENT-001–003/F-112–114 distinguish reset requests from supplied memset effects, EOF return words from low-byte 0xff, signed cursor comparison from actual bit/Huffman cursor updates, and exact destination copying from full snapshot publication. PUSH/POP omission and stopped helper boundaries are explicit; no original process or native callee executes. Eight independent tests check reset/guard/error arguments, signed returns, copy/EOF continuation/tail/canaries/loop coverage, trace boundaries and evaluator CMOVLEL/BL/unknown-flag behavior. The full suite passes 286 tests. New and preceding spatial reports regenerate byte-identically; storage, links and whitespace checks pass. RE-336 remains open; RE-340 tracks actual codecs/cursors, bad-message checks, full validation/publication, renderer semantics and real-packet/map/original-runtime/all-platform coverage.
+
+## 2026-10-11 — Message read modes and bit assembly
+
+Completed bounded RE-341–343 with `analyze_message_bits.py`: eight verified graphs across both Linux ELF64 engines, eight exact reset probes, 600 OOB and 688 compressed reader compositions. BITS-001–003/F-115–117 distinguish padded backing storage from cursize, direct OOB loads from compressed cursors, composed raw-bit instructions from supplied Huffman symbols/code lengths, and requested signed width from the mutated width used by the tail. Nonbyte and 32-bit signed-tail effects are recorded without assuming legal packet reachability or cross-platform equivalence. Prologue/frame omission and explicit caller-register restoration are documented; original processes and every native callee remain unexecuted. Eight independent tests verify reset fields/canaries, dispatch/errors, little-endian/sign/cursor results, raw/symbol assembly, helper arguments/global writes, cursor formula, signed edge examples, paired outputs and restricted MOVZWL/NEGL/SHRL evaluator behavior. All 294 tests pass; new and preceding reports regenerate byte-identically, and storage/link/whitespace checks pass. RE-340 remains open; RE-344 tracks Huffman initialization/tree/codebooks/receive/transmit/put-bit, actual writer/reader packets, legal width/mode/cursor callers and full runtime/platform/snapshot/renderer conformance.

docs/REVERSE_ENGINEERING_TODO.md

@@ -407,3 +407,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
 - [x] RE-338 Recover bounded byte-reader low-byte/EOF returns and signed cursor comparison ([CLIENT-002](research/CLIENT_MASKS.md)).
 - [x] RE-339 Compose exact unrolled mask copies with supplied bit returns/readcounts, EOF continuation and zero-tail/canary checks ([CLIENT-003](research/CLIENT_MASKS.md)).
 - [ ] RE-340 Recover actual bit/Huffman codec and readcount updates, overflow/bad-message checks, full snapshot/delta/player/entity validation/publication and renderer mask interpretation; validate real packets/maps, original-runtime and all platforms before Rust/Bevy/Solari integration.
+
+- [x] RE-341 Recover paired exact readcount/bit/mode resets and retained bytes ([BITS-001](research/MESSAGE_BITS.md)).
+- [x] RE-342 Recover bounded OOB width/error dispatch, little-endian reads and cursor/signed-tail effects ([BITS-002](research/MESSAGE_BITS.md)).
+- [x] RE-343 Compose raw-bit instructions and supplied compressed symbol assembly, cursor/global writes and remaining-width signed tails ([BITS-003](research/MESSAGE_BITS.md)).
+- [ ] RE-344 Recover Huffman tree initialization/frequencies/codebooks/receive/transmit/put-bit, full writer-reader and real-packet compositions, overflow/truncation and legal signed width/mode/cursor callers; validate snapshot/renderer integration and original-runtime/all-platform contracts.

docs/research/CLIENT_MASKS.md

@@ -31,3 +31,5 @@ The server requests only its mask's used byte span; the client guard caps that s
 Eight independent tests cover reset arguments, both unsigned guards and error arguments, signed cursor/low-byte return, exact copy requests and bytes, continued EOF requests, canaries/tails/unrolled lengths, stopped CALL boundaries, and restricted evaluator CMOVLEL/BL semantics with unknown-flag rejection. Reports include source/disassembly hashes and native trace addresses or hashes. The evaluator extension adds only signed conditional move and the BL register alias.
 
 RE-340: recover actual bit/Huffman encoding and readcount updates, message overflow/bad-read handling, all snapshot/delta/player/entity validation and publication, mask renderer interpretation and map-area validity; compose real encoded packets and original-runtime/all-platform fixtures. Implement Rust protocol/Bevy/Solari integration only against a complete validated contract.
+
+Bit-reader follow-up: [message modes and assembly](MESSAGE_BITS.md) recovers bounded OOB reads, raw bit instructions and compressed cursor/sign-tail behavior (RE-341–343). Huffman symbols/code lengths remain supplied; real packet decoding and full snapshot validation remain open.

docs/research/MESSAGE_BITS.md

@@ -0,0 +1,35 @@
+# Message read modes, raw bits and compressed assembly
+
+Reproduce with `python3 scripts/analyze_message_bits.py`; validate with `python3 scripts/test_message_bits.py`. [message-bits.json](message-bits.json) retains four hash-verified graphs per Linux ELF64 engine: MSG_BeginReading, MSG_BeginReadingOOB, MSG_ReadBits and Huff_getBit. This follows [client mask reader boundaries](CLIENT_MASKS.md).
+
+The collector evaluates bounded native instruction slices using virtual message, data and stack addresses. Prologue register saves/restores are excluded. Huff_getBit instructions are composed across stopped calls with explicit caller-register restoration; Huff_offsetReceive never decodes a tree and instead receives supplied symbol/cursor effects. Original processes and every native callee remain unexecuted. Backing bytes are deliberately padded independently of cursize; these fixtures characterize local access and comparison behavior, not safe real packet loading, valid malformed states or runtime conformance.
+
+## BITS-001 — Read-mode resets (RE-341)
+
+Eight paired probes cover two sentinel initializations of both mode-reset functions. They write readcount at message +32 and bit cursor at +36 to zero. MSG_BeginReading sets the mode word at +8 to zero, while MSG_BeginReadingOOB sets it to one. Every other byte in the supplied 40-byte record retains its sentinel, and outside canaries remain unchanged. These are native Linux layouts, not a portable Rust structure. Initialization/allocation and write-mode transitions remain separate work.
+
+## BITS-002 — Out-of-band reads (RE-342)
+
+Six hundred paired OOB compositions cover widths 8/16/32, negative counterparts, and unsupported 0/1/7/9/24/33; four constant byte patterns; initial readcounts -1/0/3; cursize 0/8; and an additional nonzero mode value two. Any sampled nonzero mode selects this path. Negative requested widths are negated and set a local signed-return flag before dispatch.
+
+Only absolute widths 8,16,32 select byte, zero-extended little-endian word and little-endian dword loads. Readcount increases by 1/2/4 and bit cursor by 8/16/32. The sampled path does not compare cursize before accessing the backing data. Padded fixtures including negative readcount and zero cursize therefore still produce values; this does not establish legal packet states or memory safety. Unsupported widths request Com_Error with code one, the absolute width and the verified format `can't read %d bits\n`. Error execution is stopped before the callee and fixture cursors remain unchanged.
+
+The signed tail tests the top requested bit and ORs a mask produced by a native 32-bit shift. For -8 and -16 this extends the sign normally. For -32 with bit 31 set, x86 masks the shift count to zero, so the tail ORs 0xffffffff: for example a supplied repeated 0x96 dword becomes 0xffffffff rather than 0x96969696. Positive 32-bit reads retain the full loaded word. Legal callers, other architectures and original-runtime observations are still required before choosing compatibility behavior.
+
+## BITS-003 — Raw remainder, compressed symbols and cursor/sign tail (RE-343)
+
+Six hundred eighty-eight paired compressed compositions cover every nonnegative width 0–32, negative widths -1/-7/-8/-9/-15/-16/-17/-24/-31/-32, four backing/symbol patterns and initial bit cursors zero/seven. The reader consumes `abs(width) % 8` raw bits first, then one Huff_offsetReceive request per remaining byte. Raw bits occupy successive low result bits; supplied symbols occupy successive higher eight-bit lanes.
+
+The composed Huff_getBit instructions select `(data[bit >> 3] >> (bit & 7)) & 1`, using signed arithmetic shift for the byte index. They increment the supplied cursor and store the same incremented word to the verified global `bloc`. There is no cursize parameter or bounds check in this helper. The collector composes these actual instructions for raw bits, while every compressed symbol and code length is explicitly supplied.
+
+Each Huff_offsetReceive request forwards a supplied tree pointer (0x40000), a stack symbol-output pointer, the data pointer and message +36 as the bit-cursor pointer. Symbols cycle from the chosen byte with increments of 37 modulo 256. Supplied code lengths are `5 + ordinal % 4`, with ordinals starting at one. These lengths are test inputs, not recovered Huffman code lengths. Actual tree initialization, codebooks, decoding/encoding and malformed-code handling remain open.
+
+After all requests, the reader writes `readcount = (signed_bit_cursor >> 3) + 1`. Our nonnegative bounded fixtures verify this formula, including width zero and exact-byte bit positions: bit zero yields readcount one, and bit eight yields two. Thus compressed readcount is not simply the number of requested payload bytes and is not a ceiling division at byte boundaries. Complete wrap/negative-cursor lifecycle is untested.
+
+The remainder is subtracted from the local width register before the signed tail. The tail tests and extends using this **remaining byte-multiple width**, not the original request. With supplied all-one raw bits, -1 returns 1 and -7 returns 127; a supplied -31 composition returns 0xff927fff. Width -32 also has the shift-count effect described above: symbols `[129,166,203,240]` assemble to 0xf0cba681 for +32 but become 0xffffffff for -32. These observations are local native behavior; they are not a complete protocol interpretation or proof that those signed widths are used by valid game messages.
+
+## Validation and next work
+
+Eight independent tests verify exact mode resets/canaries, OOB width-error dispatch, byte order/sign/cursor increments, raw-bit and symbol assembly, helper arguments/global writes and supplied cursor formula, nonbyte/32-bit signed examples, paired engine outputs, and restricted MOVZWL/NEGL/SHRL evaluator semantics. Each composition retains trace counts/hashes; full verified graphs retain instruction addresses. No native CALL instruction executes implicitly.
+
+RE-344: recover Huffman initialization/frequency tables/tree/codebooks and bounded receive/transmit/put-bit behavior; compose writers with readers and real encoded packets, including overflow/truncation; audit legal signed widths, mode/bit/readcount invariants and actual callers; validate snapshot publication/renderer semantics and all original-runtime/platform contracts before Rust implementation.

docs/research/README.md

@@ -125,3 +125,5 @@ The extension collector adds five focused tests for native tail calls, command-r
 [Snapshot area-mask lifecycle and message arguments](SNAPSHOT_AREAS.md) adds RE-333–335: bounded ring/reset, native area union/complement and length/data request spans. Reproduce with `python3 scripts/analyze_snapshot_areas.py`; check with `python3 scripts/test_snapshot_areas.py`. Full viewpoint/entity/PVS/portal/lifecycle/codec/client/renderer/runtime/platform contracts remain open.
 
 [Client snapshot area-mask parsing](CLIENT_MASKS.md) adds RE-337–339: temporary reset/length guards, byte-reader EOF and bounded unrolled data copies. Reproduce with `python3 scripts/analyze_client_masks.py`; check with `python3 scripts/test_client_masks.py`. Actual codecs/cursors, full snapshot publication and renderer/runtime/platform contracts remain open.
+
+[Message read modes and bit assembly](MESSAGE_BITS.md) adds RE-341–343: exact resets, OOB loads/errors/cursors and raw-bit/compressed-symbol/sign-tail compositions. Reproduce with `python3 scripts/analyze_message_bits.py`; check with `python3 scripts/test_message_bits.py`. Huffman trees, writers, legal caller invariants and real packet/runtime/platform conformance remain open.

scripts/analyze_collision_composition.py

@@ -64,8 +64,8 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
         if op in ['nop','nopl','nopw']:pass
         elif op in ['movb','movl','movq','movss','movabsq']:
             a,b=args.split(', ');width=8 if op in ['movq','movabsq'] else 1 if op=='movb' else 4;write(b,read(a,width),width)
-        elif op=='movzbl':
-            a,b=args.split(', ');write(b,read(a,1),4)
+        elif op in ['movzbl','movzwl']:
+            a,b=args.split(', ');write(b,read(a,1 if op=='movzbl' else 2),4)
         elif op=='sbbl':
             if below is None:raise ValueError('unmodeled carry flag')
             a,b=args.split(', ');write(b,read(b,4)-read(a,4)-int(below),4)
@@ -106,11 +106,13 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
             elif len(parts)==3:a,b,dest=parts;v,w=read(a,width),read(b,width)
             else:raise ValueError('unsupported multiply operands')
             write(dest,v*w,width);equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
-        elif op in ['sarl','shll']:
+        elif op=='negl':
+            write(args,-read(args,4),4);equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
+        elif op in ['sarl','shll','shrl']:
             a,b=args.split(', ');amount=read(a,4)&31;v=read(b,4)
             if amount:
                 signed_v=v if v<2**31 else v-2**32
-                write(b,signed_v>>amount if op=='sarl' else v<<amount,4)
+                write(b,signed_v>>amount if op=='sarl' else v>>amount if op=='shrl' else v<<amount,4)
                 equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
         elif op=='shlq':
             a,b=args.split(', ');write(b,read(b,8)<<(read(a,8)&63),8)

scripts/analyze_message_bits.py

@@ -0,0 +1,75 @@
+#!/usr/bin/env python3
+"""Synthetic native read-mode, OOB and compressed reader compositions."""
+import gzip
+import itertools
+import json
+from pathlib import Path
+from analyze_area_queries import site
+from analyze_bsp_references import global_address,trace_fields
+from analyze_chat_paths import native_string
+from analyze_collision_composition import evaluate,get_word,put_word
+from analyze_engine_control import control_flow
+from analyze_native import elf
+from analyze_qvm import write_json
+from analyze_vm_extensions import native_functions,verified
+
+
+def run(g,start,stops,m,regs):return evaluate(g,start,stops,m,regs,limit=10000,capture_registers=True)
+
+
+def reader(graphs,width,mode,pattern,cursor=0,initial_bit=0,size=0):
+    g=graphs['MSG_ReadBits'];rows=g['instructions'];message,data,stack=0x10000,0x20000,0x30000
+    start=site(g,'testl','%esi, %esi');end=next(x['address'] for x in rows if x['opcode']=='movl' and x['operands'].split('#')[0].strip()=='%ebp, %eax')
+    calls={x['address']:i for i,x in enumerate(rows) if x['opcode']=='callq'}
+    m={}
+    for x in rows:
+        if x['opcode']=='movq' and '(%rip)' in x['operands']:put_word(m,global_address(x),0x40000,8)
+    put_word(m,message+8,mode);put_word(m,message+16,data,8);put_word(m,message+28,size);put_word(m,message+32,cursor);put_word(m,message+36,initial_bit)
+    # Padding bytes model local memory access, not a valid message length.
+    for i in range(-8,24):m[data+i]=[0,255,0x96,0x81][pattern]
+    regs={'%r12':message,'%r15':width&0xffffffff,'%rsi':width&0xffffffff,'%rsp':stack}
+    trace=[];requests=[];symbols=[];raw_bits=[];children=[];error=None
+    while True:
+        t=run(g,start,list(calls)+[end],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
+        if t['stop']==end:break
+        i=calls[t['stop']];name=rows[i]['operands']
+        if '<Com_Error>' in name:
+            error={'arguments':[regs[k] for k in ['rdi','rsi','rdx']]};break
+        if '<Huff_getBit>' in name:
+            requests.append(['Huff_getBit',regs['rdi'],regs['rsi']]);h=graphs['Huff_getBit'];ret=next(x['address'] for x in h['instructions'] if x['opcode']=='retq')
+            child=run(h,h['instructions'][0]['address'],[ret],m,{'%rdi':regs['rdi'],'%rsi':regs['rsi']});m=child['memory'];regs['rax']=child['registers']['rax'];raw_bits.append(regs['rax']);children+=child['trace']
+        elif '<Huff_offsetReceive>' in name:
+            requests.append(['Huff_offsetReceive',regs['rdi'],regs['rsi'],regs['rdx'],regs['rcx']])
+            symbol=([0,255,0x96,0x81][pattern]+37*len(symbols))&255;symbols.append(symbol)
+            # Huffman tree decoding remains stopped; symbol and variable code length supplied.
+            put_word(m,regs['rsi'],symbol);put_word(m,regs['rcx'],get_word(m,regs['rcx'])+5+len(symbols)%4)
+        else:raise ValueError('unexpected helper '+name)
+        if len(requests)>36:raise ValueError('fixture request bound')
+        start=rows[i+1]['address']
+    return {'width':width,'mode_word':mode,'pattern':pattern,'initial_readcount':cursor,'initial_bit':initial_bit,'cursize':size,
+            'result_word':None if error else regs['rbp']&0xffffffff,'error':error,'final_readcount':get_word(m,message+32),'final_bit':get_word(m,message+36),
+            'remaining_width_word':regs['r15']&0xffffffff,'raw_bits':raw_bits,'supplied_symbols':symbols,'requests':requests,
+            'final_raw_bit_global':get_word(m,global_address(next(x for x in graphs['Huff_getBit']['instructions'] if '(%rip)' in x['operands']))) if raw_bits else None,
+            'child_trace':trace_fields({'trace':children},False),**trace_fields({'trace':trace},False)}
+
+
+def main():
+    out={'native':{},'limitations':'Two Linux ELF64 engines, bounded synthetic read slices only. Original processes/callees never execute. Prologue saves/restores excluded and virtual stack supplied. Huff_getBit instructions composed with explicit caller-register restoration; Huff_offsetReceive symbol/cursor effects supplied. Padded backing memory is distinct from cursize. No real Huffman tree/writer/packet/error execution, ABI, legal malformed-state reachability or all-platform conformance.'}
+    for label in ['linux--Quake3-UrT-Ded.x86_64','linux--Quake3-UrT.x86_64']:
+        r=json.loads(Path(f'docs/research/native/{label}.json').read_text());data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);meta,_=elf(data)
+        a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode());graphs={n:control_flow(fs[n]) for n in ['MSG_BeginReading','MSG_BeginReadingOOB','MSG_ReadBits','Huff_getBit']}
+        resets=[]
+        for name,initial in itertools.product(['MSG_BeginReading','MSG_BeginReadingOOB'],[0,0xa5]):
+            g=graphs[name];m={0x10000+i:initial for i in range(-1,41)};end=next(x['address'] for x in g['instructions'] if x['opcode']=='retq');t=run(g,g['instructions'][0]['address'],[end],m,{'%rdi':0x10000})
+            resets.append({'function':name,'initial_byte':initial,'record_bytes':[t['memory'][0x10000+i] for i in range(40)],'canaries':[t['memory'][0xffff],t['memory'][0x10028]],**trace_fields(t,True)})
+        oob=[reader(graphs,w,mode,p,cursor,19,size) for w,mode,p,cursor,size in itertools.product([8,16,32,-8,-16,-32,0,1,7,9,24,33],[1],[0,1,2,3],[-1,0,3],[0,8])]
+        oob += [reader(graphs,w,2,p,0,19,0) for w,p in itertools.product([8,16,32,0,7,-8],[1,2])]
+        compressed=[reader(graphs,w,0,p,0,bit,0) for w,p,bit in itertools.product(list(range(0,33))+[-1,-7,-8,-9,-15,-16,-17,-24,-31,-32],[0,1,2,3],[0,7])]
+        g=graphs['MSG_ReadBits'];err=next(x for x in g['instructions'] if x['opcode']=='movl' and x['operands'].split('#')[0].strip().endswith(', %esi') and '$0x' in x['operands']);addr=int(err['operands'].split(',')[0][1:],0)
+        bit_g=graphs['Huff_getBit'];bloc=global_address(next(x for x in bit_g['instructions'] if '(%rip)' in x['operands']))
+        out['native'][label]={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,'error_format':native_string(data,meta['sections'],addr),'bit_global_address':bloc,'reset_probes':resets,'oob_probes':oob,'compressed_probes':compressed}
+    write_json(Path('docs/research/message-bits.json'),out)
+    print('Recovered eight resets, 600 OOB and 688 compressed reader compositions')
+
+
+if __name__=='__main__':main()

scripts/test_message_bits.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Independent reset, byte-order, bit assembly, cursor and sign-tail oracles."""
+import json
+from pathlib import Path
+import unittest
+from analyze_collision_composition import evaluate,put_word
+from test_bsp_planes import graph
+
+
+class MessageBitTests(unittest.TestCase):
+    @classmethod
+    def setUpClass(cls):cls.r=json.loads(Path('docs/research/message-bits.json').read_text())
+
+    def test_reset_exact_fields_and_canaries(self):
+        for n in self.r['native'].values():
+            for p in n['reset_probes']:
+                expected=[p['initial_byte']]*40
+                for offset,value in [(8,int(p['function'].endswith('OOB'))),(32,0),(36,0)]:expected[offset:offset+4]=list(value.to_bytes(4,'little'))
+                self.assertEqual(p['record_bytes'],expected);self.assertEqual(p['canaries'],[p['initial_byte']]*2)
+
+    def test_oob_width_errors_and_cursize_independence(self):
+        for n in self.r['native'].values():
+            self.assertEqual(len(n['oob_probes']),300)
+            for p in n['oob_probes']:
+                accepted=abs(p['width']) in [8,16,32];self.assertEqual(p['error'] is None,accepted)
+                if not accepted:
+                    self.assertEqual(p['error']['arguments'][0],1);self.assertEqual(p['error']['arguments'][2],abs(p['width']))
+                    self.assertEqual(p['final_readcount'],p['initial_readcount']&0xffffffff);self.assertEqual(p['final_bit'],19)
+                self.assertEqual(p['requests'],[])
+
+    def test_oob_little_endian_signed_result_and_cursors(self):
+        for n in self.r['native'].values():
+            for p in n['oob_probes']:
+                if p['error']:continue
+                width=abs(p['width']);byte=[0,255,0x96,0x81][p['pattern']];v=int.from_bytes(bytes([byte])*(width//8),'little')
+                if p['width']<0 and v&(1<<(width-1)):v|=(0xffffffff<<(width&31))&0xffffffff
+                self.assertEqual(p['result_word'],v&0xffffffff)
+                self.assertEqual(p['final_readcount'],(p['initial_readcount']+width//8)&0xffffffff);self.assertEqual(p['final_bit'],19+width)
+
+    def test_compressed_raw_bits_symbols_and_assembly(self):
+        for n in self.r['native'].values():
+            self.assertEqual(len(n['compressed_probes']),344)
+            for p in n['compressed_probes']:
+                width=abs(p['width']);remainder=width%8;byte=[0,255,0x96,0x81][p['pattern']]
+                raw=[(byte>>((p['initial_bit']+i)%8))&1 for i in range(remainder)]
+                self.assertEqual(p['raw_bits'],raw)
+                self.assertEqual(p['final_raw_bit_global'],p['initial_bit']+remainder if remainder else None)
+                symbols=[(byte+37*i)&255 for i in range(width//8)];self.assertEqual(p['supplied_symbols'],symbols)
+                v=sum(bit<<i for i,bit in enumerate(raw))+sum(sym<<(remainder+8*i) for i,sym in enumerate(symbols))
+                remaining=width-remainder;self.assertEqual(p['remaining_width_word'],remaining)
+                # The native tail tests the mutated remaining width, not the original request.
+                if p['width']<0 and remaining and v&(1<<(remaining-1)):v|=(0xffffffff<<(remaining&31))&0xffffffff
+                self.assertEqual(p['result_word'],v&0xffffffff)
+
+    def test_compressed_cursor_formula_and_helper_arguments(self):
+        for n in self.r['native'].values():
+            for p in n['compressed_probes']:
+                width=abs(p['width']);final=p['initial_bit']+width%8+sum(5+(i+1)%4 for i in range(width//8))
+                self.assertEqual(p['final_bit'],final);self.assertEqual(p['final_readcount'],final//8+1)
+                self.assertEqual(len(p['requests']),width%8+width//8)
+                for q in p['requests']:
+                    if q[0]=='Huff_getBit':self.assertEqual(q[1:],[0x20000,0x10024])
+                    else:self.assertEqual(q[1:],[0x40000,0x30014,0x20000,0x10024])
+
+    def test_nonbyte_signed_tail_examples(self):
+        for n in self.r['native'].values():
+            rows={p['width']:p for p in n['compressed_probes'] if p['pattern']==1 and p['initial_bit']==0}
+            self.assertEqual(rows[-1]['result_word'],1);self.assertEqual(rows[-7]['result_word'],127)
+            self.assertEqual(rows[-8]['result_word'],0xffffffff);self.assertEqual(rows[-31]['result_word'],0xff927fff)
+            high={p['width']:p for p in n['compressed_probes'] if p['pattern']==3 and p['initial_bit']==0}
+            self.assertEqual(high[32]['result_word'],0xf0cba681);self.assertEqual(high[-32]['result_word'],0xffffffff)
+
+    def test_paired_outputs_and_no_calls_in_bit_helper(self):
+        a,b=self.r['native'].values()
+        for key in ['reset_probes','oob_probes','compressed_probes']:
+            def clean(rows):
+                return [{k:(None if k=='error' and v is None else {'arguments':[v['arguments'][0],v['arguments'][2]]} if k=='error' else v) for k,v in p.items() if k not in ['trace','trace_sha256','child_trace']} for p in rows]
+            self.assertEqual(clean(a[key]),clean(b[key]))
+        for n in [a,b]:self.assertFalse(any(x['opcode']=='callq' for x in n['graphs']['Huff_getBit']['instructions']))
+
+    def test_restricted_word_load_negate_logical_shift(self):
+        g=graph([('movzwl','(%rdi), %eax'),('negl','%eax'),('shrl','$0x1, %eax'),('nop','')]);m={};put_word(m,100,0x80ff,2)
+        t=evaluate(g,0,[3],m,{'%rdi':100},capture_registers=True);self.assertEqual(t['registers']['rax'],((-0x80ff)&0xffffffff)>>1)
+
+
+if __name__=='__main__':unittest.main()