nmarshall commited on
Commit
6dafc4b
·
1 Parent(s): 10b0261

Analyze client snapshot mask guards and byte readers

Browse files
docs/ASSET_PIPELINE_TODO.md CHANGED
@@ -210,3 +210,5 @@ Reference follow-up: [leaf references and area state](research/BSP_REFERENCES.md
210
  Area-state follow-up: [flooding and bit output](research/AREA_FLOODING.md) composes supplied portal matrices and generated labels, without loading actual maps. Validate area/reference ranges and lifecycle across every BSP variant before integrating conversion with runtime visibility (RE-332).
211
 
212
  Snapshot follow-up: [area-mask lifecycle](research/SNAPSHOT_AREAS.md) connects supplied area labels to bounded native frame masks and message requests. Establish actual map area limits/reference validity before runtime conversion and renderer/network integration (RE-336).
 
 
 
210
  Area-state follow-up: [flooding and bit output](research/AREA_FLOODING.md) composes supplied portal matrices and generated labels, without loading actual maps. Validate area/reference ranges and lifecycle across every BSP variant before integrating conversion with runtime visibility (RE-332).
211
 
212
  Snapshot follow-up: [area-mask lifecycle](research/SNAPSHOT_AREAS.md) connects supplied area labels to bounded native frame masks and message requests. Establish actual map area limits/reference validity before runtime conversion and renderer/network integration (RE-336).
213
+
214
+ Client-mask follow-up: [client parsing](research/CLIENT_MASKS.md) bounds accepted area-mask length to 32 in the sampled Linux parser. Map-derived area limits, real protocol decoding and renderer mask interpretation still require validation before asset/runtime integration (RE-340).
docs/FINDINGS.md CHANGED
@@ -360,3 +360,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
360
  **F-110 — Observed snapshot mask union/complement:** supplied viewpoint area calls OR into the same frame buffer, store returned length and forward the cluster to PVS. Eight native word inversions complement all 32 bytes; a recursive call forwards the same frame/list pointers. Actual viewpoint/entity/portal/PVS selection and renderer/runtime semantics remain open. [SNAP-002](research/SNAPSHOT_AREAS.md).
361
 
362
  **F-111 — Observed mask serialization requests:** a length-byte writer request precedes a data writer request for exactly frame.length bytes at frame +4. Unused tail bytes are excluded from that requested span; no encoded packet or original message writer executes. Full codec/client/snapshot protocol conformance remains open. [SNAP-003](research/SNAPSHOT_AREAS.md).
 
 
 
 
 
 
 
360
  **F-110 — Observed snapshot mask union/complement:** supplied viewpoint area calls OR into the same frame buffer, store returned length and forward the cluster to PVS. Eight native word inversions complement all 32 bytes; a recursive call forwards the same frame/list pointers. Actual viewpoint/entity/portal/PVS selection and renderer/runtime semantics remain open. [SNAP-002](research/SNAPSHOT_AREAS.md).
361
 
362
  **F-111 — Observed mask serialization requests:** a length-byte writer request precedes a data writer request for exactly frame.length bytes at frame +4. Unused tail bytes are excluded from that requested span; no encoded packet or original message writer executes. Full codec/client/snapshot protocol conformance remains open. [SNAP-003](research/SNAPSHOT_AREAS.md).
363
+
364
+ **F-112 — Observed client mask guard:** the Linux parser requests a 540-byte temporary reset, accepts unsigned mask lengths 0–32 on both sampled paths and forwards temporary +24 to MSG_ReadData; EOF length rejects with an invalid-areamask error request. Complete parsing/publication and other platforms remain open. [CLIENT-001](research/CLIENT_MASKS.md).
365
+
366
+ **F-113 — Observed byte-reader boundary:** signed post-readcount <= cursize returns the supplied result's zero-extended low byte; larger readcounts return 0xffffffff. Malformed signed fixtures are local comparison probes, not valid runtime messages; actual bit/cursor updates remain open. [CLIENT-002](research/CLIENT_MASKS.md).
367
+
368
+ **F-114 — Observed data-copy behavior:** positive lengths request one eight-bit read per byte even after overflow, storing 0xff past supplied cursize; signed nonpositive lengths exit. Supplied snapshot reset leaves unused mask bytes zero. This is bounded native instruction composition, not encoded-packet or original-runtime conformance. [CLIENT-003](research/CLIENT_MASKS.md).
docs/PROVISIONAL_TODO.md CHANGED
@@ -522,3 +522,5 @@ Research progress: [leaf references and area portal state](research/BSP_REFERENC
522
  Research progress: [area flooding and bit output](research/AREA_FLOODING.md) completes bounded RE-329–331. Broad runtime/networking milestones remain open; RE-332 tracks real-map labels, lifecycle and snapshot consumers.
523
 
524
  Research progress: [snapshot area-mask lifecycle](research/SNAPSHOT_AREAS.md) completes bounded RE-333–335. Full networking/renderer/runtime milestones remain open; RE-336 tracks real viewpoint selection, codecs, client parsing and mask consumers.
 
 
 
522
  Research progress: [area flooding and bit output](research/AREA_FLOODING.md) completes bounded RE-329–331. Broad runtime/networking milestones remain open; RE-332 tracks real-map labels, lifecycle and snapshot consumers.
523
 
524
  Research progress: [snapshot area-mask lifecycle](research/SNAPSHOT_AREAS.md) completes bounded RE-333–335. Full networking/renderer/runtime milestones remain open; RE-336 tracks real viewpoint selection, codecs, client parsing and mask consumers.
525
+
526
+ Research progress: [client area-mask parsing](research/CLIENT_MASKS.md) completes bounded RE-337–339. Broad protocol/renderer/runtime milestones remain open; RE-340 tracks actual codec/cursors, validation/publication and real packet/platform coverage.
docs/RESEARCH_LEDGER.md CHANGED
@@ -197,3 +197,7 @@ Completed bounded RE-329–331 with `analyze_area_flooding.py`: eight verified g
197
  ## 2026-10-11 — Snapshot area-mask lifecycle
198
 
199
  Completed bounded RE-333–335 with `analyze_snapshot_areas.py`: ten verified graphs, sixteen ring/reset probes, 180 native area-union/complement compositions, twelve serialization and two recursive-argument probes. SNAP-001–003/F-109–111 distinguish mask reset from complete frame initialization, supplied viewpoint union from real portal/PVS/entity selection, full-buffer complement from transmitted span, and writer requests from actual encoded packets. Helper caller-register restoration, supplied labels/cluster returns/memset effects and every stopped CALL boundary are explicit. Eight independent tests verify ring/reset/retained-length, union/complement/tail/canary, caller/recursive/serialization arguments, paired outcomes and trace boundaries. The full suite passes 278 tests; new and preceding reports regenerate byte-identically, storage and documentation checks pass. RE-332 remains open; RE-336 tracks complete snapshot/lifecycle/capacity/viewpoint/codec/client/renderer/protocol and real-map/runtime/platform coverage. No original process or native callee executes.
 
 
 
 
 
197
  ## 2026-10-11 — Snapshot area-mask lifecycle
198
 
199
  Completed bounded RE-333–335 with `analyze_snapshot_areas.py`: ten verified graphs, sixteen ring/reset probes, 180 native area-union/complement compositions, twelve serialization and two recursive-argument probes. SNAP-001–003/F-109–111 distinguish mask reset from complete frame initialization, supplied viewpoint union from real portal/PVS/entity selection, full-buffer complement from transmitted span, and writer requests from actual encoded packets. Helper caller-register restoration, supplied labels/cluster returns/memset effects and every stopped CALL boundary are explicit. Eight independent tests verify ring/reset/retained-length, union/complement/tail/canary, caller/recursive/serialization arguments, paired outcomes and trace boundaries. The full suite passes 278 tests; new and preceding reports regenerate byte-identically, storage and documentation checks pass. RE-332 remains open; RE-336 tracks complete snapshot/lifecycle/capacity/viewpoint/codec/client/renderer/protocol and real-map/runtime/platform coverage. No original process or native callee executes.
200
+
201
+ ## 2026-10-11 — Client snapshot mask parsing and reader boundaries
202
+
203
+ Completed bounded RE-337–339 with `analyze_client_masks.py`: three verified Linux client graphs, one reset-argument slice, 22 unsigned guard probes, 210 signed byte-reader probes and 53 unrolled data-reader compositions. CLIENT-001–003/F-112–114 distinguish reset requests from supplied memset effects, EOF return words from low-byte 0xff, signed cursor comparison from actual bit/Huffman cursor updates, and exact destination copying from full snapshot publication. PUSH/POP omission and stopped helper boundaries are explicit; no original process or native callee executes. Eight independent tests check reset/guard/error arguments, signed returns, copy/EOF continuation/tail/canaries/loop coverage, trace boundaries and evaluator CMOVLEL/BL/unknown-flag behavior. The full suite passes 286 tests. New and preceding spatial reports regenerate byte-identically; storage, links and whitespace checks pass. RE-336 remains open; RE-340 tracks actual codecs/cursors, bad-message checks, full validation/publication, renderer semantics and real-packet/map/original-runtime/all-platform coverage.
docs/REVERSE_ENGINEERING_TODO.md CHANGED
@@ -402,3 +402,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
402
  - [x] RE-334 Compose native viewpoint-area union and full-mask inversion, length storage/PVS requests and shared recursive arguments ([SNAP-002](research/SNAPSHOT_AREAS.md)).
403
  - [x] RE-335 Recover bounded length-byte then exact area-mask data-span message requests ([SNAP-003](research/SNAPSHOT_AREAS.md)).
404
  - [ ] RE-336 Recover full snapshot bypass/viewpoint/PVS/entity/portal and frame/client lifecycle, area/index/capacity invariants, message encoders/overflow/client parsing/renderer masks and complete delta/reliable protocol; validate real-map/original-runtime/all-platform fixtures.
 
 
 
 
 
 
402
  - [x] RE-334 Compose native viewpoint-area union and full-mask inversion, length storage/PVS requests and shared recursive arguments ([SNAP-002](research/SNAPSHOT_AREAS.md)).
403
  - [x] RE-335 Recover bounded length-byte then exact area-mask data-span message requests ([SNAP-003](research/SNAPSHOT_AREAS.md)).
404
  - [ ] RE-336 Recover full snapshot bypass/viewpoint/PVS/entity/portal and frame/client lifecycle, area/index/capacity invariants, message encoders/overflow/client parsing/renderer masks and complete delta/reliable protocol; validate real-map/original-runtime/all-platform fixtures.
405
+
406
+ - [x] RE-337 Recover Linux client snapshot reset request, both unsigned mask-length guards and destination arguments ([CLIENT-001](research/CLIENT_MASKS.md)).
407
+ - [x] RE-338 Recover bounded byte-reader low-byte/EOF returns and signed cursor comparison ([CLIENT-002](research/CLIENT_MASKS.md)).
408
+ - [x] RE-339 Compose exact unrolled mask copies with supplied bit returns/readcounts, EOF continuation and zero-tail/canary checks ([CLIENT-003](research/CLIENT_MASKS.md)).
409
+ - [ ] RE-340 Recover actual bit/Huffman codec and readcount updates, overflow/bad-message checks, full snapshot/delta/player/entity validation/publication and renderer mask interpretation; validate real packets/maps, original-runtime and all platforms before Rust/Bevy/Solari integration.
docs/research/CLIENT_MASKS.md ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Client snapshot area-mask parsing
2
+
3
+ Reproduce with `python3 scripts/analyze_client_masks.py`; validate with `python3 scripts/test_client_masks.py`. [client-masks.json](client-masks.json) retains three hash-verified graphs from the Linux ELF64 client: CL_ParseSnapshot, MSG_ReadByte and MSG_ReadData. This follows the server [mask serialization requests](SNAPSHOT_AREAS.md).
4
+
5
+ These are bounded synthetic instruction evaluations. Original processes and every native callee remain unexecuted. Frame PUSH/POP instructions are explicitly omitted, virtual stack addresses supplied, and caller registers retained across stopped helpers. MSG_ReadBits return words and post-call readcounts are supplied; neither real packet decoding nor Huffman/cursor behavior is established. Only the Linux client is sampled, not dedicated-server equivalence or other platforms.
6
+
7
+ ## CLIENT-001 — Reset, unsigned length guards and destination (RE-337)
8
+
9
+ At entry 0x4207d2, the parser requests memset of its temporary snapshot: destination virtual stack, value zero, size 0x21c (540 bytes). Three PUSH instructions are omitted for this argument probe, and the pre-subtraction stack pointer is supplied explicitly. The memset call itself does not execute. Data compositions supply its zero-fill effects, retaining outside canaries.
10
+
11
+ Two paths read the mask-length byte and test it against 32: comparisons at 0x42088f and 0x4208ed use unsigned JBE/JA respectively. Twenty-two probes accept 0,1,7,31,32 and reject 33,127,128,255,0xffffffff,0x80000000 on both paths. Rejected paths request Com_Error with error code 1, the supplied length word and the verified string `CL_ParseSnapshot: Invalid size %d for areamask.` The error function is not executed.
12
+
13
+ Both accepted paths request MSG_ReadData(message, temporary_snapshot +0x18, length) at 0x4208fc. Length zero remains accepted. The byte-reader EOF word 0xffffffff fails this unsigned guard. These offsets describe this native Linux layout, not the server's different frame layout or a Rust ABI.
14
+
15
+ ## CLIENT-002 — Byte-reader cursor boundary (RE-338)
16
+
17
+ MSG_ReadByte at 0x44dec4 requests MSG_ReadBits(message,8). Its post-call slice compares message.readcount at +0x20 against message.cursize at +0x1c using signed CMOVLEL. If readcount <= cursize, it returns the supplied result's low eight bits, zero-extended; otherwise it returns 0xffffffff. The POP is explicitly omitted; evaluation stops before RET.
18
+
19
+ Two hundred ten probes cover sizes 0,1,32,0x7fffffff,0xffffffff; cursors 0,1,2,32,33,0x80000000,0xffffffff; and six return words including 0x1234 and 0xffffffff. Equality is accepted. Negative signed cursor/size fixtures characterize the comparison locally; they do not establish legal message states. Low-byte 0xff is distinguishable from EOF 0xffffffff at this helper's return boundary.
20
+
21
+ ## CLIENT-003 — Exact requested copy and truncation (RE-339)
22
+
23
+ MSG_ReadData at 0x45307a uses a signed length <=0 early exit. For positive lengths, it requests eight bits once per destination byte through an initial byte, peeled remainder and four-byte unrolled loop. Each lane compares the supplied post-read cursor with cursize; on overflow it stores 0xff, otherwise the returned low byte. It continues requesting and storing the entire length after overflow rather than exiting at the first failed read.
24
+
25
+ Fifty-three compositions cover lengths -1,0,1 through 9,16,17,31,32 with sizes drawn from zero, one, length-minus-one and length. Supplied post-readcounts are 1 through length, and supplied result words cycle through 0,127,128,255,0x1234,0xffffffff. Negative/zero lengths issue no requests or writes. Every positive fixture has exactly length requests `(message,8)`; overflow bytes become 0xff. The unused mask tail retains the supplied reset's zeros, and frame canaries remain unchanged.
26
+
27
+ The server requests only its mask's used byte span; the client guard caps that span at 32 and targets a zero-initialized temporary buffer. This establishes compatible bounded pointer/count relationships, not a server-to-client wire round trip. Real MSG_ReadBits can use compressed bit cursors, so the supplied ordinal readcounts are explicit test inputs, not a recovered cursor update formula. Complete post-copy bad-message checks, delta validity and snapshot publication remain open.
28
+
29
+ ## Validation and next work
30
+
31
+ Eight independent tests cover reset arguments, both unsigned guards and error arguments, signed cursor/low-byte return, exact copy requests and bytes, continued EOF requests, canaries/tails/unrolled lengths, stopped CALL boundaries, and restricted evaluator CMOVLEL/BL semantics with unknown-flag rejection. Reports include source/disassembly hashes and native trace addresses or hashes. The evaluator extension adds only signed conditional move and the BL register alias.
32
+
33
+ RE-340: recover actual bit/Huffman encoding and readcount updates, message overflow/bad-read handling, all snapshot/delta/player/entity validation and publication, mask renderer interpretation and map-area validity; compose real encoded packets and original-runtime/all-platform fixtures. Implement Rust protocol/Bevy/Solari integration only against a complete validated contract.
docs/research/README.md CHANGED
@@ -123,3 +123,5 @@ The extension collector adds five focused tests for native tail calls, command-r
123
  [Area flooding, epoch guards and area-bit output](AREA_FLOODING.md) adds RE-329–331: bounded unrolled/recursive flood and portal-to-label compositions, entry revisit/epoch guards and byte OR/fill packing. Reproduce with `python3 scripts/analyze_area_flooding.py`; check with `python3 scripts/test_area_flooding.py`. Complete lifecycle/error/asymmetric/real-map/snapshot/runtime/platform contracts remain open.
124
 
125
  [Snapshot area-mask lifecycle and message arguments](SNAPSHOT_AREAS.md) adds RE-333–335: bounded ring/reset, native area union/complement and length/data request spans. Reproduce with `python3 scripts/analyze_snapshot_areas.py`; check with `python3 scripts/test_snapshot_areas.py`. Full viewpoint/entity/PVS/portal/lifecycle/codec/client/renderer/runtime/platform contracts remain open.
 
 
 
123
  [Area flooding, epoch guards and area-bit output](AREA_FLOODING.md) adds RE-329–331: bounded unrolled/recursive flood and portal-to-label compositions, entry revisit/epoch guards and byte OR/fill packing. Reproduce with `python3 scripts/analyze_area_flooding.py`; check with `python3 scripts/test_area_flooding.py`. Complete lifecycle/error/asymmetric/real-map/snapshot/runtime/platform contracts remain open.
124
 
125
  [Snapshot area-mask lifecycle and message arguments](SNAPSHOT_AREAS.md) adds RE-333–335: bounded ring/reset, native area union/complement and length/data request spans. Reproduce with `python3 scripts/analyze_snapshot_areas.py`; check with `python3 scripts/test_snapshot_areas.py`. Full viewpoint/entity/PVS/portal/lifecycle/codec/client/renderer/runtime/platform contracts remain open.
126
+
127
+ [Client snapshot area-mask parsing](CLIENT_MASKS.md) adds RE-337–339: temporary reset/length guards, byte-reader EOF and bounded unrolled data copies. Reproduce with `python3 scripts/analyze_client_masks.py`; check with `python3 scripts/test_client_masks.py`. Actual codecs/cursors, full snapshot publication and renderer/runtime/platform contracts remain open.
docs/research/SNAPSHOT_AREAS.md CHANGED
@@ -42,3 +42,5 @@ Snapshot flags, server time/delta headers, reliable commands, entity/player delt
42
  Eight independent tests check ring arithmetic and exact resets, retained length, union/complement/tail bytes, native caller arguments and length storage, supplied fill requests, order agreement, message pointer/count/payload spans, recursive argument sharing, paired outcomes and absence of executed CALL instructions in recorded traces. Full masks are retained for larger compositions with trace counts/hashes; small traces retain addresses.
43
 
44
  RE-336: compose snapshot initialization/bypasses and real point/leaf/cluster/PVS/entity/portal selection; establish valid area/index/capacity and frame/client lifecycle; recover message encoding/overflow and client snapshot parse bounds/copy behavior; connect renderer mask semantics and full snapshot/delta/reliable protocol; validate actual maps and original-runtime/all-platform fixtures. Rust/Bevy/Solari integration remains open.
 
 
 
42
  Eight independent tests check ring arithmetic and exact resets, retained length, union/complement/tail bytes, native caller arguments and length storage, supplied fill requests, order agreement, message pointer/count/payload spans, recursive argument sharing, paired outcomes and absence of executed CALL instructions in recorded traces. Full masks are retained for larger compositions with trace counts/hashes; small traces retain addresses.
43
 
44
  RE-336: compose snapshot initialization/bypasses and real point/leaf/cluster/PVS/entity/portal selection; establish valid area/index/capacity and frame/client lifecycle; recover message encoding/overflow and client snapshot parse bounds/copy behavior; connect renderer mask semantics and full snapshot/delta/reliable protocol; validate actual maps and original-runtime/all-platform fixtures. Rust/Bevy/Solari integration remains open.
45
+
46
+ Client follow-up: [client parsing](CLIENT_MASKS.md) recovers Linux reset/length guard/copy boundaries (RE-337–339). Message encoding/decoding and complete snapshot/runtime/renderer behavior remain open.
docs/research/client-masks.json ADDED
The diff for this file is too large to render. See raw diff
 
scripts/analyze_client_masks.py ADDED
@@ -0,0 +1,81 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Bounded Linux client mask guard and reader slices; bit returns are supplied."""
3
+ import copy
4
+ import gzip
5
+ import itertools
6
+ import json
7
+ from pathlib import Path
8
+ from analyze_area_queries import site
9
+ from analyze_bsp_references import trace_fields
10
+ from analyze_chat_paths import native_string
11
+ from analyze_native import elf
12
+ from analyze_collision_composition import evaluate,put_word
13
+ from analyze_engine_control import control_flow
14
+ from analyze_qvm import write_json
15
+ from analyze_vm_extensions import native_functions,verified
16
+
17
+
18
+ def run(g,start,stops,m,regs):
19
+ return evaluate(g,start,stops,m,regs,limit=10000,capture_registers=True)
20
+
21
+
22
+ def data_probe(g,length,size,frame=0x20000):
23
+ rows=g['instructions'];calls={x['address']:i for i,x in enumerate(rows) if x['opcode']=='callq'}
24
+ end=site(g,'addq','$0x8, %rsp');message=0x10000;dest=frame+24
25
+ m={frame+i:0xa5 for i in range(-1,541)}
26
+ # The CL_ParseSnapshot memset is represented by explicitly supplied effects.
27
+ for i in range(540):m[frame+i]=0
28
+ put_word(m,message+28,size);put_word(m,message+32,0)
29
+ regs={'%r15':length&0xffffffff,'%r14':dest,'%rbp':message,'%rdx':length&0xffffffff,'%rdi':message}
30
+ start=site(g,'testl','%edx, %edx');trace=[];requests=[];returns=[]
31
+ while True:
32
+ t=run(g,start,list(calls)+[end],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
33
+ if t['stop']==end:break
34
+ i=calls[t['stop']]
35
+ if '<MSG_ReadBits>' not in rows[i]['operands']:raise ValueError('unexpected helper')
36
+ requests.append([regs['rdi'],regs['rsi']]);ordinal=len(requests)
37
+ value=[0,127,128,255,0x1234,0xffffffff][(ordinal-1)%6]
38
+ returns.append(value);put_word(m,message+32,ordinal);regs['rax']=value
39
+ start=rows[i+1]['address']
40
+ if ordinal>32:raise ValueError('fixture request bound')
41
+ return {'length':length,'size':size,'supplied_bits_returns':returns,'supplied_post_readcounts':list(range(1,len(returns)+1)),
42
+ 'requests':requests,'mask_bytes':[m[dest+i] for i in range(32)],'canaries':[m[frame-1],m[frame+540]],**trace_fields({'trace':trace},True)}
43
+
44
+
45
+ def main():
46
+ label='linux--Quake3-UrT.x86_64';r=json.loads(Path(f'docs/research/native/{label}.json').read_text())
47
+ data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);metadata,_=elf(data)
48
+ a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode())
49
+ graphs={n:control_flow(fs[n]) for n in ['CL_ParseSnapshot','MSG_ReadByte','MSG_ReadData']}
50
+ parse=graphs['CL_ParseSnapshot'];rows=parse['instructions'];dest_call=next(x['address'] for x in rows if '<MSG_ReadData>' in x['operands']);error=next(x['address'] for x in rows if '<Com_Error>' in x['operands'])
51
+ starts=[x['address'] for x in rows if x['opcode']=='cmpl' and x['operands'].split('#')[0].strip()=='$0x20, %eax']
52
+ guards=[]
53
+ for start,length in itertools.product(starts,[0,1,7,31,32,33,127,128,255,0xffffffff,0x80000000]):
54
+ t=run(parse,start,[dest_call,error],{}, {'%rax':length,'%r12':0x20000,'%rbx':0x10000})
55
+ regs=t['registers'];guards.append({'start':start,'length_word':length,'accepted':t['stop']==dest_call,'arguments':[regs['rdi'],regs['rsi'],regs['rdx']],**trace_fields(t,True)})
56
+ reset_index=next(i for i,x in enumerate(rows) if '<memset@plt>' in x['operands'])
57
+ # Frame setup and pushes are excluded, but native request operands are evaluated.
58
+ reset=copy.deepcopy(parse)
59
+ omitted=[]
60
+ for x in reset['instructions'][:reset_index]:
61
+ if x['opcode']=='pushq':omitted.append(x['address']);x['opcode']='nop';x['operands']=''
62
+ t=run(reset,rows[0]['address'],[rows[reset_index]['address']],{}, {'%rsp':0x20220,'%rdi':0x10000})
63
+ byte=copy.deepcopy(graphs['MSG_ReadByte']);br=byte['instructions'];call_index=next(i for i,x in enumerate(br) if '<MSG_ReadBits>' in x['operands']);ret=next(x['address'] for x in br if x['opcode']=='retq')
64
+ byte_omitted=[]
65
+ for x in br:
66
+ if x['opcode']=='popq':byte_omitted.append(x['address']);x['opcode']='nop';x['operands']=''
67
+ byte_probes=[]
68
+ for size,cursor,value in itertools.product([0,1,32,0x7fffffff,0xffffffff],[0,1,2,32,33,0x80000000,0xffffffff],[0,127,128,255,0x1234,0xffffffff]):
69
+ m={};put_word(m,0x10000+28,size);put_word(m,0x10000+32,cursor)
70
+ b=run(byte,br[call_index+1]['address'],[ret],m,{'%rbx':0x10000,'%rax':value})
71
+ byte_probes.append({'size_word':size,'cursor_word':cursor,'supplied_bits_return':value,'result_word':b['registers']['rax'],**trace_fields(b,False)})
72
+ probes=[data_probe(graphs['MSG_ReadData'],length,size) for length in [-1,0,1,2,3,4,5,6,7,8,9,16,17,31,32] for size in sorted({0,1,max(0,length-1),max(0,length)})]
73
+ out={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,
74
+ 'reset_request':{'arguments':[t['registers'][k] for k in ['rdi','rsi','rdx']],'omitted_push_addresses':omitted,**trace_fields(t,True)},
75
+ 'error_format':native_string(data,metadata['sections'],0x569928),'guard_probes':guards,'byte_probes':byte_probes,'byte_omitted_pop_addresses':byte_omitted,'data_probes':probes,
76
+ 'limitations':'Linux client only. Original process and every native callee remain unexecuted. Frame PUSH/POP omitted explicitly; virtual stack supplied. Snapshot memset effects, MSG_ReadBits returns and post-call readcounts supplied, not encoded/decoded packets. No full CL_ParseSnapshot execution, real Huffman/bit cursor, ABI, delta/player/entity validity, publication or renderer conformance.'}
77
+ write_json(Path('docs/research/client-masks.json'),out)
78
+ print(f"Recovered {len(guards)} guards, {len(byte_probes)} byte-reader and {len(probes)} data-reader compositions")
79
+
80
+
81
+ if __name__=='__main__':main()
scripts/analyze_collision_composition.py CHANGED
@@ -14,7 +14,7 @@ from analyze_vm_extensions import native_functions,verified
14
 
15
  def register(r):
16
  aliases={'eax':('rax',32),'edi':('rdi',32),'esi':('rsi',32),'edx':('rdx',32),
17
- 'ecx':('rcx',32),'ebp':('rbp',32),'ebx':('rbx',32),'al':('rax',8),'sil':('rsi',8),'dl':('rdx',8),'cl':('rcx',8),'dil':('rdi',8),'bpl':('rbp',8)}
18
  r=r.lstrip('%')
19
  if r in aliases:return aliases[r]
20
  if re.fullmatch(r'r\d+d',r):return r[:-1],32
@@ -70,10 +70,11 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
70
  if below is None:raise ValueError('unmodeled carry flag')
71
  a,b=args.split(', ');write(b,read(b,4)-read(a,4)-int(below),4)
72
  equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
73
- elif op in ['cmovael','cmovbl']:
74
- if ge is None:raise ValueError('unmodeled unsigned flags')
 
75
  a,b=args.split(', ')
76
- if (ge if op=='cmovael' else below):write(b,read(a,4),4)
77
  elif op=='movaps':
78
  a,b=args.split(', ')
79
  if not a.startswith('%xmm') or not b.startswith('%xmm'):raise ValueError('unsupported vector move')
 
14
 
15
  def register(r):
16
  aliases={'eax':('rax',32),'edi':('rdi',32),'esi':('rsi',32),'edx':('rdx',32),
17
+ 'ecx':('rcx',32),'ebp':('rbp',32),'ebx':('rbx',32),'bl':('rbx',8),'al':('rax',8),'sil':('rsi',8),'dl':('rdx',8),'cl':('rcx',8),'dil':('rdi',8),'bpl':('rbp',8)}
18
  r=r.lstrip('%')
19
  if r in aliases:return aliases[r]
20
  if re.fullmatch(r'r\d+d',r):return r[:-1],32
 
70
  if below is None:raise ValueError('unmodeled carry flag')
71
  a,b=args.split(', ');write(b,read(b,4)-read(a,4)-int(below),4)
72
  equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
73
+ elif op in ['cmovael','cmovbl','cmovlel']:
74
+ condition=signed_le if op=='cmovlel' else ge if op=='cmovael' else below
75
+ if condition is None:raise ValueError('unmodeled conditional move flags')
76
  a,b=args.split(', ')
77
+ if condition:write(b,read(a,4),4)
78
  elif op=='movaps':
79
  a,b=args.split(', ')
80
  if not a.startswith('%xmm') or not b.startswith('%xmm'):raise ValueError('unsupported vector move')
scripts/test_client_masks.py ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Independent unsigned guards, signed cursor and exact byte-copy oracles."""
3
+ import json
4
+ from pathlib import Path
5
+ import unittest
6
+ from analyze_collision_composition import evaluate
7
+ from test_bsp_planes import graph
8
+
9
+
10
+ def signed(v):return v-(1<<32) if v&(1<<31) else v
11
+
12
+
13
+ class ClientMaskTests(unittest.TestCase):
14
+ @classmethod
15
+ def setUpClass(cls):cls.r=json.loads(Path('docs/research/client-masks.json').read_text())
16
+
17
+ def test_snapshot_reset_request(self):
18
+ p=self.r['reset_request'];self.assertEqual(p['arguments'],[0x20000,0,540]);self.assertEqual(len(p['omitted_push_addresses']),3)
19
+
20
+ def test_both_unsigned_length_guards(self):
21
+ rows=self.r['guard_probes'];self.assertEqual(len(rows),22)
22
+ self.assertEqual(len({p['start'] for p in rows}),2)
23
+ for p in rows:
24
+ self.assertEqual(p['accepted'],p['length_word']<=32)
25
+ if p['accepted']:self.assertEqual(p['arguments'],[0x10000,0x20018,p['length_word']])
26
+ else:self.assertEqual(p['arguments'],[1,0x569928,p['length_word']])
27
+
28
+ def test_signed_cursor_and_low_byte_return(self):
29
+ self.assertEqual(len(self.r['byte_probes']),210)
30
+ for p in self.r['byte_probes']:
31
+ expected=(p['supplied_bits_return']&255) if signed(p['cursor_word'])<=signed(p['size_word']) else 0xffffffff
32
+ self.assertEqual(p['result_word'],expected)
33
+
34
+ def test_data_exact_length_and_low_bytes(self):
35
+ for p in self.r['data_probes']:
36
+ count=max(0,p['length']);expected=[0]*32
37
+ for i in range(count):expected[i]=(p['supplied_bits_returns'][i]&255) if i+1<=p['size'] else 255
38
+ self.assertEqual(p['mask_bytes'],expected)
39
+ self.assertEqual(p['requests'],[[0x10000,8]]*count)
40
+ self.assertEqual(p['supplied_post_readcounts'],list(range(1,count+1)))
41
+
42
+ def test_eof_does_not_stop_remaining_requests(self):
43
+ rows=[p for p in self.r['data_probes'] if p['size']==0 and p['length']>0]
44
+ self.assertTrue(rows)
45
+ for p in rows:self.assertEqual(p['mask_bytes'],[255]*p['length']+[0]*(32-p['length']))
46
+
47
+ def test_copy_canaries_tail_and_loop_lanes(self):
48
+ lengths={p['length'] for p in self.r['data_probes']};self.assertTrue(set(range(10))<=lengths)
49
+ for p in self.r['data_probes']:
50
+ self.assertEqual(p['canaries'],[0xa5,0xa5]);self.assertEqual(p['mask_bytes'][max(0,p['length']):],[0]*(32-max(0,p['length'])))
51
+
52
+ def test_no_calls_in_evaluated_traces(self):
53
+ calls={x['address'] for g in self.r['graphs'].values() for x in g['instructions'] if x['opcode']=='callq'}
54
+ for p in self.r['guard_probes']+self.r['data_probes']+[self.r['reset_request']]:
55
+ self.assertTrue(calls.isdisjoint(p['trace']));self.assertEqual(len(p['trace']),p['trace_length'])
56
+
57
+ def test_signed_conditional_move_bl_and_unknown_flags(self):
58
+ g=graph([('cmpl','%edx, %eax'),('movl','$0xffffffff, %ebx'),('cmovlel','%ecx, %ebx'),('movb','%bl, (%rdi)'),('nop','')])
59
+ for a,b in [(0xffffffff,0),(0,0),(1,0),(0x80000000,0x7fffffff),(0x7fffffff,0x80000000)]:
60
+ t=evaluate(g,0,[4],{}, {'%rax':a,'%rdx':b,'%rcx':0x1234,'%rdi':100})
61
+ self.assertEqual(t['memory'][100],0x34 if signed(a)<=signed(b) else 255)
62
+ with self.assertRaises(ValueError):evaluate(g,2,[4],{}, {'%rbx':0,'%rcx':1,'%rdi':100})
63
+
64
+
65
+ if __name__=='__main__':unittest.main()