Analyze client snapshot mask guards and byte readers
Browse files- docs/ASSET_PIPELINE_TODO.md +2 -0
- docs/FINDINGS.md +6 -0
- docs/PROVISIONAL_TODO.md +2 -0
- docs/RESEARCH_LEDGER.md +4 -0
- docs/REVERSE_ENGINEERING_TODO.md +5 -0
- docs/research/CLIENT_MASKS.md +33 -0
- docs/research/README.md +2 -0
- docs/research/SNAPSHOT_AREAS.md +2 -0
- docs/research/client-masks.json +0 -0
- scripts/analyze_client_masks.py +81 -0
- scripts/analyze_collision_composition.py +5 -4
- scripts/test_client_masks.py +65 -0
docs/ASSET_PIPELINE_TODO.md
CHANGED
|
@@ -210,3 +210,5 @@ Reference follow-up: [leaf references and area state](research/BSP_REFERENCES.md
|
|
| 210 |
Area-state follow-up: [flooding and bit output](research/AREA_FLOODING.md) composes supplied portal matrices and generated labels, without loading actual maps. Validate area/reference ranges and lifecycle across every BSP variant before integrating conversion with runtime visibility (RE-332).
|
| 211 |
|
| 212 |
Snapshot follow-up: [area-mask lifecycle](research/SNAPSHOT_AREAS.md) connects supplied area labels to bounded native frame masks and message requests. Establish actual map area limits/reference validity before runtime conversion and renderer/network integration (RE-336).
|
|
|
|
|
|
|
|
|
| 210 |
Area-state follow-up: [flooding and bit output](research/AREA_FLOODING.md) composes supplied portal matrices and generated labels, without loading actual maps. Validate area/reference ranges and lifecycle across every BSP variant before integrating conversion with runtime visibility (RE-332).
|
| 211 |
|
| 212 |
Snapshot follow-up: [area-mask lifecycle](research/SNAPSHOT_AREAS.md) connects supplied area labels to bounded native frame masks and message requests. Establish actual map area limits/reference validity before runtime conversion and renderer/network integration (RE-336).
|
| 213 |
+
|
| 214 |
+
Client-mask follow-up: [client parsing](research/CLIENT_MASKS.md) bounds accepted area-mask length to 32 in the sampled Linux parser. Map-derived area limits, real protocol decoding and renderer mask interpretation still require validation before asset/runtime integration (RE-340).
|
docs/FINDINGS.md
CHANGED
|
@@ -360,3 +360,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
|
|
| 360 |
**F-110 — Observed snapshot mask union/complement:** supplied viewpoint area calls OR into the same frame buffer, store returned length and forward the cluster to PVS. Eight native word inversions complement all 32 bytes; a recursive call forwards the same frame/list pointers. Actual viewpoint/entity/portal/PVS selection and renderer/runtime semantics remain open. [SNAP-002](research/SNAPSHOT_AREAS.md).
|
| 361 |
|
| 362 |
**F-111 — Observed mask serialization requests:** a length-byte writer request precedes a data writer request for exactly frame.length bytes at frame +4. Unused tail bytes are excluded from that requested span; no encoded packet or original message writer executes. Full codec/client/snapshot protocol conformance remains open. [SNAP-003](research/SNAPSHOT_AREAS.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 360 |
**F-110 — Observed snapshot mask union/complement:** supplied viewpoint area calls OR into the same frame buffer, store returned length and forward the cluster to PVS. Eight native word inversions complement all 32 bytes; a recursive call forwards the same frame/list pointers. Actual viewpoint/entity/portal/PVS selection and renderer/runtime semantics remain open. [SNAP-002](research/SNAPSHOT_AREAS.md).
|
| 361 |
|
| 362 |
**F-111 — Observed mask serialization requests:** a length-byte writer request precedes a data writer request for exactly frame.length bytes at frame +4. Unused tail bytes are excluded from that requested span; no encoded packet or original message writer executes. Full codec/client/snapshot protocol conformance remains open. [SNAP-003](research/SNAPSHOT_AREAS.md).
|
| 363 |
+
|
| 364 |
+
**F-112 — Observed client mask guard:** the Linux parser requests a 540-byte temporary reset, accepts unsigned mask lengths 0–32 on both sampled paths and forwards temporary +24 to MSG_ReadData; EOF length rejects with an invalid-areamask error request. Complete parsing/publication and other platforms remain open. [CLIENT-001](research/CLIENT_MASKS.md).
|
| 365 |
+
|
| 366 |
+
**F-113 — Observed byte-reader boundary:** signed post-readcount <= cursize returns the supplied result's zero-extended low byte; larger readcounts return 0xffffffff. Malformed signed fixtures are local comparison probes, not valid runtime messages; actual bit/cursor updates remain open. [CLIENT-002](research/CLIENT_MASKS.md).
|
| 367 |
+
|
| 368 |
+
**F-114 — Observed data-copy behavior:** positive lengths request one eight-bit read per byte even after overflow, storing 0xff past supplied cursize; signed nonpositive lengths exit. Supplied snapshot reset leaves unused mask bytes zero. This is bounded native instruction composition, not encoded-packet or original-runtime conformance. [CLIENT-003](research/CLIENT_MASKS.md).
|
docs/PROVISIONAL_TODO.md
CHANGED
|
@@ -522,3 +522,5 @@ Research progress: [leaf references and area portal state](research/BSP_REFERENC
|
|
| 522 |
Research progress: [area flooding and bit output](research/AREA_FLOODING.md) completes bounded RE-329–331. Broad runtime/networking milestones remain open; RE-332 tracks real-map labels, lifecycle and snapshot consumers.
|
| 523 |
|
| 524 |
Research progress: [snapshot area-mask lifecycle](research/SNAPSHOT_AREAS.md) completes bounded RE-333–335. Full networking/renderer/runtime milestones remain open; RE-336 tracks real viewpoint selection, codecs, client parsing and mask consumers.
|
|
|
|
|
|
|
|
|
| 522 |
Research progress: [area flooding and bit output](research/AREA_FLOODING.md) completes bounded RE-329–331. Broad runtime/networking milestones remain open; RE-332 tracks real-map labels, lifecycle and snapshot consumers.
|
| 523 |
|
| 524 |
Research progress: [snapshot area-mask lifecycle](research/SNAPSHOT_AREAS.md) completes bounded RE-333–335. Full networking/renderer/runtime milestones remain open; RE-336 tracks real viewpoint selection, codecs, client parsing and mask consumers.
|
| 525 |
+
|
| 526 |
+
Research progress: [client area-mask parsing](research/CLIENT_MASKS.md) completes bounded RE-337–339. Broad protocol/renderer/runtime milestones remain open; RE-340 tracks actual codec/cursors, validation/publication and real packet/platform coverage.
|
docs/RESEARCH_LEDGER.md
CHANGED
|
@@ -197,3 +197,7 @@ Completed bounded RE-329–331 with `analyze_area_flooding.py`: eight verified g
|
|
| 197 |
## 2026-10-11 — Snapshot area-mask lifecycle
|
| 198 |
|
| 199 |
Completed bounded RE-333–335 with `analyze_snapshot_areas.py`: ten verified graphs, sixteen ring/reset probes, 180 native area-union/complement compositions, twelve serialization and two recursive-argument probes. SNAP-001–003/F-109–111 distinguish mask reset from complete frame initialization, supplied viewpoint union from real portal/PVS/entity selection, full-buffer complement from transmitted span, and writer requests from actual encoded packets. Helper caller-register restoration, supplied labels/cluster returns/memset effects and every stopped CALL boundary are explicit. Eight independent tests verify ring/reset/retained-length, union/complement/tail/canary, caller/recursive/serialization arguments, paired outcomes and trace boundaries. The full suite passes 278 tests; new and preceding reports regenerate byte-identically, storage and documentation checks pass. RE-332 remains open; RE-336 tracks complete snapshot/lifecycle/capacity/viewpoint/codec/client/renderer/protocol and real-map/runtime/platform coverage. No original process or native callee executes.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 197 |
## 2026-10-11 — Snapshot area-mask lifecycle
|
| 198 |
|
| 199 |
Completed bounded RE-333–335 with `analyze_snapshot_areas.py`: ten verified graphs, sixteen ring/reset probes, 180 native area-union/complement compositions, twelve serialization and two recursive-argument probes. SNAP-001–003/F-109–111 distinguish mask reset from complete frame initialization, supplied viewpoint union from real portal/PVS/entity selection, full-buffer complement from transmitted span, and writer requests from actual encoded packets. Helper caller-register restoration, supplied labels/cluster returns/memset effects and every stopped CALL boundary are explicit. Eight independent tests verify ring/reset/retained-length, union/complement/tail/canary, caller/recursive/serialization arguments, paired outcomes and trace boundaries. The full suite passes 278 tests; new and preceding reports regenerate byte-identically, storage and documentation checks pass. RE-332 remains open; RE-336 tracks complete snapshot/lifecycle/capacity/viewpoint/codec/client/renderer/protocol and real-map/runtime/platform coverage. No original process or native callee executes.
|
| 200 |
+
|
| 201 |
+
## 2026-10-11 — Client snapshot mask parsing and reader boundaries
|
| 202 |
+
|
| 203 |
+
Completed bounded RE-337–339 with `analyze_client_masks.py`: three verified Linux client graphs, one reset-argument slice, 22 unsigned guard probes, 210 signed byte-reader probes and 53 unrolled data-reader compositions. CLIENT-001–003/F-112–114 distinguish reset requests from supplied memset effects, EOF return words from low-byte 0xff, signed cursor comparison from actual bit/Huffman cursor updates, and exact destination copying from full snapshot publication. PUSH/POP omission and stopped helper boundaries are explicit; no original process or native callee executes. Eight independent tests check reset/guard/error arguments, signed returns, copy/EOF continuation/tail/canaries/loop coverage, trace boundaries and evaluator CMOVLEL/BL/unknown-flag behavior. The full suite passes 286 tests. New and preceding spatial reports regenerate byte-identically; storage, links and whitespace checks pass. RE-336 remains open; RE-340 tracks actual codecs/cursors, bad-message checks, full validation/publication, renderer semantics and real-packet/map/original-runtime/all-platform coverage.
|
docs/REVERSE_ENGINEERING_TODO.md
CHANGED
|
@@ -402,3 +402,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
|
|
| 402 |
- [x] RE-334 Compose native viewpoint-area union and full-mask inversion, length storage/PVS requests and shared recursive arguments ([SNAP-002](research/SNAPSHOT_AREAS.md)).
|
| 403 |
- [x] RE-335 Recover bounded length-byte then exact area-mask data-span message requests ([SNAP-003](research/SNAPSHOT_AREAS.md)).
|
| 404 |
- [ ] RE-336 Recover full snapshot bypass/viewpoint/PVS/entity/portal and frame/client lifecycle, area/index/capacity invariants, message encoders/overflow/client parsing/renderer masks and complete delta/reliable protocol; validate real-map/original-runtime/all-platform fixtures.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 402 |
- [x] RE-334 Compose native viewpoint-area union and full-mask inversion, length storage/PVS requests and shared recursive arguments ([SNAP-002](research/SNAPSHOT_AREAS.md)).
|
| 403 |
- [x] RE-335 Recover bounded length-byte then exact area-mask data-span message requests ([SNAP-003](research/SNAPSHOT_AREAS.md)).
|
| 404 |
- [ ] RE-336 Recover full snapshot bypass/viewpoint/PVS/entity/portal and frame/client lifecycle, area/index/capacity invariants, message encoders/overflow/client parsing/renderer masks and complete delta/reliable protocol; validate real-map/original-runtime/all-platform fixtures.
|
| 405 |
+
|
| 406 |
+
- [x] RE-337 Recover Linux client snapshot reset request, both unsigned mask-length guards and destination arguments ([CLIENT-001](research/CLIENT_MASKS.md)).
|
| 407 |
+
- [x] RE-338 Recover bounded byte-reader low-byte/EOF returns and signed cursor comparison ([CLIENT-002](research/CLIENT_MASKS.md)).
|
| 408 |
+
- [x] RE-339 Compose exact unrolled mask copies with supplied bit returns/readcounts, EOF continuation and zero-tail/canary checks ([CLIENT-003](research/CLIENT_MASKS.md)).
|
| 409 |
+
- [ ] RE-340 Recover actual bit/Huffman codec and readcount updates, overflow/bad-message checks, full snapshot/delta/player/entity validation/publication and renderer mask interpretation; validate real packets/maps, original-runtime and all platforms before Rust/Bevy/Solari integration.
|
docs/research/CLIENT_MASKS.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Client snapshot area-mask parsing
|
| 2 |
+
|
| 3 |
+
Reproduce with `python3 scripts/analyze_client_masks.py`; validate with `python3 scripts/test_client_masks.py`. [client-masks.json](client-masks.json) retains three hash-verified graphs from the Linux ELF64 client: CL_ParseSnapshot, MSG_ReadByte and MSG_ReadData. This follows the server [mask serialization requests](SNAPSHOT_AREAS.md).
|
| 4 |
+
|
| 5 |
+
These are bounded synthetic instruction evaluations. Original processes and every native callee remain unexecuted. Frame PUSH/POP instructions are explicitly omitted, virtual stack addresses supplied, and caller registers retained across stopped helpers. MSG_ReadBits return words and post-call readcounts are supplied; neither real packet decoding nor Huffman/cursor behavior is established. Only the Linux client is sampled, not dedicated-server equivalence or other platforms.
|
| 6 |
+
|
| 7 |
+
## CLIENT-001 — Reset, unsigned length guards and destination (RE-337)
|
| 8 |
+
|
| 9 |
+
At entry 0x4207d2, the parser requests memset of its temporary snapshot: destination virtual stack, value zero, size 0x21c (540 bytes). Three PUSH instructions are omitted for this argument probe, and the pre-subtraction stack pointer is supplied explicitly. The memset call itself does not execute. Data compositions supply its zero-fill effects, retaining outside canaries.
|
| 10 |
+
|
| 11 |
+
Two paths read the mask-length byte and test it against 32: comparisons at 0x42088f and 0x4208ed use unsigned JBE/JA respectively. Twenty-two probes accept 0,1,7,31,32 and reject 33,127,128,255,0xffffffff,0x80000000 on both paths. Rejected paths request Com_Error with error code 1, the supplied length word and the verified string `CL_ParseSnapshot: Invalid size %d for areamask.` The error function is not executed.
|
| 12 |
+
|
| 13 |
+
Both accepted paths request MSG_ReadData(message, temporary_snapshot +0x18, length) at 0x4208fc. Length zero remains accepted. The byte-reader EOF word 0xffffffff fails this unsigned guard. These offsets describe this native Linux layout, not the server's different frame layout or a Rust ABI.
|
| 14 |
+
|
| 15 |
+
## CLIENT-002 — Byte-reader cursor boundary (RE-338)
|
| 16 |
+
|
| 17 |
+
MSG_ReadByte at 0x44dec4 requests MSG_ReadBits(message,8). Its post-call slice compares message.readcount at +0x20 against message.cursize at +0x1c using signed CMOVLEL. If readcount <= cursize, it returns the supplied result's low eight bits, zero-extended; otherwise it returns 0xffffffff. The POP is explicitly omitted; evaluation stops before RET.
|
| 18 |
+
|
| 19 |
+
Two hundred ten probes cover sizes 0,1,32,0x7fffffff,0xffffffff; cursors 0,1,2,32,33,0x80000000,0xffffffff; and six return words including 0x1234 and 0xffffffff. Equality is accepted. Negative signed cursor/size fixtures characterize the comparison locally; they do not establish legal message states. Low-byte 0xff is distinguishable from EOF 0xffffffff at this helper's return boundary.
|
| 20 |
+
|
| 21 |
+
## CLIENT-003 — Exact requested copy and truncation (RE-339)
|
| 22 |
+
|
| 23 |
+
MSG_ReadData at 0x45307a uses a signed length <=0 early exit. For positive lengths, it requests eight bits once per destination byte through an initial byte, peeled remainder and four-byte unrolled loop. Each lane compares the supplied post-read cursor with cursize; on overflow it stores 0xff, otherwise the returned low byte. It continues requesting and storing the entire length after overflow rather than exiting at the first failed read.
|
| 24 |
+
|
| 25 |
+
Fifty-three compositions cover lengths -1,0,1 through 9,16,17,31,32 with sizes drawn from zero, one, length-minus-one and length. Supplied post-readcounts are 1 through length, and supplied result words cycle through 0,127,128,255,0x1234,0xffffffff. Negative/zero lengths issue no requests or writes. Every positive fixture has exactly length requests `(message,8)`; overflow bytes become 0xff. The unused mask tail retains the supplied reset's zeros, and frame canaries remain unchanged.
|
| 26 |
+
|
| 27 |
+
The server requests only its mask's used byte span; the client guard caps that span at 32 and targets a zero-initialized temporary buffer. This establishes compatible bounded pointer/count relationships, not a server-to-client wire round trip. Real MSG_ReadBits can use compressed bit cursors, so the supplied ordinal readcounts are explicit test inputs, not a recovered cursor update formula. Complete post-copy bad-message checks, delta validity and snapshot publication remain open.
|
| 28 |
+
|
| 29 |
+
## Validation and next work
|
| 30 |
+
|
| 31 |
+
Eight independent tests cover reset arguments, both unsigned guards and error arguments, signed cursor/low-byte return, exact copy requests and bytes, continued EOF requests, canaries/tails/unrolled lengths, stopped CALL boundaries, and restricted evaluator CMOVLEL/BL semantics with unknown-flag rejection. Reports include source/disassembly hashes and native trace addresses or hashes. The evaluator extension adds only signed conditional move and the BL register alias.
|
| 32 |
+
|
| 33 |
+
RE-340: recover actual bit/Huffman encoding and readcount updates, message overflow/bad-read handling, all snapshot/delta/player/entity validation and publication, mask renderer interpretation and map-area validity; compose real encoded packets and original-runtime/all-platform fixtures. Implement Rust protocol/Bevy/Solari integration only against a complete validated contract.
|
docs/research/README.md
CHANGED
|
@@ -123,3 +123,5 @@ The extension collector adds five focused tests for native tail calls, command-r
|
|
| 123 |
[Area flooding, epoch guards and area-bit output](AREA_FLOODING.md) adds RE-329–331: bounded unrolled/recursive flood and portal-to-label compositions, entry revisit/epoch guards and byte OR/fill packing. Reproduce with `python3 scripts/analyze_area_flooding.py`; check with `python3 scripts/test_area_flooding.py`. Complete lifecycle/error/asymmetric/real-map/snapshot/runtime/platform contracts remain open.
|
| 124 |
|
| 125 |
[Snapshot area-mask lifecycle and message arguments](SNAPSHOT_AREAS.md) adds RE-333–335: bounded ring/reset, native area union/complement and length/data request spans. Reproduce with `python3 scripts/analyze_snapshot_areas.py`; check with `python3 scripts/test_snapshot_areas.py`. Full viewpoint/entity/PVS/portal/lifecycle/codec/client/renderer/runtime/platform contracts remain open.
|
|
|
|
|
|
|
|
|
| 123 |
[Area flooding, epoch guards and area-bit output](AREA_FLOODING.md) adds RE-329–331: bounded unrolled/recursive flood and portal-to-label compositions, entry revisit/epoch guards and byte OR/fill packing. Reproduce with `python3 scripts/analyze_area_flooding.py`; check with `python3 scripts/test_area_flooding.py`. Complete lifecycle/error/asymmetric/real-map/snapshot/runtime/platform contracts remain open.
|
| 124 |
|
| 125 |
[Snapshot area-mask lifecycle and message arguments](SNAPSHOT_AREAS.md) adds RE-333–335: bounded ring/reset, native area union/complement and length/data request spans. Reproduce with `python3 scripts/analyze_snapshot_areas.py`; check with `python3 scripts/test_snapshot_areas.py`. Full viewpoint/entity/PVS/portal/lifecycle/codec/client/renderer/runtime/platform contracts remain open.
|
| 126 |
+
|
| 127 |
+
[Client snapshot area-mask parsing](CLIENT_MASKS.md) adds RE-337–339: temporary reset/length guards, byte-reader EOF and bounded unrolled data copies. Reproduce with `python3 scripts/analyze_client_masks.py`; check with `python3 scripts/test_client_masks.py`. Actual codecs/cursors, full snapshot publication and renderer/runtime/platform contracts remain open.
|
docs/research/SNAPSHOT_AREAS.md
CHANGED
|
@@ -42,3 +42,5 @@ Snapshot flags, server time/delta headers, reliable commands, entity/player delt
|
|
| 42 |
Eight independent tests check ring arithmetic and exact resets, retained length, union/complement/tail bytes, native caller arguments and length storage, supplied fill requests, order agreement, message pointer/count/payload spans, recursive argument sharing, paired outcomes and absence of executed CALL instructions in recorded traces. Full masks are retained for larger compositions with trace counts/hashes; small traces retain addresses.
|
| 43 |
|
| 44 |
RE-336: compose snapshot initialization/bypasses and real point/leaf/cluster/PVS/entity/portal selection; establish valid area/index/capacity and frame/client lifecycle; recover message encoding/overflow and client snapshot parse bounds/copy behavior; connect renderer mask semantics and full snapshot/delta/reliable protocol; validate actual maps and original-runtime/all-platform fixtures. Rust/Bevy/Solari integration remains open.
|
|
|
|
|
|
|
|
|
| 42 |
Eight independent tests check ring arithmetic and exact resets, retained length, union/complement/tail bytes, native caller arguments and length storage, supplied fill requests, order agreement, message pointer/count/payload spans, recursive argument sharing, paired outcomes and absence of executed CALL instructions in recorded traces. Full masks are retained for larger compositions with trace counts/hashes; small traces retain addresses.
|
| 43 |
|
| 44 |
RE-336: compose snapshot initialization/bypasses and real point/leaf/cluster/PVS/entity/portal selection; establish valid area/index/capacity and frame/client lifecycle; recover message encoding/overflow and client snapshot parse bounds/copy behavior; connect renderer mask semantics and full snapshot/delta/reliable protocol; validate actual maps and original-runtime/all-platform fixtures. Rust/Bevy/Solari integration remains open.
|
| 45 |
+
|
| 46 |
+
Client follow-up: [client parsing](CLIENT_MASKS.md) recovers Linux reset/length guard/copy boundaries (RE-337–339). Message encoding/decoding and complete snapshot/runtime/renderer behavior remain open.
|
docs/research/client-masks.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
scripts/analyze_client_masks.py
ADDED
|
@@ -0,0 +1,81 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Bounded Linux client mask guard and reader slices; bit returns are supplied."""
|
| 3 |
+
import copy
|
| 4 |
+
import gzip
|
| 5 |
+
import itertools
|
| 6 |
+
import json
|
| 7 |
+
from pathlib import Path
|
| 8 |
+
from analyze_area_queries import site
|
| 9 |
+
from analyze_bsp_references import trace_fields
|
| 10 |
+
from analyze_chat_paths import native_string
|
| 11 |
+
from analyze_native import elf
|
| 12 |
+
from analyze_collision_composition import evaluate,put_word
|
| 13 |
+
from analyze_engine_control import control_flow
|
| 14 |
+
from analyze_qvm import write_json
|
| 15 |
+
from analyze_vm_extensions import native_functions,verified
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def run(g,start,stops,m,regs):
|
| 19 |
+
return evaluate(g,start,stops,m,regs,limit=10000,capture_registers=True)
|
| 20 |
+
|
| 21 |
+
|
| 22 |
+
def data_probe(g,length,size,frame=0x20000):
|
| 23 |
+
rows=g['instructions'];calls={x['address']:i for i,x in enumerate(rows) if x['opcode']=='callq'}
|
| 24 |
+
end=site(g,'addq','$0x8, %rsp');message=0x10000;dest=frame+24
|
| 25 |
+
m={frame+i:0xa5 for i in range(-1,541)}
|
| 26 |
+
# The CL_ParseSnapshot memset is represented by explicitly supplied effects.
|
| 27 |
+
for i in range(540):m[frame+i]=0
|
| 28 |
+
put_word(m,message+28,size);put_word(m,message+32,0)
|
| 29 |
+
regs={'%r15':length&0xffffffff,'%r14':dest,'%rbp':message,'%rdx':length&0xffffffff,'%rdi':message}
|
| 30 |
+
start=site(g,'testl','%edx, %edx');trace=[];requests=[];returns=[]
|
| 31 |
+
while True:
|
| 32 |
+
t=run(g,start,list(calls)+[end],m,regs);trace+=t['trace'];m=t['memory'];regs=t['registers']
|
| 33 |
+
if t['stop']==end:break
|
| 34 |
+
i=calls[t['stop']]
|
| 35 |
+
if '<MSG_ReadBits>' not in rows[i]['operands']:raise ValueError('unexpected helper')
|
| 36 |
+
requests.append([regs['rdi'],regs['rsi']]);ordinal=len(requests)
|
| 37 |
+
value=[0,127,128,255,0x1234,0xffffffff][(ordinal-1)%6]
|
| 38 |
+
returns.append(value);put_word(m,message+32,ordinal);regs['rax']=value
|
| 39 |
+
start=rows[i+1]['address']
|
| 40 |
+
if ordinal>32:raise ValueError('fixture request bound')
|
| 41 |
+
return {'length':length,'size':size,'supplied_bits_returns':returns,'supplied_post_readcounts':list(range(1,len(returns)+1)),
|
| 42 |
+
'requests':requests,'mask_bytes':[m[dest+i] for i in range(32)],'canaries':[m[frame-1],m[frame+540]],**trace_fields({'trace':trace},True)}
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def main():
|
| 46 |
+
label='linux--Quake3-UrT.x86_64';r=json.loads(Path(f'docs/research/native/{label}.json').read_text())
|
| 47 |
+
data=verified((Path('.cache/UrbanTerror43')/r['source']['path']).read_bytes(),r['source']['sha256']);metadata,_=elf(data)
|
| 48 |
+
a=r['artifacts']['disassembly.txt'];fs=native_functions(verified(gzip.open(a['path'],'rb').read(),a['uncompressed_sha256']).decode())
|
| 49 |
+
graphs={n:control_flow(fs[n]) for n in ['CL_ParseSnapshot','MSG_ReadByte','MSG_ReadData']}
|
| 50 |
+
parse=graphs['CL_ParseSnapshot'];rows=parse['instructions'];dest_call=next(x['address'] for x in rows if '<MSG_ReadData>' in x['operands']);error=next(x['address'] for x in rows if '<Com_Error>' in x['operands'])
|
| 51 |
+
starts=[x['address'] for x in rows if x['opcode']=='cmpl' and x['operands'].split('#')[0].strip()=='$0x20, %eax']
|
| 52 |
+
guards=[]
|
| 53 |
+
for start,length in itertools.product(starts,[0,1,7,31,32,33,127,128,255,0xffffffff,0x80000000]):
|
| 54 |
+
t=run(parse,start,[dest_call,error],{}, {'%rax':length,'%r12':0x20000,'%rbx':0x10000})
|
| 55 |
+
regs=t['registers'];guards.append({'start':start,'length_word':length,'accepted':t['stop']==dest_call,'arguments':[regs['rdi'],regs['rsi'],regs['rdx']],**trace_fields(t,True)})
|
| 56 |
+
reset_index=next(i for i,x in enumerate(rows) if '<memset@plt>' in x['operands'])
|
| 57 |
+
# Frame setup and pushes are excluded, but native request operands are evaluated.
|
| 58 |
+
reset=copy.deepcopy(parse)
|
| 59 |
+
omitted=[]
|
| 60 |
+
for x in reset['instructions'][:reset_index]:
|
| 61 |
+
if x['opcode']=='pushq':omitted.append(x['address']);x['opcode']='nop';x['operands']=''
|
| 62 |
+
t=run(reset,rows[0]['address'],[rows[reset_index]['address']],{}, {'%rsp':0x20220,'%rdi':0x10000})
|
| 63 |
+
byte=copy.deepcopy(graphs['MSG_ReadByte']);br=byte['instructions'];call_index=next(i for i,x in enumerate(br) if '<MSG_ReadBits>' in x['operands']);ret=next(x['address'] for x in br if x['opcode']=='retq')
|
| 64 |
+
byte_omitted=[]
|
| 65 |
+
for x in br:
|
| 66 |
+
if x['opcode']=='popq':byte_omitted.append(x['address']);x['opcode']='nop';x['operands']=''
|
| 67 |
+
byte_probes=[]
|
| 68 |
+
for size,cursor,value in itertools.product([0,1,32,0x7fffffff,0xffffffff],[0,1,2,32,33,0x80000000,0xffffffff],[0,127,128,255,0x1234,0xffffffff]):
|
| 69 |
+
m={};put_word(m,0x10000+28,size);put_word(m,0x10000+32,cursor)
|
| 70 |
+
b=run(byte,br[call_index+1]['address'],[ret],m,{'%rbx':0x10000,'%rax':value})
|
| 71 |
+
byte_probes.append({'size_word':size,'cursor_word':cursor,'supplied_bits_return':value,'result_word':b['registers']['rax'],**trace_fields(b,False)})
|
| 72 |
+
probes=[data_probe(graphs['MSG_ReadData'],length,size) for length in [-1,0,1,2,3,4,5,6,7,8,9,16,17,31,32] for size in sorted({0,1,max(0,length-1),max(0,length)})]
|
| 73 |
+
out={'source':r['source'],'disassembly_sha256':a['uncompressed_sha256'],'graphs':graphs,
|
| 74 |
+
'reset_request':{'arguments':[t['registers'][k] for k in ['rdi','rsi','rdx']],'omitted_push_addresses':omitted,**trace_fields(t,True)},
|
| 75 |
+
'error_format':native_string(data,metadata['sections'],0x569928),'guard_probes':guards,'byte_probes':byte_probes,'byte_omitted_pop_addresses':byte_omitted,'data_probes':probes,
|
| 76 |
+
'limitations':'Linux client only. Original process and every native callee remain unexecuted. Frame PUSH/POP omitted explicitly; virtual stack supplied. Snapshot memset effects, MSG_ReadBits returns and post-call readcounts supplied, not encoded/decoded packets. No full CL_ParseSnapshot execution, real Huffman/bit cursor, ABI, delta/player/entity validity, publication or renderer conformance.'}
|
| 77 |
+
write_json(Path('docs/research/client-masks.json'),out)
|
| 78 |
+
print(f"Recovered {len(guards)} guards, {len(byte_probes)} byte-reader and {len(probes)} data-reader compositions")
|
| 79 |
+
|
| 80 |
+
|
| 81 |
+
if __name__=='__main__':main()
|
scripts/analyze_collision_composition.py
CHANGED
|
@@ -14,7 +14,7 @@ from analyze_vm_extensions import native_functions,verified
|
|
| 14 |
|
| 15 |
def register(r):
|
| 16 |
aliases={'eax':('rax',32),'edi':('rdi',32),'esi':('rsi',32),'edx':('rdx',32),
|
| 17 |
-
'ecx':('rcx',32),'ebp':('rbp',32),'ebx':('rbx',32),'al':('rax',8),'sil':('rsi',8),'dl':('rdx',8),'cl':('rcx',8),'dil':('rdi',8),'bpl':('rbp',8)}
|
| 18 |
r=r.lstrip('%')
|
| 19 |
if r in aliases:return aliases[r]
|
| 20 |
if re.fullmatch(r'r\d+d',r):return r[:-1],32
|
|
@@ -70,10 +70,11 @@ def evaluate(graph,start,stops,memory,registers,limit=128,capture_registers=Fals
|
|
| 70 |
if below is None:raise ValueError('unmodeled carry flag')
|
| 71 |
a,b=args.split(', ');write(b,read(b,4)-read(a,4)-int(below),4)
|
| 72 |
equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
|
| 73 |
-
elif op in ['cmovael','cmovbl']:
|
| 74 |
-
if
|
|
|
|
| 75 |
a,b=args.split(', ')
|
| 76 |
-
if
|
| 77 |
elif op=='movaps':
|
| 78 |
a,b=args.split(', ')
|
| 79 |
if not a.startswith('%xmm') or not b.startswith('%xmm'):raise ValueError('unsupported vector move')
|
|
|
|
| 14 |
|
| 15 |
def register(r):
|
| 16 |
aliases={'eax':('rax',32),'edi':('rdi',32),'esi':('rsi',32),'edx':('rdx',32),
|
| 17 |
+
'ecx':('rcx',32),'ebp':('rbp',32),'ebx':('rbx',32),'bl':('rbx',8),'al':('rax',8),'sil':('rsi',8),'dl':('rdx',8),'cl':('rcx',8),'dil':('rdi',8),'bpl':('rbp',8)}
|
| 18 |
r=r.lstrip('%')
|
| 19 |
if r in aliases:return aliases[r]
|
| 20 |
if re.fullmatch(r'r\d+d',r):return r[:-1],32
|
|
|
|
| 70 |
if below is None:raise ValueError('unmodeled carry flag')
|
| 71 |
a,b=args.split(', ');write(b,read(b,4)-read(a,4)-int(below),4)
|
| 72 |
equal=None;signed_le=signed_lt=negative=None;ge=greater=below=None
|
| 73 |
+
elif op in ['cmovael','cmovbl','cmovlel']:
|
| 74 |
+
condition=signed_le if op=='cmovlel' else ge if op=='cmovael' else below
|
| 75 |
+
if condition is None:raise ValueError('unmodeled conditional move flags')
|
| 76 |
a,b=args.split(', ')
|
| 77 |
+
if condition:write(b,read(a,4),4)
|
| 78 |
elif op=='movaps':
|
| 79 |
a,b=args.split(', ')
|
| 80 |
if not a.startswith('%xmm') or not b.startswith('%xmm'):raise ValueError('unsupported vector move')
|
scripts/test_client_masks.py
ADDED
|
@@ -0,0 +1,65 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Independent unsigned guards, signed cursor and exact byte-copy oracles."""
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import unittest
|
| 6 |
+
from analyze_collision_composition import evaluate
|
| 7 |
+
from test_bsp_planes import graph
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def signed(v):return v-(1<<32) if v&(1<<31) else v
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
class ClientMaskTests(unittest.TestCase):
|
| 14 |
+
@classmethod
|
| 15 |
+
def setUpClass(cls):cls.r=json.loads(Path('docs/research/client-masks.json').read_text())
|
| 16 |
+
|
| 17 |
+
def test_snapshot_reset_request(self):
|
| 18 |
+
p=self.r['reset_request'];self.assertEqual(p['arguments'],[0x20000,0,540]);self.assertEqual(len(p['omitted_push_addresses']),3)
|
| 19 |
+
|
| 20 |
+
def test_both_unsigned_length_guards(self):
|
| 21 |
+
rows=self.r['guard_probes'];self.assertEqual(len(rows),22)
|
| 22 |
+
self.assertEqual(len({p['start'] for p in rows}),2)
|
| 23 |
+
for p in rows:
|
| 24 |
+
self.assertEqual(p['accepted'],p['length_word']<=32)
|
| 25 |
+
if p['accepted']:self.assertEqual(p['arguments'],[0x10000,0x20018,p['length_word']])
|
| 26 |
+
else:self.assertEqual(p['arguments'],[1,0x569928,p['length_word']])
|
| 27 |
+
|
| 28 |
+
def test_signed_cursor_and_low_byte_return(self):
|
| 29 |
+
self.assertEqual(len(self.r['byte_probes']),210)
|
| 30 |
+
for p in self.r['byte_probes']:
|
| 31 |
+
expected=(p['supplied_bits_return']&255) if signed(p['cursor_word'])<=signed(p['size_word']) else 0xffffffff
|
| 32 |
+
self.assertEqual(p['result_word'],expected)
|
| 33 |
+
|
| 34 |
+
def test_data_exact_length_and_low_bytes(self):
|
| 35 |
+
for p in self.r['data_probes']:
|
| 36 |
+
count=max(0,p['length']);expected=[0]*32
|
| 37 |
+
for i in range(count):expected[i]=(p['supplied_bits_returns'][i]&255) if i+1<=p['size'] else 255
|
| 38 |
+
self.assertEqual(p['mask_bytes'],expected)
|
| 39 |
+
self.assertEqual(p['requests'],[[0x10000,8]]*count)
|
| 40 |
+
self.assertEqual(p['supplied_post_readcounts'],list(range(1,count+1)))
|
| 41 |
+
|
| 42 |
+
def test_eof_does_not_stop_remaining_requests(self):
|
| 43 |
+
rows=[p for p in self.r['data_probes'] if p['size']==0 and p['length']>0]
|
| 44 |
+
self.assertTrue(rows)
|
| 45 |
+
for p in rows:self.assertEqual(p['mask_bytes'],[255]*p['length']+[0]*(32-p['length']))
|
| 46 |
+
|
| 47 |
+
def test_copy_canaries_tail_and_loop_lanes(self):
|
| 48 |
+
lengths={p['length'] for p in self.r['data_probes']};self.assertTrue(set(range(10))<=lengths)
|
| 49 |
+
for p in self.r['data_probes']:
|
| 50 |
+
self.assertEqual(p['canaries'],[0xa5,0xa5]);self.assertEqual(p['mask_bytes'][max(0,p['length']):],[0]*(32-max(0,p['length'])))
|
| 51 |
+
|
| 52 |
+
def test_no_calls_in_evaluated_traces(self):
|
| 53 |
+
calls={x['address'] for g in self.r['graphs'].values() for x in g['instructions'] if x['opcode']=='callq'}
|
| 54 |
+
for p in self.r['guard_probes']+self.r['data_probes']+[self.r['reset_request']]:
|
| 55 |
+
self.assertTrue(calls.isdisjoint(p['trace']));self.assertEqual(len(p['trace']),p['trace_length'])
|
| 56 |
+
|
| 57 |
+
def test_signed_conditional_move_bl_and_unknown_flags(self):
|
| 58 |
+
g=graph([('cmpl','%edx, %eax'),('movl','$0xffffffff, %ebx'),('cmovlel','%ecx, %ebx'),('movb','%bl, (%rdi)'),('nop','')])
|
| 59 |
+
for a,b in [(0xffffffff,0),(0,0),(1,0),(0x80000000,0x7fffffff),(0x7fffffff,0x80000000)]:
|
| 60 |
+
t=evaluate(g,0,[4],{}, {'%rax':a,'%rdx':b,'%rcx':0x1234,'%rdi':100})
|
| 61 |
+
self.assertEqual(t['memory'][100],0x34 if signed(a)<=signed(b) else 255)
|
| 62 |
+
with self.assertRaises(ValueError):evaluate(g,2,[4],{}, {'%rbx':0,'%rcx':1,'%rdi':100})
|
| 63 |
+
|
| 64 |
+
|
| 65 |
+
if __name__=='__main__':unittest.main()
|