nmarshall commited on
Commit
a68f9ad
·
1 Parent(s): fd99160

Trace VM extension commands and Linux network authentication services

Browse files
docs/FINDINGS.md CHANGED
@@ -175,3 +175,8 @@ Exact native fork/source revision, runtime resource lookup, legacy launch enviro
175
  **F-026 — Observed absence within a bounded scan.** No packaged DLL or native-image prefix occurs in the extracted members. External Windows DLL imports and additional dynamic-library string candidates exist. Interior native-magic candidates fail the supported plain-image parsers; compressed/encrypted/headerless embedded implementations are not ruled out. [NAT-001/002](research/native/README.md).
176
 
177
  **F-027 — Unknown: remaining binary semantics.** The reports retain 519 unresolved QVM call sites and provisional function/string/control-flow candidates. Most native functions remain unreviewed; extra entry commands, argument structures, library patches, loader behavior, resources and cross-platform ABI parity remain explicit tasks. Static decoding has not established gameplay rules or Rust replacement behavior. [Completion and next steps](research/README.md).
 
 
 
 
 
 
175
  **F-026 — Observed absence within a bounded scan.** No packaged DLL or native-image prefix occurs in the extracted members. External Windows DLL imports and additional dynamic-library string candidates exist. Interior native-magic candidates fail the supported plain-image parsers; compressed/encrypted/headerless embedded implementations are not ruled out. [NAT-001/002](research/native/README.md).
176
 
177
  **F-027 — Unknown: remaining binary semantics.** The reports retain 519 unresolved QVM call sites and provisional function/string/control-flow candidates. Most native functions remain unreviewed; extra entry commands, argument structures, library patches, loader behavior, resources and cross-platform ABI parity remain explicit tasks. Static decoding has not established gameplay rules or Rust replacement behavior. [Completion and next steps](research/README.md).
178
+
179
+
180
+ **F-028 — Observed/inferred: four extension entry roles.** Matching QVM bodies, string references and Linux ELF64 native callers identify qagame 11 as heartbeat, 12 as shutdown, 13 as server authentication message handling, and UI 11 as authentication response handling. These are descriptive roles, not recovered source names or complete protocol specifications. All four QVM entry cases discard the callee result and return zero. [ABI-001 evidence and limits](research/VM_EXTENSIONS.md).
181
+
182
+ **F-029 — Observed: Linux ELF64 network/auth extension boundaries.** Game 600/UI 88 parse a string into a VM address buffer; game 601/UI 90 pass socket, length, packet buffer and resolved destination to `NET_SendPacket`, then return zero without checking the parser result in the dispatch case. Game 603 converts two pointers and checks the client index before calling `SV_Auth_DropClient`. The parser/send boundary uses a 12-byte native address, with IPv4 at offset 4 and port at offset 8. Other platform contracts and full error/packet semantics remain unresolved. [ABI-002/003](research/VM_EXTENSIONS.md).
docs/RESEARCH_LEDGER.md CHANGED
@@ -68,3 +68,8 @@ These are proposed future paths, not claims that the files already exist:
68
  For each checked task, record task ID, produced file/tool/specification, associated finding IDs, verification command or experiment, result, and remaining limitations. Do not check a behavior task because a guessed constant was implemented. Do not check a conversion task while required assets have unreported fallbacks. New evidence that contradicts an existing finding reopens affected tasks and records the correction.
69
 
70
  Document inspection tooling used in this pass: Python standard library (`pathlib`, `zipfile`, `struct`, `hashlib`, `csv`, `collections`, regex), including structural ELF/PE import-table reads, and macOS `file`, `strings`, and `otool -L`. One-off inspection generated the saved inventories; that original planning pass did not yet include reusable scripts. The 2026-10-11 extraction and [binary research pipeline](research/README.md) now implement repeatable tooling and tests. Their reports retain unresolved candidates and do not constitute runtime measurements.
 
 
 
 
 
 
68
  For each checked task, record task ID, produced file/tool/specification, associated finding IDs, verification command or experiment, result, and remaining limitations. Do not check a behavior task because a guessed constant was implemented. Do not check a conversion task while required assets have unreported fallbacks. New evidence that contradicts an existing finding reopens affected tasks and records the correction.
69
 
70
  Document inspection tooling used in this pass: Python standard library (`pathlib`, `zipfile`, `struct`, `hashlib`, `csv`, `collections`, regex), including structural ELF/PE import-table reads, and macOS `file`, `strings`, and `otool -L`. One-off inspection generated the saved inventories; that original planning pass did not yet include reusable scripts. The 2026-10-11 extraction and [binary research pipeline](research/README.md) now implement repeatable tooling and tests. Their reports retain unresolved candidates and do not constitute runtime measurements.
71
+
72
+
73
+ ## 2026-10-11 — VM extension follow-up
74
+
75
+ Completed RE-211–213: extra entry command evidence, six QVM network/auth call-site cross-references, and selected Linux ELF64 native helpers. Findings F-028/029 and ABI-001–003 are documented in [VM_EXTENSIONS.md](research/VM_EXTENSIONS.md). `analyze_vm_extensions.py` verifies source/disassembly hashes and emits a deterministic focused report; five new fixture/boundary tests pass. Native command scanning includes tail calls and rejects local candidates after intervening control transfers/register writes. Reports retain broader limitations and unresolved routing, error paths, helper semantics and all-platform coverage. Originals remain in ignored cache; no original binary or QVM was run.
docs/REVERSE_ENGINEERING_TODO.md CHANGED
@@ -244,9 +244,12 @@ Purpose: turn the supplied installation into an evidence-backed specification of
244
 
245
  ## 13. Follow up the first binary analysis findings
246
 
247
- - [ ] RE-205 Name and specify qagame entry commands 11–13 and UI entry command 11 using native VM callers, argument flow and deterministic reference fixtures (VM-002).
248
- - [ ] RE-206 Recover game trap 600/601/603 and UI trap 88/90 argument/result layouts, packet/address semantics and error paths on every native platform (NAT-004).
249
  - [ ] RE-207 Extend native switch recognition to i386, Windows and macOS; compare numeric cases, pointer width, validation and results with the two ELF64 tables.
250
  - [ ] RE-208 Decode all retained SDL/app STABS source/type/function records; compare SDL 1.2.15 implementations and identify modifications (NAT-003).
251
  - [ ] RE-209 Resolve all 519 currently indirect QVM calls and remaining computed jumps through stack/dataflow analysis, retaining unresolved coverage rather than inventing targets (VM-003/004).
252
  - [ ] RE-210 Decode Windows client resource leaves, native load/bind/relocation details and unclassified tail/embedded regions; extend recursion beyond supported plain-image signatures (NAT-001/002).
 
 
 
 
244
 
245
  ## 13. Follow up the first binary analysis findings
246
 
247
+ - [ ] RE-205 Name and specify qagame entry commands 11–13 and UI entry command 11 using native VM callers, argument flow and deterministic reference fixtures (VM-002). **Progress 2026-10-11:** All four roles have matching QVM/native evidence: heartbeat, shutdown, server auth messages and UI auth responses. Complete grammar, state effects, routing predicates and runtime fixtures remain open ([ABI-001](research/VM_EXTENSIONS.md)).
248
+ - [ ] RE-206 Recover game trap 600/601/603 and UI trap 88/90 argument/result layouts, packet/address semantics and error paths on every native platform (NAT-004). **Progress 2026-10-11:** Linux ELF64 argument slots, VM pointer conversions, send/parser return handling, client-index guard and 12-byte address layout documented. Invalid-index continuation, helper semantics and other platforms remain open ([ABI-002/003](research/VM_EXTENSIONS.md)).
249
  - [ ] RE-207 Extend native switch recognition to i386, Windows and macOS; compare numeric cases, pointer width, validation and results with the two ELF64 tables.
250
  - [ ] RE-208 Decode all retained SDL/app STABS source/type/function records; compare SDL 1.2.15 implementations and identify modifications (NAT-003).
251
  - [ ] RE-209 Resolve all 519 currently indirect QVM calls and remaining computed jumps through stack/dataflow analysis, retaining unresolved coverage rather than inventing targets (VM-003/004).
252
  - [ ] RE-210 Decode Windows client resource leaves, native load/bind/relocation details and unclassified tail/embedded regions; extend recursion beyond supported plain-image signatures (NAT-001/002).
253
+ - [x] RE-211 Collect source-verified QVM bodies and Linux ELF64 callers for all four extra entry commands, including native tail calls. **Evidence:** `scripts/analyze_vm_extensions.py`, `docs/research/vm-extensions.json`, ABI-001; five focused tests pass. This completes static evidence collection, not RE-205's behavioral specification.
254
+ - [x] RE-212 Cross-reference all six observed call sites for game 600/601/603 and UI 88/90 with both ELF64 game dispatchers and the ELF64 UI dispatcher. **Evidence:** generated service cases and local QVM argument windows; ABI-002; fixture tests validate trap encoding and native helper targets.
255
+ - [x] RE-213 Preserve and inspect complete Linux ELF64 address-parser/send/drop helper bodies; document address layout, port conversion and dispatch return handling. **Evidence:** ABI-003 and hash-verified generated native functions. Full helper semantics and platform parity remain part of RE-206.
docs/research/README.md CHANGED
@@ -6,6 +6,8 @@ This pass decodes every supplied QVM and inventories/disassembles all eight nati
6
 
7
  Read [QVM findings and ABI candidates](qvm/README.md), [native images, libraries and DLL dependencies](native/README.md), and [the remaining task list](../REVERSE_ENGINEERING_TODO.md). Generated reports contain source hashes and addresses. Large derived evidence belongs under `assets/analysis/` and uses the existing Xet filters; originals stay only in `.cache/`.
8
 
 
 
9
  ## Reproduce
10
 
11
  From the repository root, with the verified source installation and extracted assets present:
@@ -13,10 +15,11 @@ From the repository root, with the verified source installation and extracted as
13
  ```sh
14
  python3 scripts/analyze_binaries.py
15
  python3 scripts/test_binary_analysis.py
 
16
  python3 scripts/storage_policy.py check --require-setup
17
  ```
18
 
19
- The pipeline runs `fetch_analysis_references.py`, `analyze_qvm.py`, `analyze_native.py`, `analyze_vm_abi.py`, and `analyze_native_dispatch.py` in that order. Python 3.9+ and **LLVM objdump** with x86 ELF/COFF/Mach-O support are required. This pass used Apple LLVM 17.0.0 on macOS arm64; it disassembled x86 images without running them. Upstream references are downloaded only if absent, checked against the committed SHA-256 manifest, and cached under `.cache/research/upstream/`. A clone also needs `git lfs pull` to obtain the derived artifacts/extracted members.
20
 
21
  Generated JSON/JSONL and compressed evidence are regenerated in place; keep prose notes in Markdown. Rerunning the four analysis stages reproduced every generated report and gzip artifact byte-for-byte on the same toolchain. gzip timestamps and stored filenames are fixed. Different objdump versions can produce different disassembly text, so tool version and uncompressed evidence hashes are recorded. Reports are UTF-8 normal Git text, with individual files below the storage policy's 2 MiB limit.
22
 
@@ -37,4 +40,6 @@ Eighteen new tests cover malformed QVM/native structures, signed syscall operand
37
 
38
  RE-041 is complete for the supplied three modules: decoder, full address-keyed instruction evidence, source-table correspondence, fixture tests and exact code reconstruction exist. RE-002's member-hash/provenance deliverable was already completed by extraction and is now linked in the checklist. Other binary/gameplay research tasks remain unchecked where only a portion is addressed. No FILE item is fully reverse engineered.
39
 
40
- Next work should name the extra game/UI entry commands, finish the 519 unresolved call sites and computed branches, recover syscall argument structures across platforms, decode SDL STABS and Windows resources, then follow QVM gameplay functions into movement/weapon/entity specifications and deterministic runtime fixtures.
 
 
 
6
 
7
  Read [QVM findings and ABI candidates](qvm/README.md), [native images, libraries and DLL dependencies](native/README.md), and [the remaining task list](../REVERSE_ENGINEERING_TODO.md). Generated reports contain source hashes and addresses. Large derived evidence belongs under `assets/analysis/` and uses the existing Xet filters; originals stay only in `.cache/`.
8
 
9
+ The [extension follow-up](VM_EXTENSIONS.md) traces all four extra game/UI entry commands and five network/auth syscall numbers through QVM callers and the Linux ELF64 engines. Its [focused generated report](vm-extensions.json) preserves selected function bodies and instruction evidence; descriptive roles and partial signatures are distinguished from unresolved behavior.
10
+
11
  ## Reproduce
12
 
13
  From the repository root, with the verified source installation and extracted assets present:
 
15
  ```sh
16
  python3 scripts/analyze_binaries.py
17
  python3 scripts/test_binary_analysis.py
18
+ python3 scripts/test_vm_extensions.py
19
  python3 scripts/storage_policy.py check --require-setup
20
  ```
21
 
22
+ The pipeline runs `fetch_analysis_references.py`, `analyze_qvm.py`, `analyze_native.py`, `analyze_vm_abi.py`, `analyze_native_dispatch.py`, and `analyze_vm_extensions.py` in that order. Python 3.9+ and **LLVM objdump** with x86 ELF/COFF/Mach-O support are required. This pass used Apple LLVM 17.0.0 on macOS arm64; it disassembled x86 images without running them. Upstream references are downloaded only if absent, checked against the committed SHA-256 manifest, and cached under `.cache/research/upstream/`. A clone also needs `git lfs pull` to obtain the derived artifacts/extracted members.
23
 
24
  Generated JSON/JSONL and compressed evidence are regenerated in place; keep prose notes in Markdown. Rerunning the four analysis stages reproduced every generated report and gzip artifact byte-for-byte on the same toolchain. gzip timestamps and stored filenames are fixed. Different objdump versions can produce different disassembly text, so tool version and uncompressed evidence hashes are recorded. Reports are UTF-8 normal Git text, with individual files below the storage policy's 2 MiB limit.
25
 
 
40
 
41
  RE-041 is complete for the supplied three modules: decoder, full address-keyed instruction evidence, source-table correspondence, fixture tests and exact code reconstruction exist. RE-002's member-hash/provenance deliverable was already completed by extraction and is now linked in the checklist. Other binary/gameplay research tasks remain unchecked where only a portion is addressed. No FILE item is fully reverse engineered.
42
 
43
+ Next work should complete the extra game/UI entry command specifications and packet grammar, finish the 519 unresolved call sites and computed branches, recover syscall argument structures across platforms, decode SDL STABS and Windows resources, then follow QVM gameplay functions into movement/weapon/entity specifications and deterministic runtime fixtures.
44
+
45
+ The extension collector adds five focused tests for native tail calls, command-register clobbers and intervening control transfers, fail-closed source identities/function boundaries, and shipped command/service connections. RE-211–213 record completed static milestones; RE-205/206 remain open for full behavior and platform coverage.
docs/research/VM_EXTENSIONS.md ADDED
@@ -0,0 +1,50 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Engine extensions: entry commands, network and authentication
2
+
3
+ Static follow-up, 2026-10-11. Evidence: [generated instruction/caller report](vm-extensions.json), tied to SHA-256 identities of both QVMs and both Linux ELF64 engines. Reproduce with `python3 scripts/analyze_vm_extensions.py`; validate with `python3 scripts/test_vm_extensions.py`. The complete pipeline also runs the collector. No original game code was executed.
4
+
5
+ ## ABI-001: extra entry commands
6
+
7
+ These are descriptive recovered roles, not recovered source enum names. The native caller and QVM body independently support each role. All four QVM entry cases call one candidate function without assigning outgoing arguments, discard its result and return zero. They consume shared engine/module state rather than forwarding the native variadic arguments in the entry case.
8
+
9
+ | Module/command | Case → QVM callee instruction | Native caller | Evidence and recovered role |
10
+ | --- | --- | --- | --- |
11
+ | qagame 11 | 146 → 1036 | `SV_MasterHeartbeat` | Caller increments the heartbeat deadline by 300,000 and invokes command 11. Callee references `HEARTBEAT`, `PLAYERS` and a player-data format string. Server authentication/master heartbeat role. Deadline unit still needs its producer traced. |
12
+ | qagame 12 | 153 → 1230 | `SV_MasterShutdown` | Caller invokes the heartbeat twice, then tail-calls command 12. Callee references `SHUTDOWN`. Server authentication/master shutdown notification role. |
13
+ | qagame 13 | 160 → 166169 | `SV_ConnectionlessPacket` | Packet handler tokenizes an OOB string before invoking the game VM. Callee references `authc`, `AUTH:CLIENT`, `ACCEPT`, `REJECT`, `AUTH:MSG`, `DROP`, user authentication fields and announcements. Server authentication message dispatcher role. |
14
+ | ui 11 | 123 → 3909 | `CL_ConnectionlessPacket` | Client packet handler tail-calls UI command 11, with a source-address value occupying `%rdx/%ecx`. The entry body does not forward it. Callee references `AUTH:CLIENT`, `REJECT`, `CHALLENGE`, `authl`, `UPDATE`, `ANNOUNCEMENT` and `cl_auth_*` variables. UI authentication response dispatcher role. |
15
+
16
+ Both Linux engines contain all three server callers. Dedicated addresses of their `VM_Call` transfers: heartbeat `0x411c4e`, shutdown `0x4120c2`, packet `0x411a1e`. Client-engine server equivalents: `0x479bce`, `0x47a042`, `0x47999e`; UI packet tail transfer: `0x41e673`. QVM addresses are instruction indices, whereas native addresses are preferred virtual addresses.
17
+
18
+ The collector includes complete linear QVM partitions for these callees, their constant string references, complete native caller functions, and bounded local argument evidence. Source-level function signatures, all packet routing predicates, protocol grammar, trust validation, side effects and reachability remain unproved. In particular, naming a role does not establish an authentication implementation specification.
19
+
20
+ ## ABI-002: network syscall contracts on Linux ELF64
21
+
22
+ Native syscall arrays contain eight-byte slots here; slot zero is the trap number. QVM outgoing `ARG` offsets are four-byte-spaced (`8, 12, 16, 20`). VM pointer conversion must be explicit in a Rust replacement; these are VM addresses, not host pointers. The native cases preserve the syscall-array base in `%rbp` while translating pointers through `VM_ArgPtr`.
23
+
24
+ | Game trap / UI trap | Recovered arguments, starting at slot 1 | Native behavior and result |
25
+ | --- | --- | --- |
26
+ | 600 / 88 | input string VM pointer; output address VM pointer | Convert both pointers, call `NET_StringToAdr(input, output)`, return its 32-bit result through a zero-extended register. QVM observes the low 32 bits. |
27
+ | 601 / 90 | socket selector; packet length; packet-data VM pointer; destination-string VM pointer | Resolve destination into a native stack-local address, convert data pointer, call `NET_SendPacket(socket, length, data, address-by-value)`, return zero. The destination parser result is not tested by this dispatch case. |
28
+ | 603 / no matched UI extension | signed client index; two VM pointers | Convert both pointers, reject negative indices and indices at least `sv_maxclients` through a common continuation, form `svs.clients + index * 0x2df40`, call `SV_Auth_DropClient(client, pointer2, pointer3)`, return zero on the valid path. Meaning of the two string-like parameters and complete drop side effects remain open. |
29
+
30
+ The game service cases in the dedicated and client engines use matching argument offsets and call sequences. UI 88 and 90 implement the same address/send pattern in the client engine. This is a static cross-check, not runtime conformance or proof of Windows/macOS parity. Trap 603's invalid-index continuation and helper's early exits need explicit result/error tracing before a complete contract can be claimed.
31
+
32
+ All six observed QVM call sites are retained: game 600 at instruction 418, 601 at 1032, 603 at 165943; UI 88 at 3712 and 90 at 3462/3557. The game send call and first UI send call explicitly use socket selector zero; the second UI call loads its selector from an incoming local. Both first send sites compute a length from their packet buffer before passing it to the native service. Inferring the socket enum name or inclusive/exclusive terminator policy requires tracing the referenced helpers.
33
+
34
+ ## ABI-003: native address layout and parsing behavior
35
+
36
+ The dedicated `NET_StringToAdr` body at `0x442a06` shows a 12-byte native address value: a 32-bit type at offset 0, IPv4 bytes at offset 4, a byte-swapped 16-bit port at offset 8, with remaining bytes/padding not assigned a meaning. `NET_SendPacket` receives the first eight bytes in `%rcx` and the next four in `%r8d`. This layout is observed for these engines; Rust needs a semantic address type and explicit serializers rather than assuming its own memory layout matches.
37
+
38
+ The parser copies into a 1,024-byte buffer using `Q_strncpyz`, splits on the first colon, invokes `Sys_StringToAdr`, parses an explicit decimal port through `__strtol_internal`, truncates to 16 bits and calls `ShortSwap`. With no explicit port it uses `0x6d38` (27960). A special literal-comparison path clears twelve bytes, sets type 2, and returns one; the literal is not assigned a name in this pass. Failed system resolution or an all-ones IPv4 word sets type 1; the latter explicitly clears the return register. Do not assume all failure paths fully initialize the address or collapse them into one Rust error yet.
39
+
40
+ This gives a concrete replacement boundary for address parsing and packet sending. DNS behavior, socket selector values, loopback/broadcast handling, packet limits, framing, retries and platform-specific errors still require the `Sys_StringToAdr` and `NET_SendPacket` callees plus runtime fixtures. No network traffic was sent during this analysis.
41
+
42
+ ## Remaining work and acceptance
43
+
44
+ - Trace every routing predicate and token access into a packet grammar, including accepted/rejected malformed inputs and source verification. Separate authentication UI state from server admission/drop state.
45
+ - Recover the two drop parameter meanings, common invalid-index continuation, full address initialization and send failure paths; record deterministic input/output fixtures before implementing replacements.
46
+ - Compare corresponding native services and callers on Linux i386, Windows and macOS; establish layout and numeric ABI differences independently.
47
+ - Resolve nested QVM calls, length helpers and socket enum values. Confirm shared command-token state dependencies and reentrancy.
48
+ - Implement Rust address/message types only after semantic serialization contracts exist. Preserve compatibility policy for unavailable historical authentication services as a separate design decision.
49
+
50
+ The broad RE-205 and RE-206 tasks remain open because runtime fixtures and all-platform behavior are still missing. Focused evidence-collection milestones are complete in the main checklist. Native caller scanning reports local candidates and intentionally discards a candidate after a branch, call, return or write to the command register; alternative incoming branches and unusual instruction aliases are outside this recognizer's scope.
docs/research/vm-extensions.json ADDED
The diff for this file is too large to render. See raw diff
 
scripts/analyze_binaries.py CHANGED
@@ -9,7 +9,7 @@ def main():
9
  if not Path('docs/inventory/snapshot.json').is_file():
10
  raise SystemExit('Run from the repository root')
11
  for script in ('fetch_analysis_references.py', 'analyze_qvm.py', 'analyze_native.py',
12
- 'analyze_vm_abi.py', 'analyze_native_dispatch.py'):
13
  subprocess.run([sys.executable, str(Path('scripts')/script)], check=True)
14
 
15
 
 
9
  if not Path('docs/inventory/snapshot.json').is_file():
10
  raise SystemExit('Run from the repository root')
11
  for script in ('fetch_analysis_references.py', 'analyze_qvm.py', 'analyze_native.py',
12
+ 'analyze_vm_abi.py', 'analyze_native_dispatch.py', 'analyze_vm_extensions.py'):
13
  subprocess.run([sys.executable, str(Path('scripts')/script)], check=True)
14
 
15
 
scripts/analyze_vm_extensions.py ADDED
@@ -0,0 +1,132 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Collect hash-verified static evidence for shipped VM engine extensions.
3
+
4
+ This is a focused evidence collector, not a native decompiler or VM emulator.
5
+ Native VM_Call argument candidates are accepted only within straight-line
6
+ instruction sequences after a constant command assignment (including tail calls).
7
+ """
8
+ import gzip
9
+ import hashlib
10
+ import json
11
+ from pathlib import Path
12
+ import re
13
+ from analyze_qvm import decode, static_analysis, write_json
14
+ from analyze_native_dispatch import recover
15
+ from analyze_vm_abi import entry_dispatch
16
+
17
+
18
+ def verified(data, expected):
19
+ if hashlib.sha256(data).hexdigest() != expected:
20
+ raise ValueError('analysis source hash mismatch')
21
+ return data
22
+
23
+
24
+ def native_functions(text):
25
+ return {m[2]: {'address': int(m[1], 16), 'lines': m[3].splitlines()}
26
+ for m in re.finditer(r'(?m)^([0-9a-f]+) <([^>]+)>:\n(.*?)(?=\n[0-9a-f]+ <|\Z)', text, re.S)}
27
+
28
+
29
+ def command_callers(functions):
30
+ rows = []
31
+ for name, function in functions.items():
32
+ candidate = None
33
+ evidence = []
34
+ for line in function['lines']:
35
+ ins = re.match(r'\s*([0-9a-f]+):[^\t]*\t(\w+)\s*(.*)', line)
36
+ if not ins:
37
+ continue
38
+ address, op, operands = int(ins[1], 16), ins[2], ins[3]
39
+ assignment = re.match(r'\$0x([0-9a-f]+), %esi(?:\s|$)', operands)
40
+ if op == 'movl' and assignment:
41
+ candidate = int(assignment[1], 16)
42
+ evidence = []
43
+ if candidate is not None:
44
+ evidence.append(line.strip())
45
+ transfer = op in ('call', 'callq', 'jmp', 'jmpq')
46
+ if transfer and '<VM_Call>' in operands and candidate is not None:
47
+ rows.append({'function': name, 'instruction_address': address,
48
+ 'command_candidate': candidate, 'tail_call': op.startswith('jmp'),
49
+ 'instructions': evidence})
50
+ # Do not propagate across calls, branches, returns, or command writes.
51
+ if (op.startswith(('j', 'call', 'ret', 'loop')) or
52
+ (assignment is None and re.search(r', %(?:rsi|esi|si|sil)(?:\s|$)', operands)) or
53
+ (op.startswith('pop') and operands.strip() in ('%rsi', '%esi'))):
54
+ candidate = None
55
+ evidence = []
56
+ return rows
57
+
58
+
59
+ def qvm_function(instructions, entry, strings):
60
+ if instructions[entry]['opcode'] != 'ENTER':
61
+ raise ValueError('extension target is not a candidate function entry')
62
+ end = next((i for i in range(entry+1, len(instructions))
63
+ if instructions[i]['opcode'] == 'ENTER'), len(instructions))
64
+ body = instructions[entry:end]
65
+ return {'entry': entry, 'linear_end_exclusive': end, 'instructions': body,
66
+ 'string_reference_candidates': [{'instruction': i['index'], 'vm_address': i['operand'],
67
+ 'text': strings[i['operand']]}
68
+ for i in body if i['opcode'] == 'CONST' and i['operand'] in strings]}
69
+
70
+
71
+ def main():
72
+ result = {'schema_version': 1, 'qvm': {}, 'native': {}, 'limitations':
73
+ 'Static evidence only. Linear QVM partitions and local call/string patterns are candidates; '
74
+ 'native command scanning rejects intervening control transfers but does not prove reachability '
75
+ 'or absence of alternate incoming edges. Native coverage is the two supplied Linux ELF64 engines.'}
76
+ for module, commands, traps in [('qagame', [11, 12, 13], [600, 601, 603]), ('ui', [11], [88, 90])]:
77
+ report = json.loads(Path(f'docs/research/qvm/{module}.json').read_text())
78
+ data = verified(Path(report['source']['member']['output']).read_bytes(), report['sha256'])
79
+ header, instructions = decode(data)
80
+ _, _, _, string_rows, _ = static_analysis(data, header, instructions)
81
+ strings = {s['vm_address']: s['text'] for s in string_rows}
82
+ abi = json.loads(Path(f'docs/research/qvm/{module}.abi-candidates.json').read_text())
83
+ if abi['sha256'] != report['sha256']:
84
+ raise ValueError('ABI report source mismatch')
85
+ entries, functions, calls = [], {}, []
86
+ for dispatch in abi['entry_dispatch']:
87
+ if dispatch['command'] not in commands:
88
+ continue
89
+ start = dispatch['case_instruction']
90
+ recovered = entry_dispatch(instructions, data, header, dispatch['command'])
91
+ if recovered['case_instruction'] != start:
92
+ raise ValueError('ABI entry dispatch disagrees with source instructions')
93
+ body = instructions[start:start+5]
94
+ if [i['opcode'] for i in body] != ['CONST', 'CALL', 'POP', 'CONST', 'LEAVE'] or body[3]['operand'] != 0:
95
+ raise ValueError('unrecognized extension entry body')
96
+ target = body[0]['operand']
97
+ entries.append({'command': dispatch['command'], 'case_instruction': start,
98
+ 'callee_entry_candidate': target, 'instructions': body})
99
+ functions[str(target)] = qvm_function(instructions, target, strings)
100
+ for i in range(1, len(instructions)):
101
+ previous = instructions[i-1]
102
+ if instructions[i]['opcode'] != 'CALL' or previous['opcode'] != 'CONST' or previous['operand'] not in [-1-n for n in traps]:
103
+ continue
104
+ entry = next(j for j in range(i-1, -1, -1) if instructions[j]['opcode'] == 'ENTER')
105
+ functions[str(entry)] = qvm_function(instructions, entry, strings)
106
+ calls.append({'trap': -1-previous['operand'], 'call_instruction': i, 'containing_entry_candidate': entry,
107
+ 'local_instructions': instructions[max(entry, i-24):i+8]})
108
+ result['qvm'][module] = {'sha256': report['sha256'], 'source': report['source'],
109
+ 'extension_entries': entries, 'trap_calls': calls, 'functions': functions}
110
+ for label in ('linux--Quake3-UrT-Ded.x86_64', 'linux--Quake3-UrT.x86_64'):
111
+ report = json.loads(Path(f'docs/research/native/{label}.json').read_text())
112
+ data = verified((Path('.cache/UrbanTerror43')/report['source']['path']).read_bytes(), report['source']['sha256'])
113
+ artifact = report['artifacts']['disassembly.txt']
114
+ text = verified(gzip.open(artifact['path'], 'rb').read(), artifact['uncompressed_sha256']).decode()
115
+ functions = native_functions(text)
116
+ callers = [r for r in command_callers(functions) if r['command_candidate'] in (11, 12, 13)]
117
+ names = {r['function'] for r in callers} | {'NET_StringToAdr', 'NET_SendPacket', 'SV_Auth_DropClient'}
118
+ if not names <= functions.keys():
119
+ raise ValueError('missing required native evidence function')
120
+ cases = []
121
+ for symbol, numbers in [('SV_GameSystemCalls', [600, 601, 603])] + ([] if 'Ded' in label else [('CL_UISystemCalls', [88, 90])]):
122
+ dispatch = recover(data, text, symbol)
123
+ cases.extend({'dispatcher': symbol, **c} for c in dispatch['cases'] if set(c['numbers_sharing_target']) & set(numbers))
124
+ result['native'][label] = {'source': report['source'], 'disassembly_sha256': artifact['uncompressed_sha256'],
125
+ 'extension_caller_candidates': callers, 'service_cases': cases,
126
+ 'functions': {name: functions[name] for name in sorted(names)}}
127
+ write_json(Path('docs/research/vm-extensions.json'), result)
128
+ print('Collected extra entry commands, five network/auth services and their static callers')
129
+
130
+
131
+ if __name__ == '__main__':
132
+ main()
scripts/test_vm_extensions.py ADDED
@@ -0,0 +1,62 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Check extension evidence boundaries, provenance and shipped ABI connections."""
3
+ import json
4
+ from pathlib import Path
5
+ import unittest
6
+ from analyze_vm_extensions import command_callers, verified, qvm_function
7
+
8
+
9
+ class ExtensionTests(unittest.TestCase):
10
+ def callers(self, *lines):
11
+ return command_callers({'fixture': {'address': 1, 'lines': list(lines)}})
12
+
13
+ def test_tail_call(self):
14
+ rows = self.callers(' 1: be\tmovl\t$0xc, %esi', ' 2: 31\txorl\t%eax, %eax',
15
+ ' 3: e9\tjmp\t0x100 <VM_Call>')
16
+ self.assertEqual([(r['command_candidate'], r['tail_call']) for r in rows], [(12, True)])
17
+
18
+ def test_control_or_register_clobber_rejects_candidate(self):
19
+ for middle in ('callq\t0x200 <other>', 'jne\t0x20', 'movq\t%rax, %rsi',
20
+ 'xorl\t%esi, %esi', 'popq\t%rsi', 'retq', 'loop\t0x20'):
21
+ with self.subTest(middle=middle):
22
+ self.assertEqual(self.callers(' 1: be\tmovl\t$0xc, %esi',
23
+ ' 2: aa\t'+middle, ' 3: e8\tcallq\t0x100 <VM_Call>'), [])
24
+
25
+ def test_command_namespace_not_assumed(self):
26
+ rows = self.callers(' 1: be\tmovl\t$0xb, %esi', ' 2: e8\tcallq\t0x100 <VM_Call>',
27
+ ' 3: e8\tcallq\t0x100 <VM_Call>')
28
+ self.assertEqual(len(rows), 1)
29
+ self.assertNotIn('module', rows[0])
30
+
31
+ def test_hash_and_partition_fail_closed(self):
32
+ with self.assertRaises(ValueError):
33
+ verified(b'changed', '0'*64)
34
+ with self.assertRaises(ValueError):
35
+ qvm_function([{'opcode': 'CONST'}], 0, {})
36
+
37
+ def test_shipped_extension_connections(self):
38
+ r = json.loads(Path('docs/research/vm-extensions.json').read_text())
39
+ self.assertEqual([(e['command'], e['callee_entry_candidate']) for e in r['qvm']['qagame']['extension_entries']],
40
+ [(11, 1036), (12, 1230), (13, 166169)])
41
+ self.assertEqual(r['qvm']['ui']['extension_entries'][0]['callee_entry_candidate'], 3909)
42
+ for label, n in r['native'].items():
43
+ verified((Path('.cache/UrbanTerror43')/n['source']['path']).read_bytes(), n['source']['sha256'])
44
+ callers = {(c['function'], c['command_candidate'], c['tail_call']) for c in n['extension_caller_candidates']}
45
+ self.assertTrue({('SV_MasterHeartbeat', 11, False), ('SV_MasterShutdown', 12, True),
46
+ ('SV_ConnectionlessPacket', 13, False)} <= callers)
47
+ if 'Ded' not in label:
48
+ self.assertIn(('CL_ConnectionlessPacket', 11, True), callers)
49
+ for case in n['service_cases']:
50
+ number = case['numbers_sharing_target'][0]
51
+ expected = 'NET_StringToAdr' if number in (600, 88) else 'NET_SendPacket' if number in (601, 90) else 'SV_Auth_DropClient'
52
+ self.assertIn(expected, case['direct_call_or_jump_labels'])
53
+ for module, q in r['qvm'].items():
54
+ verified(Path(q['source']['member']['output']).read_bytes(), q['sha256'])
55
+ self.assertEqual(len(q['trap_calls']), 3)
56
+ for c in q['trap_calls']:
57
+ ins = {i['index']: i for i in c['local_instructions']}
58
+ self.assertEqual(ins[c['call_instruction']-1]['operand'], -1-c['trap'])
59
+
60
+
61
+ if __name__ == '__main__':
62
+ unittest.main()