Recover chat expansion states and substitution keys
Browse files- docs/FINDINGS.md +6 -0
- docs/PROVISIONAL_TODO.md +1 -1
- docs/RESEARCH_LEDGER.md +4 -0
- docs/REVERSE_ENGINEERING_TODO.md +5 -0
- docs/research/CHAT_DELIVERY.md +2 -0
- docs/research/CHAT_EXPANSION.md +50 -0
- docs/research/README.md +4 -1
- docs/research/chat-expansion.json +0 -0
- scripts/analyze_binaries.py +1 -1
- scripts/analyze_chat_expansion.py +95 -0
- scripts/test_chat_expansion.py +90 -0
docs/FINDINGS.md
CHANGED
|
@@ -258,3 +258,9 @@ Bounded qagame paths recover bomb-plant/inactivity correction requests, direct f
|
|
| 258 |
**F-059 — Observed chat formatter arguments:** a guarded five-entry mode table and bounded delivery paths select cchat/tcchat/me, SUB-over-DEAD and independent SPEC labels, private direction markers and byte-25 prefixes. Aligned synthetic objects produce trap-17 slots 3/2. Recorded formatter returns are placeholders, so escaping, formatted commands and rendered output remain unverified. [DEL-002](research/CHAT_DELIVERY.md).
|
| 259 |
|
| 260 |
**F-060 — Observed delivery call topology:** explicit-target paths suppress redundant self echo and skip sender echo for object flag bit 8; fanout constructs stride-880 slot objects and can invoke cleanup per iteration. Recorded calls do not establish accepted recipients or actual cleanup results. [DEL-003](research/CHAT_DELIVERY.md).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 258 |
**F-059 — Observed chat formatter arguments:** a guarded five-entry mode table and bounded delivery paths select cchat/tcchat/me, SUB-over-DEAD and independent SPEC labels, private direction markers and byte-25 prefixes. Aligned synthetic objects produce trap-17 slots 3/2. Recorded formatter returns are placeholders, so escaping, formatted commands and rendered output remain unverified. [DEL-002](research/CHAT_DELIVERY.md).
|
| 259 |
|
| 260 |
**F-060 — Observed delivery call topology:** explicit-target paths suppress redundant self echo and skip sender echo for object flag bit 8; fanout constructs stride-880 slot objects and can invoke cleanup per iteration. Recorded calls do not establish accepted recipients or actual cleanup results. [DEL-003](research/CHAT_DELIVERY.md).
|
| 261 |
+
|
| 262 |
+
**F-061 — Observed marker parser states:** dollar/hash markers collect ASCII letters and query each growing prefix; nonzero lookup clears the marker, while unmatched delimiter/end routes preserve literal arguments. Source length >150 rejects, but acceptance at 150 still requests capacity 150. Copy/append/lookup composition and actual output bounds remain open. [EXP-001](research/CHAT_EXPANSION.md).
|
| 263 |
+
|
| 264 |
+
**F-062 — Observed substitution keys:** twelve exact supplied keys use comparator 69386 with bound 99999. Bounded byte traces confirm ASCII case-insensitive terminated equality and reject shortened/extended keys. All bodies are retained; weapon/timer/location/leader/crosshair behavior remains incomplete. [EXP-002](research/CHAT_EXPANSION.md).
|
| 265 |
+
|
| 266 |
+
**F-063 — Observed health and momo differences:** health labels split at 25/50/75; health/hp dead override tests offset 4 >=3, whereas delivery's dead flag uses ==3. The momo branch clears the shared buffer, optionally writes now+3500 and 2 through record pointer 6168, and returns zero. Field ownership, formatted output and runtime wrap remain unverified. [EXP-003](research/CHAT_EXPANSION.md).
|
docs/PROVISIONAL_TODO.md
CHANGED
|
@@ -33,7 +33,7 @@ For implementation items, the minimum done condition is a documented contract, a
|
|
| 33 |
|
| 34 |
### First research batch
|
| 35 |
|
| 36 |
-
Progress: [RE-232–234 / TIME-001–003](research/FRAME_TIMING.md) now cover Linux ELF64 elapsed modification, event sources and pause predicates. The [RE-237–239 follow-up](research/EVENT_QUEUES.md) resolves the pause-field address/bot chain and adds queue/journal evidence. The [RE-241–243 announcement pass](research/AUTH_ANNOUNCEMENTS.md) recovers label tables and recipient paths; The [RE-245–247 cvar pass](research/GAME_CVARS.md) adds all 126 main registration rows and resolves auth_verbosity/callers. The [RE-249–251 change pass](research/CVAR_CHANGES.md) adds cache/notification behavior, four numeric correction branches and direct registration coverage. The [RE-253–255 query pass](research/AUTH_QUERIES.md) adds selector helpers, command-role routing and internal authentication modes. The [RE-257–259 bounds/entry pass](research/SELECTOR_BOUNDS.md) adds integer parsing, selector storage/count evidence and native/QVM/chat route connections. The [RE-261–263 chat pass](research/CHAT_PATHS.md) adds native command metadata/filter evidence and bounded message/recipient paths, including the documented argument-mapping correction. The [RE-265–267 delivery pass](research/CHAT_DELIVERY.md) adds bounded dead-chat predicates, formatter argument routing and echo/fanout calls. PROV-002 remains open for full output/reset coverage. PROV-001 remains open for complete scheduling and platform coverage.
|
| 37 |
|
| 38 |
Start with the open questions raised by the current binary evidence. These tasks depend on the linked reports, not on a working rewrite.
|
| 39 |
|
|
|
|
| 33 |
|
| 34 |
### First research batch
|
| 35 |
|
| 36 |
+
Progress: [RE-232–234 / TIME-001–003](research/FRAME_TIMING.md) now cover Linux ELF64 elapsed modification, event sources and pause predicates. The [RE-237–239 follow-up](research/EVENT_QUEUES.md) resolves the pause-field address/bot chain and adds queue/journal evidence. The [RE-241–243 announcement pass](research/AUTH_ANNOUNCEMENTS.md) recovers label tables and recipient paths; The [RE-245–247 cvar pass](research/GAME_CVARS.md) adds all 126 main registration rows and resolves auth_verbosity/callers. The [RE-249–251 change pass](research/CVAR_CHANGES.md) adds cache/notification behavior, four numeric correction branches and direct registration coverage. The [RE-253–255 query pass](research/AUTH_QUERIES.md) adds selector helpers, command-role routing and internal authentication modes. The [RE-257–259 bounds/entry pass](research/SELECTOR_BOUNDS.md) adds integer parsing, selector storage/count evidence and native/QVM/chat route connections. The [RE-261–263 chat pass](research/CHAT_PATHS.md) adds native command metadata/filter evidence and bounded message/recipient paths, including the documented argument-mapping correction. The [RE-265–267 delivery pass](research/CHAT_DELIVERY.md) adds bounded dead-chat predicates, formatter argument routing and echo/fanout calls. The [RE-269–271 expansion pass](research/CHAT_EXPANSION.md) adds marker states, twelve key comparisons and health/momo branches. PROV-002 remains open for full output/reset coverage. PROV-001 remains open for complete scheduling and platform coverage.
|
| 37 |
|
| 38 |
Start with the open questions raised by the current binary evidence. These tasks depend on the linked reports, not on a working rewrite.
|
| 39 |
|
docs/RESEARCH_LEDGER.md
CHANGED
|
@@ -129,3 +129,7 @@ Completed bounded milestones RE-261–263 with `analyze_chat_paths.py`: paired n
|
|
| 129 |
## 2026-10-11 — Dead-chat policy, formats and delivery calls
|
| 130 |
|
| 131 |
Completed bounded milestones RE-265–267 with `analyze_chat_delivery.py`: raw flag predicates, 512 dead-chat decision paths, five-entry mode table, exact formatter arguments/control strings, recipient-slot arithmetic and echo/fanout call traces. DEL-001–003/F-058–060 retain provisional field identities and explicit zero callee-return placeholders. Seven tests check branch destinations, policy distinctions, raw field alternatives, format arguments, control bytes and delivery call topology. RE-264 remains open; RE-268 tracks text expansion, raw-field/global lifetime, client consumption and full runtime/platform output. No original process/full QVM or recorded callee was executed.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 129 |
## 2026-10-11 — Dead-chat policy, formats and delivery calls
|
| 130 |
|
| 131 |
Completed bounded milestones RE-265–267 with `analyze_chat_delivery.py`: raw flag predicates, 512 dead-chat decision paths, five-entry mode table, exact formatter arguments/control strings, recipient-slot arithmetic and echo/fanout call traces. DEL-001–003/F-058–060 retain provisional field identities and explicit zero callee-return placeholders. Seven tests check branch destinations, policy distinctions, raw field alternatives, format arguments, control bytes and delivery call topology. RE-264 remains open; RE-268 tracks text expansion, raw-field/global lifetime, client consumption and full runtime/platform output. No original process/full QVM or recorded callee was executed.
|
| 132 |
+
|
| 133 |
+
## 2026-10-11 — Chat expansion states and substitution keys
|
| 134 |
+
|
| 135 |
+
Completed bounded milestones RE-269–271 with `analyze_chat_expansion.py`: parser length/marker/prefix/tail paths, twelve key comparisons and byte-loop fixtures, 36 health/four hp cases and three momo NULL/write/wrap paths. EXP-001–003/F-061–063 distinguish supplied lengths/lookup returns from executed callees, capacity from preservation, and >=3 health death predicates from ==3 delivery predicates. Seven tests cover exact key evidence, comparator bounds, character/return/tail routing, length gates, labels and side-effect writes. RE-268 remains open; RE-272 tracks full buffer/parser/substitution/ownership/runtime/platform composition. No original executable, full QVM or recorded callee executes.
|
docs/REVERSE_ENGINEERING_TODO.md
CHANGED
|
@@ -317,3 +317,8 @@ Purpose: turn the supplied installation into an evidence-backed specification of
|
|
| 317 |
- [x] RE-266 Recover mode jump table, exact formatter argument routing, labels/control bytes and signed recipient-slot calculation ([DEL-002](research/CHAT_DELIVERY.md)).
|
| 318 |
- [x] RE-267 Recover explicit-recipient self/flag-bit echo gates and count/cleanup fanout call paths ([DEL-003](research/CHAT_DELIVERY.md)).
|
| 319 |
- [ ] RE-268 Recover radio/text expansion helper 186435 and key lookup 185212, all chat flag/global writers, full prefix/copy/escaping behavior, private-message callers and client command consumption; validate output/queues/lifecycle across original runtime and target platforms.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 317 |
- [x] RE-266 Recover mode jump table, exact formatter argument routing, labels/control bytes and signed recipient-slot calculation ([DEL-002](research/CHAT_DELIVERY.md)).
|
| 318 |
- [x] RE-267 Recover explicit-recipient self/flag-bit echo gates and count/cleanup fanout call paths ([DEL-003](research/CHAT_DELIVERY.md)).
|
| 319 |
- [ ] RE-268 Recover radio/text expansion helper 186435 and key lookup 185212, all chat flag/global writers, full prefix/copy/escaping behavior, private-message callers and client command consumption; validate output/queues/lifecycle across original runtime and target platforms.
|
| 320 |
+
|
| 321 |
+
- [x] RE-269 Recover bounded radio/chat source-length gate, marker replacement, growing-key lookup, unmatched-byte and pending-tail branches ([EXP-001](research/CHAT_EXPANSION.md)).
|
| 322 |
+
- [x] RE-270 Recover all twelve substitution key comparisons and actual ASCII case-insensitive comparator fixtures; retain every lookup body ([EXP-002](research/CHAT_EXPANSION.md)).
|
| 323 |
+
- [x] RE-271 Recover health/hp label thresholds/death overrides and momo buffer clear, optional referenced-object writes and zero return ([EXP-003](research/CHAT_EXPANSION.md)).
|
| 324 |
+
- [ ] RE-272 Compose parser copy/length/append/lookup with exact capacities, prefix consumption and aliases; recover item/ammo tables, time cvars/arithmetic, location/leader/crosshair callees and momo object ownership/consumers; validate original-runtime and platform fixtures.
|
docs/research/CHAT_DELIVERY.md
CHANGED
|
@@ -57,3 +57,5 @@ The fanout tail visits `34091368 + i * 880` for indices below signed count globa
|
|
| 57 |
## Remaining work
|
| 58 |
|
| 59 |
RE-265–267 record these bounded milestones. RE-264 stays open for the complete chat contract; RE-268 adds specific follow-ups: text-expansion helper 186435 and key lookup 185212, all raw flag/global writers, prefix construction/copy/escaping, private-message callers and client command consumption, plus original-runtime and macOS/Linux comparison fixtures. Rust/Bevy behavior should be specified from these predicates while retaining the unresolved field identities and output contracts as explicit dependencies.
|
|
|
|
|
|
|
|
|
| 57 |
## Remaining work
|
| 58 |
|
| 59 |
RE-265–267 record these bounded milestones. RE-264 stays open for the complete chat contract; RE-268 adds specific follow-ups: text-expansion helper 186435 and key lookup 185212, all raw flag/global writers, prefix construction/copy/escaping, private-message callers and client command consumption, plus original-runtime and macOS/Linux comparison fixtures. Rust/Bevy behavior should be specified from these predicates while retaining the unresolved field identities and output contracts as explicit dependencies.
|
| 60 |
+
|
| 61 |
+
The [EXP-001–003 follow-up](CHAT_EXPANSION.md) recovers parser states, key comparisons and health/momo branches. Full text-expansion/copy/output behavior remains open under RE-268/272.
|
docs/research/CHAT_EXPANSION.md
ADDED
|
@@ -0,0 +1,50 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Radio/chat expansion states and substitution keys
|
| 2 |
+
|
| 3 |
+
Static pass, 2026-10-11. Reproduce with `python3 scripts/analyze_chat_expansion.py`; check with `python3 scripts/test_chat_expansion.py`. [Generated evidence](chat-expansion.json) contains verified qagame provenance, complete partitions for parser 186435, lookup 185212 and comparator 69386, plus bounded synthetic traces. No original executable or full QVM runs. Calls are recorded with zero placeholder returns; no formatted/copy/append output is produced. These findings extend [chat delivery](CHAT_DELIVERY.md).
|
| 4 |
+
|
| 5 |
+
## EXP-001: marker parsing and length gate
|
| 6 |
+
|
| 7 |
+
Parser 186435 checks a source length returned by helper 36038. A value **greater than 150** takes diagnostic `CG_UTParseRadio: radio string > MAX_MAPPING_STRING\n` through helper 2294 and leaves without the copy/clear path. Values 149/150 reach formatter 70386 with destination local 32, capacity **150**, template `%s` and the source pointer, then clear the source's first byte. Acceptance at 150 does not prove preservation of 150 bytes: capacity, formatting truncation and effective string length remain separate questions. The fixtures supply lengths; recorded copy and repeated length calls return zero, so the accepted fixture skips the character loop. It must not be interpreted as a real empty-input transformation.
|
| 8 |
+
|
| 9 |
+
Per-character slices at 186481–186676 establish the following states. Pending marker lives at local byte 182, key text at local 183, key length at local word 336; current byte is signed through SEX8.
|
| 10 |
+
|
| 11 |
+
- `$` (36) and `#` (35) both start a pending marker. An existing marker/key first gets formatter `%c%s` and append helper 36123, then marker is replaced and key cleared.
|
| 12 |
+
- While pending, ASCII letters `a`–`z` or `A`–`Z` are appended to the key with a NUL. Lookup 185212 receives the sender and **each growing prefix**, rather than waiting for a word delimiter.
|
| 13 |
+
- A nonzero lookup result is passed to append helper 36123 and clears the marker. A zero result retains it. Separate post-call fixtures supply both results explicitly; no lookup callee is executed by those parser slices.
|
| 14 |
+
- A nonletter while pending goes through `%c%s%c`, retaining marker/key/current-byte as formatter arguments, then clears the marker. A signed high byte 255 is forwarded as -1 in the character argument. Without a marker, the byte goes through `%c`.
|
| 15 |
+
- End-of-loop pending marker/key takes `%c%s` and append. No pending marker means no tail call.
|
| 16 |
+
|
| 17 |
+
Because lookup happens after each letter, a recognized short key can consume a prefix of a longer letter run. For example, the raw state machine permits matching `hp` before a later letter is inspected. Actual combined formatted output and buffer bounds require copy/length/append/lookup composition. A synthetic NUL per-character fixture verifies argument routing only; the real loop condition normally ends before its terminator.
|
| 18 |
+
|
| 19 |
+
## EXP-002: exact comparison keys and unresolved substitution bodies
|
| 20 |
+
|
| 21 |
+
All twelve key comparisons call helper **69386** with bound **99999**. Actual byte-loop traces establish ASCII case-insensitive lexicographic equality, terminated-string equality and rejection of a shorter/longer key. It folds lowercase ASCII to uppercase, reads signed bytes, and stops when its decrementing bound is exhausted. Comparison fixtures cover every exact/uppercase/shortened/extended key plus bounded prefix, embedded NUL and signed high-byte cases. This does not establish locale/Unicode matching.
|
| 22 |
+
|
| 23 |
+
| Key in comparison order | Static body evidence / remaining work |
|
| 24 |
+
| --- | --- |
|
| 25 |
+
| `health` | Four labels and dead override; bounded in EXP-003 |
|
| 26 |
+
| `hp` | Raw object field 684 formatted `%i`, then possible `0` override; bounded in EXP-003 |
|
| 27 |
+
| `clips` | Selected record word at offset 312 + 4*field144; high byte after mask/unsigned shift. Packing and item limits open |
|
| 28 |
+
| `bullets` | Same selected word, mask 65280 and signed shift 8. Packing semantics open |
|
| 29 |
+
| `weapon` | Low byte of selected word indexes stride 432 at base 4020; word interpreted by `%s`. Table/identity/bounds open |
|
| 30 |
+
| `location` | Helper 268637 requested with sender, shared buffer and capacity 150; complete location selection open |
|
| 31 |
+
| `gameitem` | Game types 7/8 choose `flag`/`bomb`; other path uses helper 36717 bit 1 for `coinbird`/`cat-bird`. Randomness and runtime modes open |
|
| 32 |
+
| `gametime` | Cvar/global arithmetic and `%01d:%02d` or `no timelimit`; full timing/wrap contract open |
|
| 33 |
+
| `roundtime` | Round-deadline arithmetic, game-type gates and `%01d:%02d` or `no roundtime`; full schedule open |
|
| 34 |
+
| `leader` | Helper 169397 and stride-880 object/record name access; return/no-leader handling open |
|
| 35 |
+
| `momo` | Buffer clear, optional referenced-object writes and zero return; bounded in EXP-003 |
|
| 36 |
+
| `crosshair` | Dead fallback or floating-point direction/trap trace/location path; vector, trace, ownership and geometry semantics open |
|
| 37 |
+
|
| 38 |
+
Lookup initializes a shared output buffer **35738436** through a capacity-150 formatting request for `unknown`. Recognized ordinary tails return that buffer pointer; unmatched keys take zero return. `momo` also returns zero after its side effect. Names in this table are exact supplied strings, not inferred command names. Static bodies are retained even where full evaluation is deferred.
|
| 39 |
+
|
| 40 |
+
## EXP-003: health boundaries and momo side effect
|
| 41 |
+
|
| 42 |
+
The `health` branch uses signed object field **684** thresholds: `<25` yields `near death`, `25–49` `badly wounded`, `50–74` `wounded`, and `>=75` `healthy`. Negative values follow the first branch. It then requests a replacement `dead` if sender-record offset **4324** is nonzero **or record offset 4 is >=3**. The `hp` branch first requests `%i` with field 684, then `0` under the same predicate. These are formatting requests; the formatter does not execute in the slices.
|
| 43 |
+
|
| 44 |
+
Thirty-six health fixtures cover -1/0/24/25/49/50/74/75/100 across four state/override combinations. Four hp fixtures preserve raw value 42 while selecting the override. The offset-4 test is **>=3 here**, unlike chat delivery's **==3** dead flag. Do not unify those predicates when porting without further evidence.
|
| 45 |
+
|
| 46 |
+
The `momo` match clears shared buffer byte zero and reads sender-record pointer **6168**. If nonzero, it writes global time **34992532 + 3500** to that referenced object's offset **84**, and **2** to offset **204**. Both NULL and nonzero pointer paths return **0**, with no callee calls in these bounded slices. A near-INT_MAX synthetic time fixture produces signed wrap in the evaluator; original interpreter/JIT conformance and timestamp lifecycle remain open. The referenced object's type, animation/UI meaning, reset/update ownership and who invokes this key are not assigned from the string alone.
|
| 47 |
+
|
| 48 |
+
## Remaining work and Rust dependencies
|
| 49 |
+
|
| 50 |
+
RE-269–271 record parser-state, key-comparison and health/momo milestones. RE-268 remains open for the complete expansion/output contract; RE-272 tracks full capacity-150 copy and append bounds, prefix consumption and buffer aliases, item/ammunition tables, timer arithmetic and cvar identities, location/leader/crosshair callees, momo object writers/consumers and original-runtime/platform fixtures. A Rust implementation will need owned-buffer semantics, exact marker/control-byte behavior and distinct field predicates, backed by completed output specifications before claiming parity.
|
docs/research/README.md
CHANGED
|
@@ -37,10 +37,11 @@ python3 scripts/test_auth_queries.py
|
|
| 37 |
python3 scripts/test_selector_bounds.py
|
| 38 |
python3 scripts/test_chat_paths.py
|
| 39 |
python3 scripts/test_chat_delivery.py
|
|
|
|
| 40 |
python3 scripts/storage_policy.py check --require-setup
|
| 41 |
```
|
| 42 |
|
| 43 |
-
The pipeline runs `fetch_analysis_references.py`, `analyze_qvm.py`, `analyze_native.py`, `analyze_vm_abi.py`, `analyze_native_dispatch.py`, `analyze_vm_extensions.py`, `analyze_native_metadata.py`, `analyze_native_coverage.py`, `analyze_auth_drop.py`, `analyze_deferred_auth.py`, `analyze_engine_control.py`, `analyze_frame_timing.py`, `analyze_event_queues.py`, `analyze_auth_announcements.py`, `analyze_game_cvars.py`, `analyze_cvar_changes.py`, `analyze_auth_queries.py`, `analyze_selector_bounds.py`, `analyze_chat_paths.py`, and `
|
| 44 |
|
| 45 |
Generated JSON/JSONL and compressed evidence are regenerated in place; keep prose notes in Markdown. Rerunning the four analysis stages reproduced every generated report and gzip artifact byte-for-byte on the same toolchain. gzip timestamps and stored filenames are fixed. Different objdump versions can produce different disassembly text, so tool version and uncompressed evidence hashes are recorded. Reports are UTF-8 normal Git text, with individual files below the storage policy's 2 MiB limit.
|
| 46 |
|
|
@@ -82,3 +83,5 @@ The extension collector adds five focused tests for native tail calls, command-r
|
|
| 82 |
[Native command admission and chat paths](CHAT_PATHS.md) adds RE-261–263: native tables/filter evidence, bounded concatenation capacity and recipient front gates. It also corrects the preceding chat third-argument/channel-mode mapping. Reproduce with `python3 scripts/analyze_chat_paths.py`; check with `python3 scripts/test_chat_paths.py`.
|
| 83 |
|
| 84 |
[Chat policy, formats and delivery calls](CHAT_DELIVERY.md) adds RE-265–267: 512 bounded policy paths, mode/formatter/control-byte evidence and echo/fanout calls. Reproduce with `python3 scripts/analyze_chat_delivery.py`; check with `python3 scripts/test_chat_delivery.py`. Field/global lifetime, actual formatted output and runtime delivery remain open.
|
|
|
|
|
|
|
|
|
| 37 |
python3 scripts/test_selector_bounds.py
|
| 38 |
python3 scripts/test_chat_paths.py
|
| 39 |
python3 scripts/test_chat_delivery.py
|
| 40 |
+
python3 scripts/test_chat_expansion.py
|
| 41 |
python3 scripts/storage_policy.py check --require-setup
|
| 42 |
```
|
| 43 |
|
| 44 |
+
The pipeline runs `fetch_analysis_references.py`, `analyze_qvm.py`, `analyze_native.py`, `analyze_vm_abi.py`, `analyze_native_dispatch.py`, `analyze_vm_extensions.py`, `analyze_native_metadata.py`, `analyze_native_coverage.py`, `analyze_auth_drop.py`, `analyze_deferred_auth.py`, `analyze_engine_control.py`, `analyze_frame_timing.py`, `analyze_event_queues.py`, `analyze_auth_announcements.py`, `analyze_game_cvars.py`, `analyze_cvar_changes.py`, `analyze_auth_queries.py`, `analyze_selector_bounds.py`, `analyze_chat_paths.py`, `analyze_chat_delivery.py`, and `analyze_chat_expansion.py` in that order. Python 3.9+ and **LLVM objdump** with x86 ELF/COFF/Mach-O support are required. This pass used Apple LLVM 17.0.0 on macOS arm64; it disassembled x86 images without running them. Upstream references are downloaded only if absent, checked against the committed SHA-256 manifest, and cached under `.cache/research/upstream/`. A clone also needs `git lfs pull` to obtain the derived artifacts/extracted members.
|
| 45 |
|
| 46 |
Generated JSON/JSONL and compressed evidence are regenerated in place; keep prose notes in Markdown. Rerunning the four analysis stages reproduced every generated report and gzip artifact byte-for-byte on the same toolchain. gzip timestamps and stored filenames are fixed. Different objdump versions can produce different disassembly text, so tool version and uncompressed evidence hashes are recorded. Reports are UTF-8 normal Git text, with individual files below the storage policy's 2 MiB limit.
|
| 47 |
|
|
|
|
| 83 |
[Native command admission and chat paths](CHAT_PATHS.md) adds RE-261–263: native tables/filter evidence, bounded concatenation capacity and recipient front gates. It also corrects the preceding chat third-argument/channel-mode mapping. Reproduce with `python3 scripts/analyze_chat_paths.py`; check with `python3 scripts/test_chat_paths.py`.
|
| 84 |
|
| 85 |
[Chat policy, formats and delivery calls](CHAT_DELIVERY.md) adds RE-265–267: 512 bounded policy paths, mode/formatter/control-byte evidence and echo/fanout calls. Reproduce with `python3 scripts/analyze_chat_delivery.py`; check with `python3 scripts/test_chat_delivery.py`. Field/global lifetime, actual formatted output and runtime delivery remain open.
|
| 86 |
+
|
| 87 |
+
[Radio/chat expansion states and keys](CHAT_EXPANSION.md) adds RE-269–271: marker and length branches, all twelve comparison keys, health labels and momo side-effect writes. Reproduce with `python3 scripts/analyze_chat_expansion.py`; check with `python3 scripts/test_chat_expansion.py`. Full composed parser output, buffer bounds and the remaining substitution bodies stay open.
|
docs/research/chat-expansion.json
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
scripts/analyze_binaries.py
CHANGED
|
@@ -14,7 +14,7 @@ def main():
|
|
| 14 |
'analyze_deferred_auth.py', 'analyze_engine_control.py', 'analyze_frame_timing.py',
|
| 15 |
'analyze_event_queues.py', 'analyze_auth_announcements.py', 'analyze_game_cvars.py',
|
| 16 |
'analyze_cvar_changes.py', 'analyze_auth_queries.py', 'analyze_selector_bounds.py',
|
| 17 |
-
'analyze_chat_paths.py', 'analyze_chat_delivery.py'):
|
| 18 |
subprocess.run([sys.executable, str(Path('scripts')/script)], check=True)
|
| 19 |
|
| 20 |
|
|
|
|
| 14 |
'analyze_deferred_auth.py', 'analyze_engine_control.py', 'analyze_frame_timing.py',
|
| 15 |
'analyze_event_queues.py', 'analyze_auth_announcements.py', 'analyze_game_cvars.py',
|
| 16 |
'analyze_cvar_changes.py', 'analyze_auth_queries.py', 'analyze_selector_bounds.py',
|
| 17 |
+
'analyze_chat_paths.py', 'analyze_chat_delivery.py', 'analyze_chat_expansion.py'):
|
| 18 |
subprocess.run([sys.executable, str(Path('scripts')/script)], check=True)
|
| 19 |
|
| 20 |
|
scripts/analyze_chat_expansion.py
ADDED
|
@@ -0,0 +1,95 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Recover bounded radio parser states, key comparisons and substitution branches."""
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
from analyze_deferred_auth import partition_function, trace_slice
|
| 6 |
+
from analyze_game_cvars import vm_string
|
| 7 |
+
from analyze_qvm import decode, write_json
|
| 8 |
+
from analyze_vm_extensions import verified
|
| 9 |
+
|
| 10 |
+
BASE=0x40000000
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def compare(ins,left,right,bound=99999):
|
| 14 |
+
a,b=0x10000,0x20000
|
| 15 |
+
memory={a+i:v for i,v in enumerate(left+b'\0')}
|
| 16 |
+
memory.update({b+i:v for i,v in enumerate(right+b'\0')})
|
| 17 |
+
return trace_slice(ins,69387,69525,{}, {48:a,52:b,56:bound},
|
| 18 |
+
byte_memory=memory,limit=4096,capture_stack=True,halt_on_leave=True)
|
| 19 |
+
|
| 20 |
+
|
| 21 |
+
def main():
|
| 22 |
+
q=json.loads(Path('docs/research/qvm/qagame.json').read_text())
|
| 23 |
+
data=verified(Path(q['source']['member']['output']).read_bytes(),q['sha256']);h,ins=decode(data)
|
| 24 |
+
functions={str(e):partition_function(ins,e,data,h) for e in [69386,185212,186435]}
|
| 25 |
+
keys=[]
|
| 26 |
+
for i in range(185212,186435):
|
| 27 |
+
if ins[i]['opcode']=='CONST' and ins[i]['operand']==69386 and ins[i+1]['opcode']=='CALL':
|
| 28 |
+
literal=ins[i-5]
|
| 29 |
+
if literal['opcode']!='CONST' or ins[i-4]['opcode']!='ARG' or ins[i-4]['operand']!=12:
|
| 30 |
+
raise ValueError('key comparison pattern changed')
|
| 31 |
+
keys.append({'key':vm_string(data,h,literal['operand']),'address':literal['operand'],
|
| 32 |
+
'compare_call':i+1,'bound':ins[i-3]['operand'],'window':ins[i-11:i+8]})
|
| 33 |
+
comparisons=[]
|
| 34 |
+
for k in keys:
|
| 35 |
+
key=k['key'].encode('ascii')
|
| 36 |
+
for candidate in [key,key.upper(),key[:-1],key+b'x']:
|
| 37 |
+
t=compare(ins,candidate,key)
|
| 38 |
+
comparisons.append({'key':k['key'],'candidate':candidate.decode('ascii'),'trace':t})
|
| 39 |
+
# Per-character slices: no copy/append/lookup callee is entered.
|
| 40 |
+
chars=[]
|
| 41 |
+
for marker,key,ch in [(0,'',ord('x')),(0,'',36),(36,'foo',35),(35,'',ord('a')),
|
| 42 |
+
(36,'h',ord('P')),(36,'foo',ord('1')),(36,'foo',32),
|
| 43 |
+
(36,'foo',255),(0,'',0)]:
|
| 44 |
+
bm={BASE+32:ch,BASE+182:marker}
|
| 45 |
+
bm.update({BASE+183+i:v for i,v in enumerate(key.encode()+b'\0')})
|
| 46 |
+
t=trace_slice(ins,186481,186676,{}, {28:0,336:len(key),376:0x50000,380:0x60000},byte_memory=bm)
|
| 47 |
+
chars.append({'pending_marker':marker,'pending_key':key,'byte':ch,'trace':t})
|
| 48 |
+
lookup_results=[]
|
| 49 |
+
for result in [0,0x70000]:
|
| 50 |
+
t=trace_slice(ins,186609,186676,{}, {340:result,380:0x60000},byte_memory={BASE+182:36})
|
| 51 |
+
lookup_results.append({'supplied_lookup_return':result,'trace':t})
|
| 52 |
+
tails=[]
|
| 53 |
+
for marker in [0,35,36]:
|
| 54 |
+
t=trace_slice(ins,186693,186719,{}, {380:0x60000},byte_memory={BASE+182:marker})
|
| 55 |
+
tails.append({'pending_marker':marker,'trace':t})
|
| 56 |
+
lengths=[]
|
| 57 |
+
for length in [149,150,151]:
|
| 58 |
+
t=trace_slice(ins,186449,186719,{}, {344:length,380:0x60000},
|
| 59 |
+
byte_memory={BASE+182:0},limit=256)
|
| 60 |
+
lengths.append({'supplied_source_length':length,'trace':t})
|
| 61 |
+
entity,record=0x10000,0x20000
|
| 62 |
+
health=[]
|
| 63 |
+
for value in [-1,0,24,25,49,50,74,75,100]:
|
| 64 |
+
for state,override in [(0,0),(3,0),(4,0),(0,1)]:
|
| 65 |
+
t=trace_slice(ins,185237,186431,{entity+684:value,entity+520:record,
|
| 66 |
+
record+4:state,record+4324:override},{1144:entity})
|
| 67 |
+
health.append({'raw_health':value,'raw_state':state,'override':override,'trace':t})
|
| 68 |
+
hp=[]
|
| 69 |
+
for state,override in [(0,0),(3,0),(4,0),(0,1)]:
|
| 70 |
+
t=trace_slice(ins,185350,186431,{entity+684:42,entity+520:record,record+4:state,
|
| 71 |
+
record+4324:override},{1144:entity})
|
| 72 |
+
hp.append({'raw_state':state,'override':override,'trace':t})
|
| 73 |
+
side_effects=[]
|
| 74 |
+
for pointer,now in [(0,1000),(0x30000,1000),(0x30000,2147483640)]:
|
| 75 |
+
t=trace_slice(ins,185987,186031,{entity+520:record,record+6168:pointer,34992532:now},
|
| 76 |
+
{1144:entity},byte_memory={},capture_stack=True,halt_on_leave=True)
|
| 77 |
+
side_effects.append({'object_pointer':pointer,'now':now,'trace':t})
|
| 78 |
+
addresses=[77862,77857,77850,77847,78120,78109,78095,78087,78079,78074,78068,
|
| 79 |
+
78066,78037,78014,78009,78000,77991,77972,77959,77936]
|
| 80 |
+
addresses.extend(k['address'] for k in keys)
|
| 81 |
+
strings={str(a):vm_string(data,h,a) for a in addresses}
|
| 82 |
+
out={'source':q['source'],'sha256':q['sha256'],'functions':functions,'keys':keys,
|
| 83 |
+
'comparison_traces':comparisons,'character_traces':chars,'lookup_result_traces':lookup_results,
|
| 84 |
+
'pending_tail_traces':tails,'source_length_traces':lengths,'health_traces':health,'hp_traces':hp,
|
| 85 |
+
'momo_traces':side_effects,'strings':strings,
|
| 86 |
+
'limitations':'Static partitions and bounded synthetic instruction slices. No original process or full '
|
| 87 |
+
'QVM executes. CALLs return zero placeholders; source lengths and isolated lookup returns are supplied '
|
| 88 |
+
'inputs, not executed callees. No composed parser output or formatting/copy/append execution. '
|
| 89 |
+
'Timer, weapon, location, leader and crosshair bodies are retained but not fully evaluated; field '
|
| 90 |
+
'identities, ownership, marker lifetime and original runtime/platform behavior remain open.'}
|
| 91 |
+
write_json(Path('docs/research/chat-expansion.json'),out)
|
| 92 |
+
print(f'Recovered {len(keys)} substitution keys, parser state and health/momo slices')
|
| 93 |
+
|
| 94 |
+
|
| 95 |
+
if __name__=='__main__':main()
|
scripts/test_chat_expansion.py
ADDED
|
@@ -0,0 +1,90 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Check radio parser slices and substitution evidence without executing callees."""
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import unittest
|
| 6 |
+
from analyze_chat_expansion import compare, BASE
|
| 7 |
+
from analyze_qvm import decode
|
| 8 |
+
from analyze_vm_extensions import verified
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class ChatExpansionTests(unittest.TestCase):
|
| 12 |
+
@classmethod
|
| 13 |
+
def setUpClass(cls):
|
| 14 |
+
cls.r=json.loads(Path('docs/research/chat-expansion.json').read_text())
|
| 15 |
+
q=cls.r
|
| 16 |
+
cls.h,cls.ins=decode(verified(Path(q['source']['member']['output']).read_bytes(),q['sha256']))
|
| 17 |
+
|
| 18 |
+
def test_keys_and_comparison_calls(self):
|
| 19 |
+
self.assertEqual([k['key'] for k in self.r['keys']],['health','hp','clips','bullets','weapon',
|
| 20 |
+
'location','gameitem','gametime','roundtime','leader','momo','crosshair'])
|
| 21 |
+
for k in self.r['keys']:
|
| 22 |
+
self.assertEqual(k['bound'],99999)
|
| 23 |
+
self.assertEqual(self.ins[k['compare_call']-1]['operand'],69386)
|
| 24 |
+
for x in self.r['comparison_traces']:
|
| 25 |
+
self.assertEqual(x['trace']['stack']==[0],x['candidate'].upper()==x['key'].upper())
|
| 26 |
+
|
| 27 |
+
def test_comparison_bounds_signed_bytes_and_nulls(self):
|
| 28 |
+
self.assertEqual(compare(self.ins,b'ab',b'ac',1)['stack'],[0])
|
| 29 |
+
self.assertEqual(compare(self.ins,b'ab',b'ac',2)['stack'],[-1])
|
| 30 |
+
self.assertEqual(compare(self.ins,b'\xff',b'A')['stack'],[-1])
|
| 31 |
+
self.assertEqual(compare(self.ins,b'a\0x',b'a\0y')['stack'],[0])
|
| 32 |
+
with self.assertRaises(ValueError):compare(self.ins,b'x'*100,b'x'*100)
|
| 33 |
+
|
| 34 |
+
def test_character_marker_and_letter_states(self):
|
| 35 |
+
c=self.r['character_traces']
|
| 36 |
+
self.assertEqual([x['trace']['calls'][0]['target'] if x['trace']['calls'] else None for x in c],
|
| 37 |
+
[70458,None,70458,185212,185212,70458,70458,70458,70458])
|
| 38 |
+
self.assertEqual(c[2]['trace']['calls'][0]['args']['8'],77857)
|
| 39 |
+
self.assertEqual(c[2]['trace']['byte_memory'][str(BASE+182)],35)
|
| 40 |
+
self.assertEqual(c[3]['trace']['byte_memory'][str(BASE+183)],ord('a'))
|
| 41 |
+
self.assertEqual(c[4]['trace']['byte_memory'][str(BASE+184)],ord('P'))
|
| 42 |
+
for x in c[5:8]:
|
| 43 |
+
self.assertEqual(x['trace']['calls'][0]['args']['8'],77850)
|
| 44 |
+
self.assertEqual(x['trace']['byte_memory'][str(BASE+182)],0)
|
| 45 |
+
self.assertEqual(c[7]['trace']['calls'][0]['args']['20'],-1)
|
| 46 |
+
|
| 47 |
+
def test_lookup_result_and_pending_tail(self):
|
| 48 |
+
a,b=self.r['lookup_result_traces']
|
| 49 |
+
self.assertEqual(a['trace']['calls'],[])
|
| 50 |
+
self.assertEqual(a['trace']['byte_memory'][str(BASE+182)],36)
|
| 51 |
+
self.assertEqual(b['trace']['calls'][0]['args']['12'],0x70000)
|
| 52 |
+
self.assertEqual(b['trace']['byte_memory'][str(BASE+182)],0)
|
| 53 |
+
self.assertEqual([len(x['trace']['calls']) for x in self.r['pending_tail_traces']],[0,2,2])
|
| 54 |
+
self.assertEqual(self.r['strings']['77857'],'%c%s')
|
| 55 |
+
self.assertEqual(self.r['strings']['77850'],'%c%s%c')
|
| 56 |
+
|
| 57 |
+
def test_length_guard_and_placeholder_copy(self):
|
| 58 |
+
rows=self.r['source_length_traces']
|
| 59 |
+
self.assertEqual([x['supplied_source_length'] for x in rows],[149,150,151])
|
| 60 |
+
self.assertEqual([[c['target'] for c in x['trace']['calls']] for x in rows],
|
| 61 |
+
[[70386,36038],[70386,36038],[2294]])
|
| 62 |
+
for x in rows[:2]:
|
| 63 |
+
self.assertEqual(x['trace']['calls'][0]['args']['12'],150)
|
| 64 |
+
self.assertEqual(x['trace']['byte_writes'],[{'address':0x60000,'value':0}])
|
| 65 |
+
self.assertEqual(rows[2]['trace']['byte_writes'],[])
|
| 66 |
+
|
| 67 |
+
def test_health_labels_boundaries_and_death_override(self):
|
| 68 |
+
for x in self.r['health_traces']:
|
| 69 |
+
v=x['raw_health'];expected='near death' if v<25 else 'badly wounded' if v<50 else 'wounded' if v<75 else 'healthy'
|
| 70 |
+
labels=[self.r['strings'][str(c['args']['16'])] for c in x['trace']['calls']]
|
| 71 |
+
self.assertEqual(labels,[expected]+(['dead'] if x['raw_state']>=3 or x['override'] else []))
|
| 72 |
+
self.assertTrue(all(c['args']['8']==35738436 and c['args']['12']==150 for c in x['trace']['calls']))
|
| 73 |
+
for x in self.r['hp_traces']:
|
| 74 |
+
calls=x['trace']['calls'];self.assertEqual(calls[0]['args']['20'],42)
|
| 75 |
+
self.assertEqual(len(calls),2 if x['raw_state']>=3 or x['override'] else 1)
|
| 76 |
+
if len(calls)==2:self.assertEqual(self.r['strings'][str(calls[1]['args']['16'])],'0')
|
| 77 |
+
|
| 78 |
+
def test_momo_writes_and_zero_return(self):
|
| 79 |
+
for x in self.r['momo_traces']:
|
| 80 |
+
t=x['trace'];self.assertEqual(t['stack'],[0]);self.assertEqual(t['calls'],[])
|
| 81 |
+
self.assertEqual(t['byte_writes'],[{'address':35738436,'value':0}])
|
| 82 |
+
p=x['object_pointer']
|
| 83 |
+
if p:
|
| 84 |
+
expected=(x['now']+3500+2**31)%2**32-2**31
|
| 85 |
+
self.assertEqual(t['memory'][str(p+84)],expected)
|
| 86 |
+
self.assertEqual(t['memory'][str(p+204)],2)
|
| 87 |
+
else:self.assertNotIn('84',t['memory'])
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
if __name__=='__main__':unittest.main()
|