paavohtl Claude Opus 4.6 (1M context) commited on
Commit
8962003
Β·
1 Parent(s): b0c95b2

docs: update desync investigation with full resolution journal

Browse files

Expands the document from "partial fix, investigation continues" to the
complete seven-bug resolution story. Adds phases 2-4 covering RNG tracing,
sub-object hashing breakthrough, hook binary search, and all root causes.
New lessons on Ghidra element indexing, buffer layout verification, and
the value of multiple test replays.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Files changed (1) hide show
  1. docs/re-notes/desync-investigation.md +141 -12
docs/re-notes/desync-investigation.md CHANGED
@@ -1,8 +1,8 @@
1
- # The debris.spr Desync: Debugging a Replay Mismatch in a Reimplemented Game Constructor
2
 
3
  ## Summary
4
 
5
- During the incremental Rust reimplementation of Worms Armageddon's DDGame constructor, a deterministic replay of a longbow match produced "Checksum Mismatch" errors starting at frame 1350. Other replays (bots 3v3, shotgun) passed. This documents the investigation that ultimately traced the root cause to a single missing sprite load (`debris.spr`) in headless mode, and the debugging methodology used to find it.
6
 
7
  ## Background
8
 
@@ -89,26 +89,155 @@ DDDisplay::load_sprite(disp, 3, 0x26E, 0, land_layer, c"debris.spr".as_ptr());
89
 
90
  This change is correct β€” the original constructor loads `debris.spr` unconditionally, and our code should match. However, it did **not** resolve the desync. An initial test appeared to show 3 of 4 mismatches fixed, but this was a `tail -5` truncation error β€” only the last mismatch was visible, creating a false impression. All 4 mismatches persist.
91
 
92
- The display state differences (sprite layer counters) we found were real and worth fixing for correctness, but they turned out not to be the root cause of the checksum mismatch. The investigation continues.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
93
 
94
  ## Debugging Tools Built
95
 
96
  The investigation produced several reusable tools:
97
 
98
- - **Hardware watchpoint system** with EBP-based stack trace walking (VEH + DR0–DR3)
99
- - **Per-frame game state hashing** via forced `SerializeGameState` calls
 
 
100
  - **Binary sub-object dumping** for display, GfxHandler, PCLandscape comparison
101
- - **`OPENWA_WATCH_DISPLAY=1`** env var to arm watchpoints on the display object during construction
102
- - Terrain collision bitmap dumping at arbitrary frames
103
 
104
  ## Key Lessons
105
 
106
- 1. **Identical struct memory β‰  identical behaviour.** The constructor's side effects on *other* objects matter as much as the struct it fills. Compare sub-objects systematically.
 
 
 
 
 
 
 
 
107
 
108
- 2. **Validate your diff tools.** Our snapshot comparison showed ~490 "differences" at known-good frames β€” pure noise from pointer canonicalization heuristics. Always baseline.
109
 
110
- 3. **There is no "visual-only" code in a deterministic game engine.** `debris.spr` seems purely decorative β€” it controls what debris particles look like when terrain is destroyed. But Worms uses a **single shared RNG** for everything: gameplay physics, AI decisions, particle effects, sound variation. There is no separate "visual RNG." Every RNG consumer must execute identically, or the streams diverge and every subsequent random outcome differs. Skipping a single sprite load changed how `GenerateDebrisParticles` consumed RNG ticks, shifting the entire RNG sequence for all downstream gameplay.
111
 
112
- 4. **Hardware watchpoints are underrated.** x86 DR0–DR3 with a VEH handler gives "what wrote this byte?" answers in seconds, without an external debugger. Adding stack traces made them even more powerful.
113
 
114
- 5. **Binary search on frames, not code.** Per-frame RNG logging narrowed 542 candidate frames to the exact frame (818) where divergence began, before any code analysis.
 
1
+ # Desync Investigation Journal: Seven Bugs, One Checksum Mismatch
2
 
3
  ## Summary
4
 
5
+ During the incremental Rust reimplementation of Worms Armageddon's DDGame constructor and gameplay hooks, a deterministic replay of a longbow match produced "Checksum Mismatch" errors. What appeared to be a single constructor bug turned out to be seven independent issues across three subsystems (constructor, replay loader, weapon dispatch), found over multiple debugging sessions using hardware watchpoints, sub-object hashing, binary search on hooks, and targeted test replays.
6
 
7
  ## Background
8
 
 
89
 
90
  This change is correct β€” the original constructor loads `debris.spr` unconditionally, and our code should match. However, it did **not** resolve the desync. An initial test appeared to show 3 of 4 mismatches fixed, but this was a `tail -5` truncation error β€” only the last mismatch was visible, creating a false impression. All 4 mismatches persist.
91
 
92
+ The display state differences (sprite layer counters) we found were real and worth fixing for correctness, but they turned out not to be the root cause of the checksum mismatch.
93
+
94
+ ---
95
+
96
+ ## Phase 2: RNG Tracing and the Debris Particle Divergence
97
+
98
+ ### Per-frame RNG tracking
99
+
100
+ With the sprite fix applied but the desync persisting, the next step was tracking the game RNG (DDGame+0x45EC) at frame boundaries. A hook on `TurnManager_ProcessFrame` logged the RNG value before and after each frame:
101
+
102
+ ```
103
+ game_f=817 rng=B714AF97 (identical in both runs)
104
+ game_f=818 rng=BE0D887F (Rust) vs ADB76793 (orig)
105
+ ```
106
+
107
+ The RNG was **identical through frame 817** and diverged during frame 818. But `AdvanceGameRNG` (0x53F320) turned out to be **mostly inlined** β€” hooking the function only caught rare non-inlined calls. The real tool was a hardware watchpoint on DDGame+0x45EC itself.
108
+
109
+ ### Hardware watchpoint on the RNG field
110
+
111
+ Arming DR0 on DDGame+0x45EC at frame 817 captured every RNG write with its EIP. Comparing the two runs:
112
+
113
+ | Write # | Rust EIP | Orig EIP | Notes |
114
+ |---------|----------|----------|-------|
115
+ | 1–24 | 0x5470BB/DE | 0x5470BB/DE | Identical (GenerateDebrisParticles inlined) |
116
+ | 25 | **0x6529960D** (hook) | **0x5470BB** (more debris!) | Divergence point |
117
+
118
+ The original had **36 debris writes** (3 Γ— GenerateDebrisParticles calls) while Rust had only **24** (2 calls). One terrain collision event was missing. Yet total debris writes across the game were identical (72 each) β€” just distributed differently across frames.
119
+
120
+ ### Caller frequency analysis
121
+
122
+ Counting all 17K+ RNG write EIPs revealed the dominant divergence source: `AdvanceGameRNG_Low16` (0x507CE0) in `CTaskFire__HandleMessage` β€” 14,085 calls (Rust) vs 12,517 (orig). The 1,568 extra calls represented projectiles processing more frames due to the cascading RNG shift.
123
+
124
+ ## Phase 3: The Flat Memory Puzzle
125
+
126
+ At this point, DDGame flat memory was verified **byte-for-byte identical** between constructors. Display, landscape, and wrapper non-pointer values all matched at frame 0. The desync was deterministic. What could differ?
127
+
128
+ ### Sub-object hashing: the breakthrough
129
+
130
+ The key insight: DDGame contains **pointers** to heap-allocated sub-objects. Flat memory comparison only shows the pointer values (which differ between runs due to heap layout), not the *content* of what they point to.
131
+
132
+ A new tool β€” `hash_pointer_targets()` β€” walks every DWORD in DDGame, follows each heap pointer, and hashes the first 256 bytes of the target with pointer canonicalization (replacing pointer-looking values with 0).
133
+
134
+ The first run produced hundreds of hash differences due to an overly aggressive pointer filter. After tuning to use the existing `is_likely_pointer` heuristic, a targeted raw dump of specific sub-objects revealed the smoking gun:
135
+
136
+ **Arrow collision region** (SpriteRegion at DDGame+0x48C):
137
+
138
+ | Offset | Rust | Original |
139
+ |--------|------|----------|
140
+ | +0x14 (this[5]) | **0** | **10** |
141
+ | +0x18 (this[6]) | **20** | **10** |
142
+
143
+ Different collision box dimensions. The Rust constructor was creating origin-based collision boxes instead of centered boxes with 10px margins.
144
+
145
+ ### The root cause: wrong SpriteRegion parameters
146
+
147
+ Tracing the SpriteRegion constructor (0x57DB20) parameters via disassembly:
148
+
149
+ ```
150
+ this[3] = p4 - p2 (width)
151
+ this[4] = EDX - p3 (height)
152
+ this[5] = ECX - p2 (x inset)
153
+ this[6] = p5 - p3 (y inset)
154
+ ```
155
+
156
+ The original computes a **centered** collision box:
157
+ - ECX = `sprite_w / 2`, EDX = `sprite_h - margin_h`
158
+ - p2 = `margin_w`, p3 = `margin_h` (where margin = max(0, dim/2 βˆ’ 10))
159
+ - p4 = `sprite_w - margin_w`, p5 = `sprite_h / 2`
160
+ - p6 = **arrow sprite pointer**
161
+
162
+ Our code passed:
163
+ - ECX = **0**, EDX = `margin_h`
164
+ - p2 = **0**, p3 = **0**
165
+ - p4 = `margin_w`, p5 = `margin_h`
166
+ - p6 = **landscape gfx_resource** (wrong object entirely β€” from an outer scope)
167
+
168
+ Six of eight parameters were wrong. The collision boxes were origin-based instead of centered, AND referenced the wrong sprite, causing different terrain collision detection for arrow projectiles.
169
+
170
+ ## Phase 4: It Wasn't Just the Constructor
171
+
172
+ ### New test replays change everything
173
+
174
+ Recording two additional replays proved crucial:
175
+ - **longbow_water** β€” longbow fired at water (no terrain collision)
176
+ - **longbow_alt_theme** β€” longbow on a different WA theme
177
+
178
+ Expected logs were generated from **unmodded WA.exe** (critical β€” our DLL's hooks can affect the game log).
179
+
180
+ Results with the original constructor:
181
+
182
+ | Replay | Status |
183
+ |--------|--------|
184
+ | longbow_water | **FAIL** |
185
+ | longbow_alt_theme | **FAIL** (Frame 0!) |
186
+
187
+ Both failed with the *original* constructor! The desync wasn't constructor-specific β€” it was caused by **gameplay hooks**.
188
+
189
+ ### Binary search on hooks
190
+
191
+ Disabling hooks in groups narrowed the search:
192
+
193
+ 1. All hooks disabled except essentials β†’ PASS
194
+ 2. Add weapon hooks β†’ FAIL
195
+ 3. Only weapon hooks, disable FireWeapon β†’ PASS for longbow_water
196
+ 4. Only replay hooks β†’ FAIL for alt_theme
197
+
198
+ Two independent hook bugs:
199
+
200
+ **ReplayLoader (replay.rs):**
201
+ - Player array strides: Ghidra showed element indices (`i*0x3C` for `u16*`, `i*0x1E` for `u32*`) but code used them as byte offsets. Correct byte stride: `i*0x78`.
202
+ - Non-zero `map_seed` path: per-team weapon config reads from stream were unimplemented. Fallback to original parser for `map_seed β‰  0`.
203
+
204
+ **FireWeapon (weapon.rs):**
205
+ - Binary search within the type-4 dispatch: type 4 β†’ subtypes 10+13 β†’ subtype 13 (Napalm Strike).
206
+ - `fire_send_team_message` wrote `team_index` to `buf[8..12]` but the original writes it to `buf[0..4]`. The HandleMessage handler reads `data[0]`, receiving 0 instead of the correct team index.
207
+ - Root cause confirmed by disassembly of the original Napalm handler at 0x51E5C0: `MOV [ESP+4], EAX` where EAX = `worm+0xFC` (team_index) and `ESP+4` = `buffer[0]`.
208
+
209
+ ### Also fixed along the way
210
+
211
+ - **DDGame allocation size**: 0x98B8 β†’ 0x98D8 (original mallocs 0x98D8 but memsets only 0x98B8)
212
+ - **Wrong function after InitPaletteGradientSprites**: called `LoadingProgressTick` (0x5717A0) instead of `DisplayGfx__InitTeamPaletteDisplayObjects` (0x5703E0)
213
 
214
  ## Debugging Tools Built
215
 
216
  The investigation produced several reusable tools:
217
 
218
+ - **Hardware watchpoint system** (`debug_watchpoint.rs`) β€” x86 DR0–DR3 with VEH handler and EBP stack trace walking
219
+ - **Sub-object hashing** (`snapshot.rs: hash_pointer_targets`) β€” follows heap pointers in a struct, hashes target content with pointer canonicalization. Integrated into the snapshot system.
220
+ - **Per-frame RNG logging** via TurnManager hook
221
+ - **`OPENWA_WATCH_DISPLAY=1`** / **`OPENWA_WATCH_FRAME=N`** β€” arm watchpoints on display or DDGame at specific frames
222
  - **Binary sub-object dumping** for display, GfxHandler, PCLandscape comparison
223
+ - **A/B constructor toggle** (`OPENWA_USE_ORIG_CTOR=1`) β€” instant switching between Rust and original constructors
 
224
 
225
  ## Key Lessons
226
 
227
+ 1. **Identical struct memory β‰  identical behaviour.** DDGame flat memory was byte-identical, but *sub-objects pointed to by DDGame* had different content. The `hash_pointer_targets` tool was purpose-built to catch this.
228
+
229
+ 2. **Validate your diff tools.** Our snapshot comparison showed ~490 "differences" at known-good frames β€” pure noise from pointer canonicalization heuristics. And our dump code once read past a heap allocation into adjacent memory, producing false diffs. Always baseline.
230
+
231
+ 3. **There is no "visual-only" code in a deterministic game engine.** `debris.spr` seems purely decorative, but Worms uses a **single shared RNG** for everything. Skipping one sprite load shifts the entire RNG sequence.
232
+
233
+ 4. **Hardware watchpoints are underrated.** x86 DR0–DR3 with a VEH handler gives "what wrote this byte?" answers in seconds, without an external debugger.
234
+
235
+ 5. **Binary search on frames, not code.** Per-frame RNG logging narrowed 542 candidate frames to the exact frame (818) where divergence began.
236
 
237
+ 6. **Binary search on hooks.** Disabling half the hooks at a time isolated two independent bugs in minutes. Then within a single hook (FireWeapon), binary search on dispatch paths (type β†’ subtype β†’ specific handler) pinpointed the exact buggy function.
238
 
239
+ 7. **Multiple test replays expose different bugs.** The original longbow replay only triggered the constructor bug. Adding longbow_water and longbow_alt_theme exposed hook bugs that were invisible with the original replay alone. Different weapons, themes, and targets exercise different code paths.
240
 
241
+ 8. **Ghidra element indexing is a trap.** The decompiler shows `(&DAT[i*0x3C])` for a `u16*` array β€” the `0x3C` is an *element* index, not a byte offset. The actual byte offset is `i * 0x3C * sizeof(u16) = i * 0x78`. This caused player data corruption for index 1+.
242
 
243
+ 9. **Check the buffer layout against disassembly.** The Napalm Strike bug was a single wrong offset in a message buffer: `buf[8]` vs `buf[0]`. The original's disassembly (`MOV [ESP+4], EAX` at 0x51E5D2) unambiguously shows the write destination. When in doubt, read the assembly.