docs: update desync investigation with full resolution journal
Browse filesExpands the document from "partial fix, investigation continues" to the
complete seven-bug resolution story. Adds phases 2-4 covering RNG tracing,
sub-object hashing breakthrough, hook binary search, and all root causes.
New lessons on Ghidra element indexing, buffer layout verification, and
the value of multiple test replays.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
docs/re-notes/desync-investigation.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
| 1 |
-
#
|
| 2 |
|
| 3 |
## Summary
|
| 4 |
|
| 5 |
-
During the incremental Rust reimplementation of Worms Armageddon's DDGame constructor, a deterministic replay of a longbow match produced "Checksum Mismatch" errors
|
| 6 |
|
| 7 |
## Background
|
| 8 |
|
|
@@ -89,26 +89,155 @@ DDDisplay::load_sprite(disp, 3, 0x26E, 0, land_layer, c"debris.spr".as_ptr());
|
|
| 89 |
|
| 90 |
This change is correct β the original constructor loads `debris.spr` unconditionally, and our code should match. However, it did **not** resolve the desync. An initial test appeared to show 3 of 4 mismatches fixed, but this was a `tail -5` truncation error β only the last mismatch was visible, creating a false impression. All 4 mismatches persist.
|
| 91 |
|
| 92 |
-
The display state differences (sprite layer counters) we found were real and worth fixing for correctness, but they turned out not to be the root cause of the checksum mismatch.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 93 |
|
| 94 |
## Debugging Tools Built
|
| 95 |
|
| 96 |
The investigation produced several reusable tools:
|
| 97 |
|
| 98 |
-
- **Hardware watchpoint system** with EBP
|
| 99 |
-
- **
|
|
|
|
|
|
|
| 100 |
- **Binary sub-object dumping** for display, GfxHandler, PCLandscape comparison
|
| 101 |
-
- **`
|
| 102 |
-
- Terrain collision bitmap dumping at arbitrary frames
|
| 103 |
|
| 104 |
## Key Lessons
|
| 105 |
|
| 106 |
-
1. **Identical struct memory β identical behaviour.**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 107 |
|
| 108 |
-
|
| 109 |
|
| 110 |
-
|
| 111 |
|
| 112 |
-
|
| 113 |
|
| 114 |
-
|
|
|
|
| 1 |
+
# Desync Investigation Journal: Seven Bugs, One Checksum Mismatch
|
| 2 |
|
| 3 |
## Summary
|
| 4 |
|
| 5 |
+
During the incremental Rust reimplementation of Worms Armageddon's DDGame constructor and gameplay hooks, a deterministic replay of a longbow match produced "Checksum Mismatch" errors. What appeared to be a single constructor bug turned out to be seven independent issues across three subsystems (constructor, replay loader, weapon dispatch), found over multiple debugging sessions using hardware watchpoints, sub-object hashing, binary search on hooks, and targeted test replays.
|
| 6 |
|
| 7 |
## Background
|
| 8 |
|
|
|
|
| 89 |
|
| 90 |
This change is correct β the original constructor loads `debris.spr` unconditionally, and our code should match. However, it did **not** resolve the desync. An initial test appeared to show 3 of 4 mismatches fixed, but this was a `tail -5` truncation error β only the last mismatch was visible, creating a false impression. All 4 mismatches persist.
|
| 91 |
|
| 92 |
+
The display state differences (sprite layer counters) we found were real and worth fixing for correctness, but they turned out not to be the root cause of the checksum mismatch.
|
| 93 |
+
|
| 94 |
+
---
|
| 95 |
+
|
| 96 |
+
## Phase 2: RNG Tracing and the Debris Particle Divergence
|
| 97 |
+
|
| 98 |
+
### Per-frame RNG tracking
|
| 99 |
+
|
| 100 |
+
With the sprite fix applied but the desync persisting, the next step was tracking the game RNG (DDGame+0x45EC) at frame boundaries. A hook on `TurnManager_ProcessFrame` logged the RNG value before and after each frame:
|
| 101 |
+
|
| 102 |
+
```
|
| 103 |
+
game_f=817 rng=B714AF97 (identical in both runs)
|
| 104 |
+
game_f=818 rng=BE0D887F (Rust) vs ADB76793 (orig)
|
| 105 |
+
```
|
| 106 |
+
|
| 107 |
+
The RNG was **identical through frame 817** and diverged during frame 818. But `AdvanceGameRNG` (0x53F320) turned out to be **mostly inlined** β hooking the function only caught rare non-inlined calls. The real tool was a hardware watchpoint on DDGame+0x45EC itself.
|
| 108 |
+
|
| 109 |
+
### Hardware watchpoint on the RNG field
|
| 110 |
+
|
| 111 |
+
Arming DR0 on DDGame+0x45EC at frame 817 captured every RNG write with its EIP. Comparing the two runs:
|
| 112 |
+
|
| 113 |
+
| Write # | Rust EIP | Orig EIP | Notes |
|
| 114 |
+
|---------|----------|----------|-------|
|
| 115 |
+
| 1β24 | 0x5470BB/DE | 0x5470BB/DE | Identical (GenerateDebrisParticles inlined) |
|
| 116 |
+
| 25 | **0x6529960D** (hook) | **0x5470BB** (more debris!) | Divergence point |
|
| 117 |
+
|
| 118 |
+
The original had **36 debris writes** (3 Γ GenerateDebrisParticles calls) while Rust had only **24** (2 calls). One terrain collision event was missing. Yet total debris writes across the game were identical (72 each) β just distributed differently across frames.
|
| 119 |
+
|
| 120 |
+
### Caller frequency analysis
|
| 121 |
+
|
| 122 |
+
Counting all 17K+ RNG write EIPs revealed the dominant divergence source: `AdvanceGameRNG_Low16` (0x507CE0) in `CTaskFire__HandleMessage` β 14,085 calls (Rust) vs 12,517 (orig). The 1,568 extra calls represented projectiles processing more frames due to the cascading RNG shift.
|
| 123 |
+
|
| 124 |
+
## Phase 3: The Flat Memory Puzzle
|
| 125 |
+
|
| 126 |
+
At this point, DDGame flat memory was verified **byte-for-byte identical** between constructors. Display, landscape, and wrapper non-pointer values all matched at frame 0. The desync was deterministic. What could differ?
|
| 127 |
+
|
| 128 |
+
### Sub-object hashing: the breakthrough
|
| 129 |
+
|
| 130 |
+
The key insight: DDGame contains **pointers** to heap-allocated sub-objects. Flat memory comparison only shows the pointer values (which differ between runs due to heap layout), not the *content* of what they point to.
|
| 131 |
+
|
| 132 |
+
A new tool β `hash_pointer_targets()` β walks every DWORD in DDGame, follows each heap pointer, and hashes the first 256 bytes of the target with pointer canonicalization (replacing pointer-looking values with 0).
|
| 133 |
+
|
| 134 |
+
The first run produced hundreds of hash differences due to an overly aggressive pointer filter. After tuning to use the existing `is_likely_pointer` heuristic, a targeted raw dump of specific sub-objects revealed the smoking gun:
|
| 135 |
+
|
| 136 |
+
**Arrow collision region** (SpriteRegion at DDGame+0x48C):
|
| 137 |
+
|
| 138 |
+
| Offset | Rust | Original |
|
| 139 |
+
|--------|------|----------|
|
| 140 |
+
| +0x14 (this[5]) | **0** | **10** |
|
| 141 |
+
| +0x18 (this[6]) | **20** | **10** |
|
| 142 |
+
|
| 143 |
+
Different collision box dimensions. The Rust constructor was creating origin-based collision boxes instead of centered boxes with 10px margins.
|
| 144 |
+
|
| 145 |
+
### The root cause: wrong SpriteRegion parameters
|
| 146 |
+
|
| 147 |
+
Tracing the SpriteRegion constructor (0x57DB20) parameters via disassembly:
|
| 148 |
+
|
| 149 |
+
```
|
| 150 |
+
this[3] = p4 - p2 (width)
|
| 151 |
+
this[4] = EDX - p3 (height)
|
| 152 |
+
this[5] = ECX - p2 (x inset)
|
| 153 |
+
this[6] = p5 - p3 (y inset)
|
| 154 |
+
```
|
| 155 |
+
|
| 156 |
+
The original computes a **centered** collision box:
|
| 157 |
+
- ECX = `sprite_w / 2`, EDX = `sprite_h - margin_h`
|
| 158 |
+
- p2 = `margin_w`, p3 = `margin_h` (where margin = max(0, dim/2 β 10))
|
| 159 |
+
- p4 = `sprite_w - margin_w`, p5 = `sprite_h / 2`
|
| 160 |
+
- p6 = **arrow sprite pointer**
|
| 161 |
+
|
| 162 |
+
Our code passed:
|
| 163 |
+
- ECX = **0**, EDX = `margin_h`
|
| 164 |
+
- p2 = **0**, p3 = **0**
|
| 165 |
+
- p4 = `margin_w`, p5 = `margin_h`
|
| 166 |
+
- p6 = **landscape gfx_resource** (wrong object entirely β from an outer scope)
|
| 167 |
+
|
| 168 |
+
Six of eight parameters were wrong. The collision boxes were origin-based instead of centered, AND referenced the wrong sprite, causing different terrain collision detection for arrow projectiles.
|
| 169 |
+
|
| 170 |
+
## Phase 4: It Wasn't Just the Constructor
|
| 171 |
+
|
| 172 |
+
### New test replays change everything
|
| 173 |
+
|
| 174 |
+
Recording two additional replays proved crucial:
|
| 175 |
+
- **longbow_water** β longbow fired at water (no terrain collision)
|
| 176 |
+
- **longbow_alt_theme** β longbow on a different WA theme
|
| 177 |
+
|
| 178 |
+
Expected logs were generated from **unmodded WA.exe** (critical β our DLL's hooks can affect the game log).
|
| 179 |
+
|
| 180 |
+
Results with the original constructor:
|
| 181 |
+
|
| 182 |
+
| Replay | Status |
|
| 183 |
+
|--------|--------|
|
| 184 |
+
| longbow_water | **FAIL** |
|
| 185 |
+
| longbow_alt_theme | **FAIL** (Frame 0!) |
|
| 186 |
+
|
| 187 |
+
Both failed with the *original* constructor! The desync wasn't constructor-specific β it was caused by **gameplay hooks**.
|
| 188 |
+
|
| 189 |
+
### Binary search on hooks
|
| 190 |
+
|
| 191 |
+
Disabling hooks in groups narrowed the search:
|
| 192 |
+
|
| 193 |
+
1. All hooks disabled except essentials β PASS
|
| 194 |
+
2. Add weapon hooks β FAIL
|
| 195 |
+
3. Only weapon hooks, disable FireWeapon β PASS for longbow_water
|
| 196 |
+
4. Only replay hooks β FAIL for alt_theme
|
| 197 |
+
|
| 198 |
+
Two independent hook bugs:
|
| 199 |
+
|
| 200 |
+
**ReplayLoader (replay.rs):**
|
| 201 |
+
- Player array strides: Ghidra showed element indices (`i*0x3C` for `u16*`, `i*0x1E` for `u32*`) but code used them as byte offsets. Correct byte stride: `i*0x78`.
|
| 202 |
+
- Non-zero `map_seed` path: per-team weapon config reads from stream were unimplemented. Fallback to original parser for `map_seed β 0`.
|
| 203 |
+
|
| 204 |
+
**FireWeapon (weapon.rs):**
|
| 205 |
+
- Binary search within the type-4 dispatch: type 4 β subtypes 10+13 β subtype 13 (Napalm Strike).
|
| 206 |
+
- `fire_send_team_message` wrote `team_index` to `buf[8..12]` but the original writes it to `buf[0..4]`. The HandleMessage handler reads `data[0]`, receiving 0 instead of the correct team index.
|
| 207 |
+
- Root cause confirmed by disassembly of the original Napalm handler at 0x51E5C0: `MOV [ESP+4], EAX` where EAX = `worm+0xFC` (team_index) and `ESP+4` = `buffer[0]`.
|
| 208 |
+
|
| 209 |
+
### Also fixed along the way
|
| 210 |
+
|
| 211 |
+
- **DDGame allocation size**: 0x98B8 β 0x98D8 (original mallocs 0x98D8 but memsets only 0x98B8)
|
| 212 |
+
- **Wrong function after InitPaletteGradientSprites**: called `LoadingProgressTick` (0x5717A0) instead of `DisplayGfx__InitTeamPaletteDisplayObjects` (0x5703E0)
|
| 213 |
|
| 214 |
## Debugging Tools Built
|
| 215 |
|
| 216 |
The investigation produced several reusable tools:
|
| 217 |
|
| 218 |
+
- **Hardware watchpoint system** (`debug_watchpoint.rs`) β x86 DR0βDR3 with VEH handler and EBP stack trace walking
|
| 219 |
+
- **Sub-object hashing** (`snapshot.rs: hash_pointer_targets`) β follows heap pointers in a struct, hashes target content with pointer canonicalization. Integrated into the snapshot system.
|
| 220 |
+
- **Per-frame RNG logging** via TurnManager hook
|
| 221 |
+
- **`OPENWA_WATCH_DISPLAY=1`** / **`OPENWA_WATCH_FRAME=N`** β arm watchpoints on display or DDGame at specific frames
|
| 222 |
- **Binary sub-object dumping** for display, GfxHandler, PCLandscape comparison
|
| 223 |
+
- **A/B constructor toggle** (`OPENWA_USE_ORIG_CTOR=1`) β instant switching between Rust and original constructors
|
|
|
|
| 224 |
|
| 225 |
## Key Lessons
|
| 226 |
|
| 227 |
+
1. **Identical struct memory β identical behaviour.** DDGame flat memory was byte-identical, but *sub-objects pointed to by DDGame* had different content. The `hash_pointer_targets` tool was purpose-built to catch this.
|
| 228 |
+
|
| 229 |
+
2. **Validate your diff tools.** Our snapshot comparison showed ~490 "differences" at known-good frames β pure noise from pointer canonicalization heuristics. And our dump code once read past a heap allocation into adjacent memory, producing false diffs. Always baseline.
|
| 230 |
+
|
| 231 |
+
3. **There is no "visual-only" code in a deterministic game engine.** `debris.spr` seems purely decorative, but Worms uses a **single shared RNG** for everything. Skipping one sprite load shifts the entire RNG sequence.
|
| 232 |
+
|
| 233 |
+
4. **Hardware watchpoints are underrated.** x86 DR0βDR3 with a VEH handler gives "what wrote this byte?" answers in seconds, without an external debugger.
|
| 234 |
+
|
| 235 |
+
5. **Binary search on frames, not code.** Per-frame RNG logging narrowed 542 candidate frames to the exact frame (818) where divergence began.
|
| 236 |
|
| 237 |
+
6. **Binary search on hooks.** Disabling half the hooks at a time isolated two independent bugs in minutes. Then within a single hook (FireWeapon), binary search on dispatch paths (type β subtype β specific handler) pinpointed the exact buggy function.
|
| 238 |
|
| 239 |
+
7. **Multiple test replays expose different bugs.** The original longbow replay only triggered the constructor bug. Adding longbow_water and longbow_alt_theme exposed hook bugs that were invisible with the original replay alone. Different weapons, themes, and targets exercise different code paths.
|
| 240 |
|
| 241 |
+
8. **Ghidra element indexing is a trap.** The decompiler shows `(&DAT[i*0x3C])` for a `u16*` array β the `0x3C` is an *element* index, not a byte offset. The actual byte offset is `i * 0x3C * sizeof(u16) = i * 0x78`. This caused player data corruption for index 1+.
|
| 242 |
|
| 243 |
+
9. **Check the buffer layout against disassembly.** The Napalm Strike bug was a single wrong offset in a message buffer: `buf[8]` vs `buf[0]`. The original's disassembly (`MOV [ESP+4], EAX` at 0x51E5D2) unambiguously shows the write destination. When in doubt, read the assembly.
|