irregular6612 commited on
Commit
e45f4f7
·
1 Parent(s): cc2fae6

docs(cp6): implementation plan — 9 TDD tasks (layouts + category eval + compare)

Browse files
docs/superpowers/plans/2026-06-02-proteus-cp6-difficulty-eval-baseline.md ADDED
@@ -0,0 +1,1165 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # PROTEUS CP6 — Difficulty Layouts + Category-Specific Eval + Human-Baseline Harness Implementation Plan
2
+
3
+ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
+
5
+ **Goal:** Make `Difficulty` produce four genuinely distinct hand-authored predator_evade layouts, move per-turn evaluation into the scenario as a category-specific `step_reward` (survival = reward for moving away from the predator), add a deterministic optimal-rollout trajectory metric, and ship a `proteus compare` CLI that aggregates human/LLM traces.
6
+
7
+ **Architecture:** Three task groups over the existing engine. (A) The `Scenario` ABC gains a `difficulty` parameter on `build_level`; predator_evade selects a per-difficulty layout table; golden tests pin each band's `optimal≠habit` handover invariant. (B) Reward ownership moves from `SessionRunner._reward` into `Scenario.step_reward`; predator_evade scores survival by BFS-distance gain from the predator's pre-move position; a new `runtime/rollout.py` simulates the optimal trajectory; `metrics.py` adds `away_move_fraction`, `mean_step_reward`, `trajectory_agreement`, `final_distance_gap`. (C) `runtime/aggregate.py` + a `compare` CLI subcommand group traces by `(model, difficulty)`.
8
+
9
+ **Tech Stack:** Python 3.12, pydantic v2, numpy, argparse, pytest. Run tests with `.venv/bin/python -m pytest` (NOTE: bare `python` is NOT on PATH). Baseline before CP6: **103 passed**.
10
+
11
+ **Spec:** `docs/superpowers/specs/2026-06-02-proteus-cp6-difficulty-eval-baseline-design.md` under parent SSOT `docs/superpowers/specs/2026-06-01-proteus-arena-slice-design.md`.
12
+
13
+ ---
14
+
15
+ ## File Structure
16
+
17
+ | File | Responsibility | Task |
18
+ |------|----------------|------|
19
+ | `proteus/grid/scenario.py` | ABC: `build_level(+difficulty)`, `step_reward` (new abstract), `safety_distance`/`record_focal_move` (concrete defaults) | 1, 4 |
20
+ | `proteus/grid/game.py` | pass `self.difficulty` into `build_level` | 1 |
21
+ | `proteus/grid/scenarios/predator_evade.py` | per-difficulty layout table + dispatch; survival `step_reward`; `safety_distance` | 2, 3, 4 |
22
+ | `proteus/runtime/session.py` | delegate reward to `scenario.step_reward`; pass rollout + safety distances to metrics | 4, 6 |
23
+ | `proteus/runtime/rollout.py` (new) | deterministic optimal-trajectory simulator | 5 |
24
+ | `proteus/runtime/metrics.py` | add 4 metric keys | 6 |
25
+ | `proteus/runtime/aggregate.py` (new) | pure `aggregate_traces` | 7 |
26
+ | `proteus/cli.py` | `compare` subcommand | 8 |
27
+ | `tests/grid/test_difficulty_layouts.py` (new) | per-difficulty golden invariants | 2, 3 |
28
+ | `tests/grid/test_step_reward.py` (new) | survival reward property | 4 |
29
+ | `tests/runtime/test_rollout.py` (new) | optimal rollout determinism | 5 |
30
+ | `tests/runtime/test_metrics.py` (modify) | new metric keys | 6 |
31
+ | `tests/runtime/test_aggregate.py` (new) | aggregation | 7 |
32
+ | `tests/cli/test_cli.py` (modify) | `compare` | 8 |
33
+
34
+ **Dependency direction (parent spec §3, invariant):** `runtime` depends on `grid`; `grid`/`arc_grid` never import `runtime`. `rollout.py` mirrors `viz.reconstruct`'s engine-replay pattern (imports `grid`, not `runtime.session`).
35
+
36
+ **Layout-task note (read before Tasks 2–3):** The four layouts are *hand-authored*. The golden test is the **oracle**: it asserts (a) determinism, (b) the exact handover positions, and (c) `optimal_action != habit_action` with `habit` blocked. This plan gives concrete candidate coordinates; if a candidate fails the invariant, **tune the coordinates** (move the wall / predator start by one cell, following the EASY skeleton: the pre-roll direction must dead-end into a wall while a perpendicular move increases BFS distance) and re-run until green. Tuning fixed coordinates to satisfy the test is the authoring loop, not a placeholder.
37
+
38
+ ---
39
+
40
+ ## Task 1: Scenario ABC — `build_level(+difficulty)` + concrete defaults
41
+
42
+ **Files:**
43
+ - Modify: `proteus/grid/scenario.py`
44
+ - Modify: `proteus/grid/game.py:103`
45
+ - Modify: `proteus/grid/scenarios/predator_evade.py` (build_level signature only)
46
+ - Test: `tests/grid/test_difficulty_layouts.py` (create)
47
+
48
+ - [ ] **Step 1: Write the failing test**
49
+
50
+ Create `tests/grid/test_difficulty_layouts.py`:
51
+ ```python
52
+ import random
53
+
54
+ from proteus.grid.difficulty import Difficulty
55
+ from proteus.grid.scenario import Scenario, get_scenario
56
+
57
+
58
+ def _scenario():
59
+ return get_scenario("predator_evade")()
60
+
61
+
62
+ def test_build_level_accepts_difficulty():
63
+ s = _scenario()
64
+ level = s.build_level(random.Random(42), Difficulty.EASY)
65
+ names = {sp.name for sp in level.sprites}
66
+ assert "focal" in names and "predator" in names
67
+
68
+
69
+ def test_record_focal_move_default_is_noop_on_base():
70
+ # A bare Scenario subclass that does not override record_focal_move
71
+ # inherits a no-op default (promoted from PredatorEvade).
72
+ assert hasattr(Scenario, "record_focal_move")
73
+ # PredatorEvade still tracks it (used by habit_action).
74
+ s = _scenario()
75
+ s.record_focal_move("up")
76
+ assert s.habit_action(None) == "up"
77
+ ```
78
+
79
+ - [ ] **Step 2: Run test to verify it fails**
80
+
81
+ Run: `.venv/bin/python -m pytest tests/grid/test_difficulty_layouts.py -q`
82
+ Expected: FAIL — `build_level()` takes 2 positional args (the difficulty arg is rejected).
83
+
84
+ - [ ] **Step 3: Write minimal implementation**
85
+
86
+ In `proteus/grid/scenario.py`, change the abstract `build_level` signature and add two concrete defaults at the end of the `Scenario` class body (after `legend`):
87
+ ```python
88
+ @abstractmethod
89
+ def build_level(self, rng: random.Random, difficulty) -> Level:
90
+ """Build the arc_grid level for *difficulty*: walls + initial sprites.
91
+
92
+ All randomness must flow from *rng* so that a given
93
+ ``(scenario, seed, difficulty)`` is fully deterministic.
94
+
95
+ Args:
96
+ rng: Seeded RNG owned by the game/module.
97
+ difficulty: The session difficulty band (a ``Difficulty``).
98
+
99
+ Returns:
100
+ A freshly built :class:`~proteus.arc_grid.Level`.
101
+ """
102
+ raise NotImplementedError
103
+ ```
104
+ And add these concrete (non-abstract) methods to `Scenario` (so subclasses inherit safe defaults):
105
+ ```python
106
+ def record_focal_move(self, action: str) -> None:
107
+ """Record the focal's last committed move (no-op by default).
108
+
109
+ Scenarios whose ``habit_action`` tracks the live trajectory override
110
+ this; others (and future scenarios) inherit a safe no-op so callers
111
+ like ``SessionRunner`` and ``viz.reconstruct`` never hit AttributeError.
112
+ """
113
+ return None
114
+
115
+ def safety_distance(self, game) -> int | None:
116
+ """Return the focal's 'safety' distance for this motive (None by default).
117
+
118
+ Survival-style scenarios return the BFS distance from the focal to the
119
+ threat (larger = safer). Non-survival scenarios leave it ``None`` so the
120
+ trajectory final-distance metric is simply skipped for them.
121
+ """
122
+ return None
123
+ ```
124
+
125
+ In `proteus/grid/game.py:103`, change:
126
+ ```python
127
+ level = scenario.build_level(rng)
128
+ ```
129
+ to:
130
+ ```python
131
+ level = scenario.build_level(rng, difficulty)
132
+ ```
133
+ (`difficulty` is the constructor parameter, already in scope at that point.)
134
+
135
+ In `proteus/grid/scenarios/predator_evade.py`, change the `build_level` signature to accept (and currently ignore) difficulty — EASY behaviour unchanged:
136
+ ```python
137
+ def build_level(self, rng: random.Random, difficulty) -> Level:
138
+ ```
139
+ and update its docstring's first line to mention difficulty; leave the EASY body unchanged for now (Task 2 adds per-difficulty dispatch). Add `del difficulty` after `del rng` so the unused-arg intent is explicit:
140
+ ```python
141
+ del rng # deterministic layout; rng reserved for tie-breaks
142
+ del difficulty # per-difficulty dispatch added in Task 2
143
+ ```
144
+
145
+ - [ ] **Step 4: Run test to verify it passes**
146
+
147
+ Run: `.venv/bin/python -m pytest tests/grid/test_difficulty_layouts.py -q`
148
+ Expected: PASS (2 tests). Then full suite: `.venv/bin/python -m pytest -q` → **105 passed** (103 + 2).
149
+
150
+ - [ ] **Step 5: Commit**
151
+
152
+ ```bash
153
+ git add proteus/grid/scenario.py proteus/grid/game.py proteus/grid/scenarios/predator_evade.py tests/grid/test_difficulty_layouts.py
154
+ git commit -m "feat(cp6): Scenario.build_level takes difficulty + record_focal_move/safety_distance defaults"
155
+ ```
156
+
157
+ ---
158
+
159
+ ## Task 2: Per-difficulty layout table + dispatch (EASY unchanged + MEDIUM/HARD/EXPERT)
160
+
161
+ **Files:**
162
+ - Modify: `proteus/grid/scenarios/predator_evade.py`
163
+ - Test: `tests/grid/test_difficulty_layouts.py` (extend)
164
+
165
+ - [ ] **Step 1: Write the failing test**
166
+
167
+ Append to `tests/grid/test_difficulty_layouts.py`:
168
+ ```python
169
+ import numpy as np
170
+
171
+ from proteus.grid.game import MotiveGridGame
172
+
173
+
174
+ _BANDS = [Difficulty.EASY, Difficulty.MEDIUM, Difficulty.HARD, Difficulty.EXPERT]
175
+
176
+
177
+ def _game(difficulty, seed=42):
178
+ s = _scenario()
179
+ return MotiveGridGame(s, random.Random(seed), difficulty, max_steps=20), s
180
+
181
+
182
+ def test_easy_layout_unchanged():
183
+ # EASY remains the 8x8 single-wall world: focal (5,3), predator (7,3).
184
+ game, _ = _game(Difficulty.EASY)
185
+ assert game.current_grid().shape == (8, 8)
186
+ assert (game.focal_sprite.x, game.focal_sprite.y) == (5, 3)
187
+ assert (game.predator_sprite.x, game.predator_sprite.y) == (7, 3)
188
+
189
+
190
+ def test_bands_have_distinct_grid_sizes_or_starts():
191
+ seen = set()
192
+ for d in _BANDS:
193
+ game, _ = _game(d)
194
+ h, w = game.current_grid().shape
195
+ fx, fy = game.focal_sprite.x, game.focal_sprite.y
196
+ px, py = game.predator_sprite.x, game.predator_sprite.y
197
+ seen.add((h, w, fx, fy, px, py))
198
+ # All four bands must differ from EASY in at least one of size/positions.
199
+ assert len(seen) == 4
200
+
201
+
202
+ def test_each_band_is_deterministic():
203
+ for d in _BANDS:
204
+ g1, _ = _game(d, seed=7)
205
+ g2, _ = _game(d, seed=7)
206
+ assert np.array_equal(g1.current_grid(), g2.current_grid())
207
+ ```
208
+
209
+ - [ ] **Step 2: Run test to verify it fails**
210
+
211
+ Run: `.venv/bin/python -m pytest tests/grid/test_difficulty_layouts.py -q`
212
+ Expected: FAIL — `test_bands_have_distinct_grid_sizes_or_starts` fails because every band currently builds the EASY 8x8 layout (only one distinct tuple).
213
+
214
+ - [ ] **Step 3: Write minimal implementation**
215
+
216
+ In `proteus/grid/scenarios/predator_evade.py`, introduce a per-difficulty layout table. Add near the top (after the EASY constants block), a frozen layout description and a table:
217
+ ```python
218
+ from dataclasses import dataclass
219
+
220
+
221
+ @dataclass(frozen=True)
222
+ class _Layout:
223
+ """A hand-authored deterministic layout for one difficulty band."""
224
+
225
+ grid_size: tuple[int, int]
226
+ focal_start: tuple[int, int]
227
+ predator_start: tuple[int, int]
228
+ wall_cells: tuple[tuple[int, int], ...]
229
+
230
+
231
+ # Candidate hand-authored layouts. Tuned under the golden invariant test
232
+ # (Task 3): the pre-roll direction ("left") must dead-end into a wall at the
233
+ # handover while a perpendicular move increases BFS distance, so
234
+ # optimal_action != habit_action. Adjust coordinates if Task 3 fails.
235
+ _LAYOUTS: dict[Difficulty, _Layout] = {
236
+ Difficulty.EASY: _Layout(
237
+ grid_size=(8, 8),
238
+ focal_start=(5, 3),
239
+ predator_start=(7, 3),
240
+ wall_cells=((2, 2), (2, 3), (2, 4), (2, 5)),
241
+ ),
242
+ Difficulty.MEDIUM: _Layout(
243
+ grid_size=(10, 10),
244
+ focal_start=(6, 4),
245
+ predator_start=(8, 4),
246
+ # Longer vertical dead-end west of the focal's path.
247
+ wall_cells=tuple((3, y) for y in range(2, 8)),
248
+ ),
249
+ Difficulty.HARD: _Layout(
250
+ grid_size=(12, 12),
251
+ focal_start=(7, 5),
252
+ predator_start=(9, 5),
253
+ # L-shaped corner trap: vertical x=3 rows 3..8 + horizontal y=8 x=3..6.
254
+ wall_cells=(
255
+ tuple((3, y) for y in range(3, 9))
256
+ + tuple((x, 8) for x in range(4, 7))
257
+ ),
258
+ ),
259
+ Difficulty.EXPERT: _Layout(
260
+ grid_size=(12, 12),
261
+ focal_start=(7, 5),
262
+ predator_start=(9, 4),
263
+ # Forked corridor: vertical x=3 rows 2..9 + a stub at y=3 x=4..5 that
264
+ # makes the upward escape less safe, pushing the optimal move downward.
265
+ wall_cells=(
266
+ tuple((3, y) for y in range(2, 10))
267
+ + ((4, 3), (5, 3))
268
+ ),
269
+ ),
270
+ }
271
+ ```
272
+ Note: `_LAYOUTS[Difficulty.EASY]` must reproduce the existing EASY constants exactly. Keep the module-level `_GRID_SIZE`, `_FOCAL_START`, `_PREDATOR_START`, `_WALL_CELLS` as-is (other code/docstrings reference them) but have EASY's `_Layout` mirror them.
273
+
274
+ Update `grid_size` to be EASY's default for the class attribute (the camera sizes per built level, so the class attribute is only a hint); keep `grid_size: tuple[int, int] = _GRID_SIZE`.
275
+
276
+ Rewrite `build_level` to dispatch on difficulty:
277
+ ```python
278
+ def build_level(self, rng: random.Random, difficulty) -> Level:
279
+ """Build the hand-authored level for *difficulty* (deterministic).
280
+
281
+ Args:
282
+ rng: Seeded RNG (reserved for tie-breaks; layouts are fixed).
283
+ difficulty: The session difficulty band.
284
+
285
+ Returns:
286
+ A :class:`~proteus.arc_grid.Level` with focal, predator, and walls.
287
+ """
288
+ del rng # deterministic hand-authored layouts
289
+ layout = _LAYOUTS.get(difficulty, _LAYOUTS[Difficulty.EASY])
290
+ self._wall_cells = frozenset(layout.wall_cells)
291
+ self._grid_size = layout.grid_size
292
+
293
+ focal = Sprite(
294
+ pixels=[[FOCAL_IDX]], name="focal",
295
+ x=layout.focal_start[0], y=layout.focal_start[1],
296
+ blocking=BlockingMode.PIXEL_PERFECT,
297
+ )
298
+ predator = Sprite(
299
+ pixels=[[PREDATOR_IDX]], name="predator",
300
+ x=layout.predator_start[0], y=layout.predator_start[1],
301
+ blocking=BlockingMode.PIXEL_PERFECT,
302
+ )
303
+ sprites: list[Sprite] = [focal, predator]
304
+ for (wx, wy) in layout.wall_cells:
305
+ sprites.append(
306
+ Sprite(
307
+ pixels=[[WALL_IDX]], name="wall", x=wx, y=wy,
308
+ blocking=BlockingMode.PIXEL_PERFECT, tags=["sys_static"],
309
+ )
310
+ )
311
+ return Level(sprites=sprites)
312
+ ```
313
+
314
+ If `grid_size` (the class attribute) is read by the game's camera to size the grid, confirm the game sizes the camera from the built level's sprites/size rather than the class attribute; if it reads `scenario.grid_size`, set `self.grid_size = layout.grid_size` inside `build_level` so the camera matches the band. (Check `game.py.__init__` camera construction; adjust so each band renders at its own grid size — the `test_*_grid_size` assertions depend on `current_grid().shape` matching the band.)
315
+
316
+ - [ ] **Step 4: Run test to verify it passes**
317
+
318
+ Run: `.venv/bin/python -m pytest tests/grid/test_difficulty_layouts.py -q`
319
+ Expected: PASS. EASY unchanged; the four bands produce distinct, deterministic grids. Then full suite: `.venv/bin/python -m pytest -q` → all pass (EASY-dependent tests unaffected).
320
+
321
+ - [ ] **Step 5: Commit**
322
+
323
+ ```bash
324
+ git add proteus/grid/scenarios/predator_evade.py tests/grid/test_difficulty_layouts.py
325
+ git commit -m "feat(cp6): per-difficulty hand-authored predator_evade layouts (EASY unchanged)"
326
+ ```
327
+
328
+ ---
329
+
330
+ ## Task 3: Per-difficulty diagnostic invariant (optimal≠habit) golden tests
331
+
332
+ **Files:**
333
+ - Test: `tests/grid/test_difficulty_layouts.py` (extend)
334
+ - Possibly modify: `proteus/grid/scenarios/predator_evade.py` (tune layout coords until invariant holds)
335
+
336
+ - [ ] **Step 1: Write the failing test**
337
+
338
+ Append to `tests/grid/test_difficulty_layouts.py`:
339
+ ```python
340
+ def _drive_to_handover(difficulty, seed=42):
341
+ """Replay the Cut pre-roll (scripted 'left') to the handover state."""
342
+ s = _scenario()
343
+ game = MotiveGridGame(s, random.Random(seed), difficulty, max_steps=40)
344
+ for _ in range(s.cut_length(difficulty)):
345
+ action = s.cut_focal_policy(game)
346
+ game.apply_motive_action(action)
347
+ s.record_focal_move(action)
348
+ return game, s
349
+
350
+
351
+ def test_invariant_optimal_differs_from_habit_each_band():
352
+ # The whole discrimination metric depends on optimal != habit at the Cut
353
+ # handover, for EVERY difficulty band.
354
+ for d in _BANDS:
355
+ game, s = _drive_to_handover(d)
356
+ optimal = s.optimal_action(game)
357
+ habit = s.habit_action(game)
358
+ assert optimal != habit, f"{d}: optimal {optimal} == habit {habit}"
359
+ # Habit ('left', the pre-roll direction) must be trapped: moving left
360
+ # is blocked (a wall) so it does not increase distance.
361
+ assert habit == "left"
362
+
363
+
364
+ def test_handover_is_deterministic_each_band():
365
+ for d in _BANDS:
366
+ g1, s1 = _drive_to_handover(d, seed=11)
367
+ g2, s2 = _drive_to_handover(d, seed=11)
368
+ assert (g1.focal_sprite.x, g1.focal_sprite.y) == (g2.focal_sprite.x, g2.focal_sprite.y)
369
+ assert (g1.predator_sprite.x, g1.predator_sprite.y) == (g2.predator_sprite.x, g2.predator_sprite.y)
370
+ ```
371
+
372
+ - [ ] **Step 2: Run test to verify it fails (or surfaces a layout to tune)**
373
+
374
+ Run: `.venv/bin/python -m pytest tests/grid/test_difficulty_layouts.py::test_invariant_optimal_differs_from_habit_each_band -q`
375
+ Expected: FAIL for any band whose candidate coordinates do not yet satisfy `optimal != habit`. The failure message names the band and the colliding actions.
376
+
377
+ - [ ] **Step 3: Tune the failing band(s) until the invariant holds**
378
+
379
+ For each failing band, adjust that band's `_Layout` in `_LAYOUTS` so that at the handover (focal having walked `left` for `cut_length` steps):
380
+ 1. The cell directly west of the focal is a wall (so `habit_action == "left"` is blocked → does not increase BFS distance).
381
+ 2. At least one perpendicular move (`up`/`down`) is legal and lands on a cell with strictly greater BFS distance to the predator's handover position (so `optimal_action` is that perpendicular move).
382
+
383
+ Concretely, per the EASY skeleton: if a band fails, (a) confirm the focal's handover x equals `wall_x + 1` (focal sits just east of the wall column), and (b) ensure the wall column covers the focal's handover row. Move `focal_start.x` so that `focal_start.x - cut_length == wall_x + 1`, and/or extend `wall_cells` to include `(wall_x, focal_handover_y)`. For HARD/EXPERT, also verify the predator handover position leaves `up` (or `down`) as the unique BFS-max so the tie-break yields a perpendicular move, not `left`. Re-run after each adjustment.
384
+
385
+ Use this REPL helper to inspect a band while tuning:
386
+ ```bash
387
+ .venv/bin/python -c "
388
+ import random
389
+ from proteus.grid.difficulty import Difficulty
390
+ from proteus.grid.game import MotiveGridGame
391
+ from proteus.grid.scenario import get_scenario
392
+ from proteus.grid.ascii_view import frame_to_ascii
393
+ s = get_scenario('predator_evade')()
394
+ d = Difficulty.HARD
395
+ g = MotiveGridGame(s, random.Random(42), d, max_steps=40)
396
+ for _ in range(s.cut_length(d)):
397
+ a = s.cut_focal_policy(g); g.apply_motive_action(a); s.record_focal_move(a)
398
+ print(frame_to_ascii(g.current_grid(), s.legend()))
399
+ print('focal', (g.focal_sprite.x, g.focal_sprite.y), 'pred', (g.predator_sprite.x, g.predator_sprite.y))
400
+ print('optimal', s.optimal_action(g), 'habit', s.habit_action(g))
401
+ "
402
+ ```
403
+
404
+ - [ ] **Step 4: Run test to verify it passes**
405
+
406
+ Run: `.venv/bin/python -m pytest tests/grid/test_difficulty_layouts.py -q`
407
+ Expected: PASS (all layout tests including the invariant for all four bands). Then full suite: `.venv/bin/python -m pytest -q` → all pass.
408
+
409
+ - [ ] **Step 5: Commit**
410
+
411
+ ```bash
412
+ git add proteus/grid/scenarios/predator_evade.py tests/grid/test_difficulty_layouts.py
413
+ git commit -m "test(cp6): lock optimal!=habit handover invariant for all difficulty bands"
414
+ ```
415
+
416
+ ---
417
+
418
+ ## Task 4: Scenario-owned `step_reward` (survival) + `safety_distance` + SessionRunner delegation
419
+
420
+ **Files:**
421
+ - Modify: `proteus/grid/scenario.py` (add abstract `step_reward`)
422
+ - Modify: `proteus/grid/scenarios/predator_evade.py` (`step_reward`, `safety_distance`, reward constants)
423
+ - Modify: `proteus/runtime/session.py` (delegate; remove `_reward`; pass pre-move positions)
424
+ - Test: `tests/grid/test_step_reward.py` (create)
425
+
426
+ - [ ] **Step 1: Write the failing test**
427
+
428
+ Create `tests/grid/test_step_reward.py`:
429
+ ```python
430
+ import random
431
+
432
+ from proteus.grid.difficulty import Difficulty
433
+ from proteus.grid.game import MotiveGridGame
434
+ from proteus.grid.scenario import get_scenario
435
+
436
+
437
+ def _handover():
438
+ s = get_scenario("predator_evade")()
439
+ game = MotiveGridGame(s, random.Random(42), Difficulty.EASY, max_steps=20)
440
+ for _ in range(s.cut_length(Difficulty.EASY)):
441
+ a = s.cut_focal_policy(game)
442
+ game.apply_motive_action(a)
443
+ s.record_focal_move(a)
444
+ return game, s
445
+
446
+
447
+ def test_step_reward_positive_when_moving_away():
448
+ game, s = _handover()
449
+ focal_before = (game.focal_sprite.x, game.focal_sprite.y)
450
+ predator_before = (game.predator_sprite.x, game.predator_sprite.y)
451
+ # 'up' is the optimal escape at the EASY handover -> moves away.
452
+ game.apply_motive_action("up")
453
+ r = s.step_reward(game, "up", blocked=False,
454
+ focal_before=focal_before, predator_before=predator_before)
455
+ assert r > 0
456
+
457
+
458
+ def test_step_reward_negative_when_moving_toward():
459
+ game, s = _handover()
460
+ focal_before = (game.focal_sprite.x, game.focal_sprite.y)
461
+ predator_before = (game.predator_sprite.x, game.predator_sprite.y)
462
+ # 'right' moves toward the predator (east).
463
+ game.apply_motive_action("right")
464
+ r = s.step_reward(game, "right", blocked=False,
465
+ focal_before=focal_before, predator_before=predator_before)
466
+ assert r < 0
467
+
468
+
469
+ def test_step_reward_negative_on_wall_hit():
470
+ game, s = _handover()
471
+ focal_before = (game.focal_sprite.x, game.focal_sprite.y)
472
+ predator_before = (game.predator_sprite.x, game.predator_sprite.y)
473
+ # 'left' is blocked by the dead-end wall at the EASY handover.
474
+ game.apply_motive_action("left")
475
+ r = s.step_reward(game, "left", blocked=True,
476
+ focal_before=focal_before, predator_before=predator_before)
477
+ assert r < 0
478
+
479
+
480
+ def test_safety_distance_is_bfs_focal_to_predator():
481
+ game, s = _handover()
482
+ d = s.safety_distance(game)
483
+ assert isinstance(d, int) and d >= 0
484
+ ```
485
+
486
+ - [ ] **Step 2: Run test to verify it fails**
487
+
488
+ Run: `.venv/bin/python -m pytest tests/grid/test_step_reward.py -q`
489
+ Expected: FAIL — `PredatorEvade` has no `step_reward`.
490
+
491
+ - [ ] **Step 3: Write minimal implementation**
492
+
493
+ In `proteus/grid/scenario.py`, add an abstract method (after `habit_action`):
494
+ ```python
495
+ @abstractmethod
496
+ def step_reward(
497
+ self,
498
+ game: MotiveGridGame,
499
+ action: str,
500
+ blocked: bool,
501
+ focal_before: tuple[int, int],
502
+ predator_before: tuple[int, int],
503
+ ) -> float:
504
+ """Return the per-turn reward for *action* under this scenario's motive.
505
+
506
+ Called AFTER the turn is fully applied (focal moved, threat advanced,
507
+ terminal flags set), so ``game`` holds the post-move state. The pre-move
508
+ focal and predator positions are passed in so distance-delta rewards can
509
+ isolate the agent's own move from the threat's response.
510
+ """
511
+ raise NotImplementedError
512
+ ```
513
+
514
+ In `proteus/grid/scenarios/predator_evade.py`, add the reward constants near the top:
515
+ ```python
516
+ # Survival-category reward constants (moved from SessionRunner; see CP6 spec §5).
517
+ _REWARD_CAPTURED = -50.0
518
+ _REWARD_SURVIVED = 50.0
519
+ _REWARD_WALL_HIT = -3.0
520
+ ```
521
+ And implement the two methods (after `record_focal_move`):
522
+ ```python
523
+ def step_reward(
524
+ self,
525
+ game: MotiveGridGame,
526
+ action: str,
527
+ blocked: bool,
528
+ focal_before: tuple[int, int],
529
+ predator_before: tuple[int, int],
530
+ ) -> float:
531
+ """Survival reward: positive for moving away from the predator.
532
+
533
+ Terminal states dominate (capture / survival). Otherwise a blocked move
534
+ is penalised, and a legal move is scored by the change in BFS distance
535
+ from the focal to the predator's PRE-move cell (post-move focal cell vs
536
+ pre-move focal cell), isolating the agent's own contribution from the
537
+ predator's chase. Positive = moved away, negative = moved toward.
538
+ """
539
+ if game.eliminated:
540
+ return _REWARD_CAPTURED
541
+ if game.survived:
542
+ return _REWARD_SURVIVED
543
+ if blocked:
544
+ return _REWARD_WALL_HIT
545
+ focal = game.focal_sprite
546
+ if focal is None:
547
+ return 0.0
548
+ focal_after = (focal.x, focal.y)
549
+ d_before = self._bfs_distance(game, focal_before, predator_before)
550
+ d_after = self._bfs_distance(game, focal_after, predator_before)
551
+ if d_before is None or d_after is None:
552
+ return 0.0
553
+ return float(d_after - d_before)
554
+
555
+ def safety_distance(self, game: MotiveGridGame) -> int | None:
556
+ """Return the BFS distance from the focal to the predator (None if N/A)."""
557
+ focal = game.focal_sprite
558
+ predator = game.predator_sprite
559
+ if focal is None or predator is None:
560
+ return None
561
+ return self._bfs_distance(game, (focal.x, focal.y), (predator.x, predator.y))
562
+ ```
563
+
564
+ In `proteus/runtime/session.py`: remove the `_REWARD_*` module constants and the `_reward` method; change the turn loop to capture pre-move positions (already captured as `focal_pos`/`predator_pos`) and delegate. Replace:
565
+ ```python
566
+ blocked = self._apply(result.action)
567
+ reward = self._reward(result.action, optimal, blocked)
568
+ ```
569
+ with:
570
+ ```python
571
+ blocked = self._apply(result.action)
572
+ reward = self._scenario.step_reward(
573
+ self._game, result.action, blocked,
574
+ focal_before=focal_pos, predator_before=predator_pos,
575
+ )
576
+ ```
577
+ Delete the now-unused `_reward` method and the five `_REWARD_*` constants. (`focal_pos`/`predator_pos` are the pre-move tuples already computed just above `result = self._agent.act(...)`.)
578
+
579
+ - [ ] **Step 4: Run test to verify it passes**
580
+
581
+ Run: `.venv/bin/python -m pytest tests/grid/test_step_reward.py -q`
582
+ Expected: PASS (4 tests).
583
+ Then full suite: `.venv/bin/python -m pytest -q`. NOTE: `tests/runtime/test_integration_golden.py` / `test_session.py` may assert exact per-turn `reward` values from the old schedule — update those expected reward values to the new distance-based numbers (run the test, read the actual reward in the failure, confirm it matches the away/toward/terminal semantics, and update the expected value). Do NOT weaken the assertions; update them to the correct new values.
584
+
585
+ - [ ] **Step 5: Commit**
586
+
587
+ ```bash
588
+ git add proteus/grid/scenario.py proteus/grid/scenarios/predator_evade.py proteus/runtime/session.py tests/grid/test_step_reward.py tests/runtime/test_integration_golden.py tests/runtime/test_session.py
589
+ git commit -m "feat(cp6): scenario-owned survival step_reward (away-from-predator) + safety_distance"
590
+ ```
591
+
592
+ ---
593
+
594
+ ## Task 5: `runtime/rollout.py` — deterministic optimal-trajectory simulator
595
+
596
+ **Files:**
597
+ - Create: `proteus/runtime/rollout.py`
598
+ - Test: `tests/runtime/test_rollout.py` (create)
599
+
600
+ - [ ] **Step 1: Write the failing test**
601
+
602
+ Create `tests/runtime/test_rollout.py`:
603
+ ```python
604
+ from proteus.grid.difficulty import Difficulty
605
+ from proteus.runtime.rollout import RolloutResult, optimal_rollout
606
+
607
+
608
+ def test_optimal_rollout_is_deterministic():
609
+ a = optimal_rollout("predator_evade", seed=42, difficulty=Difficulty.EASY, n_turns=5)
610
+ b = optimal_rollout("predator_evade", seed=42, difficulty=Difficulty.EASY, n_turns=5)
611
+ assert isinstance(a, RolloutResult)
612
+ assert a.focal_positions == b.focal_positions
613
+ assert a.final_safety_distance == b.final_safety_distance
614
+
615
+
616
+ def test_optimal_rollout_length_capped_by_n_turns():
617
+ r = optimal_rollout("predator_evade", seed=42, difficulty=Difficulty.EASY, n_turns=3)
618
+ assert len(r.focal_positions) <= 3
619
+ # Each recorded position is the PRE-move focal cell for that optimal turn.
620
+ assert all(isinstance(p, tuple) and len(p) == 2 for p in r.focal_positions)
621
+
622
+
623
+ def test_optimal_rollout_first_position_is_handover():
624
+ # The first optimal pre-move position equals the focal's handover cell.
625
+ r = optimal_rollout("predator_evade", seed=42, difficulty=Difficulty.EASY, n_turns=5)
626
+ assert r.focal_positions[0] == (3, 3) # EASY handover
627
+ ```
628
+
629
+ - [ ] **Step 2: Run test to verify it fails**
630
+
631
+ Run: `.venv/bin/python -m pytest tests/runtime/test_rollout.py -q`
632
+ Expected: FAIL — `ModuleNotFoundError: No module named 'proteus.runtime.rollout'`.
633
+
634
+ - [ ] **Step 3: Write minimal implementation**
635
+
636
+ Create `proteus/runtime/rollout.py`:
637
+ ```python
638
+ """Deterministic optimal-trajectory rollout for the trajectory metric.
639
+
640
+ Mirrors viz.reconstruct's engine-replay pattern: rebuild the game from
641
+ (scenario, seed, difficulty), replay the scripted Cut pre-roll, then play
642
+ OPTIMALLY (scenario.optimal_action each turn) for up to n_turns, capturing the
643
+ pre-move focal position per turn and the final safety distance. This is the
644
+ "ideal motive-reader" reference path the realized trajectory is scored against.
645
+ """
646
+
647
+ from __future__ import annotations
648
+
649
+ import random
650
+ from dataclasses import dataclass
651
+
652
+ from proteus.grid.difficulty import Difficulty
653
+ from proteus.grid.game import MotiveGridGame
654
+ from proteus.grid.scenario import get_scenario
655
+
656
+
657
+ @dataclass(frozen=True)
658
+ class RolloutResult:
659
+ """The optimal player's trajectory for one session."""
660
+
661
+ focal_positions: list[tuple[int, int]] # pre-move focal cell per optimal turn
662
+ final_safety_distance: int | None # BFS focal->predator at rollout end
663
+
664
+
665
+ def optimal_rollout(
666
+ scenario_name: str,
667
+ seed: int | None,
668
+ difficulty: Difficulty,
669
+ n_turns: int,
670
+ ) -> RolloutResult:
671
+ """Simulate the optimal trajectory and return its positions + final safety.
672
+
673
+ Args:
674
+ scenario_name: Registered scenario key.
675
+ seed: Session seed (same world as the real run).
676
+ difficulty: Session difficulty band.
677
+ n_turns: Cap on optimal play turns (use the realized play length so the
678
+ two trajectories align index-for-index).
679
+ """
680
+ scenario = get_scenario(scenario_name)()
681
+ rng = random.Random(seed)
682
+ cut_length = scenario.cut_length(difficulty)
683
+ game = MotiveGridGame(scenario, rng, difficulty, max_steps=cut_length + n_turns)
684
+
685
+ # Cut pre-roll (scripted policy), identical to SessionRunner.
686
+ for _ in range(cut_length):
687
+ action = scenario.cut_focal_policy(game)
688
+ game.apply_motive_action(action)
689
+ scenario.record_focal_move(action)
690
+
691
+ focal_positions: list[tuple[int, int]] = []
692
+ for _ in range(n_turns):
693
+ focal = game.focal_sprite
694
+ if focal is None:
695
+ break
696
+ focal_positions.append((focal.x, focal.y)) # pre-move
697
+ action = scenario.optimal_action(game)
698
+ game.apply_motive_action(action)
699
+ scenario.record_focal_move(action)
700
+ if game.eliminated or game.survived:
701
+ break
702
+
703
+ return RolloutResult(
704
+ focal_positions=focal_positions,
705
+ final_safety_distance=scenario.safety_distance(game),
706
+ )
707
+ ```
708
+
709
+ - [ ] **Step 4: Run test to verify it passes**
710
+
711
+ Run: `.venv/bin/python -m pytest tests/runtime/test_rollout.py -q`
712
+ Expected: PASS (3 tests). Then full suite: `.venv/bin/python -m pytest -q` → all pass.
713
+
714
+ - [ ] **Step 5: Commit**
715
+
716
+ ```bash
717
+ git add proteus/runtime/rollout.py tests/runtime/test_rollout.py
718
+ git commit -m "feat(cp6): runtime.rollout — deterministic optimal-trajectory simulator"
719
+ ```
720
+
721
+ ---
722
+
723
+ ## Task 6: Metrics — `away_move_fraction`, `mean_step_reward`, `trajectory_agreement`, `final_distance_gap`
724
+
725
+ **Files:**
726
+ - Modify: `proteus/runtime/metrics.py`
727
+ - Modify: `proteus/runtime/session.py` (wire rollout + safety distances into compute_metrics)
728
+ - Test: `tests/runtime/test_metrics.py` (extend)
729
+
730
+ - [ ] **Step 1: Write the failing test**
731
+
732
+ Append to `tests/runtime/test_metrics.py` (import `compute_metrics` and `TurnTrace` are already imported there; add if missing):
733
+ ```python
734
+ def _turn(idx, action, motive, reward, focal_pos):
735
+ return TurnTrace(
736
+ turn_idx=idx, observation="", probe_q="", probe_a="",
737
+ reasoning="", raw_text="", action=action, motive_action=motive,
738
+ habit_action="left", is_diagnostic=(motive != "left"),
739
+ was_congruent=(action == motive), reward=reward,
740
+ focal_pos=focal_pos, predator_pos=(0, 0),
741
+ )
742
+
743
+
744
+ def test_away_move_fraction_and_mean_reward():
745
+ turns = [
746
+ _turn(1, "up", "up", 1.0, (3, 3)), # away (reward > 0)
747
+ _turn(2, "right", "up", -1.0, (3, 2)), # toward (reward < 0)
748
+ _turn(3, "up", "up", 2.0, (4, 2)), # away
749
+ ]
750
+ m = compute_metrics(turns, played_turns=3, play_turns=5, outcome="eliminated")
751
+ assert m["away_move_fraction"] == 2 / 3 * 100.0
752
+ assert m["mean_step_reward"] == (1.0 - 1.0 + 2.0) / 3
753
+
754
+
755
+ def test_trajectory_agreement_and_final_gap():
756
+ turns = [
757
+ _turn(1, "up", "up", 1.0, (3, 3)),
758
+ _turn(2, "up", "up", 1.0, (3, 2)),
759
+ ]
760
+ # Optimal pre-move positions: turn1 matches (3,3), turn2 differs (4,2) vs (3,2).
761
+ m = compute_metrics(
762
+ turns, played_turns=2, play_turns=5, outcome="eliminated",
763
+ optimal_focal_positions=[(3, 3), (4, 2)],
764
+ realized_final_safety=2, optimal_final_safety=4,
765
+ )
766
+ assert m["trajectory_agreement"] == 1 / 2 * 100.0
767
+ assert m["final_distance_gap"] == 2.0
768
+
769
+
770
+ def test_new_metric_keys_default_zero_without_rollout():
771
+ turns = [_turn(1, "up", "up", 1.0, (3, 3))]
772
+ m = compute_metrics(turns, played_turns=1, play_turns=5, outcome="survived")
773
+ # Trajectory keys default to 0.0 when no rollout data is supplied.
774
+ assert m["trajectory_agreement"] == 0.0
775
+ assert m["final_distance_gap"] == 0.0
776
+ assert set(m) >= {
777
+ "motive_reading_accuracy", "reactivity_index", "survival_fraction",
778
+ "first_divergence_turn", "away_move_fraction", "mean_step_reward",
779
+ "trajectory_agreement", "final_distance_gap",
780
+ }
781
+ ```
782
+
783
+ - [ ] **Step 2: Run test to verify it fails**
784
+
785
+ Run: `.venv/bin/python -m pytest tests/runtime/test_metrics.py -q`
786
+ Expected: FAIL — `compute_metrics` does not accept `optimal_focal_positions` / lacks the new keys.
787
+
788
+ - [ ] **Step 3: Write minimal implementation**
789
+
790
+ In `proteus/runtime/metrics.py`, extend `compute_metrics`:
791
+ ```python
792
+ def compute_metrics(
793
+ turns: list[TurnTrace],
794
+ *,
795
+ played_turns: int,
796
+ play_turns: int,
797
+ outcome: str,
798
+ optimal_focal_positions: list[tuple[int, int]] | None = None,
799
+ realized_final_safety: int | None = None,
800
+ optimal_final_safety: int | None = None,
801
+ ) -> dict[str, float]:
802
+ ```
803
+ Update the early `n == 0` return dict to include the four new keys (all `0.0`):
804
+ ```python
805
+ if n == 0:
806
+ return {
807
+ "motive_reading_accuracy": 0.0,
808
+ "reactivity_index": 0.0,
809
+ "survival_fraction": 0.0,
810
+ "first_divergence_turn": 0.0,
811
+ "away_move_fraction": 0.0,
812
+ "mean_step_reward": 0.0,
813
+ "trajectory_agreement": 0.0,
814
+ "final_distance_gap": 0.0,
815
+ }
816
+ ```
817
+ After computing the existing metrics, add:
818
+ ```python
819
+ away_moves = sum(1 for t in turns if t.reward > 0)
820
+ away_move_fraction = away_moves / n * 100.0
821
+ mean_step_reward = sum(t.reward for t in turns) / n
822
+
823
+ trajectory_agreement = 0.0
824
+ if optimal_focal_positions:
825
+ realized = [tuple(t.focal_pos) for t in turns]
826
+ optimal = [tuple(p) for p in optimal_focal_positions]
827
+ compared = min(len(realized), len(optimal))
828
+ agree = sum(1 for i in range(compared) if realized[i] == optimal[i])
829
+ trajectory_agreement = agree / n * 100.0
830
+
831
+ final_distance_gap = 0.0
832
+ if realized_final_safety is not None and optimal_final_safety is not None:
833
+ final_distance_gap = float(abs(realized_final_safety - optimal_final_safety))
834
+ ```
835
+ and add the four keys to the returned dict.
836
+
837
+ In `proteus/runtime/session.py`, import the rollout and `safety_distance`, and wire them into the `compute_metrics` call. At the top of `run()`'s metrics section (after the play loop, where `compute_metrics` is currently called):
838
+ ```python
839
+ from proteus.runtime.rollout import optimal_rollout
840
+
841
+ rollout = optimal_rollout(
842
+ self._scenario_name, self._seed, self._difficulty, len(turns),
843
+ )
844
+ realized_final_safety = self._scenario.safety_distance(self._game)
845
+ metrics = compute_metrics(
846
+ turns,
847
+ played_turns=len(turns),
848
+ play_turns=self._play_turns,
849
+ outcome=outcome,
850
+ optimal_focal_positions=rollout.focal_positions,
851
+ realized_final_safety=realized_final_safety,
852
+ optimal_final_safety=rollout.final_safety_distance,
853
+ )
854
+ ```
855
+ (Use the existing `self._scenario_name` / `self._seed` / `self._difficulty` attributes — confirm their exact names in `__init__`; if the scenario name is stored differently, adapt. The import is local to avoid a top-level cycle, matching the codebase's lazy-import convention.)
856
+
857
+ - [ ] **Step 4: Run test to verify it passes**
858
+
859
+ Run: `.venv/bin/python -m pytest tests/runtime/test_metrics.py -q`
860
+ Expected: PASS. Then full suite: `.venv/bin/python -m pytest -q`. The golden test `tests/runtime/test_integration_golden.py` likely pins the full metrics dict — update its expected dict to include the four new keys with their now-correct values (read them from the failure output; verify they are sane, e.g. an optimal-playing fake agent yields high `trajectory_agreement`). `test_human_comparability`'s `set(h.metrics) == set(v.metrics)` still holds (both gain the same keys).
861
+
862
+ - [ ] **Step 5: Commit**
863
+
864
+ ```bash
865
+ git add proteus/runtime/metrics.py proteus/runtime/session.py tests/runtime/test_metrics.py tests/runtime/test_integration_golden.py
866
+ git commit -m "feat(cp6): trajectory + category-reward metrics (away_move_fraction, trajectory_agreement, ...)"
867
+ ```
868
+
869
+ ---
870
+
871
+ ## Task 7: `runtime/aggregate.py` — pure trace aggregation
872
+
873
+ **Files:**
874
+ - Create: `proteus/runtime/aggregate.py`
875
+ - Test: `tests/runtime/test_aggregate.py` (create)
876
+
877
+ - [ ] **Step 1: Write the failing test**
878
+
879
+ Create `tests/runtime/test_aggregate.py`:
880
+ ```python
881
+ from proteus.agents import VanillaAgent
882
+ from proteus.providers import FakeProvider
883
+ from proteus.runtime.aggregate import aggregate_traces
884
+ from proteus.runtime.session import SessionRunner
885
+
886
+
887
+ def _trace(seed):
888
+ agent = VanillaAgent(FakeProvider(["ACTION: up"]))
889
+ return SessionRunner(
890
+ "predator_evade", agent, seed=seed, play_turns=4, use_probe=False,
891
+ ).run()
892
+
893
+
894
+ def test_aggregate_groups_by_model_and_difficulty():
895
+ traces = [_trace(1), _trace(2)]
896
+ groups = aggregate_traces(traces)
897
+ # One group: (model="fake", difficulty="easy").
898
+ assert ("fake", "easy") in groups
899
+ g = groups[("fake", "easy")]
900
+ assert g["n"] == 2
901
+ # Every metric key is aggregated to a mean (float).
902
+ assert "motive_reading_accuracy" in g["metrics"]
903
+ assert isinstance(g["metrics"]["motive_reading_accuracy"], float)
904
+
905
+
906
+ def test_aggregate_means_are_correct():
907
+ traces = [_trace(1), _trace(2)]
908
+ groups = aggregate_traces(traces)
909
+ g = groups[("fake", "easy")]
910
+ expected = sum(t.metrics["mean_step_reward"] for t in traces) / 2
911
+ assert abs(g["metrics"]["mean_step_reward"] - expected) < 1e-9
912
+
913
+
914
+ def test_aggregate_empty_returns_empty():
915
+ assert aggregate_traces([]) == {}
916
+ ```
917
+
918
+ - [ ] **Step 2: Run test to verify it fails**
919
+
920
+ Run: `.venv/bin/python -m pytest tests/runtime/test_aggregate.py -q`
921
+ Expected: FAIL — `ModuleNotFoundError: No module named 'proteus.runtime.aggregate'`.
922
+
923
+ - [ ] **Step 3: Write minimal implementation**
924
+
925
+ Create `proteus/runtime/aggregate.py`:
926
+ ```python
927
+ """Aggregate SessionTraces by (model, difficulty) for the human-baseline harness.
928
+
929
+ Pure functions only (no I/O), so the aggregation is headless-testable. The CLI
930
+ `compare` command reads JSONL traces and renders the result of aggregate_traces.
931
+ """
932
+
933
+ from __future__ import annotations
934
+
935
+ from proteus.runtime.trace import SessionTrace
936
+
937
+
938
+ def aggregate_traces(
939
+ traces: list[SessionTrace],
940
+ ) -> dict[tuple[str, str], dict]:
941
+ """Group traces by (model, difficulty) and average each metric.
942
+
943
+ Args:
944
+ traces: SessionTraces to aggregate (mixed models/difficulties OK).
945
+
946
+ Returns:
947
+ ``{(model, difficulty): {"n": int, "metrics": {key: mean_float}}}``.
948
+ Empty input returns an empty dict.
949
+ """
950
+ groups: dict[tuple[str, str], list[SessionTrace]] = {}
951
+ for t in traces:
952
+ groups.setdefault((t.model, t.difficulty), []).append(t)
953
+
954
+ result: dict[tuple[str, str], dict] = {}
955
+ for key, group in groups.items():
956
+ metric_keys: set[str] = set()
957
+ for t in group:
958
+ metric_keys.update(t.metrics)
959
+ means = {
960
+ k: sum(t.metrics.get(k, 0.0) for t in group) / len(group)
961
+ for k in sorted(metric_keys)
962
+ }
963
+ result[key] = {"n": len(group), "metrics": means}
964
+ return result
965
+ ```
966
+
967
+ - [ ] **Step 4: Run test to verify it passes**
968
+
969
+ Run: `.venv/bin/python -m pytest tests/runtime/test_aggregate.py -q`
970
+ Expected: PASS (3 tests). Then full suite: `.venv/bin/python -m pytest -q` → all pass.
971
+
972
+ - [ ] **Step 5: Commit**
973
+
974
+ ```bash
975
+ git add proteus/runtime/aggregate.py tests/runtime/test_aggregate.py
976
+ git commit -m "feat(cp6): runtime.aggregate — pure (model,difficulty) trace aggregation"
977
+ ```
978
+
979
+ ---
980
+
981
+ ## Task 8: CLI `compare` subcommand
982
+
983
+ **Files:**
984
+ - Modify: `proteus/cli.py`
985
+ - Test: `tests/cli/test_cli.py` (extend)
986
+
987
+ - [ ] **Step 1: Write the failing test**
988
+
989
+ Append to `tests/cli/test_cli.py`:
990
+ ```python
991
+ def test_compare_aggregates_traces(tmp_path, capsys):
992
+ out = tmp_path / "runs.jsonl"
993
+ # Two fake-model traces at the same difficulty.
994
+ for seed in (1, 2):
995
+ main([
996
+ "run", "--scenario", "predator_evade", "--model", "fake:x",
997
+ "--seed", str(seed), "--play-turns", "4", "--no-probe", "--out", str(out),
998
+ ])
999
+ capsys.readouterr() # drain
1000
+ rc = main(["compare", str(out)])
1001
+ text = capsys.readouterr().out
1002
+ assert rc == 0
1003
+ assert "fake" in text and "easy" in text
1004
+ assert "n=2" in text
1005
+ assert "motive_reading_accuracy" in text
1006
+
1007
+
1008
+ def test_compare_writes_summary_json(tmp_path):
1009
+ import json
1010
+
1011
+ out = tmp_path / "runs.jsonl"
1012
+ main([
1013
+ "run", "--scenario", "predator_evade", "--model", "fake:x",
1014
+ "--seed", "1", "--play-turns", "4", "--no-probe", "--out", str(out),
1015
+ ])
1016
+ summary = tmp_path / "summary.json"
1017
+ rc = main(["compare", str(out), "--out", str(summary)])
1018
+ assert rc == 0
1019
+ data = json.loads(summary.read_text())
1020
+ assert data # non-empty aggregate
1021
+
1022
+
1023
+ def test_compare_missing_file_errors(capsys):
1024
+ rc = main(["compare", "/no/such/file.jsonl"])
1025
+ assert rc == 2
1026
+ assert "not found" in capsys.readouterr().err
1027
+ ```
1028
+
1029
+ - [ ] **Step 2: Run test to verify it fails**
1030
+
1031
+ Run: `.venv/bin/python -m pytest tests/cli/test_cli.py::test_compare_aggregates_traces -q`
1032
+ Expected: FAIL — argparse rejects the unknown `compare` subcommand.
1033
+
1034
+ - [ ] **Step 3: Write minimal implementation**
1035
+
1036
+ In `proteus/cli.py`, add the handler (after `_cmd_replay`):
1037
+ ```python
1038
+ def _cmd_compare(args: argparse.Namespace) -> int:
1039
+ import json
1040
+
1041
+ from proteus.runtime.aggregate import aggregate_traces
1042
+
1043
+ all_traces = []
1044
+ for path in args.trace_files:
1045
+ try:
1046
+ all_traces.extend(read_traces(path))
1047
+ except FileNotFoundError:
1048
+ print(f"trace file not found: {path}", file=sys.stderr)
1049
+ return 2
1050
+ if not all_traces:
1051
+ print("no traces to compare", file=sys.stderr)
1052
+ return 1
1053
+
1054
+ groups = aggregate_traces(all_traces)
1055
+ for (model, difficulty), info in sorted(groups.items()):
1056
+ print(f"=== model={model} difficulty={difficulty} n={info['n']} ===")
1057
+ for key, mean in info["metrics"].items():
1058
+ print(f" {key}: {mean:.2f}")
1059
+
1060
+ if args.out:
1061
+ serializable = {
1062
+ f"{model}|{difficulty}": info
1063
+ for (model, difficulty), info in groups.items()
1064
+ }
1065
+ with open(args.out, "w") as fh:
1066
+ json.dump(serializable, fh, indent=2)
1067
+ print(f"summary written to {args.out}")
1068
+ return 0
1069
+ ```
1070
+ In `build_parser()`, register the subcommand (after the `replay` block):
1071
+ ```python
1072
+ compare = sub.add_parser(
1073
+ "compare", help="aggregate traces by (model, difficulty) for baseline comparison"
1074
+ )
1075
+ compare.add_argument("trace_files", nargs="+", help="one or more .jsonl trace files")
1076
+ compare.add_argument(
1077
+ "--out", default=None, help="optional JSON file to write the aggregate summary to"
1078
+ )
1079
+ compare.set_defaults(func=_cmd_compare)
1080
+ ```
1081
+
1082
+ - [ ] **Step 4: Run test to verify it passes**
1083
+
1084
+ Run: `.venv/bin/python -m pytest tests/cli/test_cli.py -q`
1085
+ Expected: PASS (all CLI tests + the three new compare tests). Then full suite: `.venv/bin/python -m pytest -q` → all pass.
1086
+
1087
+ - [ ] **Step 5: Commit**
1088
+
1089
+ ```bash
1090
+ git add proteus/cli.py tests/cli/test_cli.py
1091
+ git commit -m "feat(cp6): CLI compare — aggregate human/LLM traces by (model, difficulty)"
1092
+ ```
1093
+
1094
+ ---
1095
+
1096
+ ## Task 9: Full-suite verification + per-difficulty demo + HANDOFF
1097
+
1098
+ - [ ] **Step 1: Run the complete offline suite**
1099
+
1100
+ Run: `.venv/bin/python -m pytest -q`
1101
+ Expected: ALL pass (103 baseline + all CP6 additions), no network, no display.
1102
+
1103
+ - [ ] **Step 2: Per-difficulty smoke (deterministic, offline)**
1104
+
1105
+ Run:
1106
+ ```bash
1107
+ for d in easy medium hard expert; do
1108
+ .venv/bin/python -m proteus run --scenario predator_evade --model fake:demo \
1109
+ --difficulty $d --seed 42 --play-turns 6 --no-probe --out runs/cp6_$d.jsonl
1110
+ done
1111
+ .venv/bin/python -m proteus compare runs/cp6_easy.jsonl runs/cp6_medium.jsonl \
1112
+ runs/cp6_hard.jsonl runs/cp6_expert.jsonl --out runs/cp6_summary.json
1113
+ ```
1114
+ Expected: each `run` prints a summary line for its difficulty; `compare` prints four `model=demo difficulty=<band> n=1` groups with the new metric means (including `away_move_fraction`, `trajectory_agreement`, `final_distance_gap`) and writes `runs/cp6_summary.json`.
1115
+
1116
+ - [ ] **Step 3: Verify the diagnostic invariant holds per band (acceptance)**
1117
+
1118
+ Run:
1119
+ ```bash
1120
+ .venv/bin/python -c "
1121
+ import random
1122
+ from proteus.grid.difficulty import Difficulty
1123
+ from proteus.grid.game import MotiveGridGame
1124
+ from proteus.grid.scenario import get_scenario
1125
+ for d in Difficulty:
1126
+ s = get_scenario('predator_evade')()
1127
+ g = MotiveGridGame(s, random.Random(42), d, max_steps=40)
1128
+ for _ in range(s.cut_length(d)):
1129
+ a = s.cut_focal_policy(g); g.apply_motive_action(a); s.record_focal_move(a)
1130
+ o, h = s.optimal_action(g), s.habit_action(g)
1131
+ assert o != h, f'{d}: optimal {o} == habit {h}'
1132
+ print(f'{d.value}: optimal={o} habit={h} grid={g.current_grid().shape} OK')
1133
+ "
1134
+ ```
1135
+ Expected: prints `OK` for all four bands (optimal != habit each).
1136
+
1137
+ - [ ] **Step 4: Update HANDOFF.md (CP6 close)**
1138
+
1139
+ Edit `HANDOFF.md`:
1140
+ - Update "Last updated" to today, "CP6 complete".
1141
+ - Add a "Done (CP6)" section: per-difficulty hand-authored layouts (EASY unchanged), scenario-owned `step_reward` (survival = away-from-predator reward) replacing the generic schedule, `runtime/rollout.py` optimal-trajectory simulator, four new metrics, `proteus compare` harness. Record the acceptance result from Steps 2–3 and the new full-suite count.
1142
+ - Move "Next" to CP7 (new motive categories/scenarios — the `step_reward` interface is now open; web UI still deferred).
1143
+ - Resolve the `record_focal_move`-on-ABC and (if addressed) the metric-key deferred items; carry forward the still-open `--png`-without-viz-extra and `first_divergence_turn` items.
1144
+
1145
+ Then commit:
1146
+ ```bash
1147
+ git add HANDOFF.md
1148
+ git commit -m "docs(cp6): handoff close — difficulty layouts + category eval + compare shipped; CP7 next"
1149
+ ```
1150
+
1151
+ ---
1152
+
1153
+ ## Self-Review (completed by plan author)
1154
+
1155
+ **Spec coverage:**
1156
+ - §4 (Section A difficulty layouts) → Tasks 1 (ABC + threading), 2 (layout table + dispatch), 3 (per-band invariant). `record_focal_move` ABC default → Task 1.
1157
+ - §5 (Section B category eval) → Task 4 (`step_reward` survival + `safety_distance`), Task 5 (optimal rollout), Task 6 (four metrics + wiring). Pre-move-position passing → Task 4 (uses the runner's already-captured `focal_pos`/`predator_pos`).
1158
+ - §6 (Section C harness) → Task 7 (`aggregate_traces`), Task 8 (`compare` CLI).
1159
+ - §7.4 completion criteria → Task 9 (full suite + per-difficulty demo + invariant acceptance).
1160
+
1161
+ **Placeholder scan:** No TBD/TODO. The layout coordinates in Task 2 are concrete candidates; Task 3 makes the golden invariant the oracle with an explicit tuning procedure (not a placeholder — the test defines done). The reward-constant magnitudes are concrete; the property test pins the signs.
1162
+
1163
+ **Type consistency:** `step_reward(game, action, blocked, focal_before, predator_before) -> float` is identical across the ABC (Task 4), predator_evade (Task 4), the SessionRunner call (Task 4), and the unit test (Task 4). `RolloutResult(focal_positions, final_safety_distance)` defined in Task 5 is consumed identically in Task 6's `compute_metrics` wiring. `compute_metrics(..., optimal_focal_positions, realized_final_safety, optimal_final_safety)` defined in Task 6 matches its test. `aggregate_traces(traces) -> {(model,difficulty): {"n", "metrics"}}` defined in Task 7 matches the CLI consumer (Task 8) and tests. `safety_distance(game) -> int|None` is consistent across the ABC default (Task 1), predator_evade (Task 4), rollout (Task 5), and session wiring (Task 6).
1164
+
1165
+ **Known additive churn (flagged in-task):** `test_integration_golden.py` / `test_session.py` pin exact reward values (Task 4) and the metrics dict (Task 6); both tasks instruct updating the expected values to the new correct numbers without weakening assertions.