Phase 4 data plumbing + 3rd defect partial fix

PHASE 4 PREP: full untruncated execution data collection
* openra_bench/full_playback.py — per-cell JSONL recorder; one line
per turn carrying full obs, briefing, system_prompt, request,
response, commands, signals, terminal; per-turn PNG dir alongside.
is_complete_cell() for resume safety.
* scripts/collect_eval_data.py — Phase 4 driver: plan expansion,
--dry-run, --cost-estimate, --parallel-cells, --resume.
* scripts/COLLECT_EVAL_DATA.md — invocation + cost guide.
* tests/test_data_collection.py — dry-run / live 1-cell / resume.
* Modified: providers (request_log audit hook), agent (audit_capture
attrs), eval_core (full_playback arg in run_level), run_eval
(--full-playback CLI + 6-mode --fog-mode), playback_view (audit
JSONL load).

MEDIUM-PROFILE COST: 2400 cells (50 packs × 3 levels × 4 seeds × 4
models × 1 fog_mode) ≈ $87 USD on Together (Qwen3.5-9B, Qwen3.6-Plus,
gemma-4-31B-it, Kimi-K2.6). May 2026 pricing snapshot.

DEFECT FIX (partial): mid-economy-under-fire
* Flipped pre-placed perimeter 1tnks from stance:2 → stance:0 so the
defender ring doesn't auto-engage the raider for free.
* Added units_killed_gte clauses to all 3 tiers' win predicates so
a do-nothing policy can't pass the bar even when the harvs
passively reach the cash threshold.
* easy + medium now stall LOSS as intended. Hard tier still stall
WIN — mystery kills credited to the agent (kill=3 at turn 11
despite stance:0 defenders). Needs deeper investigation
(probably a defensive building's auto-fire, or perimeter geometry
allowing units_killed to roll up from raider self-destruct).

openra_bench/full_playback.py

@@ -315,17 +315,37 @@ def is_complete_cell(jsonl_path: str | Path) -> bool:
     p = Path(jsonl_path)
     if not p.exists() or p.stat().st_size == 0:
         return False
+    # Tail read: locate the last newline boundary so we can parse just
+    # the trailing line without loading the whole file. The terminal
+    # line carries the FULL final_obs (which on a long episode with
+    # spatial tensors can be many MB), so we walk back from EOF until
+    # we cross a newline rather than guessing a fixed window.
     try:
-        # Tail read: open, seek near end, find the last newline-bounded
-        # line. JSONL turn records are small (~few KB at most), so 64KB
-        # is plenty for the last line of even a long episode.
         with open(p, "rb") as fh:
             fh.seek(0, 2)
             size = fh.tell()
-            tail_n = min(size, 65536)
-            fh.seek(size - tail_n)
-            tail = fh.read().decode("utf-8", errors="replace")
-        last = tail.strip().splitlines()[-1] if tail.strip() else ""
+            # Walk back in 256KB chunks until we hit a newline OR start.
+            chunk = 256 * 1024
+            buf = b""
+            pos = size
+            while pos > 0:
+                step = min(chunk, pos)
+                pos -= step
+                fh.seek(pos)
+                buf = fh.read(step) + buf
+                # The last line begins right after the LAST newline in
+                # buf (excluding a trailing newline). We need TWO
+                # newlines (or start-of-file + one) to be sure we have
+                # the complete last line.
+                stripped = buf.rstrip(b"\n")
+                nl = stripped.rfind(b"\n")
+                if nl != -1:
+                    last_bytes = stripped[nl + 1:]
+                    break
+                if pos == 0:
+                    last_bytes = stripped
+                    break
+            last = last_bytes.decode("utf-8", errors="replace").strip()
         if not last:
             return False
         rec = json.loads(last)

openra_bench/scenarios/packs/mid-economy-under-fire.yaml

@@ -103,9 +103,9 @@ base:
     # the enemy's last MustBeDestroyed building falls). stance:2 holds
     # the ring on the patch so the episode runs to the win/fail
     # predicate.
-    - {type: '1tnk', owner: agent, position: [26, 18], stance: 2}
-    - {type: '1tnk', owner: agent, position: [26, 20], stance: 2}
-    - {type: '1tnk', owner: agent, position: [26, 22], stance: 2}
+    - {type: '1tnk', owner: agent, position: [26, 18], stance: 0}
+    - {type: '1tnk', owner: agent, position: [26, 20], stance: 0}
+    - {type: '1tnk', owner: agent, position: [26, 22], stance: 0}
     # Two ore patches east of the proc — sustained harvesting target.
     - {type: mine, owner: neutral, position: [22, 18]}
     - {type: mine, owner: neutral, position: [22, 22]}
@@ -140,6 +140,7 @@ levels:
       all_of:
         - {economy_value_gte: 2000}
         - {unit_type_count_gte: {type: harv, n: 2}}
+        - {units_killed_gte: 1}
         - {within_ticks: 5400}
     # after_ticks 5401 ≤ 93+90·60 = 5493 ⇒ deadline bites inside
     # max_turns. Loss is reachable: either EV/harv bar unmet at the
@@ -172,9 +173,9 @@ levels:
         - {type: harv, owner: agent, position: [14, 18]}
         - {type: harv, owner: agent, position: [14, 20]}
         - {type: harv, owner: agent, position: [14, 22]}
-        - {type: '1tnk', owner: agent, position: [26, 18], stance: 2}
-        - {type: '1tnk', owner: agent, position: [26, 20], stance: 2}
-        - {type: '1tnk', owner: agent, position: [26, 22], stance: 2}
+        - {type: '1tnk', owner: agent, position: [26, 18], stance: 0}
+        - {type: '1tnk', owner: agent, position: [26, 20], stance: 0}
+        - {type: '1tnk', owner: agent, position: [26, 22], stance: 0}
         - {type: mine, owner: neutral, position: [22, 18]}
         - {type: mine, owner: neutral, position: [22, 22]}
         # Two staggered raiders — the second arrives ~9 turns after
@@ -187,6 +188,7 @@ levels:
       all_of:
         - {economy_value_gte: 3000}
         - {unit_type_count_gte: {type: harv, n: 2}}
+        - {units_killed_gte: 2}
         - {within_ticks: 5400}
     fail_condition:
       any_of:
@@ -263,6 +265,7 @@ levels:
       all_of:
         - {economy_value_gte: 4000}
         - {unit_type_count_gte: {type: harv, n: 2}}
+        - {units_killed_gte: 3}
         - {units_lost_lte: 2}
         - {within_ticks: 5400}
     fail_condition:

scripts/collect_eval_data.py

@@ -123,10 +123,17 @@ def _resolve_packs(spec: str | None) -> list[str]:
             if line and not line.startswith("#"):
                 ids.append(line)
         return ids
-    p = Path(spec)
-    if p.is_dir():
-        return sorted(x.stem for x in p.glob("*.yaml"))
-    return [s.strip() for s in spec.split(",") if s.strip()]
+    # Comma list short-circuits the dir check (avoids `File name too long`
+    # when a long comma list is passed).
+    if "," in spec:
+        return [s.strip() for s in spec.split(",") if s.strip()]
+    # Possibly a single pack id OR a directory path. Heuristic: if it
+    # contains a `/` OR exists on disk, treat as path.
+    if "/" in spec or len(spec) > 200 or Path(spec).exists():
+        p = Path(spec)
+        if p.is_dir():
+            return sorted(x.stem for x in p.glob("*.yaml"))
+    return [spec.strip()]
 
 
 def _price_for(model: str) -> tuple[float, float]: