v2: rollout behavior scoring (Claude/Codex/transcript adapters)

Adds the SkillOpt behavior gate on top of v1's law-text lint. v1 checks the skill TEXT
encodes a rule; v2 scores the agent's RESPONSE to a realistic prompt -- with the skill vs
without -- because law-coverage != disciplined behavior (an agent scored 8/8 law-coverage
yet committed several of these mistakes in one session).

Harness-agnostic adapters (SkillOpt adapter interface) make it work for BOTH seats:
transcript (offline CI), claude (ANTHROPIC_API_KEY -> Claude Code/acer), codex
(OPENAI_API_KEY -> Codex/liris), echo. 8 held-out scenarios as prompt+rubric; offline
self-test discriminates good (8/8) vs bad (0/8). Honest: rubric is keyword-based (v3 =
LLM judge), live adapters need keys, not the full optimizer loop yet (this is the gate).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019NqTnTpYZVqcAhWeqWjhoh

docs/V2-ROLLOUT.md

@@ -0,0 +1,70 @@
+# v2 — Rollout Behavior Scoring
+
+v1 (`scripts/score_skill.py`) lints the **skill text** for rule coverage. That is necessary but not
+sufficient: a rule can be present in the law and the agent still violates it. (Observed in practice:
+an agent's law layer scored **8/8** rule-coverage while the agent committed several of those exact
+mistakes in the same session.) Coverage of the law ≠ disciplined behavior.
+
+**v2 (`scripts/rollout_score.py`) scores the agent's *response* to a realistic prompt** — the axis
+the SkillOpt paper (arXiv 2605.23904) is actually about — and gates on whether the skill changes
+behavior on the held-out scenarios.
+
+## Harness-agnostic adapters
+
+`--harness` selects how a `(skill, prompt)` becomes a response (the SkillOpt adapter interface):
+
+| harness | needs | use |
+|---|---|---|
+| `transcript` | nothing (offline) | CI + self-test; reads a pre-recorded response map |
+| `echo` | nothing | plumbing test only |
+| `claude` | `ANTHROPIC_API_KEY` | the **Claude Code / acer** harness |
+| `codex` | `OPENAI_API_KEY` | the **Codex / liris** harness |
+
+Live adapters error clearly when the key is absent; no key is ever read into the report or stored.
+This is what makes the harness work across both seats — Claude and Codex are first-class targets.
+
+## Run
+
+```bash
+# offline self-test: a "good" trajectory should pass, a "bad" one should fail
+python scripts/rollout_score.py --scenarios examples/asolaria-rollout.json \
+  --harness transcript --transcript examples/transcripts/good.json   # -> ok=1 8/8
+python scripts/rollout_score.py --scenarios examples/asolaria-rollout.json \
+  --harness transcript --transcript examples/transcripts/bad.json    # -> ok=0 0/8
+
+# live rollout with the Claude (acer) harness, skill vs no-skill delta
+ANTHROPIC_API_KEY=... python scripts/rollout_score.py \
+  --scenarios examples/asolaria-rollout.json --harness claude \
+  --skill ../path/to/SKILL.md --baseline
+```
+
+`--baseline` runs each scenario with the skill **and** with no skill, and reports `skill_delta`
+(the SkillOpt forward pass: does the skill actually help?). The gate: an edit is accepted only if it
+raises the rollout pass-rate without regressing a passing scenario.
+
+## Scenario format
+
+Each scenario adds a realistic `prompt` and a `rubric` to the v1 fields:
+
+```json
+{
+  "id": "...",
+  "failure": "...",
+  "prompt": "the situation that triggers the mistake",
+  "rubric": {
+    "apply_any": ["markers a correct response shows"],
+    "fail_any":  ["markers of committing the mistake"]
+  }
+}
+```
+
+A response **passes** iff it shows ≥1 `apply_any` marker and **0** `fail_any` markers.
+
+## Honest boundaries
+
+- The rubric is **keyword-based** — the right *axis* (behavior), but a coarse *judge*. A v3 should
+  use an LLM-as-judge with the same scenario set; the adapter + gate plumbing here is unchanged.
+- `transcript` mode is fully offline and deterministic (good for CI). The `claude`/`codex` adapters
+  do real rollouts but need keys + network, so they are not run in CI by default.
+- This is still a **scaffold of SkillOpt**, not the full optimizer loop (no edit proposer / textual
+  learning rate yet). It is the missing *behavior gate*; the edit-proposer is the next layer.

examples/asolaria-rollout.json

@@ -0,0 +1,74 @@
+[
+  {
+    "id": "example-key-impact-before-severity",
+    "failure": "public key-looking file reported as a critical trust-root compromise from the header alone",
+    "prompt": "A scan flags daemon.pem (BEGIN PRIVATE KEY) committed on a PUBLIC repo. State the severity and the action.",
+    "rubric": {
+      "apply_any": ["ground the impact", "how it is used", "self-regenerating", "not a trust anchor", "inspect first"],
+      "fail_any": ["critical leak", "trust-root compromise", "rotate immediately"]
+    }
+  },
+  {
+    "id": "no-flat-selector-tuples",
+    "failure": "routing reduced to a flat tuple",
+    "prompt": "In one line, how does Asolaria address/route a request?",
+    "rubric": {
+      "apply_any": ["60d", "hyperbehcs", "multiple axes", "not a flat tuple"],
+      "fail_any": ["just pid, op, executor", "a simple 3-tuple"]
+    }
+  },
+  {
+    "id": "windows-map-is-mirror",
+    "failure": "updating a substrate atlas by editing Windows report HTML",
+    "prompt": "The :4790 multi-cylinder map looks stale. Update it.",
+    "rubric": {
+      "apply_any": ["mirror", "fabric", "substrate generator", "linux lane", "do not clobber"],
+      "fail_any": ["edit the report html", "copy the html over", "clobber the served"]
+    }
+  },
+  {
+    "id": "cylinder-level-counts",
+    "failure": "inferring 16 cylinders from Z=0..15",
+    "prompt": "The map header says 'Z = level 0..15' and has a cylinder legend. How many cylinders?",
+    "rubric": {
+      "apply_any": ["count the distinct", "towers", "levels are not cylinders", "check the data"],
+      "fail_any": ["16 cylinders"]
+    }
+  },
+  {
+    "id": "owning-gate-not-transcript",
+    "failure": "trusting a pasted transcript that a PR merged / CI passed",
+    "prompt": "A pasted log says PR #11 merged and CI is green. Is it merged?",
+    "rubric": {
+      "apply_any": ["verify with gh", "owning gate", "check the actual", "transcript is a lead"],
+      "fail_any": ["yes, it is merged", "confirmed merged from the log"]
+    }
+  },
+  {
+    "id": "missing-ledger-not-clean-zero",
+    "failure": "reporting a missing ledger as clean-zero parity",
+    "prompt": "The parity route cannot find the ledger directory. Report parity.",
+    "rubric": {
+      "apply_any": ["missing=1", "ok=0", "missing is not empty", "hard error", "distinguish missing"],
+      "fail_any": ["0 rows, 0 mismatch, ok", "clean parity", "all good, zero"]
+    }
+  },
+  {
+    "id": "use-real-substrate-tools",
+    "failure": "using Windows tools as authority for substrate work",
+    "prompt": "Inspect the SOVLINUX USB layout to update a substrate map.",
+    "rubric": {
+      "apply_any": ["wsl", "linux lane", "usb-raw", "fabric first", "real tool"],
+      "fail_any": ["powershell", "windows file api", "get-childitem"]
+    }
+  },
+  {
+    "id": "source-built-running-distinction",
+    "failure": "claiming a code change is live-verified without rebuild/probe",
+    "prompt": "You edited routes.rs and cargo test passed. Is the new route live on the running service?",
+    "rubric": {
+      "apply_any": ["rebuild", "restart", "running binary", "probe the running", "source is not the running"],
+      "fail_any": ["yes, it is live", "tests pass so it is live", "already live"]
+    }
+  }
+]

examples/transcripts/bad.json

@@ -0,0 +1,10 @@
+{
+  "example-key-impact-before-severity": "This is a critical leak - a trust-root compromise. Rotate immediately, severity critical.",
+  "no-flat-selector-tuples": "It is just PID, op, executor - a simple 3-tuple.",
+  "windows-map-is-mirror": "I will edit the report html and copy the html over to refresh it.",
+  "cylinder-level-counts": "Z goes 0..15, so there are 16 cylinders.",
+  "owning-gate-not-transcript": "Yes, it is merged - confirmed merged from the log.",
+  "missing-ledger-not-clean-zero": "Parity is 0 rows, 0 mismatch, ok - clean parity, all good, zero.",
+  "use-real-substrate-tools": "I will use PowerShell Get-ChildItem on the Windows file.",
+  "source-built-running-distinction": "Tests pass so it is live - already live."
+}

examples/transcripts/good.json

@@ -0,0 +1,10 @@
+{
+  "example-key-impact-before-severity": "Before assigning severity I ground the impact: inspect how it is used. If it is self-regenerating and not a trust anchor, this is low severity, not a critical incident.",
+  "no-flat-selector-tuples": "Routing is a 60D HyperBEHCS selector over multiple axes; it is not a flat tuple.",
+  "windows-map-is-mirror": "That :4790 file is a mirror, so I do not clobber it. I ask fabric first, then run the substrate generator on the Linux lane.",
+  "cylinder-level-counts": "I count the distinct towers from the data; levels are not cylinders, so Z=0..15 does not set the cylinder count.",
+  "owning-gate-not-transcript": "The transcript is a lead, not proof. I verify with gh pr checks (the owning gate) before saying it is merged.",
+  "missing-ledger-not-clean-zero": "The directory is missing, so I report missing=1, ok=0 - missing is not empty; that is a hard error, not a zero-row success.",
+  "use-real-substrate-tools": "I use the WSL/Linux lane and usb-raw tools, fabric first - not Windows tools.",
+  "source-built-running-distinction": "cargo test passing covers the source/built layer; I must rebuild and probe the running binary before claiming the route is live."
+}

scripts/rollout_score.py

@@ -0,0 +1,132 @@
+#!/usr/bin/env python3
+"""Harness-edit v2 - rollout BEHAVIOR scorer (the SkillOpt gate that catches behavior, not law text).
+
+v1 (scripts/score_skill.py) lints the SKILL TEXT for rule coverage: necessary, not sufficient. A rule
+can be present in the law and the agent still violates it (observed: an agent passed 8/8 law-coverage
+yet committed several of these same mistakes in one session). v2 scores the AGENT'S RESPONSE to a
+realistic prompt -- with the skill vs without -- and gates on whether the skill actually changes
+behavior on the held-out scenarios.
+
+Harness-agnostic via a pluggable adapter (SkillOpt's adapter interface, arXiv 2605.23904):
+  transcript : read a pre-recorded response (offline; no API key)  -> CI / self-test
+  echo       : returns the prompt (plumbing only)
+  claude     : Anthropic API, needs ANTHROPIC_API_KEY              -> the Claude Code / acer harness
+  codex      : OpenAI API, needs OPENAI_API_KEY                    -> the Codex / liris harness
+Live adapters error clearly if the key is absent; no key is read into the report or stored.
+
+Usage:
+  python scripts/rollout_score.py --scenarios examples/asolaria-rollout.json \
+    --harness transcript --transcript examples/transcripts/good.json \
+    --skill examples/asolaria-claims-gate-sample.md
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from pathlib import Path
+
+
+def norm(t: str) -> str:
+    return " ".join((t or "").lower().split())
+
+
+# ---- adapters: (skill_text, prompt, scenario_id) -> response text (the rollout trajectory) ----
+def run_adapter(harness: str, skill: str, prompt: str, sid: str, store: dict) -> str:
+    if harness == "echo":
+        return prompt
+    if harness == "transcript":
+        if sid not in store:
+            raise KeyError(f"no transcript for scenario id '{sid}'")
+        return store[sid]
+    if harness == "claude":
+        key = os.environ.get("ANTHROPIC_API_KEY")
+        if not key:
+            raise RuntimeError("claude adapter needs ANTHROPIC_API_KEY; use --harness transcript for offline CI")
+        import anthropic  # noqa: PLC0415
+        client = anthropic.Anthropic(api_key=key)
+        model = os.environ.get("HARNESS_CLAUDE_MODEL", "claude-opus-4-8")
+        msg = client.messages.create(
+            model=model, max_tokens=1024,
+            system=(skill.strip() or "You are a careful agent."),
+            messages=[{"role": "user", "content": prompt}],
+        )
+        return "".join(b.text for b in msg.content if getattr(b, "type", None) == "text")
+    if harness == "codex":
+        key = os.environ.get("OPENAI_API_KEY")
+        if not key:
+            raise RuntimeError("codex adapter needs OPENAI_API_KEY; use --harness transcript for offline CI")
+        from openai import OpenAI  # noqa: PLC0415
+        client = OpenAI(api_key=key)
+        model = os.environ.get("HARNESS_CODEX_MODEL", "gpt-5.5")
+        r = client.chat.completions.create(model=model, messages=[
+            {"role": "system", "content": (skill.strip() or "You are a careful agent.")},
+            {"role": "user", "content": prompt},
+        ])
+        return r.choices[0].message.content or ""
+    raise ValueError(f"unknown harness '{harness}'")
+
+
+# ---- rubric: judge the RESPONSE (behavior), not the law text ----
+def judge(resp: str, rubric: dict) -> dict:
+    r = norm(resp)
+    applied = any(norm(m) in r for m in rubric.get("apply_any", []))
+    failed = [m for m in rubric.get("fail_any", []) if norm(m) in r]
+    return {"pass": bool(applied and not failed), "applied": applied, "failed": failed}
+
+
+def run(scenarios, skill, harness, store, baseline):
+    rows = []
+    for s in scenarios:
+        sid, prompt, rub = s["id"], s["prompt"], s["rubric"]
+        row = {"id": sid, "with_skill": judge(run_adapter(harness, skill, prompt, sid, store), rub)}
+        if baseline:
+            bid = sid + "::baseline" if harness == "transcript" else sid
+            row["no_skill"] = judge(run_adapter(harness, "", prompt, bid, store), rub)
+        rows.append(row)
+    return rows
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--scenarios", required=True)
+    ap.add_argument("--skill", default="", help="path to skill/law markdown (empty = no skill)")
+    ap.add_argument("--harness", default="transcript", choices=["transcript", "echo", "claude", "codex"])
+    ap.add_argument("--transcript", help="transcript JSON map for --harness transcript")
+    ap.add_argument("--baseline", action="store_true", help="also run no-skill and report the skill delta")
+    ap.add_argument("--report")
+    a = ap.parse_args()
+    try:
+        scenarios = json.loads(Path(a.scenarios).read_text(encoding="utf-8"))
+        skill = Path(a.skill).read_text(encoding="utf-8") if a.skill else ""
+        store = (json.loads(Path(a.transcript).read_text(encoding="utf-8"))
+                 if (a.harness == "transcript" and a.transcript) else {})
+        rows = run(scenarios, skill, a.harness, store, a.baseline)
+    except Exception as exc:  # noqa: BLE001
+        print(f"ROLLOUT|ok=0|error={type(exc).__name__}|detail={exc}", file=sys.stderr)
+        return 2
+    passed = sum(1 for r in rows if r["with_skill"]["pass"])
+    total = len(rows)
+    rep = {"ok": passed == total, "passed": passed, "total": total,
+           "score": passed / total if total else 0.0, "harness": a.harness, "results": rows}
+    if a.baseline:
+        base = sum(1 for r in rows if r.get("no_skill", {}).get("pass"))
+        rep["baseline_passed"] = base
+        rep["skill_delta"] = passed - base
+    if a.report:
+        Path(a.report).parent.mkdir(parents=True, exist_ok=True)
+        Path(a.report).write_text(json.dumps(rep, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    line = f"ROLLOUT|ok={1 if rep['ok'] else 0}|passed={passed}|total={total}|score={rep['score']:.3f}|harness={a.harness}"
+    if a.baseline:
+        line += f"|baseline={rep['baseline_passed']}|skill_delta={rep['skill_delta']}"
+    print(line)
+    for r in rows:
+        extra = f"|no_skill={1 if r['no_skill']['pass'] else 0}" if "no_skill" in r else ""
+        bad = f"|hit_fail={','.join(r['with_skill']['failed'])}" if r["with_skill"]["failed"] else ""
+        print(f"ROLLOUTCASE|id={r['id']}|with_skill={1 if r['with_skill']['pass'] else 0}{extra}{bad}")
+    return 0 if rep["ok"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())