# v2 — Rollout Behavior Scoring

v1 (`scripts/score_skill.py`) lints the **skill text** for rule coverage. That is necessary but not
sufficient: a rule can be present in the law and the agent still violates it. (Observed in practice:
an agent's law layer scored **8/8** rule-coverage while the agent committed several of those exact
mistakes in the same session.) Coverage of the law ≠ disciplined behavior.

**v2 (`scripts/rollout_score.py`) scores the agent's *response* to a realistic prompt** — the axis
the SkillOpt paper (arXiv 2605.23904) is actually about — and gates on whether the skill changes
behavior on the held-out scenarios.

## Harness-agnostic adapters

`--harness` selects how a `(skill, prompt)` becomes a response (the SkillOpt adapter interface):

| harness | needs | use |
|---|---|---|
| `transcript` | nothing (offline) | CI + self-test; reads a pre-recorded response map |
| `echo` | nothing | plumbing test only |
| `claude` | `ANTHROPIC_API_KEY` | the **Claude Code / acer** harness |
| `codex` | `OPENAI_API_KEY` | the **Codex / liris** harness |

Live adapters error clearly when the key is absent; no key is ever read into the report or stored.
This is what makes the harness work across both seats — Claude and Codex are first-class targets.

## Run

```bash
# offline self-test: a "good" trajectory should pass, a "bad" one should fail
python scripts/rollout_score.py --scenarios examples/asolaria-scenarios.json \
  --harness transcript --transcript examples/transcripts/good.json   # -> ok=1 8/8
python scripts/rollout_score.py --scenarios examples/asolaria-scenarios.json \
  --harness transcript --transcript examples/transcripts/bad.json    # -> ok=0 0/8

# offline delta self-test: good with-skill trajectory vs bad no-skill baseline
python scripts/rollout_score.py --scenarios examples/asolaria-scenarios.json \
  --harness transcript --transcript examples/transcripts/good-with-bad-baseline.json \
  --baseline                                                    # -> ok=1 8/8, baseline=0, delta=8

# live rollout with the Claude (acer) harness, skill vs no-skill delta
ANTHROPIC_API_KEY=... python scripts/rollout_score.py \
  --scenarios examples/asolaria-scenarios.json --harness claude \
  --skill ../path/to/SKILL.md --baseline
```

`--baseline` runs each scenario with the skill **and** with no skill, and reports `skill_delta`
(the SkillOpt forward pass: does the skill actually help?). The gate: an edit is accepted only if it
raises the rollout pass-rate without regressing a passing scenario.

## Scenario format

`examples/asolaria-scenarios.json` is the canonical scenario file for both v1 and v2.
Each scenario keeps the v1 `must_include_any` / `must_not_include` fields and adds a realistic
`prompt` plus rollout `rubric`:

```json
{
  "id": "...",
  "failure": "...",
  "prompt": "the situation that triggers the mistake",
  "rubric": {
    "apply_any": ["markers a correct response shows"],
    "fail_any":  ["markers of committing the mistake"]
  }
}
```

A response **passes** iff it shows ≥1 `apply_any` marker and **0** `fail_any` markers.

## Honest boundaries

- The rubric is **keyword-based** — the right *axis* (behavior), but a coarse *judge*. A v3 should
  use an LLM-as-judge with the same scenario set; the adapter + gate plumbing here is unchanged.
- `transcript` mode is fully offline and deterministic (good for CI). The `claude`/`codex` adapters
  do real rollouts but need keys + network, so they are not run in CI by default.
- This is still a **scaffold of SkillOpt**, not the full optimizer loop (no edit proposer / textual
  learning rate yet). It is the missing *behavior gate*; the edit-proposer is the next layer.