v2 — Rollout Behavior Scoring

v1 (scripts/score_skill.py) lints the skill text for rule coverage. That is necessary but not sufficient: a rule can be present in the law and the agent still violates it. (Observed in practice: an agent's law layer scored 8/8 rule-coverage while the agent committed several of those exact mistakes in the same session.) Coverage of the law ≠ disciplined behavior.

v2 (scripts/rollout_score.py) scores the agent's response to a realistic prompt — the axis the SkillOpt paper (arXiv 2605.23904) is actually about — and gates on whether the skill changes behavior on the held-out scenarios.

Harness-agnostic adapters

--harness selects how a (skill, prompt) becomes a response (the SkillOpt adapter interface):

harness	needs	use
`transcript`	nothing (offline)	CI + self-test; reads a pre-recorded response map
`echo`	nothing	plumbing test only
`claude`	`ANTHROPIC_API_KEY`	the Claude Code / acer harness
`codex`	`OPENAI_API_KEY`	the Codex / liris harness

Live adapters error clearly when the key is absent; no key is ever read into the report or stored. This is what makes the harness work across both seats — Claude and Codex are first-class targets.

Run

# offline self-test: a "good" trajectory should pass, a "bad" one should fail
python scripts/rollout_score.py --scenarios examples/asolaria-scenarios.json \
  --harness transcript --transcript examples/transcripts/good.json   # -> ok=1 8/8
python scripts/rollout_score.py --scenarios examples/asolaria-scenarios.json \
  --harness transcript --transcript examples/transcripts/bad.json    # -> ok=0 0/8

# offline delta self-test: good with-skill trajectory vs bad no-skill baseline
python scripts/rollout_score.py --scenarios examples/asolaria-scenarios.json \
  --harness transcript --transcript examples/transcripts/good-with-bad-baseline.json \
  --baseline                                                    # -> ok=1 8/8, baseline=0, delta=8

# live rollout with the Claude (acer) harness, skill vs no-skill delta
ANTHROPIC_API_KEY=... python scripts/rollout_score.py \
  --scenarios examples/asolaria-scenarios.json --harness claude \
  --skill ../path/to/SKILL.md --baseline

--baseline runs each scenario with the skill and with no skill, and reports skill_delta (the SkillOpt forward pass: does the skill actually help?). The gate: an edit is accepted only if it raises the rollout pass-rate without regressing a passing scenario.

Scenario format

examples/asolaria-scenarios.json is the canonical scenario file for both v1 and v2. Each scenario keeps the v1 must_include_any / must_not_include fields and adds a realistic prompt plus rollout rubric:

{
  "id": "...",
  "failure": "...",
  "prompt": "the situation that triggers the mistake",
  "rubric": {
    "apply_any": ["markers a correct response shows"],
    "fail_any":  ["markers of committing the mistake"]
  }
}

A response passes iff it shows ≥1 apply_any marker and 0 fail_any markers.

Honest boundaries

The rubric is keyword-based — the right axis (behavior), but a coarse judge. A v3 should use an LLM-as-judge with the same scenario set; the adapter + gate plumbing here is unchanged.
transcript mode is fully offline and deterministic (good for CI). The claude/codex adapters do real rollouts but need keys + network, so they are not run in CI by default.
This is still a scaffold of SkillOpt, not the full optimizer loop (no edit proposer / textual learning rate yet). It is the missing behavior gate; the edit-proposer is the next layer.