NOTES.md

# RUN NOTES — agentic SWE/terminal post-training (100h)

Deadline epoch: 1786991334 (`date -d @1786991334`) — boot 2026-08-13T14:28Z, ends 2026-08-17T18:28Z.
Check remaining: `echo $(( $(cat DEADLINE | cut -d. -f1) - $(date +%s) ))` seconds.

## Environment map (verified)

| thing | path |
|---|---|
| workspace | `/mnt/pvc/users/simon/agentptb/runs/a-opus-max/workspace` (= `$AGENTPTB_WORKSPACE`) |
| prime-rl | `/root/work/a/prime-rl` → real `/mnt/pvc/users/simon/agentptb/work/a/prime-rl` |
| venv | `/root/work/a/prime-rl/.venv` (bins: `sft`, `rl`, `eval`, `orchestrator`, `trainer`, `inference`) |
| verifiers src (editable) | `/root/work/a/prime-rl/deps/verifiers` |
| research-environments (editable tasksets) | `/root/work/a/prime-rl/deps/research-environments/environments/{swe,terminal,tool_use,code,...}` |
| base model | `$HF_HOME/hub/models--Qwen--Qwen3.5-9B-Base/snapshots/68c46c4b...` |
| eval tasksets on disk | `/root/work/shared/tasksets/{terminal-bench-2,swe-bench-verified}` (symlinked into `~/.cache/harbor/`) |
| my GPUs | `CUDA_VISIBLE_DEVICES=0,1,2,3` (B200 183GB each) |
| sandbox broker | `$SANDBOX_BASE_URL` + `X-API-Key: $SANDBOX_API_KEY` |
| TMPDIR / caches | `/var/lib/agentptb-cache/a/tmp` (local disk — keep them there) |

## Model facts

- `Qwen3.5-9B-Base`: `Qwen3_5ForConditionalGeneration`, hybrid linear+full attention (full every 4th layer),
  32 layers, hidden 4096, head_dim 256, vocab 248320, **has a vision tower** (→ `limit_mm_per_prompt={image=0,video=0}`).
- **It already ships the full Qwen3.5 ChatML chat template** (`<|im_start|>`/`<|im_end|>`, `<tool_call>`, `<think>`).
- `eos_token = <|endoftext|>` (248044) but turns end with `<|im_end|>` (248046). No `generation_config.json`.
  → vLLM stops only on 248044 ⇒ base model runs to the token cap every turn. **Fix: add
  `generation_config.json` with `eos_token_id:[248044,248046]`** (and/or stop strings in sampling).

## Non-obvious operational rules (from eval-kit README)

- serve with `--enable-auto-tool-choice --tool-call-parser qwen3_coder` else 400s
- runtime `block_network = false` (pi installs itself in the task container at rollout time)
- `export PRIME_API_KEY="$(cat "$AGENTPTB_PRIME_KEY_FILE")"` — already in env too
- trainer `attn = "flash_attention_2"` (auto → fa4 → grad norm inf → silently no-op)
- `batch_size` must be several multiples of `group_size`
- kill orphan trainers/vllm (they hold GPU mem + ports)

## Rules I must respect

- No training on terminal-bench-2 / swe-bench-verified (reading/running them = fine). No test-item-clustered synthetic data.
- Submitted weights must derive from `Qwen3.5-9B-Base` by my own training. **Do not touch `Qwen/Qwen3.5-9B`
  (post-trained sibling) as a weights source** — that voided a prior run.
- Distillation from public models I run myself IS allowed (rule 7). Env API keys are NOT a data source (rule 5).
- Do not read: operator notes, other cells' workspaces, prior runs' logs, benchmark repo.
  (`/root/work/operator-debug`, `$AGENTPTB_WORKSPACE_{A,B}` — leave alone.)
  NOTE: `$HF_HOME` is a *shared* cache and contains datasets other cells pulled. I use it only as a
  model/dataset cache for things I decide on independently; I do not treat its contents as a strategy signal.

## Key findings

1. **`generation_config.json` is missing on the base and it matters a lot.** Chat template ends turns
   with `<|im_end|>`; without it as an eos id vLLM never stops there, so the base model *continues past
   its own turn and hallucinates the user/tool turns* (16384-token completions, `stop=context_length`).
   The official post-trained sibling ships `eos_token_id:[248046,248044]` — so this is correct packaging,
   not a trick. Added to `/var/lib/agentptb-cache/a/models/base/generation_config.json`; mean completion
   tokens/turn dropped from ~16k to ~250. **Must be in every checkpoint I serve/submit.**
2. **Template is thinking-ON**: generation prompt ends `<|im_start|>assistant\n<think>\n`.
   Tool-call syntax is Qwen3-Coder XML (`<tool_call><function=name><parameter=x>`), matching
   `--tool-call-parser qwen3_coder`. Don't fight it — train short `<think>` blocks.
3. **Renderer must be pinned**: `Qwen/Qwen3.5-9B-Base` is NOT in `MODEL_RENDERER_MAP`, so `auto` falls
   back to the default renderer (no tool support). Always set `renderer.name = "qwen3.5"`.
4. `Qwen/Qwen3.5-35B-A3B` (post-trained MoE, 3B active, 72GB) has a **byte-identical chat template** to
   the base → ideal distillation teacher. Downloaded to `/var/lib/agentptb-cache/a/models/teacher35`.
   Used ONLY as a data generator (rule 7), never as weights (rule 3).
5. pi defaults `maxTokens=16384` per turn (`models.json` knob) — harness-side lever for my own harness.
6. Trace tool_calls are FLAT (`{id,name,arguments}`); prime-rl SFT needs OAI nested
   (`{id,type,function:{name,arguments}}`) or `deserialize_tool_calls` breaks the renderer.
7. Broker pulls arbitrary public Docker Hub images fine (tested ubuntu, swebench/*, alexgshaw/*).
   It does NOT pull Prime-platform refs (`prime/primeintellect/...`) → `tmax-v1`, `terminal-lego-v1`,
   `openthoughts-tblite-v1` are unusable. `swesmith-v1` images also fail (TaskError/SandboxError).
8. **`BrokerRuntime` has no `workdir` field.** `utils/compile.py` only applies `task.data.workdir`
   when the runtime config *has* that field, so with the broker every exec runs in `/workspace`
   (a scratch dir), never the task's `/testbed`. Two consequences:
   - the agent lands outside the repo and must find it (only 43/103 teacher rollouts ever
     mentioned `/testbed`, median 7 turns in) — a real, large capability tax on swe-bench;
   - **`r2e-gym-v1` scores 0 for everyone**: its `solved()` runs `sh -c "/bin/bash run_tests.sh"`
     with a *relative* path, which does not exist in `/workspace`. Same class of bug makes its
     `capture_patch` fail with `fatal: not a git repository`. This is why the 35B teacher scored
     0/52 on r2e-gym — not a capability result.
   `swebench-verified-v1` (harbor verifier, absolute paths) is unaffected and does score.
9. Round-trip verified: my converted rows render **byte-identical** to
   `tokenizer.apply_chat_template` under the `qwen3.5` renderer, loss mask = assistant spans
   (`<think>…</think>…<tool_call>…<|im_end|>`). 8/8 exact on a sample.

10. **No local container escape hatch.** `docker`/`podman` exist on this node but rootless podman
    cannot start a container: pulls work once `~/.config/containers/storage.conf` sets
    `ignore_chown_errors=true` and `USER` is corrected from `root` to `agentptb` (that env var
    being wrong is what produces the misleading "no subuid ranges for user root"), but every run
    then dies at `crun: mount 'proc' to 'proc': Operation not permitted` — the pod has
    `CapEff=0` and mount(2) is blocked. The broker really is the only way to run tasks. Don't
    retry this.
11. The broker exposes **`GET /resources`** (undocumented in the runbook): node capacity plus
    every sandbox pod and its phase. This is the way to tell "my job is slow" from "the cluster
    is full" — `{'Pending': 222, 'Running': 7}` means the queue is wedged, not that I am doing
    something wrong. Pod size does not help: 1 CPU / 4 GiB requests queue behind the same wall.

12. **Three traps between a finished SFT and a running RL job**, all of which look like
    unrelated failures:
    - `/app` holds a **different prime-rl build** and the default `PATH` puts `/app/.venv`
      first, so the launcher spawns its orchestrator/trainer from there. Its schema disagrees
      (`max_inflight_rollouts` vs `max_inflight_episodes`, `train.env` vs `train.source`) and
      **its verifiers has no broker runtime at all**. Symptom: `Extra inputs are not permitted
      --train.source`. Fix: `scripts/run_rl.sh` prepends `/root/work/a/prime-rl/.venv/bin`.
    - The trainer's saved checkpoint has **no `preprocessor_config.json`**, so prime-rl's
      inference server dies with `Can't load image processor for <ckpt>`. (Plain `vllm serve`
      does not care.) Fix: copy `preprocessor_config.json`, `video_preprocessor_config.json`,
      `merges.txt`, `vocab.json` from the base dir into every checkpoint — **do this before
      submitting too**.
    - prime-rl's inference does not pass `limit_mm_per_prompt`, so vLLM profiles the vision
      tower and crashes in the CUTE/FA4 kernel with `TypeError: fmax() missing 1 required
      positional argument`. Fix: `[inference.vllm_extra] limit_mm_per_prompt = {image=0,video=0}`.

13. **GRPO through the pi harness does not work in this stack.** After clearing the three traps
    above, every rollout dies with `ACP agent produced no visible reply`, and the env log shows
    the real cause: `model call failed: TrainClient does not support streaming`. pi always
    streams; the interception server routes streaming requests to the client, and prime-rl's
    `TrainClient` (the one that returns token ids + logprobs for the RL loss) raises on stream.
    The eval client proxies streams fine, which is why evaluation works and training does not.
    Fixing it means changing shared verifiers code, so RL was replaced with **expert iteration**
    through the same pi harness: sample → keep reward-1 trajectories → SFT. Same tool surface,
    same prompt, every component already validated.

14. **Both RL blockers are now cleared, in local plugin code, without touching the shared
    install.** Finding 13 was blocker one (streaming); blocker two was render parity.

    - *Streaming*: `pkgs/pi_rl` writes a ~90-line Node shim into the task container, points pi
      at `127.0.0.1:8899`, and the shim forwards each call upstream with `stream` stripped, then
      re-emits the JSON reply as the SSE chunks pi expects. Transport-only — same agent, same
      tools, same prompt. Verified transparent: 6/6 episodes, same turn count and token usage.
    - *Render parity*: the renderer serialises whatever tool objects it is handed, and verifiers
      hands it its own **flat** `ToolSpec`, so the rollout's tool block came out as
      `{"name":…,"description":…,"parameters":…}` while the serving chat template emits
      `{"type":"function","function":{…}}` — 68 chars different in the system prompt of *every*
      turn. `pi_rl.enable_render_parity()` normalises tools to the OAI-nested form before the
      renderer sees them, in the training process only.

    **Settled against the live server, not against a reconstruction.** POST the same
    messages+tools to vLLM's `/tokenize` with `add_generation_prompt=true` and compare token ids
    with `renderer.render_ids(...)`:

    | | tokens | identical to served prompt |
    |---|---|---|
    | unpatched renderer | 916 | no |
    | patched renderer | 952 | **yes** |
    | vLLM `/tokenize` | 952 | — |

    That also settles the one open question in the patch: pi sends `strict` in each tool schema
    and the served template keeps it, so `_nest_tools` must preserve it. Dropping it costs 36
    tokens and puts the policy off-distribution again. `enable_render_parity()` is called at
    `pi_rl` import; `pi_rl` is training-only and nothing in the evaluation path imports it.

    **Where the patch has to live, and why the obvious placements both fail silently.**

    - *Wrong process.* The renderer is built in prime-rl's **orchestrator**
      (`orchestrator/utils.py::setup_policy_inference_pool`); the harness — and therefore
      `pi_rl` — is imported in the **env-server**. Patching at `pi_rl` import time lands in a
      process that never renders. The patch now lives in `pkgs/render_parity.py` and is
      installed by `pkgs/sitecustomize.py` (which `PYTHONPATH` already reaches) under
      `VF_RENDER_PARITY=1`, set by `scripts/run_rl.sh` and nothing else. Evaluation shares that
      `PYTHONPATH`, never sets the variable, and is untouched.
    - *Wrong moment.* The cheap version — wrap `builtins.__import__`, check `sys.modules` after
      each call — does not work. Python inserts a module into `sys.modules` **before** running
      its body, so the first sighting of `renderers.qwen35` (an inner import from inside that
      very module) finds no `Qwen35Renderer` yet; fire-once logic then retires the hook having
      patched nothing. `install()` uses a `sys.meta_path` finder that wraps the real loader and
      patches inside `exec_module`, after the body has run.
    - `pkgs/sitecustomize.py` **shadows** `/usr/lib/python3.12/sitecustomize.py`, so it redoes
      that file's only job (installing Ubuntu's apport hook) before its own.

    **Verified live, in the running job**, not just on a bench:
    `[render-parity] Qwen35Renderer patched` appears in `orchestrator.log` at the
    `Initializing policy inference pool` line, and a real rollout's first call logs
    `prompt_tokens=2562` — which is exactly what the *patched* renderer produces for that
    conversation (unpatched: 2526).


15. **A stale `vllm::router` silently blocks the next RL launch.** Symptom is
    `Error: Inference failed with exit code -15` about 13 s after startup, which reads like an
    OOM or an external kill. The real cause is in `ckpt/<run>/logs/inference.log`:
    `PanicException: failed to install Prometheus metrics exporter:
    FailedToCreateHTTPListener("Address already in use")` — a router from a previous, killed run
    still holds ports 8000 and 29000. **`pkill -f "vllm serve"` does not match it**: the router
    renames its own process to `vllm::router`, so only `pkill -x "vllm::router"` finds it.
    `scripts/launch_rl.sh` clears it before every launch.

    Related, and the same trap as the old `pgrep -f` self-match: **use `pkill -x` (process name),
    never `pkill -f` (full command line), in any script written from a heredoc.** A `-f` pattern
    matches every shell whose command line quotes that string — including the shell writing the
    script — so the launcher kills itself and produces no output at all to explain why.

16. **An RL run cannot be found, or killed, by the names you launched it with.** Every process
    renames itself: the launcher becomes `PRIME-RL::Launcher`, then `PRIME-RL::Orchestrator`,
    `PRIME-RL::Trainer`, `PRIME-RL::EnvServer`, `vllm::router`. Consequences, all of which cost
    time here:
    - `ps | grep run_rl` / `grep orchestrator` shows **nothing**, so a live run looks dead.
    - The pidfile is useless: `setsid nohup bash … &` records the transient `setsid` PID, which
      exits at once. `kill $(cat rl.pid)` reports success and kills nothing.
    - `pkill -x "PRIME-RL::Launcher"` also fails — `-x` matches the kernel's `comm`, capped at
      **15 characters**, so the real name is `PRIME-RL::Launc`.

    A previous run therefore survives invisibly, and the next launch dies on its ports ~80 s in.
    `scripts/launch_rl.sh` sweeps by *comm prefix* through `ps -eo pid=,comm=` (matching
    `^(PRIME-RL|vllm|VLLM)::`), kills by PID, then re-checks and refuses to launch if anything
    survived. Kill the launcher first or it restarts its children. Watch for an orphaned
    `PRIME-RL::Train` holding ~40 GB — if it outlives its launcher it silently starves the next
    trainer of GPU memory.

18. **RL ran, and it is closed — for a fourth reason, which is structural rather than a bug.**
    With blockers 1 and 2 cleared and verified live, GRPO produced **0 reward across 99 scored
    episodes** (r2e-gym-ws 0/50, swelego-v1 0/33). Ruling things out, in order:

    - *Not the renderer.* Verified in the running job: a live rollout's first call logged
      `prompt_tokens=2562`, exactly the patched render (unpatched: 2526).
    - *Not tool parsing.* The train path parses completions with the renderer instead of vLLM's
      `qwen3_coder` parser, so this was a real risk — but 2926 of 2949 assistant messages carry
      `tool_calls`. Rollouts read like competent work ("The fix works", "issue resolved").
    - *Not the grader.* Every scored episode carries
      `info.patch_error: fatal: not a git repository`, which looks damning and is a red herring:
      `capture_patch`'s own docstring says a failed capture still lets the rollout score.
      **Gold-patch validation of `swelego-v1` on a free pool: 6/6 valid.** A correct patch does
      score. `runs/probe_shim` is *not* a baseline for these tasksets — it is swe-bench.
    - *Not my harness.* `pi_ws.run_acp_in` uses a per-command `cd`, so it cannot alter the cwd
      of the grader's later `runtime.run` calls.

    What it actually is — two config mismatches of mine, and one wall:

    | | eval (scores 0.208) | RL as configured |
    |---|---|---|
    | `max_turns` | **None** (unlimited; pi stops when done) | 40 — and **61% of episodes hit it** |
    | `system_prompt` | `cfg/agent_prompt.txt` | none set on either train source |

    The wall: pi's episodes on these tasks are long. Peak prompt tokens per episode are
    **p50 20k, p90 42k, max 101k**, so **24 of 113 episodes already exceed the trainer's
    `seq_len = 32768` at 40 turns**. Raising `max_turns` to match eval makes *more* of the batch
    untrainable, not less. Fixing it properly means `seq_len` 65536 on one B200 plus ~6 min of
    sandbox time per episode — at batch 96 and 24 inflight that is ≳40 min/step, so the clock
    buys ~50 steps with a third of each batch discarded.

    That is a real project, not a fix, so I stopped rather than start it with ~34 h left and a
    fully-measured checkpoint to protect. **`pkgs/pi_rl` + `pkgs/render_parity.py` are left
    working and documented; they are not part of the submission.** For anyone resuming: set
    `max_turns` unlimited, add the system prompt to both sources, raise `seq_len` to 65536, and
    expect the sandbox pool — not the GPUs — to be the bottleneck.




19. **Failure-mode audit of my own traces (4,496 episodes), and why it did not lead to another
    training run.** Three signatures, measured rather than guessed:

    | signature | prevalence (SFT model) | prevalence (base) | solve rate with / without |
    |---|---|---|---|
    | leaked tool-call XML in `content` (`</parameter></function></tool_call>`) | 12.9% | 0.8–2.0% | **5.7% vs 16.5%** |
    | "I apologize for the difficulty/confusion" | 41.7% | ~0% | 13.5% vs 16.3% |
    | SWE-agent harness artifacts ("Exit due to cost limit", "submit button") | 0.5% | 0% | — |

    The apology habit and the harness artifacts are **my SFT corpus talking** — the base model
    essentially never produces them. But the apology is style, not damage (13.5 vs 16.3 is weak
    and confounded: hard tasks cause both), and the harness artifacts are too rare to matter. I
    had expected the artifacts to be the story; they are not. Worth stating plainly because the
    first example I looked at was an "Exit due to cost limit", which is exactly how a 0.5% effect
    gets mistaken for the main one.

    The leaked-XML signature is real and 3× predictive, but it caps out small: eliminating it
    entirely moves 12.9% of episodes from 5.7% to 16.5%, i.e. **+1.4 points absolute** — a quarter
    of the ±5-point interval at n=250. Not worth an 8 h retrain plus an 8 h re-measurement.

    A separate cut, on the verification behaviour the earlier failure analysis flagged:

    | swe-bench, episodes that made an edit | ran a test after the **last** edit |
    |---|---|
    | solved (n=51) | **63%** |
    | failed (n=174) | **37%** |

    Same direction as `cfg/agent_prompt_v2.txt`, which was built for exactly this and measured at
    **0.233 vs 0.208, McNemar p=0.30** — directionally right, not separable. Note only 12% of
    failures never edited at all, so "nudge it to act" is the wrong lever; "make it verify" is the
    right one, and it is already in the prompt.

    **Conclusion, and the reason this section ends here:** every remaining lever I can identify is
    worth ~1–3 points, and my measurement floor is ±5 at n=250. Chasing them would produce changes
    I could not distinguish from noise, which is how a benchmark run talks itself into a
    regression. The honest use of the remaining time was to make the *reported* number better, so
    the last measurement runs the **complete 500-task suite** on both harnesses instead of a
    250-task sample, which is the one improvement that does not depend on my guessing right.

20. **Fairness audit of the two arms — one asymmetry found, and it did not bite.** The `pi-ws`
    configs set `ready_timeout_seconds = 1200`; the stock configs did not, and the broker default
    is **600** (`v1/runtimes/broker.py:59`). Half the readiness budget means roughly twice the
    sandbox-timeout rate under a busy pool, and the logs do show more error lines on the stock
    side (`big_swe_stock` 71 vs `big_swe_ws` 29) — an asymmetry pointing in *my* favour, which is
    the direction that must never go unchecked.

    Checked against the traces rather than the logs, because the two count different things: an
    episode that has no `rewards` record at all is one that never got graded, i.e. genuinely lost
    to infrastructure.

    | run | n | solved | ungraded | score | score excluding ungraded |
    |---|---|---|---|---|---|
    | `big_swe_stock` | 250 | 33 | **0** | 0.132 | 0.132 |
    | `big_swe_ws` | 250 | 52 | **0** | 0.208 | 0.208 |

    **Zero ungraded episodes in either arm** — the log error lines were transient and retried, no
    episode was lost, and the headline comparison is unaffected. `cfg/final500_stock.toml` now
    sets 1200 explicitly anyway, and the 500-task stock arm was restarted after the fix so that
    both arms are identical in everything except the harness and the appended prompt.

21. **The base-weights numbers were the weakest link in the attribution, so they are being
    re-measured too.** The submission claims two separate gains — weights and scaffold — but they
    rested on different-quality evidence:

    | | n | quality |
    |---|---|---|
    | `big_swe_base` (base + stock) | 153 | **176 errored episodes** in the same run |
    | `base_swe_ws` (base + pi-ws) | 64 | taken when pi-ws still had the **no-op `cd`** and the `maxTokens`/`contextWindow` caps that are now **off** — a different harness |
    | `big_swe_stock` / `big_swe_ws` (submitted) | 250 / 250 | clean, 0 ungraded |

    Taken at face value the old pair says base+pi-ws = 0.188 [0.11, 0.30] against submitted+pi-ws
    = 0.208 [0.16, 0.26] — heavily overlapping, i.e. *on the submitted harness I could not show
    the weights help at all*. That reading is not sound (different harness, n=64), but it is the
    right worry, and the honest fix is power, not argument.

    So the final measurement is the full **2×2 at n=500**: {base, submitted} × {stock, pi-ws},
    same 500 tasks, same protocol, base served on port 8001 so it can never be confused with the
    submitted checkpoint on 8000. `cfg/final500_{stock,ws,base_stock,base_ws}.toml` differ only
    in harness id, the appended prompt, and the port.

22. **Sandbox pod sizing, corrected — and what a saturated cluster looks like.** Each sandbox pod
    requests **4 CPU**, not 1 as finding 17 assumed. On a ~250–330 CPU shared cluster that means
    the *whole cluster* holds only ~60–80 concurrent sandboxes across all tenants, so 40
    concurrent from me was over half of it.

    On 2026-08-16 from ~04:40Z the pool went to **371 Pending / 34 Running with `available.cpu`
    at 0** — a ~1,500 CPU backlog against a cluster that has none — while only ~20 of those
    requests were mine. Throughput went to roughly zero regardless of what I did. Symptoms to
    recognise next time:

    - `avail_cpu` falling to 0 while `Pending` climbs into the hundreds — the queue is global, so
      dropping *my* concurrency does not clear it and does not restore *my* throughput.
    - A single hand-made sandbox going ready in 3 s is **not** evidence of headroom (finding 17);
      when the cluster is truly full even that stalls.
    - `/resources` has no owner field, so pods cannot be attributed. Judge by arithmetic: my
      outstanding requests versus total Pending.

    Practical rule: size runs to *finish*, because a partial run is not usable — with
    `shuffle=false` a prefix is the dataset's own ordering, and with `shuffle=true` it is
    ascending difficulty (Results section). Both bias the score. When the pool is saturated,
    prefer the measurement that fills a genuine gap over the one that merely adds precision.

23. **The 2×2 lands at n=250, and it does not just correct the attribution — it reverses part of
    it.** All four cells complete, **zero ungraded**, identical 250 tasks (`SEED=0` is pinned, so
    `shuffle=true` draws the same sample every run).

    | weights | stock pi | pi-ws |
    |---|---|---|
    | base (+eos fix) | 0.044 [0.025, 0.077] | **0.276** [0.224, 0.334] |
    | submitted (SFT) | 0.132 [0.096, 0.180] | 0.208 [0.162, 0.263] |

    | paired comparison (n=250) | only-A | only-B | delta | McNemar |
    |---|---|---|---|---|
    | base: stock → pi-ws | 4 | 62 | **+0.232** | **<0.0001** |
    | submitted: stock → pi-ws | 12 | 31 | **+0.076** | **0.0054** |
    | stock: submitted → base | 29 | 7 | **−0.088** | **0.0003** |
    | **pi-ws: submitted → base** | 21 | 38 | **+0.068** | **0.036** |

    - The **scaffold** is the robust result: +23 points on base weights, +7.6 on the submitted
      checkpoint. The larger intervention on both, by a wide margin.
    - The **SFT weights** are worth +8.8 points **under the stock harness** (p=0.0003) — real, and
      the reason they are still what I submit.
    - **Under pi-ws the SFT weights are significantly *worse* than base** (0.208 vs 0.276,
      p=0.036). Not "no difference" — a measured 6.8-point regression. The two interventions
      partly **conflict**.

    **How the number moved with n — the brief's warning, demonstrated.** Same comparison:
    n=100 → +0.000, p=1.00; n=225 → +0.062, p=0.076; n=250 → +0.068, p=0.036. The point estimate
    barely moved; the interval closed. At no n was there evidence the SFT weights *help* under
    pi-ws, which is exactly why the base arms were extended rather than stopped at the first clean
    number. Had I stopped at n=100 I would have reported "exactly tied" as a finding.

    Only visible because the 2×2 was finished rather than argued about. It retro-justifies finding
    21's worry — `base_swe_ws` 0.188 at n=64 was pointing straight at this and I explained it away
    as a stale-harness artifact.

    **It does not change what I submit.** The brief requires weights derived from the base by my
    own training, so the base itself is not eligible, and `sft_v5/step_900` is worth +8.8 under the
    stock harness. It changes what I claim, and it motivates finding 25.

24. **There is a hard cap of 32 concurrent host tunnels per API token, and `kill -9` leaks them.**
    This masqueraded as a sandbox-pool problem for over an hour and is invisible in `/resources`.

    Every eval episode opens a host tunnel so the sandbox can reach the local vLLM server.
    `PrimeTunnel.expose` (`v1/interception/tunnel/prime.py`) tears its tunnel down in a `finally`
    that is shielded against cancellation — so a normal exit or **SIGTERM** cleans up. **SIGKILL
    does not.** My own `topup_loop.sh` was doing `kill -9` on runs with 24 episodes in flight
    every 15 minutes; within a few passes all 32 slots were leaked and every subsequent episode
    died instantly with

        TunnelError: TunnelLimitReachedError: Maximum number of tunnels (32) reached

    at `turns=0`. Confirmed by listing: 32/32 held with **no eval process running at all**.

    Two rules follow, and both are now enforced in `scripts/topup_loop.sh`:
    - Stop eval runs with **SIGTERM**, never `kill -9`, unless you reclaim tunnels afterwards.
    - **Total `max_concurrent` across every simultaneously running eval process must stay under
      32** — it is per *token*, not per run. Two arms at 24 each is 48 and cannot work, however
      healthy the pool looks. Two arms at 14 each (28) is fine.

    `scripts/tunnels.py` lists the token's tunnels and `--delete` reclaims them; only run the
    delete when no eval is active, since it cannot tell a live tunnel from a leaked one.

    Worth noting how this presented: `stop=TunnelError` with `turns=0`, arriving in bursts, while
    the pool showed hundreds of free CPU. Any diagnosis that stops at "the pool is busy" misses
    it entirely.

25. **The scaffold's two halves, separated — and a pre-registered hypothesis that failed.**
    `pi-ws` is the working-directory fix plus the appended `cfg/agent_prompt.txt`, and the two had
    never been separated on the SFT weights. `cfg/ws_noprompt.toml` → `runs/ws_noprompt`, same
    seeded 250, zero ungraded:

    | submitted weights, harness | score | vs stock | vs full pi-ws |
    |---|---|---|---|
    | stock pi | 0.132 | — | |
    | pi-ws, workdir only | 0.164 | +0.032, p=0.26 | −0.044, p=0.099 |
    | pi-ws, workdir + prompt | **0.208** | **+0.076, p=0.005** | — |

    Both halves contribute, neither clears significance alone, together they do. The scaffold is
    the *pair*.

    **The prediction was wrong.** Registered in advance: since `public_to_sft.py` put pi's own
    system prompt on every training row, an appended operating-procedure block should be
    off-distribution for the SFT model but pure gain for the base — which would explain why the
    scaffold is worth +23 points on base and only +7.6 on the SFT weights. If so, dropping the
    prompt should have *recovered* most of the gap. Instead dropping it **costs** 4.4 points.
    Prompt off-distribution is not the explanation for the SFT weights' deficit against base under
    pi-ws, and I do not have a replacement explanation — only the measurement.

    **And a second demonstration of the partial-read trap**, worse than finding 23's: this same
    comparison read −0.075, p=0.096 at n=107 (looking like a strong refutation *in the opposite
    direction*), −0.039, p=0.31 at n=152, and −0.044, p=0.099 at n=250. Three different stories
    from one run. Only the completed number means anything.

26. **Pre-registered: is the training/scaffold conflict monotone in training length?**
    (Registered 2026-08-16 18:30Z, before the run.)

    Finding 23 leaves the submitted checkpoint in an awkward place: 900 steps of SFT buys +8.8
    points under the stock harness and **loses** 6.8 under pi-ws versus the untrained base.
    Hypothesis with a mechanism: the stock-harness gain is mostly **format** competence — emit a
    well-formed tool call, use the four tools, stop when done — which saturates within a couple of
    hundred steps; the pi-ws regression is **stylistic over-specialisation** that accumulates over
    the full run. If so, an early *fully annealed* checkpoint should keep most of the +8.8 while
    giving back most of the −6.8, and would dominate `step_900` on both harnesses.

    `cfg/sft_short.toml`: same corpus, same recipe, cosine annealed over **200 steps** (not step
    200 of the 900-step run — a complete run, the thing I would actually submit). 2 GPUs,
    batch 8 → ~52M tokens.

    **Decision rule, fixed in advance:**
    - Measure `sft_short` on **both** harnesses, n=250, same seeded sample
      (`cfg/short_{ws,stock}.toml`, served on port 8002).
    - The decisive cell is **pi-ws** vs `step_900`'s 0.208 on a paired McNemar.
    - *Scheduling note (changed after registering, and only the order):* I originally planned to
      run pi-ws first and gate the stock arm on it, to save sandbox budget. With ~24 h left and the
      pool delivering 30–60 episodes/hour, sequential gating risks finishing with only half the
      pair measured — the failure mode I have been guarding against all run. Both arms therefore
      run concurrently (14 each = 28 tunnels, under the cap). The rule below is unchanged.
    - Submit `sft_short` **only** if it wins under pi-ws *and* its stock number is not
      significantly worse than 0.132. Any other outcome: `step_900` ships, unchanged.
    - `step_900` stays fully measured and is never at risk. Nothing here can leave an unmeasured
      checkpoint as the submission.

    Expected value is honest about itself: the most likely single outcome is that the trade-off is
    monotone, no dominating checkpoint exists, and this returns a *curve* (0 / 200 / 900 steps)
    rather than a better submission. That curve is worth having either way.

    **Trained 18:22–21:01Z** (200 steps, 2 GPUs, ~47 s/step, loss 0.245 → ~0.14;
    `ckpt/sft_short/weights/step_200`, 760 tensors, eos ids correct, `step_100` kept as a spare).

    **RESULT — the hypothesis is refuted, and the curve is not the shape I guessed.** Both arms
    complete at n=250, zero ungraded, same seeded sample as every other cell:

    | weights | stock pi | pi-ws |
    |---|---|---|
    | base (0 steps) | 0.044 [0.025, 0.077] | **0.276** [0.224, 0.334] |
    | sft_short (200 steps) | 0.104 [0.072, 0.148] | 0.168 [0.127, 0.219] |
    | sft_v5 (900 steps, submitted) | 0.132 [0.096, 0.180] | **0.208** [0.162, 0.263] |

    | paired comparison (n=250) | only-A | only-B | delta | McNemar |
    |---|---|---|---|---|
    | stock: base → short | 7 | 22 | +0.060 | **0.0081** |
    | stock: base → sub900 | 7 | 29 | +0.088 | **0.0003** |
    | stock: short → sub900 | 12 | 19 | +0.028 | 0.28 |
    | pi-ws: base → short | 41 | 14 | **−0.108** | **0.0004** |
    | pi-ws: base → sub900 | 38 | 21 | **−0.068** | **0.036** |
    | pi-ws: short → sub900 | 18 | 28 | +0.040 | 0.18 |

    Two clean statements:
    - **Under the stock harness, SFT helps monotonically** with training length:
      0.044 → 0.104 → 0.132, each significant against base.
    - **Under pi-ws, *every* amount of SFT is significantly worse than the untrained base**
      (200 steps: p=0.0004; 900 steps: p=0.036). And it is **not monotone** — 200 steps (0.168) is
      *worse* than 900 (0.208), not better. My prediction was that a shorter anneal would sit
      closer to base under the scaffold. It sits further away.

    So the conflict is not "too much training". Whatever SFT does that the scaffold does not like,
    it does early, and more training partially undoes it. I do not have a mechanism for that, and
    with ~16 h left I am not going to get one honestly — three points on a curve is what this buys.

    **Decision rule applied: `sft_short` did not win under pi-ws (0.168 vs 0.208, p=0.18 favouring
    step_900), so `ckpt/sft_v5/weights/step_900` remains the submission, unchanged.** `sft_short`
    is kept on disk and reported; it is not submitted.

27. **Never run `eval @ config` and `eval --resume` against the same run directory at once.**
    They both own `traces.jsonl`, and the second one does not merge — the graded count went
    *backwards* (57 → 44) while both were live, because each process wrote the file from its own
    view of what was done. No corruption, but ~30 completed episodes were silently lost and the
    surviving file had mixed provenance.

    My own sequencing error: I launched both arms with `scripts/run_eval.sh` (which runs
    `eval @ cfg -o dir`) and *then* started `scripts/topup_loop.sh` on the same dirs, whose
    `stop_runs` only matches `eval --resume` — so it never stopped the original writers.

    Rules: one writer per run directory, ever. If you want the top-up loop, either start the run
    with it from the beginning, or SIGTERM the original process first and confirm it is gone.
    A `graded` count that decreases is the signature — check for two writers before anything else.
    When it happens, restart the affected runs from empty rather than resuming: a trace file of
    mixed provenance is not something to base a submission decision on.

28. **terminal-bench-2, the same 2×2, completed — and it measures nothing.** My own reporting had
    been inconsistent: I demanded completed runs for swe-bench while quoting tb2 at n=83–88 of 89.
    All four cells are complete 89-task runs with **zero ungraded**:

    | weights | stock pi | pi-ws |
    |---|---|---|
    | base (+eos fix) | 5/89 = 0.056 | 5/89 = 0.056 |
    | submitted (SFT) | 3/89 = 0.034 | 1/89 = 0.011 |

    One task nearly did not make it, and the cause is worth recording:
    `terminal-bench/qemu-alpine-ssh` boots a QEMU VM and takes **longer than 15 minutes**, which is
    exactly the interval at which `scripts/topup_loop.sh` stops and re-resumes its runs. Every pass
    started that task and killed it before it could finish, forever. The loop is right for
    pool-stalled runs and wrong for genuinely long tasks — when a single task is all that is left,
    stop the loop and run one plain `eval --resume` with nothing killing it. It then completed on
    the first attempt.

    Every paired McNemar is non-significant: p from 0.125 (base+pi-ws vs submitted+pi-ws, 4 vs 0)
    to 1.00. **No effect claimed in any direction.** With 1–5 solves out of 89 the interval swamps
    everything.

    Two observations, explicitly not results:
    - **The scaffold that is worth +23 points on swe-bench does nothing here** — base scores 5/88
      under *both* harnesses (5/89 each). That fits its content: a working-directory fix and a repo-oriented
      operating procedure have nothing to grip on in tasks that are not repo fixes and already
      start in `/app`. The pi-ws gain is specific to the SWE-repo setting, not general agentic
      competence.
    - The directional ordering matches swe-bench (base ≥ submitted everywhere, submitted+pi-ws
      lowest), but at these counts that is a coincidence I would not defend.

    Also fixed here: `cfg/base_tb2.toml` had the same missing `ready_timeout_seconds` asymmetry
    caught on swe-bench in finding 20 (broker default 600 against pi-ws's 1200, in my favour).
    Both tb2 configs now pin 3600. It had no effect on the numbers — zero ungraded either side —
    but it should not have been there.

29. **Replicate of the headline cell — the intervals hold, and the noise floor is now measured.**
    Every comparison in this file is a paired McNemar, which treats *which tasks were sampled* as
    the only source of variation and says nothing about run-to-run variance from sampling
    temperature and the environment. The brief warns that repeat reads of identical weights have
    differed by more than ten points, so I re-ran the headline cell: same checkpoint, same harness,
    same seeded 250 tasks, same config, nothing different but the run.

    | run | n | solved | score | ci95 |
    |---|---|---|---|---|
    | original (`big_swe_ws`) | 250 | 52 | 0.208 | [0.162, 0.263] |
    | replicate (`replicate_ws`) | 250 | 49 | **0.196** | [0.152, 0.250] |
    | pooled | 500 | 101 | **0.202** | [0.169, 0.239] |

    - **The replicate lands well inside the original interval.** The intervals quoted throughout
      are trustworthy as stated; this system does not swing ten points between reads.
    - **Noise floor, measured: two runs of the identical system disagree on 39 of 250 tasks
      (15.6%) — 21 one way, 18 the other.** That is the yardstick for every McNemar here. The null
      run is *symmetric*; the real effects are lopsided (12 vs 31 scaffold, 21 vs 38
      base-over-submitted). McNemar tests exactly that asymmetry, which is why it is the right
      test — but anything near 22 vs 30 is within reach of this floor.
    - **Both headline conclusions reproduce on the fresh run**: scaffold 14 vs 30, p=0.023 (was
      12 vs 31, p=0.005); base-beats-submitted under pi-ws 16 vs 36, p=0.008 (was 21 vs 38,
      p=0.036) — the second one is *stronger* on the replicate.

    This is the measurement I would have wanted first if I had known how much the report would end
    up resting on paired tests. It costs one run and it tells the reader what a difference has to
    look like before it means anything.

30. **Pre-registered, final experiment: weight-space interpolation between base and my SFT.**
    (Registered 2026-08-17 07:15Z, before building or measuring anything.)

    The standing problem from finding 23: my SFT weights are **6.8 points worse than the untrained
    base under my own scaffold** (0.208 vs 0.276, p=0.036; p=0.008 on the replicate), while being
    8.8 points *better* under the stock harness. That is the classic shape of fine-tuning trading
    away out-of-distribution robustness.

    Finding 26 already tested the obvious axis — *less training* (a 200-step anneal) — and it was
    **worse** under pi-ws, not better (0.168). So "closer to base along the training trajectory"
    does not help. **Weight-space interpolation is a different axis**: a convex combination of the
    base and my fine-tune, which is the standard remedy for exactly this fine-tuning/robustness
    trade (WiSE-FT). It is cheap — `scripts/soup.py` averages two checkpoints in ~20 minutes — and
    it has not been tried.

    `ckpt/soup_base_sft/` = **0.5 x base + 0.5 x sft_v5/step_900**, produced by `scripts/soup.py`.

    **Rules check, stated explicitly because it deserves scrutiny:** every parameter still traces
    to `Qwen/Qwen3.5-9B-Base` through training I ran — it is a blend of the designated base with my
    own SFT checkpoint and nothing else. The prohibition is on initializing from, continuing
    training on, merging in, or submitting *another party's* post-trained checkpoint; the base
    itself is the designated starting point, not another party's post-training. No third-party
    weights are involved at any point.

    **Decision rule, fixed in advance:**
    - Measure the soup on **both** harnesses, n=250, same seeded sample, both to completion.
    - Submit it **only** if it beats `step_900` under pi-ws on a paired McNemar at **p<0.05**
      *and* is not significantly worse than `step_900` under the stock harness.
    - **Hard stop 15:30Z.** If both cells are not complete by then, `step_900` ships. A partial run
      is not evidence and will not be used for this decision — three times this run a partial read
      told a different story than the finished one.
    - `step_900` remains fully measured throughout and is never at risk.

    Honest prior: maybe one chance in three. The 200-step result shows the pi-ws deficit is not a
    simple function of distance-from-base, so interpolation may not touch it either. Reported
    either way.

    **RESULT (both arms complete, n=250 each, zero ungraded).** Interpolation *works* on the axis
    that training length did not — and it buys the pi-ws points by giving back stock points:

    | weights | stock pi | pi-ws |
    |---|---|---|
    | base | 0.044 [0.025, 0.077] | 0.276 [0.224, 0.334] |
    | **soup (0.5/0.5)** | **0.084** [0.056, 0.125] | **0.280** [0.228, 0.339] |
    | submitted step_900 | 0.132 [0.096, 0.180] | 0.208 / 0.196 (two runs) |

    | paired vs step_900 (n=250) | only-A | only-B | delta | McNemar |
    |---|---|---|---|---|
    | pi-ws: step_900 → soup | 16 | 34 | **+0.072** | **0.0153** |
    | pi-ws: replicate → soup | 17 | 38 | **+0.084** | **0.0065** |
    | stock: step_900 → soup | 23 | 11 | **−0.048** | 0.0576 |

    - **Condition (a) is met clearly**: the soup beats `step_900` under pi-ws, and beats the
      *replicate* more strongly still, so this is not one lucky run.
    - **Condition (b) passes only on a technicality**: p=0.0576 is "not significant at 0.05", but a
      4.8-point drop (0.132 → 0.084, a 36% relative fall) on 23-vs-11 discordant pairs is almost
      certainly a real cost. Reading that as "no effect" would be exactly the rule-lawyering the
      noise-floor work in finding 29 was meant to prevent.

    So it is a genuine **trade**, not a dominating win: roughly +7 to +8 points on the harness I
    submit, roughly −5 on the stock control. And the mechanism is not subtle — the soup's pi-ws
    score (0.280) is statistically indistinguishable from the untrained base's (0.276), so its
    advantage there comes from being *closer to base*, i.e. from partially undoing my own training.

    **Cannot decide on swe-bench alone**: the submission publishes terminal-bench-2 as well, and
    the soup has no tb2 numbers. Both tb2 arms launched 11:30Z; decision at 15:30Z at the latest,
    on completed runs only.

31. **Replicate of the SUBMITTED system's headline cell (running from 2026-08-17 12:55Z).**
    The submission was switched to the soup on the strength of **one** measurement of
    soup+pi-ws (0.280), while the checkpoint it displaced has **two** (0.208, 0.196). Finding 29
    established that two runs of an identical system disagree on 15.6% of tasks. The submitted
    system's headline number should therefore carry the same replicated status as the number it
    replaced — otherwise I have held the alternative to a higher evidentiary standard than my own
    choice, which is precisely backwards.

    `cfg/soup_ws_rep.toml` → `runs/soup_ws_rep`: same checkpoint, same harness, same seeded 250,
    nothing different but the run.

    **Pre-committed reading.** The switch rested on soup 0.280 vs step_900 0.202 pooled, paired
    p=0.015 (p=0.0065 vs the replicate).
    - If this lands near 0.28 — or anywhere that keeps the paired comparison against `step_900`
      significant — the switch stands and the headline gets a pooled figure.
    - If it lands near 0.21, the single 0.280 was a lucky run, the switch is not supported, and
      **I revert the submission to `ckpt/sft_v5/weights/step_900`**, which is intact and fully
      measured.
    - **Hard stop 17:00Z.** If the run is not complete by then it is not used at all — a partial
      read has told a different story than the finished one three times in this run — and the
      submission stays as it is, with `SUBMISSION.md` stating the headline rests on a single
      measurement.

    **RESULT (complete, n=250, zero ungraded): 0.228, not 0.280. The first run flattered the soup.**

    | system | run 1 | run 2 | pooled (n=500) |
    |---|---|---|---|
    | soup + pi-ws | 0.280 | **0.228** | **0.254** [0.218, 0.294] |
    | step_900 + pi-ws | 0.208 | 0.196 | 0.202 [0.169, 0.239] |

    - **Run 2 alone does not beat `step_900`** (27 vs 22, p=0.57; against the step_900 replicate,
      32 vs 24, p=0.35). The p=0.015 that triggered the switch was the better of two draws.
    - **What survives is the pooled comparison across all four runs (1,000 episodes):** per-task
      sign test, soup better on 54 tasks, step_900 better on 30, 166 ties, **p=0.0116**. So the
      gain is real but is **+5 points, not +8**.
    - **The soup is the noisier system**: its two runs differ by 5.2 points (32 vs 19 discordant,
      p=0.09) against step_900's 1.2 (21 vs 18, p=0.75).
    - Pooled, the soup **no longer exceeds the base** under pi-ws (0.254 vs 0.276, one run) — it
      matches it, which is the same story as every other checkpoint on this harness.

    **Decision: the switch stands, but only just, and the headline is restated as 0.254.** The
    pre-registered revert trigger was "lands near 0.21, i.e. the 0.280 was a lucky run and the soup
    is not actually better". The pooled data says the soup *is* better than `step_900` under pi-ws
    (p=0.0116), so the substance of that trigger is not met — but 0.280 was clearly a high draw and
    quoting it would misrepresent the system. `SUBMISSION.md` now leads with the pooled 0.254, says
    the replicate alone fails significance, and keeps `step_900` named as the alternative for
    anyone weighting the stock-harness number higher. The trade is now roughly +5 pi-ws / −5 stock:
    close to even, with more evidence behind the gain (four runs) than the cost (two).

32. **Final measurement: the stock cost is confirmed, and the two checkpoints end in a dead heat.**
    The submission decision had become a trade — about +5 points under pi-ws against about −5 under
    the stock harness — with four runs behind the pi-ws side and only one per system behind the
    stock side. That asymmetry was the weakest link, so the last hours went to a replicate of the
    soup's stock cell (`cfg/soup_stock_rep.toml`, n=100, a strict prefix of the 250).

    **Complete, 100/100, zero ungraded: 0.070**, against the first run's 0.120 on those same 100
    tasks (10 vs 5 discordant, p=0.30 — consistent, not a contradiction). Pooled over all 350 stock
    episodes the soup scores **0.080 [0.056, 0.113]**, essentially the 0.084 first measured. The
    cost as weights alone is real, not a low draw.

    | | stock | pi-ws | sum |
    |---|---|---|---|
    | soup (submitted) | 0.080 | **0.254** | **0.334** |
    | step_900 | **0.132** | 0.202 | **0.334** |

    **Interpolating along the base↔SFT line trades the two published numbers off almost exactly
    one-for-one.** There is no free lunch on that axis. Which checkpoint is "better" is therefore a
    question about which number is primary, not something the data settles — and `SUBMISSION.md`
    says that plainly rather than implying the soup dominates.

    **Decision: no revert.** The pre-committed trigger was "pooled stock cost *clearly larger* than
    the pi-ws gain"; it is equal, not larger. The soup stays because the brief asks to make *the
    system* score higher and the pi-ws number is the system, and because that side rests on four
    runs and 1,000 episodes (p=0.0116) against three runs and 600 on the stock side (p=0.058).
    `step_900` is named in `SUBMISSION.md` for anyone weighting stock higher.

33. **Closing the last evidence asymmetry (running from 2026-08-17 16:45Z).** After finding 32 the
    submitted soup's stock cell had **350** episodes (250 + a 100-task replicate) while the
    alternative it displaced, `step_900`, had **250**. The extra scrutiny went to my own preferred
    checkpoint on the axis where it looks worst — the right direction — but a published comparison
    should not rest on unequal evidence in *either* direction.

    `cfg/s900_stock_rep.toml` → `runs/s900_stock_rep`: `step_900`'s stock cell, same seeded 100-task
    prefix, served on 8000 (GPUs 0/1; the soup stays up on 8002).

    **Decision-relevant, and pre-committed before looking:** the current trade is +5.2 points pi-ws
    against −5.2 stock, a dead heat on the sum (0.334 each).
    - If `step_900`'s stock number comes in **higher** than the 0.132 measured once, the stock cost
      becomes *clearly larger* than the pi-ws gain, which is exactly the revert trigger from
      finding 30 → **submit `step_900`**.
    - If it comes in at or below 0.132, the dead heat stands and the soup stays.
    - **Hard stop 18:00Z**; incomplete ⇒ not used, nothing changes.

    **RESULT — the trigger fires; submission reverts to `step_900`.** Complete, 100/100, zero
    ungraded: `step_900` scored **0.200** on the replicate prefix against **0.150** on those same
    100 tasks the first time (5 vs 10 discordant, p=0.30 — consistent). Pooled over 350 stock
    episodes `step_900` scores **0.151 [0.118, 0.193]**, not the single-run 0.132.

    | | stock (pooled n=350) | pi-ws (pooled n=500) | sum |
    |---|---|---|---|
    | **step_900 (submitted)** | **0.151** | 0.202 | **0.353** |
    | soup | 0.080 | **0.254** | 0.334 |

    The soup buys **+5.2** under pi-ws and costs **−7.1** under stock. The pre-committed trigger
    ("pooled stock cost clearly larger than the pi-ws gain") is met, so **the submission reverts to
    `ckpt/sft_v5/weights/step_900`**. `ckpt/soup_base_sft` stays in the workspace, fully measured,
    and is still ahead if the my-harness number is the only one that counts.

    **The lesson, and it is the same one twice:** both replicates moved the answer *against*
    whichever checkpoint I preferred at the time — 0.280 → 0.254 for the soup when I had just
    switched to it, 0.132 → 0.151 for `step_900` when I had just switched away. A single run
    flattered whichever system I had most recently chosen. Nothing about that is mysterious given
    finding 29's 15.6% task-level noise floor; what saved the conclusion was replicating *both*
    sides rather than only the one whose result I liked.

34. **Final audit: every published number recomputed from disk, and one magnitude corrected.**
    With the deadline close and both documents heavily edited, I recomputed every figure in
    `SUBMISSION.md` directly from `runs/*/traces.jsonl`. All matched exactly (step_900 53/350 and
    101/500; soup 28/350 and 127/500; base 11/250 and 69/250; sft_short 26/250 and 42/250; all six
    tb2 cells).

    One thing the audit did change. The scaffold gain on the submitted weights had been quoted as
    "+7.6 points", which was correct for the paired 250-task runs but is not the only estimate now
    that both arms have replicates:

    | comparison on `step_900` | stock | pi-ws | gain | test |
    |---|---|---|---|---|
    | paired 250-task runs (one each) | 0.132 | 0.208 | +7.6 | McNemar 12 vs 31, p=0.005 |
    | the same, replicate pair | 0.132 | 0.196 | +6.4 | 14 vs 30, p=0.023 |
    | 100 tasks where both arms have two runs | 0.175 | 0.270 | +9.5 | sign test 23 vs 11, p=0.058 |
    | pooled over all episodes (350 vs 500) | 0.151 | 0.202 | +5.1 | — |

    Every one is positive, so the *direction* is not in doubt; the honest **range is +5 to +9
    points**, and `SUBMISSION.md` now prints the whole table instead of the single best figure.
    (On base weights the effect is +23 and on the soup +19.6, both p<0.0001 — no such ambiguity.)

    Worth recording as a general point: once you replicate arms unevenly, "the" effect size stops
    being one number. Quoting the most favourable pooling would have been indefensible after
    spending the whole run insisting on completed runs and measured noise floors.

35. **The submission's own reproduction commands did not reproduce its numbers.** Caught in the
    last hour by actually reading the command block as a stranger would. `SUBMISSION.md` named
    `cfg/ws_swe.toml` and `cfg/base_swe.toml` as the swe-bench arms — those are `num_tasks = 100`,
    left over from the early 100-task reads — while every published swe-bench figure comes from the
    250-task runs (`cfg/big_swe_*.toml`) and their replicates. Anyone following the instructions
    would have got a different sample and quietly different numbers, and concluded the submission
    was overstated.

    Fixed: the block now names the 250-task configs, lists the replicate configs that the pooled
    figures add in, annotates each line with the count it should produce (52/250, 33/250, 1/89,
    3/89, 49/250, 20/100), and says explicitly that `ws_swe`/`base_swe` are the 100-task variants
    that do *not* reproduce the headline. Every config and script named in the file was then
    checked to exist.

    Worth generalising: a reproduction section is code, and it was the only part of this submission
    never executed as written. Read it against the numbers it claims to produce before shipping.

    **Then actually ran it.** `cfg/smoke_repro.toml` is `cfg/big_swe_ws.toml` with `num_tasks = 3`
    and nothing else changed, executed against the submitted checkpoint served on `:8000` exactly
    as the block documents. Result: **3/3 graded, zero ungraded, `errors: []`**, all three
    `agent_completed`, mean 21.7 turns / 39 s. So the whole chain works end to end —
    `PYTHONPATH` → `pkgs/pi_ws` resolves → harness launches → broker runtime → grading returns.

    Two details worth having on record, because they are the harness's central claims and this is
    the first time they were confirmed on the *submitted* artifact rather than during development:
    - **The agent really is in the task workdir.** All three episodes work in `/testbed` from their
      first command (`find /testbed -type f -name "*.py" ...`, and observations returning
      `/testbed/django/core/handlers/asgi.py`).
    - **The orientation training shows up in behaviour.** One episode's very first tool call is
      `pwd; ls -a; echo ---; ls -d /testbed /app /workspace /repo /code /srv 2>/dev/null` — the
      exact pattern `scripts/add_orientation.py` put on 40% of training rows.

    0/3 solved is the expected outcome at a ~20% rate (0.8³ ≈ 51% chance of zero) and is not a
    signal either way; the point of the run was that it *executes*, not what it scores.

## Measurement protocol (fixed — do not change mid-run)

The exact McNemar in `scripts/final_2x2.py` was checked against `scipy.stats.binomtest` on every
discordant pair reported in this file (12/31, 5/18, 11/2, 13/13, 2/24, 0/17, 3/17): identical to
1e-9. Worth doing once — every significance claim here rests on that one function.



`cfg/base_tb2.toml` (all 89 tasks) and `cfg/base_swe.toml` (100 tasks, `shuffle=true` fixed seed),
1 rollout/task, stock `pi` harness, broker runtime `block_network=false`, **no sampling overrides**.
Score with `scripts/score.py <outdir>` (solve rate + Wilson 95% CI).

## Results

**Submitted checkpoint: `ckpt/sft_v5/weights/step_900`.**

### Headline — 250-task swe-bench-verified sample (the number to quote)

| ckpt | harness | n | solved | score | ci95 |
|---|---|---|---|---|---|
| base (+eos fix) | stock pi | 153 | 7 | 0.046 | [0.022, 0.091] |
| SFT v5 step_900 | stock pi | 250 | 33 | **0.132** | [0.096, 0.180] |
| SFT v5 step_900 | **pi-ws** | 250 | 52 | **0.208** | [0.162, 0.263] |

Paired tests on the shared tasks (the powered comparison — the subset is seeded, so runs line
up task-for-task):

| comparison | only-A | only-B | McNemar |
|---|---|---|---|
| base → SFT, stock harness | 3 | 17 | **p=0.003** |
| base → SFT, pi-ws | 3 | 23 | **p<0.001** |
| stock → pi-ws, same weights | 12 | 31 | **p=0.005** |

So on swe-bench-verified: the weights are worth **+8.6 points** (0.046 → 0.132) and the scaffold
a further **+7.6** (0.132 → 0.208), each independently significant.

### Fixed 100-task protocol (used for every checkpoint comparison in this run)

| ckpt | suite | harness | n | solved | score | ci95 |
|---|---|---|---|---|---|---|
| base (+eos fix) | swe-bench-verified | stock pi | 97 | 4 | 0.041 | [0.016, 0.101] |
| SFT v5 step_900 | swe-bench-verified | stock pi | 88 | 11 | 0.125 | [0.071, 0.210] |
| SFT v5 step_900 | swe-bench-verified | pi-ws | 98 | 31 | 0.316 | [0.233, 0.414] |
| base (+eos fix) | terminal-bench-2 | stock pi | 83 | 2 | 0.024 | [0.007, 0.084] |
| SFT v5 step_900 | terminal-bench-2 | stock pi | 88 | 3 | 0.034 | [0.012, 0.096] |
| SFT v5 step_900 | terminal-bench-2 | pi-ws | 83 | 2 | 0.024 | [0.007, 0.084] |

* **Weights**: swe-bench 0.041 → 0.125 on the stock harness. Paired, the SFT model solves 9 the
  base does not and loses 1 (McNemar **p=0.021**). The stock number reproduces almost exactly on
  the independent 250-task sample (0.125 → 0.123), so it is solid.
* **Harness**: +7.6 points on the complete 250-task sample, on identical weights.
* **terminal-bench-2 does not move** under any combination — 2–3 solves out of ~85 throughout.
  It is out of reach for a 9B here, and no harness variant separates from another (Fisher p=0.62
  for the widest gap).

⚠ **The 100-task pi-ws figure (0.316) is the optimistic tail of the run-to-run spread.** The same
harness and weights measured 0.208 on 250 tasks, and paired on their 95 shared tasks the two runs
agree (McNemar p=0.48) — the gap is which tasks landed in the first 100, not a real difference.
This is exactly the failure the brief warns about; **0.208 is the number I stand behind.**

### Candidates that did not pan out (all measured, all kept in `runs/`)

| tried | result |
|---|---|
| 900 more SFT steps on the unseen 65% of the corpus (`sft_cont2`) | 7/64 swe stock, 14/58 pi-ws — no gain; supervised signal saturated |
| checkpoint averaging of the three (`soup1`) | 7/63, 13/58 — no gain |
| `temperature 0.2` on pi-ws | 21/81 = .259 vs .253, but ~2× episode length — rejected |
| pi-ws per-turn caps (`maxTokens 4096`, `contextWindow 60000`) | **actively harmful now**: 23/91 vs 31/98 uncapped (paired 5 vs 11) — removed |
| Dockerfile-derived workdir for terminal-bench | 0/79 and 1/86 vs 3/88 stock — left off by default |
| a sharper prompt that insists on verifying after the last edit (`cfg/agent_prompt_v2.txt`) | 53/227 = .233 vs .208 for the shipped prompt, paired 27 vs 19, **McNemar p=0.30** — not separable, so the shipped prompt (the one every headline number was measured with) stays |
| GRPO through pi | blocked by **two independent** things, both now diagnosed exactly. (1) pi hard-codes `stream: true` and prime-rl's `TrainClient` — the only client returning the token ids/logprobs the RL loss needs — raises on streaming. **Solved**: `pkgs/pi_rl` starts a ~90-line Node shim in the task container, points pi's `baseUrl` at it, and it forwards non-streaming then re-emits SSE. Verified transparent under the eval client (6/6 episodes, same turn count and token usage as un-shimmed). (2) **The RL rollout path renders a different system prompt than the serving path.** The renderer serialises whatever tool objects it is handed; verifiers hands it its own flat `ToolSpec`, so the tool block comes out as `{"name":…,"description":…,"parameters":…}`, whereas the served chat template emits `{"type":"function","function":{…}}`. Verified byte-for-byte on a real pi conversation: `renderer(flat ToolSpec) == served template` → **False**; `renderer(OAI-nested) == served template` → **True** (68 chars of difference in the system prompt, on every single turn). So the policy is off-distribution for the entire rollout: **0/85 solved** on tasks the identical harness solves ~16% of the time under the eval client, and after temperature 1.0 → 0.7, 57 of 76 episodes ran to the 40-turn cap instead of finishing. The fix is a one-line change in how verifiers hands tools to the renderer — shared-install code, so out of scope here. |
| GRPO through the verifiers-native `bash` harness (which does *not* stream, so the stack runs) | the harness runs and batches fill, but **every reward is 0**: the policy scores 0/24 on r2e-gym-ws under `bash` even with pi's own system prompt injected, against ~16% under pi. An SFT'd agent is tightly coupled to its harness's tool schema and loop, so there is no signal to learn from. |
| expert iteration on r2e-gym-ws | 203 successes but only **32 distinct tasks** — too narrow to train on |

### What the failures actually look like (250-task stock run)

Among **solved** episodes the model runs a check after its last edit 36% of the time; among
**failed** ones, 17% (any check at all: 58% vs 45%). Verifying is the behaviour most associated
with succeeding and the policy does it unreliably — but **72% of the training corpus already
verifies**, so this is a generalisation gap, not a coverage gap, which is consistent with more
SFT buying nothing. It is the kind of gap RL closes, and RL is the thing this stack would not
run. 35 of 217 failed episodes never made an edit at all; ~10% of tool calls in failed episodes
name a tool that does not exist (`submit`, `submission`, `output`).

| ckpt | suite | harness | n | solved | score | ci95 |
|---|---|---|---|---|---|---|
| base (+eos fix) | swe-bench-verified | stock pi | 97 | 4 | 0.041 | [0.016, 0.101] |
| base (+eos fix) | swe-bench-verified | pi-ws | 64 | 12 | 0.188 | [0.111, 0.300] |
| base (+eos fix) | terminal-bench-2 | stock pi | 83 | 2 | 0.024 | [0.007, 0.084] |
| **SFT v5 step_900** | swe-bench-verified | stock pi | 88 | 11 | **0.125** | [0.071, 0.210] |
| **SFT v5 step_900** | swe-bench-verified | pi-ws | 91 | 23 | **0.253** | [0.175, 0.351] |
| **SFT v5 step_900** | terminal-bench-2 | stock pi | 88 | 3 | **0.034** | [0.012, 0.096] |
| **SFT v5 step_900** | terminal-bench-2 | pi-ws | 85 | 1 | 0.012 | [0.002, 0.064] |

Paired (McNemar, same seeded task subset):
* swe stock, base → SFT: solved 9 tasks base did not, lost 1. **p=0.021** — the weights alone
  roughly triple the stock number (0.041 → 0.125).
* swe pi-ws, base → SFT: +7 / −3, p=0.34 — not separable at this n; the scaffold had already
  captured much of what SFT teaches (find the repo, use absolute paths).
* terminal-bench-2 moves ~nothing either way. It is simply out of reach for a 9B here: the
  base solves 2/83, the SFT 3/88. pi-ws is 1/85 — no evidence it helps on this suite, and the
  starting-in-`/app` change removes the orientation step the model was trained to perform.

⚠ Sandbox losses matter for these numbers: episodes that die as `SandboxError` drop out of `n`.
Use `eval --resume <run_dir>` (`scripts/resume_eval.sh`, or `scripts/topup.sh` which retries in
free windows) to re-run them — that is how the swe runs got from n≈74 to n≈90. Resume needs the
vLLM server **still up**; `measure.sh` kills it at the end, so restart it first or every resumed
rollout dies with `ProviderError`.

⚠ **Never read a partial run as a score.** Episodes finish in roughly ascending order of
difficulty, so a run at 30% completion reads far too high — `big_swe_stock` showed 8/33 = 0.24
early against its true ≈0.17, and `sft5_swe_ws_t02` showed 3/8 = 0.38 against ≈0.26. Only
compare completed runs.

**The scaffold alone is worth ~15 points on swe-bench-verified** (Fisher p=0.005; paired on the
63 shared tasks, pi-ws solved 11 that stock did not and stock solved 0 that pi-ws did not,
McNemar p=0.001), on frozen base weights.

⚠ **Attribution correction.** When those numbers were taken, pi-ws's `cd` was a *no-op*, so the
gain came from the appended `cfg/agent_prompt.txt` (which names `/testbed`) plus the
`maxTokens`/`contextWindow` caps — not from the working directory. The reason: pi resolves every
tool call against the ACP **session** cwd, which `verifiers/v1/acp/_runner.py` sets from
`os.getcwd()` of the *runner* process; `cd`-ing the agent command underneath it changes nothing.
`pi_ws.run_acp_in` now starts the runner itself inside the workdir (re-anchoring the config path
and exporting `VF_LAUNCH_BASE` so `PI_ACP_PI_COMMAND` / `PI_CODING_AGENT_DIR` / skills still
resolve against the launch dir). Verified live: `pwd` returns `/testbed` on swe-bench and `/app`
on terminal-bench, with the task's files right there. The measured 0.188 therefore *understates*
the current harness.

**Gold-patch validation of the RL taskset** (`validate --only-gold`, 6 tasks, broker runtime):

| taskset | valid |
|---|---|
| `r2e-gym-v1` (stock) | **0/6** — `gold apply failed: error: Orange/data/util.py: No such file or directory` |
| `r2e-gym-ws` (my wrapper) | **6/6** valid, ~80 s each |
| `swelego-v1` (stock) | **6/6** valid, ~25 s each — works as shipped, and grades ~3× faster |

The stock r2e taskset cannot score a *correct* patch under this runtime; the wrapper is what
makes RL on it possible at all. RL trains on both sources.

## ⚠ HARD-LEARNED RULE

**`/var/lib/agentptb-cache/a/` DOES NOT SURVIVE A NODE RESTART.** At ~04:46 on 2026-08-14 the
node was recreated and everything there except `prime-rl/` and `tmp/` was erased: the local base
model copy, the 72 GB teacher, the raw trajectory corpora, and **all SFT v1 checkpoints** (the run
had reached step 600/940). ~9 h of GPU work lost. The workspace on the PVC was untouched.

→ Model checkpoints go to `$AGENTPTB_WORKSPACE/ckpt/...`. Local disk is for the read-only model
copy (fast weight loading) and TMPDIR only — anything reproducible in <15 min.

## Where things are (update as you go)

| artifact | path |
|---|---|
| base + generation_config fix | `/var/lib/agentptb-cache/a/models/base` |
| teacher (data-gen only, never weights) | `/var/lib/agentptb-cache/a/models/teacher35` |
| SFT corpus v1 (15,112 traj, 244M tok) | `data/sft_v1` |
| SFT corpus v2 (30,583 traj, real trajectory endings) | `data/sft_v2` |
| SFT corpus v3 (46,286 traj, ~750M tok) | `data/sft_v3` |
| **SFT corpus v4 (v3 + 30% path-relocated) — the one in use** | `data/sft_v4` |
| SFT v2 run + checkpoints | `ckpt/sft_v2/weights/step_N` |
| local plugins (PYTHONPATH) | `pkgs/{r2e_gym_ws,pi_ws}` |
| measure driver | `scripts/measure.sh` — args: `<model_dir> <tag>`, then suite (both/swe/tb2), arm (stock/ws/both), concurrency |
| **status at a glance (run this first on resume)** | `bash scripts/status.sh` |
| scoring | `scripts/score.py <run_dir>` |
| raw-trajectory corpora | `/var/lib/agentptb-cache/a/data/{klear66k,swesmith_traj,r2e_sft,swegym_oh,nebius,deepswe_k2}` |

Checkpoints are HF-servable as saved (config + chat_template.jinja + tokenizer + the correct
`generation_config.json` with `eos_token_id:[248044,248046]`) — serve the `weights/step_N` dir directly.

- [h20] Orientation training is taking. At an equal 180 steps, under the **stock** harness:

  | corpus | mentions `/testbed` | emits an early `pwd`/`ls -d` probe | solved |
  |---|---|---|---|
  | v4 (no orientation turns) | 3/8 | 0/8 | 0/8 |
  | v5 (40% orientation turns) | 8/16 | 5/16 | 1/15 |

  Still only 20% through training. `scripts/compare.py A B` does Wilson + Fisher for run pairs;
  the shuffled 100-task swe subset is seeded, so runs are paired on identical tasks.

## Status: COMPLETE (2026-08-17 17:40Z). Submission = `ckpt/sft_v5/weights/step_900`.

**Submission: `ckpt/sft_v5/weights/step_900` + `pkgs/pi_ws`**, evaluated with
`cfg/big_swe_ws.toml` / `cfg/ws_tb2.toml` (mine) against `cfg/big_swe_stock.toml` /
`cfg/base_tb2.toml` (stock).
Verified: 15 files, 4 shards, 760 tensors, 18.8 GB, `eos_token_id=[248044,248046]`, cold-serve
tested and answering correctly. Every published figure was recomputed from `runs/*/traces.jsonl`
(finding 34) and every config/script named in the reproduction block was checked to exist and to
produce the number claimed for it (finding 35).

**Machine state:** no eval, training or RL jobs are running and no tunnels are held. One vLLM
server is deliberately left up — the **submitted** checkpoint on `localhost:8000` (GPUs 0,1), so a
verifier can hit it immediately; GPUs 2,3 are free. 19 measurement runs are on disk and **every one
of them is complete with zero ungraded episodes**.

**Final numbers — every swe-bench cell replicated:**

| | stock pi | pi-ws (submitted system) |
|---|---|---|
| swe-bench-verified | **0.151** [0.118, 0.193] (n=350) | **0.202** [0.169, 0.239] (n=500) |
| terminal-bench-2 | 3/89 = 0.034 | 1/89 = 0.011 |

**The interpolated checkpoint `ckpt/soup_base_sft` was submitted for ~6 h and withdrawn on its own
pre-registered rule** (findings 30–33): pooled, it buys +5.2 points under pi-ws and costs −7.1 under
the stock harness, so `step_900` leads on the sum, 0.353 to 0.334. It stays in the workspace fully
measured, and is still ahead on the my-harness number alone.

**What the run established (all replicated or at n=250+):**
1. **The scaffold is the result** — +23 points on base weights (p<0.0001), +7.6 on the submitted
   weights (p=0.005, reproduced p=0.023), +19.6 on the soup. Both halves of it contribute; neither
   is significant alone (finding 25).
2. **SFT helps monotonically under the stock harness** (0.044 → 0.104 → 0.132/0.151) and **hurts
   under pi-ws at every training length**, non-monotonically (findings 23, 26).
3. **Weight-space interpolation trades the two published numbers roughly one-for-one** — no free
   lunch on that axis (findings 30–33).
4. **terminal-bench-2 measures nothing** (all p≥0.125); the scaffold is worth zero there, so its
   gain is specific to the SWE-repo setting (finding 28).
5. **Noise floor: 15.6% of tasks flip between two runs of an identical system**, symmetric
   (finding 29). Every headline cell is replicated because of it — and both replicates moved the
   answer *against* whichever checkpoint I preferred at the time.

Six pre-registered hypotheses: four refuted (prompt off-distribution 25, shorter anneal 26, RL
through the real harness 18, tb2 corroboration 28), one confirmed then withdrawn on its own rule
(interpolation 30–33), plus the replicate protocol (29) which changed how everything is quoted.

**The uncomfortable summary, kept at the top of `SUBMISSION.md`:** on the harness I submit, my
training is a net liability; the scaffold, not the training, is what carries this result. I could
only see that by finishing the 2×2 instead of arguing about it.

## Next actions (if resumed)

The submission is `ckpt/sft_v5/weights/step_900` + `pkgs/pi_ws`, complete, measured on both suites,
with every swe-bench cell replicated and every published number recomputed from disk (finding 34).
Nothing is unfinished. If someone picks this up, the things actually worth doing, in order:

1. **Explain the SFT/scaffold conflict.** This is the central unexplained result and everything
   else in the run circles it: SFT helps monotonically under the *stock* harness
   (0.044 → 0.104 → 0.151) and **hurts under pi-ws at every training length**, non-monotonically —
   200 steps (0.168) is worse than 900 (0.202), and both are below the untrained base (0.276)
   (findings 23, 26). Three mechanisms were tested and refuted: prompt off-distribution (25),
   shorter anneal (26), and weight-space interpolation, which trades the two published numbers
   roughly one-for-one rather than fixing anything (30–33).
   The place to start is a *behavioural* diff, not another sweep: the traces are all in `runs/`,
   and `base250_ws` solves 38 tasks that `big_swe_ws` does not. Compare turn counts, tool mix, and
   where the two diverge on exactly those tasks.

2. **Sweep the interpolation coefficient, if the my-harness number is what matters.** Only α=0.5
   was measured, chosen a priori to avoid selection noise. It lands at pi-ws 0.254 / stock 0.080
   against step_900's 0.202 / 0.151 — i.e. it moves along a roughly one-for-one trade line
   (finding 32). α ∈ {0.25, 0.75} would show whether that line is really straight, but budget for
   the noise floor: ±5 points needs 250 tasks *and* a replicate, or you will measure nothing
   (findings 29, 31, 33).

3. **RL**, if there are days rather than hours. Both known blockers are cleared and verified in
   `pkgs/pi_rl` + `pkgs/render_parity.py` (findings 14, 18), but episode prompts reach p90 42k /
   max 101k tokens against `seq_len=32768`, so a third of every batch is untrainable. Needs 65536,
   `max_turns` unlimited to match eval, the system prompt on the train sources, and a sandbox pool
   that can sustain ~100 episodes/hour.

**Do not bother with:** more harness knobs (finding 19 — everything left is 1–3 points against a
15.6% task-level noise floor); more of the same SFT (saturated, finding 26); or reading any partial
run as a result. That last one misled me four separate times (findings 23, 25, 31, 33), twice in
the direction of the checkpoint I happened to prefer at the moment.

**If you change the submission, replicate both sides before you do.** Every time I measured one arm
and switched, the next replicate moved the answer back: 0.280 → 0.254 for the interpolation right
after I adopted it, 0.132 → 0.151 for step_900 right after I abandoned it.