RECIPE: document per-checkpoint eval worker

RECIPE.md

@@ -144,13 +144,40 @@ the job's own public URL → start the 30-min watchdog → train (GPU 0) → fin
 + the matching API key also works if you prefer those backends; then the Jobs-proxy
 plumbing is unnecessary.)
 
-### Evaluation (built into the job)
+### Evaluation (two layers)
 
-After training, the job restarts vLLM with `--enable-lora --lora-modules trained=$ADAPTER_REPO`
-and runs all 250 test tasks twice — base model vs trained adapter — through the same
-harness/sandbox path (`purpose="eval"`, required when passing `eval_sampling`).
-Pass@1 for each lands in the adapter repo at `eval/final_results.json`.
-A standalone version (`run_eval.sh`) runs the same comparison on any checkpoint.
+**Per-checkpoint, live** — a separate eval-worker job (`run_eval_watch.sh`, 1×H200)
+polls the adapter repo; every time a new `checkpoint-N` appears it hot-loads it as a
+LoRA into its running vLLM (runtime LoRA updates, no restart) and evals it on a fixed
+100-task test subset. Base-model cells are computed once and reused. Results:
+`eval/<checkpoint-N>_results.json` + a rolling **`eval/summary.json` — the live
+learning curve**. Launch it alongside training:
+
+```bash
+uv run --with 'huggingface_hub>=1.29' python - <<'EOF'
+import os
+from huggingface_hub import run_job
+token = os.popen("hf auth token").read().strip()
+run_job(
+    image="vllm/vllm-openai:v0.31.0",
+    command=["bash", "-c",
+             "pip -q install huggingface_hub && "
+             "hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && "
+             "bash /app/run_eval_watch.sh"],
+    flavor="h200", timeout="48h",
+    env={"ADAPTER_REPO": "YOU/LFM2.5-2.6B-opencode-lora-RL", "N_TASKS": "100"},
+    secrets={"HF_TOKEN": token},
+    expose_public=[8000],
+    token=token,
+)
+EOF
+```
+
+**Final, full** — after training, the training job itself restarts vLLM with the
+final adapter and runs all 250 test tasks twice (base vs final) →
+`eval/final_results.json`. (`purpose="eval"` is required when passing
+`eval_sampling`.) A standalone version (`run_eval.sh`) runs the same comparison on
+any checkpoint.
 
 ---
 
@@ -199,6 +226,7 @@ A standalone version (`run_eval.sh`) runs the same comparison on any checkpoint.
 | `train_lora.py` | trainer + worker wiring, reward, LoRA config, upload & heartbeat callbacks |
 | `run_job.sh` | job entrypoint: setup, seam patch, 3 processes, watchdog, final eval |
 | `run_eval.sh` | standalone base-vs-adapter eval job |
+| `run_eval_watch.sh` + `eval_watch.py` | per-checkpoint eval worker (live learning curve) |
 | `eval_adapter.py` | held-out eval loop (pass@1, agreement breakdown) |
 | `prepare_tasks.py` + `data/` | pinned, hash-checked task materialization (+ `docker_image` patch) |
 | `serve_policy.sh` / `serve_env.sh` | the two servers, for multi-machine setups |