TASK.md: handoff for the next agent (state, artifacts, outstanding work, sharp edges)

TASK.md

@@ -0,0 +1,117 @@
+# TASK: OpenCode-only RL on LFM2.5-2.6B — state and next steps
+
+*Written 2026-10-28 for an agent taking over from here.*
+
+## TL;DR
+
+We reproduced the OpenCode-only RL run from
+[The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl)
+on `LiquidAI/LFM2.5-2.6B` with a LoRA adapter, end to end, entirely on HF
+infrastructure. **The pipeline is complete and battle-tested; the learning outcome
+was flat**: held-out pass@1 never separated from the base model (~12% ± 3% at n=100
+through all 10 checkpoints), and held-out tool-call efficiency did not improve.
+The immediate open item is a clean **250-task final eval** (running now — see
+"Outstanding work" #1).
+
+## Where everything lives
+
+| Artifact | Location |
+|---|---|
+| Code bundle (this project) | https://huggingface.co/SaylorTwift/lfm-opencode-lora-rl-code |
+| Recipe (architecture, versions, pitfalls) | `RECIPE.md` in that repo — **read it first** |
+| Base model | https://huggingface.co/LiquidAI/LFM2.5-2.6B |
+| Adapter repo (final + checkpoint-100…1000 + all eval JSONs) | https://huggingface.co/SaylorTwift/LFM2.5-2.6B-opencode-lora-RL (private) |
+| Learning curve | `eval/summary.json` in that repo |
+| Task data | `FineEnvs/SmolDataEnvs-harbor-train` / `-test` (pinned revisions, hash-checked by `prepare_tasks.py`) |
+| Article | https://huggingface.co/spaces/FineEnvs/multi-harness-rl |
+| Article's tutorial / env server | GitHub `adithya-s-k/FineEnvs` `05-multi-harness-rl/`; Space `FineEnvs/smoldataenv-multi-harness-harbor` |
+
+## What was run and what it produced
+
+- **Smoke run** (20 steps) and **four full/partial training runs** across the
+  debugging campaign; the last complete one (`6ae06e6f…`, h200x2, resumed from
+  checkpoint-500) trained to step 1000 and pushed checkpoints 100–1000 + final.
+- **Eval worker** evaluated every checkpoint on 100 held-out tasks (base evaluated
+  twice — both runs agree within 2%).
+- **Curve (pass@1, n=100, base = 12%):**
+  100:10, 200:13, 300:11, 400:12, 500:11, 600:8, 700:**15**, 800:10, 900:11, 1000:11.
+- **Tool calls on solved tasks vs base:** checkpoint-800 ×1.10, 900 ×1.04, 1000 ×1.02 —
+  no held-out efficiency gain (training-pool tool calls dropped ~21% between two
+  windows, but held-out cells say that was task-mix confounding).
+
+Config: LoRA r=32/α=64 on q/k/v/out_proj+w1/w2/w3, async GRPO (group of 8, OpenCode
+only), lr 1e-6, 1000 steps, temp 1.0/top_p 1.0/top_k 0, reward = correctness + ≤0.1
+tool-efficiency bonus, output cap 4096, agent budget 32 calls. Full details in RECIPE.md.
+
+## What is already proven (do not re-derive)
+
+- HF Sandboxes (`harbor[hf-sandbox]`, hf-sandbox backend, needs `docker_image` in
+  task.toml — `prepare_tasks.py` patches it) run OpenCode unmodified.
+- Capture proxy records exact tokens+logprobs; importance ratio 0.997–1.01.
+- Merged-weight LoRA sync over NCCL: 136 ms/step at 2.6B.
+- Jobs-proxy exposure (`expose_public=[8000]` + `SPACE_HOST=$JOB_ID--8000.hf.jobs`)
+  is stable for multi-day runs; the Gradio tunnel is not (it killed one run) and the
+  Jobs proxy **strips Authorization**, hence the x-api-key seam patch in the entrypoints.
+- Watchdog (30-min stall → loud fail) + per-100-step checkpoint pushes (transformers v5
+  has no PushToHubCallback) + resume-from-repo all work.
+- trl must be **1.15.0** (1.16's tree packing rejects hybrid models) on **vllm 0.31.0**,
+  with `kernels==0.16.0`.
+
+## Outstanding work, in priority order
+
+1. **Finish the clean 250-task final eval.** Job `6ae2026e3fb22a235dd409c2` is running
+   (`run_eval.sh`, now fixed and hardened: test-only split + Jobs-proxy path). It writes
+   `eval/final_v2_results.json`. A previous attempt produced `eval/final_results.json`
+   but is **invalid** (env server had degraded after 11h; 247/250 cells failed per
+   model — openenv client bug masks connection errors as `TypeError: object NoneType
+   can't be used in 'await' expression`). Overwrite/ignore `eval/final_results.json`.
+   If the job died, relaunch: `run_eval.sh` with env
+   `ADAPTER_REPO=SaylorTwift/LFM2.5-2.6B-opencode-lora-RL, N_TASKS=250, EVAL_TAG=final_v2`,
+   via `huggingface_hub.run_job(..., expose_public=[8000], flavor="h200", timeout="6h")`.
+2. **Interpret the final eval** (250 tasks, ±2% error bars). If it confirms flat:
+   the honest finding is *r=32 LoRA undertrains this model for agentic RL*.
+3. **The natural follow-up run, in order of expected value:**
+   a. **r=64 (or 128) LoRA** — one-line config change (`LoraConfig` in `train_lora.py`),
+      plus `--max-lora-rank 64` wherever `--max-lora-rank 32` appears. Everything else identical.
+   b. **Full fine-tune** (drop `peft_config`) — the article's actual setup; merged sync
+      makes weight transfers ~5.2 GB/step (was fine at 136 ms/step).
+   c. **SFT warm-start then RL** — the article's `FineEnvs/SmolDataEnvs-multiharness-sft`
+      dataset is public and pre-tokenized for LFM2.5-2.6B; their runs showed SFT+RL > RL alone
+      is the natural question. SFT took 1.5–6.5h of trainer time in the article.
+   d. If pass@1 still won't move: raise training temperature exploration, check that the
+      100-task eval subset isn't saturated/hard-skewed (compare a random-100 draw), or
+      evaluate a *bigger* base model on the same stack.
+4. **Backfill efficiency metrics for checkpoints 100–700** — delete
+   `eval/checkpoint-{100..700}_results.json` from the adapter repo and run
+   `run_eval_watch.sh` for one pass (~2h total) if the complete efficiency curve matters.
+5. **Housekeeping:** flip `SaylorTwift/LFM2.5-2.6B-opencode-lora-RL` to public when
+   publishing; `export_adapter.py` regenerates the model card. The stalled-run artifacts
+   (jobs `6ada06b…` through `6adf625f…`) can be inspected with `hf jobs inspect/logs`.
+
+## Known sharp edges (all documented in RECIPE.md's pitfalls section)
+
+- HF Jobs proxy strips `Authorization` → session id must also ride `x-api-key`
+  (patched into the OpenCode seam by every `run_*.sh` entrypoint).
+- Jobs proxy URL: `https://<JOB_ID>--8000.hf.jobs` (ports double-dashed).
+- `run_eval.sh`/`run_job.sh` must serve **only the splits they prepared**
+  (`OPENENV_DATASETS`) — a missing split dir crashes the env server at boot.
+- Long-lived env servers degrade (websocket error loop after ~11h) — prefer fresh
+  servers per eval; don't reuse the training job's server for the final eval.
+- openenv 0.10.0's MCP client masks connection failures with
+  `TypeError: object NoneType can't be used in 'await' expression` — if you see that,
+  it's a connection problem, not the code path you're reading.
+- `hf jobs logs` retains a bounded window; pull early, grep locally.
+
+## Ops cheatsheet
+
+```bash
+hf jobs ps --all                       # list jobs
+hf jobs logs SaylorTwift/<id>          # fetch logs (grep locally, logs are big)
+hf jobs inspect SaylorTwift/<id> --format json   # status incl. stage/message
+hf jobs cancel SaylorTwift/<id>
+hf download <repo> --include "eval/*"  # grab results
+# Launch pattern: huggingface_hub.run_job(..., expose_public=[8000], timeout="6h",
+#   flavor="h200", env={...}, secrets={"HF_TOKEN": token})
+# (The CLI lacks expose_public; the eval path needs it because HF Jobs
+#  auth-gated --expose breaks sandbox→proxy calls.)
+```