Add RECIPE.md: shareable setup + run recipe with pitfalls

RECIPE.md

@@ -0,0 +1,205 @@
+# Recipe: OpenCode-only agentic RL on LFM2.5-2.6B, shipped as a LoRA adapter
+
+Train a small open model *inside the OpenCode agent harness* — unmodified, running in
+cloud sandboxes — with async GRPO, and distribute the result as a ~170 MB LoRA
+adapter on top of [`LiquidAI/LFM2.5-2.6B`](https://huggingface.co/LiquidAI/LFM2.5-2.6B).
+
+Everything runs on Hugging Face infrastructure: training on HF Jobs, sandboxes on HF
+Sandboxes (Harbor's `hf-sandbox` backend). The only credential needed is an HF token.
+
+Based on the stack from [The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl)
+(OpenEnv capture proxy + Harbor + TRL), narrowed to one harness and adapted for LoRA
+and HF-only infrastructure.
+
+---
+
+## Architecture
+
+```
+                 ┌─────────────────────────── HF Job (h200x2) ───────────────────────────┐
+                 │                                                                       │
+ GRPO groups ──► │  TRL AsyncGRPOTrainer (GPU 0)                                         │
+ of 8 rollouts   │    · LoRA r=32 on q/k/v/out_proj + w1/w2/w3                            │
+                 │    · merges adapter → streams FULL weights over NCCL (136 ms/step)    │
+                 │            │                           ▲                              │
+                 │            ▼ weight sync               │ tokens + logprobs             │
+                 │  vLLM 0.31 (GPU 1) ◄── capture proxy ◄─┘ (mounted at /capture)        │
+                 │    · --return-tokens-as-token-ids                                     │
+                 │    · --logprobs-mode processed_logprobs                               │
+                 │    · --tool-call-parser lfm2 --enable-auto-tool-choice                │
+                 │            ▲                           │                              │
+                 │  OpenEnv Harbor env server (CPU) ──────┘                              │
+                 │    · task API · capture proxy · per-rollout sandbox lifecycle         │
+                 └───────────────┬───────────────────────────────────────────────────────┘
+                                 │ https://<job-id>--8000.hf.jobs/capture
+                                 │ (session id as x-api-key — Jobs proxy strips Authorization)
+                 ┌───────────────▼────────────────┐
+                 │  HF Sandboxes (1 per rollout)  │  cpu-basic Jobs running the task image
+                 │  OpenCode → runs the task →    │  pulls Kaggle data from a public bucket,
+                 │  verifier grades the answer    │  agent loop owned entirely by OpenCode
+                 └────────────────────────────────┘
+```
+
+**Data flow per rollout:** the trainer's `HarnessRolloutWorker` asks the env server
+for a rollout → env server boots an HF Sandbox with the task → OpenCode runs inside,
+calling the model through the capture proxy → proxy records exact prompt/completion
+token IDs + per-token logprobs from vLLM → sandbox verifier scores the answer →
+reward (correctness + tool-efficiency bonus) and the token-faithful trace go back to
+the trainer → GRPO updates the LoRA → merged weights stream to vLLM for the next group.
+
+---
+
+## Key design decisions (and why)
+
+| Decision | Why |
+|---|---|
+| **LoRA r=32, α=64, on `q,k,v,out_proj,w1,w2,w3`** | Distributable artifact (~170 MB). Module names verified against the model's safetensors index. Short-conv projections left out for portability. |
+| **Merged-weight sync, not vLLM `--enable-lora` sync** | Generation flows through the capture proxy addressed by a *fixed model name*; TRL's adapter sync rotates versioned adapter names that only its own built-in worker knows. Merged sync is 136 ms/step over NCCL for 2.6B — and vLLM serves adapters ~1.35× slower per token anyway. Checkpoints still contain only the adapter. |
+| **HF Sandboxes (`hf-sandbox` backend, harbor ≥0.24)** | No Daytona/E2B account; each rollout is an HF Jobs cpu-basic machine under the same token. Requires `docker_image` in task.toml — SmolDataEnvs tasks all `FROM` one prebuilt image, so it's a one-line patch. |
+| **Jobs-proxy exposure instead of a Gradio tunnel** | A Gradio tunnel died 11h into run #1 and stalled it silently. The Jobs proxy (`https://<job-id>--8000.hf.jobs/capture`) is first-party and stable. |
+| **Session id as `x-api-key`** | The Jobs proxy **strips the `Authorization` header** (verified with a header-echo probe). The capture proxy also accepts `x-api-key`; OpenCode's `@ai-sdk/openai-compatible` provider supports custom headers. One-line seam patch. |
+| **trl==1.15.0, not 1.16** | 1.16 made tree packing unconditional and *refuses hybrid models* — LFM2.5 has conv layers. 1.15.0 has `HarnessRolloutWorker` (#6947) but predates tree packing. Bonus: it matches the article's actual training era. |
+| **vllm/vllm-openai:v0.31.0 image** | Top of trl 1.15's supported band; ships the `lfm2` tool parser (for LFM2.5's `<|tool_call_start|>[fn(args)]<|tool_call_end|>` format) and predates vLLM's `mcp>=2` pin, which conflicts with openenv's fastmcp 3.x. |
+| **Reward = correctness + ≤0.1 tool-efficiency bonus** | From the article: without it, all-correct GRPO groups carry no contrast and tool calls creep up. A wrong answer scores 0 regardless, so the bonus can't reward giving up. |
+| **Watchdog + checkpoint pushes every 100 steps** | transformers v5 silently dropped `PushToHubCallback` — periodic saves stay local unless you push them yourself. A heartbeat-touched file + 30-min-stall killer turns silent stalls into loud, resumable failures. |
+
+---
+
+## Exact recipe
+
+### Versions
+
+| Component | Version | Notes |
+|---|---|---|
+| trl | **1.15.0** (`trl[peft]`) | `HarnessRolloutWorker` in `trl.experimental.async_grpo.openenv_harness` |
+| vLLM | **0.31.0** (docker image) | needs `kernels==0.16.0` pip-installed for trl's `kernels-community/flash-attn3` |
+| openenv | **0.10.0** (`openenv[harbor]`) | `harbor_env` comes from the OpenEnv **repo** (`envs/`), not the wheel — clone + `.pth` |
+| harbor | **≥0.24.0** (`harbor[hf-sandbox]`) | the version with `EnvironmentType.HF_SANDBOX` |
+| transformers | **≥5.2.0** (`--no-deps` next to vLLM's stack) | v5 removed PushToHubCallback — see callback below |
+| Base model | `LiquidAI/LFM2.5-2.6B` | `Lfm2ForCausalLM`, 30 layers (20 conv + 10 attention), tied embeddings |
+
+### Data
+
+- **Train:** 1,000 SmolDataEnvs Harbor tasks (400 medium / 600 hard, fixed order),
+  pinned to dataset revision `4719635555de…` with per-instruction SHA-256 checks.
+- **Test:** 250 held-out tasks (no notebook/question overlap), revision `b130595b57…`.
+- Task lists + prep script: `prepare_tasks.py` + `data/{train,test}.json` in the code repo.
+  It also injects `docker_image = "docker.io/savatar101/env-data-agent-train:base"` into
+  every task.toml (required by the hf-sandbox backend) and sets 1 CPU / 4 GB.
+- Task data files are pulled into each sandbox from the public bucket
+  `AdithyaSK/jupyter-agent-kaggle-all` via the task's own healthcheck hook.
+
+### Training hyperparameters
+
+| Knob | Value |
+|---|---|
+| Algorithm | async GRPO (`AsyncGRPOTrainer` + `HarnessRolloutWorker`, loop owned by the harness) |
+| LoRA | r=32, α=64, dropout 0, targets `q_proj,k_proj,v_proj,out_proj,w1,w2,w3` |
+| Steps | 1,000 (1 GRPO group of 8 rollouts ≈ 1 task per step) |
+| Learning rate | 1e-6 |
+| Sampling | temperature 1.0, top_p 1.0, top_k 0 — **no truncation** (it breaks logprob agreement; importance ratio should sit at 0.998–1.0) |
+| Output cap | 4,096 tokens, **same in training and eval** (a mismatch collapsed one of the article's Qwen runs) |
+| Agent budget | 32 model calls/rollout (`max_tool_calling_iterations` — the loop-owning default is 8!) |
+| Staleness | max 4 optimizer steps (default) |
+| Reward | `env_reward` (1/0) + `0.1 × max(0, (32 − tool_calls) / 32)` on correct answers |
+| Hardware | HF Jobs `h200x2` (~90 s/step early, ~35–50 s/step once the rollout queue saturates) |
+
+### Launch
+
+```bash
+# 0. Upload the code bundle (one time)
+hf repos create YOU/lfm-opencode-lora-rl-code --type model --private
+hf upload YOU/lfm-opencode-lora-rl-code . \
+  --include "*.py" --include "*.sh" --include "*.md" --include "data/*.json"
+
+# 1. Launch (Python API — the CLI lacks expose_public)
+uv run --with 'huggingface_hub>=1.29' python - <<'EOF'
+import os
+from huggingface_hub import run_job
+token = os.popen("hf auth token").read().strip()
+run_job(
+    image="vllm/vllm-openai:v0.31.0",
+    command=["bash", "-c",
+             "pip -q install huggingface_hub && "
+             "hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && "
+             "bash /app/run_job.sh"],
+    flavor="h200x2", timeout="72h",
+    env={"ADAPTER_REPO": "YOU/LFM2.5-2.6B-opencode-lora-RL", "MAX_STEPS": "1000"},
+    secrets={"HF_TOKEN": token},
+    expose_public=[8000],
+    token=token,
+)
+EOF
+
+# 2. Watch
+hf jobs logs -f YOU/<job-id>
+```
+
+`run_job.sh` does, in order: install git+packages → clone OpenEnv & `.pth` its `envs/` →
+**patch the OpenCode seam to send the capture session as `x-api-key`** → prepare the
+pinned tasks → start vLLM (GPU 1) → start the env server with `SPACE_HOST` pointed at
+the job's own public URL → start the 30-min watchdog → train (GPU 0) → final eval.
+
+`MAX_STEPS=20` gives a ~40-min smoke run — do that first. (`SANDBOX=daytona`/`e2b`
++ the matching API key also works if you prefer those backends; then the Jobs-proxy
+plumbing is unnecessary.)
+
+### Evaluation (built into the job)
+
+After training, the job restarts vLLM with `--enable-lora --lora-modules trained=$ADAPTER_REPO`
+and runs all 250 test tasks twice — base model vs trained adapter — through the same
+harness/sandbox path (`purpose="eval"`, required when passing `eval_sampling`).
+Pass@1 for each lands in the adapter repo at `eval/final_results.json`.
+A standalone version (`run_eval.sh`) runs the same comparison on any checkpoint.
+
+---
+
+## Pitfalls (each one cost us a run — skip them all)
+
+1. **trl 1.16 rejects LFM2.5.** `ValueError: Lfm2ForCausalLM has ['conv'] layers` — tree
+   packing is attention-only. Pin `trl==1.15.0`.
+2. **`ImportError: kernels`** — trl 1.15 loads `kernels-community/flash-attn3`;
+   `pip install kernels==0.16.0`.
+3. **`train_dataset` is required** by the AsyncGRPOTrainer ctor even with an injected
+   rollout worker. Build it from `HarborSessionFactory(...).prompt_rows()` — `create()`
+   rejects any prompt the env server doesn't serve, and there is no Hub-dataset shortcut.
+4. **`eos_token_ids` is a required worker arg** the article's snippet omits.
+5. **The rollout loop caps the agent at 8 model calls by default** — set
+   `max_tool_calling_iterations` explicitly.
+6. **LFM2.5's chat template isn't recognized** by TRL's `add_response_schema`; set
+   `tokenizer.response_template = trl.chat_template_utils.lfm2_2_5_template` yourself.
+7. **OpenCode always sends `tool_choice="auto"`** → vLLM needs
+   `--enable-auto-tool-choice --tool-call-parser lfm2`, or every rollout 400s.
+8. **The HF Jobs proxy strips `Authorization`.** Session keys must travel as
+   `x-api-key`; patch the OpenCode seam (options.headers) — `run_job.sh` does it.
+9. **transformers v5 has no `PushToHubCallback`** — `hub_strategy="every_save"` is
+   vestigial; push checkpoints in your own `on_save` callback or lose them on crash.
+10. **A tunnel is a single point of failure.** Gradio share died at hour 11. Prefer the
+    Jobs proxy; add a stall watchdog regardless (30 min without an optimizer step → fail).
+11. **`hf jobs run --env` takes one variable per flag**, and the vLLM image has no
+    `python` (only `python3`) and no `git`.
+12. **`run_rollout(eval_sampling=...)` requires `purpose="eval"`** or every cell fails
+    validation.
+
+## Outputs
+
+- **Adapter:** `YOU/LFM2.5-2.6B-opencode-lora-RL` — `adapter_model.safetensors` +
+  `adapter_config.json` (+ `checkpoint-{100,200,...}/` every 100 steps).
+  Load with `PeftModel.from_pretrained("LiquidAI/LFM2.5-2.6B", YOU/...)` or serve with
+  `vllm serve LiquidAI/LFM2.5-2.6B --enable-lora --max-lora-rank 32 --lora-modules opencode-lora=YOU/...`.
+- **Eval:** `eval/final_results.json` — pass@1 base vs trained on 250 held-out tasks,
+  plus both/only-base/only-adapter breakdown.
+- **Cost ballpark:** h200x2 for ~10–25h (task-difficulty dependent) + thousands of
+  cpu-basic sandbox-minutes. Smoke run first: `MAX_STEPS=20`, ~40 min.
+
+## Repo layout
+
+| File | Role |
+|---|---|
+| `train_lora.py` | trainer + worker wiring, reward, LoRA config, upload & heartbeat callbacks |
+| `run_job.sh` | job entrypoint: setup, seam patch, 3 processes, watchdog, final eval |
+| `run_eval.sh` | standalone base-vs-adapter eval job |
+| `eval_adapter.py` | held-out eval loop (pass@1, agreement breakdown) |
+| `prepare_tasks.py` + `data/` | pinned, hash-checked task materialization (+ `docker_image` patch) |
+| `serve_policy.sh` / `serve_env.sh` | the two servers, for multi-machine setups |
+| `export_adapter.py` | publish a checkpoint with a model card (+ optional merged model) |