# OpenCode-only RL on LFM2.5-2.6B with LoRA

Reproduction of the OpenCode-only run from
[The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl),
but training a **LoRA adapter** instead of full weights, so the result is a small,
distributable adapter on top of `LiquidAI/LFM2.5-2.6B`.

Stack: OpenEnv (capture proxy + Harbor integration) → Harbor (tasks + **HF Sandboxes**) →
TRL `AsyncGRPOTrainer` with `HarnessRolloutWorker` → vLLM policy server.
Every API here was verified against the released wheels (trl 1.16.0, openenv 0.10.0,
harbor 0.24.0), the `harbor_env` source, and the FineEnvs production env Space.

**Version pins that matter:** `trl==1.15.0` (1.16.0 made tree packing unconditional
in AsyncGRPOTrainer and refuses hybrid models — LFM2.5 has conv layers; 1.15.0 has
the harness worker but predates tree packing) and `vllm/vllm-openai:v0.31.0` (top
of trl 1.15.0's supported vLLM band, and predates its mcp>=2 requirement which
conflicts with openenv's fastmcp 3.x).

**Sandboxes are HF Sandboxes** (Harbor's `hf-sandbox` backend, harbor>=0.24): each
rollout boots an HF Jobs cpu-basic machine running the task's prebuilt image
(`savatar101/env-data-agent-train:base` — the same image every task's Dockerfile is
a one-line `FROM` of). Only `HF_TOKEN` is needed; no Daytona/E2B account.

## Why merged-weight sync (source-verified)

Generation goes harness → capture proxy → vLLM, addressed by a **fixed model name**.
TRL's adapter-only sync publishes versioned adapters (`trl-policy-v{N}`) and only the
trainer-built default worker learns the versioned name; an injected
`HarnessRolloutWorker` never gets one. So the server runs *without* `--enable-lora`
and the trainer merges the LoRA into the base and NCCL-streams full weights each step
(~5.2 GB for 2.6B — trivial on one node; vLLM also serves adapters 1.33–1.39x slower
per token). **Checkpoints still contain only the LoRA adapter** — that is what you ship.

## Files

| File | Purpose |
|---|---|
| `train_lora.py` | Worker wiring + `AsyncGRPOConfig` + `LoraConfig` (r=32 on q/k/v/out_proj, w1/2/3). Task dataset is discovered from the env server via `HarborSessionFactory.prompt_rows()` — `create()` rejects any prompt the server doesn't serve |
| `serve_policy.sh` | vLLM with capture-proxy flags + NCCL weight transfer |
| `serve_env.sh` | `openenv harbor serve` with production env vars (`MAX_CONCURRENT_ENVS`, pinned OpenCode 1.18.31, 4096 output cap) |
| `prepare_tasks.py` + `data/` | Materializes the pinned 1,000 train / 250 test tasks (hash-checked, from the FineEnvs Space) and patches `docker_image` into each task.toml — the one change HF Sandboxes require |
| `run_job.sh` | Single-container entrypoint for HF Jobs (setup + all 3 processes) |
| `export_adapter.py` | Publish any adapter checkpoint to the Hub (+ optional merged model) |

## Launch on HF Jobs

```bash
# 0. One-time: put this folder on the Hub so the job can pull it
hf repos create YOU/lfm-opencode-lora-rl-code --type model --private
hf upload YOU/lfm-opencode-lora-rl-code . \
  --include "*.py" --include "*.sh" --include "*.md" --include "data/*.json"

# 1. Smoke run (~1-2h): validates weight sync, capture contract, sandbox boots
hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 6h --detach \
  --secrets HF_TOKEN="$(hf auth token)" \
  --env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=20 \
  -- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'

# 2. Watch: merged weight transfers (no .vllm_lora/), sandbox Jobs booting under
#    your account (hf jobs ps), rollouts returning rewards, no CaptureContractError.
hf jobs logs <job-id> --follow

# 3. Full run (the article's OpenCode-only run took ~32h on 2x H100)
hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 72h --detach \
  --secrets HF_TOKEN="$(hf auth token)" \
  --env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=1000 \
  -- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'
```

Checkpoints (adapter-only) push to `ADAPTER_REPO` every 100 steps, so a dead job
loses little. `SANDBOX=daytona`/`e2b` + the matching API key secret still works.

## What you distribute

Every checkpoint contains `adapter_model.safetensors` + `adapter_config.json`
(~50 MB at r=32). `export_adapter.py` adds a model card and pushes:

```bash
python export_adapter.py checkpoints/checkpoint-1000 YOU/LFM2.5-2.6B-opencode-lora-RL [--also-merged]
```

Consumers load it with `PeftModel.from_pretrained("LiquidAI/LFM2.5-2.6B", YOU/...)` or
`vllm serve LiquidAI/LFM2.5-2.6B --enable-lora --max-lora-rank 32 --lora-modules opencode-lora=YOU/...`
(`Lfm2ForCausalLM` implements `SupportsLoRA`).

## Recipe choices carried over from the article (don't regress these)

- `top_p=1.0`, `top_k=0` — truncation biases the policy and breaks logprob agreement.
- Same 4096-token output cap in training and evaluation (`OPENENV_MAX_OUTPUT_TOKENS`
  + worker `max_tokens`). A Qwen run with 16k training answers collapsed at 4k eval.
- Correctness + <=0.1 tool-efficiency bonus on correct answers (else all-correct
  GRPO groups carry no signal and tool calls creep up).
- `num_generations=8`, one harness per group, so advantage compares actions.
- `max_tool_calling_iterations=32` on the worker — the loop-owning default would
  silently cap the agent at 8 model calls.
- `max_staleness=4` (config default) matches the article's async setup.

## Honest caveats

- `HarnessRolloutWorker` + `peft_config` and `sandbox="hf-sandbox"` are both
  untested-in-print combinations on this stack. All pieces are released and the
  interfaces line up (verified in source) — the smoke run exists to prove it before
  you spend 32+ GPU-hours.
- HF Sandboxes are experimental (per HF docs) and each rollout is an HF Job under
  your token: mind your Jobs concurrency quota (`MAX_CONCURRENT_ENVS=8` bounds it)
  and per-minute billing of thousands of short cpu-basic jobs over a full run.
- Sandbox boot time (job scheduling + first image pull) adds latency per rollout
  vs Daytona/E2B's warm pools; expect slower wall-clock than the article's 32h.
- Expect the article's single-harness shape: gains concentrate in OpenCode
  (34 → 58% there in their run) and transfer unevenly elsewhere.
- LoRA r=32 may cap peak accuracy vs full FT; no published comparison on this stack.