Pin trl==1.15.0 (pre-tree-packing, LFM2 hybrid compatible) + vllm v0.31.0 image

73712c7 verified 19 days ago

6.52 kB

# OpenCode-only RL on LFM2.5-2.6B with LoRA

Reproduction of the OpenCode-only run from
[The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl),
but training a **LoRA adapter** instead of full weights, so the result is a small,
distributable adapter on top of `LiquidAI/LFM2.5-2.6B`.

Stack: OpenEnv (capture proxy + Harbor integration) → Harbor (tasks + **HF Sandboxes**) →
TRL `AsyncGRPOTrainer` with `HarnessRolloutWorker` → vLLM policy server.
Every API here was verified against the released wheels (trl 1.16.0, openenv 0.10.0,
harbor 0.24.0), the `harbor_env` source, and the FineEnvs production env Space.

**Version pins that matter:** `trl==1.15.0` (1.16.0 made tree packing unconditional
in AsyncGRPOTrainer and refuses hybrid models — LFM2.5 has conv layers; 1.15.0 has
the harness worker but predates tree packing) and `vllm/vllm-openai:v0.31.0` (top
of trl 1.15.0's supported vLLM band, and predates its mcp>=2 requirement which
conflicts with openenv's fastmcp 3.x).

**Sandboxes are HF Sandboxes** (Harbor's `hf-sandbox` backend, harbor>=0.24): each
rollout boots an HF Jobs cpu-basic machine running the task's prebuilt image
(`savatar101/env-data-agent-train:base` — the same image every task's Dockerfile is
a one-line `FROM` of). Only `HF_TOKEN` is needed; no Daytona/E2B account.

## Why merged-weight sync (source-verified)

Generation goes harness → capture proxy → vLLM, addressed by a **fixed model name**.
TRL's adapter-only sync publishes versioned adapters (`trl-policy-v{N}`) and only the
trainer-built default worker learns the versioned name; an injected
`HarnessRolloutWorker` never gets one. So the server runs *without* `--enable-lora`
and the trainer merges the LoRA into the base and NCCL-streams full weights each step
(~5.2 GB for 2.6B — trivial on one node; vLLM also serves adapters 1.33–1.39x slower
per token). **Checkpoints still contain only the LoRA adapter** — that is what you ship.

## Files

| File | Purpose |
|---|---|
| `train_lora.py` | Worker wiring + `AsyncGRPOConfig` + `LoraConfig` (r=32 on q/k/v/out_proj, w1/2/3). Task dataset is discovered from the env server via `HarborSessionFactory.prompt_rows()` — `create()` rejects any prompt the server doesn't serve |
| `serve_policy.sh` | vLLM with capture-proxy flags + NCCL weight transfer |
| `serve_env.sh` | `openenv harbor serve` with production env vars (`MAX_CONCURRENT_ENVS`, pinned OpenCode 1.18.31, 4096 output cap) |
| `prepare_tasks.py` + `data/` | Materializes the pinned 1,000 train / 250 test tasks (hash-checked, from the FineEnvs Space) and patches `docker_image` into each task.toml — the one change HF Sandboxes require |
| `run_job.sh` | Single-container entrypoint for HF Jobs (setup + all 3 processes) |
| `export_adapter.py` | Publish any adapter checkpoint to the Hub (+ optional merged model) |

## Launch on HF Jobs

```bash
# 0. One-time: put this folder on the Hub so the job can pull it
hf repos create YOU/lfm-opencode-lora-rl-code --type model --private
hf upload YOU/lfm-opencode-lora-rl-code . \
  --include "*.py" --include "*.sh" --include "*.md" --include "data/*.json"
# 1. Smoke run (~1-2h): validates weight sync, capture contract, sandbox boots
hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 6h --detach \
  --secrets HF_TOKEN="$(hf auth token)" \
  --env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=20 \
  -- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'
# 2. Watch: merged weight transfers (no .vllm_lora/), sandbox Jobs booting under
#    your account (hf jobs ps), rollouts returning rewards, no CaptureContractError.
hf jobs logs <job-id> --follow
# 3. Full run (the article's OpenCode-only run took ~32h on 2x H100)
hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 72h --detach \
  --secrets HF_TOKEN="$(hf auth token)" \
  --env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=1000 \
  -- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'
```

Checkpoints (adapter-only) push to `ADAPTER_REPO` every 100 steps, so a dead job
loses little. `SANDBOX=daytona`/`e2b` + the matching API key secret still works.

## What you distribute

Every checkpoint contains `adapter_model.safetensors` + `adapter_config.json`
(~50 MB at r=32). `export_adapter.py` adds a model card and pushes:

```bash
python export_adapter.py checkpoints/checkpoint-1000 YOU/LFM2.5-2.6B-opencode-lora-RL [--also-merged]
```

Consumers load it with `PeftModel.from_pretrained("LiquidAI/LFM2.5-2.6B", YOU/...)` or
`vllm serve LiquidAI/LFM2.5-2.6B --enable-lora --max-lora-rank 32 --lora-modules opencode-lora=YOU/...`
(`Lfm2ForCausalLM` implements `SupportsLoRA`).

## Recipe choices carried over from the article (don't regress these)

- `top_p=1.0`, `top_k=0` — truncation biases the policy and breaks logprob agreement.
- Same 4096-token output cap in training and evaluation (`OPENENV_MAX_OUTPUT_TOKENS`
  + worker `max_tokens`). A Qwen run with 16k training answers collapsed at 4k eval.
- Correctness + <=0.1 tool-efficiency bonus on correct answers (else all-correct
  GRPO groups carry no signal and tool calls creep up).
- `num_generations=8`, one harness per group, so advantage compares actions.
- `max_tool_calling_iterations=32` on the worker — the loop-owning default would
  silently cap the agent at 8 model calls.
- `max_staleness=4` (config default) matches the article's async setup.

## Honest caveats

- `HarnessRolloutWorker` + `peft_config` and `sandbox="hf-sandbox"` are both
  untested-in-print combinations on this stack. All pieces are released and the
  interfaces line up (verified in source) — the smoke run exists to prove it before
  you spend 32+ GPU-hours.
- HF Sandboxes are experimental (per HF docs) and each rollout is an HF Job under
  your token: mind your Jobs concurrency quota (`MAX_CONCURRENT_ENVS=8` bounds it)
  and per-minute billing of thousands of short cpu-basic jobs over a full run.
- Sandbox boot time (job scheduling + first image pull) adds latency per rollout
  vs Daytona/E2B's warm pools; expect slower wall-clock than the article's 32h.
- Expect the article's single-harness shape: gains concentrate in OpenCode
  (34 → 58% there in their run) and transfer unevenly elsewhere.
- LoRA r=32 may cap peak accuracy vs full FT; no published comparison on this stack.

	# OpenCode-only RL on LFM2.5-2.6B with LoRA

	Reproduction of the OpenCode-only run from
	[The ultimate guide to multi-harness RL](https://huggingface.co/spaces/FineEnvs/multi-harness-rl),
	but training a LoRA adapter instead of full weights, so the result is a small,
	distributable adapter on top of `LiquidAI/LFM2.5-2.6B`.

	Stack: OpenEnv (capture proxy + Harbor integration) → Harbor (tasks + HF Sandboxes) →
	TRL `AsyncGRPOTrainer` with `HarnessRolloutWorker` → vLLM policy server.
	Every API here was verified against the released wheels (trl 1.16.0, openenv 0.10.0,
	harbor 0.24.0), the `harbor_env` source, and the FineEnvs production env Space.

	Version pins that matter: `trl==1.15.0` (1.16.0 made tree packing unconditional
	in AsyncGRPOTrainer and refuses hybrid models — LFM2.5 has conv layers; 1.15.0 has
	the harness worker but predates tree packing) and `vllm/vllm-openai:v0.31.0` (top
	of trl 1.15.0's supported vLLM band, and predates its mcp>=2 requirement which
	conflicts with openenv's fastmcp 3.x).

	Sandboxes are HF Sandboxes (Harbor's `hf-sandbox` backend, harbor>=0.24): each
	rollout boots an HF Jobs cpu-basic machine running the task's prebuilt image
	(`savatar101/env-data-agent-train:base` — the same image every task's Dockerfile is
	a one-line `FROM` of). Only `HF_TOKEN` is needed; no Daytona/E2B account.

	## Why merged-weight sync (source-verified)

	Generation goes harness → capture proxy → vLLM, addressed by a fixed model name.
	TRL's adapter-only sync publishes versioned adapters (`trl-policy-v{N}`) and only the
	trainer-built default worker learns the versioned name; an injected
	`HarnessRolloutWorker` never gets one. So the server runs without `--enable-lora`
	and the trainer merges the LoRA into the base and NCCL-streams full weights each step
	(~5.2 GB for 2.6B — trivial on one node; vLLM also serves adapters 1.33–1.39x slower
	per token). Checkpoints still contain only the LoRA adapter — that is what you ship.

	## Files

	\| File \| Purpose \|
	\|---\|---\|
	\| `train_lora.py` \| Worker wiring + `AsyncGRPOConfig` + `LoraConfig` (r=32 on q/k/v/out_proj, w1/2/3). Task dataset is discovered from the env server via `HarborSessionFactory.prompt_rows()` — `create()` rejects any prompt the server doesn't serve \|
	\| `serve_policy.sh` \| vLLM with capture-proxy flags + NCCL weight transfer \|
	\| `serve_env.sh` \| `openenv harbor serve` with production env vars (`MAX_CONCURRENT_ENVS`, pinned OpenCode 1.18.31, 4096 output cap) \|
	\| `prepare_tasks.py` + `data/` \| Materializes the pinned 1,000 train / 250 test tasks (hash-checked, from the FineEnvs Space) and patches `docker_image` into each task.toml — the one change HF Sandboxes require \|
	\| `run_job.sh` \| Single-container entrypoint for HF Jobs (setup + all 3 processes) \|
	\| `export_adapter.py` \| Publish any adapter checkpoint to the Hub (+ optional merged model) \|

	## Launch on HF Jobs

	```bash
	# 0. One-time: put this folder on the Hub so the job can pull it
	hf repos create YOU/lfm-opencode-lora-rl-code --type model --private
	hf upload YOU/lfm-opencode-lora-rl-code . \
	--include ".py" --include ".sh" --include ".md" --include "data/.json"

	# 1. Smoke run (~1-2h): validates weight sync, capture contract, sandbox boots
	hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 6h --detach \
	--secrets HF_TOKEN="$(hf auth token)" \
	--env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=20 \
	-- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'

	# 2. Watch: merged weight transfers (no .vllm_lora/), sandbox Jobs booting under
	# your account (hf jobs ps), rollouts returning rewards, no CaptureContractError.
	hf jobs logs <job-id> --follow

	# 3. Full run (the article's OpenCode-only run took ~32h on 2x H100)
	hf jobs run vllm/vllm-openai:v0.31.0 --flavor h200x2 --timeout 72h --detach \
	--secrets HF_TOKEN="$(hf auth token)" \
	--env ADAPTER_REPO=YOU/LFM2.5-2.6B-opencode-lora-RL MAX_STEPS=1000 \
	-- bash -c 'pip -q install huggingface_hub && hf download YOU/lfm-opencode-lora-rl-code --local-dir /app && bash /app/run_job.sh'
	```

	Checkpoints (adapter-only) push to `ADAPTER_REPO` every 100 steps, so a dead job
	loses little. `SANDBOX=daytona`/`e2b` + the matching API key secret still works.

	## What you distribute

	Every checkpoint contains `adapter_model.safetensors` + `adapter_config.json`
	(~50 MB at r=32). `export_adapter.py` adds a model card and pushes:

	```bash
	python export_adapter.py checkpoints/checkpoint-1000 YOU/LFM2.5-2.6B-opencode-lora-RL [--also-merged]
	```

	Consumers load it with `PeftModel.from_pretrained("LiquidAI/LFM2.5-2.6B", YOU/...)` or
	`vllm serve LiquidAI/LFM2.5-2.6B --enable-lora --max-lora-rank 32 --lora-modules opencode-lora=YOU/...`
	(`Lfm2ForCausalLM` implements `SupportsLoRA`).

	## Recipe choices carried over from the article (don't regress these)

	- `top_p=1.0`, `top_k=0` — truncation biases the policy and breaks logprob agreement.
	- Same 4096-token output cap in training and evaluation (`OPENENV_MAX_OUTPUT_TOKENS`
	+ worker `max_tokens`). A Qwen run with 16k training answers collapsed at 4k eval.
	- Correctness + <=0.1 tool-efficiency bonus on correct answers (else all-correct
	GRPO groups carry no signal and tool calls creep up).
	- `num_generations=8`, one harness per group, so advantage compares actions.
	- `max_tool_calling_iterations=32` on the worker — the loop-owning default would
	silently cap the agent at 8 model calls.
	- `max_staleness=4` (config default) matches the article's async setup.

	## Honest caveats

	- `HarnessRolloutWorker` + `peft_config` and `sandbox="hf-sandbox"` are both
	untested-in-print combinations on this stack. All pieces are released and the
	interfaces line up (verified in source) — the smoke run exists to prove it before
	you spend 32+ GPU-hours.
	- HF Sandboxes are experimental (per HF docs) and each rollout is an HF Job under
	your token: mind your Jobs concurrency quota (`MAX_CONCURRENT_ENVS=8` bounds it)
	and per-minute billing of thousands of short cpu-basic jobs over a full run.
	- Sandbox boot time (job scheduling + first image pull) adds latency per rollout
	vs Daytona/E2B's warm pools; expect slower wall-clock than the article's 32h.
	- Expect the article's single-harness shape: gains concentrate in OpenCode
	(34 → 58% there in their run) and transfer unevenly elsewhere.
	- LoRA r=32 may cap peak accuracy vs full FT; no published comparison on this stack.