| # InvoiceGuard Round 2 — Trajectory-level GRPO |
|
|
| This package trains a small instruction-tuned LM (default `Qwen/Qwen3-4B-Instruct-2507`) |
| on the InvoiceGuard OpenEnv with a hand-written multi-step GRPO loop: |
|
|
| - Sample **G trajectories** per training task with the current (stochastic) policy. |
| - Trajectory reward = env cumulative reward + `grader_bonus * grader_score`. |
| - **Group-relative advantage** = z-score within the G trajectories of one task. |
| - **PPO-clipped policy gradient** on every `(observation → action)` pair in |
| each trajectory, weighted by that trajectory's advantage, regularised by |
| KL against a **frozen reference** (the same model with the LoRA adapter |
| disabled — no second copy of the base model in memory). |
| - LoRA on attention projections; everything else frozen. |
|
|
| We deliberately do **not** use TRL's `GRPOTrainer` — it assumes a single-turn |
| reward, but our env is multi-turn agentic. The whole loop is in |
| [`train_grpo.py`](./train_grpo.py) and is ~300 lines. |
|
|
| ## Files |
|
|
| | File | Purpose | |
| | --- | --- | |
| | `rollout.py` | Drives the local `InvoiceGuardEnvironment` with an HF model; reuses `inference.py`'s prompt/parse helpers so trajectories are IO-identical to the OpenAI baseline. | |
| | `train_grpo.py` | The trainer. Self-contained PEP 723 UV script — runnable both as a Hugging Face Jobs payload and locally for smoke tests. | |
| | `launch_hf_job.py` | Uploads the `invoice_guard/` source to a Hub code repo, then submits `train_grpo.py` to HF Jobs with the right env vars and secrets. | |
|
|
| ## Train / eval split |
|
|
| Split is deterministic from `--seed` (default 42): |
|
|
| - **Holdout (never trained on):** `eval_holdout_canonical=3` canonical + `eval_holdout_hard=3` hard tasks. |
| - **Train:** the remaining 9 canonical + 7 hard = 16 tasks. |
|
|
| The end-of-iteration eval inside the trainer reports the average grader score |
| on the holdout, so you can see the learning curve in Trackio. The full |
| benchmark for the README plots is produced separately by |
| [`eval_round2.py`](../eval_round2.py). |
|
|
| ## Quick local smoke test (no GPU, no Hub push) |
|
|
| ```powershell |
| cd invoice_guard |
| ..\.venv\Scripts\python -m training.train_grpo ` |
| --model-name Qwen/Qwen3-4B-Instruct-2507 ` |
| --num-iterations 1 --group-size 2 --max-train-tasks 2 --no-push |
| ``` |
|
|
| This uses the in-tree env (no clone), runs one iteration over 2 tasks with |
| G=2 trajectories each, and saves the LoRA adapter to `/tmp/invoiceguard-grpo`. |
| On CPU it's slow but verifies the wiring. Use a small base model. |
|
|
| ## Launch on Hugging Face Jobs |
|
|
| Prereqs: HF Pro/Team/Enterprise plan, `hf auth login` done locally, |
| `pip install huggingface_hub`. |
|
|
| ```powershell |
| cd invoice_guard |
| ..\.venv\Scripts\python training\launch_hf_job.py ` |
| --hf-username <your-username> ` |
| --flavor a10g-large ` |
| --timeout 4h ` |
| --base-model Qwen/Qwen3-4B-Instruct-2507 ` |
| --num-iterations 3 --group-size 4 |
| ``` |
|
|
| What happens: |
|
|
| 1. The launcher uploads `invoice_guard/` to `{your-username}/invoiceguard-code` |
| (creates the repo if needed; ignores `outputs/`, `.venv/`, `.env`). |
| 2. `train_grpo.py` is submitted to HF Jobs from the uploaded Hub script URL. |
| 3. Inside the container the script `snapshot_download`s the code repo and adds |
| it to `sys.path`, then runs the GRPO loop. |
| 4. Trackio dashboard appears at `https://huggingface.co/spaces/<user>/trackio` |
| (project `invoiceguard-round2`, run `qwen3-4b-grpo`). |
| 5. On completion the LoRA adapter is pushed to |
| `{your-username}/invoiceguard-qwen3-4b-grpo`. |
|
|
| ## After training |
|
|
| Run the Round 2 benchmark against the trained model: |
|
|
| ```powershell |
| $env:API_BASE_URL = "<endpoint serving your-username/invoiceguard-qwen3-4b-grpo>" |
| $env:MODEL_NAME = "<your-username>/invoiceguard-qwen3-4b-grpo" |
| ..\.venv\Scripts\python eval_round2.py --slice all --model-tag trained_qwen3_4b_grpo |
| ..\.venv\Scripts\python eval_round2.py --compare ` |
| outputs\baseline_scores\hard__clean_qwen3_4b.json ` |
| outputs\round2\hard__trained_qwen3_4b_grpo.json |
| ``` |
|
|
| The compare output is what the README before/after plot is built from in |
| Stage H. |
|
|
| ## Cost estimate |
|
|
| | Setting | Approx | Notes | |
| | --- | --- | --- | |
| | `l40s` / `a100-large` × 2-4h | varies | Default — Qwen3-4B-Instruct, 3 iter × 16 tasks × G=4 = 192 trajectories per epoch. | |
| | `a10g-large` × 3-5h | lower cost | May require shorter prompts/generation and 4-bit LoRA. | |
| | Smoke test (CPU) | $0 | Uses in-tree env, no Hub push. | |
|
|
