README.md

---
base_model:
- Qwen/Qwen3-VL-2B-Instruct
- Qwen/Qwen3-VL-4B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags: [lora, grpo, rlhf, gui-agent, android, androidflux]
---

# AF_RL — AndroidFlux GRPO LoRA adapters

GRPO-trained LoRA adapters for **Qwen3-VL-2B-Instruct** on Android GUI error
recovery. Each adapter lives in its own subfolder.

| subfolder | data split | reward model | final reward |
|---|---|---|---|
| `scalar-firsterror-20260817` | first_error (2,662) | Bradley-Terry scalar head | +4.608 |
| `discrete-firsterror-20260817` | first_error (2,662) | judge-token margin | −2.273 |
| `scalar-preerror-20260818` | pre_error (2,669) | Bradley-Terry scalar head | +4.982 |
| `discrete-preerror-20260818` | pre_error (2,669) | judge-token margin | −5.053 |

**Reward numbers are not comparable across rows.** The two reward models emit
different scales, and the two splits are different states. Compare policies by
downstream task performance, not by these values.

- **first_error** — the screen *immediately after* the first erroneous action;
  the policy must recover.
- **pre_error** — the screen *before* the erroneous action; the policy must avoid
  the mistake.

```python
import torch
from transformers import AutoModelForImageTextToText
from peft import PeftModel

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-2B-Instruct",
    revision="89644892e4d85e24eaac8bacfd4f463576704203",
    dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(
    base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818")
```

Qwen3-VL is a vision-language model, so `AutoModelForCausalLM` /
`AutoPeftModelForCausalLM` do **not** work — use `AutoModelForImageTextToText`.
After loading, assert
`sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392`.

All four were trained from
`Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174`
for 1 epoch with identical hyperparameters — only the reward model and the data
split differ. See each subfolder's `README.md` for full details.

Intended comparison (run elsewhere, on held-out data):
base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO.

## Qwen3-VL-4B adapters (2026-09-24/25)

Four offline-GRPO LoRA adapters for **Qwen3-VL-4B-Instruct**, trained with the same recipe, reward model
(`Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced`, Bradley-Terry scalar + GPRS) and budget
(2,400 prompts, 150 steps); only the training-data mix differs. HV65 success rate (65 human-verified AndroidWorld tasks):

| subfolder | training data | clean | error_t | mean |
|---|---|---:|---:|---:|
| (base Qwen3-VL-4B-Instruct, no RL) | — | 20.0 | 21.5 | 20.8 |
| `grpo4b-ui100-20260924` | UI-Genie 100% | 23.1 | 16.9 | 20.0 |
| `grpo4b-t100-20260924` | AndroidFlux t 100% | 18.5 | 30.8 | 24.6 |
| `grpo4b-ui50-t25-t1-25-20260924` | UI 50 / t 25 / t-1 25 | 29.2 | 21.5 | 25.4 |
| `grpo4b-ui40-t35-t1-25-20260925` | UI 40 / t 35 / t-1 25 | 27.7 | 26.2 | 26.9 |

Load with `subfolder=` exactly as above, but with base `Qwen/Qwen3-VL-4B-Instruct` @
`ebb281ec70b05090aa6165b016eac8ec08e71b17`, and assert **504** LoRA tensors (not 392). Each subfolder's
`README.md` has the full recipe and data provenance; `trainer_state.json` has the per-step logs.