| --- |
| base_model: |
| - Qwen/Qwen3-VL-2B-Instruct |
| - Qwen/Qwen3-VL-4B-Instruct |
| library_name: peft |
| pipeline_tag: image-text-to-text |
| tags: [lora, grpo, rlhf, gui-agent, android, androidflux] |
| --- |
| |
| # AF_RL — AndroidFlux GRPO LoRA adapters |
| |
| GRPO-trained LoRA adapters for **Qwen3-VL-2B-Instruct** on Android GUI error |
| recovery. Each adapter lives in its own subfolder. |
| |
| | subfolder | data split | reward model | final reward | |
| |---|---|---|---| |
| | `scalar-firsterror-20260817` | first_error (2,662) | Bradley-Terry scalar head | +4.608 | |
| | `discrete-firsterror-20260817` | first_error (2,662) | judge-token margin | −2.273 | |
| | `scalar-preerror-20260818` | pre_error (2,669) | Bradley-Terry scalar head | +4.982 | |
| | `discrete-preerror-20260818` | pre_error (2,669) | judge-token margin | −5.053 | |
| |
| **Reward numbers are not comparable across rows.** The two reward models emit |
| different scales, and the two splits are different states. Compare policies by |
| downstream task performance, not by these values. |
| |
| - **first_error** — the screen *immediately after* the first erroneous action; |
| the policy must recover. |
| - **pre_error** — the screen *before* the erroneous action; the policy must avoid |
| the mistake. |
| |
| ```python |
| import torch |
| from transformers import AutoModelForImageTextToText |
| from peft import PeftModel |
| |
| base = AutoModelForImageTextToText.from_pretrained( |
| "Qwen/Qwen3-VL-2B-Instruct", |
| revision="89644892e4d85e24eaac8bacfd4f463576704203", |
| dtype=torch.bfloat16, device_map="cuda") |
| model = PeftModel.from_pretrained( |
| base, "YeMoKoo/AF_RL", subfolder="scalar-preerror-20260818") |
| ``` |
| |
| Qwen3-VL is a vision-language model, so `AutoModelForCausalLM` / |
| `AutoPeftModelForCausalLM` do **not** work — use `AutoModelForImageTextToText`. |
| After loading, assert |
| `sum("lora" in n.lower() for n, _ in model.named_parameters()) == 392`. |
|
|
| All four were trained from |
| `Gyubeum/AndroidFlux_RL_Train_Test` @ `4545695c26951a4884e8f18e6468d69cbdf9d174` |
| for 1 epoch with identical hyperparameters — only the reward model and the data |
| split differ. See each subfolder's `README.md` for full details. |
|
|
| Intended comparison (run elsewhere, on held-out data): |
| base `Qwen/Qwen3-VL-2B-Instruct` vs. scalar-GRPO vs. discrete-GRPO. |
|
|
| ## Qwen3-VL-4B adapters (2026-09-24/25) |
|
|
| Four offline-GRPO LoRA adapters for **Qwen3-VL-4B-Instruct**, trained with the same recipe, reward model |
| (`Gyubeum/Qwen3-VL-8B-Instruct-UI-Genie-scoring-128k-balanced`, Bradley-Terry scalar + GPRS) and budget |
| (2,400 prompts, 150 steps); only the training-data mix differs. HV65 success rate (65 human-verified AndroidWorld tasks): |
|
|
| | subfolder | training data | clean | error_t | mean | |
| |---|---|---:|---:|---:| |
| | (base Qwen3-VL-4B-Instruct, no RL) | — | 20.0 | 21.5 | 20.8 | |
| | `grpo4b-ui100-20260924` | UI-Genie 100% | 23.1 | 16.9 | 20.0 | |
| | `grpo4b-t100-20260924` | AndroidFlux t 100% | 18.5 | 30.8 | 24.6 | |
| | `grpo4b-ui50-t25-t1-25-20260924` | UI 50 / t 25 / t-1 25 | 29.2 | 21.5 | 25.4 | |
| | `grpo4b-ui40-t35-t1-25-20260925` | UI 40 / t 35 / t-1 25 | 27.7 | 26.2 | 26.9 | |
| |
| Load with `subfolder=` exactly as above, but with base `Qwen/Qwen3-VL-4B-Instruct` @ |
| `ebb281ec70b05090aa6165b016eac8ec08e71b17`, and assert **504** LoRA tensors (not 392). Each subfolder's |
| `README.md` has the full recipe and data provenance; `trainer_state.json` has the per-step logs. |
|
|
