xuzhl's picture
Model card: arXiv link and citation
d6768ed verified
|
Raw History Blame Contribute Delete
3.36 kB
---
license: cc-by-nc-4.0
base_model: Qwen/Qwen3-VL-4B-Instruct
library_name: peft
pipeline_tag: image-text-to-text
tags:
- lora
- gui-agent
- screen-reader
- mobilegym
---
# Aligned screen reader for MobileGym (LoRA on Qwen3-VL-4B-Instruct)
This adapter comes with the paper *Practice in Text, Act in Pixels: Correcting Where Perception Fails*
([arXiv:2611.22722](https://arxiv.org/abs/2611.22722)) and its code at https://github.com/Carlxuzhl/practice-in-text.
GUI agents can practise cheaply on a text version of an interface and act on real screenshots through a screen reader
that turns each screenshot into the same kind of list. This LoRA *aligns* the reader Qwen3-VL-4B-Instruct to that list:
for a MobileGym screenshot it writes the screen's text view, one line per visible element in the layout order (top to
bottom, left to right within a row), in the line format the code's parser reads. Policies trained only in text can then
act on screenshots without being trained on pixels.
## Results (from the paper)
- Element error rate (missed elements, wrong text or wrong state) on 1,007 held-back MobileGym screenshots: 25.62% with
the original reader, 1.56% with this adapter.
- A policy that practised only in text, acting through this reader, completes 80.9% of the confirmation tasks, against
55.6% for a policy trained directly on the same budget of about 2,000 screenshots through the original reader.
## How to use it
Use it with the code repository, which holds the prompt, the parser and the settings it was trained for:
```bash
python -m reader.merge_lora --base <Qwen3-VL-4B-Instruct directory> --adapter <this adapter's directory> --out <merged directory>
bash scripts/with_servers.sh --gpu 0 --no-policy --reader <merged directory> -- <command>
```
The reader must be called exactly as the original one: the prompt of `reader/reader.py` (in Chinese, the apps'
language), greedy decoding, repetition penalty 1.05, screenshots at half size (`configs/reader.json`).
## Training
- LoRA rank 16, alpha 32, dropout 0.05, on the q/k/v/o/gate/up/down projections of the language model; the vision
encoder and its merger are frozen.
- 2,004 calibration screenshots of 29 MobileGym templates, each labelled with its true text view (2,003 used; one was
longer than 10,240 tokens). Learning rate 1e-4, 2 epochs, one example per step with 16 accumulated, cosine schedule
with 3% warmup, last checkpoint; 251 updates, 93 minutes on one RTX A6000. Details in `training_summary.json`.
## Scope and limits
- Aligned on MobileGym only (its 29 practised templates, simulated Chinese-language apps). Expect little benefit on
other apps or platforms without aligning again on their screenshots.
- It is tied to the prompt, the output format and the screenshot scale above.
## Licence
CC BY-NC 4.0: the adapter was trained on screenshots and labels generated from MobileGym, whose data is under
CC BY-NC 4.0 (academic, non-commercial use). The base model's own licence applies to the base weights (see the
Qwen3-VL-4B-Instruct model card). The code repository is under Apache-2.0.
## Citation
```bibtex
@article{xu2026practice,
title = {Practice in Text, Act in Pixels: Correcting Where Perception Fails},
author = {Xu, Zhongling and Wang, Hairong},
journal = {arXiv preprint arXiv:2611.22722},
year = {2026}
}
```