--- license: cc-by-nc-4.0 base_model: Qwen/Qwen3-VL-4B-Instruct library_name: peft pipeline_tag: image-text-to-text tags: - lora - gui-agent - screen-reader - mobilegym --- # Aligned screen reader for MobileGym (LoRA on Qwen3-VL-4B-Instruct) This adapter comes with the paper *Practice in Text, Act in Pixels: Correcting Where Perception Fails* ([arXiv:2611.22722](https://arxiv.org/abs/2611.22722)) and its code at https://github.com/Carlxuzhl/practice-in-text. GUI agents can practise cheaply on a text version of an interface and act on real screenshots through a screen reader that turns each screenshot into the same kind of list. This LoRA *aligns* the reader Qwen3-VL-4B-Instruct to that list: for a MobileGym screenshot it writes the screen's text view, one line per visible element in the layout order (top to bottom, left to right within a row), in the line format the code's parser reads. Policies trained only in text can then act on screenshots without being trained on pixels. ## Results (from the paper) - Element error rate (missed elements, wrong text or wrong state) on 1,007 held-back MobileGym screenshots: 25.62% with the original reader, 1.56% with this adapter. - A policy that practised only in text, acting through this reader, completes 80.9% of the confirmation tasks, against 55.6% for a policy trained directly on the same budget of about 2,000 screenshots through the original reader. ## How to use it Use it with the code repository, which holds the prompt, the parser and the settings it was trained for: ```bash python -m reader.merge_lora --base --adapter --out bash scripts/with_servers.sh --gpu 0 --no-policy --reader -- ``` The reader must be called exactly as the original one: the prompt of `reader/reader.py` (in Chinese, the apps' language), greedy decoding, repetition penalty 1.05, screenshots at half size (`configs/reader.json`). ## Training - LoRA rank 16, alpha 32, dropout 0.05, on the q/k/v/o/gate/up/down projections of the language model; the vision encoder and its merger are frozen. - 2,004 calibration screenshots of 29 MobileGym templates, each labelled with its true text view (2,003 used; one was longer than 10,240 tokens). Learning rate 1e-4, 2 epochs, one example per step with 16 accumulated, cosine schedule with 3% warmup, last checkpoint; 251 updates, 93 minutes on one RTX A6000. Details in `training_summary.json`. ## Scope and limits - Aligned on MobileGym only (its 29 practised templates, simulated Chinese-language apps). Expect little benefit on other apps or platforms without aligning again on their screenshots. - It is tied to the prompt, the output format and the screenshot scale above. ## Licence CC BY-NC 4.0: the adapter was trained on screenshots and labels generated from MobileGym, whose data is under CC BY-NC 4.0 (academic, non-commercial use). The base model's own licence applies to the base weights (see the Qwen3-VL-4B-Instruct model card). The code repository is under Apache-2.0. ## Citation ```bibtex @article{xu2026practice, title = {Practice in Text, Act in Pixels: Correcting Where Perception Fails}, author = {Xu, Zhongling and Wang, Hairong}, journal = {arXiv preprint arXiv:2611.22722}, year = {2026} } ```