Instructions to use xuzhl/practice-in-text-aligned-reader with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use xuzhl/practice-in-text-aligned-reader with PEFT:
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-VL-4B-Instruct") model = PeftModel.from_pretrained(base_model, "xuzhl/practice-in-text-aligned-reader") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from xuzhl/practice-in-text-aligned-reader: direct link, hf CLI and curl.
- Browser
- Download file 3.36 kB
-
https://huggingface.co/xuzhl/practice-in-text-aligned-reader/resolve/main/README.md
- Command line
-
hf download hf://xuzhl/practice-in-text-aligned-reader/README.md
-
curl -L -o README.md https://huggingface.co/xuzhl/practice-in-text-aligned-reader/resolve/main/README.md
3.36 kB
| license: cc-by-nc-4.0 | |
| base_model: Qwen/Qwen3-VL-4B-Instruct | |
| library_name: peft | |
| pipeline_tag: image-text-to-text | |
| tags: | |
| - lora | |
| - gui-agent | |
| - screen-reader | |
| - mobilegym | |
| # Aligned screen reader for MobileGym (LoRA on Qwen3-VL-4B-Instruct) | |
| This adapter comes with the paper *Practice in Text, Act in Pixels: Correcting Where Perception Fails* | |
| ([arXiv:2611.22722](https://arxiv.org/abs/2611.22722)) and its code at https://github.com/Carlxuzhl/practice-in-text. | |
| GUI agents can practise cheaply on a text version of an interface and act on real screenshots through a screen reader | |
| that turns each screenshot into the same kind of list. This LoRA *aligns* the reader Qwen3-VL-4B-Instruct to that list: | |
| for a MobileGym screenshot it writes the screen's text view, one line per visible element in the layout order (top to | |
| bottom, left to right within a row), in the line format the code's parser reads. Policies trained only in text can then | |
| act on screenshots without being trained on pixels. | |
| ## Results (from the paper) | |
| - Element error rate (missed elements, wrong text or wrong state) on 1,007 held-back MobileGym screenshots: 25.62% with | |
| the original reader, 1.56% with this adapter. | |
| - A policy that practised only in text, acting through this reader, completes 80.9% of the confirmation tasks, against | |
| 55.6% for a policy trained directly on the same budget of about 2,000 screenshots through the original reader. | |
| ## How to use it | |
| Use it with the code repository, which holds the prompt, the parser and the settings it was trained for: | |
| ```bash | |
| python -m reader.merge_lora --base <Qwen3-VL-4B-Instruct directory> --adapter <this adapter's directory> --out <merged directory> | |
| bash scripts/with_servers.sh --gpu 0 --no-policy --reader <merged directory> -- <command> | |
| ``` | |
| The reader must be called exactly as the original one: the prompt of `reader/reader.py` (in Chinese, the apps' | |
| language), greedy decoding, repetition penalty 1.05, screenshots at half size (`configs/reader.json`). | |
| ## Training | |
| - LoRA rank 16, alpha 32, dropout 0.05, on the q/k/v/o/gate/up/down projections of the language model; the vision | |
| encoder and its merger are frozen. | |
| - 2,004 calibration screenshots of 29 MobileGym templates, each labelled with its true text view (2,003 used; one was | |
| longer than 10,240 tokens). Learning rate 1e-4, 2 epochs, one example per step with 16 accumulated, cosine schedule | |
| with 3% warmup, last checkpoint; 251 updates, 93 minutes on one RTX A6000. Details in `training_summary.json`. | |
| ## Scope and limits | |
| - Aligned on MobileGym only (its 29 practised templates, simulated Chinese-language apps). Expect little benefit on | |
| other apps or platforms without aligning again on their screenshots. | |
| - It is tied to the prompt, the output format and the screenshot scale above. | |
| ## Licence | |
| CC BY-NC 4.0: the adapter was trained on screenshots and labels generated from MobileGym, whose data is under | |
| CC BY-NC 4.0 (academic, non-commercial use). The base model's own licence applies to the base weights (see the | |
| Qwen3-VL-4B-Instruct model card). The code repository is under Apache-2.0. | |
| ## Citation | |
| ```bibtex | |
| @article{xu2026practice, | |
| title = {Practice in Text, Act in Pixels: Correcting Where Perception Fails}, | |
| author = {Xu, Zhongling and Wang, Hairong}, | |
| journal = {arXiv preprint arXiv:2611.22722}, | |
| year = {2026} | |
| } | |
| ``` | |