ui-test-vision-2b-alpha (checkpoint q3s-sv2ft-2b-390)

A small screenshot-understanding model for iOS UI testing. Given an app screenshot plus a set of questions with fixed options, it returns a probability per option in a single forward pass (no text generation). For element questions it can also return a box (top, bottom, left, right as 0–1 fractions of the image height/width) with a confidence; whether the element exists is asked as a separate yes/no question.

Status: alpha (v0.1.0-alpha). The question schema and weights may change without notice. Do not treat this as a finished or general-purpose vision model.

• Backbone: Qwen/Qwen3.5-2B + LoRA (r=32, alpha=64)
• Heads: Clef-style JointSchemaHead (option probabilities) and a BoxHead (64 DFL bins, L1 + GIoU, IoU-predicting confidence)
• Teacher: Cloudflare/clef-flash, run locally; distilled with temperature KL + cross-entropy on gold labels + Brier. Box targets come from element coordinates exported at capture time (no teacher).
• Image budget: max_pixels=430080 (~390 image tokens). This is the recommended setting. A lower pixel cap speeds up inference but hurts box placement more than classification.

Intended use

This model is meant to help decide what is on an iOS screenshot during automated UI testing: screen type, whether common chrome or states are present (tab bar, modal, loading, error, empty state, and so on), which control is primary or focused, and roughly where that element is.

Use it with the trained question schema and short, fixed-option prompts. Read the option probabilities and boxes; do not prompt it as a chatbot and do not expect free-form text.

It is not intended as a general visual assistant, a document or photo captioner, a web-UI model, an accessibility scanner, or an agent that calls tools or browses the web. It does not generate actions or code.

Training data

Training and the numbers below use synthetic iOS screenshots from our own SwiftUI gallery app, including a style-varied set with randomized visual design. Classification labels were distilled from a local Clef-Flash teacher (plus gold labels). Box labels are the view coordinates recorded when each screenshot was captured.

This release does not include the screenshot dataset. The checkpoint was not trained or evaluated on real third-party production apps.

Contents

Path	What
pytorch/adapter/	LoRA adapter (PEFT) for Qwen3.5-2B
pytorch/head.safetensors	JointSchemaHead (option probabilities)
pytorch/box_head.safetensors	BoxHead
pytorch/glint_config.json	head / LoRA / training config and training history
code/	PyTorch training and inference code (infer.py, glint/)
mlx/bf16/	merged model for Apple Silicon (mlx-vlm, bf16, ~4.5 GB weights)
mlx/4bit/	merged model for Apple Silicon (mlx-vlm, 4-bit, group 64, ~1.8 GB weights)
mlx/glint_mlx_infer.py, mlx/merge_for_mlx.py	MLX inference and the LoRA-merge script

The MLX models use the clef model type in mlx-vlm.

Evaluation (synthetic gallery screenshots)

Set	Record-exact	Box mean IoU	IoU ≥ 0.5	IoU ≥ 0.75	Box conf ECE
Gallery eval	98.1%	0.942	98.7%	94.2%	0.073
Style-varied eval	97.6%	0.830	92.1%	78.9%	0.093

Record-exact = every question on the screenshot answered correctly.

On the style-varied set, box quality varies a lot by element type. After the sv2ft fine-tune, approximate mean IoU by type was: selected tab 0.891, focused text field 0.790, error message 0.680, loading indicator 0.401. Loading indicators remain the weakest type because they occupy a very small area of the screen. Expect larger box errors on small or edge-of-screen elements (spinners, keyboards, compact error or empty-state regions). Most other boxes stay close to the higher-resolution result; a minority can shift substantially if the image is downscaled below the training pixel budget.

On a MacBook Air (M3, 24 GB) with MLX, 20 style-varied screenshots (180 questions): bf16 1.33 s/image (4.9 GB), 4-bit 1.37 s/image (2.4 GB); both gave the same argmax answer as the PyTorch model on all 180. The same 4-bit model at max_pixels=244608 (~220 image tokens) ran at 0.985 s/image on the same 20 screenshots, with 179/180 argmax agreement versus the 390-token run. Box edges moved more (median absolute edge difference 0.009, mean 0.042, max 0.586). Prefer ~390 image tokens when box location matters.

Limitations

• Synthetic training and eval only; accuracy on real apps with unfamiliar designs is unknown and likely lower.
• The question set is fixed to the schema used in training (screen type, primary button, loading, error, modal, focused field, selected tab, and so on). New question wordings or option lists are unsupported.
• Small elements, especially loading indicators, are boxed less accurately. Do not rely on a single box when the element is tiny or the confidence is low.
• This is an alpha checkpoint: both the question format and the weights may change.

License

Apache-2.0. code/glint/joint_schema_model.py is from Cloudflare's Clef-Flash release (Apache-2.0); see code/NOTICE.md. The Qwen3.5-2B base model is Apache-2.0.