| # Glint Decision Model PoC |
|
|
| Screenshot + natural-language questions + options → probability per option, computed in one forward pass |
| by a **Clef-style Joint Schema Head** on top of a **Qwen3.5 VLM backbone with LoRA** (no text generation). |
| The student is distilled from a teacher's option distributions (Clef / Clef-Flash) with a temperature KL loss, |
| plus cross-entropy (+ Brier) when a gold label exists. |
|
|
| > `glint/joint_schema_model.py` is Cloudflare's Clef-Flash code (Apache-2.0), see `NOTICE.md` and |
| > `LICENSE-CLEF-APACHE-2.0.txt`. The student head is freshly initialised (no Clef weights in the student). The released Clef-Flash weights |
| > (`Cloudflare/clef-flash`, Apache-2.0) are only run locally as the *teacher* by `scripts/label_local_clef.py`. |
|
|
| ## Layout |
|
|
| | Path | What | |
| | --- | --- | |
| | `glint/joint_schema_model.py` | Clef `JointSchemaHead`, `ClefModel`, `encode_record`, `collate_records`, `systemone_answer` (Apache-2.0, unmodified) | |
| | `glint/model.py` | build student (Qwen3.5 + LoRA + head), `max_pixels` control, save/load checkpoint | |
| | `glint/data.py` | JSONL loader, teacher normalisation (Workers AI `noul` = p(true) is expanded), augmentation | |
| | `glint/losses.py` | `α·T²·KL(teacher_T‖student_T) + (1−α)·CE_ls(gold) + β·Brier` (+ box loss, see below) | |
| | `glint/box.py` | Jev-style `box` questions: `BoxHead`, DFL bins, IoU/GIoU, box loss, metrics, answer formatting | |
| | `glint/schema.py` | `encode_glint_record`: Clef `encode_record` + box fields (identical output when there are none) | |
| | `glint/glint_model.py` | `GlintModel(ClefModel)`: joint head + optional box head on one backbone pass | |
| | `make_synthetic.py` | tiny PIL dataset of iPhone-sized dialog screenshots, ja/en mixed, **DUMMY teacher** | |
| | `train.py` | training loop (batch 1 + grad accumulation, cosine LR, JSON log lines) | |
| | `infer.py` | load checkpoint → JSON with per-option probabilities + latency | |
| | `bench.py` | train step time / inference latency / peak memory for several `max_pixels` | |
| | `scripts/evaluate.py` | accuracy, NLL, Brier, ECE(15 bins), teacher agreement | |
| | `scripts/label_workers_ai.py` | replace dummy teacher with Clef via Workers AI (**billable**, dry-run by default) | |
| | `scripts/label_local_clef.py` | replace dummy teacher with locally-run Clef-Flash (open weights, bf16 on the iGPU, no cost); stores probs + logits + latency/memory stats | |
| | `scripts/teacher_report.py` | teacher-label accuracy / NLL / ECE vs gold, per-question accuracy, errors, example distributions | |
| | `tests/` | CPU unit tests (no model download): `python -m pytest tests` | |
| | `RESULTS.md` | measured numbers on the Windows + ROCm PC | |
|
|
| ## Environment (Windows + ROCm, AMD Ryzen AI 9 HX 370 iGPU) |
|
|
| ```powershell |
| $env:HF_HOME = "<HF_CACHE_DIR>" |
| $py = "<VENV>\Scripts\python.exe" |
| cd <REPO_DIR> |
| $env:HF_HUB_OFFLINE = "1" # models are already cached |
| # TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 is set by default inside the scripts (faster SDPA, see RESULTS.md). |
| # To disable: $env:TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL = "0" |
| ``` |
|
|
| Never `pip install` anything that pulls in `torch` into `venv-rocm` (it would replace the ROCm build). |
| Use `pip install --no-deps <pkg>`. No extra packages are required beyond what the venv already has |
| (`transformers 5.20.2`, `peft 0.21.2`, `safetensors`, `pillow`, `requests`). |
|
|
| ## Quick start |
|
|
| ```powershell |
| & $py make_synthetic.py --out data\synthetic --n-train 60 --n-eval 20 |
| & $py train.py --data data\synthetic\train.jsonl --out checkpoints\syn-2b --model Qwen/Qwen3.5-2B --max-pixels 430080 --steps 200 |
| & $py infer.py --ckpt checkpoints\syn-2b --jsonl data\synthetic\eval.jsonl --limit 3 |
| & $py scripts\evaluate.py --ckpt checkpoints\syn-2b --data data\synthetic\eval.jsonl |
| & $py bench.py --data data\synthetic\eval.jsonl --pixels 430080,1048576 |
| ``` |
|
|
| Single screenshot: |
|
|
| ```powershell |
| & $py infer.py --ckpt checkpoints\syn-2b --image shot.png --questions questions.json --state '{"app":"MyApp"}' |
| ``` |
|
|
| `questions.json` uses the Clef / SystemOne schema: |
|
|
| ```json |
| {"has_error": {"type": "noul", "instructions": "エラーダイアログが表示されているか？"}, |
| "screen": {"type": "choice", "instructions": "What is the current screen state?", |
| "criteria": {"error_dialog": "An error alert is shown", "home": "Normal home screen", "loading": "Loading"}}, |
| "severity": {"type": "score", "instructions": "How severe?", "criteria": ["none", "minor", "major"]}} |
| ``` |
|
|
| ### Image token budget (`--max-pixels`) |
|
|
| Qwen3.5 processor: patch 16, merge 2 → 1 token per 32×32 px after resizing to ≤ `max_pixels`. |
| For a 1179×2556 iPhone portrait screenshot: `430080` → 390 tokens, `1048576` → 987 tokens, `244608` → 220 tokens. |
| The value is stored in the checkpoint (`glint_config.json`) and can be overridden at inference with `--max-pixels`. |
|
|
| ### Main training options |
|
|
| `--temperature 2.0` (KD temperature), `--alpha 0.7` (KD weight when both teacher and gold exist; teacher-only → pure KD, |
| gold-only → pure CE), `--beta 0.1` (Brier), `--lora-r 32`, `--lr-lora 2e-4`, `--lr-head 5e-4`, `--accum 1`, |
| `--grad-ckpt` (gradient checkpointing for larger images), `--dtype bf16|fp16|fp32`. |
|
|
| LoRA targets only the language model (`q/k/v/o_proj`, MLP, and Gated-DeltaNet `in_proj_qkv/in_proj_z/out_proj`); |
| the vision encoder is frozen. |
|
|
| ## Data format (JSONL, one record per line) |
|
|
| ```json |
| {"id": "syn-000001", |
| "images": ["images/syn-000001.png"], |
| "state": {"app": "SampleApp", "locale": "ja-JP", "a11y": ["Alert: 通信エラー", "Button: OK"]}, |
| "questions": {"screen": {"type": "choice", "instructions": "現在の画面状態は？", |
| "criteria": {"home": "ホーム画面", "error_dialog": "エラーダイアログ", "loading": "読み込み中"}}, |
| "has_error": {"type": "noul", "instructions": "エラーメッセージが見えているか？"}}, |
| "teacher": {"screen": {"error_dialog": 0.91, "home": 0.06, "loading": 0.03}, "has_error": 0.97}, |
| "gold": {"screen": "error_dialog", "has_error": "true"}, |
| "meta": {"source": "synthetic", "teacher_model": "@cf/cloudflare/clef-flash", "split_group": "SampleApp"}} |
| ``` |
|
|
| * `images` paths are relative to the JSONL file. `state` may be any string/JSON (put a11y tree / OCR text here). |
| * `teacher[qid]` is `{option_id: prob}`; for `noul` a bare float p(true) is accepted (Workers AI response shape). |
| SystemOne answer objects (`{"probabilities": ...}` / `{"noul": p}`) are accepted too. |
| * `gold` is optional; `teacher` is optional. Option ids for `noul` are `"true"/"false"`, for `score` `"0","1",...`. |
| * **The synthetic data's `teacher` is a DUMMY** (gold + noise, `meta.teacher_is_dummy=true`). |
|
|
| ## Box questions |
|
|
| Like Jev 1.14, a question of type `box` asks for one rectangle in the (first) screenshot: |
|
|
| ```json |
| {"primary_button_box": {"type": "box", "instructions": "the primary button in `screen`"}, |
| "primary_button_box_visible": {"type": "noul", "instructions": "Is the primary button visible?"}} |
| ``` |
|
|
| Answer (`infer.py`), edges as fractions of image height (`top`/`bottom`) and width (`left`/`right`) from the top-left, |
| always `top < bottom`, `left < right`: |
|
|
| ```json |
| {"primary_button_box": {"type": "box", "box": {"top": 0.534, "bottom": 0.589, "left": 0.500, "right": 0.859}, "confidence": 0.83}} |
| ``` |
|
|
| There is no "not found" answer: ask presence as a separate `noul` question (convention: `<qid>_visible`). |
|
|
| * **Data.** Box gold comes from ground truth (capture / annotation), never from the teacher; `teacher` entries for box |
| questions are ignored and the label scripts do not send box questions to Clef. |
| `gold[qid] = {"top": .., "bottom": .., "left": .., "right": .., "present": true}` (0–1; `{"box": {...}}` is accepted too; |
| inverted edges are swapped, values clamped). `{"present": false}` skips the box loss for that record, and if a |
| `<qid>_visible` noul question exists without gold it gets `"false"` (`"true"` when present). |
| * **Head.** `BoxHead` reads the same backbone hidden states as the joint head: the question span is pooled and 4 edge |
| queries (top/bottom/left/right) cross-attend to the record's tokens; image tokens get a 2D Fourier encoding of their |
| patch centre. Each edge outputs logits over `--box-bins 64` bins (DFL/GFL style); the edge is the expectation over |
| bin centres `i/(K−1)`, then top/bottom and left/right are sorted. `confidence` is a separate sigmoid head trained to |
| predict the IoU of its own box (BCE against the detached IoU); `evaluate.py` also reports the bin max-prob alternative. |
| * **Loss.** `box = dfl·DFL + l1·L1 + giou·(1−GIoU) + conf·BCE(conf, IoU)` (`--box-dfl 1 --box-l1 5 --box-giou 2 --box-conf 1`), |
| added to the total as `--box-weight 1.0 · box`. `--box-head auto` (default) builds the head only when the training data |
| contains box questions; `on`/`off` force it. Without box questions nothing changes (same encoding, losses, logs, checkpoint). |
| * **Eval.** `scripts/evaluate.py` adds a `box` block: `mean_iou`, `iou_ge_0.5`, `iou_ge_0.75`, `mean_confidence`, |
| `ece_15bin_conf_vs_iou50` (confidence vs. IoU≥0.5) and `ece_15bin_maxprob_vs_iou50`. |
| `bench.py` builds the box head (and adds the box loss) only when its data contains box questions. |
| * **Synthetic.** `make_synthetic.py --box` adds `primary_button_box` + `primary_button_box_visible` with gold from the drawn |
| button rectangle (absent on loading/home). Without `--box` the output is byte-identical to before. |
|
|
| ```powershell |
| & $py make_synthetic.py --out data\synthetic-box --n-train 60 --n-eval 20 --box |
| & $py train.py --data data\synthetic-box\train.jsonl --out checkpoints\syn-box-2b --steps 200 |
| & $py scripts\evaluate.py --ckpt checkpoints\syn-box-2b --data data\synthetic-box\eval.jsonl |
| ``` |
|
|
| ## Swapping in real teacher labels |
|
|
| ```powershell |
| # dry run (no requests, no cost): prints request sizes |
| & $py scripts\label_workers_ai.py data\synthetic\train.jsonl data\synthetic\train.clef.jsonl --model clef-flash |
| # real (billable) run |
| $env:CLOUDFLARE_ACCOUNT_ID = "..."; $env:CLOUDFLARE_API_TOKEN = "..." |
| & $py scripts\label_workers_ai.py data\real\train.jsonl data\real\train.clef.jsonl --model clef-flash --run |
| ``` |
|
|
| Image files are re-encoded to JPEG ≤ ~190 KB and sent as base64 data URLs (max 4 images per request). |
| Alternatively run the open-weight teacher locally (tested on the Windows + ROCm PC, see RESULTS.md; ~3.1 s/image at 390 image tokens, 18.0 GB peak): |
|
|
| ```powershell |
| $env:HF_HUB_OFFLINE = "0" # first run only: downloads Cloudflare/clef-flash (~19.1 GB) into HF_HOME |
| & $py scripts\label_local_clef.py data\synthetic\train.jsonl data\synthetic\train.clef.jsonl --max-pixels 430080 --stats runs\clef-label-train-390.json |
| & $py scripts\label_local_clef.py data\synthetic\eval.jsonl data\synthetic\eval.clef.jsonl --max-pixels 430080 |
| & $py scripts\teacher_report.py data\synthetic\train.clef.jsonl data\synthetic\eval.clef.jsonl |
| & $py train.py --data data\synthetic\train.clef.jsonl --out checkpoints\clef-2b-390 --steps 200 --max-pixels 430080 |
| & $py scripts\evaluate.py --ckpt checkpoints\clef-2b-390 --data data\synthetic\eval.clef.jsonl |
| ``` |
|
|
| Output records get `teacher` (`{qid: {option: prob}}`), `teacher_logits`, and `meta.teacher_model/teacher_max_pixels/teacher_image_tokens`, |
| with `meta.teacher_is_dummy=false`. Options: `--limit N` (smoke test), `--max-pixels` (Clef-Flash release default is |
| `longest_edge=16777216`, i.e. effectively native resolution, ~2.9k tokens for 1179x2556), `--offload` (accelerate |
| `device_map=auto` CPU-offload fallback; not needed on that PC, untested). |
|
|
| ## Checkpoint layout |
|
|
| ``` |
| checkpoints/<run>/adapter/ # PEFT LoRA adapter (adapter_config.json, adapter_model.safetensors) |
| checkpoints/<run>/head.safetensors # JointSchemaHead weights |
| checkpoints/<run>/box_head.safetensors # BoxHead weights (only when trained with box questions) |
| checkpoints/<run>/glint_config.json # base model, max_pixels, dtype, head/LoRA config, train args, loss history |
| ``` |
|
|
| `infer.py` merges LoRA into the backbone by default (`--no-merge` to keep it separate). |
|
|
| ## Known limitations (Windows + ROCm) |
|
|
| * `flash-linear-attention` needs Triton; `triton-windows` targets NVIDIA only, so the Gated-DeltaNet layers run the reference PyTorch path (slow). Not installed. |
| * Training is batch 1 (+ `--accum`); `collate_records` supports batching but is untested here. |
| * Synthetic data ships with a DUMMY teacher (`train.jsonl`/`eval.jsonl`); real Clef-Flash labels are in `*.clef.jsonl` |
| (generated by `scripts/label_local_clef.py`; `data/` is git-ignored, so regenerate them on another machine). |
| * The synthetic task is easy: Clef-Flash is 100% correct on it, so it does not yet tell us much about real iOS screens. |
|
|
