code/README.md

Glint Decision Model PoC

Screenshot + natural-language questions + options → probability per option, computed in one forward pass by a Clef-style Joint Schema Head on top of a Qwen3.5 VLM backbone with LoRA (no text generation). The student is distilled from a teacher's option distributions (Clef / Clef-Flash) with a temperature KL loss, plus cross-entropy (+ Brier) when a gold label exists.

glint/joint_schema_model.py is Cloudflare's Clef-Flash code (Apache-2.0), see NOTICE.md and LICENSE-CLEF-APACHE-2.0.txt. The student head is freshly initialised (no Clef weights in the student). The released Clef-Flash weights (Cloudflare/clef-flash, Apache-2.0) are only run locally as the teacher by scripts/label_local_clef.py.

Layout

Path	What
glint/joint_schema_model.py	Clef JointSchemaHead, ClefModel, encode_record, collate_records, systemone_answer (Apache-2.0, unmodified)
glint/model.py	build student (Qwen3.5 + LoRA + head), max_pixels control, save/load checkpoint
glint/data.py	JSONL loader, teacher normalisation (Workers AI noul = p(true) is expanded), augmentation
glint/losses.py	α·T²·KL(teacher_T‖student_T) + (1−α)·CE_ls(gold) + β·Brier (+ box loss, see below)
glint/box.py	Jev-style box questions: BoxHead, DFL bins, IoU/GIoU, box loss, metrics, answer formatting
glint/schema.py	encode_glint_record: Clef encode_record + box fields (identical output when there are none)
glint/glint_model.py	GlintModel(ClefModel): joint head + optional box head on one backbone pass
make_synthetic.py	tiny PIL dataset of iPhone-sized dialog screenshots, ja/en mixed, DUMMY teacher
train.py	training loop (batch 1 + grad accumulation, cosine LR, JSON log lines)
infer.py	load checkpoint → JSON with per-option probabilities + latency
bench.py	train step time / inference latency / peak memory for several max_pixels
scripts/evaluate.py	accuracy, NLL, Brier, ECE(15 bins), teacher agreement
scripts/label_workers_ai.py	replace dummy teacher with Clef via Workers AI (billable, dry-run by default)
scripts/label_local_clef.py	replace dummy teacher with locally-run Clef-Flash (open weights, bf16 on the iGPU, no cost); stores probs + logits + latency/memory stats
scripts/teacher_report.py	teacher-label accuracy / NLL / ECE vs gold, per-question accuracy, errors, example distributions
tests/	CPU unit tests (no model download): python -m pytest tests
RESULTS.md	measured numbers on the Windows + ROCm PC

Environment (Windows + ROCm, AMD Ryzen AI 9 HX 370 iGPU)

$env:HF_HOME = "<HF_CACHE_DIR>"
$py = "<VENV>\Scripts\python.exe"
cd <REPO_DIR>
$env:HF_HUB_OFFLINE = "1"   # models are already cached
# TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 is set by default inside the scripts (faster SDPA, see RESULTS.md).
# To disable: $env:TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL = "0"

Never pip install anything that pulls in torch into venv-rocm (it would replace the ROCm build). Use pip install --no-deps <pkg>. No extra packages are required beyond what the venv already has (transformers 5.20.2, peft 0.21.2, safetensors, pillow, requests).

Quick start

& $py make_synthetic.py --out data\synthetic --n-train 60 --n-eval 20
& $py train.py --data data\synthetic\train.jsonl --out checkpoints\syn-2b --model Qwen/Qwen3.5-2B --max-pixels 430080 --steps 200
& $py infer.py --ckpt checkpoints\syn-2b --jsonl data\synthetic\eval.jsonl --limit 3
& $py scripts\evaluate.py --ckpt checkpoints\syn-2b --data data\synthetic\eval.jsonl
& $py bench.py --data data\synthetic\eval.jsonl --pixels 430080,1048576

Single screenshot:

& $py infer.py --ckpt checkpoints\syn-2b --image shot.png --questions questions.json --state '{"app":"MyApp"}'

questions.json uses the Clef / SystemOne schema:

{"has_error": {"type": "noul", "instructions": "エラーダイアログが表示されているか？"},
 "screen": {"type": "choice", "instructions": "What is the current screen state?",
            "criteria": {"error_dialog": "An error alert is shown", "home": "Normal home screen", "loading": "Loading"}},
 "severity": {"type": "score", "instructions": "How severe?", "criteria": ["none", "minor", "major"]}}

Image token budget (--max-pixels)

Qwen3.5 processor: patch 16, merge 2 → 1 token per 32×32 px after resizing to ≤ max_pixels. For a 1179×2556 iPhone portrait screenshot: 430080 → 390 tokens, 1048576 → 987 tokens, 244608 → 220 tokens. The value is stored in the checkpoint (glint_config.json) and can be overridden at inference with --max-pixels.

Main training options

--temperature 2.0 (KD temperature), --alpha 0.7 (KD weight when both teacher and gold exist; teacher-only → pure KD, gold-only → pure CE), --beta 0.1 (Brier), --lora-r 32, --lr-lora 2e-4, --lr-head 5e-4, --accum 1, --grad-ckpt (gradient checkpointing for larger images), --dtype bf16|fp16|fp32.

LoRA targets only the language model (q/k/v/o_proj, MLP, and Gated-DeltaNet in_proj_qkv/in_proj_z/out_proj); the vision encoder is frozen.

Data format (JSONL, one record per line)

{"id": "syn-000001",
 "images": ["images/syn-000001.png"],
 "state": {"app": "SampleApp", "locale": "ja-JP", "a11y": ["Alert: 通信エラー", "Button: OK"]},
 "questions": {"screen": {"type": "choice", "instructions": "現在の画面状態は？",
                          "criteria": {"home": "ホーム画面", "error_dialog": "エラーダイアログ", "loading": "読み込み中"}},
               "has_error": {"type": "noul", "instructions": "エラーメッセージが見えているか？"}},
 "teacher": {"screen": {"error_dialog": 0.91, "home": 0.06, "loading": 0.03}, "has_error": 0.97},
 "gold": {"screen": "error_dialog", "has_error": "true"},
 "meta": {"source": "synthetic", "teacher_model": "@cf/cloudflare/clef-flash", "split_group": "SampleApp"}}

• images paths are relative to the JSONL file. state may be any string/JSON (put a11y tree / OCR text here).
• teacher[qid] is {option_id: prob}; for noul a bare float p(true) is accepted (Workers AI response shape). SystemOne answer objects ({"probabilities": ...} / {"noul": p}) are accepted too.
• gold is optional; teacher is optional. Option ids for noul are "true"/"false", for score "0","1",....
• The synthetic data's teacher is a DUMMY (gold + noise, meta.teacher_is_dummy=true).

Box questions

Like Jev 1.14, a question of type box asks for one rectangle in the (first) screenshot:

{"primary_button_box": {"type": "box", "instructions": "the primary button in `screen`"},
 "primary_button_box_visible": {"type": "noul", "instructions": "Is the primary button visible?"}}

Answer (infer.py), edges as fractions of image height (top/bottom) and width (left/right) from the top-left, always top < bottom, left < right:

{"primary_button_box": {"type": "box", "box": {"top": 0.534, "bottom": 0.589, "left": 0.500, "right": 0.859}, "confidence": 0.83}}

There is no "not found" answer: ask presence as a separate noul question (convention: <qid>_visible).

• Data. Box gold comes from ground truth (capture / annotation), never from the teacher; teacher entries for box questions are ignored and the label scripts do not send box questions to Clef. gold[qid] = {"top": .., "bottom": .., "left": .., "right": .., "present": true} (0–1; {"box": {...}} is accepted too; inverted edges are swapped, values clamped). {"present": false} skips the box loss for that record, and if a <qid>_visible noul question exists without gold it gets "false" ("true" when present).
• Head. BoxHead reads the same backbone hidden states as the joint head: the question span is pooled and 4 edge queries (top/bottom/left/right) cross-attend to the record's tokens; image tokens get a 2D Fourier encoding of their patch centre. Each edge outputs logits over --box-bins 64 bins (DFL/GFL style); the edge is the expectation over bin centres i/(K−1), then top/bottom and left/right are sorted. confidence is a separate sigmoid head trained to predict the IoU of its own box (BCE against the detached IoU); evaluate.py also reports the bin max-prob alternative.
• Loss. box = dfl·DFL + l1·L1 + giou·(1−GIoU) + conf·BCE(conf, IoU) (--box-dfl 1 --box-l1 5 --box-giou 2 --box-conf 1), added to the total as --box-weight 1.0 · box. --box-head auto (default) builds the head only when the training data contains box questions; on/off force it. Without box questions nothing changes (same encoding, losses, logs, checkpoint).
• Eval. scripts/evaluate.py adds a box block: mean_iou, iou_ge_0.5, iou_ge_0.75, mean_confidence, ece_15bin_conf_vs_iou50 (confidence vs. IoU≥0.5) and ece_15bin_maxprob_vs_iou50. bench.py builds the box head (and adds the box loss) only when its data contains box questions.
• Synthetic. make_synthetic.py --box adds primary_button_box + primary_button_box_visible with gold from the drawn button rectangle (absent on loading/home). Without --box the output is byte-identical to before.

& $py make_synthetic.py --out data\synthetic-box --n-train 60 --n-eval 20 --box
& $py train.py --data data\synthetic-box\train.jsonl --out checkpoints\syn-box-2b --steps 200
& $py scripts\evaluate.py --ckpt checkpoints\syn-box-2b --data data\synthetic-box\eval.jsonl

Swapping in real teacher labels

# dry run (no requests, no cost): prints request sizes
& $py scripts\label_workers_ai.py data\synthetic\train.jsonl data\synthetic\train.clef.jsonl --model clef-flash
# real (billable) run
$env:CLOUDFLARE_ACCOUNT_ID = "..."; $env:CLOUDFLARE_API_TOKEN = "..."
& $py scripts\label_workers_ai.py data\real\train.jsonl data\real\train.clef.jsonl --model clef-flash --run

Image files are re-encoded to JPEG ≤ ~190 KB and sent as base64 data URLs (max 4 images per request). Alternatively run the open-weight teacher locally (tested on the Windows + ROCm PC, see RESULTS.md; ~3.1 s/image at 390 image tokens, 18.0 GB peak):

$env:HF_HUB_OFFLINE = "0"   # first run only: downloads Cloudflare/clef-flash (~19.1 GB) into HF_HOME
& $py scripts\label_local_clef.py data\synthetic\train.jsonl data\synthetic\train.clef.jsonl --max-pixels 430080 --stats runs\clef-label-train-390.json
& $py scripts\label_local_clef.py data\synthetic\eval.jsonl  data\synthetic\eval.clef.jsonl  --max-pixels 430080
& $py scripts\teacher_report.py data\synthetic\train.clef.jsonl data\synthetic\eval.clef.jsonl
& $py train.py --data data\synthetic\train.clef.jsonl --out checkpoints\clef-2b-390 --steps 200 --max-pixels 430080
& $py scripts\evaluate.py --ckpt checkpoints\clef-2b-390 --data data\synthetic\eval.clef.jsonl

Output records get teacher ({qid: {option: prob}}), teacher_logits, and meta.teacher_model/teacher_max_pixels/teacher_image_tokens, with meta.teacher_is_dummy=false. Options: --limit N (smoke test), --max-pixels (Clef-Flash release default is longest_edge=16777216, i.e. effectively native resolution, ~2.9k tokens for 1179x2556), --offload (accelerate device_map=auto CPU-offload fallback; not needed on that PC, untested).

Checkpoint layout

checkpoints/<run>/adapter/            # PEFT LoRA adapter (adapter_config.json, adapter_model.safetensors)
checkpoints/<run>/head.safetensors    # JointSchemaHead weights
checkpoints/<run>/box_head.safetensors  # BoxHead weights (only when trained with box questions)
checkpoints/<run>/glint_config.json   # base model, max_pixels, dtype, head/LoRA config, train args, loss history

infer.py merges LoRA into the backbone by default (--no-merge to keep it separate).

Known limitations (Windows + ROCm)

• flash-linear-attention needs Triton; triton-windows targets NVIDIA only, so the Gated-DeltaNet layers run the reference PyTorch path (slow). Not installed.
• Training is batch 1 (+ --accum); collate_records supports batching but is untested here.
• Synthetic data ships with a DUMMY teacher (train.jsonl/eval.jsonl); real Clef-Flash labels are in *.clef.jsonl (generated by scripts/label_local_clef.py; data/ is git-ignored, so regenerate them on another machine).
• The synthetic task is easy: Clef-Flash is 100% correct on it, so it does not yet tell us much about real iOS screens.