# Glint Decision Model PoC

Screenshot + natural-language questions + options → probability per option, computed in one forward pass
by a **Clef-style Joint Schema Head** on top of a **Qwen3.5 VLM backbone with LoRA** (no text generation).
The student is distilled from a teacher's option distributions (Clef / Clef-Flash) with a temperature KL loss,
plus cross-entropy (+ Brier) when a gold label exists.

> `glint/joint_schema_model.py` is Cloudflare's Clef-Flash code (Apache-2.0), see `NOTICE.md` and
> `LICENSE-CLEF-APACHE-2.0.txt`. The student head is freshly initialised (no Clef weights in the student). The released Clef-Flash weights
> (`Cloudflare/clef-flash`, Apache-2.0) are only run locally as the *teacher* by `scripts/label_local_clef.py`.

## Layout

| Path | What |
| --- | --- |
| `glint/joint_schema_model.py` | Clef `JointSchemaHead`, `ClefModel`, `encode_record`, `collate_records`, `systemone_answer` (Apache-2.0, unmodified) |
| `glint/model.py` | build student (Qwen3.5 + LoRA + head), `max_pixels` control, save/load checkpoint |
| `glint/data.py` | JSONL loader, teacher normalisation (Workers AI `noul` = p(true) is expanded), augmentation |
| `glint/losses.py` | `α·T²·KL(teacher_T‖student_T) + (1−α)·CE_ls(gold) + β·Brier` (+ box loss, see below) |
| `glint/box.py` | Jev-style `box` questions: `BoxHead`, DFL bins, IoU/GIoU, box loss, metrics, answer formatting |
| `glint/schema.py` | `encode_glint_record`: Clef `encode_record` + box fields (identical output when there are none) |
| `glint/glint_model.py` | `GlintModel(ClefModel)`: joint head + optional box head on one backbone pass |
| `make_synthetic.py` | tiny PIL dataset of iPhone-sized dialog screenshots, ja/en mixed, **DUMMY teacher** |
| `train.py` | training loop (batch 1 + grad accumulation, cosine LR, JSON log lines) |
| `infer.py` | load checkpoint → JSON with per-option probabilities + latency |
| `bench.py` | train step time / inference latency / peak memory for several `max_pixels` |
| `scripts/evaluate.py` | accuracy, NLL, Brier, ECE(15 bins), teacher agreement |
| `scripts/label_workers_ai.py` | replace dummy teacher with Clef via Workers AI (**billable**, dry-run by default) |
| `scripts/label_local_clef.py` | replace dummy teacher with locally-run Clef-Flash (open weights, bf16 on the iGPU, no cost); stores probs + logits + latency/memory stats |
| `scripts/teacher_report.py` | teacher-label accuracy / NLL / ECE vs gold, per-question accuracy, errors, example distributions |
| `tests/` | CPU unit tests (no model download): `python -m pytest tests` |
| `RESULTS.md` | measured numbers on the Windows + ROCm PC |

## Environment (Windows + ROCm, AMD Ryzen AI 9 HX 370 iGPU)

```powershell
$env:HF_HOME = "<HF_CACHE_DIR>"
$py = "<VENV>\Scripts\python.exe"
cd <REPO_DIR>
$env:HF_HUB_OFFLINE = "1"   # models are already cached
# TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL=1 is set by default inside the scripts (faster SDPA, see RESULTS.md).
# To disable: $env:TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL = "0"
```

Never `pip install` anything that pulls in `torch` into `venv-rocm` (it would replace the ROCm build).
Use `pip install --no-deps <pkg>`. No extra packages are required beyond what the venv already has
(`transformers 5.20.2`, `peft 0.21.2`, `safetensors`, `pillow`, `requests`).

## Quick start

```powershell
& $py make_synthetic.py --out data\synthetic --n-train 60 --n-eval 20
& $py train.py --data data\synthetic\train.jsonl --out checkpoints\syn-2b --model Qwen/Qwen3.5-2B --max-pixels 430080 --steps 200
& $py infer.py --ckpt checkpoints\syn-2b --jsonl data\synthetic\eval.jsonl --limit 3
& $py scripts\evaluate.py --ckpt checkpoints\syn-2b --data data\synthetic\eval.jsonl
& $py bench.py --data data\synthetic\eval.jsonl --pixels 430080,1048576
```

Single screenshot:

```powershell
& $py infer.py --ckpt checkpoints\syn-2b --image shot.png --questions questions.json --state '{"app":"MyApp"}'
```

`questions.json` uses the Clef / SystemOne schema:

```json
{"has_error": {"type": "noul", "instructions": "エラーダイアログが表示されているか？"},
 "screen": {"type": "choice", "instructions": "What is the current screen state?",
            "criteria": {"error_dialog": "An error alert is shown", "home": "Normal home screen", "loading": "Loading"}},
 "severity": {"type": "score", "instructions": "How severe?", "criteria": ["none", "minor", "major"]}}
```

### Image token budget (`--max-pixels`)

Qwen3.5 processor: patch 16, merge 2 → 1 token per 32×32 px after resizing to ≤ `max_pixels`.
For a 1179×2556 iPhone portrait screenshot: `430080` → 390 tokens, `1048576` → 987 tokens, `244608` → 220 tokens.
The value is stored in the checkpoint (`glint_config.json`) and can be overridden at inference with `--max-pixels`.

### Main training options

`--temperature 2.0` (KD temperature), `--alpha 0.7` (KD weight when both teacher and gold exist; teacher-only → pure KD,
gold-only → pure CE), `--beta 0.1` (Brier), `--lora-r 32`, `--lr-lora 2e-4`, `--lr-head 5e-4`, `--accum 1`,
`--grad-ckpt` (gradient checkpointing for larger images), `--dtype bf16|fp16|fp32`.

LoRA targets only the language model (`q/k/v/o_proj`, MLP, and Gated-DeltaNet `in_proj_qkv/in_proj_z/out_proj`);
the vision encoder is frozen.

## Data format (JSONL, one record per line)

```json
{"id": "syn-000001",
 "images": ["images/syn-000001.png"],
 "state": {"app": "SampleApp", "locale": "ja-JP", "a11y": ["Alert: 通信エラー", "Button: OK"]},
 "questions": {"screen": {"type": "choice", "instructions": "現在の画面状態は？",
                          "criteria": {"home": "ホーム画面", "error_dialog": "エラーダイアログ", "loading": "読み込み中"}},
               "has_error": {"type": "noul", "instructions": "エラーメッセージが見えているか？"}},
 "teacher": {"screen": {"error_dialog": 0.91, "home": 0.06, "loading": 0.03}, "has_error": 0.97},
 "gold": {"screen": "error_dialog", "has_error": "true"},
 "meta": {"source": "synthetic", "teacher_model": "@cf/cloudflare/clef-flash", "split_group": "SampleApp"}}
```

* `images` paths are relative to the JSONL file. `state` may be any string/JSON (put a11y tree / OCR text here).
* `teacher[qid]` is `{option_id: prob}`; for `noul` a bare float p(true) is accepted (Workers AI response shape).
  SystemOne answer objects (`{"probabilities": ...}` / `{"noul": p}`) are accepted too.
* `gold` is optional; `teacher` is optional. Option ids for `noul` are `"true"/"false"`, for `score` `"0","1",...`.
* **The synthetic data's `teacher` is a DUMMY** (gold + noise, `meta.teacher_is_dummy=true`).

## Box questions

Like Jev 1.14, a question of type `box` asks for one rectangle in the (first) screenshot:

```json
{"primary_button_box": {"type": "box", "instructions": "the primary button in `screen`"},
 "primary_button_box_visible": {"type": "noul", "instructions": "Is the primary button visible?"}}
```

Answer (`infer.py`), edges as fractions of image height (`top`/`bottom`) and width (`left`/`right`) from the top-left,
always `top < bottom`, `left < right`:

```json
{"primary_button_box": {"type": "box", "box": {"top": 0.534, "bottom": 0.589, "left": 0.500, "right": 0.859}, "confidence": 0.83}}
```

There is no "not found" answer: ask presence as a separate `noul` question (convention: `<qid>_visible`).

* **Data.** Box gold comes from ground truth (capture / annotation), never from the teacher; `teacher` entries for box
  questions are ignored and the label scripts do not send box questions to Clef.
  `gold[qid] = {"top": .., "bottom": .., "left": .., "right": .., "present": true}` (0–1; `{"box": {...}}` is accepted too;
  inverted edges are swapped, values clamped). `{"present": false}` skips the box loss for that record, and if a
  `<qid>_visible` noul question exists without gold it gets `"false"` (`"true"` when present).
* **Head.** `BoxHead` reads the same backbone hidden states as the joint head: the question span is pooled and 4 edge
  queries (top/bottom/left/right) cross-attend to the record's tokens; image tokens get a 2D Fourier encoding of their
  patch centre. Each edge outputs logits over `--box-bins 64` bins (DFL/GFL style); the edge is the expectation over
  bin centres `i/(K−1)`, then top/bottom and left/right are sorted. `confidence` is a separate sigmoid head trained to
  predict the IoU of its own box (BCE against the detached IoU); `evaluate.py` also reports the bin max-prob alternative.
* **Loss.** `box = dfl·DFL + l1·L1 + giou·(1−GIoU) + conf·BCE(conf, IoU)` (`--box-dfl 1 --box-l1 5 --box-giou 2 --box-conf 1`),
  added to the total as `--box-weight 1.0 · box`. `--box-head auto` (default) builds the head only when the training data
  contains box questions; `on`/`off` force it. Without box questions nothing changes (same encoding, losses, logs, checkpoint).
* **Eval.** `scripts/evaluate.py` adds a `box` block: `mean_iou`, `iou_ge_0.5`, `iou_ge_0.75`, `mean_confidence`,
  `ece_15bin_conf_vs_iou50` (confidence vs. IoU≥0.5) and `ece_15bin_maxprob_vs_iou50`.
  `bench.py` builds the box head (and adds the box loss) only when its data contains box questions.
* **Synthetic.** `make_synthetic.py --box` adds `primary_button_box` + `primary_button_box_visible` with gold from the drawn
  button rectangle (absent on loading/home). Without `--box` the output is byte-identical to before.

```powershell
& $py make_synthetic.py --out data\synthetic-box --n-train 60 --n-eval 20 --box
& $py train.py --data data\synthetic-box\train.jsonl --out checkpoints\syn-box-2b --steps 200
& $py scripts\evaluate.py --ckpt checkpoints\syn-box-2b --data data\synthetic-box\eval.jsonl
```

## Swapping in real teacher labels

```powershell
# dry run (no requests, no cost): prints request sizes
& $py scripts\label_workers_ai.py data\synthetic\train.jsonl data\synthetic\train.clef.jsonl --model clef-flash
# real (billable) run
$env:CLOUDFLARE_ACCOUNT_ID = "..."; $env:CLOUDFLARE_API_TOKEN = "..."
& $py scripts\label_workers_ai.py data\real\train.jsonl data\real\train.clef.jsonl --model clef-flash --run
```

Image files are re-encoded to JPEG ≤ ~190 KB and sent as base64 data URLs (max 4 images per request).
Alternatively run the open-weight teacher locally (tested on the Windows + ROCm PC, see RESULTS.md; ~3.1 s/image at 390 image tokens, 18.0 GB peak):

```powershell
$env:HF_HUB_OFFLINE = "0"   # first run only: downloads Cloudflare/clef-flash (~19.1 GB) into HF_HOME
& $py scripts\label_local_clef.py data\synthetic\train.jsonl data\synthetic\train.clef.jsonl --max-pixels 430080 --stats runs\clef-label-train-390.json
& $py scripts\label_local_clef.py data\synthetic\eval.jsonl  data\synthetic\eval.clef.jsonl  --max-pixels 430080
& $py scripts\teacher_report.py data\synthetic\train.clef.jsonl data\synthetic\eval.clef.jsonl
& $py train.py --data data\synthetic\train.clef.jsonl --out checkpoints\clef-2b-390 --steps 200 --max-pixels 430080
& $py scripts\evaluate.py --ckpt checkpoints\clef-2b-390 --data data\synthetic\eval.clef.jsonl
```

Output records get `teacher` (`{qid: {option: prob}}`), `teacher_logits`, and `meta.teacher_model/teacher_max_pixels/teacher_image_tokens`,
with `meta.teacher_is_dummy=false`. Options: `--limit N` (smoke test), `--max-pixels` (Clef-Flash release default is
`longest_edge=16777216`, i.e. effectively native resolution, ~2.9k tokens for 1179x2556), `--offload` (accelerate
`device_map=auto` CPU-offload fallback; not needed on that PC, untested).

## Checkpoint layout

```
checkpoints/<run>/adapter/            # PEFT LoRA adapter (adapter_config.json, adapter_model.safetensors)
checkpoints/<run>/head.safetensors    # JointSchemaHead weights
checkpoints/<run>/box_head.safetensors  # BoxHead weights (only when trained with box questions)
checkpoints/<run>/glint_config.json   # base model, max_pixels, dtype, head/LoRA config, train args, loss history
```

`infer.py` merges LoRA into the backbone by default (`--no-merge` to keep it separate).

## Known limitations (Windows + ROCm)

* `flash-linear-attention` needs Triton; `triton-windows` targets NVIDIA only, so the Gated-DeltaNet layers run the reference PyTorch path (slow). Not installed.
* Training is batch 1 (+ `--accum`); `collate_records` supports batching but is untested here.
* Synthetic data ships with a DUMMY teacher (`train.jsonl`/`eval.jsonl`); real Clef-Flash labels are in `*.clef.jsonl`
  (generated by `scripts/label_local_clef.py`; `data/` is git-ignored, so regenerate them on another machine).
* The synthetic task is easy: Clef-Flash is 100% correct on it, so it does not yet tell us much about real iOS screens.