docs/visual_quality.md

Offline Rollout Visual Quality

zebra_sft.score_visual_quality is a standalone analysis workflow for saved Visual-Jigsaw interleaved rollouts. It is intentionally not part of canonical task evaluation and never changes an eval run.

Pairing contract

The join key is (sample_id, one_based_generated_state_index). The eval arguments.json selects the packed cache; sample_id resolves its exact manifest shard/index and source Parquet coordinate. Image index follows the ordered assistant generated_state events. Paired metrics are emitted only when expected and generated image counts match. This is a positional protocol alignment, not proof that the model assigned the same semantics to each step.

Raw BSQ comparison additionally requires identical code topology. Pixel and perceptual metrics require relative aspect-ratio error at most 10%; the generated RGB is then bicubic-resized to fixed reference coordinates. Larger mismatches remain explicit geometry failures and receive semantic metrics only.

Metrics

• Native: BSQ bit accuracy, fractional/absolute Hamming distance, exact visual token agreement.
• Pixel: MSE, PSNR, SSIM, adaptive-level MS-SSIM.
• Perceptual: LPIPS and DISTS from PIQ 0.8.0.
• Semantic: image-image cosine from DINOv2-base and CLIP ViT-B/32.
• Completeness: image-count match, blank-image rate, resolution/aspect diagnostics, and adjacent near-duplicate rate. A near duplicate requires DINO cosine at least 0.995 and LPIPS at most 0.02.

Every metric is reported both image-weighted and trajectory-balanced; the latter first averages image indices within a sample so four-image trajectories do not receive four times the weight.

Fixed dependencies

Install the analysis-only dependency without changing the training package:

mamba run -n UniAR python -m pip install -r experiments/requirements-visual-quality.txt

Local semantic checkpoints are fixed to:

• facebook/dinov2-base at f9e44c814b77203eaa57a6bdbbd535f21ede1415;
• openai/clip-vit-base-patch32 at 3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268.

Command

Run on a scheduler-assigned CUDA device. Decode and scoring are separately resumable phases; all performs both while unloading the diffusion decoder before loading metric networks.

PYTHONPATH=OURS/src:experiments/src mamba run -n UniAR \
  python -m zebra_sft.score_visual_quality \
  --eval-run experiments/output/zebra-cot-visual-jigsaw/runs/eval/<eval-run> \
  --output-dir experiments/output/uniar-sft/zebra-cot-visual-jigsaw/unversioned/runs/visual-quality/<run> \
  --phase all --resume

The output contains deterministic decoded PNGs, atomic per-sample metrics, sample_metrics.jsonl, summary.json, report.md, and COMPLETE. Existing eval artifacts remain read-only.

A campaign containing one child output per eval can be aggregated on CPU:

PYTHONPATH=OURS/src:experiments/src mamba run -n UniAR \
  python -m zebra_sft.summarize_visual_quality \
  --campaign-dir experiments/output/uniar-sft/zebra-cot-visual-jigsaw/unversioned/runs/visual-quality/<campaign>

This writes comparison.json, comparison.csv, and a compact comparison.md table using trajectory-balanced headline metrics.

For many eval runs, zebra_sft.roll_visual_quality_campaign first freezes one source snapshot per child, submits them through Delta's regular gpuA100x4 partition, polls completion, resubmits incomplete/time-limited children from their exact snapshots, and invokes the campaign summary when all are complete. The interactive partition is intentionally not used for campaigns because its partition QoS permits only one running and two submitted jobs per user.