Buckets:
| # Offline Rollout Visual Quality | |
| `zebra_sft.score_visual_quality` is a standalone analysis workflow for saved | |
| Visual-Jigsaw interleaved rollouts. It is intentionally not part of canonical | |
| task evaluation and never changes an eval run. | |
| ## Pairing contract | |
| The join key is `(sample_id, one_based_generated_state_index)`. The eval | |
| `arguments.json` selects the packed cache; `sample_id` resolves its exact | |
| manifest shard/index and source Parquet coordinate. Image index follows the | |
| ordered assistant `generated_state` events. Paired metrics are emitted only | |
| when expected and generated image counts match. This is a positional protocol | |
| alignment, not proof that the model assigned the same semantics to each step. | |
| Raw BSQ comparison additionally requires identical code topology. Pixel and | |
| perceptual metrics require relative aspect-ratio error at most 10%; the generated | |
| RGB is then bicubic-resized to fixed reference coordinates. Larger mismatches | |
| remain explicit geometry failures and receive semantic metrics only. | |
| ## Metrics | |
| - Native: BSQ bit accuracy, fractional/absolute Hamming distance, exact visual | |
| token agreement. | |
| - Pixel: MSE, PSNR, SSIM, adaptive-level MS-SSIM. | |
| - Perceptual: LPIPS and DISTS from PIQ 0.8.0. | |
| - Semantic: image-image cosine from DINOv2-base and CLIP ViT-B/32. | |
| - Completeness: image-count match, blank-image rate, resolution/aspect | |
| diagnostics, and adjacent near-duplicate rate. A near duplicate requires | |
| DINO cosine at least 0.995 and LPIPS at most 0.02. | |
| Every metric is reported both image-weighted and trajectory-balanced; the latter | |
| first averages image indices within a sample so four-image trajectories do not | |
| receive four times the weight. | |
| ## Fixed dependencies | |
| Install the analysis-only dependency without changing the training package: | |
| ```bash | |
| mamba run -n UniAR python -m pip install -r experiments/requirements-visual-quality.txt | |
| ``` | |
| Local semantic checkpoints are fixed to: | |
| - `facebook/dinov2-base` at `f9e44c814b77203eaa57a6bdbbd535f21ede1415`; | |
| - `openai/clip-vit-base-patch32` at | |
| `3d74acf9a28c67741b2f4f2ea7635f0aaf6f0268`. | |
| ## Command | |
| Run on a scheduler-assigned CUDA device. Decode and scoring are separately | |
| resumable phases; `all` performs both while unloading the diffusion decoder | |
| before loading metric networks. | |
| ```bash | |
| PYTHONPATH=OURS/src:experiments/src mamba run -n UniAR \ | |
| python -m zebra_sft.score_visual_quality \ | |
| --eval-run experiments/output/zebra-cot-visual-jigsaw/runs/eval/<eval-run> \ | |
| --output-dir experiments/output/uniar-sft/zebra-cot-visual-jigsaw/unversioned/runs/visual-quality/<run> \ | |
| --phase all --resume | |
| ``` | |
| The output contains deterministic decoded PNGs, atomic per-sample metrics, | |
| `sample_metrics.jsonl`, `summary.json`, `report.md`, and `COMPLETE`. Existing | |
| eval artifacts remain read-only. | |
| A campaign containing one child output per eval can be aggregated on CPU: | |
| ```bash | |
| PYTHONPATH=OURS/src:experiments/src mamba run -n UniAR \ | |
| python -m zebra_sft.summarize_visual_quality \ | |
| --campaign-dir experiments/output/uniar-sft/zebra-cot-visual-jigsaw/unversioned/runs/visual-quality/<campaign> | |
| ``` | |
| This writes `comparison.json`, `comparison.csv`, and a compact | |
| `comparison.md` table using trajectory-balanced headline metrics. | |
| For many eval runs, `zebra_sft.roll_visual_quality_campaign` first freezes one | |
| source snapshot per child, submits them through Delta's regular `gpuA100x4` | |
| partition, polls completion, resubmits incomplete/time-limited children from | |
| their exact snapshots, and invokes the campaign summary when all are complete. | |
| The interactive partition is intentionally not used for campaigns because its | |
| partition QoS permits only one running and two submitted jobs per user. | |
Xet Storage Details
- Size:
- 3.69 kB
- Xet hash:
- 0592910d79db55f22238ba89f833a586bd3f77fe7f63474e6b4d9a421584a6ff
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.