docs/cv_bench.md

CV-Bench evaluation

Dataset

• Official dataset: nyu-visionx/CV-Bench
• Pinned revision: bc284db50d036958861cb60cdd7b77612052ce0d
• Local root: data/CV-Bench/
• Prepared contract: data/CV-Bench/prepared/records.jsonl plus 2,638 images
• License: Apache-2.0

The benchmark contains 1,438 2D questions (counting and spatial relation from ADE20K/COCO) and 1,200 3D questions (depth and relative distance from Omni3D).

Confirmed baselines

Model	Official total	Robust total	Robust 2D	Robust 3D	Robust Count	Robust Relation	Robust Depth	Robust Distance
UniAR-SFT	83.86	84.86	81.22	88.50	70.69	95.08	95.00	82.00
UniAR-SFT + Zebra-CoT interleaved	77.37	77.37	78.16	76.58	68.78	90.77	81.00	72.17
Qwen3-VL-8B-Instruct	86.61	86.61	80.14	93.08	71.70	91.85	95.50	90.67
Qwen3.5-9B	85.98	87.27	82.63	91.92	73.22	95.54	95.00	88.83
Qwen3.8-27B	79.80	84.06	77.87	90.25	69.42	89.69	93.33	87.17

Native UniAR protocol

Use models/UniAR-SFT, uniar.CHAT_TEMPLATE, the checkpoint's default image processor, flash_attention_2, and greedy decoding. The dataset's preformatted prompt is extended with the released Cambrian evaluator suffix:

Answer with the option's letter from the given choices directly.

The released Cambrian evaluator uses temperature=0, one beam, and max_new_tokens=512. UniAR's released chat.py does not propagate tokenizer EOS IDs into generate(), which can continue through special-token loops after a valid answer. The project evaluator explicitly stops on <|im_end|> or pad while preserving the same greedy token prefix.

Zebra-CoT interleaved generalization audit

The full Zebra adapter uses the Zebra system prompt, aspect-preserving long-edge<=960 input, autonomous <image_gen> switching, direct generated-BSQ reuse, greedy text/visual decoding, no text or image-count budget, and the checkpoint's native 32,768-token context. Four single-GPU shards on yixing cover all 2,638 questions. The official score is 77.37, versus 83.86 for the released direct-answer UniAR-SFT path; 2D changes from 79.21 to 78.16 while 3D drops from 88.50 to 76.58. Depth is 81.00 and Distance 72.17.

The model emits 2,759 images on 2,635 questions. Three samples produce an invalid H/W scaffold and seven answers are unparsed. This is an end-to-end system comparison, not a weight-only ablation: Zebra continued training and mandatory autonomous visual reasoning both change. A direct-answer/visual- disabled run of the same adapter is required to separate parameter forgetting from generated-visual interference.

Canonical artifacts are experiments/output/uniar-sft/cv-bench/unversioned/runs/eval/20260821-131116_zebra_sft_cvbench_interleaved_full/. The public rollout observatory includes five fixed samples from each of Count, Relation, Depth, and Distance, with exact token sequences and 27 decoded visual thoughts: https://zebra-cot-uniar-rollouts.leckyelt.chatgpt.site.

Released Zebra-CoT checkpoint sample probe

The same 20 fixed samples were additionally run with the released Bagel-Zebra-CoT and Anole-Zebra-CoT checkpoints using their native interleaved serializations. Bagel scores 8/20 (Count 3/5, Relation 2/5, Depth 3/5, Distance 0/5) with 35 generated states; Anole scores 13/20 (2/5, 4/5, 4/5, 3/5) with 20. This is not a full 2,638-question score or a direct deterministic comparison with UniAR. The Official Zebra Models site tab exposes the trajectories, tokens, and images. Exact protocol and revisions: docs/research/20260824-122930_official_zebra_models_membership_and_cvbench_probe.md.

Qwen3-VL protocol

Use models/Qwen3-VL-8B-Instruct at revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b, its checkpoint chat template and image-processor defaults, BF16, flash_attention_2, and greedy decoding. Use the same preformatted prompt, Cambrian suffix, max_new_tokens=512, official extractor, and source-balanced scorer as the UniAR baseline. The verified full run used four batch-size-4 shards on luogeng GPUs 4–7. All 2,638 outputs begin with a valid option, so the official and conservative parsers agree exactly.

Qwen3.5-family protocol

The comparison checkpoints are external shared models:

• Qwen3.5-9B revision c202236235762e1c871ad0ccb60c8ee5ba337b9a at /mnt/ultracube/tac010/RL4SpatialIntelligence/models/Qwen3.5-9B;
• Qwen3.8-27B revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0 at /mnt/ultracube/tac010/RL4SpatialIntelligence/models/Qwen3.8-27B.

Both use Qwen3_5ForConditionalGeneration, the checkpoint-native Qwen3VLProcessor, chat template and image-processor defaults, BF16, SDPA, and enable_thinking=False. The template therefore ends the generation prefix with the official empty <think>\n\n</think>\n\n block. CV-Bench's deterministic greedy contract overrides Qwen's general non-thinking sampling recommendation: do_sample=False, one beam, and max_new_tokens=512 are retained for direct comparability. Qwen3.5-9B ran as four single-GPU batch-size-4 shards; the 27B checkpoint ran as two batch-size-2 shards, each balanced across two A6000s.

The protocol audit compared every generated answer with the released ADE/COCO/Omni first-character extractor and found zero mismatches. Both full runs contain 2,638 ordered unique indices and zero emitted <think> blocks. Batch-size-1 smoke and full-run index 0 outputs agree exactly for both models. The official total is retained for leaderboard reproduction. Cross-model analysis uses one internally consistent robust view: robust total, source- balanced robust 2D/3D, and robust task breakdowns. Never combine a robust total with official submetrics. The released extractor remains available as the official view, including its treatment of verbose answers beginning with ordinary letters such as “Based”.

Commands

# Prepare embedded parquet images once.
mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py prepare \
  --data-dir data/CV-Bench

# One GPU shard; submit through GPUQ in normal use.
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py infer \
  --data-dir data/CV-Bench \
  --model-path models/UniAR-SFT \
  --output-dir <run-dir> --shard-id 0 --num-shards 8 \
  --batch-size 1 --max-new-tokens 512 --explicit-eos \
  --attn flash_attention_2

# Merge and score after all shards finish.
mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py score \
  --output-dir <run-dir>

# Pin and download Qwen3-VL once.
mamba run -n UniAR hf download Qwen/Qwen3-VL-8B-Instruct \
  --revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \
  --local-dir models/Qwen3-VL-8B-Instruct

# One Qwen3-VL shard; submit four shards through GPUQ.
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n UniAR python experiments/src/qwen3_vl/eval_cv_bench.py infer \
  --data-dir data/CV-Bench \
  --model-path models/Qwen3-VL-8B-Instruct \
  --output-dir <run-dir> --shard-id 0 --num-shards 4 \
  --batch-size 4 --max-new-tokens 512 --attn flash_attention_2

PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n UniAR python experiments/src/qwen3_vl/eval_cv_bench.py score \
  --output-dir <run-dir>

# Qwen3.5/Qwen3.8 family (requires the Transformers 5.15 environment).
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n RL4SI-Qwen35-SFT python experiments/src/qwen3_5/eval_cv_bench.py infer \
  --data-dir data/CV-Bench --model-path <qwen3.5-family-checkpoint> \
  --output-dir <run-dir> --shard-id 0 --num-shards <2-or-4> \
  --batch-size <2-or-4> --max-new-tokens 512 --attn sdpa \
  --device-map <balanced-or-single>

PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n RL4SI-Qwen35-SFT python experiments/src/qwen3_5/eval_cv_bench.py score \
  --output-dir <run-dir>

Scoring

The official score gives equal weight to 2D and 3D, and equal weight to the two 2D sources:

2D = (accuracy_ADE20K + accuracy_COCO) / 2
3D = accuracy_Omni3D
CV-Bench = (2D + 3D) / 2

The primary result uses Cambrian's released first-character MCQ extractor. The evaluator also records a conservative parser result and parse-failure count so verbose outputs cannot silently inflate interpretation confidence. For research comparisons, report the official total separately and use robust_cv_bench_accuracy, robust_accuracy_2d, robust_accuracy_3d, and robust_task_accuracy together for every breakdown.