Buckets:

EmergentTwilight
/

MMCoT

Files

xet

EmergentTwilight/MMCoT / docs /cv_bench.md

EmergentTwilight

27 days ago

preview code

download

raw

8.66 kB

# CV-Bench evaluation

## Dataset

- Official dataset: `nyu-visionx/CV-Bench`
- Pinned revision: `bc284db50d036958861cb60cdd7b77612052ce0d`
- Local root: `data/CV-Bench/`
- Prepared contract: `data/CV-Bench/prepared/records.jsonl` plus 2,638 images
- License: Apache-2.0

The benchmark contains 1,438 2D questions (counting and spatial relation from
ADE20K/COCO) and 1,200 3D questions (depth and relative distance from Omni3D).

## Confirmed baselines

| Model | Official total | Robust total | Robust 2D | Robust 3D | Robust Count | Robust Relation | Robust Depth | Robust Distance |
|---|---:|---:|---:|---:|---:|---:|---:|---:|
| UniAR-SFT | 83.86 | 84.86 | 81.22 | 88.50 | 70.69 | 95.08 | 95.00 | 82.00 |
| UniAR-SFT + Zebra-CoT interleaved | 77.37 | 77.37 | 78.16 | 76.58 | 68.78 | 90.77 | 81.00 | 72.17 |
| Qwen3-VL-8B-Instruct | **86.61** | 86.61 | 80.14 | **93.08** | 71.70 | 91.85 | **95.50** | **90.67** |
| Qwen3.5-9B | 85.98 | **87.27** | **82.63** | 91.92 | **73.22** | **95.54** | 95.00 | 88.83 |
| Qwen3.8-27B | 79.80 | 84.06 | 77.87 | 90.25 | 69.42 | 89.69 | 93.33 | 87.17 |

## Native UniAR protocol

Use `models/UniAR-SFT`, `uniar.CHAT_TEMPLATE`, the checkpoint's
default image processor, `flash_attention_2`, and greedy decoding. The dataset's
preformatted `prompt` is extended with the released Cambrian evaluator suffix:

```text
Answer with the option's letter from the given choices directly.
```

The released Cambrian evaluator uses `temperature=0`, one beam, and
`max_new_tokens=512`. UniAR's released `chat.py` does not propagate tokenizer EOS
IDs into `generate()`, which can continue through special-token loops after a
valid answer. The project evaluator explicitly stops on `<|im_end|>` or pad while
preserving the same greedy token prefix.

## Zebra-CoT interleaved generalization audit

The full Zebra adapter uses the Zebra system prompt, aspect-preserving
long-edge<=960 input, autonomous `<image_gen>` switching, direct generated-BSQ
reuse, greedy text/visual decoding, no text or image-count budget, and the
checkpoint's native 32,768-token context. Four single-GPU shards on `yixing`
cover all 2,638 questions. The official score is **77.37**, versus **83.86** for
the released direct-answer UniAR-SFT path; 2D changes from 79.21 to 78.16 while
3D drops from 88.50 to 76.58. Depth is 81.00 and Distance 72.17.

The model emits 2,759 images on 2,635 questions. Three samples produce an
invalid H/W scaffold and seven answers are unparsed. This is an end-to-end
system comparison, not a weight-only ablation: Zebra continued training and
mandatory autonomous visual reasoning both change. A direct-answer/visual-
disabled run of the same adapter is required to separate parameter forgetting
from generated-visual interference.

Canonical artifacts are
`experiments/output/uniar-sft/cv-bench/unversioned/runs/eval/20260821-131116_zebra_sft_cvbench_interleaved_full/`.
The public rollout observatory includes five fixed samples from each of Count,
Relation, Depth, and Distance, with exact token sequences and 27 decoded visual
thoughts: `https://zebra-cot-uniar-rollouts.leckyelt.chatgpt.site`.

## Released Zebra-CoT checkpoint sample probe

The same 20 fixed samples were additionally run with the released
Bagel-Zebra-CoT and Anole-Zebra-CoT checkpoints using their native interleaved
serializations. Bagel scores 8/20 (Count 3/5, Relation 2/5, Depth 3/5, Distance
0/5) with 35 generated states; Anole scores 13/20 (2/5, 4/5, 4/5, 3/5) with 20.
This is not a full 2,638-question score or a direct deterministic comparison
with UniAR. The `Official Zebra Models` site tab exposes the trajectories,
tokens, and images. Exact protocol and revisions:
`docs/research/20260824-122930_official_zebra_models_membership_and_cvbench_probe.md`.

## Qwen3-VL protocol

Use `models/Qwen3-VL-8B-Instruct` at revision
`0c351dd01ed87e9c1b53cbc748cba10e6187ff3b`, its checkpoint chat template and
image-processor defaults, BF16, `flash_attention_2`, and greedy decoding. Use the
same preformatted prompt, Cambrian suffix, `max_new_tokens=512`, official
extractor, and source-balanced scorer as the UniAR baseline. The verified full
run used four batch-size-4 shards on `luogeng` GPUs 4–7. All 2,638 outputs begin
with a valid option, so the official and conservative parsers agree exactly.

## Qwen3.5-family protocol

The comparison checkpoints are external shared models:

- Qwen3.5-9B revision `c202236235762e1c871ad0ccb60c8ee5ba337b9a` at
  `/mnt/ultracube/tac010/RL4SpatialIntelligence/models/Qwen3.5-9B`;
- Qwen3.8-27B revision `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` at
  `/mnt/ultracube/tac010/RL4SpatialIntelligence/models/Qwen3.8-27B`.

Both use `Qwen3_5ForConditionalGeneration`, the checkpoint-native
`Qwen3VLProcessor`, chat template and image-processor defaults, BF16, SDPA, and
`enable_thinking=False`. The template therefore ends the generation prefix with
the official empty `<think>\n\n</think>\n\n` block. CV-Bench's deterministic
greedy contract overrides Qwen's general non-thinking sampling recommendation:
`do_sample=False`, one beam, and `max_new_tokens=512` are retained for direct
comparability. Qwen3.5-9B ran as four single-GPU batch-size-4 shards; the 27B
checkpoint ran as two batch-size-2 shards, each balanced across two A6000s.

The protocol audit compared every generated answer with the released
ADE/COCO/Omni first-character extractor and found zero mismatches. Both full
runs contain 2,638 ordered unique indices and zero emitted `<think>` blocks.
Batch-size-1 smoke and full-run index 0 outputs agree exactly for both models.
The official total is retained for leaderboard reproduction. Cross-model
analysis uses one internally consistent robust view: robust total, source-
balanced robust 2D/3D, and robust task breakdowns. Never combine a robust total
with official submetrics. The released extractor remains available as the
official view, including its treatment of verbose answers beginning with
ordinary letters such as “Based”.

## Commands

```bash
# Prepare embedded parquet images once.
mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py prepare \
  --data-dir data/CV-Bench
# One GPU shard; submit through GPUQ in normal use.
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py infer \
  --data-dir data/CV-Bench \
  --model-path models/UniAR-SFT \
  --output-dir <run-dir> --shard-id 0 --num-shards 8 \
  --batch-size 1 --max-new-tokens 512 --explicit-eos \
  --attn flash_attention_2
# Merge and score after all shards finish.
mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py score \
  --output-dir <run-dir>
# Pin and download Qwen3-VL once.
mamba run -n UniAR hf download Qwen/Qwen3-VL-8B-Instruct \
  --revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \
  --local-dir models/Qwen3-VL-8B-Instruct
# One Qwen3-VL shard; submit four shards through GPUQ.
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n UniAR python experiments/src/qwen3_vl/eval_cv_bench.py infer \
  --data-dir data/CV-Bench \
  --model-path models/Qwen3-VL-8B-Instruct \
  --output-dir <run-dir> --shard-id 0 --num-shards 4 \
  --batch-size 4 --max-new-tokens 512 --attn flash_attention_2
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n UniAR python experiments/src/qwen3_vl/eval_cv_bench.py score \
  --output-dir <run-dir>
# Qwen3.5/Qwen3.8 family (requires the Transformers 5.15 environment).
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n RL4SI-Qwen35-SFT python experiments/src/qwen3_5/eval_cv_bench.py infer \
  --data-dir data/CV-Bench --model-path <qwen3.5-family-checkpoint> \
  --output-dir <run-dir> --shard-id 0 --num-shards <2-or-4> \
  --batch-size <2-or-4> --max-new-tokens 512 --attn sdpa \
  --device-map <balanced-or-single>
PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
mamba run -n RL4SI-Qwen35-SFT python experiments/src/qwen3_5/eval_cv_bench.py score \
  --output-dir <run-dir>
```

## Scoring

The official score gives equal weight to 2D and 3D, and equal weight to the two
2D sources:

```text
2D = (accuracy_ADE20K + accuracy_COCO) / 2
3D = accuracy_Omni3D
CV-Bench = (2D + 3D) / 2
```

The primary result uses Cambrian's released first-character MCQ extractor. The
evaluator also records a conservative parser result and parse-failure count so
verbose outputs cannot silently inflate interpretation confidence.
For research comparisons, report the official total separately and use
`robust_cv_bench_accuracy`, `robust_accuracy_2d`, `robust_accuracy_3d`, and
`robust_task_accuracy` together for every breakdown.

Xet Storage Details

Size:: 8.66 kB
Xet hash:: dc45b10c46d92bd66f5cc464e42b911889bdb0119b8cb61aa8ec500d7a6c26b6

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.

	# CV-Bench evaluation

	## Dataset

	- Official dataset: `nyu-visionx/CV-Bench`
	- Pinned revision: `bc284db50d036958861cb60cdd7b77612052ce0d`
	- Local root: `data/CV-Bench/`
	- Prepared contract: `data/CV-Bench/prepared/records.jsonl` plus 2,638 images
	- License: Apache-2.0

	The benchmark contains 1,438 2D questions (counting and spatial relation from
	ADE20K/COCO) and 1,200 3D questions (depth and relative distance from Omni3D).

	## Confirmed baselines

	\| Model \| Official total \| Robust total \| Robust 2D \| Robust 3D \| Robust Count \| Robust Relation \| Robust Depth \| Robust Distance \|
	\|---\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|---:\|
	\| UniAR-SFT \| 83.86 \| 84.86 \| 81.22 \| 88.50 \| 70.69 \| 95.08 \| 95.00 \| 82.00 \|
	\| UniAR-SFT + Zebra-CoT interleaved \| 77.37 \| 77.37 \| 78.16 \| 76.58 \| 68.78 \| 90.77 \| 81.00 \| 72.17 \|
	\| Qwen3-VL-8B-Instruct \| 86.61 \| 86.61 \| 80.14 \| 93.08 \| 71.70 \| 91.85 \| 95.50 \| 90.67 \|
	\| Qwen3.5-9B \| 85.98 \| 87.27 \| 82.63 \| 91.92 \| 73.22 \| 95.54 \| 95.00 \| 88.83 \|
	\| Qwen3.8-27B \| 79.80 \| 84.06 \| 77.87 \| 90.25 \| 69.42 \| 89.69 \| 93.33 \| 87.17 \|

	## Native UniAR protocol

	Use `models/UniAR-SFT`, `uniar.CHAT_TEMPLATE`, the checkpoint's
	default image processor, `flash_attention_2`, and greedy decoding. The dataset's
	preformatted `prompt` is extended with the released Cambrian evaluator suffix:

	```text
	Answer with the option's letter from the given choices directly.
	```

	The released Cambrian evaluator uses `temperature=0`, one beam, and
	`max_new_tokens=512`. UniAR's released `chat.py` does not propagate tokenizer EOS
	IDs into `generate()`, which can continue through special-token loops after a
	valid answer. The project evaluator explicitly stops on `<\|im_end\|>` or pad while
	preserving the same greedy token prefix.

	## Zebra-CoT interleaved generalization audit

	The full Zebra adapter uses the Zebra system prompt, aspect-preserving
	long-edge<=960 input, autonomous `<image_gen>` switching, direct generated-BSQ
	reuse, greedy text/visual decoding, no text or image-count budget, and the
	checkpoint's native 32,768-token context. Four single-GPU shards on `yixing`
	cover all 2,638 questions. The official score is 77.37, versus 83.86 for
	the released direct-answer UniAR-SFT path; 2D changes from 79.21 to 78.16 while
	3D drops from 88.50 to 76.58. Depth is 81.00 and Distance 72.17.

	The model emits 2,759 images on 2,635 questions. Three samples produce an
	invalid H/W scaffold and seven answers are unparsed. This is an end-to-end
	system comparison, not a weight-only ablation: Zebra continued training and
	mandatory autonomous visual reasoning both change. A direct-answer/visual-
	disabled run of the same adapter is required to separate parameter forgetting
	from generated-visual interference.

	Canonical artifacts are
	`experiments/output/uniar-sft/cv-bench/unversioned/runs/eval/20260821-131116_zebra_sft_cvbench_interleaved_full/`.
	The public rollout observatory includes five fixed samples from each of Count,
	Relation, Depth, and Distance, with exact token sequences and 27 decoded visual
	thoughts: `https://zebra-cot-uniar-rollouts.leckyelt.chatgpt.site`.

	## Released Zebra-CoT checkpoint sample probe

	The same 20 fixed samples were additionally run with the released
	Bagel-Zebra-CoT and Anole-Zebra-CoT checkpoints using their native interleaved
	serializations. Bagel scores 8/20 (Count 3/5, Relation 2/5, Depth 3/5, Distance
	0/5) with 35 generated states; Anole scores 13/20 (2/5, 4/5, 4/5, 3/5) with 20.
	This is not a full 2,638-question score or a direct deterministic comparison
	with UniAR. The `Official Zebra Models` site tab exposes the trajectories,
	tokens, and images. Exact protocol and revisions:
	`docs/research/20260824-122930_official_zebra_models_membership_and_cvbench_probe.md`.

	## Qwen3-VL protocol

	Use `models/Qwen3-VL-8B-Instruct` at revision
	`0c351dd01ed87e9c1b53cbc748cba10e6187ff3b`, its checkpoint chat template and
	image-processor defaults, BF16, `flash_attention_2`, and greedy decoding. Use the
	same preformatted prompt, Cambrian suffix, `max_new_tokens=512`, official
	extractor, and source-balanced scorer as the UniAR baseline. The verified full
	run used four batch-size-4 shards on `luogeng` GPUs 4–7. All 2,638 outputs begin
	with a valid option, so the official and conservative parsers agree exactly.

	## Qwen3.5-family protocol

	The comparison checkpoints are external shared models:

	- Qwen3.5-9B revision `c202236235762e1c871ad0ccb60c8ee5ba337b9a` at
	`/mnt/ultracube/tac010/RL4SpatialIntelligence/models/Qwen3.5-9B`;
	- Qwen3.8-27B revision `1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0` at
	`/mnt/ultracube/tac010/RL4SpatialIntelligence/models/Qwen3.8-27B`.

	Both use `Qwen3_5ForConditionalGeneration`, the checkpoint-native
	`Qwen3VLProcessor`, chat template and image-processor defaults, BF16, SDPA, and
	`enable_thinking=False`. The template therefore ends the generation prefix with
	the official empty `<think>\n\n</think>\n\n` block. CV-Bench's deterministic
	greedy contract overrides Qwen's general non-thinking sampling recommendation:
	`do_sample=False`, one beam, and `max_new_tokens=512` are retained for direct
	comparability. Qwen3.5-9B ran as four single-GPU batch-size-4 shards; the 27B
	checkpoint ran as two batch-size-2 shards, each balanced across two A6000s.

	The protocol audit compared every generated answer with the released
	ADE/COCO/Omni first-character extractor and found zero mismatches. Both full
	runs contain 2,638 ordered unique indices and zero emitted `<think>` blocks.
	Batch-size-1 smoke and full-run index 0 outputs agree exactly for both models.
	The official total is retained for leaderboard reproduction. Cross-model
	analysis uses one internally consistent robust view: robust total, source-
	balanced robust 2D/3D, and robust task breakdowns. Never combine a robust total
	with official submetrics. The released extractor remains available as the
	official view, including its treatment of verbose answers beginning with
	ordinary letters such as “Based”.

	## Commands

	```bash
	# Prepare embedded parquet images once.
	mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py prepare \
	--data-dir data/CV-Bench

	# One GPU shard; submit through GPUQ in normal use.
	PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
	mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py infer \
	--data-dir data/CV-Bench \
	--model-path models/UniAR-SFT \
	--output-dir <run-dir> --shard-id 0 --num-shards 8 \
	--batch-size 1 --max-new-tokens 512 --explicit-eos \
	--attn flash_attention_2

	# Merge and score after all shards finish.
	mamba run -n UniAR python experiments/src/uniar_adapter/eval_cv_bench.py score \
	--output-dir <run-dir>

	# Pin and download Qwen3-VL once.
	mamba run -n UniAR hf download Qwen/Qwen3-VL-8B-Instruct \
	--revision 0c351dd01ed87e9c1b53cbc748cba10e6187ff3b \
	--local-dir models/Qwen3-VL-8B-Instruct

	# One Qwen3-VL shard; submit four shards through GPUQ.
	PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
	mamba run -n UniAR python experiments/src/qwen3_vl/eval_cv_bench.py infer \
	--data-dir data/CV-Bench \
	--model-path models/Qwen3-VL-8B-Instruct \
	--output-dir <run-dir> --shard-id 0 --num-shards 4 \
	--batch-size 4 --max-new-tokens 512 --attn flash_attention_2

	PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
	mamba run -n UniAR python experiments/src/qwen3_vl/eval_cv_bench.py score \
	--output-dir <run-dir>

	# Qwen3.5/Qwen3.8 family (requires the Transformers 5.15 environment).
	PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
	mamba run -n RL4SI-Qwen35-SFT python experiments/src/qwen3_5/eval_cv_bench.py infer \
	--data-dir data/CV-Bench --model-path <qwen3.5-family-checkpoint> \
	--output-dir <run-dir> --shard-id 0 --num-shards <2-or-4> \
	--batch-size <2-or-4> --max-new-tokens 512 --attn sdpa \
	--device-map <balanced-or-single>

	PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" \
	mamba run -n RL4SI-Qwen35-SFT python experiments/src/qwen3_5/eval_cv_bench.py score \
	--output-dir <run-dir>
	```

	## Scoring

	The official score gives equal weight to 2D and 3D, and equal weight to the two
	2D sources:

	```text
	2D = (accuracy_ADE20K + accuracy_COCO) / 2
	3D = accuracy_Omni3D
	CV-Bench = (2D + 3D) / 2
	```

	The primary result uses Cambrian's released first-character MCQ extractor. The
	evaluator also records a conservative parser result and parse-failure count so
	verbose outputs cannot silently inflate interpretation confidence.
	For research comparisons, report the official total separately and use
	`robust_cv_bench_accuracy`, `robust_accuracy_2d`, `robust_accuracy_3d`, and
	`robust_task_accuracy` together for every breakdown.