docs/zebra_cot.md

Zebra-CoT for UniAR interleaved SFT

Status and scope

• Local link: data/Zebra-CoT -> /mnt/ultracube/datasets/Zebra-CoT
• Dataset: 182,384 train-only examples, 18 configurations, 163 Parquet shards.
• On-disk Parquet payload: 66.10 GiB; images are embedded in Parquet rows.
• Dataset-card license: CC-BY-NC-4.0. Use is limited to non-commercial research unless separately licensed.
• Official sources: dataset, paper, training code, and the complete standalone Graph-Algorithms source.

Zebra-CoT is suitable for teaching UniAR an interleaved visual-reasoning protocol and for studying explicit visual-state rollout. It is not a substitute for Stage 1 posed multi-view supervision: most reasoning images are sketches, crops, edited boards, diagrams, or demonstrated trajectory frames rather than camera/action-conditioned novel views with calibrated geometry.

Raw 1% development subset

data/Zebra-CoT-small/ is a directly materialized 633 MiB development subset, not a symlink and not a preprocessed UniAR cache. It contains 1,825 of 182,384 rows: each of the 18 configurations contributes max(1, floor(config_rows * 0.01 + 0.5)) rows selected with seed 20260825. Every configuration retains its original Arrow schema and embedded image bytes in train-00000-of-00001.parquet. Exact source shard/row coordinates live in SPATIAL_MMCOT_SUBSET.json; all 1,825 copied rows were verified field-by-field against their source rows. Reproduce the immutable output at a new path with:

PYTHONPATH="$PWD/OURS/src:$PWD/experiments/src" mamba run -n UniAR python -m \
  zebra_sft.sample_raw_subset --source-root data/Zebra-CoT \
  --output-root data/Zebra-CoT-small-rebuild --fraction 0.01 --seed 20260825

Materialized task-specific full subsets

Three full configurations are physically materialized for controlled task-specific training; none is a symlink or hardlink. Each root preserves the original config directory, Parquet shard names, Arrow schema, row order, and embedded image bytes. A checksum-mode rsync comparison found zero differences against every source shard. SPATIAL_MMCOT_TASK_SUBSET.json records provenance and the answer audit in each root.

Root	Config	Shards	Rows	Answer contract
data/Zebra-CoT-Visual-Jigsaw	2D Visual Reasoning - Visual Jigsaw	26	21,899	100% A--D visual multiple choice
data/Zebra-CoT-Chess	Visual Logic & Strategic Games - Chess	8	20,483	100% explicit A--D; 20,480 raw answers unambiguous
data/Zebra-CoT-Maze	Visual Logic & Strategic Games - Maze	11	20,000	9,000 explicit MC + 2,500 image-labeled MC + 8,500 numeric

Each task uses a deterministic whole-group 90/5/5 split. Because every group in these three configurations contains one row, the row and group counts are identical and the requested ratios are exact after rounding:

Task	Train	Validation	Test	Group leakage
Visual Jigsaw	19,709	1,095	1,095	0
Chess	18,435	1,024	1,024	0
Maze	18,000	1,000	1,000	0

SPATIAL_MMCOT_SPLIT.json records the split method and counts. The canonical membership lives in each task's UniAR-960-cache/manifest.jsonl; it reuses the immutable packed tensors in data/Zebra-CoT-UniAR-960 through absolute shard paths instead of duplicating roughly 40 GiB of BSQ tensors. The raw Parquet payload above remains physically materialized. Rebuild a task view with experiments/src/zebra_sft/task_split.py at a fresh destination.

The current controlled Visual-Jigsaw BitFlip comparison uses the five OURS/config/sft/zebra_visual_jigsaw_control_*_bitflip_2gpu_1epoch.yaml configs. All use seed 17, the same split/order, rsLoRA, optimizer, one epoch, 50% stable noisy samples with 2% per-bit corruption, two GH200 ranks, an eight-sample microbatch under a 32,768 padded-token budget, global batch up to 16, and per-step TensorBoard/JSONL logging. Only the IG enable flag and weights vary: Vanilla; T->V/V->T .005/.1; .005/1; .05/10; and .5/100. The 2-GPU task-2 smoke completed 10/10 finite updates, reproduced a 1.996% noisy-group flip rate, peaked at 36.26 GiB allocated per GPU, and recorded about 97% average utilization per GH200. The resulting full schedule is 1,312 updates over the same 19,709 train rows. Older four-A6000 configs and runs are historical comparisons, not members of this new controlled matrix.

Within this project, unqualified "Vanilla SFT" denotes the IG-disabled member with the stable 50% noisy-sample partition and 2% per-bit BSQ input corruption; the no-corruption variant must be named explicitly. The controlled gradnorm5 follow-ups change the clipping threshold from 1.0 to 5.0 while preserving global batch 16 with single-GPU gradient accumulation. The two-epoch half-sample variant changes only training duration; the all_samples variant changes only the noisy-sample fraction to 1.0.

On DeltaAI, the controlled Vanilla and .005/.1 IG runs completed with the two-rank configs before the sequential bundle was stopped. The remaining three weight settings and the four event_selection: all variants use explicit *_bitflip_1gpu_1epoch.yaml configs: microbatch and padded-token budget remain 8/32,768, while two sequential microbatches are accumulated per optimizer update. This preserves effective global batch 16 and the same 1,312-update scheduler without DDP synchronization. All-events IG uses trajectory-balanced reduction: events are averaged inside each trajectory before averaging the contributing trajectories. The single-GH200 all-events smoke completed 10/10 finite updates, used 20 microbatches, and peaked at 36.28 GiB.

Native multi-rank SFT exposes trainer.ddp_find_unused_parameters and trainer.ddp_static_graph; historical configs retain true/false. The v1 Tetris and Visual-Jigsaw base family uses false/false. Changing these flags alone did not resolve historical failures. A confirmed Torch-2.7 interaction was the custom dataclass returned by the SFT objective: DDP's output pytree did not expose its loss to the static-graph backward sink, silently bypassing gradient synchronization. ObjectiveOutput now uses a NamedTuple. Four-process CPU regressions exercise the actual output type under normal, unused-parameter discovery, and static-graph DDP, plus checkpoint/RNG/optimizer resume parity. Pre-fix static-graph checkpoints must not be treated as valid distributed-SFT results; recovery should return to the pre-static parent checkpoint.

SFT multi-rank control traffic is isolated from DDP gradients: NCCL carries only reducer buckets, while a CPU Gloo group carries statistics, finite flags, objects, barriers, validation aggregation, and checkpoint coordination. This preserves wall-clock recovery checkpoints. Mixed NCCL gradient/control traffic alone passed a 2,000-iteration four-GPU communication test; it is not by itself proof of a defect. First-backward gradient and pre-checkpoint parameter hashes now enforce replica equality using the Gloo group.

The default is trainer.gradient_sync: ddp: native backward bucket reduction with gradient_as_bucket_view=True, no custom communication hook, and no per-bucket CPU rendezvous. The v1 family explicitly uses DDP with find_unused_parameters=false, static_graph=false, microbatch 4 per GPU, and accumulation 1 on four GPUs (global batch 16). Existing sample membership, loss normalization, BitFlip seeds, and optimizer schedules are preserved. The explicit post_backward SUM path remains a diagnostic comparison only.

The supported stack is Torch 2.7.1+cu128 and torchvision 0.22.1. Torch 2.7.0 implicitly enabled nonblocking NCCL host APIs with eager device_id, causing a main/watchdog race in ncclCommGetAsyncError/pthread_join on both clusters. Torch 2.7.1 disables that implicit mode upstream (PyTorch #153960), so SFT now uses ordinary process-group options. Keep TORCH_NCCL_USE_COMM_NONBLOCKING and NCCL_COMM_BLOCKING unset on this stack. Tests use TORCH_DISTRIBUTED_DEBUG=OFF; DETAIL changes synchronization and can mask timing-sensitive defects. CUDA stream asynchrony and watchdog monitoring stay enabled. Neither extra barriers nor wall-checkpoint removal repairs the race.

The v1 duration extension uses zebra_tetris_vanilla_{3,4,5,6}epoch.yaml; these configs differ from the Tetris vanilla base only in run identity and epoch count. The first Visual-Jigsaw relative-IG weight extension uses zebra_visual_jigsaw_ig_lambda{2,4}_3epoch.yaml, changing both directional weights together while retaining three epochs and temperature 0.25.

The v2 Tetris base pair lives at OURS/config/sft/v2/zebra_tetris_{vanilla,ig}_base.yaml. It preserves the v1 pair except for run.version/name and a constant LR schedule with 20 fixed warmup updates. V2 long runs may request exact epoch-boundary checkpoints without changing legacy checkpoint behavior.

The RPM v1 duration family is OURS/config/sft/v1/zebra_rpm_vanilla_{2,4,8}epoch.yaml. It preserves the Tetris v1 vanilla recipe (rank 32, half-sample uniform BitFlip up to 0.1, IG/EMA off, clip norm 5, four-GPU DDP/global batch 16), using the existing RPM cache's 2,700 train and all 150 validation rows. Only run name and epoch count vary across that duration family. The controlled 4-epoch relative-IG weight sweep is zebra_rpm_ig_lambda{2,4,8}_4epoch.yaml: it changes the run identity/category, enables all-event relative_per_unit IG with equal T→V and V→T weights, and enables validation IG statistics. Both temperatures remain 0.25. Data, uniform half-sample BitFlip, model, optimizer, schedule, clipping, DDP, global batch 16, and all 150 validation rows match the 4-epoch vanilla control. Each λ run starts independently from UniAR-SFT. Each duration starts independently from UniAR-SFT with its own cosine schedule. Internal validation runs at startup, every 100 updates and completion; it is teacher-forced, not rollout accuracy. Use validation for duration/IG selection and keep the 150-row test split for the final comparison. These files do not by themselves submit Slurm jobs.

The first EMA-distillation control is zebra_visual_jigsaw_ema_nec_all_events_bitflip_full_gradnorm5_1gpu_1epoch.yaml. It keeps one epoch, full-sample 2% BitFlip, global batch 16, all events, and max_grad_norm=5, while disabling IG so the only new objective is EMA Necessity. The initial EMA settings are decay .99, ratio clip 5, contrast alpha .5 in both directions, and T->V/V->T weights .05/2.0. Microbatch 2 with eight-step accumulation preserves global batch 16 on a 40-GiB A100. A two-update GH200 smoke completed with 26.73 GiB peak allocation and roughly 1,000--1,070 sequence tokens/s; EMA update count exactly matched the two successful optimizer updates. The formal task pairs one Delta training job with an after-success full 1,095-row autonomous test job.

The controlled IG Temperature center is zebra_visual_jigsaw_ig_temperature_center_tv0p5_vt0p05_all_events_bitflip_full_gradnorm5_1gpu_1epoch.yaml. It changes the strong-weight all-events anchor only to full-sample BitFlip, T->V/V->T temperatures .5/.05, max_grad_norm=5, and 200-update validation/checkpoint boundaries; weights remain .05/10, EMA distillation is explicitly disabled, and the original microbatch 8 / accumulation 2 schedule retains all 19,709 rows and 1,312 updates. The matching historical run reached 50.29 GiB after its short smoke window, so Phase A uses a Delta gpuA100x8 device rather than an A100-40GB and does not change batch packing. Each teacher-forced validation artifact includes trajectory-balanced gain and gate distributions. evaluate_task_subset --selection validation owns the complete 1,095-row autonomous validation rollout and reports one-/four-image accuracy, grid completion, image totals, and assistant text/visual token counts. The --sample-offset N --max-samples M pair selects a deterministic manifest-order window for short calibration runs while preserving each row's split-global evaluation_index; shard merging validates those nonzero indices directly. The train-512 diagnostic accepts an exact prior subset file and a complete step-000200 checkpoint adapter, recording both subset and adapter fingerprints. Temperature calibration also supports exact --resume: an atomic progress.json commits complete batch byte boundaries, cumulative event hash, sample IDs and counts; a torn JSONL tail is truncated to the last committed batch, stable request/subset metadata is revalidated, and only pending cache rows are loaded. Event output is buffered and flushed once per batch rather than once per event. Because the fixed panel is globally length-sorted, calibration keeps all of its mmap-backed shard mappings open instead of repeatedly thrashing a two-shard LRU; tensor storage remains demand-paged.

DeltaAI training reads the independently rebuilt data/Zebra-CoT-Visual-Jigsaw/UniAR-960-cache-deltaai. Its 21,899 sample IDs, source coordinates, group IDs, split membership, sequence lengths, and visual statistics exactly match the migrated task manifest; all 344 local shards are present. The old task view is retained as provenance but its absolute /mnt/ultracube shard paths are unavailable on this cluster.

The preliminary autonomous evaluation uses a fixed 100-row Visual-Jigsaw test subset: 25 examples for each target choice A--D, selected independently by a seeded SHA256 order. evaluation_subset_index=0..99 in the task cache manifest is the canonical cross-model join key; all non-members store null. SPATIAL_MMCOT_EVALUATION_SUBSET.json records exact membership and provenance. Rebuild the index with select_evaluation_subset.py; run two-rank native understanding or unbounded interleaved evaluation with evaluate_task_subset.py. The latter preserves exact token/event sequences, generated BSQ codes, deterministic RGB decodes, strict-final scoring, and per-sample artifacts for the rollout viewer.

Controlled Visual-Jigsaw comparisons use --selection test, which evaluates the complete 1,095-row test split rather than the indexed 20/100-row qualitative subsets. On one GH200, launch two evaluator processes with torchrun --nproc-per-node=2 plus --workers-per-gpu 2 --scheduling dynamic. Both processes share the assigned GPU and atomically claim the next unfinished sample, so variable rollout lengths do not create fixed-rank tail imbalance. Autonomous evaluation also stops an individual text or scaffold phase when its token suffix contains an exact period of at most 128 tokens repeated three times and spanning at least 64 repeated tokens. The minimum span avoids misclassifying short, legitimate enumerations as degeneration. This token-level guard is separate from the existing complete-image-phase cycle detector, records cycle_period with cycle_unit=text_tokens, and does not alter logits or otherwise constrain non-cyclic generation. The evaluator holds an advisory lock for the run directory for its full lifetime; a second job targeting the same output must fail before touching the shared queue or shards. Do not bypass this single-writer invariant. The final merge rejects duplicate, missing, or provenance-mismatched rows and supports exact resume from completed per-rank shards. If an interrupted legacy run contains byte-identical duplicate commits, repair it with reshard_task_subset_eval.py --deduplicate-identical; the command validates all sample artifacts and preserves the original shards under resume-backups/ before atomically installing a clean shard set. A fixed six-trajectory smoke matched single-worker token/event trajectories and generated BSQ tensors exactly; its inference-phase GPU utilization rose from 36.9% for one worker to 81.7% for two, with 40.3 GiB peak memory.

For SIQ TIMEOUT retries, the identical evaluation command reuses the same run directory without an explicit --resume: an existing run claim enables resume, committed sample bundles are validated, an incomplete journal suffix is rolled back, and all workers atomically rebuild one queue containing only unfinished samples. Sample text/JSON/packed-code files are atomic and the JSONL commit is fsynced after them; summary.json remains the single final accuracy result.

Task evaluation loads selected cache items through mmap-backed shard grouping and a small dynamic-worker LRU instead of retaining nearly the entire mixed-split cache in every process; the measured loader RSS fell from about 4.11 GB to 0.57 GB. Resume validates completed shards before model loading, loads only pending items, and can publish an already-complete run without loading the 7B model. Generated 0/1 BSQ artifacts use the versioned packed-bit schema zebra_generated_codes_packed_v1; all readers remain compatible with historical raw tensor lists. The largest measured four-image artifact shrank by 62.9x. Standalone paired RGB/BSQ quality scoring is owned by docs/visual_quality.md; it consumes completed eval artifacts but is not part of task evaluation.

Cross-phase language-KV reuse is intentionally off by default. Real GH200 FlashAttention2 diagnostic 3057393 kept the first eight greedy text tokens exact but changed a subsequent visual bit because incremental-decode hidden states were not bit-identical to a full-prefix language prefill. Production rollout therefore retains phase-local language prefills; the experimental single-cache path requires explicit reuse_language_cache=True and must not be used for benchmark comparisons. Retroactive mask_prior_thought can never use that cache because later K/V already encode the masked thought indirectly. Likewise, GH200 two-token diagnostic 3057464 found that incremental caching inside the four-layer visual decoder changed a generated bit relative to its historical full-prefix call. reuse_visual_decoder_cache therefore also defaults to false; its implementation remains an explicit experimental path only.

The Qwen3.8-27B direct-answer baseline uses the same 1,095 manifest rows through qwen3_5.eval_visual_jigsaw. It reads only the verbatim Question events and problem_image_1; Text Reasoning Trace, reasoning_image_*, and Final Answer never enter the prompt. The source-positioned raw 2400x1800 image is processed by vLLM with checkpoint defaults, followed only by an output-format request for Final answer: X. Inference uses Qwen revision 1d4bf0f2ff6012fd82039f2fa52739d0dd7c60c0, vLLM 0.28.0, and the checkpoint chat template. The completed default-template run passed no Thinking-control arguments, so this revision emitted its implicit xhigh instruction and <think> prefix. The current comparison runner explicitly passes the official template argument reasoning_effort="medium"; it does not add a custom system prompt. prompt_audit.json stores the rendered template plus a lossless run-length encoding and SHA256 of the actual post-image-expansion prompt token IDs. Primary accuracy parses only the post-</think> answer region, while explicit-Final answer: accuracy is also reported. Launch through run_visual_jigsaw_vllm.sh; partial JSONL output is resume-safe. Output generation has no per-request token cap and the engine uses the checkpoint-native 262,144-token context; Delta runs therefore request four 40GB A100s so memory is not traded against the model context.

On this exact test100, native UniAR understanding scores 23/100, the one-epoch Vanilla interleaved SFT scores 50/100, and the controlled Information-Gain SFT scores 53/100; every interleaved rollout terminates with im_end. These are task-specific exact-choice results, not full Zebra-CoT scores. The public rollout viewer exposes the first five indexed rows within each answer choice (20 total), including exact assistant token sequences, all generated states, and the GT trajectory. The full 100-row summaries remain the accuracy source of truth; the balanced 20-row display is qualitative only. The shared choice scorer treats a bare numeric answer such as 5 as equivalent to option 5, just as bare C is equivalent to option C; this is required for RPM, whose valid choices are numbered 1--8 and whose target wording varies.

Chess has three source-label anomalies: shard 0 row 1882 says A. Qf5 although Qf5 is option C; shard 2 row 2404 says C. Rc8 although Rc8 is option B; shard 1 row 1642 copies the complete option list instead of choosing one. In all three cases the reasoning trace identifies the intended move. Keep raw data immutable and either filter these rows from validation or apply an explicit scorer-side repair.

Maze is therefore not entirely multiple choice. Its 8,500 free-response rows remain deterministic and parser-friendly: connected-component count (2,000), largest component size (2,000), total hole count (1,000), minimum steps (2,500), and shortest-path count plus length (1,000). All 20,000 answers match one of the audited discrete/templates, so task-specific exact scoring is practical.

The most relevant first-stage subset is all 39,610 3D examples:

Configuration	Examples	Role
3D Multi-Hop Objects Counting	10,000	Synthetic action-conditioned object-state changes; best first pilot
3D Embodied CoT	22,666	Initial/goal observations and intermediate embodied states
3D Robot Planning	6,944	Initial/goal observations and intermediate manipulation states

The remaining domains are useful for protocol diversity and general visual reasoning, but should be added only after the spatial pilot is understood.

Local data audit

Audit performed against every row and Parquet metadata on 2026-08-19.

Check	Result
Rows	182,384
Problem-image references / physical images	210,830 / 210,830
Reasoning-image references / physical images after local repair	718,866 / 718,869
Blank question, trace, or answer	0
Trace without a reasoning image	0
Duplicate image reference	0
Duplicate/non-monotonic THOUGHT n ID	0

Known packaging defects:

• The upstream consolidated Zebra-CoT Scientific Reasoning - Graph Algorithms release contains only reasoning_image_1..9; 1,802 rows reference 8,660 unavailable reasoning_image_10..19. This was not an incomplete clone: commit f6d1defc169d19180a69f517925ed6cbb06a1c97 matched Hugging Face main. The local checkout has since been repaired as documented below.
• The authors' standalone multimodal-reasoning-lab/Graph-Algorithms repository at commit 020bc9d2a442d9989ba49344dd23c0bb4cfac90f contains reasoning_image_1..19. Its seven shards have the same row counts, and all Question, Text Reasoning Trace, and Final Answer values are exactly equal in the same row order. Its columns 10–19 contain exactly 8,660 non-null images, so they can be joined unambiguously by shard name and row index without replacing the consolidated text or images 1–9.
• Geometry has 10 rows and Physics has 1 row that place problem_image_1 inside the assistant reasoning trace.
• ARC-AGI has three physical reasoning images not referenced by its traces; ignore these extras.

Local Graph repair

The shared Zebra checkout is now repaired in place. All seven consolidated Graph Parquet files were replaced by the corresponding complete official Graph-Algorithms shards after proving every pre-existing Arrow column was value-identical row-by-row. The repaired shards contain reasoning_image_1..19, retain the same 10,000 row order/source coordinates, and expose 8,660 non-null images in the newly available columns 10--19.

Post-repair audit: 84,072 reasoning-image references, zero missing referenced images, and target file SHA256 values exactly equal to the standalone official repository at revision 020bc9d2a442d9989ba49344dd23c0bb4cfac90f.

• Repair manifest: /mnt/ultracube/datasets/Zebra-CoT/SPATIAL_MMCOT_REPAIR.json
• Original-file backup: /mnt/ultracube/datasets/Zebra-CoT-repair-backups/20260819-133848_graph-algorithms-original/
• Reproducible repair: experiments/src/zebra_sft/repair_graph_dataset.py

The checkout is intentionally dirty; do not run a destructive Git reset. The streaming loader detects the repaired schema and no longer requires the standalone overlay, while remaining backward-compatible with an unrepaired checkout.

The unpatched upstream release accepts 180,582 rows; the repaired local checkout accepts all 182,384. The 11 Geometry/Physics assistant images stored in problem_image_1 columns are retained as generated states with explicit provenance rather than excluded. Do not silently truncate trajectories or reinterpret a missing physical image as text-only reasoning.

Image sizes and aspect ratios are heterogeneous: sampled images range from small crops such as 198x116 through 2400x1800 and 2085x2999. Training must use an explicit resolution/bucketing policy rather than the original dimensions blindly.

Canonical trajectory contract

Apart from the documented Graph repair, do not edit the Parquet files. Convert each row to a model-independent typed trajectory:

{
  "sample_id": "sha256:...",
  "source": {"dataset": "Zebra-CoT", "config": "..."},
  "user": [
    {"type": "text", "text": "..."},
    {"type": "image", "role": "observation", "key": "problem_image_1"}
  ],
  "assistant": [
    {"type": "text", "role": "thought", "text": "..."},
    {"type": "image", "role": "generated_state", "key": "reasoning_image_1"},
    {"type": "text", "role": "answer", "text": "..."}
  ]
}

The generic record stores semantics, not checkpoint-specific token strings. The UniAR collator renders each generated_state into the exact checkpoint protocol. This guarantees that observation images never accidentally receive a generation marker.

UniAR rendering

Problem images are user observations and use the normal understanding path:

<|im_start|>user
question text <|vision_start|><|image_pad|>...<|vision_end|>
<|im_end|>

A problem image is optional. Text-only questions that introduce one or more assistant reasoning images are valid text-to-image/interleaved SFT examples; only the assistant reasoning-image target is required. Do not discard these Chemistry, Competitive Programming, Geometry, ARC-AGI, or Physics rows merely because the user turn has no observation image.

Semantic role follows turn position, not the source column prefix. Eleven released rows use problem_image_1 for a diagram first referenced inside the assistant trace; retain these as generated_state, insert <image_gen>, and record the source-column mismatch in provenance.

Each assistant reasoning image is a generation target:

<|im_start|>assistant
<think>thought before image 1</think>
<image_gen> generate image H W<|vision_start|>
GT_BSQ_CODES_1
<|vision_end|>
<think>thought before image 2</think>
<image_gen> generate image H W<|vision_start|>
GT_BSQ_CODES_2
<|vision_end|>
Final answer: ...
<|im_end|>

H W are visual-token grid dimensions, not pixel dimensions. With the current patch_size=16 and spatial_merge_size=2, a 512x512 target is 16 16 and contains 256 autoregressive visual positions.

The <image_gen> marker is therefore required before every reasoning_image_*, not before any problem_image_*. Use the fast tokenizer loaded by AutoProcessor and assert that the literal marker encodes to exactly one ID. The checkpoint currently maps it to ID 151669 in that runtime; slow-tokenizer metadata instead reports 151675, so no numeric ID may be hard-coded in prepared data.

Do not add a new <answer> token in the first experiment. Use ordinary Final answer: text plus ChatML end-of-turn; this isolates the experiment to the already-required image boundary.

Loss and model contract

modeling.md owns matched context views, bidirectional IG, OPD divergences and RL rewards. Legacy auxiliaries and current execution contracts belong to uniar.md. In particular, one UniAR visual prediction step is one composite token containing 4 levels x 4 merge cells x 64 bits; the bits factorize its likelihood but do not count as 1,024 autoregressive time steps.

One teacher-forced mixed sequence should provide:

Region	Supervision
System, user text, problem images	masked
Assistant thoughts and final answer	language CE
` generate image H W<	vision_start
Reasoning-image BSQ codes	per-bit two-class CE through the visual prediction head
Text after a reasoning image	language CE conditioned on ground-truth BSQ features

In canonical autonomous rollout, the student samples the complete <image_gen> generate image H W<|vision_start|> scaffold and owns <|vision_start|> as a language action/target. The controller inserts only <|vision_end|> after exactly $H\times W$ visual positions. Grid sides are validated in 1–30 only after the delimiter is sampled; default text, phase, image, and visual-token horizons are context-bound and use shared short-cycle detection rather than fixed caps.

Normalize text and visual losses separately before combining them; otherwise the many BSQ bit decisions dominate the objective. Log both losses and their gradient norms.

Freeze the BSQ encoder and SD3 visual decoder initially. Train the AR backbone with LoRA, the visual decoder/prediction head, and the <image_gen> input/output rows. Ordinary LoRA alone is insufficient because the current fast-token <image_gen> embedding is nearly untrained and the base checkpoint does not emit it reliably.

The released UniAR repository does not provide this SFT path. Its default forward computes language loss, while is_vistok_pred=True returns visual logits with loss=None; only image-generation GRPO training is public. The canonical reusable cache reader, objective, and trainer now live under OURS/src/sft/ without importing or modifying projects/UniAR/; older Zebra-specific preparation and experiment runners remain under experiments/.

Canonical on-policy distillation is implemented separately under OURS/src/opd/. It rolls out the current student from the cached $B_0$, keeps sampled text and hard BSQ states in their exact serialized prefix, and calls the binary answer verifier once. Event selection supports the first complete visual-to-text window or all complete bidirectional transitions. The eight supported routing/replay combinations are configured under OURS/config/opd/; shared EMA, distribution, model, replay-SFT, scoring, and artifact logic lives under OURS/src/shared/ rather than being imported across the SFT and OPD packages. Objective semantics remain owned by docs/modeling.md.

Cleaning and preparation plan

1. Stream Parquet rather than extracting all RGB files.
2. Overlay only reasoning_image_10..19 for Graph Algorithms from the standalone repository, keyed by shard name and row index. Assert exact equality of the three text columns before accepting every shard, and record both repository revisions in provenance.
3. Validate non-empty fields, exact placeholder-to-column mapping, non-null image bytes, unique ordered references, and decodability.
4. Remove THOUGHT n: labels while preserving the text/image order; keep the original fields in provenance metadata.
5. Build stable IDs and group splits from configuration, normalized question, and problem-image content hashes. Split before turning one trajectory into windows so alternate annotations of the same problem cannot leak across splits.
6. Preserve aspect ratio and cap the long edge at 960 through the exact released qwen-vl-utils + UniAR processor path. Derive and assert each rendered H W from the resulting image_grid_thw; only lower the per-sample cap if the full sequence would exceed 32K.
7. Precompute frozen BSQ codes into a content-addressed shared cache. The prepared manifest refers to source Parquet coordinates and cache keys rather than duplicating JPEG bytes.
8. Keep full trajectories when they fit 32K. For an overflow, lower that trajectory's aspect-preserving cap through audited buckets; reject only if text plus minimum-resolution visuals still cannot fit.
9. Emit deterministic audit metrics and a small human-readable sample gallery before training.

Historical pilot framework

The original pilot code is preserved under experiments/src/zebra_sft/:

• prepare_samples.py: exact raw Parquet fixture plus typed/normalized samples;
• trajectory.py: placeholder parsing and UniAR rendering;
• dataset.py: processor inputs, assistant-only language labels, and generated-image masks;
• objective.py: frozen BSQ encoding and joint language/visual teacher-forcing loss;
• train.py: LoRA overfit runner, BSQ cache, metrics, and adapter saving;
• evaluate.py: autonomous <image_gen> emission and teacher-forced BSQ decoding;
• continue_after_visual.py: generated-BSQ replay and text-head resumption;
• audit_bsq_roundtrip.py: RGB decode/re-encode and light-perturbation BSQ stability audit;
• rollout.py: complete repeated text/visual state machine;
• evaluate_full.py: all-step image decoding, protocol audit, and answer scoring;
• evaluate_baseline.py: constrained and native direct-answer baselines;
• summarize_comparison.py: paired metrics and per-sample comparison;
• run_overfit16.sh: canonical queue payload.

The inspectable fixture is data/Zebra-CoT-samples/:

• raw/samples.parquet preserves 16 exact source-format rows with embedded images;
• raw/records.jsonl exposes original text and image hashes;
• prepared/trajectories.jsonl uses spatial_mmcot_trajectory_v1;
• prepared/images/ contains 96 aspect-preserving 256x256 PNGs;
• prepared/rendered/ shows the exact semantic placement of every generation marker.

The current pilot freezes the BSQ tokenizer, caches its ground-truth grouped bits, and applies LoRA to the language/visual decoder projections plus embed_tokens and lm_head. There are 53,489,664 trainable parameters out of 9,680,563,696 (0.5525%). <image_gen> targets receive higher language-loss weight because they are sparse but protocol-critical. The generation-intent switch is only <image_gen>; <|vision_start|> remains a shared structural boundary for both input observations and generated images.

16-trajectory overfit result

Run: experiments/output/uniar-sft/zebra-cot-samples/unversioned/smoke/sft/20260818-222131_uniar_sft_zebra_overfit16/. Queue tasks: smoke 6da5630c, train eb5e82d4, autonomous/decode eval 6dfbb48c, and post-image continuation 464ac353. The 320-step run used one queue-assigned RTX A6000 (GPU 5), took 6m40s, and peaked at 20.26 GiB.

Metric	Base	Final
Total joint loss	4.8585	0.1439
Weighted text loss	4.5908	0.00918
Marker loss	18.1007	0.00000685
Text token accuracy	58.70%	99.75%
<image_gen> accuracy	0/80	80/80
Visual bit loss	0.2677	0.1348
Visual bit accuracy	88.04%	94.28%

After adapter reload, all 16 prompts autonomously emitted <image_gen>. The controller then generated and decoded a first visual state. Generated-BSQ replay successfully resumed the next textual thought and emitted the next <image_gen>. Teacher-forced decoded layout closely matches the target reconstruction; the autonomous image preserves the main colors/layout but contains one extra brown object. Therefore P0 confirms the SFT/control contract but not exact visual-state memorization. See docs/research/20260818-223457_zebra_cot_uniar_sft_overfit16_report.md.

Complete-rollout result

Paired run: experiments/output/uniar-sft/zebra-cot-samples/unversioned/smoke/eval/20260818-231640_zebra_overfit16_full_vs_base/. The fine-tuned path repeatedly generates text until the model emits <image_gen>, lets the controller add only H W + vision_start, greedily generates 64 BSQ positions, directly replays those generated codes without RGB re-encoding, and continues until ChatML end or a safety failure.

Mode	Final answers
Base UniAR-SFT, constrained direct answer	4/16 (25%)
Base UniAR-SFT, native free-form answer	4/16 (25%)
Fine-tuned complete interleaved rollout	8/16 (50%)

Only 11/16 fine-tuned trajectories complete the exact annotated interleave; 75/80 expected images are emitted. Mean autonomous visual-bit agreement is 78.06% and mean pixel PSNR is 17.47 dB. Agreement declines with rollout depth: 81.47%, 80.59%, 79.33%, 75.27%, 73.87%, 72.71%, 68.88%, and 68.21% for steps 1–8 respectively (later steps have fewer examples).

Contact sheets show that generated scenes often preserve coarse color/layout but fail the requested state change: removed objects persist, required objects are absent, or empty scenes remain populated. Some answers remain correct despite poor visual states, so this training-set gain cannot be attributed to useful visual world modeling. The canonical paired table is summary.md in the run; per-sample sequences and contact sheets are under finetuned_interleaved/<sample_id>/. Full analysis: docs/research/20260818-234151_zebra_cot_full_rollout_vs_baseline_report.md.

Sequence-protocol audit

The canonical v2 Zebra serialization uses this system prompt for both SFT and autonomous inference:

Solve the problem using interleaved textual and visual reasoning. Wrap text reasoning in <think>...</think>. To generate a visual state, emit <image_gen> generate image H W<|vision_start|> and continue after <|vision_end|>. Conclude with "Final answer:" followed by the answer.

The assistant turn starts with the input-owned <|im_start|>assistant\n. Each assistant event is separated by exactly one newline. Text scaffolds through the student-emitted <|vision_start|>, visual BSQ codes, the newline following the controller-owned <|vision_end|>, final answer, and <|im_end|> are student targets; only <|vision_end|> is inserted by the rollout controller. IG source masks remain semantic thought payloads or visual payload positions and exclude their wrappers. The protocol identifier, prompt, and source hashes are recorded in each cache config.json. Full-corpus v2 artifacts live at data/Zebra-CoT-UniAR-960-cache-v2; older caches remain immutable because their token sequences use the v1 prompt and whitespace.

Three independent audits of the UniAR paper, released code, Zebra raw data, and project implementation agree on the following boundary:

• generate image 8 8 is correct for the 256x256 pilot: 8 8 is the merged visual-token grid (256 / (patch16 * merge2)), not pixel dimensions;
• <image_gen> is the only generation-intent special token; generate image, spaces, and digits are ordinary text tokens, while vision_start/end are structural boundaries;
• Qwen image-pad expansion, image/grid order, M-RoPE, pre-norm visual head input, and causal visual-target shift match the released UniAR implementation;
• the complete topology does not match released UniAR SFT/inference: native T2I places marker/dimensions in the user turn and starts assistant visual output at vision_start; this project teaches assistant <think> → <image_gen> → image → <think> → ... as a new capability;
• current SFT also differs through marker-weighted assistant CE, 1:1 text/visual loss rather than the paper's 1:10, no random visual-bit flipping, generated-only visual loss, LoRA rather than full AR training, and 256-square letterboxing.

Before scaling, pin the fast tokenizer/ID, derive and assert H/W from actual processor grids, decide whether controller-forced scaffold tokens should receive language CE, add visual-noise/scheduled-sampling ablations, and repair data edges (THOUGHT N:, adjacent-image spans, answer wrappers). Full audit: docs/research/20260819-001450_zebra_cot_uniar_sequence_protocol_audit.md.

For token inspection use experiments/src/zebra_sft/render_token_alignment.py. It emits fixed-width RAW/TOK/ID rows; blank RAW cells mark inserted protocol tokens. Current raw mapping is event-level provenance for inspection, not yet a universal character-offset provenance system for all 182K rows.

Fine-tuning plan

Historical full-corpus run implementation

The completed 2026-08-20 production run used experiments/src/zebra_sft/:

• full_data.py: strict row validation, standalone Graph overlay, grouped split, exact dynamic image grids, packed BSQ cache, and batching;
• prepare_full.py: four-GPU cache orchestration and corpus audit;
• train_full.py: BF16 rank-32 rsLoRA + full visual-head synchronous DDP, validation, checkpoints, and TensorBoard;
• cached_rollout.py: cache-native dynamic H/W modality-switching controller;
• evaluate_corpus.py: held-out baseline/interleaved comparison, trajectories, metrics, and image galleries.

Long cache builds submitted through SIQ use four workers plus --auto-resume --balanced-resume. Each retry automatically starts fresh when the output is absent, repartitions only unfinished source shards when partial, or exits successfully when the atomic READY marker already exists.

The public Hub release has no validation or test split. The production cache uses deterministic grouped train/validation/test assignment. Within each configuration, hash-sorted groups reserve 1% test (32--256 groups) and 0.25% validation (8--64 groups); the remainder trains. The final audit requires every configuration in both held-out splits and zero group leakage. Full design: docs/research/20260819-012200_zebra_cot_full_sft_design.md.

Current production queue chain: completed cache 04155db3, guarded continuation train 698f61d8, held-out test 8c683c2f. The continuation writes TensorBoard, adapters, checkpoints, optimizer state, and validation metrics to experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/sft/20260820-054100_zebra_full960_resume2000_guarded/.

The original train task 7b95ef57 is valid only through step 2,000. At step 2,001 its gradient norm became NaN; the unconditional optimizer update corrupted the model, while the logger converted NaN metrics to zero. Recovery uses the clean step-2,000 adapter, a fresh lower-LR optimizer, FP32 visual-head parameters and moments, non-finite update skipping/recording, and a hard abort threshold. Four-step DDP smoke task 537479fd exactly reproduced step-2,000 validation and completed steps 2,001--2,004 with zero non-finite updates.

Final full-SFT and held-out rollout result

Guarded continuation task 698f61d8 completed global step 4,110. One inf-gradient update at step 3,179 was safely skipped; no loss/parameter corruption recurred. Fixed 36-sample validation changed from base to final as follows:

Metric	Base	Final
Joint loss	5.6757	2.7391
Text accuracy	58.99%	84.88%
<image_gen> accuracy	0%	100%
Visual-bit accuracy	87.73%	89.79%

Held-out task 8c683c2f evaluated one low-compute complete trajectory from each of the 18 configs. Base direct answers scored 1/18; fine-tuned autonomous interleaving scored 6/18. Strict annotated protocol completion was 7/18, with 38/40 expected images generated. All 38 emitted images used syntactically valid <image_gen> generate image H W<|vision_start|> scaffolds, but only 12/18 trajectories matched every annotated grid and only 10/18 reached ChatML end with a final answer. Failures comprise five extra-image markers, three text-token limits, and four emitted-grid mismatches (plus two no-image token-limit cases).

Across 33 generated steps whose grid matched the target, autonomous visual-bit agreement averaged 80.27% (median 85.10%, range 55.42--96.52%). Contact sheets show recognizable coarse appearance for natural images, but weak state-change semantics: removed objects persist, manipulation states differ, and chemistry diagrams can depict the wrong molecule. Therefore the answer gain is real on this small diagnostic, but does not establish useful visual world modeling.

A relaxed retry raised the text-segment limit from 512 to 4,096 and allowed four additional images for all eight initially incomplete samples. Five reached a final answer: Physics and ARC-AGI resolved their text limits, while Robot Planning, Connect Four, and Tetris completed with extra images. Geometry moved from text truncation into an image loop; Competitive Programming and Graph Algorithms also continued looping past the enlarged image budget. Relaxed completion is therefore 15/18 overall. Image count/HW differences remain reported diagnostics, not primary failure criteria; valid image scaffolds plus a final answer define the relaxed format.

A final native-context rerun removed both artificial budgets: text and scaffold generation may use the remaining 32,768-token model context, and generated-image count is independent of the annotation. It produced 64 images, 17/18 trajectories ended with a final answer and im_end, all emitted scaffolds were valid, and no sequence ended with a dangling <image_gen>. Graph Algorithms continued from 6 to 7 images and Geometry from 5 to 10 before ending normally. Competitive Programming instead entered an exact period-2 image/thought loop; after the full period repeated three times it was recorded as cycle_detected after image 8 at a complete vision_end boundary. This is semantic degeneration detection, not a token or image-count budget. Automatic answer accuracy remains 6/18 and strict annotated-grid protocol rises to 9/18. Canonical artifacts are under experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/eval/20260821-111254_zebra_native_context_test18/.

A latent-copy audit compares each generated BSQ state with every same-grid observation and annotated state. Of 49 comparable generated images, 29 (59.2%) are nearest an observation; among 23 cases where both an observation and the index-aligned GT are comparable, the generated state is closer to the observation in all 23. Directly replaying the closest observation reaches 87.66% bit agreement with the GT, versus 85.90% for autonomous generation; the generator beats this copy baseline in only 30.4% of cases. Consecutive generated states average 96.28% agreement and 71.4% exceed 99%, exposing visual fixed-point collapse. Robot Planning specifically emits obs1 → obs1 → obs2; its prompt already contains both initial and goal images, enabling keyframe retrieval instead of action-conditioned transition. Treat index alignment cautiously when the model emits extra actions/images. Audit artifact: experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/eval/20260821-111254_zebra_native_context_test18/visual_copy_analysis.json.

Generated-visual-context ablation

The same 18 held-out test samples were rerun under six causal interventions at the native 32,768-token limit, with no per-segment or image-count budget: deterministic RGB noise encoded to BSQ, gray-128 encoded to BSQ, image-pad word embeddings without BSQ/DeepStack, zero main/DeepStack visual features, empty vision spans, and masking all prior completed text phases during later text generation. The control and text-mask conditions use native autoregressive BSQ.

Strict automatic accuracy, gated by an explicit Final answer:, is respectively 6/18 control, 7/18 noise, 9/18 gray, 3/18 image-pad, 4/18 zero-feature, 5/18 empty-span, 1/18 text-mask. Noise/gray change T2 in 14/18 despite containing no task evidence, and the model repeatedly narrates specific evidence from gray, zero, or empty visual states. Their apparent score gains are perturbation/chance, not improved visual reasoning. Masking old text while retaining native images reduces normal completion from 17/18 to 1/18; 15/18 traces cycle and adjacent generated states are nearly frozen (median same-size BSQ agreement 99.898%). The supported conclusion is that visual state affects decoding and modality control, but the text chain carries most rollout progress and generated images are not a robust standalone cross-step memory.

cached_rollout.py, ablate_rollouts.py, and analyze_ablations.py own the implementation, artifact validation, and paired analysis. Canonical artifacts: experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/ablation/20260823-170404_zebra_test18_visual_ablation_suite/. Full protocol, examples, scoring audit, and limitations: docs/research/20260823-173015_zebra_visual_context_ablation_report.md.

BSQ RGB round-trip pilot

A four-image Zebra fixture pilot at long-edge 384 and eight decoder steps finds that RGB -> BSQ -> decoded RGB -> BSQ preserves only 71.68% of bits on average; just 0.20% of 64-bit groups remain exactly equal. A second cycle preserves 74.84% of the preceding codes. Applied to the first decoded RGB, Gaussian noise at sigma 2/255, JPEG quality 95, and 0.95x bicubic down/up change respectively 10.85%, 9.88%, and 11.31% of re-encoded bits while keeping the input RGB at 41.76, 42.84, and 45.03 dB PSNR. Contact sheets retain coarse object/layout semantics but also expose texture and fine-geometry drift, so this pilot establishes a lossy BSQ bottleneck, not semantic invariance or evidence of a private channel. Any future grounding reward must still compare direct-BSQ and rebuilt RGB paths behaviorally. Script, complete metrics, and limitations are recorded in docs/research/20260825-014710_bsq_rgb_roundtrip_pilot.md.

Official model membership audit

The public Zebra-CoT release exposes only train shards. The released Bagel configuration feeds one zebra_cot.jsonl into its iterable dataset with num_used_data: None, and the Anole card says it is fine-tuned on the corpus. Our grouped 18-example split is consequently held out from our UniAR SFT only; for released Bagel/Anole it is likely author-training membership, not an author-held-out test. The paper does not release its claimed Anole in-distribution test IDs/split or the IDs of its 512-question frontier-model evaluation. We therefore run a labelled six-domain task-coverage proxy rather than misrepresenting it as the hidden test set.

With native public serializations, Bagel/Anole score 8/20 and 13/20 on the fixed external CV-Bench probe; on our grouped 18 their strict-final scores are 5/18 and 4/18, and on the six-domain proxy both are 0/6. The public viewer contains both Zebra settings, all trajectories, native token representations, and generated images. Protocol, revisions, and limitations: docs/research/20260824-122930_official_zebra_models_membership_and_cvbench_probe.md.

The public rollout viewer exposes all 18 native-context runs alongside the old 512 and 4,096 variants, the seven-condition ablation, exact token IDs and masked attention spans, input/GT images, and all decoded visual states: https://zebra-cot-uniar-rollouts.leckyelt.chatgpt.site.

The cleaned unified TensorBoard excludes corrupted old steps 2,001--2,284, merges valid steps 1--2,000 with guarded steps 2,001--4,110, and includes all validation points, 18 output trajectories, eight relaxed retries, six contact sheets, and final test scalars:

experiments/output/uniar-sft/zebra-cot-full/unversioned/runs/sft/20260820-054100_zebra_full960_resume2000_guarded/tensorboard_merged/

The first full cache pass stopped after 166,706 committed rows because it incorrectly rejected 7,718 legal text-only-question rows and a faster rank hit the default 10-minute NCCL barrier timeout. The canonical preparation path now accepts zero observation images, supports --resume from committed manifest coordinates, uses collision-free appended cache shards, and gives preprocessing ranks a 24-hour coordination timeout.

Final full-corpus preparation result

Balanced resume task 04155db3 completed successfully. The final cache is data/Zebra-CoT-UniAR-960 -> /mnt/ultracube/datasets/Zebra-CoT-UniAR-960-v1.

Audit	Result
Source / retained rows	182,384 / 182,384 (100%)
Train / validation / test rows	180,080 / 461 / 1,843
Unique grouped problems	166,713
Group leakage	0
Text-only questions retained	7,718
Misnamed assistant-image rows retained with provenance	11
Discarded rows	0
Resolution cap	960 for all rows
Mean / maximum sequence length	2,116.06 / 29,415
Mean / maximum visual positions	1,608.35 / 18,000
Cache size	36 GiB

The final manifest has 182,384 unique sample IDs and source coordinates. All 2,853 referenced cache shards exist; there are no missing/orphan shards or bad per-shard index ranges. stats.json owns the complete per-config split audit.

The full autonomous-interleaved CV-Bench audit scores 77.37 on all 2,638 questions, versus 83.86 for the base checkpoint's released direct-answer path. The loss is concentrated in 3D (88.50 -> 76.58; Depth 81.00, Distance 72.17). The adapter generates 2,759 images on 2,635 questions, with three invalid H/W scaffolds and seven unparsed answers. This comparison still confounds continued-training regression with generated-visual interference; the same-adapter visual-disabled/direct-answer ablation remains required. Artifacts and the four-task 5-sample visual/token gallery are owned by docs/cv_bench.md.

Current next experiments

P0 — Causal coupling audit

• Build 200--500 machine-checkable one-step windows from Multi-Hop Objects, Robot Planning, games, and ARC.
• Run identical-prefix GT/generated/counterfactual/shuffled/null BSQ interventions, next-token/state probes, and the GT/generated text × visual-history 2×2.
• Compare greedy CFG=1 with temperature 0.1 at CFG 1, 2.0, and 2.5 using a correct unconditional history. Do not train another full-corpus model first.

P1 — Targeted remediation

• If GT-prefix next-state generation fails, train action-conditioned one-step transitions with change-region and semantic-state supervision. Compare the current rsLoRA against full AR-backbone/visual-decoder/head tuning as a capacity ablation while keeping the BSQ tokenizer and SD3 decoder frozen.
• If counterfactual BSQ does not control subsequent text, add post-image state QA and paired counterfactual continuations before more long-trajectory SFT. A full-versus-visual-blind dependency term must be anchored by correct, state-consistent targets so it cannot create arbitrary visual dependence.
• If both one-step directions work but autonomous histories fail, train on detached hard-BSQ self-replay with clean-teacher-forced batches retained. Do not attach the original GT continuation to a semantically different sampled state: relabel it with a simulator/state parser, train explicit critique-and-repair, or mask the inconsistent downstream target.
• Treat the existing all-position/all-level bit CE as the baseline. Test a soft BSQ-feature/delta-aware auxiliary loss with changed-region weighting and a source-copy contrastive margin; plain full-frame reconstruction is not targeted to the observed fixed-point shortcut.
• Retain native understanding/QA replay data in every continued-training branch.

P2 — World-model and transfer gate

• Prefer Stage 1 posed multi-view/action-conditioned data over more heterogeneous Zebra traces for establishing world-model competence.
• Re-enable longer multimodal rollouts only after transition, recognition, and causal-use metrics pass on held-out one-step windows.
• Test generated/GT/disabled visual inference and cross-task transfer before RL; RL should optimize when to imagine only after visual states are reliable.

Required comparisons

1. Base UniAR-SFT.
2. Zebra text-only CoT SFT with identical questions/answers.
3. Zebra interleaved SFT with visual loss but forced boundary at inference.
4. Zebra interleaved SFT with autonomous <image_gen> boundary.
5. Spatial-only 3D Zebra versus full-domain Zebra.

Evaluate exact protocol compliance, visual bit loss, decoded state correctness, final-answer accuracy with generated images, ground-truth BSQ replay, and visual generation disabled. Retain CV-Bench as the native-understanding regression test. The Zebra-CoT paper reports that its Bagel fine-tune learned visual CoT but did not improve standard benchmarks and sometimes regressed because hallucinated images corrupted later reasoning; this makes the generated/ground-truth/disabled three-way comparison mandatory rather than optional.

Before further full-corpus training, run the causal coupling audit in docs/research/20260823-144502_zebra_cross_modal_coupling_diagnosis.md on machine-checkable one-step windows. The image-to-text comparison must additionally inject valid counterfactual, shuffled, and null/copy BSQ states under an identical text prefix and report next-token NLL plus answer/action sensitivity. GT replay is an oracle bound, not evidence that the model causally uses visual state. Evaluate the 2x2 of GT/generated prior text and GT/generated prior visual state to separate textual drift, visual drift, and exposure bias. Visual evaluation must include changed-region/action correctness and unchanged-region preservation; global BSQ agreement alone is invalid because observation copying already reaches 87.66%. The native-context strict_protocol=9/18 is only an autonomous structural check (normal end plus exact annotated grid sequence); it is not GT BSQ replay.