readme.md

this repo used to show the intervention effect of different baseline values. 

theorically,  if the visual tower can perfectly model the visual world, the small intervention should not change the final VLM output.

this dir is used to check the overlap of failure cases between different baseline values.


Therefore, I will run inference in multiple online model:
1. Qwen/Qwen2.5-VL-3B-Instruct
2. Qwen/Qwen2.5-VL-7B-Instruct

3. Qwen/Qwen3.5-4B
4. Qwen/Qwen3.5-4B-Base



Datasets:
VSR: /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/02_data/VSR/V7_testing.json 
 - baseline: 0
 - baseline: 0.05
 - baseline: 0.1
 - baseline: 0.15
 - baseline: 0.2
 - baseline: 0.25

Youtube_self_depth_QA
 - Level 1: /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/02_data/Youtube_self_depth_QA/level1/test.jsonl
    - baseline: 0
    - baseline: 0.05
    - baseline: 0.1
    - baseline: 0.15
    - baseline: 0.2
    - baseline: 0.25

 - Level 2: /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/02_data/Youtube_self_depth_QA/level2/test.jsonl
    - baseline: 0
    - baseline: 0.05
    - baseline: 0.1
    - baseline: 0.15
    - baseline: 0.2
    - baseline: 0.25
 
 - Level 3: /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/02_data/Youtube_self_depth_QA/level3/test.jsonl
    - baseline: 0
    - baseline: 0.05
    - baseline: 0.1
    - baseline: 0.15
    - baseline: 0.2
    - baseline: 0.25

  
all of code with respect to load data will be saved in /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/01_3_prelimilary_intervention_different_baseline/dataset

all of inference code will be saved in /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/01_3_prelimilary_intervention_different_baseline/inference

all of inference results will be saved in /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/01_3_prelimilary_intervention_different_baseline/results
    - it will contain each model's performance in each dataset in each baseline
    - each sample's input and output will be saved in the corresponding json file

all of analysis code will be saved in /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/01_3_prelimilary_intervention_different_baseline/analysis

all runnable sh scripts will be saved in /HDD2/disheng/Spatial_VLM_Stereo/June/Spatial_VLM_Stereo/01_3_prelimilary_intervention_different_baseline/sh



---

## Implementation

At baseline `b` the model sees **one** image: the scene rendered from the
baseline-`b` viewpoint. `b = 0` is the untouched original view, i.e. the
no-intervention control. The question and the reference answer are identical
across baselines, so the per-sample answers line up directly and the failure
sets can be compared.

Baselines are keyed canonically as `0.00 0.05 0.10 0.15 0.20 0.25`. The
on-disk directories are spelled inconsistently (`0.10` vs `0.2`, and the VSR
annotation uses `0.20` for the correspondence but `0.2` for the image); all of
that is normalised in `dataset/baselines.py`, so any spelling can be passed on
the command line.

| split | n | image at baseline b | answer space |
|---|---|---|---|
| VSR (`V7_testing.json`) | 2195 | `00_data/gen_output_VSR/original_images` (b=0) / `.../generate_image/<b>` | True / False |
| Youtube level 1 | 821 | `level1/visualizations/<b>/` | A B C D |
| Youtube level 2 | 764 | `level2/visualizations/<b>/` | A B C D (option letter) |
| Youtube level 3 | 795 | `level3/visualizations/<b>/` | A B C D |

The Youtube visualisations are already rendered per baseline with the
annotation points warped into that view, so the marked points track the scene
across the sweep.

Level 2 is a four-option single choice: the question embeds four candidate
orderings as (A)-(D) and the answer is the option letter. Note that the image
labels its four *points* A-D as well, so a model may answer either with the
option letter ("C") or by writing out that option's ordering ("B > C > A > D").
Only the letter is the requested format, but the ordering is unambiguous, so it
is mapped back through the sample's own option table rather than scored wrong;
the two are told apart by whether the first standalone letter is followed by a
">". The loader detects the format from the records themselves, so the older
annotations that asked for the ordering directly (24 candidates, including the
`.ordering_bak` copies) still load and score correctly.

### Answering modes

`--scoring generate` (default, instruction-tuned models) greedily decodes and
parses the generation. `--scoring likelihood` (base models, which do not follow
the answer format) picks `argmax_c sum_t log P(c_t | prompt, c_<t)` over the
candidate answers, so every sample gets a well-defined answer and nothing
depends on output formatting. When every candidate is a single token -- which
is every split as the data currently stands -- this costs one forward pass per
sample. A multi-token candidate set (the legacy level-2 orderings) falls back to
one forward pass per candidate.

Qwen3.5 is a thinking model and its chat template opens a `<think>` block at
the generation prompt. Thinking is therefore **off** by default
(`--enable_thinking` to turn it on); it is rejected outright under
`--scoring likelihood`, where it would score the candidates as the opening
words of a reasoning trace rather than as the answer.

A generation that never reaches an answer normalises to something outside the
candidate set. Those count as wrong, but are also tallied as `num_unparsed` /
`unparsed_rate` in `metrics.json`, so a high parse-failure rate is visible
instead of being read as low accuracy. If a model's unparsed rate is high,
re-run it with `--scoring likelihood`.

### Layout

```
dataset/    baselines.py  canonical baseline naming
            datasets.py   per-baseline sample loading, prompts, answer parsing
inference/  run_inference.py   one model x one split x one baseline
analysis/   analyze_baseline_overlap.py
result/     <model_tag>/<task>/baseline_<b>/{predictions.jsonl,metrics.json,logs/}
sh/         run_all.sh     the whole grid (4 models x 4 splits x 6 baselines)
            run_sweep.sh   one model x one split, all baselines, sharded over GPUs
            run_analysis.sh
            merge_shards.py
```

`predictions.jsonl` holds one row per sample with the image path, the full
prompt, the raw model output, the parsed answer, the reference and correctness
(plus per-candidate log-probs under likelihood scoring).

### Running

```bash
./sh/run_all.sh                                     # everything, then the analysis

MODEL_ID=Qwen/Qwen2.5-VL-7B-Instruct ./sh/run_sweep.sh vsr
./sh/run_sweep.sh youtube 2
SCORING=likelihood MODEL_ID=Qwen/Qwen3.5-4B-Base ./sh/run_sweep.sh vsr

./sh/run_analysis.sh
```

Everything is resumable: a baseline whose `metrics.json` and
`predictions.jsonl` already exist is skipped (`SKIP_COMPLETED=false` to redo).
Each baseline runs data-parallel over `GPU_IDS` (default `0 1 2 3`), one
process per GPU, shards merged afterwards. Other knobs: `BASELINES`,
`BATCH_SIZE`, `MAX_NEW_TOKENS`, `IMAGE_MIN_PIXELS`/`IMAGE_MAX_PIXELS`,
`ENABLE_THINKING`, `DEBUG_EVAL`, `PYTHON_BIN`.

### What the analysis reports

Per (model, split), over the samples common to all baselines:

* **accuracy** per baseline and the spread across the sweep;
* **flip rate** — fraction of samples whose *answer* changes relative to
  baseline 0. This is the direct test of the premise: a visual tower that
  modelled the scene perfectly would answer identically at every baseline, so
  the flip rate measures the size of the effect independently of whether the
  answer was right to begin with;
* **stability** — always correct / always wrong / unstable;
* **failure overlap** — for every baseline pair, `|F_a ∩ F_b| / |F_a ∪ F_b|`
  over the failure sets, with raw counts and answer agreement. High overlap
  means the baselines fail on the *same* samples (a shared difficulty); low
  overlap means each baseline breaks a different subset, i.e. the failures are
  driven by the intervention itself.

Outputs land in `result/analysis/`: `analysis_summary.json`, one
`<model>__<task>_failure_overlap.csv` per run, and (with `--dump_per_sample`,
which `run_analysis.sh` passes) a per-sample answer/correctness matrix.

### Environment

All inference runs in `/home/disheng/miniconda3/envs/Spatial_VLM`
(python 3.10, torch 2.8, transformers 5.7). The interpreter is pinned in
`sh/env.sh`, which every script sources.

It is required, not merely preferred, and there is deliberately **no fallback
to `python`**: the base env has transformers 4.51, which has no `qwen3_5`
support, so a silent fallback would let the Qwen2.5 sweeps pass and then fail
partway through the grid. `sh/env.sh` therefore fails immediately if the
interpreter is missing, and `check_env` verifies torch/transformers import and
that `qwen3_5` is present before any model loads — once per `run_all.sh`, or
once per `run_sweep.sh` when run on its own. It prints, e.g.

```
env ok: python 3.10.19, torch 2.8.0+cu128, transformers 5.7.0, cuda yes
```

Override the interpreter with `PYTHON_BIN=/path/to/python`, or skip the check
with `CHECK_ENV=false`.