task47_0809/task8_final_with_diffusion_bank/setup.md

Task 8: setup and run

What this package contains

• scripts/: the Task 8 evaluator, official ImageNet export helper, checkpoint downloader, and aggregate-plot script.
• results/: final outputs from the completed 1,000-image Task 8 run.

Environment

Run from the WorldModelLens repository root in a CUDA-enabled PyTorch environment with torchvision installed. The Task 8 script imports the WorldModelLens library from that repository; the library is not copied here.

Checkpoint and images

Use the official I-JEPA ViT-H/14 ImageNet-1K checkpoint at checkpoints/vith14_in1k_ep300.pth.tar. The completed run used the authenticated Hugging Face ILSVRC/imagenet-1k validation split, exported locally as image files.

Run

python scripts/task8_ijepa_ema_ambiguity.py \
  --ijepa-checkpoint /path/to/vith14_in1k_ep300.pth.tar \
  --imagenet-val /path/to/imagenet/val \
  --image-manifest /path/to/fixed_1000_image_manifest.json \
  --inpainting-bank /path/to/inpainting_bank \
  --n-validation 1000 --batch-size 8 --seed 42 --cka-depths all \
  --device cuda --output-dir results

The final run compared frozen context and EMA target encoder outputs on the same full image and corresponding patches. It reports global CKA and per-sample patch-token CKA, cosine, raw L2, and normalized L2 across all 32 layers. The ambiguity-bank files are synthetic smoke-test artifacts and are not scientific inpainting results.

Build a real controlled-ambiguity bank

Use scripts/build_task8_diffusion_bank.py on the cluster before rerunning the ambiguity analysis. It requires diffusers, transformers, accelerate, and safetensors, plus access to the selected Hugging Face inpainting checkpoint. Diffusers recommends using an inpainting-fine-tuned checkpoint rather than a plain text-to-image checkpoint; the script defaults to stabilityai/stable-diffusion-2-inpainting but records any supplied model ID in every case metadata file.

Prepare a reviewed, pre-registered JSON case specification. Each case must have a unique ID, a real ImageNet source path, a mask represented by ViT-H/14 row-major target patch IDs, three distinct structural-completion prompts, and a human-assigned context-consistent candidate index. The label must be assigned independently of I-JEPA scores.

[
  {
    "case_id": "case_0001",
    "source": "/datasets/official_imagenet1k_validation/class_0815/imagenet_val_13512.jpg",
    "target_patch_ids": [85, 86, 87, 101, 102, 103],
    "candidate_prompts": [
      "a realistic continuation with a small brown bird",
      "a realistic continuation with a red flower",
      "a realistic continuation with green foliage"
    ],
    "negative_prompt": "blurry, distorted, low quality, text, watermark",
    "candidate_seeds": [5001, 5002, 5003],
    "context_consistent_candidate": 0
  }
]

Generate the bank (the destination must be new or empty):

pip install 'diffusers[torch]' transformers accelerate safetensors

python scripts/build_task8_diffusion_bank.py \
  --case-spec /datasets/task8_cases_30.json \
  --dest /datasets/task8_real_inpainting_bank_30 \
  --model-id stabilityai/stable-diffusion-2-inpainting \
  --device cuda --steps 30 --guidance-scale 7.5 --seed 42

The generator makes source.png, three candidate_*.png files, and metadata.json for every case. It composites generated pixels only into the masked patches, guaranteeing identical visible context across candidates. The root bank_manifest.json records all sources, prompts, labels, seeds, and diffusion settings. Do not add image files other than source.png and the three candidates inside a case directory because the evaluator expects exactly those four images.

The evaluator carries context_consistent_candidate into task8_metrics.json and reports labelled-case predictor top-1 accuracy with a bootstrap confidence interval under predictor_ranking_inference. It is a comparison against chance accuracy of one third; it does not replace human review of generation quality.

The evaluator first uses the repository's world_model_lens.data.load_imagenet_image when available. It now includes an equivalent resize-shorter-side/centre-crop ImageNet-normalization fallback, so the Task 8 package remains runnable if that optional module is absent from the cluster checkout.