RuipengZhang/TLVS

configuration:

• one lam_max
• one layer list (--layers)
• no sweep loops

It supports:

• cache-hit path using prompt-only build + cached pixel_values
• fallback to online image preprocess when cache is missing or prompt-only fails
• optional trace dumping

---

Usage

python amber_klgate_single.py

Common overrides:

python llava_steering.py \
  --model-path liuhaotian/llava-v1.5-7b \
  --question-file data/AMBER/query_all.json \
  --image-folder data/AMBER/images \
  --image-cache-folder data/AMBER/cache_pixel_values \
  --output-dir outputs/amber_llava_klgate \
  --probe-path probes/delta_pca_as_binary_style.npz \
  --lam-max 1.5 \
  --layers "" \
  --vs-mode coupled

Enable debug logs for the first few samples:

python llava_steering.py --debug --debug-first-n 2 --log-every 10

Save per-token trace (large output):

python llava_steering.py --save-trace

---

Inputs

Questions JSON

--question-file should be a JSON list of objects like:

[
  {"id": "1", "image": "xxx.jpg", "query": "Describe the image."}
]

Images

--image-folder/<image> must exist.

Optional cache

--image-cache-folder/<image>.pt

The cache file must be a torch.Tensor on CPU:

• shape [3, H, W] (or [1, 3, H, W] will be auto-squeezed)

---

Output

The script writes one JSON file:

amber_klgate_lam{LAM}_layers{LAYER_TAG}.json

Format:

[
  {"id": "1", "response": "...", "trace": [...]}
]

If --save-trace is not set, each item only contains:

{"id": "...", "response": "..."}

---

Notes

• Cache-hit path requires a prompt-only builder:

  • preferred: rt.build_inputs(..., skip_image_preprocess=True)
  • otherwise: internal fallback hooks are attempted
  • if none exist, it will raise a RuntimeError with guidance

• vs_mode=coupled runs at least 2 forwards per token step (img + no-img). vs_mode=decoupled adds an extra img forward for KL computation.

---