EmergentTwilight/MMCoT/projects/VLMEvalKit_Thinkmorph_llm - Storage Bucket

VLMEvalKit for Thinkmorph

This repository provides evaluation support for the ThinkMorph model based on VLMEvalKit. And this repo also supports all the benchmarks evaluated in our paper, including: VSP, VisPuzzle, ChartQA, VStar, BLINK-J, MMVP, SAT, BLINK, and CV-Bench.

Installation

1. Clone the Repository

git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git
cd VLMEvalKit_Thinkmorph

2. Install Dependencies

pip install -r requirements.txt

3. Install ThinkMorph Dependencies

Make sure you have the necessary dependencies for ThinkMorph

For UniAR evaluation, use the separate ThinkMorph-Eval environment with Torch 2.7.1+cu128 and install OURS[train,decoder] from the surrounding Spatial-MMCoT workspace. The UniAR adapter adds only the canonical system protocol and preserves the framework-generated user prompt byte-for-byte.

Quick Start

1️⃣ Configure API Keys

export OPENAI_API_KEY="your_api_key_here"

2️⃣ Edit the model configuration in vlmeval/config.py:

thinkmorph_series = {
    "thinkmorph": partial(
        ThinkMorph, 
        model_path="ThinkMorph/ThinkMorph-7B", 
        think=True, # Enable thinking mode
        understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images
        temperature=0.3, 
        max_think_token_n=4096, 
        save_dir="path_to_your_imgs_dir" # Directory to save generated images
    ),
}

2️⃣ Choose the benchmark you wanna eval, and run.

An example script is shown in run_thinkmorph.sh. If you wanna try more benchmark, check VLMEvalKit Features

python run.py \
    --data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \
    --model thinkmorph \
    --judge gpt-5 \
    --work-dir ./results

UniAR with DeepSeek judge

The registered UniAR models cover the base checkpoint, the rank-64 two-/four- epoch adapters, the BitCE weight-1/weight-3 ablations, and the absolute-IG two-epoch adapter. This fork preserves VLMEvalKit's original verifier and uses deepseek-v4-flash only when rule-based prefetch cannot extract an answer.

export DEEPSEEK_API_KEY="$(<API_KEY)"
LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \
SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \
  --data BLINK_Jigsaw \
  --model uniar-thinkmorph-v1-ig-abs-r64-val5-2epoch \
  --judge deepseek-v4-flash --reuse --work-dir <output-dir>

API_KEY is ignored by Git. Prefer DEEPSEEK_API_KEY in managed jobs; the UniAR runner may load the ignored file through DEEPSEEK_API_KEY_FILE.

The UniAR runner also sets UNIAR_ROLLOUT_DIR=<output>/rollouts. Each generated sample atomically saves prompt/assistant/full token IDs, observation and generated visual codes, grids, event metadata, decoded token text, and a COMPLETE marker. RGB decoding is intentionally deferred; the workbook's extra_records column links each prediction to its rollout directory.

Interactive retries set the stable namespace VLMEVAL_EVAL_ID=RUN, so date or live Git changes cannot redirect rank shards into a new prediction directory. VLMEVAL_DUMP_EVERY=1 commits the rank pickle after every generated sample; the upstream evaluator and verifier are unchanged. VLMEVAL_SCHEDULING=dynamic uses one distributed atomic counter: every free worker claims the next remaining query, while retry startup merges committed rank pickles before rebuilding the queue. Model generation and scoring are unchanged.

To replace scores from the retired custom verifier without rerunning model inference, point the native LLM verifier at the existing raw workbook:

python scripts/rescore_with_llm_judge.py --dataset MMVP \
  --eval-file <model>_MMVP.xlsx --judge deepseek-v4-flash --nproc 24

This preserves the prediction workbook, replaces _acc.csv, removes stale exact_matching derivatives, and publishes DeepSeek result/provenance files in the same evaluation directory.