projects/VLMEvalKit_Thinkmorph/README.md

VLMEvalKit for Thinkmorph

This repository provides evaluation support for the ThinkMorph model based on VLMEvalKit. And this repo also supports all the benchmarks evaluated in our paper, including: VSP, VisPuzzle, ChartQA, VStar, BLINK-J, MMVP, SAT, BLINK, and CV-Bench.

Installation

1. Clone the Repository

git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git
cd VLMEvalKit_Thinkmorph

2. Install Dependencies

pip install -r requirements.txt

3. Install ThinkMorph Dependencies

Make sure you have the necessary dependencies for ThinkMorph

For the local UniAR adapter, use the separate ThinkMorph-Eval environment with Torch 2.7.1+cu128 and install OURS[train,decoder] from the surrounding Spatial-MMCoT workspace. The compatible evaluation dependency set is pinned in requirements.txt; pip check must pass after installation.

Quick Start

1️⃣ Configure API Keys

export OPENAI_API_KEY="your_api_key_here"

2️⃣ Edit the model configuration in vlmeval/config.py:

thinkmorph_series = {
    "thinkmorph": partial(
        ThinkMorph, 
        model_path="ThinkMorph/ThinkMorph-7B", 
        think=True, # Enable thinking mode
        understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images
        temperature=0.3, 
        max_think_token_n=4096, 
        save_dir="path_to_your_imgs_dir" # Directory to save generated images
    ),
}

2️⃣ Choose the benchmark you wanna eval, and run.

An example script is shown in run_thinkmorph.sh. If you wanna try more benchmark, check VLMEvalKit Features

python run.py \
    --data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \
    --model thinkmorph \
    --judge gpt-5 \
    --work-dir ./results

UniAR BLINK-Jigsaw

The registered models cover base UniAR-SFT and the completed rank-64 two- and four-epoch ThinkMorph adapters. They add only the training protocol as a system prompt; the VLMEvalKit user message is passed through without modification. Run the 150-row benchmark without an API judge using:

LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \
SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \
  --data BLINK_Jigsaw \
  --model uniar-thinkmorph-v1-vanilla-r64-val5-2epoch \
  --judge exact_matching --reuse --work-dir <output-dir>

The exact matcher gives priority to a complete structured answer, otherwise matches an explicit option letter or an exact choice value only on the final non-empty output line, and then falls back to the conservative VLMEvalKit rules. It strips serialized ChatML control tokens and does not use an API or semantic judge. To update derived metrics after a matcher repair without rerunning inference:

LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \
PYTHONPATH=. python scripts/rescore_exact_matching.py \
  --dataset VStarBench --eval-file <raw-predictions.xlsx>

This atomically replaces only the derived result/accuracy files and records a hash-bound *_exact_matching_verifier.json; the raw prediction workbook is unchanged.