Buckets:
| # VLMEvalKit for Thinkmorph | |
| This repository provides evaluation support for the [ThinkMorph](https://arxiv.org/pdf/2510.27492) model based on [VLMEvalKit](https://github.com/open-compass/VLMEvalKit). And this repo also supports all the benchmarks evaluated in our paper, including: `VSP`, `VisPuzzle`, `ChartQA`, `VStar`, `BLINK-J`, `MMVP`, `SAT`, `BLINK`, and `CV-Bench`. | |
| ## Installation | |
| ### 1. Clone the Repository | |
| ``` | |
| git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git | |
| cd VLMEvalKit_Thinkmorph | |
| ``` | |
| ### 2. Install Dependencies | |
| ``` | |
| pip install -r requirements.txt | |
| ``` | |
| ### 3. Install ThinkMorph Dependencies | |
| Make sure you have the necessary dependencies for ThinkMorph | |
| For the local UniAR adapter, use the separate `ThinkMorph-Eval` environment | |
| with Torch 2.7.1+cu128 and install `OURS[train,decoder]` from the surrounding | |
| Spatial-MMCoT workspace. The compatible evaluation dependency set is pinned in | |
| `requirements.txt`; `pip check` must pass after installation. | |
| ## Quick Start | |
| 1️⃣ Configure API Keys | |
| ``` | |
| export OPENAI_API_KEY="your_api_key_here" | |
| ``` | |
| 2️⃣ Edit the model configuration in [`vlmeval/config.py`](vlmeval/config.py): | |
| ``` | |
| thinkmorph_series = { | |
| "thinkmorph": partial( | |
| ThinkMorph, | |
| model_path="ThinkMorph/ThinkMorph-7B", | |
| think=True, # Enable thinking mode | |
| understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images | |
| temperature=0.3, | |
| max_think_token_n=4096, | |
| save_dir="path_to_your_imgs_dir" # Directory to save generated images | |
| ), | |
| } | |
| ``` | |
| 2️⃣ Choose the benchmark you wanna eval, and run. | |
| An example script is shown in [run_thinkmorph.sh](run_thinkmorph.sh). | |
| If you wanna try more benchmark, check [VLMEvalKit Features](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb&view=vewUH3kRGs) | |
| ``` | |
| python run.py \ | |
| --data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \ | |
| --model thinkmorph \ | |
| --judge gpt-5 \ | |
| --work-dir ./results | |
| ``` | |
| ### UniAR BLINK-Jigsaw | |
| The registered models cover base UniAR-SFT and the completed rank-64 two- and | |
| four-epoch ThinkMorph adapters. They add only the training protocol as a system | |
| prompt; the VLMEvalKit user message is passed through without modification. | |
| Run the 150-row benchmark without an API judge using: | |
| ```bash | |
| LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \ | |
| SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \ | |
| --data BLINK_Jigsaw \ | |
| --model uniar-thinkmorph-v1-vanilla-r64-val5-2epoch \ | |
| --judge exact_matching --reuse --work-dir <output-dir> | |
| ``` | |
| The exact matcher gives priority to a complete structured answer, otherwise | |
| matches an explicit option letter or an exact choice value only on the final | |
| non-empty output line, and then falls back to the conservative VLMEvalKit | |
| rules. It strips serialized ChatML control tokens and does not use an API or | |
| semantic judge. To update derived | |
| metrics after a matcher repair without rerunning inference: | |
| ```bash | |
| LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \ | |
| PYTHONPATH=. python scripts/rescore_exact_matching.py \ | |
| --dataset VStarBench --eval-file <raw-predictions.xlsx> | |
| ``` | |
| This atomically replaces only the derived result/accuracy files and records a | |
| hash-bound `*_exact_matching_verifier.json`; the raw prediction workbook is | |
| unchanged. | |
Xet Storage Details
- Size:
- 3.48 kB
- Xet hash:
- 9bbd8b474f3d829ffe97345857c12320712c0447c304c43b06ea12a12b09ddc4
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.