Buckets:
| # VLMEvalKit for Thinkmorph | |
| This repository provides evaluation support for the [ThinkMorph](https://arxiv.org/pdf/2510.27492) model based on [VLMEvalKit](https://github.com/open-compass/VLMEvalKit). And this repo also supports all the benchmarks evaluated in our paper, including: `VSP`, `VisPuzzle`, `ChartQA`, `VStar`, `BLINK-J`, `MMVP`, `SAT`, `BLINK`, and `CV-Bench`. | |
| ## Installation | |
| ### 1. Clone the Repository | |
| ``` | |
| git clone https://github.com/hychaochao/VLMEvalKit_Thinkmorph.git | |
| cd VLMEvalKit_Thinkmorph | |
| ``` | |
| ### 2. Install Dependencies | |
| ``` | |
| pip install -r requirements.txt | |
| ``` | |
| ### 3. Install ThinkMorph Dependencies | |
| Make sure you have the necessary dependencies for ThinkMorph | |
| For UniAR evaluation, use the separate `ThinkMorph-Eval` environment with | |
| Torch 2.7.1+cu128 and install `OURS[train,decoder]` from the surrounding | |
| Spatial-MMCoT workspace. The UniAR adapter adds only the canonical system | |
| protocol and preserves the framework-generated user prompt byte-for-byte. | |
| ## Quick Start | |
| 1️⃣ Configure API Keys | |
| ``` | |
| export OPENAI_API_KEY="your_api_key_here" | |
| ``` | |
| 2️⃣ Edit the model configuration in [`vlmeval/config.py`](vlmeval/config.py): | |
| ``` | |
| thinkmorph_series = { | |
| "thinkmorph": partial( | |
| ThinkMorph, | |
| model_path="ThinkMorph/ThinkMorph-7B", | |
| think=True, # Enable thinking mode | |
| understanding_output=False, # If `False`, enables interleaved reasoning and requires `save_dir` to save generated images | |
| temperature=0.3, | |
| max_think_token_n=4096, | |
| save_dir="path_to_your_imgs_dir" # Directory to save generated images | |
| ), | |
| } | |
| ``` | |
| 2️⃣ Choose the benchmark you wanna eval, and run. | |
| An example script is shown in [run_thinkmorph.sh](run_thinkmorph.sh). | |
| If you wanna try more benchmark, check [VLMEvalKit Features](https://aicarrier.feishu.cn/wiki/Qp7wwSzQ9iK1Y6kNUJVcr6zTnPe?table=tblsdEpLieDoCxtb&view=vewUH3kRGs) | |
| ``` | |
| python run.py \ | |
| --data VSP_maze_task_main_original VisPuzzle ChartQA_h_bar ChartQA_v_bar VStarBench BLINK_Jigsaw MMVP BLINK SAT_circular CV-Bench-2D CV-Bench-3D \ | |
| --model thinkmorph \ | |
| --judge gpt-5 \ | |
| --work-dir ./results | |
| ``` | |
| ### UniAR with DeepSeek judge | |
| The registered UniAR models cover the base checkpoint, the rank-64 two-/four- | |
| epoch adapters, the BitCE weight-1/weight-3 ablations, and the absolute-IG | |
| two-epoch adapter. This fork preserves VLMEvalKit's original verifier and uses | |
| `deepseek-v4-flash` only when rule-based prefetch cannot extract an answer. | |
| ```bash | |
| export DEEPSEEK_API_KEY="$(<API_KEY)" | |
| LMUData=/work/nvme/bgjw/tcheng2/MMCoT/data/VLMEvalKit \ | |
| SPLIT_THINK=1 torchrun --standalone --nproc-per-node=4 run.py \ | |
| --data BLINK_Jigsaw \ | |
| --model uniar-thinkmorph-v1-ig-abs-r64-val5-2epoch \ | |
| --judge deepseek-v4-flash --reuse --work-dir <output-dir> | |
| ``` | |
| `API_KEY` is ignored by Git. Prefer `DEEPSEEK_API_KEY` in managed jobs; the | |
| UniAR runner may load the ignored file through `DEEPSEEK_API_KEY_FILE`. | |
| The UniAR runner also sets `UNIAR_ROLLOUT_DIR=<output>/rollouts`. Each generated | |
| sample atomically saves prompt/assistant/full token IDs, observation and | |
| generated visual codes, grids, event metadata, decoded token text, and a | |
| `COMPLETE` marker. RGB decoding is intentionally deferred; the workbook's | |
| `extra_records` column links each prediction to its rollout directory. | |
| Interactive retries set the stable namespace `VLMEVAL_EVAL_ID=RUN`, so date or | |
| live Git changes cannot redirect rank shards into a new prediction directory. | |
| `VLMEVAL_DUMP_EVERY=1` commits the rank pickle after every generated sample; | |
| the upstream evaluator and verifier are unchanged. | |
| `VLMEVAL_SCHEDULING=dynamic` uses one distributed atomic counter: every free | |
| worker claims the next remaining query, while retry startup merges committed | |
| rank pickles before rebuilding the queue. Model generation and scoring are | |
| unchanged. | |
| To replace scores from the retired custom verifier without rerunning model | |
| inference, point the native LLM verifier at the existing raw workbook: | |
| ```bash | |
| python scripts/rescore_with_llm_judge.py --dataset MMVP \ | |
| --eval-file <model>_MMVP.xlsx --judge deepseek-v4-flash --nproc 24 | |
| ``` | |
| This preserves the prediction workbook, replaces `_acc.csv`, removes stale | |
| `exact_matching` derivatives, and publishes DeepSeek result/provenance files | |
| in the same evaluation directory. | |
Xet Storage Details
- Size:
- 4.32 kB
- Xet hash:
- 169d19af640ae3d2cac9d311fe20f353709c18b17044d3afe782dbbc1095b2ee
·
Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.