| --- |
| license: apache-2.0 |
| tags: |
| - vision-language |
| - multimodal |
| - visual-reasoning |
| - self-evolving |
| - self-distillation |
| --- |
| |
| <div align="center"> |
|
|
| # SelfThinker Data: Reliability-Verified Visual Thought Pools & Benchmark Suite |
|
|
| [](https://huggingface.co/jankin123/SelfThinker-7B) |
| [](https://huggingface.co/jankin123/SelfThinker) |
| [](https://github.com/zhangquanchen/SelfThinker) |
|
|
| </div> |
|
|
| This repository contains the reliability-verified intrinsic visual-thought training sets and the benchmark TSV files for training and evaluating **SelfThinker**. |
|
|
| - **Models:** [https://huggingface.co/jankin123/SelfThinker-7B](https://huggingface.co/jankin123/SelfThinker-7B) |
| - **Data:** [https://huggingface.co/jankin123/SelfThinker](https://huggingface.co/jankin123/SelfThinker) |
| - **Code:** [https://github.com/zhangquanchen/SelfThinker](https://github.com/zhangquanchen/SelfThinker) |
|
|
| <p align="center"> |
| <img src="assets/Fig5.png" width="100%" alt="Reliability-Verified Visual Thoughts on Navigation and Visual Search"/> |
| </p> |
|
|
| --- |
|
|
| ## 1. Reliability-Verified Training Sets |
|
|
| Synthesized multi-object visual-search problems (`384x384` px canvases with 5–8 non-overlapping shapes across 5 shapes and 8 colors) where candidate intrinsic visual thoughts $\{v^{(k)}\}_{k=1}^{6}$ are scored by conditional pointwise mutual information: |
| |
| $$u(v) = \log p_\theta\big(d \mid C(I, q, v)\big) - \log p_\theta\big(d \mid C(I, q)\big)$$ |
| |
| and retained when the top candidate $v^\star$ exceeds the item-specific 95th-percentile mismatched threshold $Q_{0.95}(s_i)$. |
| |
| | Directory | Verification Direction $d$ | Verified Items | Parquet File | Command | |
| |---|---|---:|---|---| |
| | `audit_pass_l0_vsearch_trainonly/` | Ground-truth answer $a^\star$ | 1,092 / 1,500 | `Visual_Search/data/train-00000-of-00001.parquet` (111.7 MB) | `scripts/train_selfthinker.sh answer` | |
| | `audit_pass_l1_vsearch_trainonly/` | Label-free rollout majority vote $\tilde{a}$ | 1,074 / 1,500 | `Visual_Search/data/train-00000-of-00001.parquet` (107.4 MB) | `scripts/train_selfthinker.sh vote` | |
| |
| <p align="center"> |
| <img src="assets/Fig7.png" width="100%" alt="Answer Consensus vs Visual Consensus on Visual Search"/> |
| </p> |
| |
| --- |
| |
| ## 2. Benchmark Evaluation Suite (`lmudata/`) |
| |
| `lmudata/` provides all self-contained TSV files (with embedded base64 images) used by the seven-benchmark evaluation script (`scripts/eval_nine.sh`) and the multi-layer image-condition probe (`scripts/eval_image.sh`): |
| |
| | Benchmark | TSV File(s) | Items | Size | Aggregation Rule | |
| |---|---|---:|---:|---| |
| | **VSP** | `lmudata/VSP_maze_task_main_original.tsv` | 600 | 144.3 MB | Main maze task accuracy | |
| | **ChartQA** | `lmudata/ChartQA_h_bar.tsv`, `lmudata/ChartQA_v_bar.tsv` | 444 + 382 | 35.7 MB, 19.7 MB | Mean of horizontal and vertical bar subsets | |
| | **V\*** | `lmudata/VStarBench.tsv` | 191 | 162.0 MB | Direct accuracy | |
| | **MMVP** | `lmudata/MMVP.tsv` | 300 | 4.2 MB | Direct accuracy | |
| | **SAT** | `lmudata/SAT_circular.tsv` | 150 | 354.1 MB | Circular evaluation split accuracy | |
| | **BLINK** | `lmudata/BLINK.tsv` | 1,901 | 538.4 MB | Overall accuracy across 14 subtasks | |
| | **CV-Bench** | `lmudata/CV-Bench-2D.tsv`, `lmudata/CV-Bench-3D.tsv` | 1,438 + 1,200 | 139.9 MB, 158.2 MB | Mean of 2D and 3D subsets | |
|
|
| <p align="center"> |
| <img src="assets/Fig10.png" width="100%" alt="Conditions of the Multi-Layer Image-Condition Probe"/> |
| </p> |
|
|
| --- |
|
|
| ## Download & Usage |
|
|
| ```bash |
| git clone https://github.com/zhangquanchen/SelfThinker.git |
| cd SelfThinker |
| |
| huggingface-cli download jankin123/SelfThinker --local-dir data |
| huggingface-cli download jankin123/SelfThinker-7B --local-dir models |
| ``` |
|
|
