Update README.md

README.md

@@ -34,16 +34,13 @@ tags:
   <a href="#citation"><img src="https://img.shields.io/badge/Citation-BibTeX-blue" alt="BibTeX citation"></a>
 </p>
 
-## Overview
-
-This repository hosts the downstream MLLM evaluation checkpoints accompanying **[A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models](https://arxiv.org/abs/2610.05413)**.
-
-The study evaluates **70 vision encoders with three main language backbones**. The released checkpoints are organized by language backbone, vision encoder and training stage; links to the original frozen encoder weights appear alongside the corresponding MLLM weights below.
+# Overview
 
+This repository hosts the downstream MLLM checkpoints accompanying **[A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models](https://arxiv.org/abs/2610.05413)**.
 
 ## Files
 
-Checkpoints are grouped by language model, vision encoder, and training stage:
+Checkpoints are grouped by language model backbone, vision encoder, and training stage:
 
 ```text
 models/
@@ -53,71 +50,30 @@ models/
     finetune/                 # Model weights, config, and tokenizer files
 ```
 
-For example, the Qwen3 + CLIP-L/14 finetuned model is at `models/continuous/qwen3/clip_openai__l14/finetune/`. Use `qwen25`, `qwen3`, or `smollm2` for the three main language-model groups; the additional Base runs use `qwen3base`.
-
-## Download
-
-Download one encoder/model pair and its frozen vision encoder:
-
-```python
-from huggingface_hub import snapshot_download
-
-snapshot_download(
-    repo_id="336labs/VisionEncoder-to-MLLM-ModelZoo",
-    allow_patterns=["continuous/qwen3/clip_openai__l14/**"],
-    local_dir="models",
-)
-snapshot_download(
-    repo_id="openai/clip-vit-large-patch14",
-    local_dir="encoders/clip-vit-large-patch14",
-)
-```
-
-The local checkpoint directory is `models/continuous/qwen3/clip_openai__l14/finetune/`; the encoder directory is `encoders/clip-vit-large-patch14/`. Keep all files in the selected checkpoint directory, including any weight shards. Other frozen encoder downloads are listed [below](#vision-encoder-downloads).
-
-Load checkpoints through the [project code](https://github.com/JuntaoTang/MLLM-VisionEncoder-Eval) using your local encoder directory. Saved model configurations may contain paths from the training machine. A pretrain projector also needs the matching original language model; the finetune directory contains the trained MLLM weights.
-
-## Training and evaluation data
-
-A simple local data layout is:
-
-```text
-data/
-  instructions/pretrain/blip_laion_cc_sbu_558k.json
-  instructions/finetune/llava_v1_5_mix665k_drop_ge8kchars.json
-  instructions/test/<dataset>.tsv
-  images/pretrain/
-  images/finetune/
-  images/test/
-```
-
-Raw images and annotations are downloaded separately. See the [data sources and 11 evaluation benchmarks](https://huggingface.co/datasets/336labs/VisionEncoder-Eval-ReproData#datasets-required-by-the-evaluation-pipelines). Configuration and experiment commands are described in the code repository's [Setup](https://github.com/JuntaoTang/MLLM-VisionEncoder-Eval#setup) and [Running](https://github.com/JuntaoTang/MLLM-VisionEncoder-Eval#running) sections.
+Use `qwen25`, `qwen3`, or `smollm2` for the three main language-model groups.
+For example, the Qwen3 + CLIP-L/14 finetuned model is at `models/continuous/qwen3/clip_openai__l14/finetune/`. 
 
-## Model Zoo
+# Model Zoo
 
 This Model Zoo covers the **70 visual tokenizers used in our paper**: **43 language-supervised**, **22 self-supervised**, and **5 discrete** tokenizers. 
-The current release provides both pretraining and finetuning checkpoints for all **65 continuous tokenizers** with each of the three main language backbones, plus **2** Qwen3-1.7B-Base runs. The **5 discrete tokenizers** are listed separately with their checkpoint availability. 
+The current release provides both pretraining and finetuning checkpoints for all **65 continuous tokenizers** with each of the three main language backbones. The **5 discrete tokenizers** are listed separately with their checkpoint availability. 
 
-### Training data and checkpoint types
+## Training data
 
 - **Pretrain Data — LCS-558K:** the image–text alignment dataset from [LLaVA-Pretrain](https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain), using `blip_laion_cc_sbu_558k.json`. This dataset is used for MLLM projector training.
 - **Finetuning Data — LLaVA-v1.5 mix665k (filtered):** the [LLaVA-v1.5 instruction mixture](https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K/blob/main/llava_v1_5_mix665k.json), using `llava_v1_5_mix665k_drop_ge8kchars.json`. The training configuration removes 395 examples with at least 8,000 characters of conversation text.
-- **Download:** `projector` downloads the pretraining `mm_projector.bin`; `finetuned` opens the finetuned checkpoint directory, including model weights, configuration, and language-tokenizer files. The frozen vision encoder must be supplied separately using the matching architecture and weights recorded in `config.json`. Pretraining projector weights alone are not instruction-tuned MLLMs.
-
-### Vision encoder downloads
 
-The **Encoder weights** column links to the original upstream vision encoder or visual tokenizer. Each encoder name links to its model or project page. Use these frozen encoder weights together with the corresponding Stage 1 projector or Stage 2 MLLM checkpoint, preserving the architecture, feature layer and preprocessing specified by that run's `config.json`.
 
-- For Hugging Face models, retain the associated configuration and processor files. **Web-SSL MAE 3B** has sharded weights; its link opens the complete model repository.
-- **DINOv3** requires acceptance of the upstream model's terms and authentication. The **RAEv2 DINOv3-L (k=7)** entry uses the same DINOv3-L/16 backbone with the project's k=7 intermediate-layer readout; its download is the backbone used to construct that representation.
-- **TokLIP-S/L** require both the encoder checkpoint and the linked **VQ** checkpoint. For **UniAR** and **VILA-U**, retain the config files alongside the weights in the respective `bsq_encoder/` and `vision_tower/` directories.
+## Downloads
+- 'Encoder weights' links to the original upstream vision encoder or visual tokenizer. Each encoder name links to its model or project page. Use these frozen encoder weights together with the corresponding Stage 1 projector or Stage 2 MLLM checkpoint, preserving the architecture, feature layer and preprocessing specified by that run's `config.json`.
+  - For Hugging Face models, retain the associated configuration and processor files. **Web-SSL MAE 3B** has sharded weights; its link opens the complete model repository.
+  - **DINOv3** requires acceptance of the upstream model's terms and authentication. The **RAEv2 DINOv3-L (k=7)** entry uses the same DINOv3-L/16 backbone with the project's k=7 intermediate-layer readout; its download is the backbone used to construct that representation.
+  - **TokLIP-S/L** require both the encoder checkpoint and the linked **VQ** checkpoint. For **UniAR** and **VILA-U**, retain the config files alongside the weights in the respective `bsq_encoder/` and `vision_tower/` directories.
+- `projector` downloads the pretraining `mm_projector.bin`;
+- `finetuned` opens the finetuned checkpoint directory, including model weights, configuration, and language-tokenizer files. The frozen vision encoder must be supplied separately using the matching architecture and weights recorded in `config.json`. Pretraining projector weights alone are not instruction-tuned MLLMs.
 
 ### Qwen2.5-1.5B-Instruct
-
-Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct). **65 tokenizers**, each with pretraining and finetuning checkpoints.
-
-<details>
-<summary>Show 65 tokenizers and download links</summary>
+Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
 
 | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
 | --- | --- | --- | --- | --- |
@@ -187,14 +143,10 @@ Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-
 | Qwen2.5-1.5B-Instruct | [DINO ViT-S/8](https://huggingface.co/facebook/dino-vits8) | [encoder](https://huggingface.co/facebook/dino-vits8/resolve/main/model.safetensors?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen25/dino_vits8/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen25/dino_vits8/finetune) |
 | Qwen2.5-1.5B-Instruct | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen25/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen25/eupe_vit_t/finetune) |
 
-</details>
 
 ### Qwen3-1.7B
 
-Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B). **65 tokenizers**, each with pretraining and finetuning checkpoints.
-
-<details>
-<summary>Show 65 tokenizers and download links</summary>
+Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).
 
 | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
 | --- | --- | --- | --- | --- |
@@ -264,14 +216,10 @@ Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B). **65 tokeni
 | Qwen3-1.7B | [DINO ViT-S/8](https://huggingface.co/facebook/dino-vits8) | [encoder](https://huggingface.co/facebook/dino-vits8/resolve/main/model.safetensors?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen3/dino_vits8/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen3/dino_vits8/finetune) |
 | Qwen3-1.7B | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen3/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen3/eupe_vit_t/finetune) |
 
-</details>
 
 ### SmolLM2-1.7B-Instruct
 
-Base LLM: [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct). **65 tokenizers**, each with pretraining and finetuning checkpoints.
-
-<details>
-<summary>Show 65 tokenizers and download links</summary>
+Base LLM: [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct).
 
 | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
 | --- | --- | --- | --- | --- |
@@ -341,17 +289,6 @@ Base LLM: [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFa
 | SmolLM2-1.7B-Instruct | [DINO ViT-S/8](https://huggingface.co/facebook/dino-vits8) | [encoder](https://huggingface.co/facebook/dino-vits8/resolve/main/model.safetensors?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/smollm2/dino_vits8/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/smollm2/dino_vits8/finetune) |
 | SmolLM2-1.7B-Instruct | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/smollm2/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/smollm2/eupe_vit_t/finetune) |
 
-</details>
-
-### Discrete tokenizers in the paper
-
-| Vision Tokenizer | Encoder weights | MLLM checkpoint availability |
-| --- | --- | --- |
-| [UniTok-Attn (256)](https://huggingface.co/FoundationVision/unitok_tokenizer) | [encoder](https://huggingface.co/FoundationVision/unitok_tokenizer/resolve/main/unitok_tokenizer.pth?download=true) | Not released in this repository |
-| [UniAR-BSQ](https://huggingface.co/ShareLab-SII/UniAR-SFT/tree/main/bsq_encoder) | [encoder](https://huggingface.co/ShareLab-SII/UniAR-SFT/resolve/main/bsq_encoder/model.safetensors?download=true) | Not released in this repository |
-| [TokLIP-L (384)](https://huggingface.co/TencentARC/TokLIP) | [encoder](https://huggingface.co/TencentARC/TokLIP/resolve/main/TokLIP_L_384.pt?download=true) · [VQ](https://huggingface.co/peizesun/llamagen_t2i/resolve/main/vq_ds16_t2i.pt?download=true) | Not released in this repository |
-| [TokLIP-S (256)](https://huggingface.co/TencentARC/TokLIP) | [encoder](https://huggingface.co/TencentARC/TokLIP/resolve/main/TokLIP_S_256.pt?download=true) · [VQ](https://huggingface.co/peizesun/llamagen_t2i/resolve/main/vq_ds16_t2i.pt?download=true) | Not released in this repository |
-| [VILA-U (256)](https://huggingface.co/mit-han-lab/vila-u-7b-256) | [encoder](https://huggingface.co/mit-han-lab/vila-u-7b-256/resolve/main/vision_tower/model.safetensors?download=true) | Not released in this repository |
 
 ## Citation