JuntaoTang commited on
Commit
8ef427b
·
verified ·
1 Parent(s): 917313f

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +6 -27
README.md CHANGED
@@ -38,25 +38,10 @@ tags:
38
 
39
  This repository hosts the downstream MLLM checkpoints accompanying **[A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models](https://arxiv.org/abs/2610.05413)**.
40
 
41
- ## Files
42
-
43
- Checkpoints are grouped by language model backbone, vision encoder, and training stage:
44
-
45
- ```text
46
- models/
47
- continuous/<llm>/<encoder>/
48
- pretrain/mm_projector.bin
49
- pretrain/config.json
50
- finetune/ # Model weights, config, and tokenizer files
51
- ```
52
-
53
- Use `qwen25`, `qwen3`, or `smollm2` for the three main language-model groups.
54
- For example, the Qwen3 + CLIP-L/14 finetuned model is at `models/continuous/qwen3/clip_openai__l14/finetune/`.
55
-
56
  # Model Zoo
57
 
58
- This Model Zoo covers the **70 visual tokenizers used in our paper**: **43 language-supervised**, **22 self-supervised**, and **5 discrete** tokenizers.
59
- The current release provides both pretraining and finetuning checkpoints for all **65 continuous tokenizers** with each of the three main language backbones. The **5 discrete tokenizers** are listed separately with their checkpoint availability.
60
 
61
  ## Training data
62
 
@@ -65,15 +50,13 @@ The current release provides both pretraining and finetuning checkpoints for all
65
 
66
 
67
  ## Downloads
68
- - 'Encoder weights' links to the original upstream vision encoder or visual tokenizer. Each encoder name links to its model or project page. Use these frozen encoder weights together with the corresponding Stage 1 projector or Stage 2 MLLM checkpoint, preserving the architecture, feature layer and preprocessing specified by that run's `config.json`.
69
  - For Hugging Face models, retain the associated configuration and processor files. **Web-SSL MAE 3B** has sharded weights; its link opens the complete model repository.
70
  - **DINOv3** requires acceptance of the upstream model's terms and authentication. The **RAEv2 DINOv3-L (k=7)** entry uses the same DINOv3-L/16 backbone with the project's k=7 intermediate-layer readout; its download is the backbone used to construct that representation.
71
- - **TokLIP-S/L** require both the encoder checkpoint and the linked **VQ** checkpoint. For **UniAR** and **VILA-U**, retain the config files alongside the weights in the respective `bsq_encoder/` and `vision_tower/` directories.
72
  - `projector` downloads the pretraining `mm_projector.bin`;
73
  - `finetuned` opens the finetuned checkpoint directory, including model weights, configuration, and language-tokenizer files. The frozen vision encoder must be supplied separately using the matching architecture and weights recorded in `config.json`. Pretraining projector weights alone are not instruction-tuned MLLMs.
74
 
75
- ### Qwen2.5-1.5B-Instruct
76
- Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
77
 
78
  | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
79
  | --- | --- | --- | --- | --- |
@@ -144,9 +127,7 @@ Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-
144
  | Qwen2.5-1.5B-Instruct | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen25/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen25/eupe_vit_t/finetune) |
145
 
146
 
147
- ### Qwen3-1.7B
148
-
149
- Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).
150
 
151
  | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
152
  | --- | --- | --- | --- | --- |
@@ -217,9 +198,7 @@ Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).
217
  | Qwen3-1.7B | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen3/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen3/eupe_vit_t/finetune) |
218
 
219
 
220
- ### SmolLM2-1.7B-Instruct
221
-
222
- Base LLM: [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct).
223
 
224
  | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
225
  | --- | --- | --- | --- | --- |
 
38
 
39
  This repository hosts the downstream MLLM checkpoints accompanying **[A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models](https://arxiv.org/abs/2610.05413)**.
40
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
41
  # Model Zoo
42
 
43
+ This Model Zoo covers the **70 visual encoders used in our paper**: **43 language-supervised**, **22 self-supervised**, and **5 discrete** tokenizers.
44
+ The current release provides both pretraining and finetuning checkpoints for all **65 continuous vision encoders** with each of the three main language backbones.
45
 
46
  ## Training data
47
 
 
50
 
51
 
52
  ## Downloads
53
+ - `Encoder weights` links to the original upstream vision encoder or visual tokenizer. Each encoder name links to its model or project page. Use these frozen encoder weights together with the corresponding Stage 1 projector or Stage 2 MLLM checkpoint, preserving the architecture, feature layer and preprocessing specified by that run's `config.json`.
54
  - For Hugging Face models, retain the associated configuration and processor files. **Web-SSL MAE 3B** has sharded weights; its link opens the complete model repository.
55
  - **DINOv3** requires acceptance of the upstream model's terms and authentication. The **RAEv2 DINOv3-L (k=7)** entry uses the same DINOv3-L/16 backbone with the project's k=7 intermediate-layer readout; its download is the backbone used to construct that representation.
 
56
  - `projector` downloads the pretraining `mm_projector.bin`;
57
  - `finetuned` opens the finetuned checkpoint directory, including model weights, configuration, and language-tokenizer files. The frozen vision encoder must be supplied separately using the matching architecture and weights recorded in `config.json`. Pretraining projector weights alone are not instruction-tuned MLLMs.
58
 
59
+ ### [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)
 
60
 
61
  | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
62
  | --- | --- | --- | --- | --- |
 
127
  | Qwen2.5-1.5B-Instruct | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen25/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen25/eupe_vit_t/finetune) |
128
 
129
 
130
+ ### [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
 
 
131
 
132
  | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
133
  | --- | --- | --- | --- | --- |
 
198
  | Qwen3-1.7B | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen3/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen3/eupe_vit_t/finetune) |
199
 
200
 
201
+ ### [SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct)
 
 
202
 
203
  | Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
204
  | --- | --- | --- | --- | --- |