Update README.md
Browse files
README.md
CHANGED
|
@@ -38,25 +38,10 @@ tags:
|
|
| 38 |
|
| 39 |
This repository hosts the downstream MLLM checkpoints accompanying **[A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models](https://arxiv.org/abs/2610.05413)**.
|
| 40 |
|
| 41 |
-
## Files
|
| 42 |
-
|
| 43 |
-
Checkpoints are grouped by language model backbone, vision encoder, and training stage:
|
| 44 |
-
|
| 45 |
-
```text
|
| 46 |
-
models/
|
| 47 |
-
continuous/<llm>/<encoder>/
|
| 48 |
-
pretrain/mm_projector.bin
|
| 49 |
-
pretrain/config.json
|
| 50 |
-
finetune/ # Model weights, config, and tokenizer files
|
| 51 |
-
```
|
| 52 |
-
|
| 53 |
-
Use `qwen25`, `qwen3`, or `smollm2` for the three main language-model groups.
|
| 54 |
-
For example, the Qwen3 + CLIP-L/14 finetuned model is at `models/continuous/qwen3/clip_openai__l14/finetune/`.
|
| 55 |
-
|
| 56 |
# Model Zoo
|
| 57 |
|
| 58 |
-
This Model Zoo covers the **70 visual
|
| 59 |
-
The current release provides both pretraining and finetuning checkpoints for all **65 continuous
|
| 60 |
|
| 61 |
## Training data
|
| 62 |
|
|
@@ -65,15 +50,13 @@ The current release provides both pretraining and finetuning checkpoints for all
|
|
| 65 |
|
| 66 |
|
| 67 |
## Downloads
|
| 68 |
-
-
|
| 69 |
- For Hugging Face models, retain the associated configuration and processor files. **Web-SSL MAE 3B** has sharded weights; its link opens the complete model repository.
|
| 70 |
- **DINOv3** requires acceptance of the upstream model's terms and authentication. The **RAEv2 DINOv3-L (k=7)** entry uses the same DINOv3-L/16 backbone with the project's k=7 intermediate-layer readout; its download is the backbone used to construct that representation.
|
| 71 |
-
- **TokLIP-S/L** require both the encoder checkpoint and the linked **VQ** checkpoint. For **UniAR** and **VILA-U**, retain the config files alongside the weights in the respective `bsq_encoder/` and `vision_tower/` directories.
|
| 72 |
- `projector` downloads the pretraining `mm_projector.bin`;
|
| 73 |
- `finetuned` opens the finetuned checkpoint directory, including model weights, configuration, and language-tokenizer files. The frozen vision encoder must be supplied separately using the matching architecture and weights recorded in `config.json`. Pretraining projector weights alone are not instruction-tuned MLLMs.
|
| 74 |
|
| 75 |
-
### Qwen2.5-1.5B-Instruct
|
| 76 |
-
Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct).
|
| 77 |
|
| 78 |
| Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
|
| 79 |
| --- | --- | --- | --- | --- |
|
|
@@ -144,9 +127,7 @@ Base LLM: [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-
|
|
| 144 |
| Qwen2.5-1.5B-Instruct | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen25/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen25/eupe_vit_t/finetune) |
|
| 145 |
|
| 146 |
|
| 147 |
-
### Qwen3-1.7B
|
| 148 |
-
|
| 149 |
-
Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).
|
| 150 |
|
| 151 |
| Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
|
| 152 |
| --- | --- | --- | --- | --- |
|
|
@@ -217,9 +198,7 @@ Base LLM: [Qwen/Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B).
|
|
| 217 |
| Qwen3-1.7B | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen3/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen3/eupe_vit_t/finetune) |
|
| 218 |
|
| 219 |
|
| 220 |
-
### SmolLM2-1.7B-Instruct
|
| 221 |
-
|
| 222 |
-
Base LLM: [HuggingFaceTB/SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct).
|
| 223 |
|
| 224 |
| Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
|
| 225 |
| --- | --- | --- | --- | --- |
|
|
|
|
| 38 |
|
| 39 |
This repository hosts the downstream MLLM checkpoints accompanying **[A Strong Baseline for Evaluating Vision Encoders in Multimodal Large Language Models](https://arxiv.org/abs/2610.05413)**.
|
| 40 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 41 |
# Model Zoo
|
| 42 |
|
| 43 |
+
This Model Zoo covers the **70 visual encoders used in our paper**: **43 language-supervised**, **22 self-supervised**, and **5 discrete** tokenizers.
|
| 44 |
+
The current release provides both pretraining and finetuning checkpoints for all **65 continuous vision encoders** with each of the three main language backbones.
|
| 45 |
|
| 46 |
## Training data
|
| 47 |
|
|
|
|
| 50 |
|
| 51 |
|
| 52 |
## Downloads
|
| 53 |
+
- `Encoder weights` links to the original upstream vision encoder or visual tokenizer. Each encoder name links to its model or project page. Use these frozen encoder weights together with the corresponding Stage 1 projector or Stage 2 MLLM checkpoint, preserving the architecture, feature layer and preprocessing specified by that run's `config.json`.
|
| 54 |
- For Hugging Face models, retain the associated configuration and processor files. **Web-SSL MAE 3B** has sharded weights; its link opens the complete model repository.
|
| 55 |
- **DINOv3** requires acceptance of the upstream model's terms and authentication. The **RAEv2 DINOv3-L (k=7)** entry uses the same DINOv3-L/16 backbone with the project's k=7 intermediate-layer readout; its download is the backbone used to construct that representation.
|
|
|
|
| 56 |
- `projector` downloads the pretraining `mm_projector.bin`;
|
| 57 |
- `finetuned` opens the finetuned checkpoint directory, including model weights, configuration, and language-tokenizer files. The frozen vision encoder must be supplied separately using the matching architecture and weights recorded in `config.json`. Pretraining projector weights alone are not instruction-tuned MLLMs.
|
| 58 |
|
| 59 |
+
### [Qwen/Qwen2.5-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct)
|
|
|
|
| 60 |
|
| 61 |
| Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
|
| 62 |
| --- | --- | --- | --- | --- |
|
|
|
|
| 127 |
| Qwen2.5-1.5B-Instruct | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen25/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen25/eupe_vit_t/finetune) |
|
| 128 |
|
| 129 |
|
| 130 |
+
### [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)
|
|
|
|
|
|
|
| 131 |
|
| 132 |
| Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
|
| 133 |
| --- | --- | --- | --- | --- |
|
|
|
|
| 198 |
| Qwen3-1.7B | [EUPE ViT-T](https://huggingface.co/facebook/EUPE-ViT-T) | [encoder](https://huggingface.co/facebook/EUPE-ViT-T/resolve/main/EUPE-ViT-T.pt?download=true) | [projector](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/resolve/main/continuous/qwen3/eupe_vit_t/pretrain/mm_projector.bin) | [finetuned](https://huggingface.co/336labs/VisionEncoder-to-MLLM-ModelZoo/tree/main/continuous/qwen3/eupe_vit_t/finetune) |
|
| 199 |
|
| 200 |
|
| 201 |
+
### [SmolLM2-1.7B-Instruct](https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct)
|
|
|
|
|
|
|
| 202 |
|
| 203 |
| Base LLM | Vision Encoder / Tokenizer | Encoder weights | Stage 1 Pretrained weights | Stage 2 Finetuned weights |
|
| 204 |
| --- | --- | --- | --- | --- |
|