Visual Document Retrieval
Transformers
Safetensors
sentence-transformers
multilingual
qwen3_5
feature-extraction
text
image
multimodal-embedding
vidore
colbert
colqwen3_5
multilingual-embedding
multi-vector
custom_code
Instructions to use webAI-Official/webAI-ColVec1.1-4b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use webAI-Official/webAI-ColVec1.1-4b with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModel processor = AutoProcessor.from_pretrained("webAI-Official/webAI-ColVec1.1-4b", trust_remote_code=True) model = AutoModel.from_pretrained("webAI-Official/webAI-ColVec1.1-4b", trust_remote_code=True, device_map="auto") - sentence-transformers
How to use webAI-Official/webAI-ColVec1.1-4b with sentence-transformers:
from sentence_transformers import MultiVectorEncoder model = MultiVectorEncoder("webAI-Official/webAI-ColVec1.1-4b", trust_remote_code=True) queries = ["Which planet is known as the Red Planet?"] documents = [ "Venus is often called Earth's twin because of its similar size and proximity.", "Mars, known for its reddish appearance, is often referred to as the Red Planet.", "Jupiter, the largest planet in our solar system, has a prominent red spot.", ] query_embeddings = model.encode_query(queries) document_embeddings = model.encode_document(documents) similarities = model.similarity(query_embeddings, document_embeddings) print(similarities) - Notebooks
- Google Colab
- Kaggle
Update 4B model card
Browse files
README.md
CHANGED
|
@@ -41,19 +41,15 @@ been removed from the released checkpoint.
|
|
| 41 |
|
| 42 |
### Training data
|
| 43 |
|
| 44 |
-
We created filtered, balanced, and multilingual curated subsets from
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
| [VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data) |
|
| 54 |
-
| [VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data) |
|
| 55 |
-
| [VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train) |
|
| 56 |
-
| [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq) |
|
| 57 |
|
| 58 |
## 🛠️ Model specifications
|
| 59 |
|
|
@@ -81,10 +77,10 @@ subset.** The companion 9B-backbone model was trained on a separate
|
|
| 81 |
|
| 82 |
## 📊 Evaluation results
|
| 83 |
|
| 84 |
-
The table reports NDCG@10
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
|
| 89 |
The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
|
| 90 |
with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
|
|
@@ -97,8 +93,8 @@ Model encoding runs in `bfloat16`. Before MaxSim scoring, query and document
|
|
| 97 |
embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
|
| 98 |
results use this FP32 scoring path.
|
| 99 |
|
| 100 |
-
| Model |
|
| 101 |
-
| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 102 |
| **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
|
| 103 |
| [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
|
| 104 |
| [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
|
|
@@ -235,18 +231,19 @@ change retrieval scores.
|
|
| 235 |
|
| 236 |
### Strengths
|
| 237 |
|
| 238 |
-
-
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
-
|
| 242 |
-
-
|
| 243 |
-
|
|
|
|
|
|
|
| 244 |
|
| 245 |
### Limitations
|
| 246 |
|
| 247 |
-
-
|
| 248 |
-
|
| 249 |
-
SDPA.
|
| 250 |
|
| 251 |
## License
|
| 252 |
|
|
|
|
| 41 |
|
| 42 |
### Training data
|
| 43 |
|
| 44 |
+
We created filtered, balanced, and multilingual curated subsets from six public
|
| 45 |
+
datasets: [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set),
|
| 46 |
+
[Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir),
|
| 47 |
+
[VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data),
|
| 48 |
+
[VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data),
|
| 49 |
+
[VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train),
|
| 50 |
+
and [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq). This
|
| 51 |
+
Qwen3.5-4B-backbone model was trained on a 500,000-sample curated subset as well
|
| 52 |
+
as synthetically generated data.
|
|
|
|
|
|
|
|
|
|
|
|
|
| 53 |
|
| 54 |
## 🛠️ Model specifications
|
| 55 |
|
|
|
|
| 77 |
|
| 78 |
## 📊 Evaluation results
|
| 79 |
|
| 80 |
+
The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as
|
| 81 |
+
percentages rather than values between 0 and 1 (for example, 0.80 is shown as
|
| 82 |
+
80.00). Each task value is the mean of its six language subsets; the public
|
| 83 |
+
average is the unweighted mean of the eight public task values.
|
| 84 |
|
| 85 |
The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
|
| 86 |
with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
|
|
|
|
| 93 |
embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
|
| 94 |
results use this FP32 scoring path.
|
| 95 |
|
| 96 |
+
| Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** |
|
| 97 |
+
| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
|
| 98 |
| **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
|
| 99 |
| [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
|
| 100 |
| [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
|
|
|
|
| 231 |
|
| 232 |
### Strengths
|
| 233 |
|
| 234 |
+
- **Performance:** State-of-the-art retrieval performance among 4B models on
|
| 235 |
+
the public ViDoRe V3 tasks, with excellent multimodal document retrieval
|
| 236 |
+
results.
|
| 237 |
+
- **Complex layouts:** Excellent handling of chart-rich PDFs and
|
| 238 |
+
domain-specific documents.
|
| 239 |
+
- **End-to-end retrieval:** OCR-free retrieval on unseen multimodal documents
|
| 240 |
+
without using an intermediate vision-language model to generate summaries.
|
| 241 |
+
- **Multilingualism:** Strong performance on non-English document inputs.
|
| 242 |
|
| 243 |
### Limitations
|
| 244 |
|
| 245 |
+
- **Storage Cost:** Still larger than single-vector baselines despite the
|
| 246 |
+
smaller token dimension.
|
|
|
|
| 247 |
|
| 248 |
## License
|
| 249 |
|