File size: 6,821 Bytes
e730d84 7238977 e730d84 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 | ---
license: mit
language:
- en
- es
- de
- fr
- it
- pt
base_model:
- athrael-soju/UltraVBERT
base_model_relation: finetune
datasets:
- athrael-soju/vdr-train-500k
tags:
- ultravbert
- sentence-transformers
- vision-language
- multimodal
- document-retrieval
- late-interaction
- multi-vector
- colbert
- colpali
- vidore
pipeline_tag: visual-document-retrieval
library_name: sentence-transformers
---
<!-- vllm-ci-copy -->
> [!NOTE]
> This is a copy of [athrael-soju/ColUltraVBERT](https://huggingface.co/athrael-soju/ColUltraVBERT) at commit `486d2f2`, made on 2026-10-30. vLLM CI uses this checkpoint and the original is on a personal account, so we keep a copy here to reduce supply-chain risk.
<img src="colultravbert-hero.png" width="100%" alt="ColUltraVBERT">
# ColUltraVBERT
ColUltraVBERT is a 255M-parameter late-interaction retriever for visual documents, fine-tuned from
[UltraVBERT](https://huggingface.co/athrael-soju/UltraVBERT). It encodes each query token and each page token as a
768-d vector and ranks pages by MaxSim. Truncating the vectors to 128, 256, 320 or 512 dimensions gives a smaller
index.
## Models
<table style="display: table; width: 100%;">
<thead>
<tr><th align="left">Model</th><th align="left">Role</th><th align="left">Status</th></tr>
</thead>
<tbody>
<tr><td><a href="https://huggingface.co/athrael-soju/UltraVBERT">UltraVBERT</a></td><td>Base encoder aligned to page images</td><td>Available</td></tr>
<tr><td><a href="https://huggingface.co/athrael-soju/BiUltraVBERT">BiUltraVBERT</a></td><td>Single-vector retriever for visual documents</td><td>Available</td></tr>
<tr><td><a href="https://huggingface.co/athrael-soju/ColUltraVBERT">ColUltraVBERT</a></td><td>Late-interaction retriever for visual documents</td><td>Available · <a href="https://huggingface.co/spaces/athrael-soju/ColUltraVBERT">Demo</a></td></tr>
</tbody>
</table>
## Data
<table style="display: table; width: 100%;">
<thead>
<tr><th align="left">Dataset</th><th align="left">Content</th></tr>
</thead>
<tbody>
<tr><td><a href="https://huggingface.co/datasets/athrael-soju/vdr-train-500k">VDR Train 500k</a></td><td>499,704 query-page pairs; queries in English (73%), Spanish, German, French, Italian and Portuguese</td></tr>
</tbody>
</table>
## Usage
Requires `sentence-transformers>=6.1` and `transformers>=5.21`, which supports UltraVBERT natively
([docs](https://huggingface.co/docs/transformers/model_doc/ultravbert)). On 5.20.x, pass `trust_remote_code=True` to load the code in this repository.
```python
from PIL import Image
from sentence_transformers import MultiVectorEncoder
model = MultiVectorEncoder("athrael-soju/ColUltraVBERT")
pages = [Image.open("page1.png"), Image.open("page2.png")]
queries = ["What was the 2023 revenue?", "Which table lists regional margins?"]
page_embeddings = model.encode_document(pages)
query_embeddings = model.encode_query(queries)
# One row per query, one column per page.
scores = model.similarity(query_embeddings, page_embeddings)
```
For a smaller index, keep the first 128, 256, 320 or 512 dimensions of each vector and L2-normalize it again.
## Model
A linear layer projects each of UltraVBERT's 768-d token states to a 768-d retrieval vector. Vectors are L2-normalized
and padding tokens are dropped. Pages are resized to at most 3,072 px on the longest edge and split into up to 25 tiles
of 512 px, plus a downscaled view of the whole page: at most 1,664 image vectors per page, plus the prompt and
tile-tag vectors.
## Training
The released weights are a per-block merge of four fine-tunes of UltraVBERT on VDR Train 500k that differ in
hard negatives, distillation and seed. Each fine-tune trains rank-32 LoRA adapters on the text encoder and the output
projection for one epoch at batch size 256, with Matryoshka losses at 128, 256, 320, 512 and 768 dimensions; the vision
encoder and connector stay frozen.
## ViDoRe
ColUltraVBERT's scores are its official MTEB results
([results#783](https://github.com/embeddings-benchmark/results/pull/783)): MTEB 2.26.3, 768-d vectors, bf16 and SDPA
attention. The MTEB maintainers scored the private ViDoRe V3 tasks, Nuclear and Telecom
([results#792](https://github.com/embeddings-benchmark/results/pull/792)). ColModernVBERT's V1 and V2 scores are our
MTEB 2.26.3 runs of [`ModernVBERT/colmodernvbert`](https://huggingface.co/ModernVBERT/colmodernvbert) at revision
`9052fc0e`; its V3 scores are its MTEB results.
| ViDoRe V1 (nDCG@5) | ColModernVBERT | ColUltraVBERT |
| --- | ---: | ---: |
| ArxivQA | 75.0 | 80.1 |
| DocVQA | 56.2 | 57.7 |
| InfoVQA | 86.5 | 89.5 |
| TabFQuAD | 76.1 | 92.5 |
| TAT-DQA | 73.9 | 75.1 |
| ShiftProject | 64.8 | 81.2 |
| SyntheticDocQA AI | 97.5 | 99.6 |
| SyntheticDocQA Energy | 94.2 | 95.7 |
| SyntheticDocQA Government Reports | 93.3 | 95.1 |
| SyntheticDocQA Healthcare Industry | 96.9 | 97.9 |
| **Mean** | **81.4** | **86.4** |
| ViDoRe V2 (nDCG@5) | ColModernVBERT | ColUltraVBERT |
| --- | ---: | ---: |
| ESG Reports | 40.5 | 49.8 |
| Economics Reports | 32.6 | 51.2 |
| BioMedical Lectures | 34.8 | 54.3 |
| ESG Reports HL | 54.8 | 64.0 |
| **Mean** | **40.7** | **54.8** |
| ViDoRe V3 (nDCG@10) | ColModernVBERT | ColUltraVBERT |
| --- | ---: | ---: |
| Computer Science | 37.6 | 66.5 |
| Energy | 32.5 | 51.9 |
| Finance EN | 28.8 | 53.3 |
| Finance FR | 19.8 | 33.3 |
| HR | 19.8 | 51.1 |
| Industrial | 15.7 | 37.6 |
| Pharmaceuticals | 32.2 | 57.6 |
| Physics | 22.6 | 37.7 |
| Nuclear (private) | 21.5 | 42.0 |
| Telecom (private) | 30.3 | 62.0 |
| **Mean, public** | **26.1** | **48.6** |
| **Mean, private** | **25.9** | **52.0** |
| **Mean, all** | **26.1** | **49.3** |
### Smaller indexes
Scores come from cached page features; at 768 dimensions and 3,072 px they are within 0.1 of the official means. For
2,048 px pages with 16 tiles, set `size.longest_edge` to 2048 and `max_image_tiles` to 16 in `processor_config.json`; a
page then produces at most 1,088 image vectors.
| Vector size | Page resolution | V1 | V2 | V3 public |
| ---: | --- | ---: | ---: | ---: |
| 768 | 3,072 px, 25 tiles | 86.4 | 54.8 | 48.7 |
| 512 | 3,072 px, 25 tiles | 86.3 | 54.3 | 48.4 |
| 320 | 3,072 px, 25 tiles | 86.0 | 54.2 | 48.2 |
| 256 | 3,072 px, 25 tiles | 86.0 | 54.3 | 48.0 |
| 128 | 3,072 px, 25 tiles | 85.7 | 54.3 | 46.8 |
| 768 | 2,048 px, 16 tiles | 86.3 | 53.3 | 47.6 |
| 512 | 2,048 px, 16 tiles | 86.3 | 52.9 | 47.5 |
| 320 | 2,048 px, 16 tiles | 86.4 | 52.4 | 47.2 |
| 256 | 2,048 px, 16 tiles | 86.4 | 52.1 | 46.9 |
| 128 | 2,048 px, 16 tiles | 85.6 | 52.6 | 45.9 |
## Limits
- A page produces up to 1,664 image vectors, 2.4 MiB at 768 dimensions in bf16, plus its prompt and tile-tag vectors.
## License
`MIT`. The UltraBERT-150M and SigLIP2-B/16-512 components are Apache-2.0, with their license text in
`LICENSE-APACHE`.
|