--- pipeline_tag: visual-document-retrieval library_name: transformers language: - multilingual license: other license_name: webai-non-commercial-license-v1.0 license_link: https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b/blob/main/LICENSE.md base_model: Qwen/Qwen3.5-4B datasets: - vidore/colpali_train_set - Tevatron/docmatix-ir - openbmb/VisRAG-Ret-Train-In-domain-data - openbmb/VisRAG-Ret-Train-Synthetic-data - llamaindex/vdr-multilingual-train - Tevatron/wiki-ss-nq tags: - text - image - multimodal-embedding - visual-document-retrieval - vidore - colbert - colqwen3_5 - multilingual-embedding --- # webAI-Official/webAI-ColVec1.1-4b ## ⚡ Summary **webAI-Official/webAI-ColVec1.1-4b** is a ColBERT-style multimodal embedding model based on [Qwen/Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B). It maps text queries and visual documents (images or rendered PDF pages) into aligned, L2-normalized multi-vector embeddings for late-interaction retrieval. The model uses bidirectional attention in Qwen3.5's full-attention layers and a learned 640-dimensional projection head. The unused language-model head has been removed from the released checkpoint. ### Training data We created filtered, balanced, and multilingual curated subsets from six public datasets: [ColPali Train Set](https://huggingface.co/datasets/vidore/colpali_train_set), [Docmatix-IR](https://huggingface.co/datasets/Tevatron/docmatix-ir), [VisRAG In-Domain](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-In-domain-data), [VisRAG Synthetic](https://huggingface.co/datasets/openbmb/VisRAG-Ret-Train-Synthetic-data), [VDR Multilingual Train](https://huggingface.co/datasets/llamaindex/vdr-multilingual-train), and [Wiki-SS-NQ](https://huggingface.co/datasets/Tevatron/wiki-ss-nq). This Qwen3.5-4B-backbone model was trained on a 500,000-sample curated subset as well as synthetically generated data. ## 🛠️ Model specifications | Feature | Detail | | :--- | :--- | | **Architecture** | Qwen3.5-4B vision-language model + 640-dimensional linear projection | | **Released parameters** | 4,540,904,576 | | **Method** | ColBERT-style late interaction with MaxSim scoring | | **Output** | L2-normalized multi-vector embeddings `(sequence_length, 640)` | | **Modalities** | Text queries and document images | | **Attention** | Bidirectional full-attention layers; selectable SDPA, FlashAttention 2, or FlashAttention 3 kernel | | **Visual-token budget** | 1,792 tokens per image in the released processor | | **Training** | LoRA adapters and a fully trained projection layer, merged for release | | **Weights** | `bfloat16`; language-model head removed | ### Key properties - **Unified encoder:** The same model encodes text and document images. - **Token-level retrieval:** Multi-vector embeddings preserve fine-grained layout and content signals that single-vector pooling can discard. - **Compact projection:** Hidden states are projected to 640 dimensions without an activation function. - **Bidirectional retrieval attention:** Selecting SDPA, FlashAttention 2, or FlashAttention 3 changes the execution kernel, not the model's bidirectional attention mode. ## 📊 Evaluation results The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as percentages rather than values between 0 and 1 (for example, 0.80 is shown as 80.00). Each task value is the mean of its six language subsets; the public average is the unweighted mean of the eight public task values. The evaluation software versions and setup are documented under [Reproducing the evaluation environment](#reproducing-the-evaluation-environment). For the reported evaluation, SDPA was selected as the attention implementation, the released processor used a 1,792 visual-token budget, and the batch size was 32. Model encoding used `bfloat16`. Before MaxSim scoring, query and document embeddings were moved to CPU and converted to `float32`; all reported ViDoRe results use this FP32 scoring path. Because floating-point calculations and kernel execution can vary across accelerator hardware, independent evaluations may produce slightly different results. The submitted MTEB artifacts are the canonical source for the reported scores. All table values are shown to two decimal places. ColVec1.1 values are rounded from the submitted artifacts using round-half-up. Comparator values were read from the live [ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29) on July 22, 2026. | Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** | | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | | **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 67.88 | 51.50 | **65.32** | | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | **68.19** | **51.73** | 64.72 | | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 | | **webAI-ColVec1.1-4b (this model)** | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 67.25 | 51.24 | 64.24 | | [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | 67.45 | 50.18 | 63.72 | | [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 67.19 | 50.84 | 63.54 | | [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 65.68 | 50.15 | 63.39 | | [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 66.36 | 50.13 | 61.60 | The current MTEB leaderboard entries named `webAI-ColVec1-4b` and `webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1 checkpoints. ## 💻 Usage The processor provides the current retrieval API: - `process_images(images)` prepares one or more document images. - `process_queries(texts)` prepares one or more natural-language queries. - `score_retrieval(query_embeddings, document_embeddings)` computes a MaxSim score matrix with shape `(number_of_queries, number_of_documents)`. ### Quick start Create and activate a Python 3.12 virtual environment, then install the validated PyTorch CUDA 12.8 build and the minimal packages required for SDPA inference. If you are already using an isolated Python environment, skip the first two commands. ```bash python3.12 -m venv .venv source .venv/bin/activate python -m pip install \ torch==2.9.0 torchvision==0.24.0 \ --index-url https://download.pytorch.org/whl/cu128 python -m pip install \ "transformers>=5.14.1,<6.0.0" \ accelerate pillow requests safetensors ``` The following example uses SDPA, the portable default and the attention implementation used for the reported evaluation. It does not require FlashAttention. ```python from io import BytesIO import requests import torch from PIL import Image from transformers import AutoModel, AutoProcessor MODEL_ID = "webAI-Official/webAI-ColVec1.1-4b" DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu" # Portable default and the backend used by the published evaluation: ATTN_IMPLEMENTATION = "sdpa" processor = AutoProcessor.from_pretrained( MODEL_ID, trust_remote_code=True, max_num_visual_tokens=1792, ) model = AutoModel.from_pretrained( MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation=ATTN_IMPLEMENTATION, device_map=DEVICE, ).eval() queries = [ "When was the United States Declaration of Independence proclaimed?", "Who printed the edition of Romeo and Juliet?", ] document_urls = [ "https://upload.wikimedia.org/wikipedia/commons/8/89/US-original-Declaration-1776.jpg", "https://upload.wikimedia.org/wikipedia/commons/thumb/4/4c/Romeoandjuliet1597.jpg/500px-Romeoandjuliet1597.jpg", ] def load_image(url: str) -> Image.Image: response = requests.get( url, headers={"User-Agent": "Mozilla/5.0"}, timeout=30, ) response.raise_for_status() return Image.open(BytesIO(response.content)).convert("RGB") device = next(model.parameters()).device query_inputs = processor.process_queries(queries) document_inputs = processor.process_images( [load_image(url) for url in document_urls] ) query_inputs = { key: value.to(device) if isinstance(value, torch.Tensor) else value for key, value in query_inputs.items() } document_inputs = { key: value.to(device) if isinstance(value, torch.Tensor) else value for key, value in document_inputs.items() } with torch.inference_mode(): query_batch = model(**query_inputs) document_batch = model(**document_inputs) query_embeddings = [embedding.cpu() for embedding in query_batch] document_embeddings = [embedding.cpu() for embedding in document_batch] scores = processor.score_retrieval( query_embeddings, document_embeddings, output_dtype=torch.float32, ) print(scores) print("Best document per query:", scores.argmax(dim=1)) ``` The released processor uses a 1,792 visual-token budget by default. To reduce memory use, pass a lower `max_num_visual_tokens` value to `AutoProcessor.from_pretrained`; this changes document granularity and may change retrieval scores. ### Optional acceleration The model can run with compatible PyTorch and CUDA builds using SDPA, FlashAttention 2, or FlashAttention 3. CPU execution is supported through PyTorch's SDPA math fallback, but is generally impractical for a model of this size. Qwen3.5 uses a hybrid stack of full-attention and GatedDeltaNet linear-attention layers. These kernels serve different parts of the model: - `flash-attn` can accelerate the full-attention layers when selected. - `causal-conv1d` and `flash-linear-attention` (`fla`) accelerate the GatedDeltaNet layers. Transformers can fall back to PyTorch implementations without them. - `tilelang` provides optimized GPU kernels for some FLA operations. FLA uses these kernels when the operation and hardware are supported and uses another implementation otherwise. Pin `apache-tvm-ffi<0.1.10` alongside it to keep TileLang's TVM dependency compatible. [flash-attn](https://github.com/Dao-AILab/flash-attention) and [causal-conv1d](https://github.com/Dao-AILab/causal-conv1d) ship as prebuilt wheels tied to a specific Python, PyTorch, CUDA, and C++ ABI combination, so install the build that matches your environment; [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) and [TileLang](https://github.com/tile-ai/tilelang) install from PyPI. The exact versions used for the reported scores are pinned in [Reproducing the evaluation environment](#reproducing-the-evaluation-environment). `causal-conv1d`, `flash-linear-attention`, and `tilelang` are detected automatically once installed, so the GatedDeltaNet layers need no configuration change. Only the full-attention backend is selected explicitly, as a one-line change to the model loading code in [Quick start](#quick-start): ```python ATTN_IMPLEMENTATION = "flash_attention_2" # or "flash_attention_3" ``` FlashAttention 2 and FlashAttention 3 both preserve the model's bidirectional attention, and FlashAttention 3 can improve throughput on Hopper GPUs (H100/H200). PyTorch's built-in SDPA remains the more portable choice because it does not require a separate FlashAttention package or ABI-compatible wheel. Changing the full-attention implementation may introduce small floating-point differences and affect only those layers, not the GatedDeltaNet layers. ### Reproducing the evaluation environment The complete pinned Python environment is provided in [`evaluation-requirements-cu128.txt`](./evaluation-requirements-cu128.txt), which reproduces the recorded Linux x86-64, CPython 3.12, CUDA 12.8, and PyTorch 2.9 environment used for evaluation. Its pinned wheel URLs are specific to that platform, so a different environment needs matching wheels or a source build. Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), ensure Git is available, and then run: ```bash uv venv --python 3.12 .venv source .venv/bin/activate uv pip install \ torch==2.9.0 torchvision==0.24.0 \ --index-url https://download.pytorch.org/whl/cu128 uv pip install -r evaluation-requirements-cu128.txt uv pip check ``` The requirements include FlashAttention 2 as an optional supported backend; its presence does not change the SDPA configuration used for the reported scores. The public ViDoRe V3 scores used the following core software versions: ```text Python 3.12 PyTorch 2.9.0 + CUDA 12.8 Transformers 5.14.1 MTEB 2.18.6 (commit d56a414b45ebad0d03495de000b4880d8b028d4a) Sentence Transformers 5.6.0 causal-conv1d 1.6.2.post1 flash-linear-attention 0.5.1 TileLang 0.1.9 Attention implementation: SDPA ``` ## ⚖️ Strengths and limitations ### Strengths - **Performance:** State-of-the-art retrieval performance among 4B models on the public ViDoRe V3 tasks, with excellent multimodal document retrieval results. - **Complex layouts:** Excellent handling of chart-rich PDFs and domain-specific documents. - **End-to-end retrieval:** OCR-free retrieval on unseen multimodal documents without using an intermediate vision-language model to generate summaries. - **Multilingualism:** Strong performance on non-English document inputs. ### Limitations - **Storage cost:** Still larger than single-vector baselines despite the smaller token dimension. ## License Model weights are distributed under the [webAI Non-Commercial License v1.0](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b/blob/main/LICENSE.md). See the repository's `NOTICES.md` for upstream attribution. ## 📚 Citation ```bibtex @misc{webai_colvec1_1_4b, title = {webAI-ColVec1.1-4b: A Bidirectional Multi-Vector Model for Visual Document Retrieval}, author = {webAI}, year = {2026}, url = {https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b} } ```