Update model card: SDPA scores, quick start, eval environment

README.md

@@ -37,9 +37,9 @@ L2-normalized multi-vector embeddings for late-interaction retrieval.
 
 The model uses bidirectional attention in Qwen3.5's full-attention layers and
 a learned 640-dimensional projection head. The unused language-model head has
-been removed from the released checkpoint. Consequently, the released
-embedding model contains ~8.4B parameters, reflected by the
-`webAI-ColVec1.1-8b` repository name, while its original backbone is
+been removed from the released checkpoint. As a result, the released
+embedding model contains approximately 8.4B parameters, as reflected in the
+`webAI-ColVec1.1-8b` repository name, even though its original backbone is
 Qwen3.5-9B.
 
 ### Training data
@@ -63,7 +63,7 @@ as synthetically generated data.
 | **Method** | ColBERT-style late interaction with MaxSim scoring |
 | **Output** | L2-normalized multi-vector embeddings `(sequence_length, 640)` |
 | **Modalities** | Text queries and document images |
-| **Attention** | Bidirectional full-attention layers; selectable FlashAttention 2, FlashAttention 3, or SDPA kernel |
+| **Attention** | Bidirectional full-attention layers; selectable SDPA, FlashAttention 2, or FlashAttention 3 kernel |
 | **Visual-token budget** | 1,792 tokens per image in the released processor |
 | **Training** | LoRA adapters and a fully trained projection layer, merged for release |
 | **Weights** | `bfloat16`; language-model head removed |
@@ -75,32 +75,42 @@ as synthetically generated data.
   layout and content signals that single-vector pooling can discard.
 - **Compact projection:** Hidden states are projected to 640 dimensions
   without an activation function.
-- **Bidirectional retrieval attention:** Selecting FlashAttention 2 or 3
-  changes the execution kernel, not the model's bidirectional attention mode.
+- **Bidirectional retrieval attention:** Selecting SDPA, FlashAttention 2, or
+  FlashAttention 3 changes the execution kernel, not the model's bidirectional
+  attention mode.
 
 ## 📊 Evaluation results
 
 The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as
 percentages rather than values between 0 and 1 (for example, 0.80 is shown as
-80.00). Each task value is the mean of its six language subsets; the public average is the unweighted mean of the eight public task values.
-
-The result artifacts record MTEB 2.18.5, Transformers 5.14.1, PyTorch 2.9.0
-with CUDA 12.8, `bfloat16`, FlashAttention 2.8.3, and batch size 32. The release
-processor is configured with a 1,792 visual-token budget. Comparator values
-were read from the live
+80.00). Each task value is the mean of its six language subsets; the public
+average is the unweighted mean of the eight public task values.
+
+The evaluation software versions and setup are documented under
+[Reproducing the evaluation environment](#reproducing-the-evaluation-environment).
+For the reported evaluation, SDPA was selected as the attention
+implementation, the released processor used a 1,792 visual-token budget, and
+the batch size was 32.
+
+Model encoding used `bfloat16`. Before MaxSim scoring, query and document
+embeddings were moved to CPU and converted to `float32`; all reported ViDoRe
+results use this FP32 scoring path. Because floating-point calculations and
+kernel execution can vary across accelerator hardware, independent
+evaluations may produce slightly different results. The submitted MTEB
+artifacts are the canonical source for the reported scores.
+
+All table values are shown to two decimal places. ColVec1.1 values are rounded
+from the submitted artifacts using round-half-up. Comparator values were read
+from the live
 [ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)
 on July 22, 2026.
 
-Model encoding runs in `bfloat16`. Before MaxSim scoring, query and document
-embeddings are moved to CPU and converted to `float32`. All reported ViDoRe
-results use this FP32 scoring path.
-
 | Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** |
 | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
-| **webAI-ColVec1.1-8b (this model)** | 80.14 | 70.09 | **71.89** | **55.10** | 68.49 | **57.41** | 67.88 | 51.29 | **65.29** |
-| [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | **57.41** | **68.19** | **51.73** | 64.72 |
+| **webAI-ColVec1.1-8b (this model)** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 67.88 | 51.50 | **65.32** |
+| [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | **68.19** | **51.73** | 64.72 |
 | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
-| **[webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b)** | 80.35 | 69.26 | 69.12 | 53.21 | 67.02 | 56.30 | 67.07 | 51.36 | 64.21 |
+| **[webAI-ColVec1.1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-4b)** | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 67.25 | 51.24 | 64.24 |
 | [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | 67.45 | 50.18 | 63.72 |
 | [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 67.19 | 50.84 | 63.54 |
 | [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 65.68 | 50.15 | 63.39 |
@@ -119,31 +129,29 @@ The processor provides the current retrieval API:
 - `score_retrieval(query_embeddings, document_embeddings)` computes a MaxSim
   score matrix with shape `(number_of_queries, number_of_documents)`.
 
-### Prerequisites and attention backends
+### Quick start
 
-The public ViDoRe V3 numbers are reproducible with this pinned FlashAttention 2
-environment:
+Create and activate a Python 3.12 virtual environment, then install the
+validated PyTorch CUDA 12.8 build and the minimal packages required for SDPA
+inference. If you are already using an isolated Python environment, skip the
+first two commands.
 
-```text
-Python 3.12
-PyTorch 2.9.0 + CUDA 12.8
-Transformers 5.14.1
-MTEB 2.18.5
-Sentence Transformers 5.6.0
-FlashAttention 2.8.3
-```
+```bash
+python3.12 -m venv .venv
+source .venv/bin/activate
 
-The model also supports FlashAttention 3 on Hopper GPUs (H100/H200). See the
-[Dao-AILab FlashAttention repository](https://github.com/dao-ailab/flash-attention)
-for FlashAttention 3 installation instructions, then select
-`flash_attention_3` when loading in a compatible Hopper environment.
+python -m pip install \
+  torch==2.9.0 torchvision==0.24.0 \
+  --index-url https://download.pytorch.org/whl/cu128
 
-FlashAttention 2 and FlashAttention 3 both preserve bidirectional attention.
-FlashAttention 3 can improve throughput on Hopper GPUs, but the published
-scores use FlashAttention 2; changing kernels can produce small floating-point
-differences. Use FlashAttention 2 when reproducing the table.
+python -m pip install \
+  "transformers>=5.14.1,<6.0.0" \
+  accelerate pillow requests safetensors
+```
 
-### Inference code
+The following example uses SDPA, the portable default and the attention
+implementation used for the reported evaluation. It does not require
+FlashAttention.
 
 ```python
 from io import BytesIO
@@ -155,11 +163,9 @@ from transformers import AutoModel, AutoProcessor
 
 MODEL_ID = "webAI-Official/webAI-ColVec1.1-8b"
 DEVICE = "cuda:0" if torch.cuda.is_available() else "cpu"
-ATTN_IMPLEMENTATION = (
-    "flash_attention_2" if torch.cuda.is_available() else "sdpa"
-)
-# On an H100/H200 with FlashAttention 3 installed, use:
-# ATTN_IMPLEMENTATION = "flash_attention_3"
+
+# Portable default and the backend used by the published evaluation:
+ATTN_IMPLEMENTATION = "sdpa"
 
 processor = AutoProcessor.from_pretrained(
     MODEL_ID,
@@ -224,11 +230,97 @@ print(scores)
 print("Best document per query:", scores.argmax(dim=1))
 ```
 
-The processor loads the release's 1,792 visual-token budget by default. To
-reduce memory use, pass a lower `max_num_visual_tokens` value to
+The released processor uses a 1,792 visual-token budget by default. To reduce
+memory use, pass a lower `max_num_visual_tokens` value to
 `AutoProcessor.from_pretrained`; this changes document granularity and may
 change retrieval scores.
 
+### Optional acceleration
+
+The model can run with compatible PyTorch and CUDA builds using SDPA,
+FlashAttention 2, or FlashAttention 3. CPU execution is supported through
+PyTorch's SDPA math fallback, but is generally impractical for a model of this
+size.
+
+Qwen3.5 uses a hybrid stack of full-attention and GatedDeltaNet
+linear-attention layers. These kernels serve different parts of the model:
+
+- `flash-attn` can accelerate the full-attention layers when selected.
+- `causal-conv1d` and `flash-linear-attention` (`fla`) accelerate the
+  GatedDeltaNet layers. Transformers can fall back to PyTorch implementations
+  without them.
+- `tilelang` provides optimized GPU kernels for some FLA operations. FLA uses
+  these kernels when the operation and hardware are supported and uses another
+  implementation otherwise. Pin `apache-tvm-ffi<0.1.10` alongside it to keep
+  TileLang's TVM dependency compatible.
+
+[flash-attn](https://github.com/Dao-AILab/flash-attention) and
+[causal-conv1d](https://github.com/Dao-AILab/causal-conv1d) ship as prebuilt
+wheels tied to a specific Python, PyTorch, CUDA, and C++ ABI combination, so
+install the build that matches your environment;
+[flash-linear-attention](https://github.com/fla-org/flash-linear-attention) and
+[TileLang](https://github.com/tile-ai/tilelang) install from PyPI. The exact
+versions used for the reported scores are pinned in
+[Reproducing the evaluation environment](#reproducing-the-evaluation-environment).
+
+`causal-conv1d`, `flash-linear-attention`, and `tilelang` are detected
+automatically once installed, so the GatedDeltaNet layers need no configuration
+change. Only the full-attention backend is selected explicitly, as a one-line
+change to the model loading code in [Quick start](#quick-start):
+
+```python
+ATTN_IMPLEMENTATION = "flash_attention_2"  # or "flash_attention_3"
+```
+
+FlashAttention 2 and FlashAttention 3 both preserve the model's bidirectional
+attention, and FlashAttention 3 can improve throughput on Hopper GPUs
+(H100/H200). PyTorch's built-in SDPA remains the more portable choice because
+it does not require a separate FlashAttention package or ABI-compatible wheel.
+Changing the full-attention implementation may introduce small floating-point
+differences and affect only those layers, not the GatedDeltaNet layers.
+
+### Reproducing the evaluation environment
+
+The complete pinned Python environment is provided in
+[`evaluation-requirements-cu128.txt`](./evaluation-requirements-cu128.txt),
+which reproduces the recorded Linux x86-64, CPython 3.12, CUDA 12.8, and
+PyTorch 2.9 environment used for evaluation. Its pinned wheel URLs are specific
+to that platform, so a different environment needs matching wheels or a source
+build.
+
+Install [`uv`](https://docs.astral.sh/uv/getting-started/installation/), ensure
+Git is available, and then run:
+
+```bash
+uv venv --python 3.12 .venv
+source .venv/bin/activate
+
+uv pip install \
+  torch==2.9.0 torchvision==0.24.0 \
+  --index-url https://download.pytorch.org/whl/cu128
+
+uv pip install -r evaluation-requirements-cu128.txt
+uv pip check
+```
+
+The requirements include FlashAttention 2 as an optional supported backend;
+its presence does not change the SDPA configuration used for the reported
+scores.
+
+The public ViDoRe V3 scores used the following core software versions:
+
+```text
+Python 3.12
+PyTorch 2.9.0 + CUDA 12.8
+Transformers 5.14.1
+MTEB 2.18.6 (commit d56a414b45ebad0d03495de000b4880d8b028d4a)
+Sentence Transformers 5.6.0
+causal-conv1d 1.6.2.post1
+flash-linear-attention 0.5.1
+TileLang 0.1.9
+Attention implementation: SDPA
+```
+
 ## ⚖️ Strengths and limitations
 
 ### Strengths
@@ -243,7 +335,7 @@ change retrieval scores.
 
 ### Limitations
 
-- **Storage Cost:** Still larger than single-vector baselines despite the
+- **Storage cost:** Still larger than single-vector baselines despite the
   smaller token dimension.
 
 ## License