tomaarsen HF Staff commited on
Commit
3be3e40
·
verified ·
1 Parent(s): 5545ad0

Add Sentence Transformers usage section to the model card

Browse files
Files changed (1) hide show
  1. README.md +43 -0
README.md CHANGED
@@ -6,6 +6,8 @@ pipeline_tag: visual-document-retrieval
6
  library_name: transformers
7
  tags:
8
  - transformers
 
 
9
  - multimodal_embedding
10
  - embedding
11
  - colpali
@@ -30,6 +32,47 @@ The model is trained using a multi-stage strategy that combines large-scale text
30
 
31
  ## Usage
32
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
33
  **Requirements**
34
  ```
35
  pillow
 
6
  library_name: transformers
7
  tags:
8
  - transformers
9
+ - sentence-transformers
10
+ - multi-vector
11
  - multimodal_embedding
12
  - embedding
13
  - colpali
 
32
 
33
  ## Usage
34
 
35
+ ### Sentence Transformers
36
+
37
+ This model can be used with [Sentence Transformers](https://www.sbert.net/) as a multi-vector (ColBERT-style late interaction) retriever via the `MultiVectorEncoder`:
38
+
39
+ ```bash
40
+ pip install "sentence-transformers[image]>=6.0.0"
41
+ ```
42
+
43
+ ```python
44
+ from sentence_transformers import MultiVectorEncoder
45
+
46
+ model = MultiVectorEncoder(
47
+ "OpenSearch-AI/Ops-Colqwen3-4B",
48
+ trust_remote_code=True,
49
+ model_kwargs={"dtype": "bfloat16"},
50
+ )
51
+
52
+ queries = [
53
+ "What is the variable represented on the y-axis of the graph?",
54
+ "Total outlay is maximum in which year?",
55
+ ]
56
+ images = [
57
+ "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc1.jpg",
58
+ "https://huggingface.co/datasets/sentence-transformers/example-documents/resolve/main/doc2.jpg",
59
+ ]
60
+
61
+ query_embeddings = model.encode_query(queries)
62
+ image_embeddings = model.encode_document(images)
63
+ print(query_embeddings[0].shape, image_embeddings[0].shape)
64
+ # torch.Size([25, 2560]) torch.Size([1254, 2560])
65
+
66
+ # Diagonal should have higher scores
67
+ scores = model.similarity(query_embeddings, image_embeddings)
68
+ print(scores)
69
+ # tensor([[17.4668, 12.9785],
70
+ # [ 7.0088, 15.9492]], device='cuda:0')
71
+ ```
72
+
73
+ ### Transformers
74
+
75
+
76
  **Requirements**
77
  ```
78
  pillow