nebhailema commited on
Commit
c524857
·
verified ·
1 Parent(s): 65347df

docs: link each model in the full-corpus tables

Browse files
Files changed (1) hide show
  1. README.md +88 -75
README.md CHANGED
@@ -144,7 +144,7 @@ print(similarities)
144
  ```
145
  ## Evaluation
146
 
147
- Full-corpus passage retrieval on [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR) for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. `avg` is the query-weighted average across those languages. The best value in that column is **bold**.
148
 
149
  Reranker rows rerank the top 50 passages from a frozen dense retriever.
150
 
@@ -154,79 +154,79 @@ Reranker rows rerank the top 50 passages from a frozen dense retriever.
154
 
155
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
156
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
157
- | dense mini | 0.7805 | 0.6940 | 0.5643 | 0.6270 | 0.6637 | 0.7081 | 0.7755 | 0.6024 | 0.7233 |
158
- | e5-small | 0.8741 | 0.8014 | 0.7418 | 0.8162 | 0.8088 | 0.8451 | 0.8571 | 0.7359 | 0.8379 |
159
- | e5-base | 0.8886 | 0.8691 | 0.7810 | 0.8000 | 0.8526 | 0.8738 | 0.8571 | 0.7590 | 0.8629 |
160
- | e5-large | 0.8948 | **0.8809** | 0.7974 | 0.8324 | 0.8841 | **0.8993** | 0.7755 | 0.7550 | 0.8738 |
161
- | M3 dense | 0.7067 | 0.6264 | 0.4630 | 0.5892 | 0.5894 | 0.6488 | 0.7551 | 0.5231 | 0.6501 |
162
- | SPLADE mini | 0.7759 | 0.7446 | 0.7603 | 0.7405 | 0.7705 | 0.7546 | 0.8163 | 0.6657 | 0.7599 |
163
- | M3 sparse | 0.7826 | 0.7356 | 0.7593 | 0.7189 | 0.7627 | 0.7463 | 0.8367 | 0.6536 | 0.7598 |
164
- | multivector mini | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | 0.9388 | 0.7841 | 0.8471 |
165
- | multivector e5-small | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
166
- | multivector e5-base | 0.9034 | 0.8718 | 0.8246 | 0.8432 | 0.8695 | 0.8719 | 0.9388 | 0.7922 | 0.8789 |
167
- | M3 multi | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
168
- | reranker mini | 0.8647 | 0.8159 | 0.7462 | 0.7946 | 0.8245 | 0.8305 | 0.9184 | 0.7359 | 0.8336 |
169
- | reranker e5-small | 0.8907 | 0.8430 | 0.8279 | 0.8541 | 0.8673 | 0.8693 | **0.9592** | 0.7871 | 0.8690 |
170
- | reranker e5-base | **0.9082** | 0.8727 | **0.8497** | 0.8486 | **0.8943** | 0.8725 | 0.8980 | **0.7932** | **0.8853** |
171
 
172
  #### recall@10
173
 
174
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
175
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
176
- | dense mini | 0.8285 | 0.7644 | 0.6362 | 0.6973 | 0.7334 | 0.7827 | 0.8163 | 0.6737 | 0.7820 |
177
- | e5-small | 0.9072 | 0.8718 | 0.8007 | 0.8811 | 0.8774 | 0.9012 | 0.8980 | 0.7982 | 0.8844 |
178
- | e5-base | 0.9209 | 0.9070 | 0.8410 | 0.8649 | 0.9224 | 0.9133 | 0.8571 | 0.8233 | 0.9042 |
179
- | e5-large | 0.9273 | **0.9242** | 0.8551 | 0.8811 | **0.9359** | **0.9344** | 0.8163 | 0.8143 | 0.9131 |
180
- | M3 dense | 0.7584 | 0.6931 | 0.5381 | 0.6595 | 0.6704 | 0.7208 | 0.7755 | 0.5944 | 0.7112 |
181
- | SPLADE mini | 0.8293 | 0.7924 | 0.8115 | 0.8162 | 0.8425 | 0.8317 | 0.8776 | 0.7440 | 0.8192 |
182
- | M3 sparse | 0.8279 | 0.8087 | 0.8137 | 0.8000 | 0.8256 | 0.8107 | 0.8776 | 0.7169 | 0.8138 |
183
- | multivector mini | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
184
- | multivector e5-small | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
185
- | multivector e5-base | 0.9300 | 0.9206 | 0.8758 | 0.8649 | 0.9123 | 0.9152 | 0.9388 | 0.8394 | 0.9140 |
186
- | M3 multi | 0.8896 | 0.8484 | 0.8529 | 0.8865 | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
187
- | reranker mini | 0.8872 | 0.8547 | 0.7800 | 0.8324 | 0.8605 | 0.8662 | **0.9592** | 0.7620 | 0.8615 |
188
- | reranker e5-small | 0.9206 | 0.8917 | 0.8704 | 0.8811 | 0.9168 | 0.9191 | **0.9592** | 0.8414 | 0.9072 |
189
- | reranker e5-base | **0.9345** | 0.9224 | **0.8987** | **0.8919** | 0.9314 | **0.9344** | **0.9592** | **0.8454** | **0.9229** |
190
 
191
  #### ndcg@10
192
 
193
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
194
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
195
- | dense mini | 0.7129 | 0.5933 | 0.4881 | 0.5533 | 0.5709 | 0.6234 | 0.6532 | 0.5160 | 0.6464 |
196
- | e5-small | 0.8086 | 0.7077 | 0.6464 | 0.7233 | 0.7270 | 0.7653 | 0.7810 | 0.6609 | 0.7635 |
197
- | e5-base | 0.8264 | 0.7661 | 0.6851 | 0.7362 | 0.7691 | 0.7929 | 0.7583 | 0.6855 | 0.7897 |
198
- | e5-large | 0.8294 | **0.7962** | 0.7089 | 0.7499 | **0.8061** | **0.8147** | 0.7093 | 0.6677 | 0.7997 |
199
- | M3 dense | 0.6340 | 0.5278 | 0.4034 | 0.5053 | 0.5045 | 0.5699 | 0.6650 | 0.4406 | 0.5734 |
200
- | SPLADE mini | 0.6816 | 0.6213 | 0.6629 | 0.6763 | 0.6768 | 0.6579 | 0.7696 | 0.5729 | 0.6633 |
201
- | M3 sparse | 0.6987 | 0.6469 | 0.6776 | 0.6599 | 0.6835 | 0.6593 | 0.7754 | 0.5562 | 0.6750 |
202
- | multivector mini | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
203
- | multivector e5-small | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
204
- | multivector e5-base | 0.8428 | 0.7853 | 0.7429 | 0.7801 | 0.7867 | 0.7989 | 0.8569 | **0.7137** | 0.8098 |
205
- | M3 multi | 0.7690 | 0.6967 | 0.7113 | 0.7824 | 0.7315 | 0.7365 | 0.8879 | 0.6265 | 0.7410 |
206
- | reranker mini | 0.8082 | 0.7483 | 0.6906 | 0.7416 | 0.7613 | 0.7523 | 0.8987 | 0.6671 | 0.7722 |
207
- | reranker e5-small | 0.8284 | 0.7397 | 0.7435 | 0.7852 | 0.7814 | 0.7815 | **0.9037** | 0.6999 | 0.7946 |
208
- | reranker e5-base | **0.8449** | 0.7681 | **0.7668** | **0.7903** | 0.7968 | 0.7947 | 0.8769 | 0.7018 | **0.8107** |
209
 
210
  #### mrr@10
211
 
212
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
213
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
214
- | dense mini | 0.6759 | 0.5387 | 0.4412 | 0.5077 | 0.5192 | 0.5729 | 0.6004 | 0.4658 | 0.6032 |
215
- | e5-small | 0.7766 | 0.6548 | 0.5970 | 0.6736 | 0.6790 | 0.7214 | 0.7437 | 0.6169 | 0.7246 |
216
- | e5-base | 0.7957 | 0.7201 | 0.6350 | 0.6957 | 0.7202 | 0.7534 | 0.7245 | 0.6418 | 0.7527 |
217
- | e5-large | 0.7976 | **0.7544** | 0.6620 | 0.7077 | **0.7643** | **0.7756** | 0.6741 | 0.6208 | 0.7629 |
218
- | M3 dense | 0.5941 | 0.4746 | 0.3609 | 0.4563 | 0.4518 | 0.5221 | 0.6291 | 0.3917 | 0.5295 |
219
- | SPLADE mini | 0.6338 | 0.5656 | 0.6148 | 0.6319 | 0.6237 | 0.6027 | 0.7358 | 0.5185 | 0.6130 |
220
- | M3 sparse | 0.6568 | 0.5950 | 0.6337 | 0.6157 | 0.6376 | 0.6108 | 0.7417 | 0.5047 | 0.6302 |
221
- | multivector mini | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
222
- | multivector e5-small | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
223
- | multivector e5-base | 0.8145 | 0.7413 | 0.7000 | 0.7524 | 0.7459 | 0.7612 | 0.8296 | **0.6731** | **0.7760** |
224
- | M3 multi | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | 0.8716 | 0.5789 | 0.6982 |
225
- | reranker mini | 0.7824 | 0.7139 | 0.6618 | 0.7122 | 0.7296 | 0.7152 | 0.8793 | 0.6361 | 0.7431 |
226
- | reranker e5-small | 0.7984 | 0.6900 | 0.7023 | 0.7539 | 0.7379 | 0.7369 | **0.8850** | 0.6541 | 0.7580 |
227
- | reranker e5-base | **0.8156** | 0.7178 | **0.7240** | **0.7572** | 0.7527 | 0.7498 | 0.8509 | 0.6553 | 0.7741 |
228
-
229
- In Multivector, **bold** is the best multivector model in that column.
230
 
231
  ### Multivector
232
 
@@ -234,37 +234,37 @@ In Multivector, **bold** is the best multivector model in that column.
234
 
235
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
236
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
237
- | multivector mini | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | **0.9388** | 0.7841 | 0.8471 |
238
- | multivector e5-small | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
239
- | multivector e5-base | **0.9034** | **0.8718** | **0.8246** | 0.8432 | **0.8695** | **0.8719** | **0.9388** | **0.7922** | **0.8789** |
240
- | M3 multi | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
241
 
242
  #### recall@10
243
 
244
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
245
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
246
- | multivector mini | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
247
- | multivector e5-small | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
248
- | multivector e5-base | **0.9300** | **0.9206** | **0.8758** | 0.8649 | **0.9123** | **0.9152** | 0.9388 | **0.8394** | **0.9140** |
249
- | M3 multi | 0.8896 | 0.8484 | 0.8529 | **0.8865** | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
250
 
251
  #### ndcg@10
252
 
253
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
254
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
255
- | multivector mini | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
256
- | multivector e5-small | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
257
- | multivector e5-base | **0.8428** | **0.7853** | **0.7429** | 0.7801 | **0.7867** | **0.7989** | 0.8569 | **0.7137** | **0.8098** |
258
- | M3 multi | 0.7690 | 0.6967 | 0.7113 | **0.7824** | 0.7315 | 0.7365 | **0.8879** | 0.6265 | 0.7410 |
259
 
260
  #### mrr@10
261
 
262
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
263
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
264
- | multivector mini | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
265
- | multivector e5-small | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
266
- | multivector e5-base | **0.8145** | **0.7413** | **0.7000** | **0.7524** | **0.7459** | **0.7612** | 0.8296 | **0.6731** | **0.7760** |
267
- | M3 multi | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | **0.8716** | 0.5789 | 0.6982 |
268
 
269
  ## Training Details
270
 
@@ -288,6 +288,19 @@ In Multivector, **bold** is the best multivector model in that column.
288
 
289
  ### BibTeX
290
 
 
 
 
 
 
 
 
 
 
 
 
 
 
291
  #### CachedMultiVectorMultipleNegativesRankingLoss
292
  ```bibtex
293
  @misc{gao2021scaling,
 
144
  ```
145
  ## Evaluation
146
 
147
+ Full-corpus passage retrieval on [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR) for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. `avg` is the query-weighted average across those languages. The best value in that column is bold.
148
 
149
  Reranker rows rerank the top 50 passages from a frozen dense retriever.
150
 
 
154
 
155
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
156
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
157
+ | [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.7805 | 0.6940 | 0.5643 | 0.6270 | 0.6637 | 0.7081 | 0.7755 | 0.6024 | 0.7233 |
158
+ | [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.8741 | 0.8014 | 0.7418 | 0.8162 | 0.8088 | 0.8451 | 0.8571 | 0.7359 | 0.8379 |
159
+ | [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.8886 | 0.8691 | 0.7810 | 0.8000 | 0.8526 | 0.8738 | 0.8571 | 0.7590 | 0.8629 |
160
+ | [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.8948 | **0.8809** | 0.7974 | 0.8324 | 0.8841 | **0.8993** | 0.7755 | 0.7550 | 0.8738 |
161
+ | [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7067 | 0.6264 | 0.4630 | 0.5892 | 0.5894 | 0.6488 | 0.7551 | 0.5231 | 0.6501 |
162
+ | [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.7759 | 0.7446 | 0.7603 | 0.7405 | 0.7705 | 0.7546 | 0.8163 | 0.6657 | 0.7599 |
163
+ | [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7826 | 0.7356 | 0.7593 | 0.7189 | 0.7627 | 0.7463 | 0.8367 | 0.6536 | 0.7598 |
164
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | 0.9388 | 0.7841 | 0.8471 |
165
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
166
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.9034 | 0.8718 | 0.8246 | 0.8432 | 0.8695 | 0.8719 | 0.9388 | 0.7922 | 0.8789 |
167
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
168
+ | [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8647 | 0.8159 | 0.7462 | 0.7946 | 0.8245 | 0.8305 | 0.9184 | 0.7359 | 0.8336 |
169
+ | [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.8907 | 0.8430 | 0.8279 | 0.8541 | 0.8673 | 0.8693 | **0.9592** | 0.7871 | 0.8690 |
170
+ | [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.9082** | 0.8727 | **0.8497** | 0.8486 | **0.8943** | 0.8725 | 0.8980 | **0.7932** | **0.8853** |
171
 
172
  #### recall@10
173
 
174
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
175
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
176
+ | [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.8285 | 0.7644 | 0.6362 | 0.6973 | 0.7334 | 0.7827 | 0.8163 | 0.6737 | 0.7820 |
177
+ | [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.9072 | 0.8718 | 0.8007 | 0.8811 | 0.8774 | 0.9012 | 0.8980 | 0.7982 | 0.8844 |
178
+ | [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.9209 | 0.9070 | 0.8410 | 0.8649 | 0.9224 | 0.9133 | 0.8571 | 0.8233 | 0.9042 |
179
+ | [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.9273 | **0.9242** | 0.8551 | 0.8811 | **0.9359** | **0.9344** | 0.8163 | 0.8143 | 0.9131 |
180
+ | [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7584 | 0.6931 | 0.5381 | 0.6595 | 0.6704 | 0.7208 | 0.7755 | 0.5944 | 0.7112 |
181
+ | [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.8293 | 0.7924 | 0.8115 | 0.8162 | 0.8425 | 0.8317 | 0.8776 | 0.7440 | 0.8192 |
182
+ | [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8279 | 0.8087 | 0.8137 | 0.8000 | 0.8256 | 0.8107 | 0.8776 | 0.7169 | 0.8138 |
183
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
184
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
185
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.9300 | 0.9206 | 0.8758 | 0.8649 | 0.9123 | 0.9152 | 0.9388 | 0.8394 | 0.9140 |
186
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8896 | 0.8484 | 0.8529 | 0.8865 | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
187
+ | [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8872 | 0.8547 | 0.7800 | 0.8324 | 0.8605 | 0.8662 | **0.9592** | 0.7620 | 0.8615 |
188
+ | [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.9206 | 0.8917 | 0.8704 | 0.8811 | 0.9168 | 0.9191 | **0.9592** | 0.8414 | 0.9072 |
189
+ | [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.9345** | 0.9224 | **0.8987** | **0.8919** | 0.9314 | **0.9344** | **0.9592** | **0.8454** | **0.9229** |
190
 
191
  #### ndcg@10
192
 
193
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
194
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
195
+ | [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.7129 | 0.5933 | 0.4881 | 0.5533 | 0.5709 | 0.6234 | 0.6532 | 0.5160 | 0.6464 |
196
+ | [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.8086 | 0.7077 | 0.6464 | 0.7233 | 0.7270 | 0.7653 | 0.7810 | 0.6609 | 0.7635 |
197
+ | [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.8264 | 0.7661 | 0.6851 | 0.7362 | 0.7691 | 0.7929 | 0.7583 | 0.6855 | 0.7897 |
198
+ | [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.8294 | **0.7962** | 0.7089 | 0.7499 | **0.8061** | **0.8147** | 0.7093 | 0.6677 | 0.7997 |
199
+ | [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6340 | 0.5278 | 0.4034 | 0.5053 | 0.5045 | 0.5699 | 0.6650 | 0.4406 | 0.5734 |
200
+ | [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.6816 | 0.6213 | 0.6629 | 0.6763 | 0.6768 | 0.6579 | 0.7696 | 0.5729 | 0.6633 |
201
+ | [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6987 | 0.6469 | 0.6776 | 0.6599 | 0.6835 | 0.6593 | 0.7754 | 0.5562 | 0.6750 |
202
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
203
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
204
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.8428 | 0.7853 | 0.7429 | 0.7801 | 0.7867 | 0.7989 | 0.8569 | **0.7137** | 0.8098 |
205
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7690 | 0.6967 | 0.7113 | 0.7824 | 0.7315 | 0.7365 | 0.8879 | 0.6265 | 0.7410 |
206
+ | [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8082 | 0.7483 | 0.6906 | 0.7416 | 0.7613 | 0.7523 | 0.8987 | 0.6671 | 0.7722 |
207
+ | [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.8284 | 0.7397 | 0.7435 | 0.7852 | 0.7814 | 0.7815 | **0.9037** | 0.6999 | 0.7946 |
208
+ | [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.8449** | 0.7681 | **0.7668** | **0.7903** | 0.7968 | 0.7947 | 0.8769 | 0.7018 | **0.8107** |
209
 
210
  #### mrr@10
211
 
212
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
213
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
214
+ | [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.6759 | 0.5387 | 0.4412 | 0.5077 | 0.5192 | 0.5729 | 0.6004 | 0.4658 | 0.6032 |
215
+ | [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.7766 | 0.6548 | 0.5970 | 0.6736 | 0.6790 | 0.7214 | 0.7437 | 0.6169 | 0.7246 |
216
+ | [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.7957 | 0.7201 | 0.6350 | 0.6957 | 0.7202 | 0.7534 | 0.7245 | 0.6418 | 0.7527 |
217
+ | [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.7976 | **0.7544** | 0.6620 | 0.7077 | **0.7643** | **0.7756** | 0.6741 | 0.6208 | 0.7629 |
218
+ | [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.5941 | 0.4746 | 0.3609 | 0.4563 | 0.4518 | 0.5221 | 0.6291 | 0.3917 | 0.5295 |
219
+ | [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.6338 | 0.5656 | 0.6148 | 0.6319 | 0.6237 | 0.6027 | 0.7358 | 0.5185 | 0.6130 |
220
+ | [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6568 | 0.5950 | 0.6337 | 0.6157 | 0.6376 | 0.6108 | 0.7417 | 0.5047 | 0.6302 |
221
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
222
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
223
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.8145 | 0.7413 | 0.7000 | 0.7524 | 0.7459 | 0.7612 | 0.8296 | **0.6731** | **0.7760** |
224
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | 0.8716 | 0.5789 | 0.6982 |
225
+ | [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.7824 | 0.7139 | 0.6618 | 0.7122 | 0.7296 | 0.7152 | 0.8793 | 0.6361 | 0.7431 |
226
+ | [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.7984 | 0.6900 | 0.7023 | 0.7539 | 0.7379 | 0.7369 | **0.8850** | 0.6541 | 0.7580 |
227
+ | [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.8156** | 0.7178 | **0.7240** | **0.7572** | 0.7527 | 0.7498 | 0.8509 | 0.6553 | 0.7741 |
228
+
229
+ In Multivector, bold is the best multivector model in that column.
230
 
231
  ### Multivector
232
 
 
234
 
235
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
236
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
237
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | **0.9388** | 0.7841 | 0.8471 |
238
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
239
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.9034** | **0.8718** | **0.8246** | 0.8432 | **0.8695** | **0.8719** | **0.9388** | **0.7922** | **0.8789** |
240
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
241
 
242
  #### recall@10
243
 
244
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
245
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
246
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
247
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
248
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.9300** | **0.9206** | **0.8758** | 0.8649 | **0.9123** | **0.9152** | 0.9388 | **0.8394** | **0.9140** |
249
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8896 | 0.8484 | 0.8529 | **0.8865** | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
250
 
251
  #### ndcg@10
252
 
253
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
254
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
255
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
256
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
257
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.8428** | **0.7853** | **0.7429** | 0.7801 | **0.7867** | **0.7989** | 0.8569 | **0.7137** | **0.8098** |
258
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7690 | 0.6967 | 0.7113 | **0.7824** | 0.7315 | 0.7365 | **0.8879** | 0.6265 | 0.7410 |
259
 
260
  #### mrr@10
261
 
262
  | model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
263
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
264
+ | [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
265
+ | [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
266
+ | [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.8145** | **0.7413** | **0.7000** | **0.7524** | **0.7459** | **0.7612** | 0.8296 | **0.6731** | **0.7760** |
267
+ | [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | **0.8716** | 0.5789 | 0.6982 |
268
 
269
  ## Training Details
270
 
 
288
 
289
  ### BibTeX
290
 
291
+ #### Sentence Transformers
292
+ ```bibtex
293
+ @inproceedings{reimers-2019-sentence-bert,
294
+ title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
295
+ author = "Reimers, Nils and Gurevych, Iryna",
296
+ booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
297
+ month = "11",
298
+ year = "2019",
299
+ publisher = "Association for Computational Linguistics",
300
+ url = "https://arxiv.org/abs/1908.10084",
301
+ }
302
+ ```
303
+
304
  #### CachedMultiVectorMultipleNegativesRankingLoss
305
  ```bibtex
306
  @misc{gao2021scaling,