nebhailema's picture
docs: put the comparison under each heading
0c21349 verified
|
Raw History Blame Contribute Delete
28.5 kB
---
language:
- am
- ha
- om
- sn
- so
- sw
- yo
- zu
license: apache-2.0
tags:
- sentence-transformers
- colbert
- late-interaction
- multi-vector
- information-retrieval
- african-languages
- multilingual
- generated_from_trainer
- dataset_size:112864
- loss:CachedMultiVectorMultipleNegativesRankingLoss
base_model: intfloat/multilingual-e5-base
widget:
- text: የኮንጎ ዲሞክራስያዊ ሪፑብሊክ ሞሴ ካቱምቢ (Moise Katumbi) የተባሉ ተቃዋሚ ፖለቲከኛ ባዕዳን ቅጥረኞችን ተጠቅመዋል
በሚለው ወሬ ላይ ምርመራ እንደጀመረች የመንግስት ባለስልጣኖች ትላንት ረቡዕ ተናግረዋል።የፍትህ ሚኒስትር አሌክስ ታምብዌ ምዋምባ
በርካታ የቀድሞ አሜሪካውያን ወታደሮች በካታንጋ ክፍለ-ሀገር የካቱምቢ ጠባቂዎች ሆነው እንየሰሩ መሆናቸውን የሚያሳይ “የሰነድ
መረጃ” አለን ብለዋል።ካታንጋ የተባለችው የነሃስ ማዕዳን ያላት ክፍለ-ሀገር አስተዳዳሪ ሆነው ሰርተው የነበሩት ካቱምቢ ባለፈው
መስከረም ወር ከፕረዚዳንት ጆሴፍ ካቢላ መንግስት ወጥተው ከተቃዋሚዎች ጋር ተቀላቅለዋል።አንድ ተቃዋሚ ቡድን ካቱምቢ በያዝነው
አመት በሚካሄደው ፕረዚዳንታዊ ምርጫ እንዲወዳደሩ የሚጠይቅ የስም ዝርዝር ባለፈው እሁድ አቅርቦ ካቱምቢ ተቀብለውታል።ተቃዋሚው
ፖለቲከኛ ባዕዳን ቅጥረኞችን እየመለመሉ ነው የሚለውን ክስ አስተባብለዋል። እኔን በፖለቲካ ለመጉዳት የታቀደ ክስ ነው ብለዋል።
- text: 'ዓመጺ፡ ኣብ መንጎ ደቂ ዓዲ ዝጸንሐ ዘይምርድዳእ ብዕርቂ ተፈቲሑ
ኣብ ዓመጺ - ንኡስ ዞባ በሪኽ ኣብ መንጎ ደቂ ዓዲ ዝጸንሐ ዘይምርድዳእ፡ ብ22 ለካቲት ኣብ ዝተኻየደ ስነ-ስርዓት ብዕርቂ
ተፈቲሑ።
ወኪል ሚኒስትሪ ዞባዊ ምምሕዳር፡ መራሕቲ ሃይማኖት፡ ኣባላት ባይቶን ተወለድቲ ዓድን ኣብ ዝተረኽቡሉ ስነ- ስርዓት፡ ኣመሓዳሪ
ዞባ ማእከል ሜጀር ጀነራል ሮሞዳን ዑስማን ኣውልያይ ኣብ ዘስመዖ ቃል፡ ናጽነት ፍረ ሓያል ሓድነትን ክቡር መስዋእትነትን ምዃኑ
ድሕሪ ምጥቃስ፡ እዚ ክብርታት’ዚ’ውን ኣብ ህንጸት ሃገር መሰረታዊ ብምዃኑ፡ ዜጋታት ዕጥቆም ክገብርዎ ኣዘኻኺሩ።'
- text: ማንሲቲ ኣንጻር ባርሳ ኣብ ዝሓለፉ ሳምንታት ኣታውን ወጻእን ብቕዓት ከተርኢ ዝጸንሐት ማንሲቲ ኣብ ዝሓለፉ ክልተ ጸወታታት
ገፊሕ ዓወት ክትስንድ በቒዓ ኢያ። እዚ ድማ ናብቲ ንቡር ብቕዓታ ከምዝተመልሰት ዘርኢ ምዃኑ ኣሰልጣኒኣ ማኑዬል ፐለግሪኒ ይገልጽ።
ዝበዝሐ ስዕረት ማንሲቲ ምስ ዘይምህላው ያያ ቱረ ተዛሚዱ’ዩ ዝግለጽ። ምስ ባርሳ ኣብ 2009 ነዚ ውድድር ዝተዓወተሉ ቱረ ኣብዚ
ናይ ሎሚ ጸወታ ብምኽንያት መቅጻዕቲ ኣይክስለፍን ኢዩ። እዚ ድማ ንመስመር ምክፍፋል’ታ ጋንታ ከፍኩሶ ከምዝኽእል ይንገር። ይኹን
እምበር ማንሲቲ ኣብ ልዕሊ ክለባት ባየርሙኒክን ሮማን ዘመዝገበቶ ዓወታት ብዘይ ያያ ቱረ ብምንባሩ ፡ ኣብዚ ናይ ሎሚ ምሸት
ጸወታ’ውን ተመሳሳሊ ውጽኢት ክምዝገብ ትጽቢት ደገፍቲ ማን ሲቲ ኢዩ። ከም ፈርናንዶ፡ ፈርናንዲኖን ሚልነርን ዝኣመሰሉ ተጻወትቲ
ንቦታ ቱረ ክሽፍኑ እምንቶ ኣሰልጣኒ ፐለግሪኒ ኢዩ።
- text: 'ሳራዬቮ (Сарајево) የቦስኒያ-ሄርጸጎቪና ዋና ከተማ ነው።
የሚኖርበት የህዝብ ቁጥር (1998 ዓ.ም.) 602,500 ሆኖ ይገመታል። ከተማው ላይ ተቀምጦ ይገኛል።
በታሪክ ብዙ ሰፈሮች በሥፍራው አካባቢ ይገኙ ሲሆን ዘመናዊው ከተማ በኦቶማን ቱርክ መንግሥት በ1453 አ.ም. አካባቢ ተሠራ።
ዋና ከተሞች
የአውሮፓ ከተሞች
ቦስኒያና ሄርጸጎቪና'
- text: ኣብ ሕብሪ ጸጕርና ዘሎ ፍልልይ ክመጽእ ዝኸኣለ ብኸመይ ኢዩ? ዘ ዎርልድ ቡክ ኢንሳይክሎፔድያ ከምዝ ብምባል ይገልጽ፦
“ሕብሪ ጸጕርና ብቐንዱ ዝውሰን ብመጠን ብዝሒ ናይቲ መላኒን ዝበሃል ሓውሲ ቡናውን ጸሊምን ቀለም ኢዩ።” መላኒን ተፈጥሮኣዊ
ቀለም ኢዩ፡ ኣብ ጸጕርና፡ ቈርበትና፡ ከምኡውን ኣብ ዓይንና ኸኣ ይርከብ። እዝ ቐለም እዝ ብብዝሒ እንተደኣ ሃልዩካ ጸጕርኻ
ጸሊም ይኸውን። ብዝሒ ናይዝ ቐለም እናወሓደ ምስ ዝኸይድ ከኣ ናይ ጸጕርኻ ቐለም ካብ ጸሊም ናብ ቡናዊ ወይ ቀይሕ ወይ ጨዓይ
ይቕየር። ፈጺሙ እንተ ዘይብልካ ድማ ጸጕርኻ ብሪቕርቕ ዝብል ጻዕዳ ይኸውን። ኣብ ጸጕሪ ዘጋጥም ጸገም ሓደ ምስሓውን ምምላጥን
ኢዩ። እዝ ጸገማት እዝ ካብ ጥንቲ ዝነበረ ኢዩ።
datasets:
- nebhailema/AfriXIR
pipeline_tag: feature-extraction
library_name: sentence-transformers
---
# AfriXIR multi-vector retriever
This is a [Multi-Vector Encoder](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html) model finetuned from [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) on the [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR) dataset using the [sentence-transformers](https://www.SBERT.net) library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for passage retrieval.
## Model Details
### Model Description
- **Model Type:** Multi-Vector Encoder
- **Base model:** [intfloat/multilingual-e5-base](https://huggingface.co/intfloat/multilingual-e5-base) <!-- at revision d128750597153bb5987e10b1c3493a34e5a4502a -->
- **Maximum Sequence Length:** 512 tokens
- **Maximum Query Length:** 512 tokens
- **Maximum Document Length:** 512 tokens
- **Output Dimensionality:** 128 dimensions
- **Similarity Function:** MaxSim
- **Supported Modality:** Text
- **Training Dataset:**
- [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR)
- **Languages:** am, ha, om, sn, so, sw, yo, zu
- **License:** apache-2.0
### Model Sources
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net)
- **Documentation:** [Multi-Vector Encoder Documentation](https://www.sbert.net/docs/multi_vector_encoder/usage/usage.html)
- **Repository:** [Sentence Transformers on GitHub](https://github.com/huggingface/sentence-transformers)
- **Hugging Face:** [Multi-Vector Encoders on Hugging Face](https://huggingface.co/models?library=sentence-transformers&other=multi-vector)
### Full Model Architecture
```
MultiVectorEncoder(
(0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 512, 'document_length': 512, 'architecture': 'XLMRobertaModel'})
(1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
(2): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
(3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)
```
## Usage
### Direct Usage (Sentence Transformers)
First install the Sentence Transformers library:
```bash
pip install -U sentence-transformers
```
Then you can load this model and run inference.
```python
from sentence_transformers import MultiVectorEncoder
# Download from the 🤗 Hub
model = MultiVectorEncoder("nebhailema/afrixir-multivector-e5-base")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
'ስሩዕ ክብድን ዝበለ ልምምድ ኤሮቢክስ ምዝውታር ኣካላትና እንታይ ክንረክብ ይሕግዘና?',
]
documents = [
'ስሩዕን ክብድን ዝበለ ልምምድ ኤሮቢክስ ምዝውታር ኣካላትና እኹል ኦክስጅን ክንረክብ ይሕግዘና። ‘adenosine triphosphate’ ዝተባህለ ጥረ ነገር ብብዝሒ ኣብ ዋህዮታት ጭዋዳታትና ክፈሪ ይርአ። ‘adenosine triphosphate’ ጭዋዳታትና እኹል ጸዓት/ ሓይሊ ንኽረክብ ዝሕግዝ ባእታ’ዩ። ብኡኡ ኣቢሉ’ውን ደምና እኹል ኦክስጅን ይረክብ። በንጻሩ ውሑድ ‘adenosine triphosphate’ ዘፍሪ ሰብ ቀጻሊ ኣካላዊ ስንፈት/ ድኻም ይረኣዮ። እኹል ሰብ ኣብ ሰሙን ካብ 3 ክሳብ 5 ግዜ ብዘይ ምቁራጽ 15 ደቓይቕ ዝከውን ልምምድ ኤሮቢክስ ምስ ዘካይድ እኹል’ዩ። ዕድሚኦም ልዕሊ 35 ዓመት ዝኹኑ ደቂተባዕትዮ ይኹኒ ደቀኣንስትዮ ቅድሚ ምጅማር መደብ ልምምድ ኤሮቢክስ፡ ሕክምናዊ መርመራ ከካይዱ/ ከካይዳ ከምዘለዎም/ ከምዘለወን ክኢላታት ይመኽሩ።',
'እታ ነዊሕ እትጓዓዝ ዑፍ - ኣልባጥሮስ\nዝዝምብያ ኣዕዋፍ፡ ብሒደት ጕልበት እየን ኣብ ኣየር ዝንሳፈፋ። እታ ነዊሕ ጕዕዞ እትጓዓዝ ኣልባጥሮስ ከኣ፡ ዝበለጸት ኣብነት’ያ። እዛ ዑፍ እዝኣ፡ 3.4 ሜተር ዝንውሓቱ ኣኽናፍ ዘለዋን 8.5 ኪሎ ግራም ዝኽብደታን ኰይና፡ ብዙሕ ጕልበት ከይተጠቕመት ንኣሽሓት ኪሎ ሜትራት ክትበርር ትኽእል እያ። ከምኡ ኽትገብር ዝኸኣለት፡ ሳላ ቕርጺ ሰብነታን ሳላ እቲ እትዝምብየሉ መገድን እዩ። ነዝ ዝስዕብ ሕሰበሉ፦ እታ ዑፍ ክትበርር ከላን ንኣኽናፋ ምሉእ ብምሉእ ክትዝርግሖ ኸላን፡ ንኣኽናፋ ኣሲሩ ዝሕዞ ፍሉይ ጅማት ትጥቀም እያ። እዝ ኸኣ ጭዋዳታታ ንኸይደክም ይሕግዛ እዩ። ብዘይካ’ዝ፡ ንነዊሕ ሰዓታት ክትዝምቢ እትኽእል፡ ነቲ ኻብ ውቅያኖሳት ዝመጽእ ንፋስ ብውሕልነት ስለ እትጥቀመሉ’ያ።',
'ሰሪ ኣ\nግጥማት ሰሪ ኣ ኢጣልያ ብቐዳም ኣብ ዝቐጸለሉ፡ ሮማ ክልተ ነጥቢ ከሲራ። ሮማ ምስ መራሒት ዘላ ጁቬንቱስ ናይ ሽዱሽተ ነጥቢ ፍልልይ እዩ ዘለዋ። ነዚ ንምጽባብ ብቐዳም ኣንጻር ኢምፖሊ ተጻዊታ ብምዕዋት፡ ኣብ ልዕሊ እታ ሰንበት ምስ ኡዲነዘ እትገጥም ጁቬንቱስ ጸቕጢ ክትፈጥር ትኽእሎ ነርዋ። ይኹን እምበር ሮማ ኣብ ሜዳኣ ምስ ኢምፓሊ 1ብ1 ብምፍልላያ ነቲ ዕድል ኣባኺናቶ። ሮማ ሳልሳይ ተኸታታሊ ጸወታኣ ነጥቢ ተማቒላ ትወጽእ ብምህላዋ፡ ጁቬቱስ መሪሕነት ሰሪ ኣ ኢጣልያ ቀሲና ክትቅልሶ ኣኽኢልዋ ኣሎ።\n\nብዘይካዚ ብቐዳም ጀኖኣ ምስ ፊዮረንቲና 1ብ1 ተፈላልየን። ዝበዝሕ ግጥማት’ቲ ሊግ ትማሊ ሰንበት እዩ ተኻይዱ።\n\nጁቬንቱስ ሓደ ጸወታ ካብ ሮማ ድሒራ ብ49 ነጥቢ ቀዳመይቲ፡ ሮማ ብ43 ነጥቢ ካልኣይቲ ናፖሊ ብ36 ነጥቢ ሳልሰይቲ ኮይነን ይመርሓ።',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (41, 128) (306, 128)
# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[35.1270, 19.4954, 17.0075]])
```
## Evaluation
Full-corpus passage retrieval on [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR) for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. `avg` is the query-weighted average across those languages. The best value in that column is bold.
Reranker rows rerank the top 50 passages from a frozen dense retriever.
### Multivector
Multivector retrievers compared with each other on recall@5, recall@10, nDCG@10, and MRR@10.
#### recall@5
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | **0.9388** | 0.7841 | 0.8471 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.9034** | **0.8718** | **0.8246** | 0.8432 | **0.8695** | **0.8719** | **0.9388** | **0.7922** | **0.8789** |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
#### recall@10
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.9300** | **0.9206** | **0.8758** | 0.8649 | **0.9123** | **0.9152** | 0.9388 | **0.8394** | **0.9140** |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8896 | 0.8484 | 0.8529 | **0.8865** | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
#### ndcg@10
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.8428** | **0.7853** | **0.7429** | 0.7801 | **0.7867** | **0.7989** | 0.8569 | **0.7137** | **0.8098** |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7690 | 0.6967 | 0.7113 | **0.7824** | 0.7315 | 0.7365 | **0.8879** | 0.6265 | 0.7410 |
#### mrr@10
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | **0.8145** | **0.7413** | **0.7000** | **0.7524** | **0.7459** | **0.7612** | 0.8296 | **0.6731** | **0.7760** |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | **0.8716** | 0.5789 | 0.6982 |
### All models
Dense, sparse, multivector, and reranker models compared with each other on recall@5, recall@10, nDCG@10, and MRR@10.
#### recall@5
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.7805 | 0.6940 | 0.5643 | 0.6270 | 0.6637 | 0.7081 | 0.7755 | 0.6024 | 0.7233 |
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.8741 | 0.8014 | 0.7418 | 0.8162 | 0.8088 | 0.8451 | 0.8571 | 0.7359 | 0.8379 |
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.8886 | 0.8691 | 0.7810 | 0.8000 | 0.8526 | 0.8738 | 0.8571 | 0.7590 | 0.8629 |
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.8948 | **0.8809** | 0.7974 | 0.8324 | 0.8841 | **0.8993** | 0.7755 | 0.7550 | 0.8738 |
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7067 | 0.6264 | 0.4630 | 0.5892 | 0.5894 | 0.6488 | 0.7551 | 0.5231 | 0.6501 |
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.7759 | 0.7446 | 0.7603 | 0.7405 | 0.7705 | 0.7546 | 0.8163 | 0.6657 | 0.7599 |
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7826 | 0.7356 | 0.7593 | 0.7189 | 0.7627 | 0.7463 | 0.8367 | 0.6536 | 0.7598 |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8710 | 0.8177 | 0.8126 | 0.8162 | 0.8358 | 0.8311 | 0.9388 | 0.7841 | 0.8471 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8868 | 0.8204 | 0.7832 | 0.8108 | 0.8515 | 0.8662 | 0.8980 | 0.7691 | 0.8578 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.9034 | 0.8718 | 0.8246 | 0.8432 | 0.8695 | 0.8719 | 0.9388 | 0.7922 | 0.8789 |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8485 | 0.7960 | 0.7919 | **0.8649** | 0.8211 | 0.8260 | 0.9184 | 0.7108 | 0.8246 |
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8647 | 0.8159 | 0.7462 | 0.7946 | 0.8245 | 0.8305 | 0.9184 | 0.7359 | 0.8336 |
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.8907 | 0.8430 | 0.8279 | 0.8541 | 0.8673 | 0.8693 | **0.9592** | 0.7871 | 0.8690 |
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.9082** | 0.8727 | **0.8497** | 0.8486 | **0.8943** | 0.8725 | 0.8980 | **0.7932** | **0.8853** |
#### recall@10
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.8285 | 0.7644 | 0.6362 | 0.6973 | 0.7334 | 0.7827 | 0.8163 | 0.6737 | 0.7820 |
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.9072 | 0.8718 | 0.8007 | 0.8811 | 0.8774 | 0.9012 | 0.8980 | 0.7982 | 0.8844 |
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.9209 | 0.9070 | 0.8410 | 0.8649 | 0.9224 | 0.9133 | 0.8571 | 0.8233 | 0.9042 |
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.9273 | **0.9242** | 0.8551 | 0.8811 | **0.9359** | **0.9344** | 0.8163 | 0.8143 | 0.9131 |
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7584 | 0.6931 | 0.5381 | 0.6595 | 0.6704 | 0.7208 | 0.7755 | 0.5944 | 0.7112 |
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.8293 | 0.7924 | 0.8115 | 0.8162 | 0.8425 | 0.8317 | 0.8776 | 0.7440 | 0.8192 |
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8279 | 0.8087 | 0.8137 | 0.8000 | 0.8256 | 0.8107 | 0.8776 | 0.7169 | 0.8138 |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.9008 | 0.8809 | 0.8725 | 0.8541 | 0.8898 | 0.8802 | 0.9388 | 0.8323 | 0.8877 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.9183 | 0.8755 | 0.8573 | 0.8757 | 0.9033 | 0.9146 | **0.9592** | 0.8102 | 0.8995 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.9300 | 0.9206 | 0.8758 | 0.8649 | 0.9123 | 0.9152 | 0.9388 | 0.8394 | 0.9140 |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.8896 | 0.8484 | 0.8529 | 0.8865 | 0.8763 | 0.8878 | 0.9388 | 0.7751 | 0.8732 |
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8872 | 0.8547 | 0.7800 | 0.8324 | 0.8605 | 0.8662 | **0.9592** | 0.7620 | 0.8615 |
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.9206 | 0.8917 | 0.8704 | 0.8811 | 0.9168 | 0.9191 | **0.9592** | 0.8414 | 0.9072 |
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.9345** | 0.9224 | **0.8987** | **0.8919** | 0.9314 | **0.9344** | **0.9592** | **0.8454** | **0.9229** |
#### ndcg@10
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.7129 | 0.5933 | 0.4881 | 0.5533 | 0.5709 | 0.6234 | 0.6532 | 0.5160 | 0.6464 |
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.8086 | 0.7077 | 0.6464 | 0.7233 | 0.7270 | 0.7653 | 0.7810 | 0.6609 | 0.7635 |
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.8264 | 0.7661 | 0.6851 | 0.7362 | 0.7691 | 0.7929 | 0.7583 | 0.6855 | 0.7897 |
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.8294 | **0.7962** | 0.7089 | 0.7499 | **0.8061** | **0.8147** | 0.7093 | 0.6677 | 0.7997 |
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6340 | 0.5278 | 0.4034 | 0.5053 | 0.5045 | 0.5699 | 0.6650 | 0.4406 | 0.5734 |
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.6816 | 0.6213 | 0.6629 | 0.6763 | 0.6768 | 0.6579 | 0.7696 | 0.5729 | 0.6633 |
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6987 | 0.6469 | 0.6776 | 0.6599 | 0.6835 | 0.6593 | 0.7754 | 0.5562 | 0.6750 |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.8028 | 0.7163 | 0.7364 | 0.7599 | 0.7551 | 0.7352 | 0.8785 | 0.6876 | 0.7690 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.8272 | 0.7280 | 0.7167 | 0.7688 | 0.7617 | 0.7801 | 0.8876 | 0.6858 | 0.7880 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.8428 | 0.7853 | 0.7429 | 0.7801 | 0.7867 | 0.7989 | 0.8569 | **0.7137** | 0.8098 |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7690 | 0.6967 | 0.7113 | 0.7824 | 0.7315 | 0.7365 | 0.8879 | 0.6265 | 0.7410 |
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.8082 | 0.7483 | 0.6906 | 0.7416 | 0.7613 | 0.7523 | 0.8987 | 0.6671 | 0.7722 |
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.8284 | 0.7397 | 0.7435 | 0.7852 | 0.7814 | 0.7815 | **0.9037** | 0.6999 | 0.7946 |
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.8449** | 0.7681 | **0.7668** | **0.7903** | 0.7968 | 0.7947 | 0.8769 | 0.7018 | **0.8107** |
#### mrr@10
| model | amharic | hausa | oromo | shona | somali | swahili | tigrinya | yoruba | avg |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| [dense mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini) | 0.6759 | 0.5387 | 0.4412 | 0.5077 | 0.5192 | 0.5729 | 0.6004 | 0.4658 | 0.6032 |
| [e5-small](https://huggingface.co/nebhailema/afrixir-e5-small) | 0.7766 | 0.6548 | 0.5970 | 0.6736 | 0.6790 | 0.7214 | 0.7437 | 0.6169 | 0.7246 |
| [e5-base](https://huggingface.co/nebhailema/afrixir-e5-base) | 0.7957 | 0.7201 | 0.6350 | 0.6957 | 0.7202 | 0.7534 | 0.7245 | 0.6418 | 0.7527 |
| [e5-large](https://huggingface.co/nebhailema/afrixir-e5-large) | 0.7976 | **0.7544** | 0.6620 | 0.7077 | **0.7643** | **0.7756** | 0.6741 | 0.6208 | 0.7629 |
| [M3 dense](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.5941 | 0.4746 | 0.3609 | 0.4563 | 0.4518 | 0.5221 | 0.6291 | 0.3917 | 0.5295 |
| [SPLADE mini](https://huggingface.co/nebhailema/afrixir-splade-mini-mnrl) | 0.6338 | 0.5656 | 0.6148 | 0.6319 | 0.6237 | 0.6027 | 0.7358 | 0.5185 | 0.6130 |
| [M3 sparse](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.6568 | 0.5950 | 0.6337 | 0.6157 | 0.6376 | 0.6108 | 0.7417 | 0.5047 | 0.6302 |
| [multivector mini](https://huggingface.co/nebhailema/afrixir-multivector-mini) | 0.7709 | 0.6631 | 0.6930 | 0.7291 | 0.7118 | 0.6884 | 0.8571 | 0.6407 | 0.7305 |
| [multivector e5-small](https://huggingface.co/nebhailema/afrixir-multivector-e5-small) | 0.7977 | 0.6803 | 0.6723 | 0.7352 | 0.7160 | 0.7366 | 0.8652 | 0.6452 | 0.7520 |
| [multivector e5-base](https://huggingface.co/nebhailema/afrixir-multivector-e5-base) | 0.8145 | 0.7413 | 0.7000 | 0.7524 | 0.7459 | 0.7612 | 0.8296 | **0.6731** | **0.7760** |
| [M3 multi](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-m3) | 0.7296 | 0.6476 | 0.6660 | 0.7481 | 0.6850 | 0.6880 | 0.8716 | 0.5789 | 0.6982 |
| [reranker mini](https://huggingface.co/nebhailema/afrixir-afro-xlmr-mini-reranker) | 0.7824 | 0.7139 | 0.6618 | 0.7122 | 0.7296 | 0.7152 | 0.8793 | 0.6361 | 0.7431 |
| [reranker e5-small](https://huggingface.co/nebhailema/afrixir-e5-small-reranker) | 0.7984 | 0.6900 | 0.7023 | 0.7539 | 0.7379 | 0.7369 | **0.8850** | 0.6541 | 0.7580 |
| [reranker e5-base](https://huggingface.co/nebhailema/afrixir-e5-base-reranker) | **0.8156** | 0.7178 | **0.7240** | **0.7572** | 0.7527 | 0.7498 | 0.8509 | 0.6553 | 0.7741 |
## Training Details
### Training Dataset
#### AfriXIR
* Dataset: [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR)
* Size: 112,864 training samples
* Columns: <code>anchor</code>, <code>positive</code>, <code>negative_0</code>, <code>negative_1</code>, <code>negative_2</code>, <code>negative_3</code>, <code>negative_4</code>, <code>negative_5</code>, <code>negative_6</code>, and <code>negative_7</code>
### Evaluation Dataset
#### AfriXIR
* Dataset: [AfriXIR](https://huggingface.co/datasets/nebhailema/AfriXIR)
* Size: 12,546 evaluation samples
* Columns: <code>anchor</code>, <code>positive</code>, <code>negative_0</code>, <code>negative_1</code>, <code>negative_2</code>, <code>negative_3</code>, <code>negative_4</code>, <code>negative_5</code>, <code>negative_6</code>, and <code>negative_7</code>
## Citation
### BibTeX
#### Sentence Transformers
```bibtex
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
```
#### CachedMultiVectorMultipleNegativesRankingLoss
```bibtex
@misc{gao2021scaling,
title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
year={2021},
eprint={2101.06983},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
```