AfriXIR multi-vector retriever

This is a Multi-Vector Encoder model finetuned from intfloat/multilingual-e5-base on the AfriXIR dataset using the sentence-transformers library. It maps inputs to sequences of 128-dimensional token-level vectors and scores them with late interaction (MaxSim), useful for passage retrieval.

Model Details

Model Description

  • Model Type: Multi-Vector Encoder
  • Base model: intfloat/multilingual-e5-base
  • Maximum Sequence Length: 512 tokens
    • Maximum Query Length: 512 tokens
    • Maximum Document Length: 512 tokens
  • Output Dimensionality: 128 dimensions
  • Similarity Function: MaxSim
  • Supported Modality: Text
  • Training Dataset:
  • Languages: am, ha, om, sn, so, sw, yo, zu
  • License: apache-2.0

Model Sources

Full Model Architecture

MultiVectorEncoder(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'query_length': 512, 'document_length': 512, 'architecture': 'XLMRobertaModel'})
  (1): Dense({'in_features': 768, 'out_features': 128, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
  (2): MultiVectorMask({'skiplist_words': ['!', '"', '#', '$', '%', '&', "'", '(', ')', '*', '+', ',', '-', '.', '/', ':', ';', '<', '=', '>', '?', '@', '[', '\\', ']', '^', '_', '`', '{', '|', '}', '~'], 'skiplist_tasks': ['document'], 'keep_only_token_ids': None})
  (3): Normalize({'module_input_name': 'token_embeddings', 'module_output_name': 'token_embeddings'})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import MultiVectorEncoder

# Download from the 🤗 Hub
model = MultiVectorEncoder("nebhailema/afrixir-multivector-e5-base")
# Run inference: each input becomes a sequence of per-token vectors (variable length).
queries = [
    'ስሩዕ ክብድን ዝበለ ልምምድ ኤሮቢክስ ምዝውታር ኣካላትና እንታይ ክንረክብ ይሕግዘና?',
]
documents = [
    'ስሩዕን ክብድን ዝበለ ልምምድ ኤሮቢክስ ምዝውታር ኣካላትና እኹል ኦክስጅን ክንረክብ ይሕግዘና። ‘adenosine triphosphate’ ዝተባህለ ጥረ ነገር ብብዝሒ ኣብ ዋህዮታት ጭዋዳታትና ክፈሪ ይርአ። ‘adenosine triphosphate’ ጭዋዳታትና እኹል ጸዓት/ ሓይሊ ንኽረክብ ዝሕግዝ ባእታ’ዩ። ብኡኡ ኣቢሉ’ውን ደምና እኹል ኦክስጅን ይረክብ። በንጻሩ ውሑድ ‘adenosine triphosphate’ ዘፍሪ ሰብ ቀጻሊ ኣካላዊ ስንፈት/ ድኻም ይረኣዮ። እኹል ሰብ ኣብ ሰሙን ካብ 3 ክሳብ 5 ግዜ ብዘይ ምቁራጽ 15 ደቓይቕ ዝከውን ልምምድ ኤሮቢክስ ምስ ዘካይድ እኹል’ዩ። ዕድሚኦም ልዕሊ 35 ዓመት ዝኹኑ ደቂተባዕትዮ ይኹኒ ደቀኣንስትዮ ቅድሚ ምጅማር መደብ ልምምድ ኤሮቢክስ፡ ሕክምናዊ መርመራ ከካይዱ/ ከካይዳ ከምዘለዎም/ ከምዘለወን ክኢላታት ይመኽሩ።',
    'እታ ነዊሕ እትጓዓዝ ዑፍ - ኣልባጥሮስ\nዝዝምብያ ኣዕዋፍ፡ ብሒደት ጕልበት እየን ኣብ ኣየር ዝንሳፈፋ። እታ ነዊሕ ጕዕዞ እትጓዓዝ ኣልባጥሮስ ከኣ፡ ዝበለጸት ኣብነት’ያ። እዛ ዑፍ እዝኣ፡ 3.4 ሜተር ዝንውሓቱ ኣኽናፍ ዘለዋን 8.5 ኪሎ ግራም ዝኽብደታን ኰይና፡ ብዙሕ ጕልበት ከይተጠቕመት ንኣሽሓት ኪሎ ሜትራት ክትበርር ትኽእል እያ። ከምኡ ኽትገብር ዝኸኣለት፡ ሳላ ቕርጺ ሰብነታን ሳላ እቲ እትዝምብየሉ መገድን እዩ። ነዝ ዝስዕብ ሕሰበሉ፦ እታ ዑፍ ክትበርር ከላን ንኣኽናፋ ምሉእ ብምሉእ ክትዝርግሖ ኸላን፡ ንኣኽናፋ ኣሲሩ ዝሕዞ ፍሉይ ጅማት ትጥቀም እያ። እዝ ኸኣ ጭዋዳታታ ንኸይደክም ይሕግዛ እዩ። ብዘይካ’ዝ፡ ንነዊሕ ሰዓታት ክትዝምቢ እትኽእል፡ ነቲ ኻብ ውቅያኖሳት ዝመጽእ ንፋስ ብውሕልነት ስለ እትጥቀመሉ’ያ።',
    'ሰሪ ኣ\nግጥማት ሰሪ ኣ ኢጣልያ ብቐዳም ኣብ ዝቐጸለሉ፡ ሮማ ክልተ ነጥቢ ከሲራ። ሮማ ምስ መራሒት ዘላ ጁቬንቱስ ናይ ሽዱሽተ ነጥቢ ፍልልይ እዩ ዘለዋ። ነዚ ንምጽባብ ብቐዳም ኣንጻር ኢምፖሊ ተጻዊታ ብምዕዋት፡ ኣብ ልዕሊ እታ ሰንበት ምስ ኡዲነዘ እትገጥም ጁቬንቱስ ጸቕጢ ክትፈጥር ትኽእሎ ነርዋ። ይኹን እምበር ሮማ ኣብ ሜዳኣ ምስ ኢምፓሊ 1ብ1 ብምፍልላያ ነቲ ዕድል ኣባኺናቶ። ሮማ ሳልሳይ ተኸታታሊ ጸወታኣ ነጥቢ ተማቒላ ትወጽእ ብምህላዋ፡ ጁቬቱስ መሪሕነት ሰሪ ኣ ኢጣልያ ቀሲና ክትቅልሶ ኣኽኢልዋ ኣሎ።\n\nብዘይካዚ ብቐዳም ጀኖኣ ምስ ፊዮረንቲና 1ብ1 ተፈላልየን። ዝበዝሕ ግጥማት’ቲ ሊግ ትማሊ ሰንበት እዩ ተኻይዱ።\n\nጁቬንቱስ ሓደ ጸወታ ካብ ሮማ ድሒራ ብ49 ነጥቢ ቀዳመይቲ፡ ሮማ ብ43 ነጥቢ ካልኣይቲ ናፖሊ ብ36 ነጥቢ ሳልሰይቲ ኮይነን ይመርሓ።',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings[0].shape, document_embeddings[0].shape)
# (41, 128) (306, 128)

# Get the MaxSim similarity scores
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[35.1270, 19.4954, 17.0075]])

Evaluation

Full-corpus passage retrieval on AfriXIR for Amharic, Hausa, Oromo, Shona, Somali, Swahili, Tigrinya, and Yoruba. avg is the query-weighted average across those languages. The best value in that column is bold.

Reranker rows rerank the top 50 passages from a frozen dense retriever.

Multivector

Multivector retrievers compared with each other on recall@5, recall@10, nDCG@10, and MRR@10.

recall@5

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
multivector mini 0.8710 0.8177 0.8126 0.8162 0.8358 0.8311 0.9388 0.7841 0.8471
multivector e5-small 0.8868 0.8204 0.7832 0.8108 0.8515 0.8662 0.8980 0.7691 0.8578
multivector e5-base 0.9034 0.8718 0.8246 0.8432 0.8695 0.8719 0.9388 0.7922 0.8789
M3 multi 0.8485 0.7960 0.7919 0.8649 0.8211 0.8260 0.9184 0.7108 0.8246

recall@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
multivector mini 0.9008 0.8809 0.8725 0.8541 0.8898 0.8802 0.9388 0.8323 0.8877
multivector e5-small 0.9183 0.8755 0.8573 0.8757 0.9033 0.9146 0.9592 0.8102 0.8995
multivector e5-base 0.9300 0.9206 0.8758 0.8649 0.9123 0.9152 0.9388 0.8394 0.9140
M3 multi 0.8896 0.8484 0.8529 0.8865 0.8763 0.8878 0.9388 0.7751 0.8732

ndcg@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
multivector mini 0.8028 0.7163 0.7364 0.7599 0.7551 0.7352 0.8785 0.6876 0.7690
multivector e5-small 0.8272 0.7280 0.7167 0.7688 0.7617 0.7801 0.8876 0.6858 0.7880
multivector e5-base 0.8428 0.7853 0.7429 0.7801 0.7867 0.7989 0.8569 0.7137 0.8098
M3 multi 0.7690 0.6967 0.7113 0.7824 0.7315 0.7365 0.8879 0.6265 0.7410

mrr@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
multivector mini 0.7709 0.6631 0.6930 0.7291 0.7118 0.6884 0.8571 0.6407 0.7305
multivector e5-small 0.7977 0.6803 0.6723 0.7352 0.7160 0.7366 0.8652 0.6452 0.7520
multivector e5-base 0.8145 0.7413 0.7000 0.7524 0.7459 0.7612 0.8296 0.6731 0.7760
M3 multi 0.7296 0.6476 0.6660 0.7481 0.6850 0.6880 0.8716 0.5789 0.6982

All models

Dense, sparse, multivector, and reranker models compared with each other on recall@5, recall@10, nDCG@10, and MRR@10.

recall@5

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.7805 0.6940 0.5643 0.6270 0.6637 0.7081 0.7755 0.6024 0.7233
e5-small 0.8741 0.8014 0.7418 0.8162 0.8088 0.8451 0.8571 0.7359 0.8379
e5-base 0.8886 0.8691 0.7810 0.8000 0.8526 0.8738 0.8571 0.7590 0.8629
e5-large 0.8948 0.8809 0.7974 0.8324 0.8841 0.8993 0.7755 0.7550 0.8738
M3 dense 0.7067 0.6264 0.4630 0.5892 0.5894 0.6488 0.7551 0.5231 0.6501
SPLADE mini 0.7759 0.7446 0.7603 0.7405 0.7705 0.7546 0.8163 0.6657 0.7599
M3 sparse 0.7826 0.7356 0.7593 0.7189 0.7627 0.7463 0.8367 0.6536 0.7598
multivector mini 0.8710 0.8177 0.8126 0.8162 0.8358 0.8311 0.9388 0.7841 0.8471
multivector e5-small 0.8868 0.8204 0.7832 0.8108 0.8515 0.8662 0.8980 0.7691 0.8578
multivector e5-base 0.9034 0.8718 0.8246 0.8432 0.8695 0.8719 0.9388 0.7922 0.8789
M3 multi 0.8485 0.7960 0.7919 0.8649 0.8211 0.8260 0.9184 0.7108 0.8246
reranker mini 0.8647 0.8159 0.7462 0.7946 0.8245 0.8305 0.9184 0.7359 0.8336
reranker e5-small 0.8907 0.8430 0.8279 0.8541 0.8673 0.8693 0.9592 0.7871 0.8690
reranker e5-base 0.9082 0.8727 0.8497 0.8486 0.8943 0.8725 0.8980 0.7932 0.8853

recall@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.8285 0.7644 0.6362 0.6973 0.7334 0.7827 0.8163 0.6737 0.7820
e5-small 0.9072 0.8718 0.8007 0.8811 0.8774 0.9012 0.8980 0.7982 0.8844
e5-base 0.9209 0.9070 0.8410 0.8649 0.9224 0.9133 0.8571 0.8233 0.9042
e5-large 0.9273 0.9242 0.8551 0.8811 0.9359 0.9344 0.8163 0.8143 0.9131
M3 dense 0.7584 0.6931 0.5381 0.6595 0.6704 0.7208 0.7755 0.5944 0.7112
SPLADE mini 0.8293 0.7924 0.8115 0.8162 0.8425 0.8317 0.8776 0.7440 0.8192
M3 sparse 0.8279 0.8087 0.8137 0.8000 0.8256 0.8107 0.8776 0.7169 0.8138
multivector mini 0.9008 0.8809 0.8725 0.8541 0.8898 0.8802 0.9388 0.8323 0.8877
multivector e5-small 0.9183 0.8755 0.8573 0.8757 0.9033 0.9146 0.9592 0.8102 0.8995
multivector e5-base 0.9300 0.9206 0.8758 0.8649 0.9123 0.9152 0.9388 0.8394 0.9140
M3 multi 0.8896 0.8484 0.8529 0.8865 0.8763 0.8878 0.9388 0.7751 0.8732
reranker mini 0.8872 0.8547 0.7800 0.8324 0.8605 0.8662 0.9592 0.7620 0.8615
reranker e5-small 0.9206 0.8917 0.8704 0.8811 0.9168 0.9191 0.9592 0.8414 0.9072
reranker e5-base 0.9345 0.9224 0.8987 0.8919 0.9314 0.9344 0.9592 0.8454 0.9229

ndcg@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.7129 0.5933 0.4881 0.5533 0.5709 0.6234 0.6532 0.5160 0.6464
e5-small 0.8086 0.7077 0.6464 0.7233 0.7270 0.7653 0.7810 0.6609 0.7635
e5-base 0.8264 0.7661 0.6851 0.7362 0.7691 0.7929 0.7583 0.6855 0.7897
e5-large 0.8294 0.7962 0.7089 0.7499 0.8061 0.8147 0.7093 0.6677 0.7997
M3 dense 0.6340 0.5278 0.4034 0.5053 0.5045 0.5699 0.6650 0.4406 0.5734
SPLADE mini 0.6816 0.6213 0.6629 0.6763 0.6768 0.6579 0.7696 0.5729 0.6633
M3 sparse 0.6987 0.6469 0.6776 0.6599 0.6835 0.6593 0.7754 0.5562 0.6750
multivector mini 0.8028 0.7163 0.7364 0.7599 0.7551 0.7352 0.8785 0.6876 0.7690
multivector e5-small 0.8272 0.7280 0.7167 0.7688 0.7617 0.7801 0.8876 0.6858 0.7880
multivector e5-base 0.8428 0.7853 0.7429 0.7801 0.7867 0.7989 0.8569 0.7137 0.8098
M3 multi 0.7690 0.6967 0.7113 0.7824 0.7315 0.7365 0.8879 0.6265 0.7410
reranker mini 0.8082 0.7483 0.6906 0.7416 0.7613 0.7523 0.8987 0.6671 0.7722
reranker e5-small 0.8284 0.7397 0.7435 0.7852 0.7814 0.7815 0.9037 0.6999 0.7946
reranker e5-base 0.8449 0.7681 0.7668 0.7903 0.7968 0.7947 0.8769 0.7018 0.8107

mrr@10

model amharic hausa oromo shona somali swahili tigrinya yoruba avg
dense mini 0.6759 0.5387 0.4412 0.5077 0.5192 0.5729 0.6004 0.4658 0.6032
e5-small 0.7766 0.6548 0.5970 0.6736 0.6790 0.7214 0.7437 0.6169 0.7246
e5-base 0.7957 0.7201 0.6350 0.6957 0.7202 0.7534 0.7245 0.6418 0.7527
e5-large 0.7976 0.7544 0.6620 0.7077 0.7643 0.7756 0.6741 0.6208 0.7629
M3 dense 0.5941 0.4746 0.3609 0.4563 0.4518 0.5221 0.6291 0.3917 0.5295
SPLADE mini 0.6338 0.5656 0.6148 0.6319 0.6237 0.6027 0.7358 0.5185 0.6130
M3 sparse 0.6568 0.5950 0.6337 0.6157 0.6376 0.6108 0.7417 0.5047 0.6302
multivector mini 0.7709 0.6631 0.6930 0.7291 0.7118 0.6884 0.8571 0.6407 0.7305
multivector e5-small 0.7977 0.6803 0.6723 0.7352 0.7160 0.7366 0.8652 0.6452 0.7520
multivector e5-base 0.8145 0.7413 0.7000 0.7524 0.7459 0.7612 0.8296 0.6731 0.7760
M3 multi 0.7296 0.6476 0.6660 0.7481 0.6850 0.6880 0.8716 0.5789 0.6982
reranker mini 0.7824 0.7139 0.6618 0.7122 0.7296 0.7152 0.8793 0.6361 0.7431
reranker e5-small 0.7984 0.6900 0.7023 0.7539 0.7379 0.7369 0.8850 0.6541 0.7580
reranker e5-base 0.8156 0.7178 0.7240 0.7572 0.7527 0.7498 0.8509 0.6553 0.7741

Training Details

Training Dataset

AfriXIR

  • Dataset: AfriXIR
  • Size: 112,864 training samples
  • Columns: anchor, positive, negative_0, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, and negative_7

Evaluation Dataset

AfriXIR

  • Dataset: AfriXIR
  • Size: 12,546 evaluation samples
  • Columns: anchor, positive, negative_0, negative_1, negative_2, negative_3, negative_4, negative_5, negative_6, and negative_7

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultiVectorMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}
Downloads last month
3
Safetensors
Model size
0.3B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for nebhailema/afrixir-multivector-e5-base

Finetuned
(198)
this model

Dataset used to train nebhailema/afrixir-multivector-e5-base

Collections including nebhailema/afrixir-multivector-e5-base

Papers for nebhailema/afrixir-multivector-e5-base