Native-Bird2: image–text retrieval for Igbo, Hausa, Yoruba and English

Native-Bird2 puts Igbo, Hausa, Yoruba and English captions and photos into one shared embedding space, so a sentence in any of the four languages can find matching images, and an image can find matching captions.

It is the multimodal extension of Native-Bird, our cross-lingual sentence embedding model for Nigerian languages.

Text encoder	Modularcomputing/Native-Bird (768-d, LaBSE fine-tune), frozen
Image encoder	google/siglip-base-patch16-224 (SigLIP ViT-B/16), frozen
Trained part	two linear projection heads (image 768→512, text 768→512), L2-normalised, plus a learnable temperature
Objective	symmetric InfoNCE (CLIP-style), batch 256; captions of the same image in other languages are masked out of the negatives
This repo ships	the projection heads only (~3 MB). Both encoders are pulled from their own Hub repos.

Headline (809 held-out test images): Igbo text→image Recall@5 69.6 vs 12.0 for multilingual CLIP zero-shot; Hausa 59.4 vs 4.2; Yoruba 52.8 vs 2.8.

Igbo image search on held-out images. Left: top-3 for the Igbo query. Right: top-3 for the same query written in English.

Files

file	what it is
model.safetensors	projection-head weights: img.net.weight [512, 768], img.net.bias, txt.net.weight [512, 768], txt.net.bias, logit_scale (log space)
best_native_bird.pt	the same heads as the original PyTorch checkpoint (model, cfg, d_img, d_txt), loaded by search.py
config.json	head shapes, encoder ids, temperature
configs/default.yaml	full training/eval config
search.py, nbmm/common.py	text→image search and the shared model code
requirements.txt	Python dependencies
metrics.json, results/	every benchmark number below, training logs, Igbo demo output, text-encoder bitext check
data/processed/test.parquet	the 809-image test split (image ids + filtered captions), used as the default search gallery
data/processed/split_summary.json, data/processed/filter_stats.json	split counts and caption-filter statistics
images/	Igbo search grid and loss curves

Training data

AfriMM/AFRICaption (Apache-2.0): 8,091 Flickr8k images, one English caption per image, machine-translated into African languages. We use the eng, ibo, hau and yor columns. Images come from the public Flickr8k mirror mustafa2ak/Flickr8k-Images (8,091 / 8,091 image ids matched and readable).

Caption filtering (Igbo, Hausa, Yoruba; English kept as-is). Rules applied in order, first failure = drop:

1. R1: empty or fewer than 2 tokens
2. R2: repetition loop (same token ≥3× in a row, or one token >50% of a ≥4-token caption; 2× reduplication such as Igbo anụnụ anụnụ is kept)
3. R3: untranslated English (≥50% of tokens of length ≥3 found in the English source, or an exact copy)
4. R4: character-length ratio vs English outside [0.45, 2.2]
5. R5: LaBSE cosine(caption, English) below max(0.50, 5th percentile of survivors)

lang	kept	dropped	R2 loop	R3 copy	R4 length	R5 semantic	R5 threshold
ibo	7,436	655	6	250	7	392	0.742
hau	7,655	436	12	11	10	403	0.603
yor	7,611	480	24	16	39	401	0.510

Split: image-level 80/10/10 with seed 42; no image appears in more than one split (checked).

split	images	eng	ibo	hau	yor	pairs
train	6,473	6,473	5,943	6,116	6,083	24,615
val	809	809	742	770	764	3,085
test	809	809	751	769	764	3,093

test_unfiltered re-scores the same 809 test images with the raw, unfiltered captions.

Training

• 1× H100, about 27 minutes for the whole experiment (feature caching, both arms, baselines, evaluation).
• Image features and frozen text features are computed once and cached, so the heads train in well under a second per epoch.
• 20 epochs, batch 256, AdamW (lr 1e-3, weight decay 0.01, 100 warm-up steps, cosine schedule), linear 512-d heads, learnable temperature initialised at ln(1/0.07) and clamped at ln(100). Each image contributes one pair per kept language.
• Checkpoint selection by validation mean Recall (4 languages × both directions × R@1/5/10):
  • frozen Native-Bird text arm: best at epoch 3 (val 60.46), the released checkpoint
  • unfreezing the top-2 Native-Bird layers (text lr 1e-5): best at epoch 2 (val 60.35), a near tie, so no gain for about 50× the training time per epoch (9.7 s vs 0.2 s)
• Both arms overfit after epoch 2–3: training loss keeps falling while validation recall drops (frozen arm: 60.46 at epoch 3 → 54.74 at epoch 20).

Benchmark

Test = 809 held-out images. The gallery is every test caption in that language (eng 809, ibo 751, hau 769, yor 764), so chance R@1 is about 0.13%. i2t = image→text, t2i = text→image. Mean R = average of the six recall numbers.

Baselines: untrained heads (random-init, seeded, same encoders) and mCLIP zero-shot (sentence-transformers/clip-ViT-B-32 image tower + sentence-transformers/clip-ViT-B-32-multilingual-v1 text tower).

Test split (filtered captions)

model	lang	i2t R@1	i2t R@5	i2t R@10	t2i R@1	t2i R@5	t2i R@10	mean R
untrained heads	eng	0.37	1.11	1.48	0.12	0.87	1.73	0.95
untrained heads	ibo	0.0	0.8	2.4	0.13	0.8	1.6	0.96
untrained heads	hau	0.26	0.78	1.69	0.13	1.04	1.69	0.93
untrained heads	yor	0.26	0.92	2.36	0.26	0.92	1.05	0.96
mCLIP zero-shot	eng	58.34	84.67	92.21	57.85	83.31	89.99	77.73
mCLIP zero-shot	ibo	6.13	14.38	20.91	4.26	11.98	16.11	12.29
mCLIP zero-shot	hau	3.38	6.63	8.71	1.82	4.16	6.11	5.13
mCLIP zero-shot	yor	2.36	4.84	6.41	1.31	2.75	4.84	3.75
Native-Bird2 (frozen text, released)	eng	46.48	77.38	88.5	44.99	76.64	87.02	70.17
Native-Bird2 (frozen text, released)	ibo	39.28	70.71	82.29	39.01	69.64	81.49	63.74
Native-Bird2 (frozen text, released)	hau	33.16	61.64	74.64	32.25	59.43	74.9	56.0
Native-Bird2 (frozen text, released)	yor	25.52	53.8	64.66	23.56	52.75	65.84	47.69
Native-Bird unfreeze top-2	eng	44.99	78.74	89.0	46.23	76.14	87.89	70.5
Native-Bird unfreeze top-2	ibo	37.95	70.57	83.49	39.68	70.17	83.36	64.2
Native-Bird unfreeze top-2	hau	31.73	60.47	74.9	30.3	60.73	74.38	55.42
Native-Bird unfreeze top-2	yor	23.3	52.88	64.66	24.87	53.27	64.4	47.23

test_unfiltered (same 809 images, raw captions, 809 per language)

model	lang	i2t R@1	i2t R@5	i2t R@10	t2i R@1	t2i R@5	t2i R@10	mean R
frozen text (released)	eng	46.48	77.38	88.5	44.99	76.64	87.02	70.17
frozen text (released)	ibo	36.59	67.37	79.6	35.6	66.75	79.23	60.86
frozen text (released)	hau	31.52	59.21	72.06	30.9	57.73	72.81	54.04
frozen text (released)	yor	24.6	52.78	63.54	22.5	51.79	64.4	46.6
unfreeze top-2	eng	44.99	78.74	89.0	46.23	76.14	87.89	70.5
unfreeze top-2	ibo	35.72	67.86	81.09	36.96	67.12	80.22	61.5
unfreeze top-2	hau	29.91	58.71	73.05	28.31	58.22	72.44	53.44
unfreeze top-2	yor	22.74	50.31	62.42	23.98	52.41	62.79	45.77

Takeaways: on Igbo, Hausa and Yoruba the model beats multilingual CLIP by a wide margin (Igbo mean R 63.74 vs 12.29); on English, mCLIP (trained on hundreds of millions of image–text pairs) is still ahead (77.73 vs 70.17). Scores on the raw captions drop only 1–3 points, so the filter mainly cleans evaluation rather than propping up the model.

Igbo image search demo

Released checkpoint, gallery = the 809 test images. All 751 held-out Igbo test captions used as queries, scoring where the caption's own image lands:

query language	queries	top-1	top-5	top-10	median rank
Igbo	751	37.4%	68.4%	80.6%	2
English (same images)	809	45.0%	76.6%	87.0%	n/a

Natural Igbo queries (top-5 overlap with the results for the same query in English):

Igbo query	English	top-5 overlap with English query
Nkịta na-agba ọsọ n'elu ahịhịa	A dog running on the grass	3/5
Ụmụaka na-egwu egwu n'ime mmiri	Children playing in the water	5/5
Otu nwoke na-agba ịnyịnya ígwè	A man riding a bicycle	4/5
Otu nwanyị yi uwe na-acha uhie uhie	A woman wearing a red dress	4/5
Ndị mmadụ nọ n'elu ugwu	People on top of a mountain	4/5
Otu nwata nwoke na-awụli elu	A boy jumping in the air	3/5
Nkịta abụọ na-egwu egwu	Two dogs playing together	2/5
Ọtụtụ mmadụ nọ n'okporo ụzọ	A crowd of people on a street	4/5

Full output, including per-query image ids and scores: results/igbo_search.json.

Usage

Standalone (no repo code)

import torch, torch.nn.functional as F
from PIL import Image
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from sentence_transformers import SentenceTransformer
from transformers import AutoModel, AutoImageProcessor

w = load_file(hf_hub_download("Modularcomputing/Native-Bird2", "model.safetensors"))
text_enc = SentenceTransformer("Modularcomputing/Native-Bird")
siglip = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()
proc = AutoImageProcessor.from_pretrained("google/siglip-base-patch16-224")

@torch.no_grad()
def embed_text(texts):
    e = text_enc.encode(texts, convert_to_tensor=True, normalize_embeddings=True).float().cpu()
    return F.normalize(e @ w["txt.net.weight"].T + w["txt.net.bias"], dim=-1)

@torch.no_grad()
def embed_images(paths):
    px = proc(images=[Image.open(p).convert("RGB") for p in paths], return_tensors="pt")["pixel_values"]
    f = siglip.get_image_features(pixel_values=px)
    if not torch.is_tensor(f):  # some transformers versions return a ModelOutput
        f = f.pooler_output
    return F.normalize(f.float() @ w["img.net.weight"].T + w["img.net.bias"], dim=-1)

images = ["dog.jpg", "beach.jpg", "market.jpg"]
scores = embed_text(["Nkịta na-agba ọsọ n'elu ahịhịa"]) @ embed_images(images).T   # cosine similarity
print(images[scores.argmax().item()])

The same space works in every direction: Hausa or Yoruba queries, English queries, or image→caption retrieval.

With search.py (gallery = the 809 test images)

git clone https://huggingface.co/Modularcomputing/Native-Bird2 && cd Native-Bird2
pip install torch torchvision && pip install -r requirements.txt
# search.py reads the Flickr8k images from data/raw/flickr8k/Flicker8k_Dataset/
python -c "from huggingface_hub import snapshot_download; snapshot_download('mustafa2ak/Flickr8k-Images', repo_type='dataset', allow_patterns=['Flicker8k_Dataset/*.jpg'], local_dir='data/raw/flickr8k')"
python search.py --ckpt best_native_bird.pt "Nkịta abụọ na-egwu egwu" "A dog running on the grass" -k 5

Image features are computed once and cached under data/features/. Runs on CPU or GPU.

Limitations

• Machine-translated captions. Igbo, Hausa and Yoruba captions in AfriCaption are machine translations of English Flickr8k captions, not native-speaker descriptions. The filter removes obvious failures (loops, untranslated English, wrong length, low semantic match), but errors, literal phrasing and code-mixing remain, and the model learned from them.
• Narrow image domain. Flickr8k is about 8k mostly North-American/European everyday photos (people, dogs, sports, outdoors). Expect weaker results on other scenes, on African contexts, on documents, products or fine-grained categories.
• Small training set (6,473 images). Both arms overfit after epoch 2–3; the released checkpoint is the early-stopped one.
• Yoruba is the weakest language (test mean R 47.69 vs 63.74 Igbo). Native-Bird's own Yoruba↔English bitext retrieval is also lowest (acc@1 79.23 vs 96.66 Igbo, results/text_encoder_crosslingual_test.json), and LaBSE scores Yoruba lower, so the filter's bottom-5% Yoruba tail drops some acceptable translations.
• The filter is LaBSE-based. LaBSE is not the model under test, but Native-Bird is a LaBSE fine-tune, so the filter is not fully independent of the text encoder.
• English is below mCLIP zero-shot (mean R 70.17 vs 77.73); use a large English CLIP model if you only need English.
• Retrieval over a gallery of ~800 candidates; numbers on larger galleries will be lower.

Citation

Please cite AfriCaption, the dataset this model is trained on (ACL Anthology, arXiv:2510.17405):

@inproceedings{oduwole-etal-2026-africaption,
    title = "{A}fri{C}aption: Establishing a New Paradigm for Image Captioning in {A}frican Languages",
    author = "Oduwole, Mardiyyah and Mireku, Prince and Adebanjo, Fatimo and Olajide, Oluwatosin and Aliyu, Mahi Aminu and Novikova, Jekaterina",
    booktitle = "Proceedings of the 7th Workshop on {A}frican Natural Language Processing ({A}frica{NLP} 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.africanlp-main.5/",
    doi = "10.18653/v1/2026.africanlp-main.5",
    pages = "44--55"
}

Also see SigLIP and Native-Bird.

Built by Modularcomputing.