README.md

---
license: apache-2.0
language:
  - ig
  - ha
  - yo
  - en
library_name: pytorch
tags:
  - image-text-retrieval
  - multimodal
  - contrastive
  - nigerian-languages
  - igbo
  - hausa
  - yoruba
  - siglip
  - native-bird
datasets:
  - AfriMM/AFRICaption
base_model:
  - Modularcomputing/Native-Bird
  - google/siglip-base-patch16-224
metrics:
  - recall
---

# Native-Bird2: image–text retrieval for Igbo, Hausa, Yoruba and English

Native-Bird2 puts **Igbo, Hausa, Yoruba and English captions and photos into one shared embedding space**, so a sentence in
any of the four languages can find matching images, and an image can find matching captions.

It is the multimodal extension of [Native-Bird](https://huggingface.co/Modularcomputing/Native-Bird), our cross-lingual
sentence embedding model for Nigerian languages.

| | |
|---|---|
| Text encoder | [`Modularcomputing/Native-Bird`](https://huggingface.co/Modularcomputing/Native-Bird) (768-d, LaBSE fine-tune), **frozen** |
| Image encoder | [`google/siglip-base-patch16-224`](https://huggingface.co/google/siglip-base-patch16-224) (SigLIP ViT-B/16), **frozen** |
| Trained part | two linear projection heads (image 768→512, text 768→512), L2-normalised, plus a learnable temperature |
| Objective | symmetric InfoNCE (CLIP-style), batch 256; captions of the *same* image in other languages are masked out of the negatives |
| This repo ships | the projection heads only (~3 MB). Both encoders are pulled from their own Hub repos. |

**Headline (809 held-out test images):** Igbo text→image Recall@5 **69.6** vs **12.0** for multilingual CLIP zero-shot;
Hausa 59.4 vs 4.2; Yoruba 52.8 vs 2.8.

![Igbo image search: top-3 images for Igbo queries vs the same query in English](images/igbo_search_grid.png)

*Igbo image search on held-out images. Left: top-3 for the Igbo query. Right: top-3 for the same query written in English.*

## Files

| file | what it is |
|---|---|
| `model.safetensors` | projection-head weights: `img.net.weight` [512, 768], `img.net.bias`, `txt.net.weight` [512, 768], `txt.net.bias`, `logit_scale` (log space) |
| `best_native_bird.pt` | the same heads as the original PyTorch checkpoint (`model`, `cfg`, `d_img`, `d_txt`), loaded by `search.py` |
| `config.json` | head shapes, encoder ids, temperature |
| `configs/default.yaml` | full training/eval config |
| `search.py`, `nbmm/common.py` | text→image search and the shared model code |
| `requirements.txt` | Python dependencies |
| `metrics.json`, `results/` | every benchmark number below, training logs, Igbo demo output, text-encoder bitext check |
| `data/processed/test.parquet` | the 809-image test split (image ids + filtered captions), used as the default search gallery |
| `data/processed/split_summary.json`, `data/processed/filter_stats.json` | split counts and caption-filter statistics |
| `images/` | Igbo search grid and loss curves |

## Training data

**[AfriMM/AFRICaption](https://huggingface.co/datasets/AfriMM/AFRICaption)** (Apache-2.0): 8,091 Flickr8k images, one English
caption per image, machine-translated into African languages. We use the `eng`, `ibo`, `hau` and `yor` columns.
Images come from the public Flickr8k mirror `mustafa2ak/Flickr8k-Images` (8,091 / 8,091 image ids matched and readable).

**Caption filtering** (Igbo, Hausa, Yoruba; English kept as-is). Rules applied in order, first failure = drop:

1. R1: empty or fewer than 2 tokens
2. R2: repetition loop (same token ≥3× in a row, or one token >50% of a ≥4-token caption; 2× reduplication such as Igbo *anụnụ anụnụ* is kept)
3. R3: untranslated English (≥50% of tokens of length ≥3 found in the English source, or an exact copy)
4. R4: character-length ratio vs English outside [0.45, 2.2]
5. R5: LaBSE cosine(caption, English) below max(0.50, 5th percentile of survivors)

| lang | kept | dropped | R2 loop | R3 copy | R4 length | R5 semantic | R5 threshold |
|---|---|---|---|---|---|---|---|
| ibo | 7,436 | 655 | 6 | 250 | 7 | 392 | 0.742 |
| hau | 7,655 | 436 | 12 | 11 | 10 | 403 | 0.603 |
| yor | 7,611 | 480 | 24 | 16 | 39 | 401 | 0.510 |

**Split:** image-level 80/10/10 with seed 42; no image appears in more than one split (checked).

| split | images | eng | ibo | hau | yor | pairs |
|---|---|---|---|---|---|---|
| train | 6,473 | 6,473 | 5,943 | 6,116 | 6,083 | 24,615 |
| val | 809 | 809 | 742 | 770 | 764 | 3,085 |
| test | 809 | 809 | 751 | 769 | 764 | 3,093 |

`test_unfiltered` re-scores the same 809 test images with the raw, unfiltered captions.

## Training

- 1× H100, about 27 minutes for the whole experiment (feature caching, both arms, baselines, evaluation).
- Image features and frozen text features are computed once and cached, so the heads train in well under a second per epoch.
- 20 epochs, batch 256, AdamW (lr 1e-3, weight decay 0.01, 100 warm-up steps, cosine schedule), linear 512-d heads,
  learnable temperature initialised at ln(1/0.07) and clamped at ln(100). Each image contributes one pair per kept language.
- **Checkpoint selection by validation mean Recall** (4 languages × both directions × R@1/5/10):
  - frozen Native-Bird text arm: best at **epoch 3** (val 60.46), **the released checkpoint**
  - unfreezing the top-2 Native-Bird layers (text lr 1e-5): best at epoch 2 (val 60.35), a near tie, so no gain for about 50× the training time per epoch (9.7 s vs 0.2 s)
- Both arms **overfit after epoch 2–3**: training loss keeps falling while validation recall drops (frozen arm: 60.46 at epoch 3 → 54.74 at epoch 20).

![Training loss and validation mean recall per epoch](images/loss_curves.png)

## Benchmark

Test = 809 held-out images. The gallery is every test caption in that language (eng 809, ibo 751, hau 769, yor 764), so
chance R@1 is about 0.13%. i2t = image→text, t2i = text→image. Mean R = average of the six recall numbers.

Baselines: **untrained heads** (random-init, seeded, same encoders) and **mCLIP zero-shot**
(`sentence-transformers/clip-ViT-B-32` image tower + `sentence-transformers/clip-ViT-B-32-multilingual-v1` text tower).

### Test split (filtered captions)

| model | lang | i2t R@1 | i2t R@5 | i2t R@10 | t2i R@1 | t2i R@5 | t2i R@10 | mean R |
|---|---|---|---|---|---|---|---|---|
| untrained heads | eng | 0.37 | 1.11 | 1.48 | 0.12 | 0.87 | 1.73 | 0.95 |
| untrained heads | ibo | 0.0 | 0.8 | 2.4 | 0.13 | 0.8 | 1.6 | 0.96 |
| untrained heads | hau | 0.26 | 0.78 | 1.69 | 0.13 | 1.04 | 1.69 | 0.93 |
| untrained heads | yor | 0.26 | 0.92 | 2.36 | 0.26 | 0.92 | 1.05 | 0.96 |
| mCLIP zero-shot | eng | 58.34 | 84.67 | 92.21 | 57.85 | 83.31 | 89.99 | 77.73 |
| mCLIP zero-shot | ibo | 6.13 | 14.38 | 20.91 | 4.26 | 11.98 | 16.11 | 12.29 |
| mCLIP zero-shot | hau | 3.38 | 6.63 | 8.71 | 1.82 | 4.16 | 6.11 | 5.13 |
| mCLIP zero-shot | yor | 2.36 | 4.84 | 6.41 | 1.31 | 2.75 | 4.84 | 3.75 |
| **Native-Bird2 (frozen text, released)** | eng | 46.48 | 77.38 | 88.5 | 44.99 | 76.64 | 87.02 | 70.17 |
| **Native-Bird2 (frozen text, released)** | ibo | **39.28** | **70.71** | **82.29** | **39.01** | **69.64** | **81.49** | **63.74** |
| **Native-Bird2 (frozen text, released)** | hau | **33.16** | **61.64** | **74.64** | **32.25** | **59.43** | **74.9** | **56.0** |
| **Native-Bird2 (frozen text, released)** | yor | **25.52** | **53.8** | **64.66** | **23.56** | **52.75** | **65.84** | **47.69** |
| Native-Bird unfreeze top-2 | eng | 44.99 | 78.74 | 89.0 | 46.23 | 76.14 | 87.89 | 70.5 |
| Native-Bird unfreeze top-2 | ibo | 37.95 | 70.57 | 83.49 | 39.68 | 70.17 | 83.36 | 64.2 |
| Native-Bird unfreeze top-2 | hau | 31.73 | 60.47 | 74.9 | 30.3 | 60.73 | 74.38 | 55.42 |
| Native-Bird unfreeze top-2 | yor | 23.3 | 52.88 | 64.66 | 24.87 | 53.27 | 64.4 | 47.23 |

### test_unfiltered (same 809 images, raw captions, 809 per language)

| model | lang | i2t R@1 | i2t R@5 | i2t R@10 | t2i R@1 | t2i R@5 | t2i R@10 | mean R |
|---|---|---|---|---|---|---|---|---|
| frozen text (released) | eng | 46.48 | 77.38 | 88.5 | 44.99 | 76.64 | 87.02 | 70.17 |
| frozen text (released) | ibo | 36.59 | 67.37 | 79.6 | 35.6 | 66.75 | 79.23 | 60.86 |
| frozen text (released) | hau | 31.52 | 59.21 | 72.06 | 30.9 | 57.73 | 72.81 | 54.04 |
| frozen text (released) | yor | 24.6 | 52.78 | 63.54 | 22.5 | 51.79 | 64.4 | 46.6 |
| unfreeze top-2 | eng | 44.99 | 78.74 | 89.0 | 46.23 | 76.14 | 87.89 | 70.5 |
| unfreeze top-2 | ibo | 35.72 | 67.86 | 81.09 | 36.96 | 67.12 | 80.22 | 61.5 |
| unfreeze top-2 | hau | 29.91 | 58.71 | 73.05 | 28.31 | 58.22 | 72.44 | 53.44 |
| unfreeze top-2 | yor | 22.74 | 50.31 | 62.42 | 23.98 | 52.41 | 62.79 | 45.77 |

Takeaways: on Igbo, Hausa and Yoruba the model beats multilingual CLIP by a wide margin (Igbo mean R 63.74 vs 12.29);
on English, mCLIP (trained on hundreds of millions of image–text pairs) is still ahead (77.73 vs 70.17). Scores on the raw
captions drop only 1–3 points, so the filter mainly cleans evaluation rather than propping up the model.

## Igbo image search demo

Released checkpoint, gallery = the 809 test images. **All 751 held-out Igbo test captions** used as queries, scoring where
the caption's own image lands:

| query language | queries | top-1 | top-5 | top-10 | median rank |
|---|---|---|---|---|---|
| Igbo | 751 | **37.4%** | **68.4%** | **80.6%** | **2** |
| English (same images) | 809 | 45.0% | 76.6% | 87.0% | n/a |

Natural Igbo queries (top-5 overlap with the results for the same query in English):

| Igbo query | English | top-5 overlap with English query |
|---|---|---|
| Nkịta na-agba ọsọ n'elu ahịhịa | A dog running on the grass | 3/5 |
| Ụmụaka na-egwu egwu n'ime mmiri | Children playing in the water | 5/5 |
| Otu nwoke na-agba ịnyịnya ígwè | A man riding a bicycle | 4/5 |
| Otu nwanyị yi uwe na-acha uhie uhie | A woman wearing a red dress | 4/5 |
| Ndị mmadụ nọ n'elu ugwu | People on top of a mountain | 4/5 |
| Otu nwata nwoke na-awụli elu | A boy jumping in the air | 3/5 |
| Nkịta abụọ na-egwu egwu | Two dogs playing together | 2/5 |
| Ọtụtụ mmadụ nọ n'okporo ụzọ | A crowd of people on a street | 4/5 |

Full output, including per-query image ids and scores: `results/igbo_search.json`.

## Usage

### Standalone (no repo code)

```python
import torch, torch.nn.functional as F
from PIL import Image
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from sentence_transformers import SentenceTransformer
from transformers import AutoModel, AutoImageProcessor

w = load_file(hf_hub_download("Modularcomputing/Native-Bird2", "model.safetensors"))
text_enc = SentenceTransformer("Modularcomputing/Native-Bird")
siglip = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()
proc = AutoImageProcessor.from_pretrained("google/siglip-base-patch16-224")

@torch.no_grad()
def embed_text(texts):
    e = text_enc.encode(texts, convert_to_tensor=True, normalize_embeddings=True).float().cpu()
    return F.normalize(e @ w["txt.net.weight"].T + w["txt.net.bias"], dim=-1)

@torch.no_grad()
def embed_images(paths):
    px = proc(images=[Image.open(p).convert("RGB") for p in paths], return_tensors="pt")["pixel_values"]
    f = siglip.get_image_features(pixel_values=px)
    if not torch.is_tensor(f):  # some transformers versions return a ModelOutput
        f = f.pooler_output
    return F.normalize(f.float() @ w["img.net.weight"].T + w["img.net.bias"], dim=-1)

images = ["dog.jpg", "beach.jpg", "market.jpg"]
scores = embed_text(["Nkịta na-agba ọsọ n'elu ahịhịa"]) @ embed_images(images).T   # cosine similarity
print(images[scores.argmax().item()])
```

The same space works in every direction: Hausa or Yoruba queries, English queries, or image→caption retrieval.

### With `search.py` (gallery = the 809 test images)

```bash
git clone https://huggingface.co/Modularcomputing/Native-Bird2 && cd Native-Bird2
pip install torch torchvision && pip install -r requirements.txt
# search.py reads the Flickr8k images from data/raw/flickr8k/Flicker8k_Dataset/
python -c "from huggingface_hub import snapshot_download; snapshot_download('mustafa2ak/Flickr8k-Images', repo_type='dataset', allow_patterns=['Flicker8k_Dataset/*.jpg'], local_dir='data/raw/flickr8k')"
python search.py --ckpt best_native_bird.pt "Nkịta abụọ na-egwu egwu" "A dog running on the grass" -k 5
```

Image features are computed once and cached under `data/features/`. Runs on CPU or GPU.

## Limitations

- **Machine-translated captions.** Igbo, Hausa and Yoruba captions in AfriCaption are machine translations of English
  Flickr8k captions, not native-speaker descriptions. The filter removes obvious failures (loops, untranslated English,
  wrong length, low semantic match), but errors, literal phrasing and code-mixing remain, and the model learned from them.
- **Narrow image domain.** Flickr8k is about 8k mostly North-American/European everyday photos (people, dogs, sports,
  outdoors). Expect weaker results on other scenes, on African contexts, on documents, products or fine-grained categories.
- **Small training set** (6,473 images). Both arms overfit after epoch 2–3; the released checkpoint is the early-stopped one.
- **Yoruba is the weakest language** (test mean R 47.69 vs 63.74 Igbo). Native-Bird's own Yoruba↔English bitext retrieval is
  also lowest (acc@1 79.23 vs 96.66 Igbo, `results/text_encoder_crosslingual_test.json`), and LaBSE scores Yoruba lower, so
  the filter's bottom-5% Yoruba tail drops some acceptable translations.
- **The filter is LaBSE-based.** LaBSE is not the model under test, but Native-Bird is a LaBSE fine-tune, so the filter is not
  fully independent of the text encoder.
- **English is below mCLIP zero-shot** (mean R 70.17 vs 77.73); use a large English CLIP model if you only need English.
- Retrieval over a gallery of ~800 candidates; numbers on larger galleries will be lower.

## Citation

Please cite AfriCaption, the dataset this model is trained on ([ACL Anthology](https://aclanthology.org/2026.africanlp-main.5/), [arXiv:2510.17405](https://arxiv.org/abs/2510.17405)):

```bibtex
@inproceedings{oduwole-etal-2026-africaption,
    title = "{A}fri{C}aption: Establishing a New Paradigm for Image Captioning in {A}frican Languages",
    author = "Oduwole, Mardiyyah and Mireku, Prince and Adebanjo, Fatimo and Olajide, Oluwatosin and Aliyu, Mahi Aminu and Novikova, Jekaterina",
    booktitle = "Proceedings of the 7th Workshop on {A}frican Natural Language Processing ({A}frica{NLP} 2026)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.africanlp-main.5/",
    doi = "10.18653/v1/2026.africanlp-main.5",
    pages = "44--55"
}
```

Also see [SigLIP](https://huggingface.co/google/siglip-base-patch16-224) and [Native-Bird](https://huggingface.co/Modularcomputing/Native-Bird).

Built by [Modularcomputing](https://huggingface.co/Modularcomputing).