Upload via Autoresearch export_data

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+images/igbo_search_grid.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,266 @@
 ---
 license: apache-2.0
+language:
+  - ig
+  - ha
+  - yo
+  - en
+library_name: pytorch
+tags:
+  - image-text-retrieval
+  - multimodal
+  - contrastive
+  - nigerian-languages
+  - igbo
+  - hausa
+  - yoruba
+  - siglip
+  - native-bird
+datasets:
+  - AfriMM/AFRICaption
+base_model:
+  - Modularcomputing/Native-Bird
+  - google/siglip-base-patch16-224
+metrics:
+  - recall
 ---
+
+# Native-Bird2: image–text retrieval for Igbo, Hausa, Yoruba and English
+
+Native-Bird2 puts **Igbo, Hausa, Yoruba and English captions and photos into one shared embedding space**, so a sentence in
+any of the four languages can find matching images, and an image can find matching captions.
+
+It is the multimodal extension of [Native-Bird](https://huggingface.co/Modularcomputing/Native-Bird), our cross-lingual
+sentence embedding model for Nigerian languages.
+
+| | |
+|---|---|
+| Text encoder | [`Modularcomputing/Native-Bird`](https://huggingface.co/Modularcomputing/Native-Bird) (768-d, LaBSE fine-tune), **frozen** |
+| Image encoder | [`google/siglip-base-patch16-224`](https://huggingface.co/google/siglip-base-patch16-224) (SigLIP ViT-B/16), **frozen** |
+| Trained part | two linear projection heads (image 768→512, text 768→512), L2-normalised, plus a learnable temperature |
+| Objective | symmetric InfoNCE (CLIP-style), batch 256; captions of the *same* image in other languages are masked out of the negatives |
+| This repo ships | the projection heads only (~3 MB). Both encoders are pulled from their own Hub repos. |
+
+**Headline (809 held-out test images):** Igbo text→image Recall@5 **69.6** vs **12.0** for multilingual CLIP zero-shot;
+Hausa 59.4 vs 4.2; Yoruba 52.8 vs 2.8.
+
+![Igbo image search: top-3 images for Igbo queries vs the same query in English](images/igbo_search_grid.png)
+
+*Igbo image search on held-out images. Left: top-3 for the Igbo query. Right: top-3 for the same query written in English.*
+
+## Files
+
+| file | what it is |
+|---|---|
+| `model.safetensors` | projection-head weights: `img.net.weight` [512, 768], `img.net.bias`, `txt.net.weight` [512, 768], `txt.net.bias`, `logit_scale` (log space) |
+| `best_native_bird.pt` | the same heads as the original PyTorch checkpoint (`model`, `cfg`, `d_img`, `d_txt`), loaded by `search.py` |
+| `config.json` | head shapes, encoder ids, temperature |
+| `configs/default.yaml` | full training/eval config |
+| `search.py`, `nbmm/common.py` | text→image search and the shared model code |
+| `requirements.txt` | Python dependencies |
+| `metrics.json`, `results/` | every benchmark number below, training logs, Igbo demo output, text-encoder bitext check |
+| `data/processed/test.parquet` | the 809-image test split (image ids + filtered captions), used as the default search gallery |
+| `data/processed/split_summary.json`, `data/processed/filter_stats.json` | split counts and caption-filter statistics |
+| `images/` | Igbo search grid and loss curves |
+
+## Training data
+
+**[AfriMM/AFRICaption](https://huggingface.co/datasets/AfriMM/AFRICaption)** (Apache-2.0): 8,091 Flickr8k images, one English
+caption per image, machine-translated into African languages. We use the `eng`, `ibo`, `hau` and `yor` columns.
+Images come from the public Flickr8k mirror `mustafa2ak/Flickr8k-Images` (8,091 / 8,091 image ids matched and readable).
+
+**Caption filtering** (Igbo, Hausa, Yoruba; English kept as-is). Rules applied in order, first failure = drop:
+
+1. R1: empty or fewer than 2 tokens
+2. R2: repetition loop (same token ≥3× in a row, or one token >50% of a ≥4-token caption; 2× reduplication such as Igbo *anụnụ anụnụ* is kept)
+3. R3: untranslated English (≥50% of tokens of length ≥3 found in the English source, or an exact copy)
+4. R4: character-length ratio vs English outside [0.45, 2.2]
+5. R5: LaBSE cosine(caption, English) below max(0.50, 5th percentile of survivors)
+
+| lang | kept | dropped | R2 loop | R3 copy | R4 length | R5 semantic | R5 threshold |
+|---|---|---|---|---|---|---|---|
+| ibo | 7,436 | 655 | 6 | 250 | 7 | 392 | 0.742 |
+| hau | 7,655 | 436 | 12 | 11 | 10 | 403 | 0.603 |
+| yor | 7,611 | 480 | 24 | 16 | 39 | 401 | 0.510 |
+
+**Split:** image-level 80/10/10 with seed 42; no image appears in more than one split (checked).
+
+| split | images | eng | ibo | hau | yor | pairs |
+|---|---|---|---|---|---|---|
+| train | 6,473 | 6,473 | 5,943 | 6,116 | 6,083 | 24,615 |
+| val | 809 | 809 | 742 | 770 | 764 | 3,085 |
+| test | 809 | 809 | 751 | 769 | 764 | 3,093 |
+
+`test_unfiltered` re-scores the same 809 test images with the raw, unfiltered captions.
+
+## Training
+
+- 1× H100, about 27 minutes for the whole experiment (feature caching, both arms, baselines, evaluation).
+- Image features and frozen text features are computed once and cached, so the heads train in well under a second per epoch.
+- 20 epochs, batch 256, AdamW (lr 1e-3, weight decay 0.01, 100 warm-up steps, cosine schedule), linear 512-d heads,
+  learnable temperature initialised at ln(1/0.07) and clamped at ln(100). Each image contributes one pair per kept language.
+- **Checkpoint selection by validation mean Recall** (4 languages × both directions × R@1/5/10):
+  - frozen Native-Bird text arm: best at **epoch 3** (val 60.46), **the released checkpoint**
+  - unfreezing the top-2 Native-Bird layers (text lr 1e-5): best at epoch 2 (val 60.35), a near tie, so no gain for about 50× the training time per epoch (9.7 s vs 0.2 s)
+- Both arms **overfit after epoch 2–3**: training loss keeps falling while validation recall drops (frozen arm: 60.46 at epoch 3 → 54.74 at epoch 20).
+
+![Training loss and validation mean recall per epoch](images/loss_curves.png)
+
+## Benchmark
+
+Test = 809 held-out images. The gallery is every test caption in that language (eng 809, ibo 751, hau 769, yor 764), so
+chance R@1 is about 0.13%. i2t = image→text, t2i = text→image. Mean R = average of the six recall numbers.
+
+Baselines: **untrained heads** (random-init, seeded, same encoders) and **mCLIP zero-shot**
+(`sentence-transformers/clip-ViT-B-32` image tower + `sentence-transformers/clip-ViT-B-32-multilingual-v1` text tower).
+
+### Test split (filtered captions)
+
+| model | lang | i2t R@1 | i2t R@5 | i2t R@10 | t2i R@1 | t2i R@5 | t2i R@10 | mean R |
+|---|---|---|---|---|---|---|---|---|
+| untrained heads | eng | 0.37 | 1.11 | 1.48 | 0.12 | 0.87 | 1.73 | 0.95 |
+| untrained heads | ibo | 0.0 | 0.8 | 2.4 | 0.13 | 0.8 | 1.6 | 0.96 |
+| untrained heads | hau | 0.26 | 0.78 | 1.69 | 0.13 | 1.04 | 1.69 | 0.93 |
+| untrained heads | yor | 0.26 | 0.92 | 2.36 | 0.26 | 0.92 | 1.05 | 0.96 |
+| mCLIP zero-shot | eng | 58.34 | 84.67 | 92.21 | 57.85 | 83.31 | 89.99 | 77.73 |
+| mCLIP zero-shot | ibo | 6.13 | 14.38 | 20.91 | 4.26 | 11.98 | 16.11 | 12.29 |
+| mCLIP zero-shot | hau | 3.38 | 6.63 | 8.71 | 1.82 | 4.16 | 6.11 | 5.13 |
+| mCLIP zero-shot | yor | 2.36 | 4.84 | 6.41 | 1.31 | 2.75 | 4.84 | 3.75 |
+| **Native-Bird2 (frozen text, released)** | eng | 46.48 | 77.38 | 88.5 | 44.99 | 76.64 | 87.02 | 70.17 |
+| **Native-Bird2 (frozen text, released)** | ibo | **39.28** | **70.71** | **82.29** | **39.01** | **69.64** | **81.49** | **63.74** |
+| **Native-Bird2 (frozen text, released)** | hau | **33.16** | **61.64** | **74.64** | **32.25** | **59.43** | **74.9** | **56.0** |
+| **Native-Bird2 (frozen text, released)** | yor | **25.52** | **53.8** | **64.66** | **23.56** | **52.75** | **65.84** | **47.69** |
+| Native-Bird unfreeze top-2 | eng | 44.99 | 78.74 | 89.0 | 46.23 | 76.14 | 87.89 | 70.5 |
+| Native-Bird unfreeze top-2 | ibo | 37.95 | 70.57 | 83.49 | 39.68 | 70.17 | 83.36 | 64.2 |
+| Native-Bird unfreeze top-2 | hau | 31.73 | 60.47 | 74.9 | 30.3 | 60.73 | 74.38 | 55.42 |
+| Native-Bird unfreeze top-2 | yor | 23.3 | 52.88 | 64.66 | 24.87 | 53.27 | 64.4 | 47.23 |
+
+### test_unfiltered (same 809 images, raw captions, 809 per language)
+
+| model | lang | i2t R@1 | i2t R@5 | i2t R@10 | t2i R@1 | t2i R@5 | t2i R@10 | mean R |
+|---|---|---|---|---|---|---|---|---|
+| frozen text (released) | eng | 46.48 | 77.38 | 88.5 | 44.99 | 76.64 | 87.02 | 70.17 |
+| frozen text (released) | ibo | 36.59 | 67.37 | 79.6 | 35.6 | 66.75 | 79.23 | 60.86 |
+| frozen text (released) | hau | 31.52 | 59.21 | 72.06 | 30.9 | 57.73 | 72.81 | 54.04 |
+| frozen text (released) | yor | 24.6 | 52.78 | 63.54 | 22.5 | 51.79 | 64.4 | 46.6 |
+| unfreeze top-2 | eng | 44.99 | 78.74 | 89.0 | 46.23 | 76.14 | 87.89 | 70.5 |
+| unfreeze top-2 | ibo | 35.72 | 67.86 | 81.09 | 36.96 | 67.12 | 80.22 | 61.5 |
+| unfreeze top-2 | hau | 29.91 | 58.71 | 73.05 | 28.31 | 58.22 | 72.44 | 53.44 |
+| unfreeze top-2 | yor | 22.74 | 50.31 | 62.42 | 23.98 | 52.41 | 62.79 | 45.77 |
+
+Takeaways: on Igbo, Hausa and Yoruba the model beats multilingual CLIP by a wide margin (Igbo mean R 63.74 vs 12.29);
+on English, mCLIP (trained on hundreds of millions of image–text pairs) is still ahead (77.73 vs 70.17). Scores on the raw
+captions drop only 1–3 points, so the filter mainly cleans evaluation rather than propping up the model.
+
+## Igbo image search demo
+
+Released checkpoint, gallery = the 809 test images. **All 751 held-out Igbo test captions** used as queries, scoring where
+the caption's own image lands:
+
+| query language | queries | top-1 | top-5 | top-10 | median rank |
+|---|---|---|---|---|---|
+| Igbo | 751 | **37.4%** | **68.4%** | **80.6%** | **2** |
+| English (same images) | 809 | 45.0% | 76.6% | 87.0% | n/a |
+
+Natural Igbo queries (top-5 overlap with the results for the same query in English):
+
+| Igbo query | English | top-5 overlap with English query |
+|---|---|---|
+| Nkịta na-agba ọsọ n'elu ahịhịa | A dog running on the grass | 3/5 |
+| Ụmụaka na-egwu egwu n'ime mmiri | Children playing in the water | 5/5 |
+| Otu nwoke na-agba ịnyịnya ígwè | A man riding a bicycle | 4/5 |
+| Otu nwanyị yi uwe na-acha uhie uhie | A woman wearing a red dress | 4/5 |
+| Ndị mmadụ nọ n'elu ugwu | People on top of a mountain | 4/5 |
+| Otu nwata nwoke na-awụli elu | A boy jumping in the air | 3/5 |
+| Nkịta abụọ na-egwu egwu | Two dogs playing together | 2/5 |
+| Ọtụtụ mmadụ nọ n'okporo ụzọ | A crowd of people on a street | 4/5 |
+
+Full output, including per-query image ids and scores: `results/igbo_search.json`.
+
+## Usage
+
+### Standalone (no repo code)
+
+```python
+import torch, torch.nn.functional as F
+from PIL import Image
+from huggingface_hub import hf_hub_download
+from safetensors.torch import load_file
+from sentence_transformers import SentenceTransformer
+from transformers import AutoModel, AutoImageProcessor
+
+w = load_file(hf_hub_download("Modularcomputing/Native-Bird2", "model.safetensors"))
+text_enc = SentenceTransformer("Modularcomputing/Native-Bird")
+siglip = AutoModel.from_pretrained("google/siglip-base-patch16-224").eval()
+proc = AutoImageProcessor.from_pretrained("google/siglip-base-patch16-224")
+
+@torch.no_grad()
+def embed_text(texts):
+    e = text_enc.encode(texts, convert_to_tensor=True, normalize_embeddings=True).float().cpu()
+    return F.normalize(e @ w["txt.net.weight"].T + w["txt.net.bias"], dim=-1)
+
+@torch.no_grad()
+def embed_images(paths):
+    px = proc(images=[Image.open(p).convert("RGB") for p in paths], return_tensors="pt")["pixel_values"]
+    f = siglip.get_image_features(pixel_values=px)
+    if not torch.is_tensor(f):  # some transformers versions return a ModelOutput
+        f = f.pooler_output
+    return F.normalize(f.float() @ w["img.net.weight"].T + w["img.net.bias"], dim=-1)
+
+images = ["dog.jpg", "beach.jpg", "market.jpg"]
+scores = embed_text(["Nkịta na-agba ọsọ n'elu ahịhịa"]) @ embed_images(images).T   # cosine similarity
+print(images[scores.argmax().item()])
+```
+
+The same space works in every direction: Hausa or Yoruba queries, English queries, or image→caption retrieval.
+
+### With `search.py` (gallery = the 809 test images)
+
+```bash
+git clone https://huggingface.co/Modularcomputing/Native-Bird2 && cd Native-Bird2
+pip install torch torchvision && pip install -r requirements.txt
+# search.py reads the Flickr8k images from data/raw/flickr8k/Flicker8k_Dataset/
+python -c "from huggingface_hub import snapshot_download; snapshot_download('mustafa2ak/Flickr8k-Images', repo_type='dataset', allow_patterns=['Flicker8k_Dataset/*.jpg'], local_dir='data/raw/flickr8k')"
+python search.py --ckpt best_native_bird.pt "Nkịta abụọ na-egwu egwu" "A dog running on the grass" -k 5
+```
+
+Image features are computed once and cached under `data/features/`. Runs on CPU or GPU.
+
+## Limitations
+
+- **Machine-translated captions.** Igbo, Hausa and Yoruba captions in AfriCaption are machine translations of English
+  Flickr8k captions, not native-speaker descriptions. The filter removes obvious failures (loops, untranslated English,
+  wrong length, low semantic match), but errors, literal phrasing and code-mixing remain, and the model learned from them.
+- **Narrow image domain.** Flickr8k is about 8k mostly North-American/European everyday photos (people, dogs, sports,
+  outdoors). Expect weaker results on other scenes, on African contexts, on documents, products or fine-grained categories.
+- **Small training set** (6,473 images). Both arms overfit after epoch 2–3; the released checkpoint is the early-stopped one.
+- **Yoruba is the weakest language** (test mean R 47.69 vs 63.74 Igbo). Native-Bird's own Yoruba↔English bitext retrieval is
+  also lowest (acc@1 79.23 vs 96.66 Igbo, `results/text_encoder_crosslingual_test.json`), and LaBSE scores Yoruba lower, so
+  the filter's bottom-5% Yoruba tail drops some acceptable translations.
+- **The filter is LaBSE-based.** LaBSE is not the model under test, but Native-Bird is a LaBSE fine-tune, so the filter is not
+  fully independent of the text encoder.
+- **English is below mCLIP zero-shot** (mean R 70.17 vs 77.73); use a large English CLIP model if you only need English.
+- Retrieval over a gallery of ~800 candidates; numbers on larger galleries will be lower.
+
+## Citation
+
+Please cite AfriCaption, the dataset this model is trained on ([ACL Anthology](https://aclanthology.org/2026.africanlp-main.5/), [arXiv:2510.17405](https://arxiv.org/abs/2510.17405)):
+
+```bibtex
+@inproceedings{oduwole-etal-2026-africaption,
+    title = "{A}fri{C}aption: Establishing a New Paradigm for Image Captioning in {A}frican Languages",
+    author = "Oduwole, Mardiyyah and Mireku, Prince and Adebanjo, Fatimo and Olajide, Oluwatosin and Aliyu, Mahi Aminu and Novikova, Jekaterina",
+    booktitle = "Proceedings of the 7th Workshop on {A}frican Natural Language Processing ({A}frica{NLP} 2026)",
+    month = mar,
+    year = "2026",
+    address = "Rabat, Morocco",
+    publisher = "Association for Computational Linguistics",
+    url = "https://aclanthology.org/2026.africanlp-main.5/",
+    doi = "10.18653/v1/2026.africanlp-main.5",
+    pages = "44--55"
+}
+```
+
+Also see [SigLIP](https://huggingface.co/google/siglip-base-patch16-224) and [Native-Bird](https://huggingface.co/Modularcomputing/Native-Bird).
+
+Built by [Modularcomputing](https://huggingface.co/Modularcomputing).

best_native_bird.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:209316f3376f73b94836fe7129964620b8a6837db0eaf2191131ddc61830bf30
+size 3153457

config.json

@@ -0,0 +1,22 @@
+{
+  "architecture": "two linear projection heads (img, txt) -> L2-normalised shared space",
+  "image_encoder": "google/siglip-base-patch16-224",
+  "text_encoder": "Modularcomputing/Native-Bird",
+  "text_encoder_frozen": true,
+  "d_img": 768,
+  "d_txt": 768,
+  "proj_dim": 512,
+  "proj_hidden": 0,
+  "logit_scale_log": 2.903928756713867,
+  "logit_scale": 18.24568748474121,
+  "best_epoch": 3,
+  "max_text_len": 64,
+  "text_prompt": "",
+  "weights": {
+    "img.net.weight": "[proj_dim, d_img]",
+    "img.net.bias": "[proj_dim]",
+    "txt.net.weight": "[proj_dim, d_txt]",
+    "txt.net.bias": "[proj_dim]",
+    "logit_scale": "scalar, log space"
+  }
+}

configs/default.yaml

@@ -0,0 +1,51 @@
+# Native-Bird image-text alignment (AfriCaption / Flickr8k)
+# All paths are relative to the project root.
+seed: 42
+data:
+  dir: data/processed            # train/val/test parquet from scripts/04_split.py
+  image_dir: data/raw/flickr8k/Flicker8k_Dataset
+  langs: [eng, ibo, hau, yor]    # captions used for training (each image contributes one pair per kept language)
+  max_text_len: 64
+  subset: null                   # int -> use only first N images per split (smoke tests)
+
+image_encoder:
+  # Any open_clip model or HF CLIP/SigLIP checkpoint. Frozen; features are cached to disk once.
+  backend: hf                    # hf | open_clip
+  name: google/siglip-base-patch16-224
+  cache_dir: data/features       # cached image features: <cache_dir>/<safe_name>/<split>.pt
+
+text_encoder:
+  # Pick the arm. Each arm is a sentence-transformers model id.
+  arm: native_bird               # native_bird | embeddinggemma2 | embeddinggemma1 | labse | custom
+  arms:
+    native_bird:     {name: Modularcomputing/Native-Bird, prompt: ""}
+    embeddinggemma2: {name: google/embeddinggemma-2, prompt: "task: search result | query: ", text_only: true}
+    embeddinggemma1: {name: google/embeddinggemma-300m, prompt: "task: search result | query: "}
+    labse:           {name: sentence-transformers/LaBSE, prompt: ""}
+    custom:          {name: null, prompt: ""}
+  unfreeze_last_n_layers: 0      # 0 = fully frozen (text embeddings are cached); >0 fine-tunes the top-N transformer layers
+  cache_dir: data/features
+
+model:
+  proj_dim: 512
+  proj_hidden: 0                 # 0 = linear projection; >0 = 2-layer MLP with this hidden size
+  dropout: 0.1
+  init_logit_scale: 2.659        # ln(1/0.07), CLIP default; learnable, clamped at ln(100)
+
+train:
+  epochs: 20
+  batch_size: 256                # contrastive batch; bigger = more negatives
+  lr: 1.0e-3                     # projection heads
+  text_lr: 1.0e-5                # unfrozen text layers (only if unfreeze_last_n_layers > 0)
+  weight_decay: 0.01
+  warmup_steps: 100
+  same_image_negatives: mask     # mask: captions of the same image in other languages are not treated as negatives
+  device: auto                   # auto | cuda | cpu
+  num_workers: 4
+  out_dir: runs/${text_encoder.arm}
+  eval_every: 1
+
+eval:
+  split: test
+  ks: [1, 5, 10]
+  langs: [eng, ibo, hau, yor]

data/processed/filter_stats.json

@@ -0,0 +1,259 @@
+{
+ "stats": {
+  "ibo": {
+   "total": 8091,
+   "kept": 7436,
+   "dropped": 655,
+   "by_rule": {
+    "R2_loop": 6,
+    "R3_copy": 250,
+    "R4_length": 7,
+    "R5_semantic": 392
+   },
+   "semantic_threshold": 0.7416,
+   "labse_sim_pctiles_all": {
+    "1": 0.639,
+    "5": 0.742,
+    "25": 0.83,
+    "50": 0.871
+   },
+   "agreement_with_afrimm_filtered": {
+    "afrimm_kept": 8078,
+    "both_kept": 7436,
+    "we_drop_afrimm_keeps": 642,
+    "we_keep_afrimm_drops": 0
+   }
+  },
+  "hau": {
+   "total": 8091,
+   "kept": 7655,
+   "dropped": 436,
+   "by_rule": {
+    "R2_loop": 12,
+    "R3_copy": 11,
+    "R4_length": 10,
+    "R5_semantic": 403
+   },
+   "semantic_threshold": 0.6025,
+   "labse_sim_pctiles_all": {
+    "1": 0.489,
+    "5": 0.601,
+    "25": 0.726,
+    "50": 0.796
+   },
+   "agreement_with_afrimm_filtered": {
+    "afrimm_kept": 7943,
+    "both_kept": 7655,
+    "we_drop_afrimm_keeps": 288,
+    "we_keep_afrimm_drops": 0
+   }
+  },
+  "yor": {
+   "total": 8091,
+   "kept": 7611,
+   "dropped": 480,
+   "by_rule": {
+    "R2_loop": 24,
+    "R3_copy": 16,
+    "R4_length": 39,
+    "R5_semantic": 401
+   },
+   "semantic_threshold": 0.5099,
+   "labse_sim_pctiles_all": {
+    "1": 0.33,
+    "5": 0.501,
+    "25": 0.658,
+    "50": 0.737
+   },
+   "agreement_with_afrimm_filtered": {
+    "afrimm_kept": 7526,
+    "both_kept": 7490,
+    "we_drop_afrimm_keeps": 36,
+    "we_keep_afrimm_drops": 121
+   }
+  },
+  "params": {
+   "LEN_LO": 0.45,
+   "LEN_HI": 2.2,
+   "ABS_FLOOR": 0.5,
+   "PCT": 5,
+   "scorer": "sentence-transformers/LaBSE"
+  }
+ },
+ "examples": {
+  "ibo": {
+   "R2_loop": [
+    [
+     "A little girl is carefully coloring in the pictures of a coloring book .",
+     "Nwatakịrị nwanyị na-eji nlezianya agba agba agba"
+    ],
+    [
+     "A little girl with blonde hair and a blue shirt is blowing bubbles outside with someone .",
+     "Nwatakịrị nwanyị nwere ntutu isi na-acha odo odo na uwe elu na-acha anụnụ anụnụ na-afụ afụ afụ n'èzí na mmadụ ."
+    ],
+    [
+     "A blue car drives down a mostly dirt road .",
+     "A-acha anụnụ anụnụ ụgbọala ụgbọala ụgbọala ụgbọala ala a ukwuu n'ime unyi ụzọ ."
+    ]
+   ],
+   "R3_copy": [
+    [
+     "A man is rock climbing .",
+     "Otu nwoke bụ rock climbing ."
+    ],
+    [
+     "A snowboarder sits on a slope with skiers and boarders nearby .",
+     "A snowboarder na-anọdụ na a mkpọda na skiers na boarders nso ."
+    ],
+    [
+     "Young boy with a black \" ADHD \" t-shirt nodding off",
+     "Nwa okorobịa nwere \"ADHD\" t-shirt nodding off"
+    ]
+   ],
+   "R4_length": [
+    [
+     "A child in a swing .",
+     "Nwatakịrị nọ n'ebe a na-agbagharị agbagharị ."
+    ],
+    [
+     "a team of rowers rowing in a green and red boat",
+     "Otu ìgwè ndị na-agba ụgbọ mmiri na-agba ụgbọ mmiri na-acha akwụkwọ ndụ akwụkwọ ndụ na nke na-acha uhie uhie"
+    ],
+    [
+     "A surfer is riding a wave .",
+     "Onye na - eme njem n' ime mmiri na - agbagharị n' ebili mmiri."
+    ]
+   ],
+   "R5_semantic": [
+    [
+     "a man is standing in front of a skyscraper",
+     "Otu nwoke na-eguzo n'ihu ụlọ elu."
+    ],
+    [
+     "Woman with three children fishing over boardwalk in the evening .",
+     "Otu nwaanyị nwere ụmụ atọ na - akụ azụ̀ n' elu osimiri n' anyasị."
+    ],
+    [
+     "A boy jumping in a fountain .",
+     "Otu nwata nwoke na-awụlikwa elu n'isi iyi ."
+    ]
+   ]
+  },
+  "hau": {
+   "R2_loop": [
+    [
+     "Hockey goalie boy in red jacket crouches by goal , with stick .",
+     "Wani dan wasan Hockey mai tsaron gida a cikin jacket ja ja ja da kullun da ƙofa, tare da sanda."
+    ],
+    [
+     "A young boy plays on a swing .",
+     "Yaro yana wasa a kan ɗan ɗan ɗan ɗan"
+    ],
+    [
+     "A person swinging on a swing",
+     "Wani mutum yana ɗaukan ɗaukan ɗaukan"
+    ]
+   ],
+   "R3_copy": [
+    [
+     "a man dressed as Jack Sparrow from Pirates of the Caribbean",
+     "Wani mutumin da ya saka tufafin Jack Sparrow daga Pirates of the Caribbean"
+    ],
+    [
+     "A child sleds .",
+     "Yaro ya yi sleds ."
+    ],
+    [
+     "Woman in a blue and white striped shirt with a straw cowboy hat on .",
+     "Wata mace a cikin blue da fari striped shirt da straw cowboy hat a kan."
+    ]
+   ],
+   "R4_length": [
+    [
+     "Young man with a nose ring and a hat .",
+     "Matashi mai ɗan"
+    ],
+    [
+     "Child with striped shirt , red shorts and orange cap standing on pavement near cars .",
+     "Yaro da shiti mai launin"
+    ],
+    [
+     "The two children swinging together on a swing .",
+     "Yara biyu suna ɗaukan"
+    ]
+   ],
+   "R5_semantic": [
+    [
+     "A dog leaps to catch a ball in a field .",
+     "Karnuka suna tsallake don su kama ƙwallo a fili ."
+    ],
+    [
+     "A child jumps off a high diving board into the pool .",
+     "Yaro ya tsallake daga tafiyar ruwan da ke cikin ruwan da ke da tsaye ."
+    ],
+    [
+     "A boy jumping in a fountain .",
+     "Yaro yana tsallake cikin ruwan marar ruwa ."
+    ]
+   ]
+  },
+  "yor": {
+   "R2_loop": [
+    [
+     "A woman is riding a three wheeler bicycle behind a pack of dogs .",
+     "Obìnrin kan ń gun kẹ̀kẹ́ kẹ̀kẹ́ mẹ́ta lẹ́yìn àwọn ajá kan ."
+    ],
+    [
+     "Two little kids in strollers have face makeup on like the band \" Kiss \" .",
+     "Àwọn ọmọ kéékèèké méjì tí wọ́n wọ kẹ̀kẹ́ kẹ̀kẹ́ máa ń dojú kọ bí ẹgbẹ́ náà \" Kiss \" ."
+    ],
+    [
+     "Two women riding a tandem bicycle go by a wall painting .",
+     "Àwọn obìnrin méjì tí wọ́n ń gun kẹ̀kẹ́ kẹ̀kẹ́ alágbèéká máa ń lọ lẹ́gbẹ̀ẹ́ ògiri ."
+    ]
+   ],
+   "R3_copy": [
+    [
+     "a man dressed as Jack Sparrow from Pirates of the Caribbean",
+     "Ọkùnrin kan tí ó múra gẹ́gẹ́ bíi Jack Sparrow láti Pirates of the Caribbean"
+    ],
+    [
+     "A motocross biker about to descend",
+     "Motocross biker fẹ́ sọ̀kalẹ̀"
+    ],
+    [
+     "person on a wakeboard in the air",
+     "eniyan lori a wakeboard ni air"
+    ]
+   ],
+   "R4_length": [
+    [
+     "A thin brown horse standing and a small black horse sitting on sand .",
+     "Ẹṣin aláwọ̀"
+    ],
+    [
+     "A black and brown dog is laying on a white shaggy carpet .",
+     "Ajá aláwọ̀ dúdú àti aláwọ̀"
+    ],
+    [
+     "The skinny brown dog is walking on a fallen tree .",
+     "Ajá aláwọ̀ aláwọ"
+    ]
+   ],
+   "R5_semantic": [
+    [
+     "Two dogs playing together on a beach .",
+     "Àwọn ajá méjì ń ṣeré pa pọ̀ ní etíkun ."
+    ],
+    [
+     "an ice climber in a blue jacket and black pants is scaling a frozen ice wall .",
+     "Ẹni tó ń gun yìnyín nínú jákẹ́ẹ̀tì aláwọ̀ búlúù àti ṣòkòtò dúdú ń gbé ògiri yìnyín dídì ."
+    ],
+    [
+     "A light-colored dog runs on the beach .",
+     "Ajá aláwọ̀ fẹ́lẹ́fẹ́lẹ́ máa ń ṣiṣẹ́ ní etí òkun."
+    ]
+   ]
+  }
+ }
+}

data/processed/split_summary.json

@@ -0,0 +1,37 @@
+{
+ "seed": 42,
+ "fractions": [
+  0.8,
+  0.1,
+  0.1
+ ],
+ "images_total": 8091,
+ "train": {
+  "images": 6473,
+  "captions_eng": 6473,
+  "captions_ibo": 5943,
+  "captions_hau": 6116,
+  "captions_yor": 6083,
+  "pairs_total": 24615
+ },
+ "val": {
+  "images": 809,
+  "captions_eng": 809,
+  "captions_ibo": 742,
+  "captions_hau": 770,
+  "captions_yor": 764,
+  "pairs_total": 3085
+ },
+ "test": {
+  "images": 809,
+  "captions_eng": 809,
+  "captions_ibo": 751,
+  "captions_hau": 769,
+  "captions_yor": 764,
+  "pairs_total": 3093
+ },
+ "test_unfiltered": {
+  "images": 809
+ },
+ "leakage_check": "passed: no image_id in more than one split"
+}

data/processed/test.parquet

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c648645e35903692a55b59221c3c2f7cbd6de862e90df637ca0ad88d8a360be3
+size 126674

metrics.json

@@ -0,0 +1,322 @@
+{
+  "best_arm_by_val": "native_bird",
+  "image_encoder": "google/siglip-base-patch16-224 (frozen)",
+  "text_encoder": "Modularcomputing/Native-Bird",
+  "runs": {
+    "baseline_mclip|test": {
+      "eng": {
+        "n": 809,
+        "i2t_R@1": 58.34,
+        "t2i_R@1": 57.85,
+        "i2t_R@5": 84.67,
+        "t2i_R@5": 83.31,
+        "i2t_R@10": 92.21,
+        "t2i_R@10": 89.99,
+        "mean_R": 77.73
+      },
+      "ibo": {
+        "n": 751,
+        "i2t_R@1": 6.13,
+        "t2i_R@1": 4.26,
+        "i2t_R@5": 14.38,
+        "t2i_R@5": 11.98,
+        "i2t_R@10": 20.91,
+        "t2i_R@10": 16.11,
+        "mean_R": 12.29
+      },
+      "hau": {
+        "n": 769,
+        "i2t_R@1": 3.38,
+        "t2i_R@1": 1.82,
+        "i2t_R@5": 6.63,
+        "t2i_R@5": 4.16,
+        "i2t_R@10": 8.71,
+        "t2i_R@10": 6.11,
+        "mean_R": 5.13
+      },
+      "yor": {
+        "n": 764,
+        "i2t_R@1": 2.36,
+        "t2i_R@1": 1.31,
+        "i2t_R@5": 4.84,
+        "t2i_R@5": 2.75,
+        "i2t_R@10": 6.41,
+        "t2i_R@10": 4.84,
+        "mean_R": 3.75
+      }
+    },
+    "baseline_untrained_native_bird|test": {
+      "eng": {
+        "n": 809,
+        "i2t_R@1": 0.37,
+        "t2i_R@1": 0.12,
+        "i2t_R@5": 1.11,
+        "t2i_R@5": 0.87,
+        "i2t_R@10": 1.48,
+        "t2i_R@10": 1.73,
+        "mean_R": 0.95
+      },
+      "ibo": {
+        "n": 751,
+        "i2t_R@1": 0.0,
+        "t2i_R@1": 0.13,
+        "i2t_R@5": 0.8,
+        "t2i_R@5": 0.8,
+        "i2t_R@10": 2.4,
+        "t2i_R@10": 1.6,
+        "mean_R": 0.96
+      },
+      "hau": {
+        "n": 769,
+        "i2t_R@1": 0.26,
+        "t2i_R@1": 0.13,
+        "i2t_R@5": 0.78,
+        "t2i_R@5": 1.04,
+        "i2t_R@10": 1.69,
+        "t2i_R@10": 1.69,
+        "mean_R": 0.93
+      },
+      "yor": {
+        "n": 764,
+        "i2t_R@1": 0.26,
+        "t2i_R@1": 0.26,
+        "i2t_R@5": 0.92,
+        "t2i_R@5": 0.92,
+        "i2t_R@10": 2.36,
+        "t2i_R@10": 1.05,
+        "mean_R": 0.96
+      }
+    },
+    "trained_native_bird|test": {
+      "eng": {
+        "n": 809,
+        "i2t_R@1": 46.48,
+        "t2i_R@1": 44.99,
+        "i2t_R@5": 77.38,
+        "t2i_R@5": 76.64,
+        "i2t_R@10": 88.5,
+        "t2i_R@10": 87.02,
+        "mean_R": 70.17
+      },
+      "ibo": {
+        "n": 751,
+        "i2t_R@1": 39.28,
+        "t2i_R@1": 39.01,
+        "i2t_R@5": 70.71,
+        "t2i_R@5": 69.64,
+        "i2t_R@10": 82.29,
+        "t2i_R@10": 81.49,
+        "mean_R": 63.74
+      },
+      "hau": {
+        "n": 769,
+        "i2t_R@1": 33.16,
+        "t2i_R@1": 32.25,
+        "i2t_R@5": 61.64,
+        "t2i_R@5": 59.43,
+        "i2t_R@10": 74.64,
+        "t2i_R@10": 74.9,
+        "mean_R": 56.0
+      },
+      "yor": {
+        "n": 764,
+        "i2t_R@1": 25.52,
+        "t2i_R@1": 23.56,
+        "i2t_R@5": 53.8,
+        "t2i_R@5": 52.75,
+        "i2t_R@10": 64.66,
+        "t2i_R@10": 65.84,
+        "mean_R": 47.69
+      }
+    },
+    "trained_native_bird|test_unfiltered": {
+      "eng": {
+        "n": 809,
+        "i2t_R@1": 46.48,
+        "t2i_R@1": 44.99,
+        "i2t_R@5": 77.38,
+        "t2i_R@5": 76.64,
+        "i2t_R@10": 88.5,
+        "t2i_R@10": 87.02,
+        "mean_R": 70.17
+      },
+      "ibo": {
+        "n": 809,
+        "i2t_R@1": 36.59,
+        "t2i_R@1": 35.6,
+        "i2t_R@5": 67.37,
+        "t2i_R@5": 66.75,
+        "i2t_R@10": 79.6,
+        "t2i_R@10": 79.23,
+        "mean_R": 60.86
+      },
+      "hau": {
+        "n": 809,
+        "i2t_R@1": 31.52,
+        "t2i_R@1": 30.9,
+        "i2t_R@5": 59.21,
+        "t2i_R@5": 57.73,
+        "i2t_R@10": 72.06,
+        "t2i_R@10": 72.81,
+        "mean_R": 54.04
+      },
+      "yor": {
+        "n": 809,
+        "i2t_R@1": 24.6,
+        "t2i_R@1": 22.5,
+        "i2t_R@5": 52.78,
+        "t2i_R@5": 51.79,
+        "i2t_R@10": 63.54,
+        "t2i_R@10": 64.4,
+        "mean_R": 46.6
+      }
+    },
+    "trained_native_bird_unfreeze2|test": {
+      "eng": {
+        "n": 809,
+        "i2t_R@1": 44.99,
+        "t2i_R@1": 46.23,
+        "i2t_R@5": 78.74,
+        "t2i_R@5": 76.14,
+        "i2t_R@10": 89.0,
+        "t2i_R@10": 87.89,
+        "mean_R": 70.5
+      },
+      "ibo": {
+        "n": 751,
+        "i2t_R@1": 37.95,
+        "t2i_R@1": 39.68,
+        "i2t_R@5": 70.57,
+        "t2i_R@5": 70.17,
+        "i2t_R@10": 83.49,
+        "t2i_R@10": 83.36,
+        "mean_R": 64.2
+      },
+      "hau": {
+        "n": 769,
+        "i2t_R@1": 31.73,
+        "t2i_R@1": 30.3,
+        "i2t_R@5": 60.47,
+        "t2i_R@5": 60.73,
+        "i2t_R@10": 74.9,
+        "t2i_R@10": 74.38,
+        "mean_R": 55.42
+      },
+      "yor": {
+        "n": 764,
+        "i2t_R@1": 23.3,
+        "t2i_R@1": 24.87,
+        "i2t_R@5": 52.88,
+        "t2i_R@5": 53.27,
+        "i2t_R@10": 64.66,
+        "t2i_R@10": 64.4,
+        "mean_R": 47.23
+      }
+    },
+    "trained_native_bird_unfreeze2|test_unfiltered": {
+      "eng": {
+        "n": 809,
+        "i2t_R@1": 44.99,
+        "t2i_R@1": 46.23,
+        "i2t_R@5": 78.74,
+        "t2i_R@5": 76.14,
+        "i2t_R@10": 89.0,
+        "t2i_R@10": 87.89,
+        "mean_R": 70.5
+      },
+      "ibo": {
+        "n": 809,
+        "i2t_R@1": 35.72,
+        "t2i_R@1": 36.96,
+        "i2t_R@5": 67.86,
+        "t2i_R@5": 67.12,
+        "i2t_R@10": 81.09,
+        "t2i_R@10": 80.22,
+        "mean_R": 61.5
+      },
+      "hau": {
+        "n": 809,
+        "i2t_R@1": 29.91,
+        "t2i_R@1": 28.31,
+        "i2t_R@5": 58.71,
+        "t2i_R@5": 58.22,
+        "i2t_R@10": 73.05,
+        "t2i_R@10": 72.44,
+        "mean_R": 53.44
+      },
+      "yor": {
+        "n": 809,
+        "i2t_R@1": 22.74,
+        "t2i_R@1": 23.98,
+        "i2t_R@5": 50.31,
+        "t2i_R@5": 52.41,
+        "i2t_R@10": 62.42,
+        "t2i_R@10": 62.79,
+        "mean_R": 45.77
+      }
+    }
+  },
+  "train": {
+    "native_bird": {
+      "best_val_mean_R": 60.46,
+      "best_epoch": 3,
+      "epochs": 20
+    },
+    "native_bird_unfreeze2": {
+      "best_val_mean_R": 60.35,
+      "best_epoch": 2,
+      "epochs": 20
+    }
+  },
+  "text_crosslingual_test": {
+    "native_bird": {
+      "avg_acc@1_cap2eng": 87.93,
+      "ibo": 96.66,
+      "hau": 87.89,
+      "yor": 79.23
+    },
+    "labse": {
+      "avg_acc@1_cap2eng": 89.91,
+      "ibo": 97.16,
+      "hau": 89.99,
+      "yor": 82.57
+    }
+  },
+  "igbo_search": {
+    "all_ibo_captions": {
+      "n": 751,
+      "hit@1": 37.42,
+      "hit@5": 68.44,
+      "hit@10": 80.56,
+      "median_rank": 2
+    },
+    "all_eng_captions": {
+      "n": 809,
+      "hit@1": 44.99,
+      "hit@5": 76.64,
+      "hit@10": 87.02,
+      "median_rank": 2
+    },
+    "heldout_sample_top5": "4/6"
+  },
+  "smoke_mean_R_128imgs": {
+    "untrained": {
+      "eng": 4.6,
+      "ibo": 3.5,
+      "hau": 2.8,
+      "yor": 5.3
+    },
+    "mclip": {
+      "eng": 92.3,
+      "ibo": 22.5,
+      "hau": 7.7,
+      "yor": 6.8
+    },
+    "trained_native_bird_1ep": {
+      "eng": 42.1,
+      "ibo": 42.4,
+      "hau": 38.2,
+      "yor": 33.9
+    }
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e8187e82f1ee49691be1b5a7ef271cc087b57b5da336c02d213d7d7755c94dcb
+size 3150260

nbmm/common.py

@@ -0,0 +1,259 @@
+"""Shared pieces: config, encoders, feature caching, projection heads, loss, retrieval metrics."""
+from __future__ import annotations
+import os, re, json, random, hashlib
+from dataclasses import dataclass
+import numpy as np, pandas as pd, torch, torch.nn as nn, torch.nn.functional as F, yaml
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+
+
+# ----------------------------------------------------------------------------- config
+def _set(d, dotted, val):
+    ks = dotted.split(".")
+    for k in ks[:-1]:
+        d = d.setdefault(k, {})
+    d[ks[-1]] = yaml.safe_load(val) if isinstance(val, str) else val
+
+
+def _get(d, dotted):
+    for k in dotted.split("."):
+        d = d[k]
+    return d
+
+
+def load_config(path, overrides=()):
+    cfg = yaml.safe_load(open(path))
+    for o in overrides:  # key.sub=value
+        k, v = o.split("=", 1)
+        _set(cfg, k, v)
+    s = yaml.safe_dump(cfg)
+    s = re.sub(r"\$\{([^}]+)\}", lambda m: str(_get(cfg, m.group(1))), s)
+    return yaml.safe_load(s)
+
+
+def rel(p):
+    return p if os.path.isabs(p) else os.path.join(ROOT, p)
+
+
+def seed_all(s):
+    random.seed(s); np.random.seed(s); torch.manual_seed(s); torch.cuda.manual_seed_all(s)
+
+
+def pick_device(name="auto"):
+    if name != "auto":
+        return torch.device(name)
+    if torch.cuda.is_available():
+        return torch.device("cuda")
+    if getattr(torch.backends, "mps", None) and torch.backends.mps.is_available():
+        return torch.device("mps")
+    return torch.device("cpu")
+
+
+def safe(name):
+    return re.sub(r"[^A-Za-z0-9._-]+", "_", name)
+
+
+# ----------------------------------------------------------------------------- data
+def load_split(cfg, split):
+    df = pd.read_parquet(rel(os.path.join(cfg["data"]["dir"], f"{split}.parquet")))
+    n = cfg["data"].get("subset")
+    if n:
+        df = df.iloc[: int(n)].reset_index(drop=True)
+    return df
+
+
+def long_pairs(df, langs):
+    """Wide (one row per image, one column per language, NaN = filtered out) -> long (image_idx, lang, text)."""
+    rows = []
+    for i, r in df.iterrows():
+        for l in langs:
+            t = r.get(l)
+            if isinstance(t, str) and t.strip():
+                rows.append((i, l, t))
+    return pd.DataFrame(rows, columns=["img_idx", "lang", "text"])
+
+
+# ----------------------------------------------------------------------------- image encoder (frozen)
+class ImageEncoder:
+    def __init__(self, cfg, device):
+        ic = cfg["image_encoder"]
+        self.backend, self.name, self.device = ic["backend"], ic["name"], device
+        if self.backend == "hf":
+            from transformers import AutoModel, AutoImageProcessor
+            self.model = AutoModel.from_pretrained(self.name).to(device).eval()
+            self.proc = AutoImageProcessor.from_pretrained(self.name)
+        elif self.backend == "open_clip":
+            import open_clip
+            arch, pre = self.name.split("/", 1)
+            self.model, _, self.proc = open_clip.create_model_and_transforms(arch, pretrained=pre)
+            self.model = self.model.to(device).eval()
+        else:
+            raise ValueError(self.backend)
+
+    @torch.no_grad()
+    def encode(self, pil_images):
+        if self.backend == "hf":
+            px = self.proc(images=pil_images, return_tensors="pt")["pixel_values"].to(self.device)
+            out = self.model.get_image_features(pixel_values=px)
+            if not torch.is_tensor(out):  # some versions return a ModelOutput
+                out = getattr(out, "pooler_output", None) if getattr(out, "pooler_output", None) is not None else out[0]
+        else:
+            px = torch.stack([self.proc(im) for im in pil_images]).to(self.device)
+            out = self.model.encode_image(px)
+        return out.float().cpu()
+
+
+def image_features(cfg, df, split, device, batch_size=32):
+    """Cached frozen image features for the rows of df, keyed by image_id."""
+    from PIL import Image
+    ic = cfg["image_encoder"]
+    d = rel(os.path.join(ic["cache_dir"], "img_" + safe(ic["name"])))
+    os.makedirs(d, exist_ok=True)
+    path = os.path.join(d, "all.pt")
+    cache = torch.load(path) if os.path.exists(path) else {}
+    need = [i for i in df.image_id if i not in cache]
+    if need:
+        enc = ImageEncoder(cfg, device)
+        idir = rel(cfg["data"]["image_dir"])
+        import time
+        last = time.time()
+        for s in range(0, len(need), batch_size):
+            ids = need[s: s + batch_size]
+            ims = [Image.open(os.path.join(idir, i)).convert("RGB") for i in ids]
+            for i, f in zip(ids, enc.encode(ims)):
+                cache[i] = f
+            if time.time() - last > 20:  # checkpoint the cache so an interrupted run resumes
+                print(f"  image feats {s + len(ids)}/{len(need)}", flush=True)
+                torch.save(cache, path); last = time.time()
+        torch.save(cache, path)
+        del enc
+    return torch.stack([cache[i] for i in df.image_id])
+
+
+# ----------------------------------------------------------------------------- text encoder
+def text_arm(cfg):
+    tc = cfg["text_encoder"]
+    a = dict(tc["arms"][tc["arm"]])
+    if not a.get("name"):
+        raise ValueError(f"text_encoder.arms.{tc['arm']}.name is empty")
+    return a
+
+
+def load_sentence_model(arm, device):
+    from sentence_transformers import SentenceTransformer
+    kw = {}
+    if arm.get("text_only"):  # EmbeddingGemma 2: skip vision/audio towers (270M text-only)
+        kw["config_kwargs"] = {"vision_config": None, "audio_config": None}
+    if "embeddinggemma" in arm["name"]:
+        # EmbeddingGemma: bf16 or fp32 only (fp16 overflows)
+        dt = torch.bfloat16 if (device.type == "cuda" and torch.cuda.is_bf16_supported()) else torch.float32
+        kw["model_kwargs"] = {"torch_dtype": dt}
+    return SentenceTransformer(arm["name"], device=str(device), **kw)
+
+
+def text_features(cfg, texts, device, batch_size=64):
+    """Cached frozen sentence embeddings (L2-normalised by the model's own head if it has one)."""
+    arm = text_arm(cfg)
+    d = rel(os.path.join(cfg["text_encoder"]["cache_dir"], "txt_" + safe(arm["name"])))
+    os.makedirs(d, exist_ok=True)
+    path = os.path.join(d, "all.pt")
+    cache = torch.load(path) if os.path.exists(path) else {}
+    key = lambda t: hashlib.md5((arm.get("prompt", "") + t).encode()).hexdigest()
+    need = sorted({t for t in texts if key(t) not in cache})
+    if need:
+        m = load_sentence_model(arm, device)
+        for s in range(0, len(need), 512):
+            chunk = need[s: s + 512]
+            e = m.encode([arm.get("prompt", "") + t for t in chunk], batch_size=batch_size,
+                         convert_to_tensor=True, normalize_embeddings=True).float().cpu()
+            for t, v in zip(chunk, e):
+                cache[key(t)] = v
+            print(f"  text feats {s + len(chunk)}/{len(need)}", flush=True)
+            torch.save(cache, path)
+        del m
+    return torch.stack([cache[key(t)] for t in texts])
+
+
+class TrainableText(nn.Module):
+    """Sentence-transformers model with only its top-N transformer layers trainable."""
+
+    def __init__(self, cfg, device):
+        super().__init__()
+        self.arm = text_arm(cfg)
+        self.st = load_sentence_model(self.arm, device)
+        n = cfg["text_encoder"]["unfreeze_last_n_layers"]
+        for p in self.st.parameters():
+            p.requires_grad = False
+        layers = None
+        for mod in self.st.modules():  # find the deepest ModuleList of transformer blocks
+            for name, child in mod.named_children():
+                if isinstance(child, nn.ModuleList) and name in ("layer", "layers") and len(child) > 1:
+                    layers = child
+        if layers is None:
+            raise RuntimeError("could not locate transformer layers to unfreeze")
+        for blk in list(layers)[-n:]:
+            for p in blk.parameters():
+                p.requires_grad = True
+        self.max_len = cfg["data"]["max_text_len"]
+        self.device = device
+
+    def forward(self, texts):
+        texts = [self.arm.get("prompt", "") + t for t in texts]
+        feats = (self.st.preprocess if hasattr(self.st, "preprocess") else self.st.tokenize)(texts)
+        feats = {k: v[:, : self.max_len].to(self.device) if torch.is_tensor(v) and v.dim() == 2 else v for k, v in feats.items()}
+        return self.st(feats)["sentence_embedding"].float()
+
+
+# ----------------------------------------------------------------------------- heads + loss
+class Head(nn.Module):
+    def __init__(self, din, dout, hidden=0, dropout=0.0):
+        super().__init__()
+        self.net = nn.Linear(din, dout) if not hidden else nn.Sequential(
+            nn.Linear(din, hidden), nn.GELU(), nn.Dropout(dropout), nn.Linear(hidden, dout))
+
+    def forward(self, x):
+        return F.normalize(self.net(x), dim=-1)
+
+
+class AlignModel(nn.Module):
+    def __init__(self, d_img, d_txt, cfg):
+        super().__init__()
+        m = cfg["model"]
+        self.img = Head(d_img, m["proj_dim"], m["proj_hidden"], m["dropout"])
+        self.txt = Head(d_txt, m["proj_dim"], m["proj_hidden"], m["dropout"])
+        self.logit_scale = nn.Parameter(torch.tensor(float(m["init_logit_scale"])))
+
+
+def clip_loss(zi, zt, img_ids, logit_scale, mask_same_image=True):
+    """Symmetric InfoNCE. With multilingual captions several texts can share an image in one batch;
+    those off-diagonal same-image pairs are masked out of the negatives."""
+    s = logit_scale.clamp(max=4.6052).exp()
+    logits = s * zi @ zt.t()
+    if mask_same_image:
+        same = img_ids[:, None] == img_ids[None, :]
+        eye = torch.eye(len(img_ids), dtype=torch.bool, device=logits.device)
+        logits = logits.masked_fill(same & ~eye, float("-inf"))
+    y = torch.arange(len(img_ids), device=logits.device)
+    return 0.5 * (F.cross_entropy(logits, y) + F.cross_entropy(logits.t(), y))
+
+
+# ----------------------------------------------------------------------------- metrics
+@torch.no_grad()
+def retrieval_recall(zi, zt, ks=(1, 5, 10)):
+    """zi[i] and zt[i] are a matched image/caption pair (one caption per image within a language)."""
+    sim = zi @ zt.t()
+    n = sim.shape[0]
+    gt = torch.arange(n)
+    r_i2t = (sim.argsort(dim=1, descending=True) == gt[:, None]).float().argmax(dim=1)
+    r_t2i = (sim.t().argsort(dim=1, descending=True) == gt[:, None]).float().argmax(dim=1)
+    out = {"n": n}
+    for k in ks:
+        out[f"i2t_R@{k}"] = round(100 * (r_i2t < k).float().mean().item(), 2)
+        out[f"t2i_R@{k}"] = round(100 * (r_t2i < k).float().mean().item(), 2)
+    out["mean_R"] = round(float(np.mean([out[f"{d}_R@{k}"] for d in ("i2t", "t2i") for k in ks])), 2)
+    return out
+
+
+def dump(obj, path):
+    os.makedirs(os.path.dirname(path), exist_ok=True)
+    json.dump(obj, open(path, "w"), indent=2)

requirements.txt

@@ -0,0 +1,15 @@
+# Python >= 3.10. Install torch for your CUDA version first, e.g.:
+#   pip install torch torchvision --index-url https://download.pytorch.org/whl/cu124
+torch>=2.4
+torchvision
+transformers>=5.20      # needed for google/embeddinggemma-2 (model_type embedding_gemma2); tested with 5.20.1
+sentence-transformers>=6.1   # tested with 6.1.0
+huggingface_hub
+datasets
+pandas
+pyarrow
+numpy
+pyyaml
+pillow
+sentencepiece
+open_clip_torch         # only if image_encoder.backend=open_clip

results/RESULTS.md

@@ -0,0 +1,56 @@
+# Native-Bird x SigLIP image-text alignment: full run (2026-10-18, 1x H100)
+
+Test = 809 held-out images; gallery = every test caption in that language (eng 809, ibo 751, hau 769, yor 764). Chance R@1 ~0.13%.
+Best checkpoint by val mean R: **native_bird (frozen text)**, epoch 3 (val 60.46) vs unfreeze-top-2 epoch 2 (val 60.35): a near tie.
+
+| run | lang | i2t R@1 | i2t R@5 | i2t R@10 | t2i R@1 | t2i R@5 | t2i R@10 | mean R |
+|---|---|---|---|---|---|---|---|---|
+| untrained heads | eng | 0.37 | 1.11 | 1.48 | 0.12 | 0.87 | 1.73 | 0.95 |
+| untrained heads | ibo | 0.0 | 0.8 | 2.4 | 0.13 | 0.8 | 1.6 | 0.96 |
+| untrained heads | hau | 0.26 | 0.78 | 1.69 | 0.13 | 1.04 | 1.69 | 0.93 |
+| untrained heads | yor | 0.26 | 0.92 | 2.36 | 0.26 | 0.92 | 1.05 | 0.96 |
+| mCLIP zero-shot | eng | 58.34 | 84.67 | 92.21 | 57.85 | 83.31 | 89.99 | 77.73 |
+| mCLIP zero-shot | ibo | 6.13 | 14.38 | 20.91 | 4.26 | 11.98 | 16.11 | 12.29 |
+| mCLIP zero-shot | hau | 3.38 | 6.63 | 8.71 | 1.82 | 4.16 | 6.11 | 5.13 |
+| mCLIP zero-shot | yor | 2.36 | 4.84 | 6.41 | 1.31 | 2.75 | 4.84 | 3.75 |
+| Native-Bird frozen | eng | 46.48 | 77.38 | 88.5 | 44.99 | 76.64 | 87.02 | 70.17 |
+| Native-Bird frozen | ibo | 39.28 | 70.71 | 82.29 | 39.01 | 69.64 | 81.49 | 63.74 |
+| Native-Bird frozen | hau | 33.16 | 61.64 | 74.64 | 32.25 | 59.43 | 74.9 | 56.0 |
+| Native-Bird frozen | yor | 25.52 | 53.8 | 64.66 | 23.56 | 52.75 | 65.84 | 47.69 |
+| Native-Bird unfreeze top-2 | eng | 44.99 | 78.74 | 89.0 | 46.23 | 76.14 | 87.89 | 70.5 |
+| Native-Bird unfreeze top-2 | ibo | 37.95 | 70.57 | 83.49 | 39.68 | 70.17 | 83.36 | 64.2 |
+| Native-Bird unfreeze top-2 | hau | 31.73 | 60.47 | 74.9 | 30.3 | 60.73 | 74.38 | 55.42 |
+| Native-Bird unfreeze top-2 | yor | 23.3 | 52.88 | 64.66 | 24.87 | 53.27 | 64.4 | 47.23 |
+| frozen, test_unfiltered | eng | 46.48 | 77.38 | 88.5 | 44.99 | 76.64 | 87.02 | 70.17 |
+| frozen, test_unfiltered | ibo | 36.59 | 67.37 | 79.6 | 35.6 | 66.75 | 79.23 | 60.86 |
+| frozen, test_unfiltered | hau | 31.52 | 59.21 | 72.06 | 30.9 | 57.73 | 72.81 | 54.04 |
+| frozen, test_unfiltered | yor | 24.6 | 52.78 | 63.54 | 22.5 | 51.79 | 64.4 | 46.6 |
+| unfreeze-2, test_unfiltered | eng | 44.99 | 78.74 | 89.0 | 46.23 | 76.14 | 87.89 | 70.5 |
+| unfreeze-2, test_unfiltered | ibo | 35.72 | 67.86 | 81.09 | 36.96 | 67.12 | 80.22 | 61.5 |
+| unfreeze-2, test_unfiltered | hau | 29.91 | 58.71 | 73.05 | 28.31 | 58.22 | 72.44 | 53.44 |
+| unfreeze-2, test_unfiltered | yor | 22.74 | 50.31 | 62.42 | 23.98 | 52.41 | 62.79 | 45.77 |
+
+Smoke (CPU, 128 images, 1 epoch, mean R): untrained 4.6/3.5/2.8/5.3, mCLIP 92.3/22.5/7.7/6.8, trained Native-Bird 42.1/42.4/38.2/33.9 (eng/ibo/hau/yor). Not comparable to the 809-image gallery.
+
+## Igbo image search (best checkpoint, 809 test images)
+All 751 held-out Igbo captions as queries: hit@1 37.42, hit@5 68.44, hit@10 80.56, median rank 2. English: hit@1 44.99, hit@5 76.64, hit@10 87.02.
+
+| Igbo query | English | Igbo top-5 | top-5 overlap with English query |
+|---|---|---|---|
+| Nkịta na-agba ọsọ n'elu ahịhịa | A dog running on the grass | 697582336_601462e052.jpg, 2114739371_83aa8bdb0e.jpg, 1514957266_a19827c538.jpg, 240583223_e26e17ee96.jpg, 3358380484_b99b48f0c9.jpg | 3/5 |
+| Ụmụaka na-egwu egwu n'ime mmiri | Children playing in the water | 1095476286_87d4f8664e.jpg, 3041689520_c481bdb20e.jpg, 2951092164_4940b9a517.jpg, 3545427060_c16a8b7dfd.jpg, 3747543364_bf5b548527.jpg | 5/5 |
+| Otu nwoke na-agba ịnyịnya ígwè | A man riding a bicycle | 3649916507_b88a3d2082.jpg, 3391924827_53b31542ce.jpg, 3079073247_97e88e2eb7.jpg, 3665169936_01ae94c9fd.jpg, 2192411521_9c7e488c5e.jpg | 4/5 |
+| Otu nwanyị yi uwe na-acha uhie uhie | A woman wearing a red dress | 495055747_a75872762a.jpg, 613900608_2e49415772.jpg, 3534952095_975cca0056.jpg, 2695085448_a11833df95.jpg, 1821238649_2fda79d6d7.jpg | 4/5 |
+| Ndị mmadụ nọ n'elu ugwu | People on top of a mountain | 2176980976_7054c99621.jpg, 3340575518_137ce2695f.jpg, 1428578577_82864facae.jpg, 3229821595_77ace81c6b.jpg, 95783195_e1ba3f57ca.jpg | 4/5 |
+| Otu nwata nwoke na-awụli elu | A boy jumping in the air | 3348811097_0e09baa26f.jpg, 2061354254_faa5bd294b.jpg, 3229913073_e7857a5966.jpg, 3741827382_71e93298d0.jpg, 3396036947_0af6c3aab7.jpg | 3/5 |
+| Nkịta abụọ na-egwu egwu | Two dogs playing together | 3473534758_1ae3847781.jpg, 522486784_978021d537.jpg, 78984436_ad96eaa802.jpg, 255741044_1102982213.jpg, 3264678536_46601d25f0.jpg | 2/5 |
+| Ọtụtụ mmadụ nọ n'okporo ụzọ | A crowd of people on a street | 3638374272_444f5e0457.jpg, 256444892_efcb3bd824.jpg, 3225037367_a71fa86319.jpg, 3433470650_a8b1c27173.jpg, 1685990174_09c4fb7df8.jpg | 4/5 |
+
+| held-out Igbo caption | English reference | GT rank | in top-5 |
+|---|---|---|---|
+| Ndị ikom abụọ na-egwuri egwu na egwuregwu egwuregwu ọnụ . | Two men are playing in a sports game together . | 8 | no |
+| A isi awọ agba nkịta na-eje ije na mmiri ájá na a osimiri . | A grey colored dog walks in wet sand at a beach . | 14 | no |
+| Nkịta na-ejide Frisbee na ndị na-ekiri ya na ndabere . | A dog catching a Frisbee with onlookers in the background . | 3 | yes |
+| Ụmụ agbọghọ abụọ na-acha odo odo na-egwuri egwu ọnụ . | Two blond girls are playing together . | 1 | yes |
+| Nkịta na-awakpo nwoke yi ngwá ọrụ nchebe . | A dog attacking a man wearing protective gear . | 3 | yes |
+| Nkịta ojii na aja aja na-agba ọsọ gafee ahịhịa. | A black and brown dog running across the grass . | 4 | yes |

results/igbo_search.json

@@ -0,0 +1,588 @@
+{
+  "ckpt": "runs/native_bird/best.pt",
+  "gallery": 809,
+  "natural_queries": [
+    {
+      "ibo": "Nkịta na-agba ọsọ n'elu ahịhịa",
+      "eng": "A dog running on the grass",
+      "ibo_top5": [
+        {
+          "image_id": "697582336_601462e052.jpg",
+          "score": 0.6106
+        },
+        {
+          "image_id": "2114739371_83aa8bdb0e.jpg",
+          "score": 0.5652
+        },
+        {
+          "image_id": "1514957266_a19827c538.jpg",
+          "score": 0.4906
+        },
+        {
+          "image_id": "240583223_e26e17ee96.jpg",
+          "score": 0.4844
+        },
+        {
+          "image_id": "3358380484_b99b48f0c9.jpg",
+          "score": 0.4838
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "697582336_601462e052.jpg",
+          "score": 0.5392
+        },
+        {
+          "image_id": "240583223_e26e17ee96.jpg",
+          "score": 0.5132
+        },
+        {
+          "image_id": "3518675890_2f65e23ff9.jpg",
+          "score": 0.4845
+        },
+        {
+          "image_id": "2114739371_83aa8bdb0e.jpg",
+          "score": 0.4705
+        },
+        {
+          "image_id": "3435035138_af32890a4c.jpg",
+          "score": 0.4673
+        }
+      ],
+      "top5_overlap": 3
+    },
+    {
+      "ibo": "Ụmụaka na-egwu egwu n'ime mmiri",
+      "eng": "Children playing in the water",
+      "ibo_top5": [
+        {
+          "image_id": "1095476286_87d4f8664e.jpg",
+          "score": 0.5337
+        },
+        {
+          "image_id": "3041689520_c481bdb20e.jpg",
+          "score": 0.5287
+        },
+        {
+          "image_id": "2951092164_4940b9a517.jpg",
+          "score": 0.4685
+        },
+        {
+          "image_id": "3545427060_c16a8b7dfd.jpg",
+          "score": 0.4635
+        },
+        {
+          "image_id": "3747543364_bf5b548527.jpg",
+          "score": 0.4539
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "1095476286_87d4f8664e.jpg",
+          "score": 0.5283
+        },
+        {
+          "image_id": "3041689520_c481bdb20e.jpg",
+          "score": 0.504
+        },
+        {
+          "image_id": "2951092164_4940b9a517.jpg",
+          "score": 0.4709
+        },
+        {
+          "image_id": "3545427060_c16a8b7dfd.jpg",
+          "score": 0.4709
+        },
+        {
+          "image_id": "3747543364_bf5b548527.jpg",
+          "score": 0.4692
+        }
+      ],
+      "top5_overlap": 5
+    },
+    {
+      "ibo": "Otu nwoke na-agba ịnyịnya ígwè",
+      "eng": "A man riding a bicycle",
+      "ibo_top5": [
+        {
+          "image_id": "3649916507_b88a3d2082.jpg",
+          "score": 0.5071
+        },
+        {
+          "image_id": "3391924827_53b31542ce.jpg",
+          "score": 0.5004
+        },
+        {
+          "image_id": "3079073247_97e88e2eb7.jpg",
+          "score": 0.4955
+        },
+        {
+          "image_id": "3665169936_01ae94c9fd.jpg",
+          "score": 0.4919
+        },
+        {
+          "image_id": "2192411521_9c7e488c5e.jpg",
+          "score": 0.4795
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "2192411521_9c7e488c5e.jpg",
+          "score": 0.5054
+        },
+        {
+          "image_id": "3649916507_b88a3d2082.jpg",
+          "score": 0.5044
+        },
+        {
+          "image_id": "3391924827_53b31542ce.jpg",
+          "score": 0.4932
+        },
+        {
+          "image_id": "3665169936_01ae94c9fd.jpg",
+          "score": 0.4931
+        },
+        {
+          "image_id": "2308108566_2cba6bca53.jpg",
+          "score": 0.4526
+        }
+      ],
+      "top5_overlap": 4
+    },
+    {
+      "ibo": "Otu nwanyị yi uwe na-acha uhie uhie",
+      "eng": "A woman wearing a red dress",
+      "ibo_top5": [
+        {
+          "image_id": "495055747_a75872762a.jpg",
+          "score": 0.5312
+        },
+        {
+          "image_id": "613900608_2e49415772.jpg",
+          "score": 0.5263
+        },
+        {
+          "image_id": "3534952095_975cca0056.jpg",
+          "score": 0.4633
+        },
+        {
+          "image_id": "2695085448_a11833df95.jpg",
+          "score": 0.4492
+        },
+        {
+          "image_id": "1821238649_2fda79d6d7.jpg",
+          "score": 0.4281
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "495055747_a75872762a.jpg",
+          "score": 0.5347
+        },
+        {
+          "image_id": "613900608_2e49415772.jpg",
+          "score": 0.5008
+        },
+        {
+          "image_id": "3534952095_975cca0056.jpg",
+          "score": 0.4484
+        },
+        {
+          "image_id": "3183195653_11b66acb34.jpg",
+          "score": 0.4395
+        },
+        {
+          "image_id": "2695085448_a11833df95.jpg",
+          "score": 0.4367
+        }
+      ],
+      "top5_overlap": 4
+    },
+    {
+      "ibo": "Ndị mmadụ nọ n'elu ugwu",
+      "eng": "People on top of a mountain",
+      "ibo_top5": [
+        {
+          "image_id": "2176980976_7054c99621.jpg",
+          "score": 0.6118
+        },
+        {
+          "image_id": "3340575518_137ce2695f.jpg",
+          "score": 0.566
+        },
+        {
+          "image_id": "1428578577_82864facae.jpg",
+          "score": 0.5028
+        },
+        {
+          "image_id": "3229821595_77ace81c6b.jpg",
+          "score": 0.4851
+        },
+        {
+          "image_id": "95783195_e1ba3f57ca.jpg",
+          "score": 0.4772
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "2176980976_7054c99621.jpg",
+          "score": 0.6013
+        },
+        {
+          "image_id": "3340575518_137ce2695f.jpg",
+          "score": 0.5809
+        },
+        {
+          "image_id": "1428578577_82864facae.jpg",
+          "score": 0.4962
+        },
+        {
+          "image_id": "224273695_0b517bd0eb.jpg",
+          "score": 0.4638
+        },
+        {
+          "image_id": "3229821595_77ace81c6b.jpg",
+          "score": 0.4631
+        }
+      ],
+      "top5_overlap": 4
+    },
+    {
+      "ibo": "Otu nwata nwoke na-awụli elu",
+      "eng": "A boy jumping in the air",
+      "ibo_top5": [
+        {
+          "image_id": "3348811097_0e09baa26f.jpg",
+          "score": 0.5431
+        },
+        {
+          "image_id": "2061354254_faa5bd294b.jpg",
+          "score": 0.4968
+        },
+        {
+          "image_id": "3229913073_e7857a5966.jpg",
+          "score": 0.4844
+        },
+        {
+          "image_id": "3741827382_71e93298d0.jpg",
+          "score": 0.4737
+        },
+        {
+          "image_id": "3396036947_0af6c3aab7.jpg",
+          "score": 0.4445
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "3348811097_0e09baa26f.jpg",
+          "score": 0.5075
+        },
+        {
+          "image_id": "2461616306_3ee7ac1b4b.jpg",
+          "score": 0.492
+        },
+        {
+          "image_id": "210126070_0d43b300b9.jpg",
+          "score": 0.4805
+        },
+        {
+          "image_id": "3229913073_e7857a5966.jpg",
+          "score": 0.4607
+        },
+        {
+          "image_id": "2061354254_faa5bd294b.jpg",
+          "score": 0.4519
+        }
+      ],
+      "top5_overlap": 3
+    },
+    {
+      "ibo": "Nkịta abụọ na-egwu egwu",
+      "eng": "Two dogs playing together",
+      "ibo_top5": [
+        {
+          "image_id": "3473534758_1ae3847781.jpg",
+          "score": 0.5172
+        },
+        {
+          "image_id": "522486784_978021d537.jpg",
+          "score": 0.4586
+        },
+        {
+          "image_id": "78984436_ad96eaa802.jpg",
+          "score": 0.4584
+        },
+        {
+          "image_id": "255741044_1102982213.jpg",
+          "score": 0.4466
+        },
+        {
+          "image_id": "3264678536_46601d25f0.jpg",
+          "score": 0.4434
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "522486784_978021d537.jpg",
+          "score": 0.5093
+        },
+        {
+          "image_id": "3473534758_1ae3847781.jpg",
+          "score": 0.5004
+        },
+        {
+          "image_id": "649596742_5ba84ce946.jpg",
+          "score": 0.4764
+        },
+        {
+          "image_id": "2313230479_13f87c6bf3.jpg",
+          "score": 0.4346
+        },
+        {
+          "image_id": "872135364_8c1e47d163.jpg",
+          "score": 0.424
+        }
+      ],
+      "top5_overlap": 2
+    },
+    {
+      "ibo": "Ọtụtụ mmadụ nọ n'okporo ụzọ",
+      "eng": "A crowd of people on a street",
+      "ibo_top5": [
+        {
+          "image_id": "3638374272_444f5e0457.jpg",
+          "score": 0.4871
+        },
+        {
+          "image_id": "256444892_efcb3bd824.jpg",
+          "score": 0.486
+        },
+        {
+          "image_id": "3225037367_a71fa86319.jpg",
+          "score": 0.4641
+        },
+        {
+          "image_id": "3433470650_a8b1c27173.jpg",
+          "score": 0.4622
+        },
+        {
+          "image_id": "1685990174_09c4fb7df8.jpg",
+          "score": 0.4556
+        }
+      ],
+      "eng_top5": [
+        {
+          "image_id": "3225037367_a71fa86319.jpg",
+          "score": 0.5139
+        },
+        {
+          "image_id": "3433470650_a8b1c27173.jpg",
+          "score": 0.5117
+        },
+        {
+          "image_id": "1685990174_09c4fb7df8.jpg",
+          "score": 0.5078
+        },
+        {
+          "image_id": "3638374272_444f5e0457.jpg",
+          "score": 0.5008
+        },
+        {
+          "image_id": "3627676364_1dc9294ec5.jpg",
+          "score": 0.4698
+        }
+      ],
+      "top5_overlap": 4
+    }
+  ],
+  "heldout_caption_queries": [
+    {
+      "ibo": "Ndị ikom abụọ na-egwuri egwu na egwuregwu egwuregwu ọnụ .",
+      "eng_reference": "Two men are playing in a sports game together .",
+      "gt_image_id": "3649307685_60c1294d2a.jpg",
+      "gt_rank": 8,
+      "gt_in_top5": false,
+      "top5": [
+        {
+          "image_id": "3498997518_c2b16f0a0e.jpg",
+          "score": 0.4755
+        },
+        {
+          "image_id": "3613027188_1645ca1976.jpg",
+          "score": 0.465
+        },
+        {
+          "image_id": "3262075846_5695021d84.jpg",
+          "score": 0.4334
+        },
+        {
+          "image_id": "3506607642_40037b3fbf.jpg",
+          "score": 0.4289
+        },
+        {
+          "image_id": "269986132_91b71e8aaa.jpg",
+          "score": 0.4134
+        }
+      ]
+    },
+    {
+      "ibo": "A isi awọ agba nkịta na-eje ije na mmiri ájá na a osimiri .",
+      "eng_reference": "A grey colored dog walks in wet sand at a beach .",
+      "gt_image_id": "1526325728_74eb4153d8.jpg",
+      "gt_rank": 14,
+      "gt_in_top5": false,
+      "top5": [
+        {
+          "image_id": "3235746553_a40416c00e.jpg",
+          "score": 0.6606
+        },
+        {
+          "image_id": "2417745327_a2c2705043.jpg",
+          "score": 0.6486
+        },
+        {
+          "image_id": "140430106_2978fda105.jpg",
+          "score": 0.6157
+        },
+        {
+          "image_id": "3571675421_7e07ac07c5.jpg",
+          "score": 0.5766
+        },
+        {
+          "image_id": "2281006675_fde04e93dd.jpg",
+          "score": 0.5653
+        }
+      ]
+    },
+    {
+      "ibo": "Nkịta na-ejide Frisbee na ndị na-ekiri ya na ndabere .",
+      "eng_reference": "A dog catching a Frisbee with onlookers in the background .",
+      "gt_image_id": "2857609295_16aaa85293.jpg",
+      "gt_rank": 3,
+      "gt_in_top5": true,
+      "top5": [
+        {
+          "image_id": "436013859_793d870b6f.jpg",
+          "score": 0.6381
+        },
+        {
+          "image_id": "2942133798_e57c862a90.jpg",
+          "score": 0.5135
+        },
+        {
+          "image_id": "2857609295_16aaa85293.jpg",
+          "score": 0.5105
+        },
+        {
+          "image_id": "241346215_037e18403a.jpg",
+          "score": 0.4531
+        },
+        {
+          "image_id": "2165461920_1a4144eb2b.jpg",
+          "score": 0.4091
+        }
+      ]
+    },
+    {
+      "ibo": "Ụmụ agbọghọ abụọ na-acha odo odo na-egwuri egwu ọnụ .",
+      "eng_reference": "Two blond girls are playing together .",
+      "gt_image_id": "2599131872_65789d86d5.jpg",
+      "gt_rank": 1,
+      "gt_in_top5": true,
+      "top5": [
+        {
+          "image_id": "2599131872_65789d86d5.jpg",
+          "score": 0.515
+        },
+        {
+          "image_id": "3661072592_2e693cd5a0.jpg",
+          "score": 0.4701
+        },
+        {
+          "image_id": "1425485485_d7c97a5470.jpg",
+          "score": 0.4576
+        },
+        {
+          "image_id": "1424237335_b3be9920ba.jpg",
+          "score": 0.4476
+        },
+        {
+          "image_id": "1316247213_1d2c726dd5.jpg",
+          "score": 0.4279
+        }
+      ]
+    },
+    {
+      "ibo": "Nkịta na-awakpo nwoke yi ngwá ọrụ nchebe .",
+      "eng_reference": "A dog attacking a man wearing protective gear .",
+      "gt_image_id": "2629346153_155ba73ae9.jpg",
+      "gt_rank": 3,
+      "gt_in_top5": true,
+      "top5": [
+        {
+          "image_id": "1562392511_522a26063b.jpg",
+          "score": 0.4444
+        },
+        {
+          "image_id": "3442844140_15aa45e9b8.jpg",
+          "score": 0.4008
+        },
+        {
+          "image_id": "2629346153_155ba73ae9.jpg",
+          "score": 0.35
+        },
+        {
+          "image_id": "2437266971_b91a8f9a00.jpg",
+          "score": 0.3412
+        },
+        {
+          "image_id": "1835511273_790eaae6e6.jpg",
+          "score": 0.3388
+        }
+      ]
+    },
+    {
+      "ibo": "Nkịta ojii na aja aja na-agba ọsọ gafee ahịhịa.",
+      "eng_reference": "A black and brown dog running across the grass .",
+      "gt_image_id": "697582336_601462e052.jpg",
+      "gt_rank": 4,
+      "gt_in_top5": true,
+      "top5": [
+        {
+          "image_id": "435739506_2daf7f4887.jpg",
+          "score": 0.4904
+        },
+        {
+          "image_id": "333973142_abcd151002.jpg",
+          "score": 0.4688
+        },
+        {
+          "image_id": "472860064_a96a228796.jpg",
+          "score": 0.4584
+        },
+        {
+          "image_id": "697582336_601462e052.jpg",
+          "score": 0.4566
+        },
+        {
+          "image_id": "3025334206_76888792e5.jpg",
+          "score": 0.4396
+        }
+      ]
+    }
+  ],
+  "all_ibo_captions": {
+    "n": 751,
+    "hit@1": 37.42,
+    "hit@5": 68.44,
+    "hit@10": 80.56,
+    "median_rank": 2
+  },
+  "all_eng_captions": {
+    "n": 809,
+    "hit@1": 44.99,
+    "hit@5": 76.64,
+    "hit@10": 87.02,
+    "median_rank": 2
+  }
+}

results/text_encoder_crosslingual_test.json

@@ -0,0 +1,67 @@
+{
+  "split": "test_unfiltered",
+  "n": 809,
+  "seed": 0,
+  "models": {
+    "native_bird": {
+      "ibo": {
+        "cos_matched_mean": 0.7389,
+        "cos_unmatched_mean": 0.2169,
+        "margin": 0.522,
+        "acc@1_cap2eng": 96.66,
+        "acc@5_cap2eng": 99.26,
+        "acc@1_eng2cap": 95.06
+      },
+      "hau": {
+        "cos_matched_mean": 0.6745,
+        "cos_unmatched_mean": 0.1988,
+        "margin": 0.4757,
+        "acc@1_cap2eng": 87.89,
+        "acc@5_cap2eng": 97.16,
+        "acc@1_eng2cap": 85.66
+      },
+      "yor": {
+        "cos_matched_mean": 0.65,
+        "cos_unmatched_mean": 0.2092,
+        "margin": 0.4408,
+        "acc@1_cap2eng": 79.23,
+        "acc@5_cap2eng": 93.94,
+        "acc@1_eng2cap": 79.73
+      },
+      "avg_acc@1_cap2eng": 87.93,
+      "avg_margin": 0.4795,
+      "model_id": "Modularcomputing/Native-Bird",
+      "sec": 27.2
+    },
+    "labse": {
+      "ibo": {
+        "cos_matched_mean": 0.859,
+        "cos_unmatched_mean": 0.3248,
+        "margin": 0.5342,
+        "acc@1_cap2eng": 97.16,
+        "acc@5_cap2eng": 99.63,
+        "acc@1_eng2cap": 97.03
+      },
+      "hau": {
+        "cos_matched_mean": 0.7849,
+        "cos_unmatched_mean": 0.3233,
+        "margin": 0.4616,
+        "acc@1_cap2eng": 89.99,
+        "acc@5_cap2eng": 97.4,
+        "acc@1_eng2cap": 86.53
+      },
+      "yor": {
+        "cos_matched_mean": 0.7253,
+        "cos_unmatched_mean": 0.2938,
+        "margin": 0.4314,
+        "acc@1_cap2eng": 82.57,
+        "acc@5_cap2eng": 93.7,
+        "acc@1_eng2cap": 78.99
+      },
+      "avg_acc@1_cap2eng": 89.91,
+      "avg_margin": 0.4757,
+      "model_id": "sentence-transformers/LaBSE",
+      "sec": 12.8
+    }
+  }
+}

results/train_log_native_bird.json

@@ -0,0 +1,985 @@
+{
+  "history": [
+    {
+      "epoch": 1,
+      "loss": 3.0705511197447777,
+      "sec": 0.3,
+      "logit_scale": 15.06,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 41.9,
+          "t2i_R@1": 44.87,
+          "i2t_R@5": 75.77,
+          "t2i_R@5": 77.01,
+          "i2t_R@10": 86.16,
+          "t2i_R@10": 85.91,
+          "mean_R": 68.6
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 34.1,
+          "t2i_R@1": 37.74,
+          "i2t_R@5": 69.14,
+          "t2i_R@5": 67.79,
+          "i2t_R@10": 80.46,
+          "t2i_R@10": 81.67,
+          "mean_R": 61.82
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 28.31,
+          "t2i_R@1": 27.66,
+          "i2t_R@5": 56.1,
+          "t2i_R@5": 59.35,
+          "i2t_R@10": 71.3,
+          "t2i_R@10": 72.86,
+          "mean_R": 52.6
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 20.16,
+          "t2i_R@1": 20.94,
+          "i2t_R@5": 48.04,
+          "t2i_R@5": 46.6,
+          "i2t_R@10": 59.82,
+          "t2i_R@10": 61.13,
+          "mean_R": 42.78
+        }
+      },
+      "val_mean_R": 56.45
+    },
+    {
+      "epoch": 2,
+      "loss": 1.4599019723633926,
+      "sec": 0.2,
+      "logit_scale": 16.69,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 47.34,
+          "t2i_R@1": 48.83,
+          "i2t_R@5": 79.23,
+          "t2i_R@5": 78.37,
+          "i2t_R@10": 88.01,
+          "t2i_R@10": 87.27,
+          "mean_R": 71.51
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 38.54,
+          "t2i_R@1": 40.43,
+          "i2t_R@5": 71.97,
+          "t2i_R@5": 71.56,
+          "i2t_R@10": 82.75,
+          "t2i_R@10": 82.88,
+          "mean_R": 64.69
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.82,
+          "t2i_R@1": 33.25,
+          "i2t_R@5": 62.21,
+          "t2i_R@5": 62.34,
+          "i2t_R@10": 76.23,
+          "t2i_R@10": 76.75,
+          "mean_R": 57.1
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.08,
+          "t2i_R@1": 23.95,
+          "i2t_R@5": 53.01,
+          "t2i_R@5": 52.49,
+          "i2t_R@10": 65.71,
+          "t2i_R@10": 66.49,
+          "mean_R": 47.62
+        }
+      },
+      "val_mean_R": 60.23
+    },
+    {
+      "epoch": 3,
+      "loss": 1.124440540249149,
+      "sec": 0.2,
+      "logit_scale": 18.25,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 49.57,
+          "t2i_R@1": 47.47,
+          "i2t_R@5": 79.6,
+          "t2i_R@5": 78.37,
+          "i2t_R@10": 88.26,
+          "t2i_R@10": 87.39,
+          "mean_R": 71.78
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 40.84,
+          "t2i_R@1": 40.43,
+          "i2t_R@5": 71.7,
+          "t2i_R@5": 70.62,
+          "i2t_R@10": 83.02,
+          "t2i_R@10": 81.67,
+          "mean_R": 64.71
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 33.12,
+          "t2i_R@1": 31.3,
+          "i2t_R@5": 60.91,
+          "t2i_R@5": 62.99,
+          "i2t_R@10": 75.58,
+          "t2i_R@10": 76.23,
+          "mean_R": 56.69
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.35,
+          "t2i_R@1": 25.0,
+          "i2t_R@5": 54.06,
+          "t2i_R@5": 53.4,
+          "i2t_R@10": 68.32,
+          "t2i_R@10": 66.88,
+          "mean_R": 48.67
+        }
+      },
+      "val_mean_R": 60.46
+    },
+    {
+      "epoch": 4,
+      "loss": 0.9302061789979538,
+      "sec": 0.2,
+      "logit_scale": 19.77,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 46.23,
+          "t2i_R@1": 46.48,
+          "i2t_R@5": 78.49,
+          "t2i_R@5": 77.75,
+          "i2t_R@10": 87.52,
+          "t2i_R@10": 86.53,
+          "mean_R": 70.5
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 38.95,
+          "t2i_R@1": 40.43,
+          "i2t_R@5": 71.02,
+          "t2i_R@5": 70.89,
+          "i2t_R@10": 83.83,
+          "t2i_R@10": 82.88,
+          "mean_R": 64.67
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.56,
+          "t2i_R@1": 31.43,
+          "i2t_R@5": 61.04,
+          "t2i_R@5": 61.04,
+          "i2t_R@10": 74.68,
+          "t2i_R@10": 74.03,
+          "mean_R": 55.63
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.61,
+          "t2i_R@1": 23.04,
+          "i2t_R@5": 54.06,
+          "t2i_R@5": 52.75,
+          "i2t_R@10": 65.71,
+          "t2i_R@10": 64.79,
+          "mean_R": 47.49
+        }
+      },
+      "val_mean_R": 59.57
+    },
+    {
+      "epoch": 5,
+      "loss": 0.7919718983272711,
+      "sec": 0.2,
+      "logit_scale": 21.23,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.86,
+          "t2i_R@1": 46.11,
+          "i2t_R@5": 77.13,
+          "t2i_R@5": 77.5,
+          "i2t_R@10": 86.77,
+          "t2i_R@10": 86.28,
+          "mean_R": 69.94
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 38.27,
+          "t2i_R@1": 39.62,
+          "i2t_R@5": 68.6,
+          "t2i_R@5": 71.29,
+          "i2t_R@10": 81.27,
+          "t2i_R@10": 80.32,
+          "mean_R": 63.23
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 33.38,
+          "t2i_R@1": 30.52,
+          "i2t_R@5": 60.78,
+          "t2i_R@5": 62.47,
+          "i2t_R@10": 73.77,
+          "t2i_R@10": 73.51,
+          "mean_R": 55.74
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.35,
+          "t2i_R@1": 23.95,
+          "i2t_R@5": 52.88,
+          "t2i_R@5": 52.62,
+          "i2t_R@10": 64.4,
+          "t2i_R@10": 65.05,
+          "mean_R": 47.21
+        }
+      },
+      "val_mean_R": 59.03
+    },
+    {
+      "epoch": 6,
+      "loss": 0.6851718326409658,
+      "sec": 0.2,
+      "logit_scale": 22.6,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.98,
+          "t2i_R@1": 45.36,
+          "i2t_R@5": 77.01,
+          "t2i_R@5": 75.15,
+          "i2t_R@10": 85.29,
+          "t2i_R@10": 83.93,
+          "mean_R": 68.79
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.47,
+          "t2i_R@1": 37.33,
+          "i2t_R@5": 69.0,
+          "t2i_R@5": 68.46,
+          "i2t_R@10": 80.86,
+          "t2i_R@10": 80.46,
+          "mean_R": 62.26
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 30.65,
+          "t2i_R@1": 29.35,
+          "i2t_R@5": 60.0,
+          "t2i_R@5": 59.35,
+          "i2t_R@10": 72.47,
+          "t2i_R@10": 72.34,
+          "mean_R": 54.03
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.48,
+          "t2i_R@1": 23.43,
+          "i2t_R@5": 52.88,
+          "t2i_R@5": 51.31,
+          "i2t_R@10": 65.31,
+          "t2i_R@10": 63.48,
+          "mean_R": 46.82
+        }
+      },
+      "val_mean_R": 57.98
+    },
+    {
+      "epoch": 7,
+      "loss": 0.5936599510411421,
+      "sec": 0.2,
+      "logit_scale": 23.9,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.36,
+          "t2i_R@1": 43.88,
+          "i2t_R@5": 75.4,
+          "t2i_R@5": 73.55,
+          "i2t_R@10": 85.04,
+          "t2i_R@10": 84.43,
+          "mean_R": 67.94
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 36.52,
+          "t2i_R@1": 35.04,
+          "i2t_R@5": 68.46,
+          "t2i_R@5": 68.6,
+          "i2t_R@10": 80.73,
+          "t2i_R@10": 79.25,
+          "mean_R": 61.43
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.17,
+          "t2i_R@1": 30.39,
+          "i2t_R@5": 59.48,
+          "t2i_R@5": 60.26,
+          "i2t_R@10": 72.21,
+          "t2i_R@10": 71.69,
+          "mean_R": 54.2
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.61,
+          "t2i_R@1": 22.77,
+          "i2t_R@5": 50.52,
+          "t2i_R@5": 49.74,
+          "i2t_R@10": 64.4,
+          "t2i_R@10": 63.74,
+          "mean_R": 45.96
+        }
+      },
+      "val_mean_R": 57.38
+    },
+    {
+      "epoch": 8,
+      "loss": 0.5110944022114078,
+      "sec": 0.3,
+      "logit_scale": 25.08,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 43.26,
+          "t2i_R@1": 42.77,
+          "i2t_R@5": 73.67,
+          "t2i_R@5": 73.79,
+          "i2t_R@10": 84.43,
+          "t2i_R@10": 85.04,
+          "mean_R": 67.16
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 35.44,
+          "t2i_R@1": 36.39,
+          "i2t_R@5": 67.65,
+          "t2i_R@5": 65.9,
+          "i2t_R@10": 78.98,
+          "t2i_R@10": 78.84,
+          "mean_R": 60.53
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.56,
+          "t2i_R@1": 29.22,
+          "i2t_R@5": 58.31,
+          "t2i_R@5": 58.44,
+          "i2t_R@10": 73.12,
+          "t2i_R@10": 73.38,
+          "mean_R": 54.0
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.08,
+          "t2i_R@1": 24.35,
+          "i2t_R@5": 52.62,
+          "t2i_R@5": 51.31,
+          "i2t_R@10": 64.53,
+          "t2i_R@10": 62.3,
+          "mean_R": 46.53
+        }
+      },
+      "val_mean_R": 57.05
+    },
+    {
+      "epoch": 9,
+      "loss": 0.4497916651889682,
+      "sec": 0.2,
+      "logit_scale": 26.14,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 41.66,
+          "t2i_R@1": 41.78,
+          "i2t_R@5": 73.18,
+          "t2i_R@5": 73.42,
+          "i2t_R@10": 84.3,
+          "t2i_R@10": 83.93,
+          "mean_R": 66.38
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 34.5,
+          "t2i_R@1": 35.31,
+          "i2t_R@5": 66.85,
+          "t2i_R@5": 66.98,
+          "i2t_R@10": 79.11,
+          "t2i_R@10": 78.57,
+          "mean_R": 60.22
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 30.39,
+          "t2i_R@1": 29.35,
+          "i2t_R@5": 58.44,
+          "t2i_R@5": 58.96,
+          "i2t_R@10": 71.43,
+          "t2i_R@10": 70.39,
+          "mean_R": 53.16
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 23.69,
+          "t2i_R@1": 22.38,
+          "i2t_R@5": 51.57,
+          "t2i_R@5": 50.52,
+          "i2t_R@10": 61.26,
+          "t2i_R@10": 62.57,
+          "mean_R": 45.33
+        }
+      },
+      "val_mean_R": 56.27
+    },
+    {
+      "epoch": 10,
+      "loss": 0.3943763903031747,
+      "sec": 0.2,
+      "logit_scale": 27.06,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.91,
+          "t2i_R@1": 40.91,
+          "i2t_R@5": 73.55,
+          "t2i_R@5": 72.19,
+          "i2t_R@10": 83.19,
+          "t2i_R@10": 82.69,
+          "mean_R": 65.57
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.02,
+          "t2i_R@1": 33.02,
+          "i2t_R@5": 64.29,
+          "t2i_R@5": 63.07,
+          "i2t_R@10": 79.51,
+          "t2i_R@10": 77.36,
+          "mean_R": 58.38
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 29.74,
+          "t2i_R@1": 27.4,
+          "i2t_R@5": 58.05,
+          "t2i_R@5": 58.31,
+          "i2t_R@10": 71.82,
+          "t2i_R@10": 69.87,
+          "mean_R": 52.53
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.0,
+          "t2i_R@1": 23.69,
+          "i2t_R@5": 51.18,
+          "t2i_R@5": 50.79,
+          "i2t_R@10": 62.96,
+          "t2i_R@10": 61.78,
+          "mean_R": 45.9
+        }
+      },
+      "val_mean_R": 55.59
+    },
+    {
+      "epoch": 11,
+      "loss": 0.3472120384685695,
+      "sec": 0.2,
+      "logit_scale": 27.87,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 42.4,
+          "t2i_R@1": 40.54,
+          "i2t_R@5": 72.19,
+          "t2i_R@5": 72.81,
+          "i2t_R@10": 84.18,
+          "t2i_R@10": 82.69,
+          "mean_R": 65.8
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 35.31,
+          "t2i_R@1": 33.56,
+          "i2t_R@5": 64.96,
+          "t2i_R@5": 65.23,
+          "i2t_R@10": 77.22,
+          "t2i_R@10": 76.15,
+          "mean_R": 58.74
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 29.74,
+          "t2i_R@1": 28.7,
+          "i2t_R@5": 58.18,
+          "t2i_R@5": 58.18,
+          "i2t_R@10": 71.56,
+          "t2i_R@10": 70.13,
+          "mean_R": 52.75
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.65,
+          "t2i_R@1": 22.64,
+          "i2t_R@5": 50.79,
+          "t2i_R@5": 49.35,
+          "i2t_R@10": 63.22,
+          "t2i_R@10": 61.13,
+          "mean_R": 45.46
+        }
+      },
+      "val_mean_R": 55.69
+    },
+    {
+      "epoch": 12,
+      "loss": 0.30427658356105286,
+      "sec": 0.3,
+      "logit_scale": 28.54,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 42.89,
+          "t2i_R@1": 40.79,
+          "i2t_R@5": 72.31,
+          "t2i_R@5": 71.57,
+          "i2t_R@10": 83.19,
+          "t2i_R@10": 83.07,
+          "mean_R": 65.64
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.69,
+          "t2i_R@1": 32.75,
+          "i2t_R@5": 65.9,
+          "t2i_R@5": 64.96,
+          "i2t_R@10": 79.38,
+          "t2i_R@10": 78.3,
+          "mean_R": 59.16
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 29.61,
+          "t2i_R@1": 28.83,
+          "i2t_R@5": 57.92,
+          "t2i_R@5": 57.14,
+          "i2t_R@10": 70.0,
+          "t2i_R@10": 71.04,
+          "mean_R": 52.42
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.74,
+          "t2i_R@1": 21.99,
+          "i2t_R@5": 50.65,
+          "t2i_R@5": 47.77,
+          "i2t_R@10": 61.65,
+          "t2i_R@10": 61.26,
+          "mean_R": 44.68
+        }
+      },
+      "val_mean_R": 55.48
+    },
+    {
+      "epoch": 13,
+      "loss": 0.2706154971383512,
+      "sec": 0.2,
+      "logit_scale": 29.08,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.3,
+          "t2i_R@1": 39.8,
+          "i2t_R@5": 73.18,
+          "t2i_R@5": 71.69,
+          "i2t_R@10": 83.56,
+          "t2i_R@10": 83.19,
+          "mean_R": 65.29
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 34.37,
+          "t2i_R@1": 33.96,
+          "i2t_R@5": 64.15,
+          "t2i_R@5": 64.15,
+          "i2t_R@10": 77.49,
+          "t2i_R@10": 75.2,
+          "mean_R": 58.22
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 28.18,
+          "t2i_R@1": 28.18,
+          "i2t_R@5": 58.05,
+          "t2i_R@5": 58.7,
+          "i2t_R@10": 70.39,
+          "t2i_R@10": 69.74,
+          "mean_R": 52.21
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.39,
+          "t2i_R@1": 21.73,
+          "i2t_R@5": 50.92,
+          "t2i_R@5": 48.17,
+          "i2t_R@10": 62.17,
+          "t2i_R@10": 61.78,
+          "mean_R": 45.03
+        }
+      },
+      "val_mean_R": 55.19
+    },
+    {
+      "epoch": 14,
+      "loss": 0.24166721773023406,
+      "sec": 0.3,
+      "logit_scale": 29.51,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.67,
+          "t2i_R@1": 40.54,
+          "i2t_R@5": 72.44,
+          "t2i_R@5": 71.2,
+          "i2t_R@10": 82.69,
+          "t2i_R@10": 82.45,
+          "mean_R": 65.0
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.56,
+          "t2i_R@1": 34.37,
+          "i2t_R@5": 63.61,
+          "t2i_R@5": 64.96,
+          "i2t_R@10": 77.36,
+          "t2i_R@10": 75.74,
+          "mean_R": 58.27
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 29.74,
+          "t2i_R@1": 28.05,
+          "i2t_R@5": 57.92,
+          "t2i_R@5": 57.4,
+          "i2t_R@10": 70.0,
+          "t2i_R@10": 69.87,
+          "mean_R": 52.16
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.48,
+          "t2i_R@1": 21.86,
+          "i2t_R@5": 50.92,
+          "t2i_R@5": 47.25,
+          "i2t_R@10": 62.43,
+          "t2i_R@10": 60.73,
+          "mean_R": 44.61
+        }
+      },
+      "val_mean_R": 55.01
+    },
+    {
+      "epoch": 15,
+      "loss": 0.21996013897781572,
+      "sec": 0.2,
+      "logit_scale": 29.82,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 41.04,
+          "t2i_R@1": 40.54,
+          "i2t_R@5": 71.94,
+          "t2i_R@5": 71.82,
+          "i2t_R@10": 82.69,
+          "t2i_R@10": 82.32,
+          "mean_R": 65.06
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.69,
+          "t2i_R@1": 34.64,
+          "i2t_R@5": 64.69,
+          "t2i_R@5": 63.21,
+          "i2t_R@10": 77.49,
+          "t2i_R@10": 76.28,
+          "mean_R": 58.33
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 29.87,
+          "t2i_R@1": 28.05,
+          "i2t_R@5": 57.4,
+          "t2i_R@5": 57.53,
+          "i2t_R@10": 70.26,
+          "t2i_R@10": 69.61,
+          "mean_R": 52.12
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.87,
+          "t2i_R@1": 23.69,
+          "i2t_R@5": 50.13,
+          "t2i_R@5": 47.38,
+          "i2t_R@10": 62.43,
+          "t2i_R@10": 61.52,
+          "mean_R": 45.0
+        }
+      },
+      "val_mean_R": 55.13
+    },
+    {
+      "epoch": 16,
+      "loss": 0.20187547306219736,
+      "sec": 0.3,
+      "logit_scale": 30.03,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.54,
+          "t2i_R@1": 40.17,
+          "i2t_R@5": 71.45,
+          "t2i_R@5": 71.08,
+          "i2t_R@10": 83.19,
+          "t2i_R@10": 82.08,
+          "mean_R": 64.75
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 32.75,
+          "t2i_R@1": 33.69,
+          "i2t_R@5": 64.15,
+          "t2i_R@5": 64.02,
+          "i2t_R@10": 77.49,
+          "t2i_R@10": 75.74,
+          "mean_R": 57.97
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 29.09,
+          "t2i_R@1": 27.66,
+          "i2t_R@5": 58.31,
+          "t2i_R@5": 56.1,
+          "i2t_R@10": 70.13,
+          "t2i_R@10": 69.48,
+          "mean_R": 51.79
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.61,
+          "t2i_R@1": 22.38,
+          "i2t_R@5": 50.65,
+          "t2i_R@5": 47.12,
+          "i2t_R@10": 61.78,
+          "t2i_R@10": 60.73,
+          "mean_R": 44.54
+        }
+      },
+      "val_mean_R": 54.76
+    },
+    {
+      "epoch": 17,
+      "loss": 0.18857671103129783,
+      "sec": 0.3,
+      "logit_scale": 30.17,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.3,
+          "t2i_R@1": 39.68,
+          "i2t_R@5": 71.32,
+          "t2i_R@5": 70.58,
+          "i2t_R@10": 82.82,
+          "t2i_R@10": 81.71,
+          "mean_R": 64.4
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 34.1,
+          "t2i_R@1": 34.1,
+          "i2t_R@5": 63.48,
+          "t2i_R@5": 63.21,
+          "i2t_R@10": 76.95,
+          "t2i_R@10": 75.61,
+          "mean_R": 57.91
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 28.7,
+          "t2i_R@1": 28.18,
+          "i2t_R@5": 58.44,
+          "t2i_R@5": 57.4,
+          "i2t_R@10": 69.74,
+          "t2i_R@10": 68.83,
+          "mean_R": 51.88
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.21,
+          "t2i_R@1": 21.86,
+          "i2t_R@5": 50.0,
+          "t2i_R@5": 47.12,
+          "i2t_R@10": 61.52,
+          "t2i_R@10": 60.08,
+          "mean_R": 44.13
+        }
+      },
+      "val_mean_R": 54.58
+    },
+    {
+      "epoch": 18,
+      "loss": 0.18112863538165888,
+      "sec": 0.2,
+      "logit_scale": 30.24,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.67,
+          "t2i_R@1": 40.42,
+          "i2t_R@5": 71.08,
+          "t2i_R@5": 70.58,
+          "i2t_R@10": 83.07,
+          "t2i_R@10": 81.46,
+          "mean_R": 64.55
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.96,
+          "t2i_R@1": 34.77,
+          "i2t_R@5": 64.29,
+          "t2i_R@5": 64.02,
+          "i2t_R@10": 76.68,
+          "t2i_R@10": 75.34,
+          "mean_R": 58.18
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 28.7,
+          "t2i_R@1": 28.18,
+          "i2t_R@5": 57.92,
+          "t2i_R@5": 57.4,
+          "i2t_R@10": 70.0,
+          "t2i_R@10": 69.35,
+          "mean_R": 51.93
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.35,
+          "t2i_R@1": 22.25,
+          "i2t_R@5": 49.74,
+          "t2i_R@5": 47.38,
+          "i2t_R@10": 61.65,
+          "t2i_R@10": 60.21,
+          "mean_R": 44.26
+        }
+      },
+      "val_mean_R": 54.73
+    },
+    {
+      "epoch": 19,
+      "loss": 0.17619465151801705,
+      "sec": 0.3,
+      "logit_scale": 30.27,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.67,
+          "t2i_R@1": 40.79,
+          "i2t_R@5": 71.32,
+          "t2i_R@5": 70.46,
+          "i2t_R@10": 83.19,
+          "t2i_R@10": 81.71,
+          "mean_R": 64.69
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.56,
+          "t2i_R@1": 34.23,
+          "i2t_R@5": 64.56,
+          "t2i_R@5": 64.29,
+          "i2t_R@10": 77.22,
+          "t2i_R@10": 75.34,
+          "mean_R": 58.2
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 28.57,
+          "t2i_R@1": 28.05,
+          "i2t_R@5": 58.44,
+          "t2i_R@5": 57.01,
+          "i2t_R@10": 69.74,
+          "t2i_R@10": 69.22,
+          "mean_R": 51.84
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.08,
+          "t2i_R@1": 22.38,
+          "i2t_R@5": 49.87,
+          "t2i_R@5": 47.51,
+          "i2t_R@10": 61.52,
+          "t2i_R@10": 60.47,
+          "mean_R": 44.3
+        }
+      },
+      "val_mean_R": 54.76
+    },
+    {
+      "epoch": 20,
+      "loss": 0.17272056515018144,
+      "sec": 0.2,
+      "logit_scale": 30.27,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 40.79,
+          "t2i_R@1": 40.79,
+          "i2t_R@5": 71.45,
+          "t2i_R@5": 70.21,
+          "i2t_R@10": 83.19,
+          "t2i_R@10": 81.71,
+          "mean_R": 64.69
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 33.42,
+          "t2i_R@1": 34.23,
+          "i2t_R@5": 64.42,
+          "t2i_R@5": 64.15,
+          "i2t_R@10": 77.22,
+          "t2i_R@10": 75.47,
+          "mean_R": 58.15
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 28.7,
+          "t2i_R@1": 28.05,
+          "i2t_R@5": 58.31,
+          "t2i_R@5": 57.01,
+          "i2t_R@10": 69.87,
+          "t2i_R@10": 69.22,
+          "mean_R": 51.86
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.08,
+          "t2i_R@1": 22.38,
+          "i2t_R@5": 50.0,
+          "t2i_R@5": 47.38,
+          "i2t_R@10": 61.39,
+          "t2i_R@10": 60.47,
+          "mean_R": 44.28
+        }
+      },
+      "val_mean_R": 54.74
+    }
+  ],
+  "best_val_mean_R": 60.462500000000006
+}

results/train_log_native_bird_unfreeze2.json

@@ -0,0 +1,985 @@
+{
+  "history": [
+    {
+      "epoch": 1,
+      "loss": 2.0317564327269793,
+      "sec": 10.2,
+      "logit_scale": 16.4,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 42.77,
+          "t2i_R@1": 46.97,
+          "i2t_R@5": 76.02,
+          "t2i_R@5": 75.77,
+          "i2t_R@10": 87.76,
+          "t2i_R@10": 87.39,
+          "mean_R": 69.45
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 36.39,
+          "t2i_R@1": 36.93,
+          "i2t_R@5": 68.06,
+          "t2i_R@5": 69.14,
+          "i2t_R@10": 81.67,
+          "t2i_R@10": 82.88,
+          "mean_R": 62.51
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 27.27,
+          "t2i_R@1": 28.96,
+          "i2t_R@5": 58.7,
+          "t2i_R@5": 59.22,
+          "i2t_R@10": 72.86,
+          "t2i_R@10": 73.64,
+          "mean_R": 53.44
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 20.55,
+          "t2i_R@1": 21.73,
+          "i2t_R@5": 47.38,
+          "t2i_R@5": 48.43,
+          "i2t_R@10": 62.43,
+          "t2i_R@10": 62.96,
+          "mean_R": 43.91
+        }
+      },
+      "val_mean_R": 57.33
+    },
+    {
+      "epoch": 2,
+      "loss": 1.0098096902171771,
+      "sec": 10.0,
+      "logit_scale": 19.17,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 47.84,
+          "t2i_R@1": 47.22,
+          "i2t_R@5": 79.98,
+          "t2i_R@5": 79.23,
+          "i2t_R@10": 89.74,
+          "t2i_R@10": 88.38,
+          "mean_R": 72.06
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 39.49,
+          "t2i_R@1": 39.76,
+          "i2t_R@5": 72.78,
+          "t2i_R@5": 72.1,
+          "i2t_R@10": 83.56,
+          "t2i_R@10": 83.69,
+          "mean_R": 65.23
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.08,
+          "t2i_R@1": 30.26,
+          "i2t_R@5": 62.21,
+          "t2i_R@5": 63.12,
+          "i2t_R@10": 75.84,
+          "t2i_R@10": 75.06,
+          "mean_R": 56.43
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.13,
+          "t2i_R@1": 25.65,
+          "i2t_R@5": 53.4,
+          "t2i_R@5": 51.44,
+          "i2t_R@10": 65.05,
+          "t2i_R@10": 65.45,
+          "mean_R": 47.69
+        }
+      },
+      "val_mean_R": 60.35
+    },
+    {
+      "epoch": 3,
+      "loss": 0.783551151243349,
+      "sec": 9.7,
+      "logit_scale": 21.89,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.24,
+          "t2i_R@1": 47.47,
+          "i2t_R@5": 79.11,
+          "t2i_R@5": 77.5,
+          "i2t_R@10": 89.25,
+          "t2i_R@10": 87.52,
+          "mean_R": 71.02
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 36.52,
+          "t2i_R@1": 39.22,
+          "i2t_R@5": 71.16,
+          "t2i_R@5": 70.89,
+          "i2t_R@10": 83.02,
+          "t2i_R@10": 83.96,
+          "mean_R": 64.13
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 30.91,
+          "t2i_R@1": 31.17,
+          "i2t_R@5": 62.6,
+          "t2i_R@5": 62.99,
+          "i2t_R@10": 74.81,
+          "t2i_R@10": 76.49,
+          "mean_R": 56.5
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.48,
+          "t2i_R@1": 24.87,
+          "i2t_R@5": 53.27,
+          "t2i_R@5": 52.75,
+          "i2t_R@10": 67.93,
+          "t2i_R@10": 67.41,
+          "mean_R": 48.45
+        }
+      },
+      "val_mean_R": 60.02
+    },
+    {
+      "epoch": 4,
+      "loss": 0.6509892633184791,
+      "sec": 9.7,
+      "logit_scale": 24.4,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 47.34,
+          "t2i_R@1": 46.72,
+          "i2t_R@5": 78.99,
+          "t2i_R@5": 77.5,
+          "i2t_R@10": 87.52,
+          "t2i_R@10": 86.03,
+          "mean_R": 70.68
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.87,
+          "t2i_R@1": 39.22,
+          "i2t_R@5": 69.95,
+          "t2i_R@5": 70.35,
+          "i2t_R@10": 81.67,
+          "t2i_R@10": 82.61,
+          "mean_R": 63.61
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 30.52,
+          "t2i_R@1": 29.35,
+          "i2t_R@5": 61.69,
+          "t2i_R@5": 60.65,
+          "i2t_R@10": 73.51,
+          "t2i_R@10": 74.68,
+          "mean_R": 55.07
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.61,
+          "t2i_R@1": 25.26,
+          "i2t_R@5": 54.97,
+          "t2i_R@5": 54.45,
+          "i2t_R@10": 67.54,
+          "t2i_R@10": 65.58,
+          "mean_R": 48.73
+        }
+      },
+      "val_mean_R": 59.52
+    },
+    {
+      "epoch": 5,
+      "loss": 0.5509887392011782,
+      "sec": 9.7,
+      "logit_scale": 26.76,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 47.34,
+          "t2i_R@1": 45.86,
+          "i2t_R@5": 77.38,
+          "t2i_R@5": 78.62,
+          "i2t_R@10": 88.63,
+          "t2i_R@10": 86.9,
+          "mean_R": 70.79
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 38.54,
+          "t2i_R@1": 37.74,
+          "i2t_R@5": 69.81,
+          "t2i_R@5": 70.22,
+          "i2t_R@10": 81.0,
+          "t2i_R@10": 81.13,
+          "mean_R": 63.07
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.82,
+          "t2i_R@1": 31.69,
+          "i2t_R@5": 60.65,
+          "t2i_R@5": 63.38,
+          "i2t_R@10": 74.94,
+          "t2i_R@10": 76.23,
+          "mean_R": 56.45
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.74,
+          "t2i_R@1": 24.48,
+          "i2t_R@5": 52.88,
+          "t2i_R@5": 53.4,
+          "i2t_R@10": 65.84,
+          "t2i_R@10": 67.41,
+          "mean_R": 48.12
+        }
+      },
+      "val_mean_R": 59.61
+    },
+    {
+      "epoch": 6,
+      "loss": 0.475721658517917,
+      "sec": 9.6,
+      "logit_scale": 28.87,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.12,
+          "t2i_R@1": 45.36,
+          "i2t_R@5": 78.24,
+          "t2i_R@5": 77.01,
+          "i2t_R@10": 86.9,
+          "t2i_R@10": 86.4,
+          "mean_R": 69.84
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 39.35,
+          "t2i_R@1": 36.79,
+          "i2t_R@5": 69.27,
+          "t2i_R@5": 69.68,
+          "i2t_R@10": 81.67,
+          "t2i_R@10": 81.27,
+          "mean_R": 63.0
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.56,
+          "t2i_R@1": 29.87,
+          "i2t_R@5": 61.43,
+          "t2i_R@5": 64.03,
+          "i2t_R@10": 74.29,
+          "t2i_R@10": 75.45,
+          "mean_R": 56.1
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.52,
+          "t2i_R@1": 23.43,
+          "i2t_R@5": 55.1,
+          "t2i_R@5": 54.45,
+          "i2t_R@10": 66.1,
+          "t2i_R@10": 67.41,
+          "mean_R": 48.67
+        }
+      },
+      "val_mean_R": 59.4
+    },
+    {
+      "epoch": 7,
+      "loss": 0.4140009465627372,
+      "sec": 9.6,
+      "logit_scale": 30.88,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 46.23,
+          "t2i_R@1": 46.48,
+          "i2t_R@5": 77.87,
+          "t2i_R@5": 75.65,
+          "i2t_R@10": 87.89,
+          "t2i_R@10": 86.65,
+          "mean_R": 70.13
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 35.98,
+          "t2i_R@1": 37.6,
+          "i2t_R@5": 72.78,
+          "t2i_R@5": 71.16,
+          "i2t_R@10": 82.88,
+          "t2i_R@10": 81.54,
+          "mean_R": 63.66
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 30.65,
+          "t2i_R@1": 30.78,
+          "i2t_R@5": 63.25,
+          "t2i_R@5": 64.68,
+          "i2t_R@10": 74.94,
+          "t2i_R@10": 75.06,
+          "mean_R": 56.56
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 24.35,
+          "t2i_R@1": 23.69,
+          "i2t_R@5": 57.07,
+          "t2i_R@5": 56.15,
+          "i2t_R@10": 68.32,
+          "t2i_R@10": 68.32,
+          "mean_R": 49.65
+        }
+      },
+      "val_mean_R": 60.0
+    },
+    {
+      "epoch": 8,
+      "loss": 0.35873290702390176,
+      "sec": 9.7,
+      "logit_scale": 32.71,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 44.38,
+          "t2i_R@1": 44.75,
+          "i2t_R@5": 76.64,
+          "t2i_R@5": 74.41,
+          "i2t_R@10": 86.53,
+          "t2i_R@10": 85.78,
+          "mean_R": 68.75
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 38.68,
+          "t2i_R@1": 38.27,
+          "i2t_R@5": 70.75,
+          "t2i_R@5": 70.49,
+          "i2t_R@10": 81.54,
+          "t2i_R@10": 80.32,
+          "mean_R": 63.34
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 30.91,
+          "t2i_R@1": 30.78,
+          "i2t_R@5": 61.43,
+          "t2i_R@5": 62.21,
+          "i2t_R@10": 73.12,
+          "t2i_R@10": 74.68,
+          "mean_R": 55.52
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.26,
+          "t2i_R@1": 25.92,
+          "i2t_R@5": 54.45,
+          "t2i_R@5": 54.84,
+          "i2t_R@10": 66.75,
+          "t2i_R@10": 67.15,
+          "mean_R": 49.06
+        }
+      },
+      "val_mean_R": 59.17
+    },
+    {
+      "epoch": 9,
+      "loss": 0.3211887921206653,
+      "sec": 9.6,
+      "logit_scale": 34.37,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 44.5,
+          "t2i_R@1": 44.0,
+          "i2t_R@5": 76.89,
+          "t2i_R@5": 76.14,
+          "i2t_R@10": 86.03,
+          "t2i_R@10": 86.4,
+          "mean_R": 68.99
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.74,
+          "t2i_R@1": 38.14,
+          "i2t_R@5": 70.75,
+          "t2i_R@5": 70.75,
+          "i2t_R@10": 81.81,
+          "t2i_R@10": 81.81,
+          "mean_R": 63.5
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.21,
+          "t2i_R@1": 29.48,
+          "i2t_R@5": 61.17,
+          "t2i_R@5": 63.51,
+          "i2t_R@10": 75.32,
+          "t2i_R@10": 74.94,
+          "mean_R": 56.1
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 25.92,
+          "t2i_R@1": 25.26,
+          "i2t_R@5": 56.15,
+          "t2i_R@5": 54.06,
+          "i2t_R@10": 67.41,
+          "t2i_R@10": 67.41,
+          "mean_R": 49.37
+        }
+      },
+      "val_mean_R": 59.49
+    },
+    {
+      "epoch": 10,
+      "loss": 0.28735482529737055,
+      "sec": 9.9,
+      "logit_scale": 35.84,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.49,
+          "t2i_R@1": 45.36,
+          "i2t_R@5": 76.27,
+          "t2i_R@5": 76.76,
+          "i2t_R@10": 85.91,
+          "t2i_R@10": 85.91,
+          "mean_R": 69.28
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 36.66,
+          "t2i_R@1": 37.33,
+          "i2t_R@5": 70.62,
+          "t2i_R@5": 69.95,
+          "i2t_R@10": 81.94,
+          "t2i_R@10": 81.4,
+          "mean_R": 62.98
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.95,
+          "t2i_R@1": 30.78,
+          "i2t_R@5": 62.73,
+          "t2i_R@5": 63.51,
+          "i2t_R@10": 73.64,
+          "t2i_R@10": 74.29,
+          "mean_R": 56.15
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.31,
+          "t2i_R@1": 25.0,
+          "i2t_R@5": 54.06,
+          "t2i_R@5": 52.36,
+          "i2t_R@10": 65.97,
+          "t2i_R@10": 66.62,
+          "mean_R": 48.39
+        }
+      },
+      "val_mean_R": 59.2
+    },
+    {
+      "epoch": 11,
+      "loss": 0.2505949118640274,
+      "sec": 9.7,
+      "logit_scale": 37.15,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 44.75,
+          "t2i_R@1": 44.0,
+          "i2t_R@5": 75.9,
+          "t2i_R@5": 75.15,
+          "i2t_R@10": 85.04,
+          "t2i_R@10": 85.54,
+          "mean_R": 68.4
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 35.04,
+          "t2i_R@1": 35.31,
+          "i2t_R@5": 69.41,
+          "t2i_R@5": 68.33,
+          "i2t_R@10": 81.81,
+          "t2i_R@10": 81.27,
+          "mean_R": 61.86
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.04,
+          "t2i_R@1": 29.09,
+          "i2t_R@5": 60.13,
+          "t2i_R@5": 60.78,
+          "i2t_R@10": 73.38,
+          "t2i_R@10": 73.25,
+          "mean_R": 54.61
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.57,
+          "t2i_R@1": 24.48,
+          "i2t_R@5": 52.23,
+          "t2i_R@5": 51.83,
+          "i2t_R@10": 65.71,
+          "t2i_R@10": 66.62,
+          "mean_R": 47.91
+        }
+      },
+      "val_mean_R": 58.2
+    },
+    {
+      "epoch": 12,
+      "loss": 0.22653604112565517,
+      "sec": 9.7,
+      "logit_scale": 38.25,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 43.51,
+          "t2i_R@1": 43.14,
+          "i2t_R@5": 75.4,
+          "t2i_R@5": 74.66,
+          "i2t_R@10": 86.16,
+          "t2i_R@10": 84.43,
+          "mean_R": 67.88
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 34.5,
+          "t2i_R@1": 36.79,
+          "i2t_R@5": 69.95,
+          "t2i_R@5": 69.27,
+          "i2t_R@10": 80.32,
+          "t2i_R@10": 80.19,
+          "mean_R": 61.84
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.73,
+          "t2i_R@1": 30.39,
+          "i2t_R@5": 60.26,
+          "t2i_R@5": 61.43,
+          "i2t_R@10": 72.73,
+          "t2i_R@10": 73.64,
+          "mean_R": 55.2
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 28.14,
+          "t2i_R@1": 24.21,
+          "i2t_R@5": 53.14,
+          "t2i_R@5": 52.88,
+          "i2t_R@10": 65.45,
+          "t2i_R@10": 67.41,
+          "mean_R": 48.54
+        }
+      },
+      "val_mean_R": 58.37
+    },
+    {
+      "epoch": 13,
+      "loss": 0.20553195266984403,
+      "sec": 9.6,
+      "logit_scale": 39.17,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 45.24,
+          "t2i_R@1": 44.0,
+          "i2t_R@5": 74.41,
+          "t2i_R@5": 73.79,
+          "i2t_R@10": 84.8,
+          "t2i_R@10": 84.3,
+          "mean_R": 67.76
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 35.98,
+          "t2i_R@1": 36.25,
+          "i2t_R@5": 69.68,
+          "t2i_R@5": 69.14,
+          "i2t_R@10": 81.94,
+          "t2i_R@10": 80.73,
+          "mean_R": 62.29
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.99,
+          "t2i_R@1": 29.74,
+          "i2t_R@5": 61.3,
+          "t2i_R@5": 60.65,
+          "i2t_R@10": 74.16,
+          "t2i_R@10": 74.42,
+          "mean_R": 55.54
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 27.09,
+          "t2i_R@1": 25.52,
+          "i2t_R@5": 54.32,
+          "t2i_R@5": 52.88,
+          "i2t_R@10": 66.23,
+          "t2i_R@10": 66.88,
+          "mean_R": 48.82
+        }
+      },
+      "val_mean_R": 58.6
+    },
+    {
+      "epoch": 14,
+      "loss": 0.18276729714125395,
+      "sec": 9.6,
+      "logit_scale": 39.98,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 43.14,
+          "t2i_R@1": 42.15,
+          "i2t_R@5": 73.92,
+          "t2i_R@5": 74.66,
+          "i2t_R@10": 84.43,
+          "t2i_R@10": 84.05,
+          "mean_R": 67.06
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.47,
+          "t2i_R@1": 36.93,
+          "i2t_R@5": 69.27,
+          "t2i_R@5": 69.54,
+          "i2t_R@10": 81.67,
+          "t2i_R@10": 80.32,
+          "mean_R": 62.53
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.69,
+          "t2i_R@1": 29.22,
+          "i2t_R@5": 60.91,
+          "t2i_R@5": 61.95,
+          "i2t_R@10": 73.51,
+          "t2i_R@10": 74.03,
+          "mean_R": 55.22
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.7,
+          "t2i_R@1": 25.26,
+          "i2t_R@5": 55.1,
+          "t2i_R@5": 52.09,
+          "i2t_R@10": 65.45,
+          "t2i_R@10": 66.1,
+          "mean_R": 48.45
+        }
+      },
+      "val_mean_R": 58.31
+    },
+    {
+      "epoch": 15,
+      "loss": 0.16345966514199972,
+      "sec": 9.5,
+      "logit_scale": 40.63,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 43.14,
+          "t2i_R@1": 43.39,
+          "i2t_R@5": 75.28,
+          "t2i_R@5": 74.41,
+          "i2t_R@10": 85.54,
+          "t2i_R@10": 84.43,
+          "mean_R": 67.7
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 36.52,
+          "t2i_R@1": 36.25,
+          "i2t_R@5": 70.08,
+          "t2i_R@5": 69.14,
+          "i2t_R@10": 82.08,
+          "t2i_R@10": 80.59,
+          "mean_R": 62.44
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.34,
+          "t2i_R@1": 29.22,
+          "i2t_R@5": 63.25,
+          "t2i_R@5": 61.95,
+          "i2t_R@10": 73.51,
+          "t2i_R@10": 74.55,
+          "mean_R": 55.8
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.05,
+          "t2i_R@1": 25.39,
+          "i2t_R@5": 55.5,
+          "t2i_R@5": 53.14,
+          "i2t_R@10": 66.88,
+          "t2i_R@10": 65.84,
+          "mean_R": 48.8
+        }
+      },
+      "val_mean_R": 58.69
+    },
+    {
+      "epoch": 16,
+      "loss": 0.14981218008324504,
+      "sec": 9.6,
+      "logit_scale": 41.1,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 42.52,
+          "t2i_R@1": 44.13,
+          "i2t_R@5": 75.15,
+          "t2i_R@5": 75.15,
+          "i2t_R@10": 84.3,
+          "t2i_R@10": 84.43,
+          "mean_R": 67.61
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.33,
+          "t2i_R@1": 36.52,
+          "i2t_R@5": 69.95,
+          "t2i_R@5": 68.87,
+          "i2t_R@10": 81.4,
+          "t2i_R@10": 79.78,
+          "mean_R": 62.31
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.08,
+          "t2i_R@1": 30.26,
+          "i2t_R@5": 61.56,
+          "t2i_R@5": 60.78,
+          "i2t_R@10": 74.55,
+          "t2i_R@10": 74.94,
+          "mean_R": 55.7
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 27.75,
+          "t2i_R@1": 25.92,
+          "i2t_R@5": 56.02,
+          "t2i_R@5": 53.4,
+          "i2t_R@10": 66.75,
+          "t2i_R@10": 66.62,
+          "mean_R": 49.41
+        }
+      },
+      "val_mean_R": 58.76
+    },
+    {
+      "epoch": 17,
+      "loss": 0.14285055936003724,
+      "sec": 9.8,
+      "logit_scale": 41.4,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 43.14,
+          "t2i_R@1": 42.65,
+          "i2t_R@5": 75.4,
+          "t2i_R@5": 75.53,
+          "i2t_R@10": 84.8,
+          "t2i_R@10": 84.18,
+          "mean_R": 67.62
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.2,
+          "t2i_R@1": 37.6,
+          "i2t_R@5": 69.68,
+          "t2i_R@5": 68.19,
+          "i2t_R@10": 81.54,
+          "t2i_R@10": 80.19,
+          "mean_R": 62.4
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.47,
+          "t2i_R@1": 30.78,
+          "i2t_R@5": 62.08,
+          "t2i_R@5": 61.3,
+          "i2t_R@10": 74.55,
+          "t2i_R@10": 74.55,
+          "mean_R": 55.96
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.57,
+          "t2i_R@1": 25.26,
+          "i2t_R@5": 55.63,
+          "t2i_R@5": 54.06,
+          "i2t_R@10": 67.28,
+          "t2i_R@10": 67.28,
+          "mean_R": 49.35
+        }
+      },
+      "val_mean_R": 58.83
+    },
+    {
+      "epoch": 18,
+      "loss": 0.13162445152799288,
+      "sec": 10.0,
+      "logit_scale": 41.57,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 42.03,
+          "t2i_R@1": 43.39,
+          "i2t_R@5": 74.91,
+          "t2i_R@5": 75.28,
+          "i2t_R@10": 85.17,
+          "t2i_R@10": 84.67,
+          "mean_R": 67.58
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 36.52,
+          "t2i_R@1": 36.66,
+          "i2t_R@5": 69.14,
+          "t2i_R@5": 69.54,
+          "i2t_R@10": 80.86,
+          "t2i_R@10": 79.92,
+          "mean_R": 62.11
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.6,
+          "t2i_R@1": 30.0,
+          "i2t_R@5": 61.95,
+          "t2i_R@5": 61.17,
+          "i2t_R@10": 74.29,
+          "t2i_R@10": 74.42,
+          "mean_R": 55.74
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.57,
+          "t2i_R@1": 25.39,
+          "i2t_R@5": 55.76,
+          "t2i_R@5": 53.8,
+          "i2t_R@10": 67.28,
+          "t2i_R@10": 66.49,
+          "mean_R": 49.22
+        }
+      },
+      "val_mean_R": 58.66
+    },
+    {
+      "epoch": 19,
+      "loss": 0.13018665796456239,
+      "sec": 9.8,
+      "logit_scale": 41.64,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 43.26,
+          "t2i_R@1": 43.51,
+          "i2t_R@5": 75.4,
+          "t2i_R@5": 75.15,
+          "i2t_R@10": 85.54,
+          "t2i_R@10": 85.04,
+          "mean_R": 67.98
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.47,
+          "t2i_R@1": 37.2,
+          "i2t_R@5": 70.08,
+          "t2i_R@5": 69.41,
+          "i2t_R@10": 81.54,
+          "t2i_R@10": 79.92,
+          "mean_R": 62.6
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 31.95,
+          "t2i_R@1": 30.39,
+          "i2t_R@5": 61.82,
+          "t2i_R@5": 61.3,
+          "i2t_R@10": 74.29,
+          "t2i_R@10": 74.42,
+          "mean_R": 55.7
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 27.09,
+          "t2i_R@1": 25.65,
+          "i2t_R@5": 56.15,
+          "t2i_R@5": 53.93,
+          "i2t_R@10": 67.28,
+          "t2i_R@10": 67.02,
+          "mean_R": 49.52
+        }
+      },
+      "val_mean_R": 58.95
+    },
+    {
+      "epoch": 20,
+      "loss": 0.12976117172123244,
+      "sec": 9.8,
+      "logit_scale": 41.65,
+      "val": {
+        "eng": {
+          "n": 809,
+          "i2t_R@1": 42.89,
+          "t2i_R@1": 43.51,
+          "i2t_R@5": 75.65,
+          "t2i_R@5": 75.15,
+          "i2t_R@10": 85.54,
+          "t2i_R@10": 85.04,
+          "mean_R": 67.96
+        },
+        "ibo": {
+          "n": 742,
+          "i2t_R@1": 37.06,
+          "t2i_R@1": 37.2,
+          "i2t_R@5": 69.95,
+          "t2i_R@5": 69.54,
+          "i2t_R@10": 81.67,
+          "t2i_R@10": 80.19,
+          "mean_R": 62.6
+        },
+        "hau": {
+          "n": 770,
+          "i2t_R@1": 32.47,
+          "t2i_R@1": 30.13,
+          "i2t_R@5": 61.95,
+          "t2i_R@5": 61.04,
+          "i2t_R@10": 74.42,
+          "t2i_R@10": 74.42,
+          "mean_R": 55.74
+        },
+        "yor": {
+          "n": 764,
+          "i2t_R@1": 26.96,
+          "t2i_R@1": 25.39,
+          "i2t_R@5": 55.89,
+          "t2i_R@5": 54.06,
+          "i2t_R@10": 67.67,
+          "t2i_R@10": 67.02,
+          "mean_R": 49.5
+        }
+      },
+      "val_mean_R": 58.95
+    }
+  ],
+  "best_val_mean_R": 60.352500000000006
+}

search.py

@@ -0,0 +1,83 @@
+"""Text -> image search with a trained Native-Bird x SigLIP alignment checkpoint.
+
+python search.py --ckpt runs/native_bird/best.pt "Nkịta na-agba ọsọ n'ahịhịa" "A dog running on grass" -k 5
+python search.py --ckpt runs/native_bird/best.pt --split test --json out.json "query 1" "query 2"
+
+The gallery is every image of --split (default test, 809 images). Image features come from the same frozen
+image encoder the checkpoint was trained with (cached in data/features/, computed on first use; needs the
+Flickr8k images at data.image_dir). Works on CPU or GPU.
+"""
+import argparse, json, os, torch
+from nbmm.common import (seed_all, pick_device, load_split, image_features, text_features, load_sentence_model,
+                         text_arm, TrainableText, AlignModel, rel, _set)
+
+
+class Searcher:
+    def __init__(self, ckpt, split="test", device="auto", overrides=()):
+        ck = torch.load(rel(ckpt), map_location="cpu", weights_only=False)
+        self.cfg = cfg = ck["cfg"]
+        for o in overrides:
+            k, v = o.split("=", 1); _set(cfg, k, v)
+        seed_all(cfg["seed"])
+        self.dev = pick_device(device)
+        self.df = load_split(cfg, split)
+        self.ids = self.df.image_id.tolist()
+        I = image_features(cfg, self.df, split, self.dev)
+        self.model = AlignModel(ck["d_img"], ck["d_txt"], cfg).to(self.dev).eval()
+        self.model.load_state_dict(ck["model"])
+        self.unfreeze = cfg["text_encoder"]["unfreeze_last_n_layers"] > 0
+        if self.unfreeze:
+            self.txt = TrainableText(cfg, self.dev).to(self.dev).eval()
+            self.txt.load_state_dict(ck["text_trainable"], strict=False)
+        else:
+            self.arm = text_arm(cfg)
+            self.st = load_sentence_model(self.arm, self.dev)
+        with torch.no_grad():
+            self.zi = self.model.img(I.to(self.dev))
+
+    @torch.no_grad()
+    def embed_text(self, texts):
+        if self.unfreeze:
+            e = self.txt(texts)
+        else:
+            e = self.st.encode([self.arm.get("prompt", "") + t for t in texts], convert_to_tensor=True,
+                               normalize_embeddings=True).float().to(self.dev)
+        return self.model.txt(e)
+
+    @torch.no_grad()
+    def search(self, texts, k=5):
+        sims = self.embed_text(texts) @ self.zi.t()
+        out = []
+        for t, row in zip(texts, sims):
+            order = row.argsort(descending=True)
+            top = order[:k].tolist()
+            out.append({"query": t, "top": [{"image_id": self.ids[j], "score": round(row[j].item(), 4)} for j in top],
+                        "_order": order})
+        return out
+
+    def rank_of(self, res, image_id):
+        j = self.ids.index(image_id)
+        return int((res["_order"] == j).nonzero()[0].item()) + 1  # 1-based
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--ckpt", required=True)
+    ap.add_argument("--split", default="test")
+    ap.add_argument("-k", type=int, default=5)
+    ap.add_argument("--json", default=None)
+    ap.add_argument("queries", nargs="+")
+    a = ap.parse_args()
+    s = Searcher(a.ckpt, a.split)
+    res = s.search(a.queries, a.k)
+    for r in res:
+        r.pop("_order")
+        print(r["query"])
+        for i, h in enumerate(r["top"], 1):
+            print(f"  {i}. {h['image_id']}  {h['score']}")
+    if a.json:
+        json.dump(res, open(a.json, "w"), indent=2, ensure_ascii=False)
+
+
+if __name__ == "__main__":
+    main()