image2wolof / README.md
milkymap's picture
image2wolof: SigLIP2-base + 6-layer decoder trained on Flickr30k Wolof captions
033e5de verified
|
Raw History Blame Contribute Delete
3.35 kB
---
license: apache-2.0
language:
- wo
library_name: pytorch
pipeline_tag: image-to-text
tags:
- image-captioning
- wolof
- low-resource
- siglip2
datasets:
- milkymap/flickr30k-wolof
base_model: google/siglip2-base-patch16-224
metrics:
- bleu
- chrf
---
# image2wolof
Image captioning in Wolof. A frozen [SigLIP2-base](https://huggingface.co/google/siglip2-base-patch16-224) encoder
(patch tokens, 196 x 768, pinned revision `75de2d55`) feeds a six-layer transformer decoder (d 512, 8 heads, ff 2048,
SentencePiece unigram vocabulary of 4,000 tied to the output layer, 27.9 M trainable parameters) trained from scratch.
Code, training pipeline and paper: [github.com/milkymap/image2wolof](https://github.com/milkymap/image2wolof)
([PDF](https://github.com/milkymap/image2wolof/blob/main/paper/image2wolof.pdf)).
## Files
| file | |
|---|---|
| `best.pt` | decoder weights, decoder config, vision backbone spec (model id, revision, preprocessing), tokenizer sha256 |
| `tokenizer.model` | SentencePiece unigram tokenizer (vocab 4,000), must sit next to `best.pt` |
| `config.json` | the same configuration in readable form |
## Usage
```bash
git clone https://github.com/milkymap/image2wolof && cd image2wolof && uv sync
hf download milkymap/image2wolof --local-dir runs/flickr30k
uv run image2wolof serve runs/flickr30k/best.pt --port 8080 # upload page + POST /caption
```
```python
from PIL import Image
from image2wolof.services.captioner import Captioner
captioner = Captioner("runs/flickr30k/best.pt", device="cuda")
print(captioner([Image.open("photo.jpg")]))
```
The SigLIP2 encoder is downloaded from the Hub at the pinned revision on first use.
## Training data
[`milkymap/flickr30k-wolof`](https://huggingface.co/datasets/milkymap/flickr30k-wolof): the 29,703 Flickr30k training images,
each with five Wolof captions obtained by machine translation of the five English captions, guided by a glossary of terms and
rules. Every caption is a training target (148,515 image-caption pairs). AdamW 3e-4, weight decay 0.01, warmup then cosine decay,
bf16, label smoothing 0.1, batch 128, early stopping on the five-caption validation loss: best epoch 6 of 10.
## Results
Greedy decoding, five references per image, sacreBLEU 2.6.0.
| split | images | five-caption loss | BLEU | chrF |
|---|---:|---:|---:|---:|
| validation | 1,014 | 2.809 | 40.01 | 46.07 |
| test | 995 | 2.792 | 41.28 | 46.19 |
Features and decoding run in bfloat16: on another GPU the loss is identical to three decimals and BLEU / chrF move by a few tenths.
## Limitations
- The references are machine translations, not human Wolof: scores measure agreement with these silver references, not
fluency judged by native speakers, and there is no large-scale human evaluation.
- Flickr30k shows mostly North American and European scenes; the model has seen almost no photographs from Senegal and may
describe local objects, clothing or places poorly.
- Known errors include confused objects, swapped roles between people, invented details and occasional repetitions.
## Citation
```bibtex
@misc{image2wolof2026,
title = {image2wolof: Wolof Image Captioning from Machine-Translated Captions with a Frozen SigLIP 2 Encoder},
author = {Ibrahima Ba},
year = {2026},
month = {oct},
url = {https://github.com/milkymap/image2wolof}
}
```