File size: 3,348 Bytes
033e5de
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
---
license: apache-2.0
language:
- wo
library_name: pytorch
pipeline_tag: image-to-text
tags:
- image-captioning
- wolof
- low-resource
- siglip2
datasets:
- milkymap/flickr30k-wolof
base_model: google/siglip2-base-patch16-224
metrics:
- bleu
- chrf
---

# image2wolof

Image captioning in Wolof. A frozen [SigLIP2-base](https://huggingface.co/google/siglip2-base-patch16-224) encoder
(patch tokens, 196 x 768, pinned revision `75de2d55`) feeds a six-layer transformer decoder (d 512, 8 heads, ff 2048,
SentencePiece unigram vocabulary of 4,000 tied to the output layer, 27.9 M trainable parameters) trained from scratch.

Code, training pipeline and paper: [github.com/milkymap/image2wolof](https://github.com/milkymap/image2wolof)
([PDF](https://github.com/milkymap/image2wolof/blob/main/paper/image2wolof.pdf)).

## Files

| file | |
|---|---|
| `best.pt` | decoder weights, decoder config, vision backbone spec (model id, revision, preprocessing), tokenizer sha256 |
| `tokenizer.model` | SentencePiece unigram tokenizer (vocab 4,000), must sit next to `best.pt` |
| `config.json` | the same configuration in readable form |

## Usage

```bash
git clone https://github.com/milkymap/image2wolof && cd image2wolof && uv sync
hf download milkymap/image2wolof --local-dir runs/flickr30k
uv run image2wolof serve runs/flickr30k/best.pt --port 8080      # upload page + POST /caption
```

```python
from PIL import Image
from image2wolof.services.captioner import Captioner

captioner = Captioner("runs/flickr30k/best.pt", device="cuda")
print(captioner([Image.open("photo.jpg")]))
```

The SigLIP2 encoder is downloaded from the Hub at the pinned revision on first use.

## Training data

[`milkymap/flickr30k-wolof`](https://huggingface.co/datasets/milkymap/flickr30k-wolof): the 29,703 Flickr30k training images,
each with five Wolof captions obtained by machine translation of the five English captions, guided by a glossary of terms and
rules. Every caption is a training target (148,515 image-caption pairs). AdamW 3e-4, weight decay 0.01, warmup then cosine decay,
bf16, label smoothing 0.1, batch 128, early stopping on the five-caption validation loss: best epoch 6 of 10.

## Results

Greedy decoding, five references per image, sacreBLEU 2.6.0.

| split | images | five-caption loss | BLEU | chrF |
|---|---:|---:|---:|---:|
| validation | 1,014 | 2.809 | 40.01 | 46.07 |
| test | 995 | 2.792 | 41.28 | 46.19 |

Features and decoding run in bfloat16: on another GPU the loss is identical to three decimals and BLEU / chrF move by a few tenths.

## Limitations

- The references are machine translations, not human Wolof: scores measure agreement with these silver references, not
  fluency judged by native speakers, and there is no large-scale human evaluation.
- Flickr30k shows mostly North American and European scenes; the model has seen almost no photographs from Senegal and may
  describe local objects, clothing or places poorly.
- Known errors include confused objects, swapped roles between people, invented details and occasional repetitions.

## Citation

```bibtex
@misc{image2wolof2026,
  title  = {image2wolof: Wolof Image Captioning from Machine-Translated Captions with a Frozen SigLIP 2 Encoder},
  author = {Ibrahima Ba},
  year   = {2026},
  month  = {oct},
  url    = {https://github.com/milkymap/image2wolof}
}
```