image2wolof

Image captioning in Wolof. A frozen SigLIP2-base encoder (patch tokens, 196 x 768, pinned revision 75de2d55) feeds a six-layer transformer decoder (d 512, 8 heads, ff 2048, SentencePiece unigram vocabulary of 4,000 tied to the output layer, 27.9 M trainable parameters) trained from scratch.

Code, training pipeline and paper: github.com/milkymap/image2wolof (PDF).

Files

file
best.pt decoder weights, decoder config, vision backbone spec (model id, revision, preprocessing), tokenizer sha256
tokenizer.model SentencePiece unigram tokenizer (vocab 4,000), must sit next to best.pt
config.json the same configuration in readable form

Usage

git clone https://github.com/milkymap/image2wolof && cd image2wolof && uv sync
hf download milkymap/image2wolof --local-dir runs/flickr30k
uv run image2wolof serve runs/flickr30k/best.pt --port 8080      # upload page + POST /caption
from PIL import Image
from image2wolof.services.captioner import Captioner

captioner = Captioner("runs/flickr30k/best.pt", device="cuda")
print(captioner([Image.open("photo.jpg")]))

The SigLIP2 encoder is downloaded from the Hub at the pinned revision on first use.

Training data

milkymap/flickr30k-wolof: the 29,703 Flickr30k training images, each with five Wolof captions obtained by machine translation of the five English captions, guided by a glossary of terms and rules. Every caption is a training target (148,515 image-caption pairs). AdamW 3e-4, weight decay 0.01, warmup then cosine decay, bf16, label smoothing 0.1, batch 128, early stopping on the five-caption validation loss: best epoch 6 of 10.

Results

Greedy decoding, five references per image, sacreBLEU 2.6.0.

split images five-caption loss BLEU chrF
validation 1,014 2.809 40.01 46.07
test 995 2.792 41.28 46.19

Features and decoding run in bfloat16: on another GPU the loss is identical to three decimals and BLEU / chrF move by a few tenths.

Limitations

  • The references are machine translations, not human Wolof: scores measure agreement with these silver references, not fluency judged by native speakers, and there is no large-scale human evaluation.
  • Flickr30k shows mostly North American and European scenes; the model has seen almost no photographs from Senegal and may describe local objects, clothing or places poorly.
  • Known errors include confused objects, swapped roles between people, invented details and occasional repetitions.

Citation

@misc{image2wolof2026,
  title  = {image2wolof: Wolof Image Captioning from Machine-Translated Captions with a Frozen SigLIP 2 Encoder},
  author = {Ibrahima Ba},
  year   = {2026},
  month  = {oct},
  url    = {https://github.com/milkymap/image2wolof}
}
Downloads last month
22
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for milkymap/image2wolof

Finetuned
(153)
this model

Dataset used to train milkymap/image2wolof