--- license: apache-2.0 language: - wo library_name: pytorch pipeline_tag: image-to-text tags: - image-captioning - wolof - low-resource - siglip2 datasets: - milkymap/flickr30k-wolof base_model: google/siglip2-base-patch16-224 metrics: - bleu - chrf --- # image2wolof Image captioning in Wolof. A frozen [SigLIP2-base](https://huggingface.co/google/siglip2-base-patch16-224) encoder (patch tokens, 196 x 768, pinned revision `75de2d55`) feeds a six-layer transformer decoder (d 512, 8 heads, ff 2048, SentencePiece unigram vocabulary of 4,000 tied to the output layer, 27.9 M trainable parameters) trained from scratch. Code, training pipeline and paper: [github.com/milkymap/image2wolof](https://github.com/milkymap/image2wolof) ([PDF](https://github.com/milkymap/image2wolof/blob/main/paper/image2wolof.pdf)). ## Files | file | | |---|---| | `best.pt` | decoder weights, decoder config, vision backbone spec (model id, revision, preprocessing), tokenizer sha256 | | `tokenizer.model` | SentencePiece unigram tokenizer (vocab 4,000), must sit next to `best.pt` | | `config.json` | the same configuration in readable form | ## Usage ```bash git clone https://github.com/milkymap/image2wolof && cd image2wolof && uv sync hf download milkymap/image2wolof --local-dir runs/flickr30k uv run image2wolof serve runs/flickr30k/best.pt --port 8080 # upload page + POST /caption ``` ```python from PIL import Image from image2wolof.services.captioner import Captioner captioner = Captioner("runs/flickr30k/best.pt", device="cuda") print(captioner([Image.open("photo.jpg")])) ``` The SigLIP2 encoder is downloaded from the Hub at the pinned revision on first use. ## Training data [`milkymap/flickr30k-wolof`](https://huggingface.co/datasets/milkymap/flickr30k-wolof): the 29,703 Flickr30k training images, each with five Wolof captions obtained by machine translation of the five English captions, guided by a glossary of terms and rules. Every caption is a training target (148,515 image-caption pairs). AdamW 3e-4, weight decay 0.01, warmup then cosine decay, bf16, label smoothing 0.1, batch 128, early stopping on the five-caption validation loss: best epoch 6 of 10. ## Results Greedy decoding, five references per image, sacreBLEU 2.6.0. | split | images | five-caption loss | BLEU | chrF | |---|---:|---:|---:|---:| | validation | 1,014 | 2.809 | 40.01 | 46.07 | | test | 995 | 2.792 | 41.28 | 46.19 | Features and decoding run in bfloat16: on another GPU the loss is identical to three decimals and BLEU / chrF move by a few tenths. ## Limitations - The references are machine translations, not human Wolof: scores measure agreement with these silver references, not fluency judged by native speakers, and there is no large-scale human evaluation. - Flickr30k shows mostly North American and European scenes; the model has seen almost no photographs from Senegal and may describe local objects, clothing or places poorly. - Known errors include confused objects, swapped roles between people, invented details and occasional repetitions. ## Citation ```bibtex @misc{image2wolof2026, title = {image2wolof: Wolof Image Captioning from Machine-Translated Captions with a Frozen SigLIP 2 Encoder}, author = {Ibrahima Ba}, year = {2026}, month = {oct}, url = {https://github.com/milkymap/image2wolof} } ```