|
Download README.md from milkymap/image2wolof: direct link, hf CLI and curl.
- Browser
- Download file 3.35 kB
-
https://huggingface.co/milkymap/image2wolof/resolve/main/README.md
- Command line
-
hf download hf://milkymap/image2wolof/README.md
-
curl -L -o README.md https://huggingface.co/milkymap/image2wolof/resolve/main/README.md
3.35 kB
| license: apache-2.0 | |
| language: | |
| - wo | |
| library_name: pytorch | |
| pipeline_tag: image-to-text | |
| tags: | |
| - image-captioning | |
| - wolof | |
| - low-resource | |
| - siglip2 | |
| datasets: | |
| - milkymap/flickr30k-wolof | |
| base_model: google/siglip2-base-patch16-224 | |
| metrics: | |
| - bleu | |
| - chrf | |
| # image2wolof | |
| Image captioning in Wolof. A frozen [SigLIP2-base](https://huggingface.co/google/siglip2-base-patch16-224) encoder | |
| (patch tokens, 196 x 768, pinned revision `75de2d55`) feeds a six-layer transformer decoder (d 512, 8 heads, ff 2048, | |
| SentencePiece unigram vocabulary of 4,000 tied to the output layer, 27.9 M trainable parameters) trained from scratch. | |
| Code, training pipeline and paper: [github.com/milkymap/image2wolof](https://github.com/milkymap/image2wolof) | |
| ([PDF](https://github.com/milkymap/image2wolof/blob/main/paper/image2wolof.pdf)). | |
| ## Files | |
| | file | | | |
| |---|---| | |
| | `best.pt` | decoder weights, decoder config, vision backbone spec (model id, revision, preprocessing), tokenizer sha256 | | |
| | `tokenizer.model` | SentencePiece unigram tokenizer (vocab 4,000), must sit next to `best.pt` | | |
| | `config.json` | the same configuration in readable form | | |
| ## Usage | |
| ```bash | |
| git clone https://github.com/milkymap/image2wolof && cd image2wolof && uv sync | |
| hf download milkymap/image2wolof --local-dir runs/flickr30k | |
| uv run image2wolof serve runs/flickr30k/best.pt --port 8080 # upload page + POST /caption | |
| ``` | |
| ```python | |
| from PIL import Image | |
| from image2wolof.services.captioner import Captioner | |
| captioner = Captioner("runs/flickr30k/best.pt", device="cuda") | |
| print(captioner([Image.open("photo.jpg")])) | |
| ``` | |
| The SigLIP2 encoder is downloaded from the Hub at the pinned revision on first use. | |
| ## Training data | |
| [`milkymap/flickr30k-wolof`](https://huggingface.co/datasets/milkymap/flickr30k-wolof): the 29,703 Flickr30k training images, | |
| each with five Wolof captions obtained by machine translation of the five English captions, guided by a glossary of terms and | |
| rules. Every caption is a training target (148,515 image-caption pairs). AdamW 3e-4, weight decay 0.01, warmup then cosine decay, | |
| bf16, label smoothing 0.1, batch 128, early stopping on the five-caption validation loss: best epoch 6 of 10. | |
| ## Results | |
| Greedy decoding, five references per image, sacreBLEU 2.6.0. | |
| | split | images | five-caption loss | BLEU | chrF | | |
| |---|---:|---:|---:|---:| | |
| | validation | 1,014 | 2.809 | 40.01 | 46.07 | | |
| | test | 995 | 2.792 | 41.28 | 46.19 | | |
| Features and decoding run in bfloat16: on another GPU the loss is identical to three decimals and BLEU / chrF move by a few tenths. | |
| ## Limitations | |
| - The references are machine translations, not human Wolof: scores measure agreement with these silver references, not | |
| fluency judged by native speakers, and there is no large-scale human evaluation. | |
| - Flickr30k shows mostly North American and European scenes; the model has seen almost no photographs from Senegal and may | |
| describe local objects, clothing or places poorly. | |
| - Known errors include confused objects, swapped roles between people, invented details and occasional repetitions. | |
| ## Citation | |
| ```bibtex | |
| @misc{image2wolof2026, | |
| title = {image2wolof: Wolof Image Captioning from Machine-Translated Captions with a Frozen SigLIP 2 Encoder}, | |
| author = {Ibrahima Ba}, | |
| year = {2026}, | |
| month = {oct}, | |
| url = {https://github.com/milkymap/image2wolof} | |
| } | |
| ``` | |