milkymap commited on
Commit
033e5de
·
verified ·
1 Parent(s): 529a020

image2wolof: SigLIP2-base + 6-layer decoder trained on Flickr30k Wolof captions

Browse files
Files changed (4) hide show
  1. README.md +91 -0
  2. best.pt +3 -0
  3. config.json +40 -0
  4. tokenizer.model +3 -0
README.md ADDED
@@ -0,0 +1,91 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - wo
5
+ library_name: pytorch
6
+ pipeline_tag: image-to-text
7
+ tags:
8
+ - image-captioning
9
+ - wolof
10
+ - low-resource
11
+ - siglip2
12
+ datasets:
13
+ - milkymap/flickr30k-wolof
14
+ base_model: google/siglip2-base-patch16-224
15
+ metrics:
16
+ - bleu
17
+ - chrf
18
+ ---
19
+
20
+ # image2wolof
21
+
22
+ Image captioning in Wolof. A frozen [SigLIP2-base](https://huggingface.co/google/siglip2-base-patch16-224) encoder
23
+ (patch tokens, 196 x 768, pinned revision `75de2d55`) feeds a six-layer transformer decoder (d 512, 8 heads, ff 2048,
24
+ SentencePiece unigram vocabulary of 4,000 tied to the output layer, 27.9 M trainable parameters) trained from scratch.
25
+
26
+ Code, training pipeline and paper: [github.com/milkymap/image2wolof](https://github.com/milkymap/image2wolof)
27
+ ([PDF](https://github.com/milkymap/image2wolof/blob/main/paper/image2wolof.pdf)).
28
+
29
+ ## Files
30
+
31
+ | file | |
32
+ |---|---|
33
+ | `best.pt` | decoder weights, decoder config, vision backbone spec (model id, revision, preprocessing), tokenizer sha256 |
34
+ | `tokenizer.model` | SentencePiece unigram tokenizer (vocab 4,000), must sit next to `best.pt` |
35
+ | `config.json` | the same configuration in readable form |
36
+
37
+ ## Usage
38
+
39
+ ```bash
40
+ git clone https://github.com/milkymap/image2wolof && cd image2wolof && uv sync
41
+ hf download milkymap/image2wolof --local-dir runs/flickr30k
42
+ uv run image2wolof serve runs/flickr30k/best.pt --port 8080 # upload page + POST /caption
43
+ ```
44
+
45
+ ```python
46
+ from PIL import Image
47
+ from image2wolof.services.captioner import Captioner
48
+
49
+ captioner = Captioner("runs/flickr30k/best.pt", device="cuda")
50
+ print(captioner([Image.open("photo.jpg")]))
51
+ ```
52
+
53
+ The SigLIP2 encoder is downloaded from the Hub at the pinned revision on first use.
54
+
55
+ ## Training data
56
+
57
+ [`milkymap/flickr30k-wolof`](https://huggingface.co/datasets/milkymap/flickr30k-wolof): the 29,703 Flickr30k training images,
58
+ each with five Wolof captions obtained by machine translation of the five English captions, guided by a glossary of terms and
59
+ rules. Every caption is a training target (148,515 image-caption pairs). AdamW 3e-4, weight decay 0.01, warmup then cosine decay,
60
+ bf16, label smoothing 0.1, batch 128, early stopping on the five-caption validation loss: best epoch 6 of 10.
61
+
62
+ ## Results
63
+
64
+ Greedy decoding, five references per image, sacreBLEU 2.6.0.
65
+
66
+ | split | images | five-caption loss | BLEU | chrF |
67
+ |---|---:|---:|---:|---:|
68
+ | validation | 1,014 | 2.809 | 40.01 | 46.07 |
69
+ | test | 995 | 2.792 | 41.28 | 46.19 |
70
+
71
+ Features and decoding run in bfloat16: on another GPU the loss is identical to three decimals and BLEU / chrF move by a few tenths.
72
+
73
+ ## Limitations
74
+
75
+ - The references are machine translations, not human Wolof: scores measure agreement with these silver references, not
76
+ fluency judged by native speakers, and there is no large-scale human evaluation.
77
+ - Flickr30k shows mostly North American and European scenes; the model has seen almost no photographs from Senegal and may
78
+ describe local objects, clothing or places poorly.
79
+ - Known errors include confused objects, swapped roles between people, invented details and occasional repetitions.
80
+
81
+ ## Citation
82
+
83
+ ```bibtex
84
+ @misc{image2wolof2026,
85
+ title = {image2wolof: Wolof Image Captioning from Machine-Translated Captions with a Frozen SigLIP 2 Encoder},
86
+ author = {Ibrahima Ba},
87
+ year = {2026},
88
+ month = {oct},
89
+ url = {https://github.com/milkymap/image2wolof}
90
+ }
91
+ ```
best.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0136da26f4a3cb9b771f651770cd699150791200478a0ea3e81da5967a04aa4f
3
+ size 111763866
config.json ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architecture": "SigLIP2-base (frozen) + transformer decoder",
3
+ "vocab_size": 4000,
4
+ "d": 512,
5
+ "layers": 6,
6
+ "heads": 8,
7
+ "ff": 2048,
8
+ "dropout": 0.1,
9
+ "memory_layers": 0,
10
+ "feature_dim": 768,
11
+ "grid_size": 14,
12
+ "add_memory_positions": false,
13
+ "vision_backbone": {
14
+ "model_id": "google/siglip2-base-patch16-224",
15
+ "revision": "75de2d55ec2d0b4efc50b3e9ad70dba96a7b2fa2",
16
+ "feature": "last_hidden_state",
17
+ "shape": [
18
+ 196,
19
+ 768
20
+ ]
21
+ },
22
+ "tokenizer": {
23
+ "type": "sentencepiece unigram",
24
+ "vocab_size": 4000,
25
+ "sha256": "8130a57672117a7faf0aa07b4f82f839fd74fd13a533b5da516e7e847410cdd1"
26
+ },
27
+ "epoch": 6,
28
+ "validation_loss": 2.8089,
29
+ "training": {
30
+ "epochs_max": 20,
31
+ "patience": 4,
32
+ "batch_size": 128,
33
+ "lr": 0.0003,
34
+ "optimizer": "AdamW",
35
+ "weight_decay": 0.01,
36
+ "label_smoothing": 0.1,
37
+ "precision": "bf16",
38
+ "seed": 42
39
+ }
40
+ }
tokenizer.model ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8130a57672117a7faf0aa07b4f82f839fd74fd13a533b5da516e7e847410cdd1
3
+ size 66600