Tossana-Caption-Khmer · Tossana ទស្សនៈ Khmer image captioning (PyTorch)

Tossana ("seeing") describes photos in Khmer, in two lengths: a short caption (one sentence) or a detailed description (2–4 sentences: objects, colors, positions). 121M parameters: a pretrained SigLIP image encoder (google/siglip-base-patch16-224) and a 6-layer decoder that writes the same 3,402 Khmer cluster tokens as Sota and Chakkhu. Made by Ing Muyleang. Preview release (research and evaluation). This is v3; v2 stays in the repo history.

	Link
This repo	PyTorch weights (model.safetensors), loads with 🤗 transformers
ONNX version (CPU, phone, browser)	https://huggingface.co/muyleanging/Tossana-Caption-Khmer-ONNX
Demo (short / detailed, ONNX / PyTorch)	https://huggingface.co/spaces/muyleanging/Tossana-Caption-Demo
Website	https://muyleanging.github.io/tossana-web/

Results

BLEU-4 / CIDEr on Khmer letter clusters (Khmer has no spaces between words), beam 5, same test photos for every version.

Short captions

Test set	Photos	v1 (ViT-S, 33M)	v2 (SigLIP, 104M)	v3 (this, 121M)
COCO test (objects, indoor scenes), translated to Khmer	1,000	—	49.6 / 115.1	50.8 / 116.0
Flickr8k test (everyday scenes), translated to Khmer	1,000	45.3 / 71.0	51.1 / 86.9	52.1 / 88.1
phonsobon (Cambodian scenes), v1 test	57	22.6 / 63.5	27.9 / 72.2	29.2 / 108.3
phonsobon (Cambodian scenes), v2 test (one long reference per photo)	853	11.0 / 18.9	11.2 / 18.5	10.4 / 15.1
All 2,910 photos	2,910	— / 50.5	— / 81.3	34.5 / 81.2

Detailed descriptions (new in v3), 231 held-out DOCCI photos with long human descriptions translated to Khmer: v3 18.3 / 18.6 · v2 (short captions only) 0.9 / 3.1.

Examples (photos taken with a phone, not in training):

Photo	v2	v3 short	v3 detailed (start)
iced coffee in a plastic cup on a desk	ការិយាល័យដែលមានកែវកាហ្វេនៅលើតុ	កែវប្លាស្ទិកដែលដាក់នៅលើតុស	រូបថតនេះបង្ហាញពីទិដ្ឋភាពលម្អិតនៃកែវប្លាស្ទិកដែលមានគម្របប្លាស្ទិកច្បាស់លាស់ដែលដាក់លើតុសមួយ។ …
lecture: a teacher in front of a big screen	បុរសម្នាក់ពាក់មួកពណ៌ខ្មៅកំពុងអង្គុយនៅមុខកុំព្យូទ័រធំ	មនុស្សម្នាក់កំពុងឈរនៅមុខអេក្រង់កុំព្យូទ័រ	រូបថតនេះបង្ហាញពីមនុស្សម្នាក់កំពុងឈរនៅក្នុងបន្ទប់មួយ…
the Bayon temple	មុខប្រាសាទបាយ័នក្នុងក្រុងអង្គរធំ…	មុខប្រាសាទបាយ័នក្នុងក្រុងអង្គរធំប្រទេសកម្ពុជា	រូបថតបង្ហាញពីមុខថ្មធំមួយ… ផ្ទៃខាងក្រោយគឺជាផ្ទៃមេឃពណ៌ខៀវច្បាស់…

⚠️ Not checked by people yet. Flickr8k, COCO and DOCCI references are machine translations; phonsobon references come from the dataset as published. Treat the numbers as a progress check. The detailed mode is a preview: it can invent details (e.g. a room or an object that is not there) and does not know most brand names yet.

Use it

pip install torch transformers pillow huggingface_hub

from huggingface_hub import snapshot_download
import sys; sys.path.insert(0, snapshot_download("muyleanging/Tossana-Caption-Khmer"))
from tossana_caption import Tossana

tossana = Tossana("muyleanging/Tossana-Caption-Khmer")
print(tossana("photo.jpg"))                    # short caption
print(tossana("photo.jpg", style="detail"))    # detailed description (2–4 sentences)

The style is chosen by a token after the start token (<short> or <detail>, ids 3402 / 3403 in tokens.json). The detailed mode blocks repeated 4-cluster sequences (no_repeat_ngram_size=4) so long text does not loop. A standard VisionEncoderDecoderModel (SigLIP vision encoder + BERT-style decoder). tossana_caption.py turns token ids into Khmer with tokens.json (one id = one Khmer cluster).

Architecture

Part	Details	Parameters
Image encoder	SigLIP-base/16 vision tower (pretrained on image–text pairs), 224×224 → 196 tokens, d = 768	92.9M
Projection + caption decoder	768 → 512, then 6 Transformer decoder layers with cross-attention, d = 512, 8 heads	27.8M
Vocabulary	3,402 Khmer cluster tokens (shared with Sota and Chakkhu) + <short>, <detail>; up to 256 clusters	—

Training

• Short captions (62,519 pairs, no news captions): COCO (15,000 photos × 2) and Flickr8k translated to Khmer with MADLAD-400; phonsobon/khmer_Images_Captioning v1 + v2 (Cambodian scenes, ×4); Darayut/khmer-historical-heritage-image-captioning (×2). Known MADLAD word errors fixed in 2,478 captions (table → តុ not តារាង; objects "sit" → ដាក់ not អង្គុយ; ពួកគេ for one person).
• Detailed descriptions (47,490, new in v3): DOCCI (google/docci, human-written), Localized Narratives for the COCO training photos, PixMo-Cap (allenai/pixmo-cap), and Qwen2.5-VL-7B-Instruct descriptions of the Cambodian and Flickr8k training photos; each cut to ≤ 4 sentences, translated sentence by sentence with MADLAD-400.
• Splits: a photo is in exactly one split across all sources (test > dev > train).
• Training: 16,000 steps, batch 64, AdamW (lr 3e-5 encoder / 3e-4 decoder), cosine, bf16, one rented RTX 5090 (≈45 min); encoder started from v2. Best dev (short + detailed CIDEr) at step 10,000.

Limitations

• Can still name the wrong object (a Monster Energy can → "beer bottle") and does not know most brands (Samsung, Starbucks).
• The detailed mode can invent details that are not in the photo. A guard is being researched.
• Cambodian street scenes (phonsobon v2 test) are a little lower than v2 (15.1 vs 18.5 CIDEr).
• Test captions are not yet checked by people; the scores will change when they are.

Citation

@misc{ing2026tossana,
  author    = {Ing, Muyleang},
  title     = {Tossana: A Small Khmer Image Captioning Model},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/muyleanging/Tossana-Caption-Khmer}
}

Data and model credits: phonsobon (khmer_Images_Captioning, v2), Darayut (khmer-historical-heritage-image-captioning), Flickr8k (Hodosh, Young & Hockenmaier, JAIR 2013), COCO (Lin et al., 2014; Karpathy split), MADLAD-400 (Kudugunta et al., 2023), SigLIP (Zhai et al., 2023), DOCCI (Onoe et al., 2024), Localized Narratives (Pont-Tuset et al., 2020), PixMo-Cap (Deitke et al., 2024), Qwen2.5-VL (Qwen team, 2025).

License

Preview: personal use, evaluation and research. See LICENSE.

Family

Akkhara អក្ខរៈ (types) · Chakkhu ចក្ខុ (reads) · Sota សោតៈ (listens) · Tossana ទស្សនៈ (sees) · VoxPocket (speaks)