File size: 7,155 Bytes
1c3b951
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
a4ff752
 
 
 
1c3b951
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
---
license: apache-2.0
library_name: pytorch
pipeline_tag: image-to-text
datasets:
  - jxie/coco_captions
tags:
  - image-captioning
  - video
  - dinov2
  - tiny
  - edge
  - mcp
---

# tiny-vision v1.0

**Let your AI assistant watch videos.** Many assistants, Claude included, can look at
images but can't open a video file. tiny-vision turns a local video into a scene-by-scene
description the assistant can read, through MCP (Claude Desktop, Claude Code, Cursor,
VS Code, LM Studio and other MCP apps) or as plain text for any LLM. It runs locally, so
nothing is uploaded.

It's a **36M-parameter** image and video captioner, fast on a laptop CPU. It uses Meta's
DINOv2-small as a frozen image encoder (its weights are unchanged, so this is not a
fine-tune of DINOv2); the adapter and caption decoder on top (14M parameters) were
trained from scratch on COCO.

- **89% of Florence-2-base's COCO caption score at 1/6 the size**
- **8× faster on GPU, 75× faster on CPU** (54 ms per image on an Apple M5 CPU)
- 1.2 GB peak GPU memory, 144 MB weights

## Benchmarks

**COCO Karpathy test (5,000 images)**

| Decoding | BLEU-4 | CIDEr |
|---|---|---|
| greedy | 0.342 | 1.145 |
| beam 3 (default) | 0.351 | 1.163 |

**Against other small open models.** Same 1,000 COCO test images, same scorer, Apple M5
(MPS GPU batch 16; CPU = single image). Reproduce with `bench.py`.

| Model | Params | CIDEr | BLEU-4 | GPU img/s | GPU peak memory | CPU ms/img |
|---|---|---|---|---|---|---|
| **tiny-vision v1.0** (greedy) | **36M** | 1.196 | 0.339 | **52** | **1.2 GB** | **54** |
| **tiny-vision v1.0** (beam 3) | 36M | 1.213 | 0.348 | 24 | 1.2 GB | 101 |
| Florence-2-base | 231M | 1.348 | 0.364 | 6.7 | 7.0 GB | 4,063 |
| Florence-2-base (beam 3) | 231M | 1.364 | 0.368 | 5.3 | 7.0 GB | 4,228 |
| SmolVLM-256M-Instruct ¹ | 256M | 0.126 | 0.051 | 0.5 | 23 GB | 55,800 |

¹ SmolVLM is a chat model. Even when asked for one short sentence, it writes long
descriptions, which caption metrics punish, so its quality score isn't a fair comparison.
Its speed and memory numbers are. Florence-2 is a multi-task model (detection, OCR,
grounding and more) and does more than captioning.

**Progress across versions** (greedy decoding)

| Version | Encoder | Train data | COCO CIDEr | Flickr30k CIDEr |
|---|---|---|---|---|
| v0.1 | from scratch | Flickr8k (6k images) | 0.061 | 0.131 |
| v0.2 (unreleased) | from scratch | Flickr30k (29k) | 0.121 | 0.204 |
| **v1.0** | frozen DINOv2-small | COCO (113k) | **1.145** | 0.423 |

Metrics are corpus BLEU-4 and CIDEr-D, computed with a pure-Python reimplementation of the
coco-caption scorers. It matches the reference code exactly on the same tokens, but
tokenization is simple lowercase plus punctuation stripping rather than the PTB tokenizer,
so scores are close to, but not identical with, numbers in papers.

## Architecture

```
image / video frames → DINOv2-small (frozen) → 2×2 pool → adapter → 6-layer decoder → text
                        22M, Meta, Apache-2.0    64 tokens   MLP      prefix-LM, 384-dim
```

| Part | Details | Trained here |
|---|---|---|
| Vision encoder | DINOv2-small ViT-S/14, 224px → 16×16 patches, frozen | no (pretrained by Meta) |
| Pooling + time | 2×2 average pool → 64 tokens per frame, sin-cos time embedding per video frame | — |
| Adapter | 2-layer MLP (384 → 384) | yes |
| Caption decoder | 6-layer prefix-LM transformer, 384-dim, RMSNorm + SwiGLU, 8,000-word vocabulary, tied embeddings | yes |

14M of the 36M parameters are trained, all on COCO. The encoder weights are bundled in
`model.safetensors`, so the model runs fully offline after download.

## Training

- Data: COCO Karpathy train split (113k images, 566,747 captions)
- 40,000 steps, batch 64 (~4.5 caption-epochs), AdamW (lr 3e-4, warmup 1,500, cosine to 3e-5),
  weight decay 0.05, bfloat16 autocast, random-resized-crop augmentation
- One Apple M5 MacBook Pro GPU, ~3 hours. Final validation loss 2.117

## Video: scene-by-scene summaries

`summarize_video` (MCP) and `caption.py clip.mp4 --summary` (CLI) do three things:
- split the video into shots (hard cuts plus slow camera drift)
- caption the start, middle and end of each shot
- group consecutive shots that show the same thing into scenes

The output is compact text that an LLM turns into a real summary:

```
video: clip.mp4 · 00:09 long · 3 scenes (3 shots) · avg scene 3s
recurring: dogs (2 scenes)

scene 1 [00:00–00:03] (3s)
  seen: a group of dogs running in a field. | three dogs running in a field with a bench in the background.
  keywords: dogs, running, field, bench
...
```

Over MCP, the connected assistant writes the summary. From the CLI, `--summarize` uses
a local Qwen2.5-1.5B-Instruct. Frames are captioned independently and there's no audio
yet, so it describes what's seen, not what's said or how things move.

## Example outputs (COCO test, beam 3)

| tiny-vision | Human reference |
|---|---|
| a man riding a motorcycle down a dirt road. | A man with a red helmet on a small moped on a dirt road. |
| a man riding a bike next to a train on a track. | A man on a bicycle riding next to a train |
| a group of people riding bikes down a street. | people on bicycles ride down a busy street |
| a bathroom with a shower, toilet, and sink. | A bathroom with an enclosed shower next to a sink and a toilet. |
| a woman sitting at a table with a cup of coffee. | A young girl inhales with the intent of blowing out a candle. *(wrong)* |

## Usage

```bash
git clone https://github.com/amar683/tiny-vision && cd tiny-vision
uv venv && uv pip install -r requirements.txt
python caption.py --run amartomar683/tiny-vision photo.jpg
python caption.py --run amartomar683/tiny-vision clip.mp4 --scenes
python eval.py --run amartomar683/tiny-vision --dataset jxie/coco_captions --beams 3
```

As an MCP server (`mcp_server.py`), it provides the `summarize_video`, `caption_image` and
`describe_video` tools. Add it to any app that runs local MCP servers, and that app's LLM
can work with local images and videos. Then ask things like *"what happens in
~/Downloads/talk.mp4?"*.

## Limitations

- **COCO-style captions only**: one short sentence about everyday scenes. No OCR, no precise
  counting, no names of people, places or brands, and no question answering.
- Domain shift: it scores lower on Flickr30k (CIDEr 0.42) than on COCO. Unusual content
  (screenshots, documents, art, medical images) gives confident but wrong captions.
- English only, word-level vocabulary (unknown words become `<unk>`).
- Video frames are captioned independently. It doesn't understand actions across frames.
- Like any model trained on COCO, it inherits COCO's biases (for example, guessing gender
  from appearance).

## License and data

- Code and the trained adapter/decoder weights: Apache-2.0.
- Encoder: DINOv2-small by Meta AI, Apache-2.0.
- Training data: COCO captions (annotations CC BY 4.0). The images come from Flickr under
  their own licenses; see the COCO terms of use.

Previous version: [amartomar683/tiny-vision-flickr8k](https://huggingface.co/amartomar683/tiny-vision-flickr8k) (v0.1, from scratch).