Duoia/duovlm-40m-v1

DuoVLM-40M (v1)

A 39.34M-trainable-parameter English vision-language model for short questions about a single image: frozen CLIP ViT-B/16 vision tower + a 1.31M connector + a 38.02M language model trained from scratch on MiniPile. It answers things like "How many people are in the image?", "What sport is the person performing?", "Render a clear and concise summary of the photo."

This is not a transformers model — it ships with its own inference code (code/) and a one-command self-check. 中文说明见 README.zh.md；用法细节见 docs/USAGE.md。

Headline numbers (measured on this exact checkpoint)

Metric	Value
Trainable parameters	39,335,936 (LLM 38,023,680 + connector 1,312,256)
Weights on disk	78.7 MB (bf16)
Language model	12 layers, d=512, GQA 8Q/2KV, SwiGLU 1408, vocab 8192, ctx 512, RoPE, tied embeddings
VQAv2 test, 1,812 items	29.53% strict normalized match
Same items with image positions zeroed ("blind")	19.65%
Visual gain	+9.88 pp
Reference floors	always-answer-yes 18.54%, text-only prior 21.85%
Latency	0.1–0.3 s/question with precomputed features; ~1.8 s cold (CLIP runs on the fly)
Memory	0.26 GiB VRAM (bf16), also runs CPU-only (1–3 s/question)

Accuracy per question type — the deltas are the honest part (where the image actually helps):

type	n	normal	blind	Δ
yes/no	696	46.3%	42.0%	+4.3
other (open)	625	17.8%	4.5%	+13.3
number (counting)	228	27.2%	4.8%	+22.4
color	191	16.8%	12.6%	+4.2
spatial	72	11.1%	1.4%	+9.7

Training cost, end to end: Stage 1 (1.0B tokens from scratch) ≈ 4.4 h on one RTX 4060 Laptop, Stage 2 (connector only) 7.7 min, Stage 3 (instruction tuning) 8.0 min.

Quick start

pip install -r requirements.txt          # torch / transformers / litgpt==0.5.13

# 1) ask a question about any image (English questions only)
python code/ask_cli.py --image photo.jpg --question "How many people are in the image?"
python code/ask_cli.py --image photo.jpg --question "Render a clear and concise summary of the photo."
python code/ask_cli.py --image photo.jpg --question "How many people are in the image?" --blind  # side-by-side blind control

# 2) the interactive web page (plain HTML, no CSS; English UI; one-click sample images)
python code/webapp.py                    # then open http://localhost:8000

# 3) self-check: checksums, parameter counts, structure, inference, determinism, mini-protocol
python code/verify_package.py

The first run downloads the CLIP vision tower (~600 MB) from HuggingFace. Offline? Point DUOVLM_CLIP_DIR at a local copy of openai/clip-vit-base-patch16.

Real outputs (captured stdout, not hand-written)

From protocol/mini_vqa_100.jsonl, run with this package:

Q: Has the carrot been peeled?                    → 'no'        (reference: no)   ✓
Q: How many bags can you see?                     → '1'         (reference: 1)    ✓   blind: 'twigs' ✗
Q: What color is the bench?                       → 'blue'      (reference: brown) ✗  blind: 'black' ✗
Q: What modes of transportation are in this picture? → 'coconut' (reference: bus)   ✗  blind: 'trees'
Q: Render a clear and concise summary of the photo.  → 'a close up of a brown lemur'  ✓

Note both kinds of rows: counting works, colours do not, and open-ended answers can be plain wrong.

Honest limitations

• Colours are unreliable. What color is the bench? → blue when the bench is brown, and the answer is not even stable: recomputing the image features with the on-the-fly CLIP path (bf16 last-bit differences only) flipped red → green on another image. It guesses from priors.
• No OCR. Cannot read text in images. No fine-grained attributes (brands, models).
• English only. Trained on English text; other languages degrade to noise.
• No world knowledge. Ask the language model a fact without an image ("What colour is a banana?") and it answers blue — 38M parameters pretrained on 1.0B tokens store templates, not facts.
• Keep outputs short. Answers are 1–3 words. Beyond ~30 generated tokens the 39M model degenerates; the decoder in code/decoding.py enforces no_repeat_ngram=3 plus a loop cut and a degeneracy guard, which is required, not optional.
• Single image, single turn. No multi-image or dialogue context.

What's in the box

weights/duovlm-s3-final.pth     the checkpoint (LLM + connector; self-contained)
weights/model_config.yaml       architecture spec
tokenizer/                      our 8k BPE tokenizer (8 specials: bos eos eot padding user assistant system image)
code/duovlm.py                  model + image-token injection + save/load
code/duovlm_infer.py            inference wrapper (env-var overridable paths, CUDA or CPU)
code/decoding.py                n-gram ban + repetition penalty + loop cut + degeneracy guard
code/ask_cli.py                 CLI
code/webapp.py + code/index.html  interactive page (drag/drop/paste, samples, blind control)
code/verify_package.py          one-command self-check (PASS/FAIL)
protocol/mini_vqa_100.jsonl    100 VQAv2 test items, images included
protocol/images/               99 images for those items
LICENSE                        Apache License 2.0 (full text)
SHA256SUMS                     checksums of every file above

Reproducing the headline number

29.53% is on the full VQAv2 test slice (1,812 items; VQAv2 is split by image). The bundled 100-item mini-protocol is a same-distribution sample — use it to confirm the pipeline runs and to compare with the numbers above; for the exact headline figure use the full slice:

python code/verify_package.py            # prints its own accuracy on the 100 bundled items

verify_package.py only prints accuracy: PASS/FAIL is decided by objective checks (checksums, parameter counts, weight structure, determinism), because a 39M model legitimately gets two-thirds of questions wrong and that must not make a good package look broken.

License

Apache License 2.0 — full text in LICENSE.

The weights are a derivative work of the upstream datasets and the CLIP tower listed in NOTICE. Each upstream keeps its own terms; where an upstream is stricter, that upstream governs. Verify every upstream page (in particular the terms of openai/clip-vit-base-patch16, VQAv2 and COCO) before redistributing this package.