docs/USAGE.md

USAGE (English) / 使用说明

DuoVLM-40M is not a transformers model. There is no AutoModel.from_pretrained path: the vision encoder is a stock CLIP, but the language model is a 12-layer GPT written against litgpt==0.5.13, and image features are injected by overwriting 196 placeholder-token embeddings. All of that lives in code/.

1. Install

python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
# torch with your CUDA build, e.g.:
# pip install torch --index-url https://download.pytorch.org/whl/cu124

CPU-only machines work: set DUOVLM_DEVICE=cpu (runs in fp32, ~1–3 s per question).

2. Environment variables (nothing is hard-coded to the author's machine)

var	default	meaning
DUOVLM_WEIGHTS	weights/duovlm-s3-final.pth	checkpoint path
DUOVLM_CFG	<weights dir>/model_config.yaml	architecture spec
DUOVLM_TOKENIZER	tokenizer/	our 8k BPE tokenizer
DUOVLM_CLIP	openai/clip-vit-base-patch16	vision tower (HuggingFace id)
DUOVLM_CLIP_DIR	(unset)	local CLIP directory; wins over DUOVLM_CLIP (offline use)
DUOVLM_DEVICE	auto	cuda or cpu

3. Ask a question

python code/ask_cli.py --image cat.jpg --question "What is the cat doing?"
python code/ask_cli.py --image cat.jpg                        # no question = describe the image
python code/ask_cli.py --image cat.jpg --question "..." --blind  # also run the "no image" control
python code/ask_cli.py --dir ./photos --question "What is happening?"
python code/ask_cli.py --text "The history of the city begins in" --continue-n 60   # text-only continuation
python code/ask_cli.py --image cat.jpg --question "..." --json   # machine-readable output

Questions must be English. The model never saw another language.

4. Python API

import sys; sys.path.insert(0, "code")
from duovlm_infer import DuoVLMInfer

vlm = DuoVLMInfer()                     # loads once (~1 s on GPU, ~10 s CPU)
r = vlm.ask("photo.jpg", "How many people are in the image?")
print(r["answer"], r["ms"], "ms", r["stats"])
# {'answer': 'two', 'tokens': 1, 'ms': 45, 'emb_ms': 30, 'source': 'image', 'stats': {'eos_rows': 1}}

ask() accepts a path or a PIL.Image (handy for HTTP uploads). blind=True keeps the template but zeroes the 196 image positions — the honest "no image" control. no_image=True removes the image positions entirely (out of distribution; only good for demonstrating that the language model itself knows nothing).

dict you get back: answer, tokens, ms (generation only), emb_ms (CLIP), source (image / blind(zeros) / no-image / text), stats (eos_rows self-stopped, loop_rows loop cut, deg_rows degeneracy cut).

5. Decoding rules (not optional)

Sequence template, fixed:

<|bos|> [IMAGE]×196 <|user|> question <|eot|> <|assistant|> answer <|eos|>

• no_repeat_ngram=3 is required. Without it this 39M model repeats itself until max_new with certainty (measured: greedy never makes <|eos|> the argmax past the first sentence).
• max_new: 12 for Q&A, 30–40 for captions. Do not ask it for paragraphs.
• code/decoding.py adds a repetition penalty, a loop cut (stops and trims when the tail becomes periodic, period ≤ 12), and a degeneracy guard (stops when the local 2-gram distinct ratio collapses — this catches the "reworded loop" that an n-gram ban alone lets through: a very small, small, and very small area. The ocean is a small, large area ...).
• temperature > 0 + top_k for caption variety; keep the n-gram ban on.

6. Interactive page

python code/webapp.py            # http://localhost:8000  (add --port 8080, --no-warmup)

Plain HTML, no CSS, English UI, nothing stored. Pick an image (file dialog, or Ctrl+V to paste a screenshot), or click one of the bundled samples; type an English question or click a preset button; optionally tick "blind" to see both answers side by side. The answer block reports token count, latency, whether the loop cut fired, and which source was used. Requires fastapi + uvicorn.

7. Verify the package

python code/verify_package.py          # full: checksums, params, structure, inference, determinism, mini-protocol
python code/verify_package.py --quick  # skips the CLIP download

Objective checks decide PASS/FAIL; model accuracy is printed only (a 39M model legitimately misses two out of three questions). protocol/mini_vqa_100.jsonl + protocol/images/ let you recompute an accuracy figure on 100 items without the 18 GB feature cache used for the full run.

8. Troubleshooting

symptom	cause / fix
ModuleNotFoundError: No module named 'torch'	you ran a different interpreter. Use python -m venv from step 1, or the absolute path of that env's python.
ModuleNotFoundError: No module named 'litgpt' / attribute errors inside litgpt	version drift: pin litgpt==0.5.13 exactly (other versions change GPT internals).
first run hangs on "loading CLIP"	it is downloading ~600 MB. Set DUOVLM_CLIP_DIR=/path/to/clip-vit-base-patch16 for offline.
OSError: Can't load ... clip / 401	no network and no DUOVLM_CLIP_DIR.
answers differ by a word from README.md	expected: bf16 kernels differ across GPUs/CUDA, and a different CLIP revision shifts features in the last bits. Greedy decoding is deterministic per machine, not across machines.
it answers nonsense to a Chinese question	the model is English-only; this is documented behavior, not a bug.
long outputs turn into loops	you bypassed decoding.py (ngram=0, loop_break=False). Use the defaults.
CPU run is slow / dtype complaints	set DUOVLM_DEVICE=cpu; the wrapper then uses fp32.