| # USAGE (English) / 使用说明 |
|
|
| DuoVLM-40M is **not a transformers model**. There is no `AutoModel.from_pretrained` path: the |
| vision encoder is a stock CLIP, but the language model is a 12-layer GPT written against |
| `litgpt==0.5.13`, and image features are injected by overwriting 196 placeholder-token embeddings. |
| All of that lives in `code/`. |
|
|
| ## 1. Install |
|
|
| ```bash |
| python -m venv .venv && . .venv/bin/activate |
| pip install -r requirements.txt |
| # torch with your CUDA build, e.g.: |
| # pip install torch --index-url https://download.pytorch.org/whl/cu124 |
| ``` |
|
|
| CPU-only machines work: set `DUOVLM_DEVICE=cpu` (runs in fp32, ~1–3 s per question). |
|
|
| ## 2. Environment variables (nothing is hard-coded to the author's machine) |
|
|
| | var | default | meaning | |
| |---|---|---| |
| | `DUOVLM_WEIGHTS` | `weights/duovlm-s3-final.pth` | checkpoint path | |
| | `DUOVLM_CFG` | `<weights dir>/model_config.yaml` | architecture spec | |
| | `DUOVLM_TOKENIZER` | `tokenizer/` | our 8k BPE tokenizer | |
| | `DUOVLM_CLIP` | `openai/clip-vit-base-patch16` | vision tower (HuggingFace id) | |
| | `DUOVLM_CLIP_DIR` | *(unset)* | local CLIP directory; **wins over `DUOVLM_CLIP`** (offline use) | |
| | `DUOVLM_DEVICE` | auto | `cuda` or `cpu` | |
| |
| ## 3. Ask a question |
| |
| ```bash |
| python code/ask_cli.py --image cat.jpg --question "What is the cat doing?" |
| python code/ask_cli.py --image cat.jpg # no question = describe the image |
| python code/ask_cli.py --image cat.jpg --question "..." --blind # also run the "no image" control |
| python code/ask_cli.py --dir ./photos --question "What is happening?" |
| python code/ask_cli.py --text "The history of the city begins in" --continue-n 60 # text-only continuation |
| python code/ask_cli.py --image cat.jpg --question "..." --json # machine-readable output |
| ``` |
| |
| **Questions must be English.** The model never saw another language. |
|
|
| ## 4. Python API |
|
|
| ```python |
| import sys; sys.path.insert(0, "code") |
| from duovlm_infer import DuoVLMInfer |
| |
| vlm = DuoVLMInfer() # loads once (~1 s on GPU, ~10 s CPU) |
| r = vlm.ask("photo.jpg", "How many people are in the image?") |
| print(r["answer"], r["ms"], "ms", r["stats"]) |
| # {'answer': 'two', 'tokens': 1, 'ms': 45, 'emb_ms': 30, 'source': 'image', 'stats': {'eos_rows': 1}} |
| ``` |
|
|
| `ask()` accepts a path **or a `PIL.Image`** (handy for HTTP uploads). `blind=True` keeps the template |
| but zeroes the 196 image positions — the honest "no image" control. `no_image=True` removes the image |
| positions entirely (out of distribution; only good for demonstrating that the language model itself |
| knows nothing). |
|
|
| `dict` you get back: `answer`, `tokens`, `ms` (generation only), `emb_ms` (CLIP), `source` |
| (`image` / `blind(zeros)` / `no-image` / `text`), `stats` (`eos_rows` self-stopped, `loop_rows` |
| loop cut, `deg_rows` degeneracy cut). |
|
|
| ## 5. Decoding rules (not optional) |
|
|
| Sequence template, fixed: |
|
|
| ``` |
| <|bos|> [IMAGE]×196 <|user|> question <|eot|> <|assistant|> answer <|eos|> |
| ``` |
|
|
| - **`no_repeat_ngram=3` is required.** Without it this 39M model repeats itself until `max_new` |
| with certainty (measured: greedy never makes `<|eos|>` the argmax past the first sentence). |
| - `max_new`: **12** for Q&A, **30–40** for captions. Do not ask it for paragraphs. |
| - `code/decoding.py` adds a repetition penalty, a **loop cut** (stops and trims when the tail becomes |
| periodic, period ≤ 12), and a **degeneracy guard** (stops when the local 2-gram distinct ratio |
| collapses — this catches the "reworded loop" that an n-gram ban alone lets through: |
| `a very small, small, and very small area. The ocean is a small, large area ...`). |
| - `temperature > 0` + `top_k` for caption variety; keep the n-gram ban on. |
|
|
| ## 6. Interactive page |
|
|
| ```bash |
| python code/webapp.py # http://localhost:8000 (add --port 8080, --no-warmup) |
| ``` |
|
|
| Plain HTML, no CSS, English UI, nothing stored. Pick an image (file dialog, or `Ctrl+V` to paste a |
| screenshot), or click one of the bundled samples; type an English question or click a preset button; |
| optionally tick "blind" to see both answers side by side. The answer block reports token count, |
| latency, whether the loop cut fired, and which source was used. Requires `fastapi` + `uvicorn`. |
|
|
| ## 7. Verify the package |
|
|
| ```bash |
| python code/verify_package.py # full: checksums, params, structure, inference, determinism, mini-protocol |
| python code/verify_package.py --quick # skips the CLIP download |
| ``` |
|
|
| Objective checks decide PASS/FAIL; model accuracy is printed only (a 39M model legitimately misses |
| two out of three questions). `protocol/mini_vqa_100.jsonl` + `protocol/images/` let you recompute an |
| accuracy figure on 100 items without the 18 GB feature cache used for the full run. |
|
|
| ## 8. Troubleshooting |
|
|
| | symptom | cause / fix | |
| |---|---| |
| | `ModuleNotFoundError: No module named 'torch'` | you ran a different interpreter. Use `python -m venv` from step 1, or the absolute path of that env's python. | |
| | `ModuleNotFoundError: No module named 'litgpt'` / attribute errors inside litgpt | version drift: pin `litgpt==0.5.13` exactly (other versions change GPT internals). | |
| | first run hangs on "loading CLIP" | it is downloading ~600 MB. Set `DUOVLM_CLIP_DIR=/path/to/clip-vit-base-patch16` for offline. | |
| | `OSError: Can't load ... clip` / 401 | no network and no `DUOVLM_CLIP_DIR`. | |
| | answers differ by a word from `README.md` | expected: bf16 kernels differ across GPUs/CUDA, and a different CLIP `revision` shifts features in the last bits. Greedy decoding is deterministic per machine, not across machines. | |
| | it answers nonsense to a Chinese question | the model is English-only; this is documented behavior, not a bug. | |
| | long outputs turn into loops | you bypassed `decoding.py` (`ngram=0`, `loop_break=False`). Use the defaults. | |
| | CPU run is slow / dtype complaints | set `DUOVLM_DEVICE=cpu`; the wrapper then uses fp32. | |
|
|
