# USAGE (English) / 使用说明

DuoVLM-40M is **not a transformers model**. There is no `AutoModel.from_pretrained` path: the
vision encoder is a stock CLIP, but the language model is a 12-layer GPT written against
`litgpt==0.5.13`, and image features are injected by overwriting 196 placeholder-token embeddings.
All of that lives in `code/`.

## 1. Install

```bash
python -m venv .venv && . .venv/bin/activate
pip install -r requirements.txt
# torch with your CUDA build, e.g.:
# pip install torch --index-url https://download.pytorch.org/whl/cu124
```

CPU-only machines work: set `DUOVLM_DEVICE=cpu` (runs in fp32, ~1–3 s per question).

## 2. Environment variables (nothing is hard-coded to the author's machine)

| var | default | meaning |
|---|---|---|
| `DUOVLM_WEIGHTS` | `weights/duovlm-s3-final.pth` | checkpoint path |
| `DUOVLM_CFG` | `<weights dir>/model_config.yaml` | architecture spec |
| `DUOVLM_TOKENIZER` | `tokenizer/` | our 8k BPE tokenizer |
| `DUOVLM_CLIP` | `openai/clip-vit-base-patch16` | vision tower (HuggingFace id) |
| `DUOVLM_CLIP_DIR` | *(unset)* | local CLIP directory; **wins over `DUOVLM_CLIP`** (offline use) |
| `DUOVLM_DEVICE` | auto | `cuda` or `cpu` |

## 3. Ask a question

```bash
python code/ask_cli.py --image cat.jpg --question "What is the cat doing?"
python code/ask_cli.py --image cat.jpg                        # no question = describe the image
python code/ask_cli.py --image cat.jpg --question "..." --blind  # also run the "no image" control
python code/ask_cli.py --dir ./photos --question "What is happening?"
python code/ask_cli.py --text "The history of the city begins in" --continue-n 60   # text-only continuation
python code/ask_cli.py --image cat.jpg --question "..." --json   # machine-readable output
```

**Questions must be English.** The model never saw another language.

## 4. Python API

```python
import sys; sys.path.insert(0, "code")
from duovlm_infer import DuoVLMInfer

vlm = DuoVLMInfer()                     # loads once (~1 s on GPU, ~10 s CPU)
r = vlm.ask("photo.jpg", "How many people are in the image?")
print(r["answer"], r["ms"], "ms", r["stats"])
# {'answer': 'two', 'tokens': 1, 'ms': 45, 'emb_ms': 30, 'source': 'image', 'stats': {'eos_rows': 1}}
```

`ask()` accepts a path **or a `PIL.Image`** (handy for HTTP uploads). `blind=True` keeps the template
but zeroes the 196 image positions — the honest "no image" control. `no_image=True` removes the image
positions entirely (out of distribution; only good for demonstrating that the language model itself
knows nothing).

`dict` you get back: `answer`, `tokens`, `ms` (generation only), `emb_ms` (CLIP), `source`
(`image` / `blind(zeros)` / `no-image` / `text`), `stats` (`eos_rows` self-stopped, `loop_rows`
loop cut, `deg_rows` degeneracy cut).

## 5. Decoding rules (not optional)

Sequence template, fixed:

```
<|bos|> [IMAGE]×196 <|user|> question <|eot|> <|assistant|> answer <|eos|>
```

- **`no_repeat_ngram=3` is required.** Without it this 39M model repeats itself until `max_new`
  with certainty (measured: greedy never makes `<|eos|>` the argmax past the first sentence).
- `max_new`: **12** for Q&A, **30–40** for captions. Do not ask it for paragraphs.
- `code/decoding.py` adds a repetition penalty, a **loop cut** (stops and trims when the tail becomes
  periodic, period ≤ 12), and a **degeneracy guard** (stops when the local 2-gram distinct ratio
  collapses — this catches the "reworded loop" that an n-gram ban alone lets through:
  `a very small, small, and very small area. The ocean is a small, large area ...`).
- `temperature > 0` + `top_k` for caption variety; keep the n-gram ban on.

## 6. Interactive page

```bash
python code/webapp.py            # http://localhost:8000  (add --port 8080, --no-warmup)
```

Plain HTML, no CSS, English UI, nothing stored. Pick an image (file dialog, or `Ctrl+V` to paste a
screenshot), or click one of the bundled samples; type an English question or click a preset button;
optionally tick "blind" to see both answers side by side. The answer block reports token count,
latency, whether the loop cut fired, and which source was used. Requires `fastapi` + `uvicorn`.

## 7. Verify the package

```bash
python code/verify_package.py          # full: checksums, params, structure, inference, determinism, mini-protocol
python code/verify_package.py --quick  # skips the CLIP download
```

Objective checks decide PASS/FAIL; model accuracy is printed only (a 39M model legitimately misses
two out of three questions). `protocol/mini_vqa_100.jsonl` + `protocol/images/` let you recompute an
accuracy figure on 100 items without the 18 GB feature cache used for the full run.

## 8. Troubleshooting

| symptom | cause / fix |
|---|---|
| `ModuleNotFoundError: No module named 'torch'` | you ran a different interpreter. Use `python -m venv` from step 1, or the absolute path of that env's python. |
| `ModuleNotFoundError: No module named 'litgpt'` / attribute errors inside litgpt | version drift: pin `litgpt==0.5.13` exactly (other versions change GPT internals). |
| first run hangs on "loading CLIP" | it is downloading ~600 MB. Set `DUOVLM_CLIP_DIR=/path/to/clip-vit-base-patch16` for offline. |
| `OSError: Can't load ... clip` / 401 | no network and no `DUOVLM_CLIP_DIR`. |
| answers differ by a word from `README.md` | expected: bf16 kernels differ across GPUs/CUDA, and a different CLIP `revision` shifts features in the last bits. Greedy decoding is deterministic per machine, not across machines. |
| it answers nonsense to a Chinese question | the model is English-only; this is documented behavior, not a bug. |
| long outputs turn into loops | you bypassed `decoding.py` (`ngram=0`, `loop_break=False`). Use the defaults. |
| CPU run is slow / dtype complaints | set `DUOVLM_DEVICE=cpu`; the wrapper then uses fp32. |