array commited on
Commit
a0a4e1b
·
verified ·
1 Parent(s): f9b2818

Model card: add Serving with vLLM section

Browse files

Documents that vLLM serves this checkpoint natively, that the chat template now
supplies the latent block automatically, how to check the latent path is
actually active (8.9 vs 101.2 mean output tokens with/without), and the
max_pixels setting needed to reproduce the paper's numbers.

Files changed (1) hide show
  1. README.md +51 -0
README.md CHANGED
@@ -115,6 +115,57 @@ response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
115
  print(response)
116
  ```
117
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
118
  ## Citation
119
 
120
  ```bibtex
 
115
  print(response)
116
  ```
117
 
118
+ ## Serving with vLLM
119
+
120
+ vLLM serves this checkpoint with its **native** Qwen2.5-VL implementation — no
121
+ custom model code and no `--trust-remote-code`. The latent tokens are ordinary
122
+ entries in an extended vocabulary (`vocab_size` 151669) with trained embeddings,
123
+ and the latent-specific code paths in the training model are gated on a
124
+ training-only flag, so they never run during generation.
125
+
126
+ ```bash
127
+ vllm serve array/Qwen2.5-VL-Mull --max-model-len 32768
128
+ ```
129
+
130
+ The chat template pre-fills the assistant turn with `<think>` + 20 x
131
+ `<|latent_pad|>` + `</think>`, so ordinary chat requests get the latent block
132
+ automatically:
133
+
134
+ ```python
135
+ from openai import OpenAI
136
+ client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
137
+ client.chat.completions.create(
138
+ model="array/Qwen2.5-VL-Mull",
139
+ messages=[{"role": "user", "content": [
140
+ {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
141
+ {"type": "text", "text": question},
142
+ ]}],
143
+ temperature=0, max_tokens=512,
144
+ # optional: vary the latent budget, or set 0 to disable it for an ablation
145
+ extra_body={"chat_template_kwargs": {"num_latents": 20}},
146
+ )
147
+ ```
148
+
149
+ **Check that the latent path is active.** If you serve this model with a stock
150
+ Qwen2.5-VL chat template, the prompt ends at `<|im_start|>assistant\n` and the
151
+ model receives *zero* latent tokens. It still answers, so the mistake is easy to
152
+ miss — but it reverts to writing its reasoning out as text. On 24 SAT items the
153
+ same checkpoint averaged **8.9 output tokens** with the latent block and
154
+ **101.2** without. Verify with
155
+ `list(output.prompt_token_ids).count(151665) == 20`.
156
+
157
+ **Reproducing the paper's numbers.** The evaluations run
158
+ `max_pixels=12845056`, while `preprocessor_config.json` defaults to 401408:
159
+
160
+ ```bash
161
+ vllm serve array/Qwen2.5-VL-Mull --max-model-len 32768 \
162
+ --mm-processor-kwargs '{"min_pixels": 3136, "max_pixels": 12845056}'
163
+ ```
164
+
165
+ A [recipe](https://recipes.vllm.ai/array/Qwen2.5-VL-Mull), serving scripts and an
166
+ HF-vs-vLLM parity harness live in
167
+ [`serving/`](https://github.com/arijitray1993/mull/tree/main/serving).
168
+
169
  ## Citation
170
 
171
  ```bibtex