Image-Text-to-Text
Transformers
Safetensors
qwen2_5_vl
multimodal
vision-language
spatial-reasoning
latent-reasoning
conversational
custom_code
text-generation-inference
Instructions to use array/Qwen2.5-VL-Mull with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use array/Qwen2.5-VL-Mull with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="array/Qwen2.5-VL-Mull", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("array/Qwen2.5-VL-Mull", trust_remote_code=True) model = AutoModelForMultimodalLM.from_pretrained("array/Qwen2.5-VL-Mull", trust_remote_code=True, device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use array/Qwen2.5-VL-Mull with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "array/Qwen2.5-VL-Mull" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "array/Qwen2.5-VL-Mull", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/array/Qwen2.5-VL-Mull
- SGLang
How to use array/Qwen2.5-VL-Mull with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "array/Qwen2.5-VL-Mull" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "array/Qwen2.5-VL-Mull", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "array/Qwen2.5-VL-Mull" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "array/Qwen2.5-VL-Mull", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use array/Qwen2.5-VL-Mull with Docker Model Runner:
docker model run hf.co/array/Qwen2.5-VL-Mull
Model card: add Serving with vLLM section
Browse filesDocuments that vLLM serves this checkpoint natively, that the chat template now
supplies the latent block automatically, how to check the latent path is
actually active (8.9 vs 101.2 mean output tokens with/without), and the
max_pixels setting needed to reproduce the paper's numbers.
README.md
CHANGED
|
@@ -115,6 +115,57 @@ response = processor.batch_decode(generated_ids, skip_special_tokens=True)[0]
|
|
| 115 |
print(response)
|
| 116 |
```
|
| 117 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
## Citation
|
| 119 |
|
| 120 |
```bibtex
|
|
|
|
| 115 |
print(response)
|
| 116 |
```
|
| 117 |
|
| 118 |
+
## Serving with vLLM
|
| 119 |
+
|
| 120 |
+
vLLM serves this checkpoint with its **native** Qwen2.5-VL implementation — no
|
| 121 |
+
custom model code and no `--trust-remote-code`. The latent tokens are ordinary
|
| 122 |
+
entries in an extended vocabulary (`vocab_size` 151669) with trained embeddings,
|
| 123 |
+
and the latent-specific code paths in the training model are gated on a
|
| 124 |
+
training-only flag, so they never run during generation.
|
| 125 |
+
|
| 126 |
+
```bash
|
| 127 |
+
vllm serve array/Qwen2.5-VL-Mull --max-model-len 32768
|
| 128 |
+
```
|
| 129 |
+
|
| 130 |
+
The chat template pre-fills the assistant turn with `<think>` + 20 x
|
| 131 |
+
`<|latent_pad|>` + `</think>`, so ordinary chat requests get the latent block
|
| 132 |
+
automatically:
|
| 133 |
+
|
| 134 |
+
```python
|
| 135 |
+
from openai import OpenAI
|
| 136 |
+
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
| 137 |
+
client.chat.completions.create(
|
| 138 |
+
model="array/Qwen2.5-VL-Mull",
|
| 139 |
+
messages=[{"role": "user", "content": [
|
| 140 |
+
{"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
|
| 141 |
+
{"type": "text", "text": question},
|
| 142 |
+
]}],
|
| 143 |
+
temperature=0, max_tokens=512,
|
| 144 |
+
# optional: vary the latent budget, or set 0 to disable it for an ablation
|
| 145 |
+
extra_body={"chat_template_kwargs": {"num_latents": 20}},
|
| 146 |
+
)
|
| 147 |
+
```
|
| 148 |
+
|
| 149 |
+
**Check that the latent path is active.** If you serve this model with a stock
|
| 150 |
+
Qwen2.5-VL chat template, the prompt ends at `<|im_start|>assistant\n` and the
|
| 151 |
+
model receives *zero* latent tokens. It still answers, so the mistake is easy to
|
| 152 |
+
miss — but it reverts to writing its reasoning out as text. On 24 SAT items the
|
| 153 |
+
same checkpoint averaged **8.9 output tokens** with the latent block and
|
| 154 |
+
**101.2** without. Verify with
|
| 155 |
+
`list(output.prompt_token_ids).count(151665) == 20`.
|
| 156 |
+
|
| 157 |
+
**Reproducing the paper's numbers.** The evaluations run
|
| 158 |
+
`max_pixels=12845056`, while `preprocessor_config.json` defaults to 401408:
|
| 159 |
+
|
| 160 |
+
```bash
|
| 161 |
+
vllm serve array/Qwen2.5-VL-Mull --max-model-len 32768 \
|
| 162 |
+
--mm-processor-kwargs '{"min_pixels": 3136, "max_pixels": 12845056}'
|
| 163 |
+
```
|
| 164 |
+
|
| 165 |
+
A [recipe](https://recipes.vllm.ai/array/Qwen2.5-VL-Mull), serving scripts and an
|
| 166 |
+
HF-vs-vLLM parity harness live in
|
| 167 |
+
[`serving/`](https://github.com/arijitray1993/mull/tree/main/serving).
|
| 168 |
+
|
| 169 |
## Citation
|
| 170 |
|
| 171 |
```bibtex
|