Instructions to use ATH-MaaS/Ovis2.5-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ATH-MaaS/Ovis2.5-9B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ATH-MaaS/Ovis2.5-9B", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("ATH-MaaS/Ovis2.5-9B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ATH-MaaS/Ovis2.5-9B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ATH-MaaS/Ovis2.5-9B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ATH-MaaS/Ovis2.5-9B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ATH-MaaS/Ovis2.5-9B

SGLang

How to use ATH-MaaS/Ovis2.5-9B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ATH-MaaS/Ovis2.5-9B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ATH-MaaS/Ovis2.5-9B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ATH-MaaS/Ovis2.5-9B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ATH-MaaS/Ovis2.5-9B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ATH-MaaS/Ovis2.5-9B with Docker Model Runner:
```
docker model run hf.co/ATH-MaaS/Ovis2.5-9B
```

Fix loading with Transformers v5

#14

by AbhayShukla19 - opened 26 days ago

base: refs/heads/main

←

from: refs/pr/14

Discussion Files changed

+30

-10

AbhayShukla19

26 days ago

This PR makes modeling_ovis2_5.py work with Transformers v5 while keeping behaviour on v4 unchanged.

Fixes

is_parallelizable no longer exists in v5, so loading fails with AttributeError: 'Qwen3ForCausalLM' object has no attribute 'is_parallelizable'. The line is removed.
Ovis2_5.__init__ never called post_init(), so v5 doesn't register the tied lm_head. The checkpoint stores no lm_head weights, so lm_head is left uninitialized and the model produces garbage (as reported in discussion #8).
The tie_weights() override now accepts v5's keyword arguments and calls the base implementation before tying the LLM.
v5 materializes non-persistent buffers with torch.empty_like, so indicator_token_indices and the ViT rotary inv_freq held uninitialized memory (out-of-range visual token ids, CUDA device-side asserts). Both are now recomputed in _init_weights.

Testing (RTX PRO 6000, bf16, greedy decoding, image prompt)

transformers 4.57.6: output is identical before and after this change.
transformers 5.17.0: the original code fails to load; with this change the output matches 4.57.6 (per-token log-probs within ~0.1; the text diverges late on a near-tied token, as expected from bf16 numerics).
vLLM's HF-vs-vLLM comparison tests for Ovis2.5-2B (single image, multi-image, video) pass with this change on transformers 5.17.0.

Fix loading with Transformers v58c9cba5f

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment