Instructions to use nvidia/VideoITG-8B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/VideoITG-8B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="nvidia/VideoITG-8B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import EagleQwenG
model = EagleQwenG.from_pretrained("nvidia/VideoITG-8B", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/VideoITG-8B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/VideoITG-8B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/VideoITG-8B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/VideoITG-8B

SGLang

How to use nvidia/VideoITG-8B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/VideoITG-8B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/VideoITG-8B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/VideoITG-8B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/VideoITG-8B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use nvidia/VideoITG-8B with Docker Model Runner:
```
docker model run hf.co/nvidia/VideoITG-8B
```

VideoITG-8B / README.md

Zhiding

Update README.md

18ab80c verified 12 months ago

preview code

Raw

History Blame Contribute Delete

3.22 kB

metadata

license: other
license_name: nvlicense
license_link: LICENSE
pipeline_tag: image-text-to-text
library_name: transformers
base_model:
  - Qwen/Qwen2-7B-Instruct
  - google/siglip-so400m-patch14-384
base_model_relation: merge
language:
  - multilingual
tags:
  - VideoITG
  - Eagle
  - VLM

VideoITG-8B

[🌐Homepage] [💻GitHub] [📜Tech Report] [🤗VideoITG-40K]

Introduction

VideoITG-8B is a multimodal video understanding model trained with instructed temporal grounding, equipped with the ability to enhance Video Large Language Models through intelligent frame selection. The model tackles the complexities of real-world video scenarios by aligning frame sampling with user instructions. Please check our paper for more details.

Model Details

Model name: VideoITG-8B
Architecture: Customized Eagle-8B base model, fine-tuned with Instructed Temporal Grounding
Model type: Multimodal Large Language Model with Video Understanding
Languages: English (primary), multilingual (partially)

Model Performance

Model	Base Model	Frames	LongVideoBench	MLVU	VideoMME	CG-Bench
VideoITG-7B	InternVL2.5-8B	32	61.9 (+2.9%)	75.0 (+7.8%)	67.3 (+4.0%)	46.7 (+7.0%)
VideoITG-7B	InternVL2.5-26B	32	63.0 (+1.0%)	78.9 (+6.1%)	69.9 (+2.5%)	48.7 (+6.0%)
VideoITG-7B	LLaVA-Video-7B	32	61.6 (+3.6%)	74.6 (+8.6%)	66.1 (+3.0%)	42.8 (+9.0%)
VideoITG-7B	LLaVA-Video-7B	64	60.9 (+7.4%)	76.3 (+7.6%)	66.4 (+1.9%)	42.9 (+8.1%)

Key Features

Instructed Temporal Grounding: Intelligently selects video frames based on user instructions
Plug-and-Play: Seamlessly integrates with existing video language models
Superior Temporal Understanding: Excels in tasks requiring precise temporal grounding

License

Code: Apache 2.0 License
Model: NVIDIA License - Research preview for non-commercial use only

Citation

If you find this project useful, please cite our work:

@article{wang2025videoitg,
  title     = {VideoITG: Multimodal Video Understanding with Instructed Temporal Grounding},
  author    = {Shihao Wang and Guo Chen and De-An Huang and Zhiqi Li and Minghan Li and Guilin Liu and Jose M. Alvarez and Lei Zhang and Zhiding Yu},
  journal   = {arXiv preprint arXiv:2507.13353},
  year      = {2025}
}

Acknowledgement

Eagle: The codebase we built upon
LMMs-Eval: Many thanks to the LMMs-Lab for the easy-to-use evaluation tools
LLaVA-OneVision and LLaVA-Video: We train our models with data from these great open-source projects