| --- |
| library_name: transformers |
| pipeline_tag: image-text-to-text |
| license: apache-2.0 |
| tags: |
| - multimodal |
| - vision-language |
| - image-text-to-text |
| - video-text-to-text |
| - llava |
| - llava-onevision-2 |
| - qwen3 |
| language: |
| - en |
| - zh |
| --- |
| |
| # LLaVA-OneVision-2-8B-Instruct |
|
|
| > [!IMPORTANT] |
| > 🌟 **The core work on this project is led by [Glint Lab](https://www.glint-mvt.com).** |
| > For an interactive online experience, visit **[ve2s.ai](https://ve2s.ai/)**. |
|
|
| A multimodal vision-language model that handles **single images, multi-image, and video** inputs, built on a Qwen3-8B language backbone with a OneVision-style vision encoder. |
|
|
| The model is distributed as a HuggingFace `transformers` checkpoint with custom code (`trust_remote_code=True`). |
|
|
| ## Requirements |
|
|
| ### Base (image + frame-sampling video) |
|
|
| ```bash |
| pip install "transformers>=5.7.0" "torch>=2.4" pillow requests decord |
| ``` |
|
|
| ### Optional: codec video backend |
|
|
| The model ships a second video backend (`video_backend="codec"`) that replaces |
| uniform frame sampling with codec-aware **canvas packing** driven by motion |
| vectors and bit-cost — typically yielding stronger long-video accuracy at the |
| same token budget. To enable it you need two extra pieces: |
|
|
| ```bash |
| # 1. The cv-preinfer CLI (PyPI: codec-video-prep) drives canvas extraction. |
| pip install codec-video-prep opencv-python |
| |
| # 2. A working `ffmpeg` binary must be on PATH. |
| # Verify with: ffmpeg -version |
| ``` |
|
|
| **ffmpeg version:** ffmpeg **4.4.x – 7.x** is recommended. |
|
|
| The codec backend additionally needs **POSIX `flock`** (already present on |
| Linux/macOS) for the on-disk result cache, and roughly **2 GB free disk** under |
| `$ONLINE_CODEC_CACHE_DIR` (defaults to `$HF_HOME/online_codec`) per |
| processed video. |
|
|
| ## Quick start |
|
|
| The repository ships a ready-to-run `demo_inference.py` that covers both image and video paths. |
|
|
| ```bash |
| # Image (default sample image; no auth required) |
| python demo_inference.py |
| |
| # Image, custom file + prompt |
| python demo_inference.py --mode image --media /path/to/cat.jpg \ |
| --prompt "What is the cat doing?" |
| |
| # Video (16 uniformly-sampled frames; max-pixels caps per-frame resolution for memory) |
| python demo_inference.py --mode video --media /path/to/clip.mp4 \ |
| --num-frames 16 --max-pixels 200704 \ |
| --prompt "Describe what happens in this video." |
| ``` |
|
|
| ## Programmatic use |
|
|
| ```python |
| import torch |
| from transformers import AutoProcessor, AutoModelForImageTextToText |
| from PIL import Image |
| |
| MODEL_ID = "lmms-lab-encoder/LLaVA-OneVision-2-8B-Instruct" |
| |
| processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True) |
| model = AutoModelForImageTextToText.from_pretrained( |
| MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda", |
| ).eval() |
| |
| # ----- Image ----- |
| image = Image.open("cat.jpg").convert("RGB") |
| messages = [{"role": "user", "content": [ |
| {"type": "image"}, |
| {"type": "text", "text": "Describe this image in detail."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = processor(text=[text], images=[image], return_tensors="pt", padding=True) |
| inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} |
| |
| out = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| |
| # ----- Video ----- |
| # Lower max_pixels if you hit OOM on long videos. |
| processor.video_processor.max_pixels = 200704 |
| |
| messages = [{"role": "user", "content": [ |
| {"type": "video"}, |
| {"type": "text", "text": "Describe what happens in this video."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| inputs = processor( |
| text=[text], videos=["clip.mp4"], return_tensors="pt", padding=True, |
| num_frames=16, # exact frame count; or use target_fps / max_frames |
| ) |
| inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} |
| out = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| ### Video — codec backend (recommended for long videos) |
|
|
| The codec backend is exposed as a single processor kwarg |
| (`video_backend="codec"`). Everything else — canvas extraction via |
| `cv-preinfer`, on-disk caching, patch-position bookkeeping, chat-template |
| rewriting — happens inside `processor(...)`: |
|
|
| ```python |
| # Make sure: `pip install codec-video-prep opencv-python` and ffmpeg on PATH. |
| messages = [{"role": "user", "content": [ |
| {"type": "video"}, |
| {"type": "text", "text": "Describe what happens in this long video."}, |
| ]}] |
| text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) |
| |
| inputs = processor( |
| text=[text], |
| videos=["long_clip.mp4"], |
| video_backend="codec", |
| max_pixels=150000, # per-canvas pixel budget; lower if OOM |
| return_tensors="pt", |
| padding=True, |
| # Optional: override codec defaults from preprocessor_config.json |
| # codec_config={"target_canvas": 32, "group_size": 32, "images_per_group": 4}, |
| ) |
| inputs = {k: v.to("cuda") if hasattr(v, "to") else v for k, v in inputs.items()} |
| |
| out = model.generate(**inputs, max_new_tokens=256, do_sample=False) |
| print(processor.tokenizer.decode(out[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True)) |
| ``` |
|
|
| Defaults for the codec pipeline live in `preprocessor_config.json` under the |
| `"codec"` key (`target_canvas=32`, `group_size=32`, `images_per_group=4`, |
| `patch=14`, `min_group_frames=8`, `max_group_frames=64`); they can be |
| overridden per call via `codec_config={...}`. |
|
|
| **Short-video behaviour:** if the input video has fewer frames than |
| `target_canvas` requires (or fewer than `min_group_frames`), a `UserWarning` |
| is emitted and inference proceeds with however many canvases `cv-preinfer` |
| can actually form. For very short clips, falling back to the frame-sampling |
| backend is usually a better choice. |
|
|
| ## Notes |
|
|
| - The vision tower is a OneVision-style encoder; the language backbone is **Qwen3-8B**. |
| - `chat_template.jinja` follows the Qwen3 chat format and emits `<|vision_start|>...<|vision_end|>` placeholders; the processor expands them per-frame (frames backend) or per-canvas-patch-run (codec backend). |
| - Two video backends are available via `processor(..., video_backend=...)`: **`"frames"`** (default, uniform sampling) and **`"codec"`** (canvas packing via `cv-preinfer`, requires `codec-video-prep` + `ffmpeg`). |
| - Inference was validated to be bit-exact at the pixel level and prefix-identical at the token level against the original reference implementation, on both backends. |
|
|
| ## Citation |
|
|
| If you find **LLaVA-OneVision-2** useful in your research, please cite the following paper: |
|
|
| ```bibtex |
| @misc{an2026llavaonevision2nextgenerationperceptualintelligence, |
| title = {LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence}, |
| author = {Xiang An and Yin Xie and Feilong Tang and Yunyao Yan and Huajie Tan and Didi Zhu and Changrui Chen and Xiuwei Zhao and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Kaichen Zhang and Wenkang Zhang and Zheng Cheng and Nansen Zhang and Chunsheng Wu and Chunjiang Ge and Zimin Ran and Dehua Song and Chunyuan Li and Shikun Feng and Ming Hu and Zhangquan Chen and Junbo Niu and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng}, |
| year = {2026}, |
| eprint = {2605.25979}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CV}, |
| url = {https://arxiv.org/abs/2605.25979} |
| } |
| ``` |
|
|
| ```bibtex |
| @misc{an2025llavaonevision15fullyopenframework, |
| title = {LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training}, |
| author = {Xiang An and Yin Xie and Kaicheng Yang and Wenkang Zhang and Xiuwei Zhao and Zheng Cheng and Yirui Wang and Songcen Xu and Changrui Chen and Didi Zhu and Chunsheng Wu and Huajie Tan and Chunyuan Li and Jing Yang and Jie Yu and Xiyao Wang and Bin Qin and Yumeng Wang and Zizhen Yan and Ziyong Feng and Ziwei Liu and Bo Li and Jiankang Deng}, |
| year = {2025}, |
| eprint = {2509.23661}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CV}, |
| url = {https://arxiv.org/abs/2509.23661} |
| } |
| ``` |
|
|
| ```bibtex |
| @misc{tang2026onevisionencodercodecalignedsparsityfoundational, |
| title = {OneVision-Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence}, |
| author = {Feilong Tang and Xiang An and Yunyao Yan and Yin Xie and Bin Qin and Kaicheng Yang and Yifei Shen and Yuanhan Zhang and Chunyuan Li and Shikun Feng and Changrui Chen and Huajie Tan and Ming Hu and Manyuan Zhang and Bo Li and Ziyong Feng and Ziwei Liu and Zongyuan Ge and Jiankang Deng}, |
| year = {2026}, |
| eprint = {2602.08683}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CV}, |
| url = {https://arxiv.org/abs/2602.08683} |
| } |
| ``` |
|
|
|
|
| ## License |
|
|
| Apache-2.0 (model weights and code in this repository). The Qwen3-8B base is subject to its own license — see [Qwen/Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B). |
|
|
