Image-Text-to-Text
Transformers
Safetensors
llava_onevision2_5
text-generation
multimodal
video-understanding
streaming
llava-onevision
conversational
custom_code
Instructions to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("image-text-to-text", model="lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True) messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
- SGLang
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }' - Docker Model Runner
How to use lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct with Docker Model Runner:
docker model run hf.co/lmms-lab-encoder/LLaVA-OneVision-2.5-4B-Instruct
Model card: average VideoMME splits, drop video-spatial rows, benchmark figure and results highlight
Browse files- README.md +3 -16
- README_CN.md +3 -16
- assets/video_benchmarks.png +0 -3
README.md
CHANGED
|
@@ -31,19 +31,16 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
|
|
| 31 |
- **Offline and streaming in one model.** `apply_chat_template(..., streaming=True)` switches mode; eight protocol tokens carry the per-frame decision to stay silent, respond or caption. Nothing else about the input changes.
|
| 32 |
- **Codec-aligned video.** Besides uniform frame sampling, *codec mode* uses the encoder's bit-cost to pack the informative regions of a video into a handful of canvases. It beats frame sampling on **17 of 21** video benchmarks.
|
| 33 |
- **9× spatial compression.** A patch-16 vision tower with 3×3 merging gives one visual token per 48×48 pixels, so long videos fit comfortably in the 262k context.
|
| 34 |
-
- **Strong video, grounding and spatial results.** Against Qwen3.5-4B, frame sampling leads on **17 of 21** video benchmarks and codec on **16 of 18**, with the largest margins on temporal grounding (TimeLens Charades +12.5) and spatial video (VSI-Bench +9.4). On still images it is strongest on grounding and spatial tasks; OCR and document reading are not its focus.
|
| 35 |
|
| 36 |
## Performance
|
| 37 |
|
| 38 |
-
<p align="center">
|
| 39 |
-
<img src="assets/video_benchmarks.png" width="92%" alt="Video benchmarks: LLaVA-OneVision-2.5-4B vs Qwen3.5-4B">
|
| 40 |
-
</p>
|
| 41 |
-
|
| 42 |
### Video
|
| 43 |
|
| 44 |
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 45 |
|:---|:---:|:---:|:---:|
|
| 46 |
| **General video understanding** | | | |
|
|
|
|
|
|
|
| 47 |
| MotionBench | 60.1 <sup>128f×153,664</sup> | 62.4 <sup>128f×153,664</sup> | **64.2** <sup>TC128×313,600</sup> |
|
| 48 |
| TOMATO \* | 34.9 <sup>16f×153,664, fps=2</sup> | 33.6 <sup>model default</sup> | **38.6** <sup>TC128×313,600</sup> |
|
| 49 |
| MVBench | 64.0 <sup>64f×153,664</sup> | 72.0 <sup>64f×153,664</sup> | **75.5** <sup>TC64×313,600</sup> |
|
|
@@ -54,16 +51,6 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
|
|
| 54 |
| EgoSchema subset | 68.2 <sup>64f×153,664</sup> | **80.0** <sup>64f×153,664</sup> | -- |
|
| 55 |
| Video-Index 512f | 19.6 <sup>task frames, ≤262,144/f</sup> | **20.0** <sup>task frames, ≤262,144/f</sup> | 19.8 <sup>TC≤256×313,600</sup> |
|
| 56 |
| MMOU test-mini | -- | 46.0 <sup>model default</sup> | -- |
|
| 57 |
-
| **VideoMME** | | | |
|
| 58 |
-
| short, no subtitles | **81.7** <sup>128f×321,489</sup> | 79.7 <sup>128f×321,489</sup> | 81.1 <sup>TC256×313,600</sup> |
|
| 59 |
-
| medium, no subtitles | 70.3 <sup>256f×136,900</sup> | 70.6 <sup>256f×136,900</sup> | **74.1** <sup>TC512×313,600</sup> |
|
| 60 |
-
| long, no subtitles | **63.0** <sup>1450f×120,000</sup> | 59.0 <sup>750f×250,000</sup> | 60.6 <sup>TC512×313,600</sup> |
|
| 61 |
-
| short, subtitles | -- | 82.4 <sup>128f×321,489</sup> | **82.7** <sup>TC256×313,600</sup> |
|
| 62 |
-
| medium, subtitles | -- | 75.4 <sup>256f×136,900</sup> | **75.6** <sup>TC256×313,600</sup> |
|
| 63 |
-
| long, subtitles | -- | 65.8 <sup>750f×250,000</sup> | **65.9** <sup>TC256×313,600</sup> |
|
| 64 |
-
| **Video spatial** | | | |
|
| 65 |
-
| VSI-Bench | 59.2 <sup>128f×153,664</sup> | **68.6** <sup>128f×153,664</sup> | 67.7 <sup>TC128×313,600</sup> |
|
| 66 |
-
| ReVSI 64f | 47.1 <sup>64f×153,664</sup> | **55.9** <sup>64f×153,664</sup> | 55.5 <sup>TC64×313,600</sup> |
|
| 67 |
| **Long video** | | | |
|
| 68 |
| LVBench | 56.5 <sup>2048f×120,000</sup> | **62.0** <sup>950f×200,000</sup> | 61.6 <sup>TC1024×156,800</sup> |
|
| 69 |
| MLVU dev | 73.8 <sup>512f×72,900</sup> | **75.3** <sup>512f×72,900</sup> | -- |
|
|
@@ -87,7 +74,7 @@ LLaVA-OneVision-2.5-4B-Instruct is a 4B vision-language model for single-image,
|
|
| 87 |
| OCRBench | **85.2** | 79.8 | EmbSpatial | 74.7 | **83.0** |
|
| 88 |
| DocVQA val | **95.2** | 92.9 | MMSI-Bench | 31.4 | **33.3** |
|
| 89 |
|
| 90 |
-
<sub>Compared with Qwen3.5-4B, thinking disabled. **Bold** is the best score in each row; `--` means not available. The superscript on each video score is the setting that produced it: frames × pixels per frame, or for codec, canvases (TC) × pixels per canvas. Each side reports its best score over the settings tried, from single runs. `*` On TOMATO the two sides use different frame settings. The image table shows selected benchmarks; full results, settings and protocol will be given in the technical report.</sub>
|
| 91 |
|
| 92 |
## Model Overview
|
| 93 |
|
|
|
|
| 31 |
- **Offline and streaming in one model.** `apply_chat_template(..., streaming=True)` switches mode; eight protocol tokens carry the per-frame decision to stay silent, respond or caption. Nothing else about the input changes.
|
| 32 |
- **Codec-aligned video.** Besides uniform frame sampling, *codec mode* uses the encoder's bit-cost to pack the informative regions of a video into a handful of canvases. It beats frame sampling on **17 of 21** video benchmarks.
|
| 33 |
- **9× spatial compression.** A patch-16 vision tower with 3×3 merging gives one visual token per 48×48 pixels, so long videos fit comfortably in the 262k context.
|
|
|
|
| 34 |
|
| 35 |
## Performance
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
### Video
|
| 38 |
|
| 39 |
| Benchmark | Qwen3.5-4B | OV2.5-4B (frames) | OV2.5-4B (codec) |
|
| 40 |
|:---|:---:|:---:|:---:|
|
| 41 |
| **General video understanding** | | | |
|
| 42 |
+
| VideoMME (no subtitles) | 71.7 | 69.7 | **71.9** |
|
| 43 |
+
| VideoMME (subtitles) | -- | 74.6 | **74.7** |
|
| 44 |
| MotionBench | 60.1 <sup>128f×153,664</sup> | 62.4 <sup>128f×153,664</sup> | **64.2** <sup>TC128×313,600</sup> |
|
| 45 |
| TOMATO \* | 34.9 <sup>16f×153,664, fps=2</sup> | 33.6 <sup>model default</sup> | **38.6** <sup>TC128×313,600</sup> |
|
| 46 |
| MVBench | 64.0 <sup>64f×153,664</sup> | 72.0 <sup>64f×153,664</sup> | **75.5** <sup>TC64×313,600</sup> |
|
|
|
|
| 51 |
| EgoSchema subset | 68.2 <sup>64f×153,664</sup> | **80.0** <sup>64f×153,664</sup> | -- |
|
| 52 |
| Video-Index 512f | 19.6 <sup>task frames, ≤262,144/f</sup> | **20.0** <sup>task frames, ≤262,144/f</sup> | 19.8 <sup>TC≤256×313,600</sup> |
|
| 53 |
| MMOU test-mini | -- | 46.0 <sup>model default</sup> | -- |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
| **Long video** | | | |
|
| 55 |
| LVBench | 56.5 <sup>2048f×120,000</sup> | **62.0** <sup>950f×200,000</sup> | 61.6 <sup>TC1024×156,800</sup> |
|
| 56 |
| MLVU dev | 73.8 <sup>512f×72,900</sup> | **75.3** <sup>512f×72,900</sup> | -- |
|
|
|
|
| 74 |
| OCRBench | **85.2** | 79.8 | EmbSpatial | 74.7 | **83.0** |
|
| 75 |
| DocVQA val | **95.2** | 92.9 | MMSI-Bench | 31.4 | **33.3** |
|
| 76 |
|
| 77 |
+
<sub>Compared with Qwen3.5-4B, thinking disabled. **Bold** is the best score in each row; `--` means not available. The superscript on each video score is the setting that produced it: frames × pixels per frame, or for codec, canvases (TC) × pixels per canvas. VideoMME is the mean over the short, medium and long splits. Each side reports its best score over the settings tried, from single runs. `*` On TOMATO the two sides use different frame settings. The image table shows selected benchmarks; full results, settings and protocol will be given in the technical report.</sub>
|
| 78 |
|
| 79 |
## Model Overview
|
| 80 |
|
README_CN.md
CHANGED
|
@@ -31,19 +31,16 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
|
|
| 31 |
- **离线与流式合一。** `apply_chat_template(..., streaming=True)` 即可切换模式;八个协议 token 承载每一帧"沉默 / 回答 / caption"的决策,输入格式的其余部分完全相同。
|
| 32 |
- **Codec 对齐的视频输入。** 除均匀抽帧外,**codec 模式**利用编码器的 bit-cost,把视频中信息量大的区域打包进少量画布。在 21 个视频 bench 中,它有 **17 个**优于抽帧。
|
| 33 |
- **9 倍空间压缩。** patch 16 的视觉塔加 3×3 合并,每 48×48 像素对应一个视觉 token,长视频可以从容放进 262k 上下文。
|
| 34 |
-
- **视频、定位与空间能力突出。** 对比 Qwen3.5-4B,抽帧在 21 个视频 bench 中领先 **17 个**,codec 在 18 个中领先 **16 个**,优势最大的是时间定位(TimeLens Charades +12.5)和视频空间(VSI-Bench +9.4)。静态图像上的优势集中在定位与空间任务;OCR 与文档阅读不是本模型的重点。
|
| 35 |
|
| 36 |
## 评测结果
|
| 37 |
|
| 38 |
-
<p align="center">
|
| 39 |
-
<img src="assets/video_benchmarks.png" width="92%" alt="视频评测:LLaVA-OneVision-2.5-4B 对比 Qwen3.5-4B">
|
| 40 |
-
</p>
|
| 41 |
-
|
| 42 |
### 视频
|
| 43 |
|
| 44 |
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 45 |
|:---|:---:|:---:|:---:|
|
| 46 |
| **通用视频理解** | | | |
|
|
|
|
|
|
|
| 47 |
| MotionBench | 60.1 <sup>128f×153,664</sup> | 62.4 <sup>128f×153,664</sup> | **64.2** <sup>TC128×313,600</sup> |
|
| 48 |
| TOMATO \* | 34.9 <sup>16f×153,664, fps=2</sup> | 33.6 <sup>model default</sup> | **38.6** <sup>TC128×313,600</sup> |
|
| 49 |
| MVBench | 64.0 <sup>64f×153,664</sup> | 72.0 <sup>64f×153,664</sup> | **75.5** <sup>TC64×313,600</sup> |
|
|
@@ -54,16 +51,6 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
|
|
| 54 |
| EgoSchema subset | 68.2 <sup>64f×153,664</sup> | **80.0** <sup>64f×153,664</sup> | -- |
|
| 55 |
| Video-Index 512f | 19.6 <sup>task frames, ≤262,144/f</sup> | **20.0** <sup>task frames, ≤262,144/f</sup> | 19.8 <sup>TC≤256×313,600</sup> |
|
| 56 |
| MMOU test-mini | -- | 46.0 <sup>model default</sup> | -- |
|
| 57 |
-
| **VideoMME** | | | |
|
| 58 |
-
| short,无字幕 | **81.7** <sup>128f×321,489</sup> | 79.7 <sup>128f×321,489</sup> | 81.1 <sup>TC256×313,600</sup> |
|
| 59 |
-
| medium,无字幕 | 70.3 <sup>256f×136,900</sup> | 70.6 <sup>256f×136,900</sup> | **74.1** <sup>TC512×313,600</sup> |
|
| 60 |
-
| long,无字幕 | **63.0** <sup>1450f×120,000</sup> | 59.0 <sup>750f×250,000</sup> | 60.6 <sup>TC512×313,600</sup> |
|
| 61 |
-
| short,有字幕 | -- | 82.4 <sup>128f×321,489</sup> | **82.7** <sup>TC256×313,600</sup> |
|
| 62 |
-
| medium,有字幕 | -- | 75.4 <sup>256f×136,900</sup> | **75.6** <sup>TC256×313,600</sup> |
|
| 63 |
-
| long,有字幕 | -- | 65.8 <sup>750f×250,000</sup> | **65.9** <sup>TC256×313,600</sup> |
|
| 64 |
-
| **视频空间** | | | |
|
| 65 |
-
| VSI-Bench | 59.2 <sup>128f×153,664</sup> | **68.6** <sup>128f×153,664</sup> | 67.7 <sup>TC128×313,600</sup> |
|
| 66 |
-
| ReVSI 64f | 47.1 <sup>64f×153,664</sup> | **55.9** <sup>64f×153,664</sup> | 55.5 <sup>TC64×313,600</sup> |
|
| 67 |
| **长视频** | | | |
|
| 68 |
| LVBench | 56.5 <sup>2048f×120,000</sup> | **62.0** <sup>950f×200,000</sup> | 61.6 <sup>TC1024×156,800</sup> |
|
| 69 |
| MLVU dev | 73.8 <sup>512f×72,900</sup> | **75.3** <sup>512f×72,900</sup> | -- |
|
|
@@ -87,7 +74,7 @@ LLaVA-OneVision-2.5-4B-Instruct 是一个 4B 视觉语言模型,支持单图
|
|
| 87 |
| OCRBench | **85.2** | 79.8 | EmbSpatial | 74.7 | **83.0** |
|
| 88 |
| DocVQA val | **95.2** | 92.9 | MMSI-Bench | 31.4 | **33.3** |
|
| 89 |
|
| 90 |
-
<sub>对比对象为关闭 thinking 的 Qwen3.5-4B。**加粗**为该行最高分,`--` 表示暂无结果。视频分数的上标是取得该分数时的设置:帧数 × 每帧像素;codec 为画布数(TC)× 每画布像素。两边各取已测设置中的最高分,均为单次运行。`*` TOMATO 两边帧数设置不同。图像表为代表性 bench 的节选,完整结果、设置与评测口径见后续技术报告。</sub>
|
| 91 |
|
| 92 |
## 模型概览
|
| 93 |
|
|
|
|
| 31 |
- **离线与流式合一。** `apply_chat_template(..., streaming=True)` 即可切换模式;八个协议 token 承载每一帧"沉默 / 回答 / caption"的决策,输入格式的其余部分完全相同。
|
| 32 |
- **Codec 对齐的视频输入。** 除均匀抽帧外,**codec 模式**利用编码器的 bit-cost,把视频中信息量大的区域打包进少量画布。在 21 个视频 bench 中,它有 **17 个**优于抽帧。
|
| 33 |
- **9 倍空间压缩。** patch 16 的视觉塔加 3×3 合并,每 48×48 像素对应一个视觉 token,长视频可以从容放进 262k 上下文。
|
|
|
|
| 34 |
|
| 35 |
## 评测结果
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
### 视频
|
| 38 |
|
| 39 |
| Benchmark | Qwen3.5-4B | OV2.5-4B(抽帧) | OV2.5-4B(codec) |
|
| 40 |
|:---|:---:|:---:|:---:|
|
| 41 |
| **通用视频理解** | | | |
|
| 42 |
+
| VideoMME(无字幕) | 71.7 | 69.7 | **71.9** |
|
| 43 |
+
| VideoMME(有字幕) | -- | 74.6 | **74.7** |
|
| 44 |
| MotionBench | 60.1 <sup>128f×153,664</sup> | 62.4 <sup>128f×153,664</sup> | **64.2** <sup>TC128×313,600</sup> |
|
| 45 |
| TOMATO \* | 34.9 <sup>16f×153,664, fps=2</sup> | 33.6 <sup>model default</sup> | **38.6** <sup>TC128×313,600</sup> |
|
| 46 |
| MVBench | 64.0 <sup>64f×153,664</sup> | 72.0 <sup>64f×153,664</sup> | **75.5** <sup>TC64×313,600</sup> |
|
|
|
|
| 51 |
| EgoSchema subset | 68.2 <sup>64f×153,664</sup> | **80.0** <sup>64f×153,664</sup> | -- |
|
| 52 |
| Video-Index 512f | 19.6 <sup>task frames, ≤262,144/f</sup> | **20.0** <sup>task frames, ≤262,144/f</sup> | 19.8 <sup>TC≤256×313,600</sup> |
|
| 53 |
| MMOU test-mini | -- | 46.0 <sup>model default</sup> | -- |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 54 |
| **长视频** | | | |
|
| 55 |
| LVBench | 56.5 <sup>2048f×120,000</sup> | **62.0** <sup>950f×200,000</sup> | 61.6 <sup>TC1024×156,800</sup> |
|
| 56 |
| MLVU dev | 73.8 <sup>512f×72,900</sup> | **75.3** <sup>512f×72,900</sup> | -- |
|
|
|
|
| 74 |
| OCRBench | **85.2** | 79.8 | EmbSpatial | 74.7 | **83.0** |
|
| 75 |
| DocVQA val | **95.2** | 92.9 | MMSI-Bench | 31.4 | **33.3** |
|
| 76 |
|
| 77 |
+
<sub>对比对象为关闭 thinking 的 Qwen3.5-4B。**加粗**为该行最高分,`--` 表示暂无结果。视频分数的上标是取得该分数时的设置:帧数 × 每帧像素;codec 为画布数(TC)× 每画布像素。VideoMME 为 short、medium、long 三个子集的平均。两边各取已测设置中的最高分,均为单次运行。`*` TOMATO 两边帧数设置不同。图像表为代表性 bench 的节选,完整结果、设置与评测口径见后续技术报告。</sub>
|
| 78 |
|
| 79 |
## 模型概览
|
| 80 |
|
assets/video_benchmarks.png
DELETED
Git LFS Details
|