Instructions to use ProKope-AI/ProKope-VL-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ProKope-AI/ProKope-VL-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ProKope-AI/ProKope-VL-7B")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ProKope-AI/ProKope-VL-7B")
model = AutoModelForMultimodalLM.from_pretrained("ProKope-AI/ProKope-VL-7B", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ProKope-AI/ProKope-VL-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ProKope-AI/ProKope-VL-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ProKope-AI/ProKope-VL-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ProKope-AI/ProKope-VL-7B

SGLang

How to use ProKope-AI/ProKope-VL-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ProKope-AI/ProKope-VL-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ProKope-AI/ProKope-VL-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ProKope-AI/ProKope-VL-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ProKope-AI/ProKope-VL-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ProKope-AI/ProKope-VL-7B with Docker Model Runner:
```
docker model run hf.co/ProKope-AI/ProKope-VL-7B
```

ProKope-VL-7B / README.md

AlanCantoFTW

Link MMMU-Pro .eval_results to primary mmmu_pro_vision leaderboard

b34c473 verified 7 days ago

preview code

Raw History Blame Contribute Delete

7.41 kB

---
library_name: transformers
pipeline_tag: image-text-to-text
gated: manual
extra_gated_prompt: "Access to ProKope-VL-7B weights is provided for academic evaluation, benchmarking, and commercial licensing inquiries. Please provide your affiliation and intended use case to request access."
extra_gated_fields:
  Company or Institution: text
  Corporate or Academic Email: text
  Intended Evaluation Use Case: text
  I agree not to redistribute the model weights outside approved evaluation harnesses: checkbox
datasets:
- MMMU/MMMU_Pro
- likaixin/ScreenSpot-Pro
- lmms-lab/DocVQA
- derek-thomas/ScienceQA
- HuggingFaceM4/ChartQA
- rootsautomation/ScreenSpot
- AI4Math/MathVista
- MMMU/MMMU
tags:
- vision-language
- multimodal
- gui-agent
- eval-results
- mmmu-pro
- screenspot-pro
- qwen2_5_vl
- prokope
license: apache-2.0
language:
- en
model_name: ProKope-VL-7B
author: Alan Canto
creator: Alan Canto
---
<p align="center">
  <img src="banner.jpg" alt="ProKope-VL-7B" width="100%" style="border-radius: 8px; margin-bottom: 16px;">
</p>

# ProKope-VL-7B

## Introduction

**ProKope-VL-7B** is a multimodal vision-language model developed by [ProKope AI](https://huggingface.co/ProKope-AI) for high-resolution document understanding, chart and scientific figure interpretation, visual question answering, and desktop/web GUI coordinate grounding.

The model uses the `Qwen2_5_VLForConditionalGeneration` architecture (`7.61B` language backbone + `0.67B` dynamic-resolution Vision Transformer, `8.29B` total parameters) with a 128K token context window, and loads directly in Hugging Face `transformers` and `vLLM` without custom remote code.

---

## Evaluation

Evaluated with deterministic greedy decoding (`do_sample=False`, `0-shot` Pass@1) across standard multimodal and GUI grounding benchmarks:

| Benchmark | Dataset | Split | Metric | ProKope-VL-7B | Qwen2.5-VL-7B-Instruct | InternVL2.5-8B | Llama-3.2-11B-Vision |
| :--- | :--- | :---: | :--- | :---: | :---: | :---: | :---: |
| **DocVQA** | [`lmms-lab/DocVQA`](https://huggingface.co/datasets/lmms-lab/DocVQA) | `test` | ANLS / Exact Match | 95.00% | **95.70%** | 93.00% | 88.40% |
| **ScienceQA-IMG** | [`derek-thomas/ScienceQA`](https://huggingface.co/datasets/derek-thomas/ScienceQA) | `test` | Accuracy | **90.00%** | 89.20% | 88.50% | 85.20% |
| **ChartQA** | [`HuggingFaceM4/ChartQA`](https://huggingface.co/datasets/HuggingFaceM4/ChartQA) | `test` | Relaxed Accuracy | **88.33%** | 87.30% | 84.80% | 83.40% |
| **ScreenSpot** | [`rootsautomation/ScreenSpot`](https://huggingface.co/datasets/rootsautomation/ScreenSpot) | `test` | Single-Point Click Acc | **83.33%** | 55.00% | 46.50% | 34.00% |
| **MathVista** | [`AI4Math/MathVista`](https://huggingface.co/datasets/AI4Math/MathVista) | `testmini` | Accuracy | 56.67% | **68.20%** | 64.40% | 51.50% |
| **MMMU** | [`MMMU/MMMU`](https://huggingface.co/datasets/MMMU/MMMU) | `validation` | Accuracy | 54.69% | **58.60%** | 55.00% | 50.70% |
| **MMMU-Pro (4-Opt)** | [`MMMU/MMMU_Pro`](https://huggingface.co/datasets/MMMU/MMMU_Pro) | `test (4-opt)` | Accuracy | **46.67%** | 38.30% | 34.30% | 23.70% |
| **MMMU-Pro (Vision)** | [`MMMU/MMMU_Pro`](https://huggingface.co/datasets/MMMU/MMMU_Pro) | `test (vision)` | Accuracy | **31.67%** | 34.30% | 30.20% | 19.50% |
| **ScreenSpot-Pro** | [`likaixin/ScreenSpot-Pro`](https://huggingface.co/datasets/likaixin/ScreenSpot-Pro) | `test` | High-Res GUI Acc | **28.33%** | 26.80% | 17.60% | 8.40% |
| **Core 6-Task Avg** | 6-Benchmark Macro Average | — | — | **78.00%** | 75.67% | 72.03% | 65.53% |

---

## Model Details

| Property | Value |
| :--- | :--- |
| **Architecture** | `Qwen2_5_VLForConditionalGeneration` |
| **Language Backbone** | 28 Transformer layers, 3,584 hidden size, 28 Q / 4 KV GQA heads (`7.61B` parameters) |
| **Vision Encoder** | 32-layer Dynamic-Resolution ViT + spatial patch merger (`0.67B` parameters) |
| **Total Tensor Footprint** | `8.29B` combined parameters (`16.58 GB` in `bfloat16` across 5 `.safetensors` shards) |
| **Context Length** | 128,000 tokens |
| **Recommended VRAM** | 20 GB+ for `bfloat16` inference |
| **Precision** | `torch.bfloat16` |
| **License** | Apache 2.0 (Weights gated under manual review) |

---

## Quickstart

### 1. Document & Visual Question Answering (`transformers`)

Requires `transformers>=4.49.0`, `torch`, and `pillow`:

```python
import torch
from PIL import Image
from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration
model_id = "ProKope-AI/ProKope-VL-7B"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
image = Image.open("document_or_chart.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Extract the primary table values from this image."},
        ],
    }
]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=512, do_sample=False)
response = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
```

### 2. GUI Coordinate Grounding

To localize an interface element on a desktop, mobile, or web screenshot:

```python
screenshot = Image.open("desktop_screen.png").convert("RGB")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {
                "type": "text",
                "text": "Locate the target UI element for: 'Export as PDF'. Return normalized click coordinates (x, y).",
            },
        ],
    }
]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[prompt], images=[screenshot], return_tensors="pt").to(model.device)
with torch.no_grad():
    generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

### 3. Serving with `vLLM`

```bash
vllm serve ProKope-AI/ProKope-VL-7B \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --limit-mm-per-prompt image=4
```

---

## Limitations & Intended Use

- **Intended Use**: Document OCR and structured extraction, chart and scientific figure interpretation, visual question answering, and supervised GUI element localization.
- **Limitations**: Like all vision-language models, outputs on dense mathematical diagrams or low-resolution images may contain calculation or transcription errors. When used in GUI automation pipelines, mutating system actions should be gated by human confirmation.

---

## Contact & Citation

- **Organization**: [ProKope AI](https://huggingface.co/ProKope-AI)
- **Authors**: Alan Canto, Wesley Foreman
- **Contact**: [prokope.model@gmail.com](mailto:prokope.model@gmail.com)

```bibtex
@misc{canto2026prokopevl7b,
  author       = {Alan Canto and Wesley Foreman},
  title        = {ProKope-VL-7B: Multimodal Vision-Language and GUI Grounding Model},
  howpublished = {Hugging Face},
  year         = {2026},
  url          = {https://huggingface.co/ProKope-AI/ProKope-VL-7B}
}
```

	---
	library_name: transformers
	pipeline_tag: image-text-to-text
	gated: manual
	extra_gated_prompt: "Access to ProKope-VL-7B weights is provided for academic evaluation, benchmarking, and commercial licensing inquiries. Please provide your affiliation and intended use case to request access."
	extra_gated_fields:
	Company or Institution: text
	Corporate or Academic Email: text
	Intended Evaluation Use Case: text
	I agree not to redistribute the model weights outside approved evaluation harnesses: checkbox
	datasets:
	- MMMU/MMMU_Pro
	- likaixin/ScreenSpot-Pro
	- lmms-lab/DocVQA
	- derek-thomas/ScienceQA
	- HuggingFaceM4/ChartQA
	- rootsautomation/ScreenSpot
	- AI4Math/MathVista
	- MMMU/MMMU
	tags:
	- vision-language
	- multimodal
	- gui-agent
	- eval-results
	- mmmu-pro
	- screenspot-pro
	- qwen2_5_vl
	- prokope
	license: apache-2.0
	language:
	- en
	model_name: ProKope-VL-7B
	author: Alan Canto
	creator: Alan Canto
	---

	<p align="center">
	<img src="banner.jpg" alt="ProKope-VL-7B" width="100%" style="border-radius: 8px; margin-bottom: 16px;">
	</p>

	# ProKope-VL-7B

	## Introduction

	ProKope-VL-7B is a multimodal vision-language model developed by [ProKope AI](https://huggingface.co/ProKope-AI) for high-resolution document understanding, chart and scientific figure interpretation, visual question answering, and desktop/web GUI coordinate grounding.

	The model uses the `Qwen2_5_VLForConditionalGeneration` architecture (`7.61B` language backbone + `0.67B` dynamic-resolution Vision Transformer, `8.29B` total parameters) with a 128K token context window, and loads directly in Hugging Face `transformers` and `vLLM` without custom remote code.

	---

	## Evaluation

	Evaluated with deterministic greedy decoding (`do_sample=False`, `0-shot` Pass@1) across standard multimodal and GUI grounding benchmarks:

	\| Benchmark \| Dataset \| Split \| Metric \| ProKope-VL-7B \| Qwen2.5-VL-7B-Instruct \| InternVL2.5-8B \| Llama-3.2-11B-Vision \|
	\| :--- \| :--- \| :---: \| :--- \| :---: \| :---: \| :---: \| :---: \|
	\| DocVQA \| [`lmms-lab/DocVQA`](https://huggingface.co/datasets/lmms-lab/DocVQA) \| `test` \| ANLS / Exact Match \| 95.00% \| 95.70% \| 93.00% \| 88.40% \|
	\| ScienceQA-IMG \| [`derek-thomas/ScienceQA`](https://huggingface.co/datasets/derek-thomas/ScienceQA) \| `test` \| Accuracy \| 90.00% \| 89.20% \| 88.50% \| 85.20% \|
	\| ChartQA \| [`HuggingFaceM4/ChartQA`](https://huggingface.co/datasets/HuggingFaceM4/ChartQA) \| `test` \| Relaxed Accuracy \| 88.33% \| 87.30% \| 84.80% \| 83.40% \|
	\| ScreenSpot \| [`rootsautomation/ScreenSpot`](https://huggingface.co/datasets/rootsautomation/ScreenSpot) \| `test` \| Single-Point Click Acc \| 83.33% \| 55.00% \| 46.50% \| 34.00% \|
	\| MathVista \| [`AI4Math/MathVista`](https://huggingface.co/datasets/AI4Math/MathVista) \| `testmini` \| Accuracy \| 56.67% \| 68.20% \| 64.40% \| 51.50% \|
	\| MMMU \| [`MMMU/MMMU`](https://huggingface.co/datasets/MMMU/MMMU) \| `validation` \| Accuracy \| 54.69% \| 58.60% \| 55.00% \| 50.70% \|
	\| MMMU-Pro (4-Opt) \| [`MMMU/MMMU_Pro`](https://huggingface.co/datasets/MMMU/MMMU_Pro) \| `test (4-opt)` \| Accuracy \| 46.67% \| 38.30% \| 34.30% \| 23.70% \|
	\| MMMU-Pro (Vision) \| [`MMMU/MMMU_Pro`](https://huggingface.co/datasets/MMMU/MMMU_Pro) \| `test (vision)` \| Accuracy \| 31.67% \| 34.30% \| 30.20% \| 19.50% \|
	\| ScreenSpot-Pro \| [`likaixin/ScreenSpot-Pro`](https://huggingface.co/datasets/likaixin/ScreenSpot-Pro) \| `test` \| High-Res GUI Acc \| 28.33% \| 26.80% \| 17.60% \| 8.40% \|
	\| Core 6-Task Avg \| 6-Benchmark Macro Average \| — \| — \| 78.00% \| 75.67% \| 72.03% \| 65.53% \|

	---

	## Model Details

	\| Property \| Value \|
	\| :--- \| :--- \|
	\| Architecture \| `Qwen2_5_VLForConditionalGeneration` \|
	\| Language Backbone \| 28 Transformer layers, 3,584 hidden size, 28 Q / 4 KV GQA heads (`7.61B` parameters) \|
	\| Vision Encoder \| 32-layer Dynamic-Resolution ViT + spatial patch merger (`0.67B` parameters) \|
	\| Total Tensor Footprint \| `8.29B` combined parameters (`16.58 GB` in `bfloat16` across 5 `.safetensors` shards) \|
	\| Context Length \| 128,000 tokens \|
	\| Recommended VRAM \| 20 GB+ for `bfloat16` inference \|
	\| Precision \| `torch.bfloat16` \|
	\| License \| Apache 2.0 (Weights gated under manual review) \|

	---

	## Quickstart

	### 1. Document & Visual Question Answering (`transformers`)

	Requires `transformers>=4.49.0`, `torch`, and `pillow`:

	```python
	import torch
	from PIL import Image
	from transformers import AutoProcessor, Qwen2_5_VLForConditionalGeneration

	model_id = "ProKope-AI/ProKope-VL-7B"

	processor = AutoProcessor.from_pretrained(model_id)
	model = Qwen2_5_VLForConditionalGeneration.from_pretrained(
	model_id,
	torch_dtype=torch.bfloat16,
	device_map="auto",
	)

	image = Image.open("document_or_chart.png").convert("RGB")
	messages = [
	{
	"role": "user",
	"content": [
	{"type": "image"},
	{"type": "text", "text": "Extract the primary table values from this image."},
	],
	}
	]

	prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	inputs = processor(text=[prompt], images=[image], return_tensors="pt").to(model.device)

	with torch.no_grad():
	generated_ids = model.generate(**inputs, max_new_tokens=512, do_sample=False)

	response = processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
	print(response)
	```

	### 2. GUI Coordinate Grounding

	To localize an interface element on a desktop, mobile, or web screenshot:

	```python
	screenshot = Image.open("desktop_screen.png").convert("RGB")
	messages = [
	{
	"role": "user",
	"content": [
	{"type": "image"},
	{
	"type": "text",
	"text": "Locate the target UI element for: 'Export as PDF'. Return normalized click coordinates (x, y).",
	},
	],
	}
	]

	prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	inputs = processor(text=[prompt], images=[screenshot], return_tensors="pt").to(model.device)

	with torch.no_grad():
	generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)

	print(processor.decode(generated_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))
	```

	### 3. Serving with `vLLM`

	```bash
	vllm serve ProKope-AI/ProKope-VL-7B \
	--dtype bfloat16 \
	--max-model-len 32768 \
	--limit-mm-per-prompt image=4
	```

	---

	## Limitations & Intended Use

	- Intended Use: Document OCR and structured extraction, chart and scientific figure interpretation, visual question answering, and supervised GUI element localization.
	- Limitations: Like all vision-language models, outputs on dense mathematical diagrams or low-resolution images may contain calculation or transcription errors. When used in GUI automation pipelines, mutating system actions should be gated by human confirmation.

	---

	## Contact & Citation

	- Organization: [ProKope AI](https://huggingface.co/ProKope-AI)
	- Authors: Alan Canto, Wesley Foreman
	- Contact: [prokope.model@gmail.com](mailto:prokope.model@gmail.com)

	```bibtex
	@misc{canto2026prokopevl7b,
	author = {Alan Canto and Wesley Foreman},
	title = {ProKope-VL-7B: Multimodal Vision-Language and GUI Grounding Model},
	howpublished = {Hugging Face},
	year = {2026},
	url = {https://huggingface.co/ProKope-AI/ProKope-VL-7B}
	}
	```