---
license: apache-2.0
language:
- en
pipeline_tag: image-text-to-text
library_name: transformers
tags:
- multimodal
- document-understanding
---

# DocPilot-4B

DocPilot-4B is a 4B action-tag model for long visual document understanding.
Weights are in the `DocPilot-4B` folder.

- Paper: https://arxiv.org/abs/2610.21327
- Code: https://github.com/HKUSTDial/DocPilot

## Requirements

```bash
pip install vllm transformers
```

## Quickstart

```python
from transformers import AutoProcessor, AutoModelForImageTextToText

model = AutoModelForImageTextToText.from_pretrained(
    "SetonLiang2/DocPilot",
    subfolder="DocPilot-4B",
    torch_dtype="auto",
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(
    "SetonLiang2/DocPilot",
    subfolder="DocPilot-4B",
)
```

## Deployment

vLLM reads a local folder directly. Download the subfolder, then serve it:

```bash
huggingface-cli download SetonLiang2/DocPilot --include "DocPilot-4B/*" --local-dir ./DocPilot
vllm serve ./DocPilot/DocPilot-4B --host 0.0.0.0 --port 8088 --dtype bfloat16
```

Point DocPilot at this server with `protocol: vllm_tags`:

```yaml
llm:
  provider: openai
  protocol: vllm_tags
  model: ./DocPilot/DocPilot-4B
  base_url: http://127.0.0.1:8088/v1
  api_key: EMPTY
```

## Citation

```bibtex
@article{liang2026docpilot,
  title={DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding},
  author={Liang, Zhuowen and Lin, Xiaotian and Luo, Yuyu and Wang, Haixun and Tang, Nan},
  journal={arXiv preprint arXiv:2610.21327},
  year={2026}
}
```