InSight-doc-8B

Agentic Visual Perception for Long-Document Understanding

📄 Paper | 💻 Code | 🧩 SFT Data | 🎯 RL Data | 🎬 Replay Demo | 🚀 Live Demo

Understand the big picture.  Focus on the right details.  Answer from the evidence.

Overview

InSight-doc-8B is an 8B-class vision-language agent for long-document understanding. It is initialized from Qwen/Qwen3-VL-8B-Instruct and trained to use visual resolution as an adaptive inference-time resource: start from low-resolution page views, call a zoom-in tool on selected regions, then answer from the returned high-resolution evidence.

This checkpoint is a Hugging Face-compatible Qwen3-VL model. To reproduce the reported agentic behavior, use it with the InSight-doc agent loop and image_zoom_in_tool rather than plain single-turn inference.

Model Details

Item	Value
Model	InSight-doc-8B
Base model	Qwen/Qwen3-VL-8B-Instruct
Architecture	Qwen3-VL vision-language model with tool-use SFT and RL
Primary task	Long-document visual question answering
Agent tool	image_zoom_in_tool for region-level visual evidence acquisition
Training data	InSight-doc-SFT-18k and InSight-doc-RL-19k
License	Apache-2.0

Training Summary

InSight-doc-8B is trained in two stages:

1. Supervised fine-tuning. Full-parameter SFT from Qwen3-VL-8B-Instruct on 17,913 curated zoom-in trajectories. The released recipe freezes the vision tower, uses max sequence length 65,536, sequence parallel size 4, global batch size 32, cosine LR 5e-6 -> 5e-7, and two epochs.
2. Reinforcement learning. RL starts from the SFT checkpoint and trains on 19,236 hard prompts with the same zoom-in agent interface. The released RL recipe uses weighted refill source sampling, temperature 0.7, top-p 0.8, top-k 20, presence penalty 1.5, and 2,000 total RL steps.

The SFT data contains 14,216 answerable and 3,697 unanswerable trajectories. The RL data contains 10,579 answerable and 8,657 unanswerable prompts, with the released sampler targeting an effective 86%/14% answerable/unanswerable mix.

Evaluation Highlights

Across DUDE, MP-DocVQA, MMLongBench-Doc, and LongDocURL, InSight-doc-8B improves over Qwen3-VL-8B by 4.3-16.4 accuracy points under medium-to-low input resolution settings. On MMLongBench-Doc and LongDocURL, it reduces hallucination on unanswerable questions by 40%+ and lowers latency by 41%-68% (1.7x-3.1x speedup) while maintaining an accuracy lead.

Across the four document VQA benchmarks, InSight-doc shifts the accuracy-efficiency Pareto frontier upward and leftward, achieving higher accuracy with shorter sequences and lower latency. Darker points indicate higher initial input DPI.

See the paper and release repository for complete benchmark tables and the exact evaluation configuration.

Demos

• Replay demo: compares InSight-doc-8B and Qwen3-VL-8B side by side on replayed examples.
• Live demo: runs an interactive InSight-doc-8B demo.

Usage

Agentic Evaluation

The recommended path is the release evaluation script, which serves the model with vLLM and runs the shared InSight-doc agent/evaluator:

git clone --recurse-submodules https://github.com/m-Just/InSight-doc.git
cd InSight-doc

pip install -e .
pip install -e ./verl

export MODEL_PATH=InSight-doc/InSight-doc-8B
export VAL_FILES='/path/to/longdocurl.parquet,/path/to/mmlongbench.parquet'
export RESCALES='0.25 0.35 0.5'
export EVAL_CUDA_VISIBLE_DEVICES=0,1,2,3
export OPENAI_API_KEY=...
export OPENAI_BASE_URL=https://.../v1

bash scripts/evaluate_insight_doc.sh

Important vLLM defaults in the release config are 4 replicas, 1 GPU per replica, max_model_len=262144, max_num_seqs=64, chunked prefill, prefix caching, and the same sampling settings used by RL validation.

Plain Transformers Loading

The checkpoint can also be loaded as a regular Qwen3-VL model for basic experimentation, but this does not execute the zoom-in tool loop:

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "InSight-doc/InSight-doc-8B"

processor = AutoProcessor.from_pretrained(model_id)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
)

Use explicit bf16 loading for inference; the uploaded config currently records dtype: float32, so dtype="auto" may load fp32 weights in plain Transformers.

Input Format

The agentic policy expects multimodal chat messages containing low-resolution page images and a tool schema for image_zoom_in_tool. During generation, the model may emit structured tool calls that specify an image index and a 2D bounding box. The runtime crops the requested region from the original page and returns it as a new visual observation.

The released SFT dataset includes the tool schema in a tools column. For RL and evaluation, the same schema is loaded from the release configuration at recipe/vsearch/config/image_zoom_in_tool_config.yaml.

License

The model weights are released under the Apache License 2.0. The checkpoint is derived from Qwen/Qwen3-VL-8B-Instruct, which is also released under Apache-2.0.

Citation

@article{li2026insightdoc,
  title={InSight-doc: Agentic Visual Perception for Long-Document Understanding},
  author={Li, Kaican and Xie, Weiyan and Yao, Lewei and Wu, Jiannan and Hong, Lanqing and Huang, Yongxiang and Zhang, Nevin L.},
  journal={arXiv preprint arXiv:2608.10628},
  year={2026}
}