\documentclass{article}

% if you need to pass options to natbib, use, e.g.:
%     \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2026

% The authors should use one of these tracks.
% Before accepting by the NeurIPS conference, select one of the options below.
% 0. "default" for submission
% \usepackage{neurips_2026}
\PassOptionsToPackage{numbers, compress}{natbib}
% \usepackage{neurips_2026}
% the "default" option is equal to the "main" option, which is used for the Main Track with double-blind reviewing.
% 1. "main" option is used for the Main Track
%  \usepackage[main]{neurips_2026}
% 2. "position" option is used for the Position Paper Track
%  \usepackage[position]{neurips_2026}
% 3. "eandd" option is used for the Evaluations & Datasets Track
 % \usepackage[eandd]{neurips_2026}

% "preprint" option is used for arXiv or other preprint submissions
 \usepackage[preprint]{neurips_2026}

\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc}    % use 8-bit T1 fonts
\usepackage{hyperref}       % hyperlinks
\usepackage{url}            % simple URL typesetting
\usepackage{booktabs}       % professional-quality tables
\usepackage{amsfonts}       % blackboard math symbols
\usepackage{nicefrac}       % compact symbols for 1/2, etc.
\usepackage{microtype}      % microtypography
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{bm}
\usepackage{graphicx}
\usepackage{amsthm}
\usepackage{subcaption}
\usepackage{algorithmicx}
\usepackage{algpseudocode}
\usepackage{algorithm}
\usepackage{pifont}
\usepackage{diagbox}
\usepackage{multirow}
\usepackage[table]{xcolor}
\usepackage{array}
\usepackage{makecell}

\newcommand{\cmark}{\ding{51}}%
\newcommand{\xmark}{\ding{55}}%

\usepackage{comment}

\title{Efficient Post-Training for Fine-Grained Multimodal \\ Foundation Models}


\author{%
\textbf{Xiaohuan Pei}$^{1}$ \quad
\textbf{Jiyuan Zhang}$^{2}$ \quad
\textbf{Yuanfan Guo}$^{2}$ \quad
\textbf{Weiguo Feng}$^{2}$ \\
\textbf{Tao Huang}$^{3}$ \quad
\textbf{Cho-Jui Hsieh}$^{4}$ \quad
\textbf{Chang Xu}$^{1}$ \\[0.5em]
$^{1}$The University of Sydney \quad
$^{2}$ByteDance \\
$^{3}$Shanghai Jiao Tong University \quad
$^{4}$University of California, Los Angeles \\[0.3em]
\texttt{xiaohuan.pei@sydney.edu.au} 
% \texttt{\{xiaohuan.pei\}@sydney.edu.au} \quad
% \texttt{\{xxx,xxx,xxx\}@xxx.edu} \\
% \texttt{t.huang@sjtu.edu.cn} \quad
% \texttt{chohsieh@cs.ucla.edu}
}


\begin{document}


\maketitle


\begin{abstract}
Fine-grained visual grounding remains a major bottleneck for multimodal foundation models, limiting dense-text reading, small-object localization, document understanding, GUI agents, and embodied perception. Existing methods improve grounding by increasing visual resolution, adding region-level modules, or performing end-to-end retraining, yet they provide limited control over how query-relevant regional cues are selected from intermediate representations and reused by deeper language-generating layers. We observe that effective grounding requires intermediate region-level perception to remain aligned with deep-layer generation. Based on this observation, we propose \textbf{GroundFlow}, a lightweight post-training method for intermediate-guided region-aware token reuse. GroundFlow supervises query-relevant visual token selection with calibrated attention maps, aligns middle- and deep-layer distributions through temperature-scaled KL distillation, and injects retrieved visual-token importance into deep layers via importance-weighted attention. During inference, the retrieved regional tokens are reused in a second pass to strengthen fine-grained visual grounding. Experiments on mainstream grounding-related multimodal benchmarks show that GroundFlow consistently improves foundation models, achieving substantial gains on fine-grained visual grounding evaluation (\textit{e.g.}, +20.4 points on V*Star at the 4B scale).
Ablations show that the gains arise from the coupling of calibrated region-level supervision, intermediate-to-deep distributional alignment, and importance-guided deep attention, which establishes GroundFlow as a scalable post-training paradigm for fine-grained visual grounding through region-aware token retrieval and reuse.
\end{abstract}

\section{Introduction}
\label{sec:introduction}

% \begin{figure}[t]
% \centering
% \begin{subfigure}[c]{0.49\linewidth}
% \centering
% \includegraphics[height=6.0cm,keepaspectratio]{figures/fig1_groundflow_pipeline.pdf}
% \caption{Retrieval-and-reuse pipeline.}
% \label{fig:teaser_pipeline}
% \end{subfigure}\hfill
% \begin{subfigure}[c]{0.49\linewidth}
% \centering
% \includegraphics[height=6.0cm,keepaspectratio]{figures/fig1_radar.pdf}
% \caption{Benchmark performance comparison.}
% \label{fig:teaser_radar}
% \end{subfigure}
% \caption{Previous vs. Ours
% }
% \label{fig:teaser_overview}
% \end{figure}

Large multimodal foundation models have achieved remarkable progress in visual reasoning, document understanding, scene understanding, and instruction following~\cite{liu2023visual,an2025llava,oquab2023dinov2,Qwen-VL,Qwen2-VL,Qwen2.5-VL}.
Despite this progress, fine-grained visual grounding remains a major challenge.
Tasks involving small objects, dense text, spatial relations, or region-level reasoning require models to identify a small number of query-relevant visual regions from many redundant visual tokens.

However, current models often rely on coarse visual semantics, leading to weakly grounded responses in high-resolution and cluttered scenes~\cite{shi2026sdrpn}.
Recently, some work has improved fine-grained grounding through end-to-end retraining, high-resolution continued training, stronger vision encoders, region-level modules, or auxiliary localization supervision~\cite{zhang2025mllms,liu2024llavanext,lu2024deepseek,zhao2024mg}.
While effective, these approaches remain expensive, data-intensive, and intrusive, and leave the transfer of text-related grounding regions from intermediate representations to deep-layer generation underexplored.

% Consequently, query-relevant regions may be implicitly captured in middle layers but diluted before contributing to final generation.
% We argue that fine-grained grounding can be improved by retrieving query-relevant grounding regions from intermediate representations and reusing them for deep-layer generation. 
% This provides a targeted alternative to full end-to-end retraining, as intermediate layers often preserve richer spatial structures while deeper layers focus more on linguistic prediction~\cite{}. 
We argue that fine-grained grounding can be improved without full end-to-end retraining by retrieving query-relevant grounding regions from spatially rich intermediate representations and reusing them for deep-layer generation.
Based on this observation, we propose \textbf{GroundFlow}, a lightweight post-training framework that connects intermediate region localization with deep-layer generation.
GroundFlow trains a middle-layer \emph{region-aware token predictor} with calibrated region-level attention maps to estimate query-relevant visual tokens. 
It aligns the predictor-side intermediate distribution with the deep backbone distribution through temperature-scaled KL distillation, and introduces \emph{Importance-Weighted Attention} (IWA) to inject predicted token importance into deep-layer attention. 
During inference, the first forward retrieves query-relevant visual tokens, while the second forward reuses shallow-layer KV caches and inserts the retrieved tokens as additional regional context. 

We summarize our contributions as follows:
(1) We directly train a middle-layer grounding predictor on hidden-state visual tokens with calibrated region-level supervision, avoiding intrusive full-model retraining.
(2) We align intermediate grounding prediction with deep-layer generation through temperature-scaled KL distillation.
(3) We introduce IWA and two-forward token reuse, allowing retrieved visual tokens to guide deep attention during training and provide additional regional context during inference.

\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{figures/prev.png}
\caption{
\textbf{Previous vs. Ours: } 
Previous methods use end-to-end supervision, so grounding signals reach the model only through the final output. Our method instead adopts end-to-middle supervision and explicit shallow–to-deep interaction, enabling middle-layer grounding representations to directly guide deep-layer generation.
}
\label{fig:main}
\end{figure*}





\section{Related Work}
\label{sec:related_work}


\subsection{General Fine-Grained Multimodal Perception}
Classic Multimodal Large Language Models (MLLMs)\cite{liu2023llava,liu2023improvedllava, dai2023instructblip}, such as LLaVA, typically employ CLIP\cite{radford2021learning} or its variants~\cite{radford2021learning,tschannen2025siglip,zhai2023sigmoid,bolya2025perception} as vision encoders to align visual features with language representations. However, constrained by the limited input resolution of these encoders, such models struggle with tasks that demand fine-grained perception, including OCR~\cite{liu2024ocrbench}, document understanding~\cite{masry2022chartqa,mathew2020docvqa,mathew2022infographicvqa}, and GUI localization~\cite{li2025screenspot}. To mitigate this, subsequent research has introduced split-and-tile strategies~\cite{zhu2025internvl3, li2024llava, liu2024llavanext}, multiple vision encoders with multi-resolution capabilities~\cite{zhang2024llava, lu2024deepseek,zhao2024mg}, and native-resolution Vision Transformers (ViTs)\cite{Qwen2-VL,dehghani2023patch}, which significantly enhance fine-grained perception. Nonetheless, because standard vision encoders compress images at a fixed ratio, processing high-resolution inputs generates thousands of visual tokens, leading to substantial computational overhead. Moreover, minute visual details are often overwhelmed by a vast number of background tokens, impeding the LLM's ability to retrieve relevant information\cite{liu2025hide}.

\subsection{Grounding-Based Multimodal Perception and Importance-Reweighted Attention}
To address these challenges, recent works have proposed a ground-then-perceive framework~\cite{zhang2025mllms, jiang2025token, hrbench, shi2025scaling, shi2026sdrpn, shi2026q}. In this approach, small visual evidence is first localized using external models~\cite{jiang2025token}, sliding windows~\cite{hrbench}, or the MLLM's internal attention mechanisms~\cite{zhang2025mllms}. These grounded regions are then re-encoded to provide complementary inputs that bolster the model's perceptual accuracy. Beyond utilizing grounded crops as complementary inputs, some works treat saliency scores as a frozen preprocessing step to reweight attention inside the MLLM, thereby amplifying important visual tokens at every backbone layer through a hand-designed scalar bias~\cite{kang2025see}. Despite their effectiveness, these methods typically rely on handcrafted pipelines and multiple prefilling passes, which introduce significant efficiency bottlenecks and restrict their overall generality.
Another line of research~\cite{zheng2025deepeyes, zhang2025thyme, lai2025mini, wei2026zooming} draws inspiration from OpenAI's Thinking-with-Images paradigm~\cite{openai2025thinking}, proposing an interleaved text-and-image reasoning framework optimized via RL-based verified rewards~\cite{shao2024deepseekmath, guo2025deepseek}. However, this paradigm is computationally expensive during both training and inference. End-to-end optimization requires substantial GPU memory and carefully curated datasets, while the long chain-of-thought reasoning heavily burdens the decoding stage.
Departing from these computationally intensive and handcrafted pipelines, GroundFlow introduces a lightweight post-training procedure that jointly leverages grounded areas as complementary inputs and applies a trainable additive attention bias to the highlighted foreground regions. By avoiding redundant computations, this dual approach ensures a highly efficient visual grounding process without compromising the inherent generality of the MLLM.


\definecolor{selectcolor}{RGB}{200,50,0}     % red    — grounding predictor / token retrieval
\definecolor{aligncolor}{RGB}{0,100,200}     % blue   — intermediate-to-deep KL distillation
\definecolor{reusecolor}{RGB}{0,150,50}      % green  — IWA bias / token reuse
\definecolor{ourslavender}{RGB}{184,174,234} % lavender — highlight for GroundFlow rows in tables

\section{Method}
\label{sec:method}

\begin{figure*}[t]
\centering
\includegraphics[width=\linewidth]{figures/post1_.png}
\caption{
\textbf{Overview of GroundFlow.}
GroundFlow attaches a middle-layer grounding predictor to a frozen multimodal backbone for query-relevant visual token retrieval.
The predictor is supervised by calibrated region-level attention maps and aligned with deep-layer generation through intermediate-to-deep KL distillation.
GroundFlow further injects predicted token importance $s_j$ into deep-layer attention via an importance-weighted attention (IWA) bias $\gamma_l\!\cdot\!s_j$.
During inference, the first forward retrieves query-relevant visual tokens, while the second forward reuses them as additional regional context for grounded generation.
}
\label{fig:main}
\end{figure*}

We aim to improve fine-grained visual grounding without full end-to-end retraining by retrieving query-relevant grounding regions from intermediate representations and reusing them for deep-layer generation.
As illustrated in Figure~\ref{fig:main}, GroundFlow consists of three coupled components.
First, a middle-layer grounding predictor estimates query-relevant visual tokens using calibrated region-level attention maps.
Second, intermediate-to-deep KL distillation aligns the predictor-side distribution with the deep backbone distribution.
Third, importance-weighted attention injects the predicted token importance into deep-layer attention as learnable biases.
During inference, GroundFlow performs two sequential forwards: the first forward retrieves query-relevant visual tokens, and the second forward reuses shallow-prefix KV caches while inserting the retrieved tokens as additional regional context.

% ============================================================
\subsection{Preliminary}
\label{subsec:preliminary}
% ============================================================

Let
$\mathbf{X}^{(0)}=[\mathbf{H}_{\mathrm{sys}},\mathbf{H}_{v},\mathbf{H}_{t}]
\in\mathbb{R}^{N\times d}$
denote the multimodal input sequence, where
$\mathbf{H}_{\mathrm{sys}}$, $\mathbf{H}_{v}$, and $\mathbf{H}_{t}$
are the system, visual, and text embeddings.
Here, $N$ is the sequence length and $d$ is the hidden dimension.
We denote the set of visual-token positions by
$\mathcal{I}_{v}\subseteq\{1,\ldots,N\}$ and the valid language-supervision positions by
$\mathcal{I}_{y}=\{t:y_{t+1}\neq -100\}$.
The backbone contains $L$ causal decoder layers
$\{f_l\}_{l=0}^{L-1}$, a final normalization layer
$\mathrm{Norm}(\cdot)$, and a language-model head
$W_{\mathrm{lm}}\in\mathbb{R}^{|\mathcal{Y}|\times d}$,
where $|\mathcal{Y}|$ is the vocabulary size.
GroundFlow attaches a grounding predictor at an intermediate layer $K$.
The first $K$ backbone layers form a shared prefix:
\begin{equation}
    \mathbf{X}^{(K)}
    =
    f_{K-1}\circ\cdots\circ f_0(\mathbf{X}^{(0)}).
\end{equation}
From $\mathbf{X}^{(K)}$, the computation separates into a deep backbone path and a shallow grounding-predictor path.
The predictor contains $T$ decoder layers
$\{\hat f_k\}_{k=0}^{T-1}$, initialized from the corresponding backbone layers
$\{f_K,\ldots,f_{K+T-1}\}$:
\begin{equation}
    \mathbf{H}_{\mathrm{bb}}^{(K)}
    =
    \mathbf{H}_{\mathrm{gp}}^{(0)}
    =
    \mathbf{X}^{(K)},
    \qquad
    \mathbf{H}_{\mathrm{bb}}^{(l+1)}
    =
    f_l(\mathbf{H}_{\mathrm{bb}}^{(l)}),
    \quad
    l=K,\ldots,L-1,
\end{equation}
\begin{equation}
    \mathbf{H}_{\mathrm{gp}}^{(k+1)}
    =
    \hat f_k(\mathbf{H}_{\mathrm{gp}}^{(k)}),
    \quad
    k=0,\ldots,T-1 .
    \label{eq:dual_path}
\end{equation}
The deep backbone and grounding predictor share the same language-model head, producing
\begin{equation}
    \mathbf{z}_{\mathrm{bb}}
    =
    W_{\mathrm{lm}}\mathrm{Norm}(\mathbf{H}_{\mathrm{bb}}^{(L)}),
    \qquad
    \mathbf{z}_{\mathrm{gp}}
    =
    W_{\mathrm{lm}}\mathrm{Norm}(\mathbf{H}_{\mathrm{gp}}^{(T)}),
    \label{eq:logits}
\end{equation}
where
$\mathbf{z}_{\mathrm{bb}},\mathbf{z}_{\mathrm{gp}}\in\mathbb{R}^{N\times|\mathcal{Y}|}$
are the backbone and predictor logits.

% ============================================================
\subsection{Grounding Predictor with Calibrated Supervision}
\label{subsec:grounding_predictor}
% ============================================================

The grounding predictor estimates which visual tokens correspond to query-relevant grounding regions.
Let
$\tilde{\mathbf{H}}_{\mathrm{gp}}=\mathrm{LayerNorm}(\mathbf{H}_{\mathrm{gp}}^{(T)})$
be the normalized predictor output.
At the predictor's terminal layer, let
$\mathbf{Q}_{h,q}\in\mathbb{R}^{d_h}$ be the query vector of text position $q$ under head $h$, and
$\mathbf{K}_{h,j}\in\mathbb{R}^{d_h}$ be the key vector of visual position $j$.
Here, $h\in\{1,\ldots,N_h\}$ indexes attention heads, $N_h$ is the number of heads, and $d_h$ is the head dimension.
For a visual token $j\in\mathcal{I}_v$, the predictor computes its grounding score by aggregating text-to-visual attention logits:
\begin{equation}
    S_j
    =
    \frac{1}{N_h|\mathcal{I}_{y}|}
    \sum_{h=1}^{N_h}
    \sum_{q\in\mathcal{I}_{y}}
    \frac{\mathbf{Q}_{h,q}^{\top}\mathbf{K}_{h,j}}{\sqrt{d_h}},
    \qquad
    s_j=\sigma(S_j),
    \label{eq:saliency}
\end{equation}
where $S_j$ is the pre-sigmoid grounding logit, $s_j\in[0,1]$ is the predicted importance score, and $\sigma(\cdot)$ is the sigmoid function.
For non-visual positions $j\notin\mathcal{I}_v$, we set $s_j=0$.

The predictor is supervised by calibrated region-level attention maps.
Let
$\mathbf{S}=\{S_j:j\in\mathcal{I}_v\}\in\mathbb{R}^{|\mathcal{I}_v|}$
be the pre-sigmoid grounding logits over visual tokens, and let
$\mathbf{Y}_{\mathrm{reg}}\in\{0,1\}^{|\mathcal{I}_v|}$
be the calibrated pseudo region map.
The grounding supervision loss is
\begin{equation}
    \mathcal{L}_{\mathrm{gp}}
    =
    \mathrm{BCEWithLogits}
    \bigl(\mathbf{S},\mathbf{Y}_{\mathrm{reg}}\bigr).
    \label{eq:grounding_supervision}
\end{equation}
This loss encourages high scores for visual tokens inside query-relevant grounding regions and low scores otherwise.

% ============================================================
\subsection{Intermediate-to-Deep KL Distillation}
\label{subsec:kl_distillation}
% ============================================================

Grounding supervision localizes query-relevant regions, but it does not by itself align intermediate region prediction with deep-layer generation.
GroundFlow therefore introduces intermediate-to-deep KL distillation between the predictor and backbone logits.

Let $\tau>0$ be the distillation temperature.
For a valid language-supervision position $t\in\mathcal{I}_{y}$ and vocabulary token $v\in\{1,\ldots,|\mathcal{Y}|\}$, we define
\begin{equation}
    P_{\mathrm{bb},t}^{(\tau)}(v)
    =
    \frac{\exp(z_{\mathrm{bb},t,v}/\tau)}
         {\sum_{v'=1}^{|\mathcal{Y}|}\exp(z_{\mathrm{bb},t,v'}/\tau)},
    \qquad
    P_{\mathrm{gp},t}^{(\tau)}(v)
    =
    \frac{\exp(z_{\mathrm{gp},t,v}/\tau)}
         {\sum_{v'=1}^{|\mathcal{Y}|}\exp(z_{\mathrm{gp},t,v'}/\tau)} .
    \label{eq:softmax_tau}
\end{equation}
The intermediate-to-deep alignment loss is
\begin{equation}
    \mathcal{L}_{\mathrm{kl}}
    =
    \frac{1}{|\mathcal{I}_{y}|}
    \sum_{t\in\mathcal{I}_{y}}
    D_{\mathrm{KL}}
    \left(
    P_{\mathrm{bb},t}^{(\tau)}
    \,\Vert\,
    P_{\mathrm{gp},t}^{(\tau)}
    \right).
    \label{eq:kl_align}
\end{equation}
During post-training, the backbone weights are frozen.
However, the backbone logits are not detached, so gradients through the deep path can update the IWA scalars and the grounding scores that enter the backbone attention.
This allows the grounding predictor and the importance-weighted attention pathway to be optimized jointly while preserving the pretrained backbone parameters.

% ============================================================
\subsection{Importance-Weighted Attention (IWA)}
\label{subsec:iwa}
% ============================================================

The predicted importance score $s_j$ identifies query-relevant visual tokens, but the deep backbone does not directly use this signal under standard causal attention.
GroundFlow introduces \emph{Importance-Weighted Attention} (IWA), which adds the predicted importance as a learnable bias to deep-layer attention logits.

For a deep backbone layer $l\in\{K,\ldots,L-1\}$, let
$\mathbf{Q}_{i}^{(l)},\mathbf{K}_{j}^{(l)}\in\mathbb{R}^{d_h}$
be the query and key vectors for positions $i$ and $j$.
The standard causal attention logit is
\begin{equation}
    A_{ij}^{(l)}
    =
    \frac{\mathbf{Q}_{i}^{(l)\top}\mathbf{K}_{j}^{(l)}}{\sqrt{d_h}}
    +
    M_{ij}^{\mathrm{causal}},
    \label{eq:base_attn}
\end{equation}
where $M_{ij}^{\mathrm{causal}}$ is the causal attention mask.
GroundFlow augments this logit as
\begin{equation}
    \hat{A}_{ij}^{(l)}
    =
    \frac{\mathbf{Q}_{i}^{(l)\top}\mathbf{K}_{j}^{(l)}}{\sqrt{d_h}}
    +
    M_{ij}^{\mathrm{causal}}
    +
    \gamma_l s_j,
    \qquad
    l=K,\ldots,L-1 ,
    \label{eq:iwa_attn}
\end{equation}
where $\gamma_l\in\mathbb{R}$ is a learnable scalar for layer $l$ and is initialized as $\gamma_l=0$.
Since $s_j=0$ for non-visual positions, IWA only biases attention toward query-relevant visual tokens.
The bias is additive before softmax and does not change the causal mask or remove any token.

% ============================================================
\subsection{Training Objective}
\label{subsec:training_objective}
% ============================================================

GroundFlow optimizes the grounding predictor and the IWA scalars while keeping the pretrained backbone weights frozen.
The trainable parameters are
\begin{equation}
    \boldsymbol{\Theta}
    =
    \left\{
    \{\hat f_k\}_{k=0}^{T-1},
    \{\gamma_l\}_{l=K}^{L-1}
    \right\}.
    \label{eq:trainable_parameters}
\end{equation}
Here, $\{\hat f_k\}$ denotes the grounding-predictor layers and $\{\gamma_l\}$ denotes the IWA scalars.
The loss is
\begin{equation}
    \mathcal{L}_{\mathrm{GroundFlow}}
    =
    \alpha \mathcal{L}_{\mathrm{gp}}
    +
    (1-\alpha)\mathcal{L}_{\mathrm{kl}},
    \label{eq:groundflow_objective}
\end{equation}
where $\alpha\in[0,1]$ is a scalar trade-off, $\mathcal{L}_{\mathrm{gp}}$ is the grounding supervision loss in Equation~\ref{eq:grounding_supervision}, and $\mathcal{L}_{\mathrm{kl}}$ is the intermediate-to-deep KL loss in Equation~\ref{eq:kl_align}.
The temperature $\tau$ and the trade-off $\alpha$ are hyperparameters.
Although IWA has no separate loss term, its scalars $\gamma_l$ are learned through the gradient of $\mathcal{L}_{\mathrm{kl}}$, because the IWA bias affects the deep backbone logits used in the alignment loss.

% ============================================================
\subsection{Two-Forward Retrieval-and-Reuse Inference}
\label{subsec:two_forward_inference}
% ============================================================

After training, GroundFlow uses a deterministic two-forward inference procedure.

\textbf{First forward: region retrieval.}
The shared prefix $\{f_l\}_{l<K}$ processes the original multimodal input and the grounding predictor produces visual-token importance scores $\{s_j\}_{j\in\mathcal{I}_v}$.
These scores define a region mask by thresholding a smoothed grounding map:
\begin{equation}
    \mathbf{M}
    =
    \mathbb{1}
    \left[
    \mathrm{GaussBlur}(\mathbf{s})>\theta
    \right],
    \qquad
    \mathbf{X}_{\mathrm{roi}}
    =
    \mathrm{ViT}
    \left(
    \mathrm{Crop}(\mathbf{I},\mathrm{bbox}(\mathbf{M}))
    \right),
    \label{eq:roi_extract}
\end{equation}
where $\mathbf{s}=\{s_j:j\in\mathcal{I}_v\}$ is the grounding map over visual tokens, $\theta$ is the retrieval threshold, $\mathbf{M}$ is the binary region mask, $\mathbf{I}$ is the input image, $\mathrm{bbox}(\mathbf{M})$ returns the bounding box of the selected region, and $\mathbf{X}_{\mathrm{roi}}$ denotes the re-encoded ROI features.

\textbf{Second forward: region reuse.}
The retrieved ROI features $\mathbf{X}_{\mathrm{roi}}$ are inserted as additional regional context between the original visual tokens and the text tokens.
The second forward reuses the shallow-prefix KV caches for unchanged prefix tokens from the first forward, while computing the retrieved ROI tokens and subsequent tokens under the updated sequence.
The deep backbone then runs from layer $K$ to $L-1$ with the IWA bias $\gamma_l s_j$ activated for final generation.

\paragraph{Resolution-invariant configuration.}
For each model scale, the same trained GroundFlow configuration is reused across different input resolutions without retraining.
Only the visual encoding resolution and the retrieval threshold $\theta$ are adjusted at evaluation time.
This allows GroundFlow to transfer its learned grounding predictor and IWA scalars across multiple input sizes.

% ============================================================
\subsection{Gradient Flow at the Bifurcation Point}
\label{subsec:gradient_flow}
% ============================================================

The three GroundFlow components are coupled at the shared prefix
$\mathbf{X}^{(K)}$, which receives gradient from both the backbone and the
predictor paths.
The gradient with respect to a shared-prefix parameter
$\theta_{f_l},\ l<K$ decomposes as
\begin{equation}
    \frac{\partial\mathcal{L}_{\mathrm{GroundFlow}}}{\partial\theta_{f_l}}
    \;=\;
    \left(
    \underbrace{\frac{\partial\mathcal{L}_{\mathrm{align}}}{\partial\mathbf{H}_{\mathrm{bb}}^{(K)}}}_{\text{from deep backbone}}
    +
    \underbrace{\frac{\partial(\mathcal{L}_{\mathrm{select}}+\mathcal{L}_{\mathrm{align}})}{\partial\mathbf{H}_{\mathrm{pr}}^{(0)}}}_{\text{from middle-layer predictor}}
    \right)
    \cdot
    \prod_{m=l+1}^{K-1}\frac{\partial\mathbf{X}^{(m+1)}}{\partial\mathbf{X}^{(m)}}
    \cdot
    \frac{\partial\mathbf{X}^{(l+1)}}{\partial\theta_{f_l}}.
    \label{eq:prefix_grad}
\end{equation}
The gradient with respect to a per-layer IWA scalar
$\textcolor{reusecolor}{\gamma_l}$ is simpler and acquires a closed form
because the bias enters the attention logit linearly:
\begin{equation}
    \frac{\partial\mathcal{L}_{\mathrm{GroundFlow}}}{\partial\textcolor{reusecolor}{\gamma_l}}
    \;=\;
    \sum_{i,j}
    \hat{\alpha}_{ij}^{(l)}
    \!\left(
    \frac{\partial\mathcal{L}_{\mathrm{align}}}{\partial o_i^{(l)}}\!\cdot\! v_j^{(l)}
    -
    \!\sum_{k}\hat{\alpha}_{ik}^{(l)}
    \frac{\partial\mathcal{L}_{\mathrm{align}}}{\partial o_i^{(l)}}\!\cdot\! v_k^{(l)}
    \!\right)
    \textcolor{selectcolor}{s_j},
    \label{eq:gamma_grad}
\end{equation}
where $o_i^{(l)}=\sum_j\hat{\alpha}_{ij}^{(l)}v_j^{(l)}$.
Equation~\ref{eq:gamma_grad} shows that
$\textcolor{reusecolor}{\gamma_l}$ increases whenever attending more strongly
to high-saliency tokens reduces $\mathcal{L}_{\mathrm{align}}$, so the
language-modeling signal trains the per-layer importance bias end-to-end.

\section{Experiments}
\label{sec:experiments}

\subsection{Experimental Setup}
\label{subsec:experimental_setup}

\textbf{Models.}
We instantiate GroundFlow at two model scales, denoted as GroundFlow-$4$B and GroundFlow-$8$B, built on the corresponding Qwen3-VL-Instruct backbones.
For both scales, we set $K=24$ and $T=3$, where $K$ denotes the decoder layer at which the grounding predictor branches from the backbone, and $T$ denotes the number of predictor layers. The grounding predictor is a $3$-layer shallow path that bifurcates at decoder layer $24$ and shares the language-model head with the backbone.
% The $4$B and $8$B GroundFlow results are training-complete; 
% the $32$B
% GroundFlow run is in progress at submission time and only the Vanilla
% $32$B baseline is reported.

\textbf{Baselines.}
We compare against three baselines that progressively ablate the components
of GroundFlow:
\textsc{Vanilla} (no auxiliary post-training),
\textsc{SD-RPN} (region-aware predictor trained on calibrated ROI BCE only,
with the backbone path detached and no IWA bias), and
\textsc{GroundFlow w/o IWA} (BCE supervision and KL alignment active with
$\alpha{=}2/3$, but
$\gamma_l{=}0$, i.e.\ retrieval and KL alignment are trained but the
retrieved importance is not injected back into the deep attention).
For a fair comparison, all methods are post-trained on the same $185$K
multimodal corpus, use the same shared prefix and predictor architecture,
and are evaluated under identical inference pipelines for the four input
resolutions.

\textbf{Benchmarks.}
We evaluate GroundFlow on a diverse set of grounding-related multimodal benchmarks, including OCRBench~\cite{liu2024ocrbench}, InfoVQA~\cite{mathew2022infographicvqa}, DocVQA~\cite{mathew2020docvqa}, ChartQA~\cite{masry2022chartqa}, TextVQA~\cite{singh2019towards}, V*Bench~\cite{vstar}, HRBench at 4K and 8K resolutions~\cite{hrbench}, RealWorldQA~\cite{}, and MME-RealWorld~\cite{zhang2024mme}.
These benchmarks cover document understanding, OCR-centric reasoning, chart interpretation, fine-grained visual grounding, and high-resolution perception.
We report official benchmark metrics and use the average score as the overall measure.

\textbf{Evaluation protocol.}
We evaluate GroundFlow under four progressively increased input resolutions, $\{576,1024,2048,4096\}$, corresponding to progressively higher-resolution visual encoding.
For each model scale, we use the same trained GroundFlow configuration across all resolutions without retraining.
During evaluation, the learned IWA scalars are kept fixed, and the retrieval threshold for the first forward is selected from $\theta\in\{0.10,0.15\}$.
The second forward then reuses the shared-prefix KV cache and incorporates the retrieved regional tokens for final generation.

\subsection{Main Results}
\label{subsec:main_results}

Table~\ref{tab:main_results} presents the main comparison across model
scales and benchmarks at the standard input resolution of $576$ visual
tokens.
GroundFlow consistently improves the multimodal foundation model on both
document/OCR and fine-grained-grounding/high-resolution groups.
The advantage is most pronounced on V*Bench, OCRBench, and HRBench, where
the calibrated region-level supervision and the IWA bias most directly
amplify the visual tokens that carry text or fine-grained spatial detail.

\begin{table*}[t]
\centering
\caption{
\textbf{Main results at input resolution $576$.}
We compare GroundFlow with the vanilla backbone and related post-training variants under the same training data and inference setting.
GroundFlow consistently improves grounding-related performance across model scales.
We report official metrics for each benchmark and the average score for overall comparison.
% Best results within each model-scale group are marked in \textbf{bold}.
}
\label{tab:main_results}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{} l | c c c c c | c c c c c || c @{}}
\toprule
\multirow{2}{*}{\textbf{Method}}
& \multicolumn{5}{c|}{\textbf{Document \& OCR}}
& \multicolumn{5}{c||}{\textbf{Fine-grained grounding \& high-resolution}}
& \multirow{2}{*}{\textbf{Avg.}} \\
\cmidrule(lr){2-6}\cmidrule(lr){7-11}
& OCR & Info & Doc & Chart & Text
& V* & HR4K & HR8K & RWQA & MME-RW & \\
\midrule

\multicolumn{12}{@{}c}{\emph{4B-scale model}} \\
\midrule
Qwen3-VL-4B-Instruct
& 839 & 68.0 & 91.2 & 83.9 & 79.3
& 62.8 & 63.6 & 56.1 & 68.6 & 52.1
& 70.5 \\
SD-RPN
& 843 & 75.2 & 92.5 & 83.7 & 78.3
& 79.6 & 75.2 & 67.4 & 71.5 & 55.3
& 76.30 \\
GroundFlow w/o IWA
& 847 & 74.9 & 92.6 & 83.6 & 78.2
& 81.2 & 73.3 & 67.0 & 72.4 & 53.1
& 76.10 \\
\rowcolor{ourslavender!50}
\textbf{GroundFlow}
& \textbf{852} & \textbf{76.0} & \textbf{92.7} & 83.5 & \textbf{78.4}
& \textbf{83.2} & 74.7 & 66.6 & \textbf{72.8} & 54.9
& \textbf{76.80} \\

\midrule
\multicolumn{12}{@{}c}{\emph{8B-scale model}} \\
\midrule
Qwen3-VL-8B-Instruct
& 849 & 72.9 & 93.0 & 84.8 & 81.5
& 67.0 & 66.2 & 59.1 & 69.4 & 53.3
& 72.0 \\
SD-RPN
& 862 & 80.3 & 94.4 & 85.1 & 79.9
& 82.2 & 74.1 & 68.9 & 71.6 & 56.7
& 77.9 \\
GroundFlow w/o IWA
& 862 & 80.3 & 94.4 & 85.2 & 79.7
& 80.1 & \textbf{75.5} & \textbf{69.4} & \textbf{73.1} & 56.1
& 78.0 \\
\rowcolor{ourslavender!50}
\textbf{GroundFlow}
& \textbf{866} & \textbf{80.3} & \textbf{94.6} & \textbf{85.4} & \textbf{79.8}
& 81.2 & 75.0 & \textbf{69.5} & 72.7 & 55.8
& \textbf{78.09} \\
% \midrule
% \multicolumn{12}{@{}l}{\emph{Qwen3-VL-32B (dense)} — GroundFlow rows training in progress} \\
% \midrule
% Vanilla
% & 862 & 77.1 & 93.8 & 86.9 & 82.2
% & 73.3 & 62.9 & 59.8 & 74.4 & 58.4
% & --- \\
% SD-RPN
% & --- & --- & --- & --- & ---
% & --- & --- & --- & --- & ---
% & --- \\
% \rowcolor{ourslavender!50}
% \textbf{GroundFlow}
% & --- & 79.1 & --- & 84.7 & 78.7
% & --- & 68.4 & 61.3 & --- & ---
% & --- \\
\bottomrule
\end{tabular}%
}
\vspace{-0.5em}
\end{table*}

\paragraph{Analysis.}
On GroundFlow-$4$B, our method improves V*Bench from $62.8$ to $83.2$, OCRBench from $839$ to $852$, and InfoVQA from $68.0$ to $76.0$.
The largest gain appears on V*Bench, which suggests that retrieving query-relevant regions and reusing them through IWA is especially effective for small or cluttered visual details.
The GroundFlow w/o IWA variant already improves over the grounding-predictor baseline on OCRBench and V*Bench, showing that intermediate-to-deep KL alignment provides a useful training signal.
Activating IWA brings further gains on OCR-heavy and fine-grained grounding benchmarks.
On GroundFlow-$8$B, our method reaches $866$ on OCRBench and $94.6$ on DocVQA, outperforming the corresponding grounding-predictor baseline and vanilla backbone.
It also achieves the best average score among all compared variants.
For high-resolution evaluation, GroundFlow-$8$B matches or improves HRBench performance while maintaining strong OCR and document understanding results.

\paragraph{Resolution scaling.}
Table~\ref{tab:resolution_scaling} reports the same configurations
evaluated across input resolutions of $\{576,1024,2048,4096\}$ visual
tokens.
The trained $\boldsymbol{\Theta}^{\star}$ is reused without retraining;
only the Pass-$1$/Pass-$2$ routing threshold is selected from
$\theta\in\{0.10,0.15\}$.
On $4$B, V*Bench rises from $83.2$ at $576$ to $92.1$ at $4096$ tokens
under the same GroundFlow configuration, while HR8K rises from $66.6$ to
$76.9$.
On $8$B, MME-RealWorld improves from $55.8$ at $576$ to $63.0$ at $4096$
tokens, and InfoVQA from $80.3$ to $83.6$.
The advantage of the retrieval-and-reuse recipe is preserved across all
four resolutions, indicating that the trained operator is
resolution-invariant.

\begin{table}[t]
\centering
\caption{
\textbf{Resolution scaling of GroundFlow.}
We evaluate GroundFlow under increasing input resolutions across grounding-related multimodal benchmarks.
}
\label{tab:resolution_scaling}
\resizebox{0.95\linewidth}{!}{
\begin{tabular}{@{}l|c|cccccccccc@{}}
\toprule
\textbf{Method} & \textbf{Res.}
& OCR & Info & Doc & Chart & Text
& V* & HR4K & HR8K & RWQA & MME-RW \\
\midrule
\multirow{4}{*}{GroundFlow-4B}
& $576$  & 852 & 76.0 & 92.7 & 83.5 & 78.4
       & 83.2 & 74.7 & 66.6 & 72.8 & 54.9 \\
& $1024$ & 854 & 80.3 & 93.9 & \underline{84.0} & 78.2
       & 84.8 & 78.6 & 73.5 & 73.1 & 60.1 \\
& $2048$ & 853 & 81.9 & 94.6 & \underline{84.0} & \underline{80.4}
       & 86.9 & 80.0 & 74.8 & \underline{73.6} & 62.8 \\
& $4096$ & \underline{856} & \underline{83.3} & \underline{94.8} & \underline{84.0} & \underline{80.4}
       & \underline{92.1} & \underline{82.0} & \underline{76.9} & \underline{73.6} & \underline{64.1} \\
\midrule
\multirow{4}{*}{GroundFlow-8B}
& $576$  & \underline{866} & 80.3 & 94.6 & \underline{85.4} & \underline{79.8}
       & 81.2 & 75.0 & 69.5 & 72.7 & 55.8 \\
& $1024$ & 860 & 82.8 & 95.0 & 85.3 & \underline{79.8}
       & 78.0 & \underline{79.1} & 70.5 & \underline{72.9} & 59.7 \\
& $2048$ & 861 & 83.0 & 95.2 & 85.3 & \underline{79.8}
       & 83.2 & 77.9 & 71.9 & 71.9 & 61.9 \\
& $4096$ & 861 & \underline{83.6} & \underline{95.4} & 85.3 & \underline{79.8}
       & \underline{86.9} & 78.6 & \underline{73.4} & 71.9 & \underline{63.0} \\
\bottomrule
\end{tabular}%
}
\end{table}


\section{Ablation Studies}
\label{sec:ablation}

We analyze GroundFlow from three complementary aspects:
\emph{(i)} the trade-off between grounding supervision and distribution alignment, controlled by the scalar $\alpha$ in Equation~\ref{eq:groundflow_objective};
\emph{(ii)} the contribution of each component, including the grounding predictor, intermediate-to-deep KL alignment, and Importance-Weighted Attention (IWA);
and \emph{(iii)} the robustness of retrieval during inference under different alignment temperatures and retrieval thresholds.
Unless otherwise specified, all ablations use the same training data, backbone, and evaluation protocol, and vary only the factor under study.

% ============================================================
\subsection{Effect of the Grounding-Alignment Trade-off $\alpha$}
\label{sec:abl_alpha}
% ============================================================

The scalar $\alpha$ in Equation~\ref{eq:groundflow_objective} controls the relative strength of calibrated region-level supervision and intermediate-to-deep KL alignment.
We sweep $\alpha$ on Qwen3-VL-$4$B at input resolution $576$, while keeping the IWA scalars and the alignment temperature fixed.
Figure~\ref{fig:abl_alpha} shows that GroundFlow is broadly stable across a wide range of $\alpha$ values.
Document- and OCR-related benchmarks remain robust when $\alpha$ varies, while fine-grained grounding is more sensitive to the balance between region supervision and distribution alignment.
The best performance is obtained at an interior value of $\alpha$, rather than at either extreme.
This indicates that calibrated region-level supervision and intermediate-to-deep KL alignment are complementary: the former improves query-relevant region localization, while the latter preserves consistency with deep-layer generation.

\begin{figure*}[t]
  \centering
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_ocr.pdf}
    \caption{OCRBench}
    \label{fig:abl_alpha_ocr}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_info.pdf}
    \caption{InfoVQA}
    \label{fig:abl_alpha_info}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_vstar.pdf}
    \caption{V*Bench}
    \label{fig:abl_alpha_vstar}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_hr4k.pdf}
    \caption{HRBench-$4$K}
    \label{fig:abl_alpha_hr4k}
  \end{subfigure}
  \caption{
  \textbf{Effect of the grounding-alignment trade-off $\alpha$.}
  We sweep $\alpha$ on GroundFlow-$4$B at input resolution $576$.
  GroundFlow achieves the best overall performance at an interior trade-off, showing that region-level supervision and intermediate-to-deep alignment are complementary.
  }
  \label{fig:abl_alpha}
\end{figure*}

\begin{figure*}[t]
  \centering
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_ocr.pdf}
    \caption{OCRBench}
    \label{fig:abl_comp_ocr}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_info.pdf}
    \caption{InfoVQA}
    \label{fig:abl_comp_info}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_vstar.pdf}
    \caption{V*Bench}
    \label{fig:abl_comp_vstar}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_hr4k.pdf}
    \caption{HRBench-$4$K}
    \label{fig:abl_comp_hr4k}
  \end{subfigure}
  \caption{
  \textbf{Component analysis of GroundFlow.}
  We progressively add the grounding predictor, intermediate-to-deep KL alignment, and IWA to the vanilla backbone.
  Each component contributes to the final performance, with IWA providing consistent additional gains after region retrieval and distribution alignment.
  }
  \label{fig:abl_components}
\end{figure*}

% ============================================================
\subsection{Component Analysis}
\label{sec:abl_components}
% ============================================================

We next study the contribution of each GroundFlow component on GroundFlow-$4$B at input resolution $576$.
Starting from the vanilla backbone, we progressively add:
\emph{(i)} the grounding predictor with calibrated region-level supervision,
\emph{(ii)} intermediate-to-deep KL alignment,
and \emph{(iii)} IWA.
Figure~\ref{fig:abl_components} reports the results on OCRBench, InfoVQA, V*Bench, and HRBench-$4$K.
The grounding predictor provides the main improvement on fine-grained grounding, showing the importance of explicitly retrieving query-relevant visual tokens from intermediate representations.
Adding intermediate-to-deep KL alignment further improves generation consistency, while IWA provides an additional gain by injecting predicted token importance into deep-layer attention.
These results show that GroundFlow benefits from the joint coupling of region retrieval, distribution alignment, and importance-guided deep attention.

% ============================================================
\subsection{Effect of the Alignment Temperature $\tau$}
\label{sec:abl_tau}
% ============================================================

We further study the effect of the alignment temperature $\tau$ in Equation~\ref{eq:kl_align}.
The temperature controls the softness of the intermediate-to-deep KL target.
We sweep $\tau$ on Qwen3-VL-$8$B at input resolution $576$, while keeping the grounding-alignment trade-off and IWA scalars fixed.
Figure~\ref{fig:abl_tau} shows that GroundFlow is robust to the choice of $\tau$.
The best performance is generally obtained around $\tau\in[1.5,2.0]$, and performance remains stable across a moderate temperature range.
This suggests that GroundFlow does not rely on a highly sensitive temperature setting, while still benefiting from softened distribution alignment.

\begin{figure*}[t]
  \centering
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig4_tau_ocr.pdf}
    \caption{OCRBench}
    \label{fig:abl_tau_ocr}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig4_tau_info.pdf}
    \caption{InfoVQA}
    \label{fig:abl_tau_info}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig4_tau_vstar.pdf}
    \caption{V*Bench}
    \label{fig:abl_tau_vstar}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig4_tau_hr4k.pdf}
    \caption{HRBench-$4$K}
    \label{fig:abl_tau_hr4k}
  \end{subfigure}
  \caption{
  \textbf{Effect of the alignment temperature $\tau$.}
  We sweep $\tau$ on Qwen3-VL-$8$B at input resolution $576$ with fixed IWA scalars and loss trade-off.
  GroundFlow is robust to $\tau$, with the best performance around $\tau\in[1.5,2.0]$.
  }
  \label{fig:abl_tau}
\end{figure*}

% ============================================================
\subsection{Robustness to Retrieval Thresholds}
\label{sec:abl_threshold}
% ============================================================

Finally, we evaluate the robustness of the first-forward retrieval threshold $\theta$ in Equation~\ref{eq:roi_extract}.
The threshold controls the region mask produced from the predicted grounding map and therefore affects the regional tokens reused in the second forward.
We vary $\theta$ on Qwen3-VL-$4$B at input resolution $576$ while keeping the trained GroundFlow configuration unchanged.
Figure~\ref{fig:abl_threshold} shows that GroundFlow remains stable across a broad threshold range.
OCRBench maintains a stable plateau under moderate thresholds, while V*Bench favors a slightly lower threshold that retains more fine-grained regional information.
Performance degrades only when the threshold becomes overly restrictive, which may remove query-relevant visual tokens before the second forward.
Overall, the retrieval process is not sensitive to the precise threshold and does not require retraining for different threshold choices.

\begin{figure*}[t]
  \centering
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_ocr.pdf}
    \caption{OCRBench}
    \label{fig:abl_thresh_ocr}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_info.pdf}
    \caption{InfoVQA}
    \label{fig:abl_thresh_info}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_vstar.pdf}
    \caption{V*Bench}
    \label{fig:abl_thresh_vstar}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.24\textwidth}
    \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_hr4k.pdf}
    \caption{HRBench-$4$K}
    \label{fig:abl_thresh_hr4k}
  \end{subfigure}
  \caption{
  \textbf{Robustness to retrieval thresholds $\theta$.}
  We vary the first-forward retrieval threshold while keeping the trained GroundFlow configuration fixed.
  % GroundFlow remains stable across a broad threshold range, indicating that the retrieval-and-reuse procedure is not sensitive to a precise threshold choice.
  }
  \label{fig:abl_threshold}
  \vspace{-1em}
\end{figure*}


\section{Conclusion}
\label{sec:conclusion}

We presented \textbf{GroundFlow}, a lightweight post-training framework for fine-grained multimodal grounding. GroundFlow retrieves query-relevant visual tokens from intermediate representations, aligns intermediate prediction with deep-layer generation through KL distillation, and injects token importance into deep attention via IWA. During inference, the retrieved regional tokens are reused in a second forward with shared-prefix KV cache reuse, providing additional regional context for grounded generation. By connecting intermediate grounding prediction with deep language generation, GroundFlow offers a scalable alternative to efficient post-training for improving fine-grained visual grounding.






% \subsection{Fine-Grained Multimodal Grounding.}
% \label{subsec:fine_grained_grounding}

% Fine-grained visual grounding has been studied from multiple angles in
% multimodal foundation models, including high-resolution visual encoders
% that scale up the input image, multi-resolution and any-resolution image
% processing that adapts to dense documents, region-level modules that
% explicitly attach localization heads to the language backbone, auxiliary
% spatial supervision through bounding boxes or pseudo-masks, and continued
% pre-training on grounding-heavy data such as document, chart, and GUI
% corpora~\citep{}.
% These directions improve grounding through model-side or data-side
% expansion, but they typically optimize the final response under
% end-to-end supervision and do not explicitly model how query-relevant
% regional cues that emerge in intermediate representations should be
% selected and reused by deeper language-generating layers.
% Different from these methods, GroundFlow operates as a lightweight
% post-training procedure that reuses retrieved visual tokens through an
% additive attention bias, leaving the backbone architecture and the vision
% encoder unchanged.

% \subsection{Region Proposal, Self-Distillation, and Importance-Reweighted Attention.}
% \label{subsec:region_aware_methods}

% Existing region-aware mechanisms typically focus on a single aspect of
% region-aware training: pure region-proposal training, pure
% intermediate-to-deep KL self-distillation, or importance-reweighted
% attention~\citep{}.
% Region-proposal methods such as SD-RPN train a shallow predictor branch
% that outputs a binary ROI map under BCE supervision and discard the
% backbone path during training~\citep{}.
% Self-speculative decoding methods distill a shallow predictor from the
% deep backbone through a teacher-fixed KL, but do not consume the
% predictor's spatial saliency back into the backbone~\citep{}.
% Importance-reweighted attention methods amplify visual tokens at every
% backbone layer through a hand-designed scalar bias, but treat the
% saliency scores as a frozen pre-processing step rather than a trainable
% signal~\citep{}.
% Although these methods have achieved promising accuracy-grounding
% trade-offs, they are usually developed as separate auxiliary recipes with
% manually specified branch depths, distillation weights, and attention
% biases.
% Different from them, our GroundFlow trains the calibrated region-aware
% selection, the intermediate-to-deep KL alignment, and the
% importance-weighted attention jointly under a single scalar trade-off, so
% that retrieved visual tokens are not only learned through ROI supervision
% but also reused inside the deep backbone attention during both training
% and inference.


% \begin{ack}
% Use unnumbered first level headings for the acknowledgments. All acknowledgments
% go at the end of the paper before the list of references. Moreover, you are required to declare
% funding (financial activities supporting the submitted work) and competing interests (related financial activities outside the submitted work).
% More information about this disclosure can be found at: \url{https://neurips.cc/Conferences/2026/PaperInformation/FundingDisclosure}.

% Do {\bf not} include this section in the anonymized submission, only in the final paper. You can use the \texttt{ack} environment provided in the style file to automatically hide this section in the anonymized submission.
% \end{ack}

% \section*{References}


% References follow the acknowledgments in the camera-ready paper. Use unnumbered first-level heading for
% the references. Any choice of citation style is acceptable as long as you are
% consistent. It is permissible to reduce the font size to \verb+small+ (9 point)
% when listing the references.
% Note that the Reference section does not count towards the page limit.
% \medskip


% {
% \small


% [1] Alexander, J.A.\ \& Mozer, M.C.\ (1995) Template-based algorithms for
% connectionist rule extraction. In G.\ Tesauro, D.S.\ Touretzky and T.K.\ Leen
% (eds.), {\it Advances in Neural Information Processing Systems 7},
% pp.\ 609--616. Cambridge, MA: MIT Press.


% [2] Bower, J.M.\ \& Beeman, D.\ (1995) {\it The Book of GENESIS: Exploring
%   Realistic Neural Models with the GEneral NEural SImulation System.}  New York:
% TELOS/Springer--Verlag.


% [3] Hasselmo, M.E., Schnell, E.\ \& Barkai, E.\ (1995) Dynamics of learning and
% recall at excitatory recurrent synapses and cholinergic modulation in rat
% hippocampal region CA3. {\it Journal of Neuroscience} {\bf 15}(7):5249-5262.
% }


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

% \appendix

% \section{Technical appendices and supplementary material}
% Technical appendices with additional results, figures, graphs, and proofs may be submitted with the paper submission before the full submission deadline (see above). You can upload a ZIP file for videos or code, but do not upload a separate PDF file for the appendix. There is no page limit for the technical appendices.

% Note: Think of the appendix as ``optional reading'' for reviewers. The paper must be able to stand alone without the appendix; for example, adding critical experiments that support the main claims to an appendix is inappropriate.

\bibliographystyle{plainnat}
\bibliography{references}
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%



%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\newpage

\appendix

\section{Technical appendices and supplementary material}

\subsection{Experimental Details}
\label{sec:exp_details}

\paragraph{Training data and spatial supervision.}
We post-train GroundFlow using $185$K image-text samples with pseudo spatial supervision.
For each sample, we construct a region-of-interest map from teacher attention patterns and use it as the supervision signal for intermediate visual evidence selection.
This avoids the need for manually annotated bounding boxes while still providing dense spatial guidance for text-rich, document-centric, and fine-grained visual inputs.

\paragraph{Model architecture.}
GroundFlow is built on Qwen3-VL backbones.
We attach a lightweight intermediate grounding branch after the $24$-th decoder layer.
The branch contains three auxiliary decoder layers initialized from the corresponding backbone layers.
Given the intermediate hidden states, the grounding branch estimates a spatial importance map over visual tokens by aggregating text-to-vision query-key interactions across attention heads.
This map is used for two purposes.
First, it supervises intermediate visual evidence selection during post-training.
Second, it identifies query-relevant regions for high-resolution re-encoding at inference time.

\paragraph{Post-training objective.}
GroundFlow combines spatial supervision with distributional alignment between intermediate and deep representations.
The training loss is defined as
\begin{equation}
\mathcal{L}
=
\alpha \mathcal{L}_{\mathrm{ROI}}
+
(1-\alpha)\mathcal{L}_{\mathrm{KL}},
\end{equation}
where $\mathcal{L}_{\mathrm{ROI}}$ is the binary spatial supervision loss and $\mathcal{L}_{\mathrm{KL}}$ aligns the intermediate branch distribution with the deeper language-generating distribution using temperature-scaled KL distillation.
The coefficient $\alpha$ controls the balance between spatial evidence learning and distributional alignment.
For the 4B-scale model, we use $\alpha=0.667$ and temperature $\tau=2.0$.
For the 8B-scale model, we use $\alpha=0.667$ and temperature $\tau=1.5$.
We do not add a separate language-modeling cross-entropy term, which keeps the post-training objective focused on spatial evidence selection and representation alignment.

\paragraph{Intermediate-to-deep attention guidance.}
To propagate intermediate grounding cues into deeper language-generating layers, GroundFlow uses intermediate-to-deep attention guidance.
The grounding branch first predicts visual-token importance from intermediate hidden states.
The resulting importance scores are converted into a lightweight additive attention bias for the subsequent backbone layers.
This design encourages deep layers to attend more strongly to query-relevant visual evidence without adding extra visual tokens, external proposal modules, or additional inference-time model components.
The guidance is parameter-efficient and only introduces a small set of learnable layer-wise scaling factors.

\paragraph{Inference.}
At inference time, GroundFlow first predicts a region-of-interest map from the intermediate grounding branch.
The map is smoothed and thresholded to obtain a compact visual region, which is then cropped from the original image and re-encoded at high resolution.
The re-encoded regional features are inserted into the multimodal sequence after the original visual tokens.
To avoid redundant computation, the prefix key-value states before the grounding branch are reused when processing the enhanced sequence.
The later backbone layers then perform a full forward pass over the combined global and regional visual evidence.

\paragraph{Optimization.}
All models are post-trained for one epoch with AdamW, a learning rate of $1\times10^{-4}$, cosine decay, $3\%$ warmup, and gradient clipping at $1.0$.
We use bf16 precision and distributed data parallel training.
For the 4B-scale model, training uses an effective batch size of $128$ and runs for $1450$ optimization steps.
The same pseudo-ROI training set, branch design, and optimization recipe are used across model scales unless otherwise specified.

\paragraph{Evaluation.}
We evaluate all methods under the same training data and inference setting.
The main evaluation includes document and OCR-oriented benchmarks, including OCRBench, InfoVQA, DocVQA, ChartQA, and TextVQA.
We also evaluate fine-grained grounding and high-resolution perception on V*Bench, HRBench4K, HRBench8K, RealWorldQA, and MME-RealWorld.
All results are reported using the official metric of each benchmark.
For the average score, we normalize benchmark-specific metrics to comparable percentage-style scales, while OCRBench is divided by $10$ before averaging.

\begin{table*}[t]
\centering
\caption{
\textbf{Post-training and inference details of GroundFlow.}
We report the main settings used for the 4B- and 8B-scale models.
}
\label{tab:training_details}
\resizebox{\textwidth}{!}{%
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Configuration} & \textbf{4B-scale model} & \textbf{8B-scale model} \\
\midrule
Base backbone & Qwen3-VL-4B-Instruct & Qwen3-VL-8B-Instruct \\
Input resolution & $576$ visual tokens & $576$ visual tokens \\
Post-training data & $185$K pseudo-ROI samples & $185$K pseudo-ROI samples \\
Grounding branch position & Layer $24$ & Layer $24$ \\
Grounding branch depth & $3$ decoder layers & $3$ decoder layers \\
Spatial map construction & Text-to-vision QK aggregation & Text-to-vision QK aggregation \\
Spatial supervision & Pseudo ROI map & Pseudo ROI map \\
Spatial loss & Binary cross-entropy & Binary cross-entropy \\
Distribution alignment & Temperature-scaled KL & Temperature-scaled KL \\
ROI loss weight $\alpha$ & $0.667$ & $0.667$ \\
Distillation temperature $\tau$ & $2.0$ & $1.5$ \\
Language-modeling CE loss & $0.0$ & $0.0$ \\
Intermediate-to-deep guidance & Additive attention bias & Additive attention bias \\
Guidance parameters & Layer-wise scalars & Layer-wise scalars \\
Regional enhancement & ROI crop re-encoding & ROI crop re-encoding \\
Computation reuse & Prefix KV reuse & Prefix KV reuse \\
Optimizer & AdamW & AdamW \\
Learning rate & $1\times10^{-4}$ & $1\times10^{-4}$ \\
Scheduler & Cosine decay & Cosine decay \\
Warmup ratio & $0.03$ & $0.03$ \\
Weight decay & $0.0$ & $0.0$ \\
Max gradient norm & $1.0$ & $1.0$ \\
Precision & bf16 & bf16 \\
Training epochs & $1$ & $1$ \\
Optimization steps & $1450$ & $1450$ \\
Effective batch size & $128$ & $128$ \\
Training parallelism & Distributed data parallel & Distributed data parallel \\
\bottomrule
\end{tabular}%
}
\vspace{-0.5em}
\end{table*}

\paragraph{Training dynamics.}
Figure~\ref{fig:app-train-dynamics} reports the training loss and gradient norm for
the $4$B and $8$B GroundFlow post-training runs. After a short transient the loss
settles at a non-zero plateau ($\approx 0.5$--$0.8$ nats): it is dominated by the
intermediate-to-deep KL term, and the two distributions being aligned ---the
intermediate grounding branch and the deeper language-generating distribution---
remain meaningfully different throughout, which is exactly the residual signal the
alignment term transfers into the deep layers. Gradient norms are stable over the
full schedule at both scales (no spikes, no divergence). Hyperparameters are as in
Table~\ref{tab:training_details}.

\begin{figure*}[t]
  \centering
  \begin{subfigure}[t]{0.49\linewidth}
    \centering\includegraphics[width=\linewidth]{figures/figA_loss_4b.pdf}
    \caption{Qwen3-VL-$4$B --- training loss}
    \label{fig:app-loss-4b}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.49\linewidth}
    \centering\includegraphics[width=\linewidth]{figures/figA_loss_8b.pdf}
    \caption{Qwen3-VL-$8$B --- training loss}
    \label{fig:app-loss-8b}
  \end{subfigure}

  \vspace{0.4em}
  \begin{subfigure}[t]{0.49\linewidth}
    \centering\includegraphics[width=\linewidth]{figures/figA_gn_4b.pdf}
    \caption{Qwen3-VL-$4$B --- gradient norm}
    \label{fig:app-gn-4b}
  \end{subfigure}\hfill
  \begin{subfigure}[t]{0.49\linewidth}
    \centering\includegraphics[width=\linewidth]{figures/figA_gn_8b.pdf}
    \caption{Qwen3-VL-$8$B --- gradient norm}
    \label{fig:app-gn-8b}
  \end{subfigure}
  \caption{\textbf{Training dynamics of GroundFlow.} Training loss (top) and
  gradient norm (bottom) for the $4$B and $8$B post-training runs; curves are
  EMA-smoothed with the raw trace shown faintly. After a brief transient the loss
  settles at the irreducible intermediate-to-deep KL component
  ($\approx 0.5$--$0.8$ nats; bold numbers mark the final value), and the gradient
  norm stays stable over the full schedule. All runs: $185$K pseudo-ROI samples,
  one epoch, $1450$ optimization steps (Table~\ref{tab:training_details}).}
  \label{fig:app-train-dynamics}
\end{figure*}

\subsection{Limitations}
\label{sec:limitations}

GroundFlow demonstrates that intermediate spatial supervision and intermediate-to-deep guidance can improve fine-grained grounding, but several directions remain open.
First, the current spatial supervision is obtained from pseudo ROI maps derived from teacher attention patterns.
This makes the method scalable and annotation-efficient, but the quality of the supervision is still bounded by the teacher signal.
Cleaner pseudo-label filtering, human-verified region annotations, or instance-level spatial supervision may further improve the learned grounding maps. Second, our regional enhancement currently uses a simple crop-based strategy.
This keeps the inference pipeline lightweight and easy to reproduce, but a single compact region may be suboptimal when the relevant evidence is distributed across multiple distant areas. Future work may explore multi-region re-encoding, adaptive region selection, or stronger layout-aware region composition.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

\newpage
% \input{checklist.tex}


\end{document}
% % % % % % % % % % % % % % % % % % % 
\subsection{Fine-Grained Multimodal Perception and Grounding}
Conventional Multimodal Large Language Models (MLLMs)~\cite{liu2023llava}, such as LLaVA, typically employ CLIP or its variants as vision encoders to align visual features with language representations. However, constrained by the limited input resolution of these encoders, such models struggle with tasks that demand fine-grained perception, including OCR, document understanding, and GUI localization. To mitigate this, subsequent research has introduced split-and-tile strategies and native-resolution Vision Transformers (ViTs), which significantly enhance fine-grained perception. Nonetheless, because standard vision encoders compress images at a fixed ratio, processing high-resolution inputs generates thousands of visual tokens, leading to substantial computational overhead. Moreover, minute visual details are often overwhelmed by a vast number of background tokens, impeding the LLM's ability to retrieve relevant information. To overcome these challenges, recent works have proposed a ground-then-perceive framework. In this approach, small visual evidence is first localized using external models, sliding windows, or the MLLM's internal attention mechanisms. These grounded regions are then re-encoded to provide complementary inputs that bolster the model's perceptual accuracy. Despite their effectiveness, these methods typically rely on handcrafted pipelines and multiple prefilling passes, which introduce significant efficiency bottlenecks and restrict their overall generality. 
To overcome these limitations, GroundFlow introduces a lightweight post-training procedure that reuses retrieved visual tokens through an additive attention bias. By avoiding redundant computations, this approach ensures a highly efficient grounding process without compromising the inherent generality of the MLLM.
% % % % % % \bibliographystyle{plain}
\bibliography{references}