| \documentclass{article} |
|
|
| |
| |
| |
|
|
| |
| |
| |
| |
| \PassOptionsToPackage{numbers, compress}{natbib} |
| |
| |
| |
| |
| |
| |
| |
| |
|
|
| |
| \usepackage[preprint]{neurips_2026} |
|
|
| \usepackage[utf8]{inputenc} |
| \usepackage[T1]{fontenc} |
| \usepackage{hyperref} |
| \usepackage{url} |
| \usepackage{booktabs} |
| \usepackage{amsfonts} |
| \usepackage{nicefrac} |
| \usepackage{microtype} |
| \usepackage{amsmath} |
| \usepackage{amssymb} |
| \usepackage{bm} |
| \usepackage{graphicx} |
| \usepackage{amsthm} |
| \usepackage{subcaption} |
| \usepackage{algorithmicx} |
| \usepackage{algpseudocode} |
| \usepackage{algorithm} |
| \usepackage{pifont} |
| \usepackage{diagbox} |
| \usepackage{multirow} |
| \usepackage[table]{xcolor} |
| \usepackage{array} |
| \usepackage{makecell} |
|
|
| \newcommand{\cmark}{\ding{51}} |
| \newcommand{\xmark}{\ding{55}} |
|
|
| \usepackage{comment} |
|
|
| \title{Efficient Post-Training for Fine-Grained Multimodal \\ Foundation Models} |
|
|
|
|
| \author{ |
| \textbf{Xiaohuan Pei}$^{1}$ \quad |
| \textbf{Jiyuan Zhang}$^{2}$ \quad |
| \textbf{Yuanfan Guo}$^{2}$ \quad |
| \textbf{Weiguo Feng}$^{2}$ \\ |
| \textbf{Tao Huang}$^{3}$ \quad |
| \textbf{Cho-Jui Hsieh}$^{4}$ \quad |
| \textbf{Chang Xu}$^{1}$ \\[0.5em] |
| $^{1}$The University of Sydney \quad |
| $^{2}$ByteDance \\ |
| $^{3}$Shanghai Jiao Tong University \quad |
| $^{4}$University of California, Los Angeles \\[0.3em] |
| \texttt{xiaohuan.pei@sydney.edu.au} |
| |
| |
| |
| |
| } |
|
|
|
|
| \begin{document} |
|
|
|
|
| \maketitle |
|
|
|
|
| \begin{abstract} |
| Fine-grained visual grounding remains a major bottleneck for multimodal foundation models, limiting dense-text reading, small-object localization, document understanding, GUI agents, and embodied perception. Existing methods improve grounding by increasing visual resolution, adding region-level modules, or performing end-to-end retraining, yet they provide limited control over how query-relevant regional cues are selected from intermediate representations and reused by deeper language-generating layers. We observe that effective grounding requires intermediate region-level perception to remain aligned with deep-layer generation. Based on this observation, we propose \textbf{GroundFlow}, a lightweight post-training method for intermediate-guided region-aware token reuse. GroundFlow supervises query-relevant visual token selection with calibrated attention maps, aligns middle- and deep-layer distributions through temperature-scaled KL distillation, and injects retrieved visual-token importance into deep layers via importance-weighted attention. During inference, the retrieved regional tokens are reused in a second pass to strengthen fine-grained visual grounding. Experiments on mainstream grounding-related multimodal benchmarks show that GroundFlow consistently improves foundation models, achieving substantial gains on fine-grained visual grounding evaluation (\textit{e.g.}, +20.4 points on V*Star at the 4B scale). |
| Ablations show that the gains arise from the coupling of calibrated region-level supervision, intermediate-to-deep distributional alignment, and importance-guided deep attention, which establishes GroundFlow as a scalable post-training paradigm for fine-grained visual grounding through region-aware token retrieval and reuse. |
| \end{abstract} |
|
|
| \section{Introduction} |
| \label{sec:introduction} |
|
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|
| Large multimodal foundation models have achieved remarkable progress in visual reasoning, document understanding, scene understanding, and instruction following~\cite{liu2023visual,an2025llava,oquab2023dinov2,Qwen-VL,Qwen2-VL,Qwen2.5-VL}. |
| Despite this progress, fine-grained visual grounding remains a major challenge. |
| Tasks involving small objects, dense text, spatial relations, or region-level reasoning require models to identify a small number of query-relevant visual regions from many redundant visual tokens. |
|
|
| However, current models often rely on coarse visual semantics, leading to weakly grounded responses in high-resolution and cluttered scenes~\cite{shi2026sdrpn}. |
| Recently, some work has improved fine-grained grounding through end-to-end retraining, high-resolution continued training, stronger vision encoders, region-level modules, or auxiliary localization supervision~\cite{zhang2025mllms,liu2024llavanext,lu2024deepseek,zhao2024mg}. |
| While effective, these approaches remain expensive, data-intensive, and intrusive, and leave the transfer of text-related grounding regions from intermediate representations to deep-layer generation underexplored. |
|
|
| |
| |
| |
| We argue that fine-grained grounding can be improved without full end-to-end retraining by retrieving query-relevant grounding regions from spatially rich intermediate representations and reusing them for deep-layer generation. |
| Based on this observation, we propose \textbf{GroundFlow}, a lightweight post-training framework that connects intermediate region localization with deep-layer generation. |
| GroundFlow trains a middle-layer \emph{region-aware token predictor} with calibrated region-level attention maps to estimate query-relevant visual tokens. |
| It aligns the predictor-side intermediate distribution with the deep backbone distribution through temperature-scaled KL distillation, and introduces \emph{Importance-Weighted Attention} (IWA) to inject predicted token importance into deep-layer attention. |
| During inference, the first forward retrieves query-relevant visual tokens, while the second forward reuses shallow-layer KV caches and inserts the retrieved tokens as additional regional context. |
|
|
| We summarize our contributions as follows: |
| (1) We directly train a middle-layer grounding predictor on hidden-state visual tokens with calibrated region-level supervision, avoiding intrusive full-model retraining. |
| (2) We align intermediate grounding prediction with deep-layer generation through temperature-scaled KL distillation. |
| (3) We introduce IWA and two-forward token reuse, allowing retrieved visual tokens to guide deep attention during training and provide additional regional context during inference. |
|
|
| \begin{figure*}[t] |
| \centering |
| \includegraphics[width=\linewidth]{figures/prev.png} |
| \caption{ |
| \textbf{Previous vs. Ours: } |
| Previous methods use end-to-end supervision, so grounding signals reach the model only through the final output. Our method instead adopts end-to-middle supervision and explicit shallow–to-deep interaction, enabling middle-layer grounding representations to directly guide deep-layer generation. |
| } |
| \label{fig:main} |
| \end{figure*} |
|
|
|
|
|
|
|
|
|
|
| \section{Related Work} |
| \label{sec:related_work} |
|
|
|
|
| \subsection{General Fine-Grained Multimodal Perception} |
| Classic Multimodal Large Language Models (MLLMs)\cite{liu2023llava,liu2023improvedllava, dai2023instructblip}, such as LLaVA, typically employ CLIP\cite{radford2021learning} or its variants~\cite{radford2021learning,tschannen2025siglip,zhai2023sigmoid,bolya2025perception} as vision encoders to align visual features with language representations. However, constrained by the limited input resolution of these encoders, such models struggle with tasks that demand fine-grained perception, including OCR~\cite{liu2024ocrbench}, document understanding~\cite{masry2022chartqa,mathew2020docvqa,mathew2022infographicvqa}, and GUI localization~\cite{li2025screenspot}. To mitigate this, subsequent research has introduced split-and-tile strategies~\cite{zhu2025internvl3, li2024llava, liu2024llavanext}, multiple vision encoders with multi-resolution capabilities~\cite{zhang2024llava, lu2024deepseek,zhao2024mg}, and native-resolution Vision Transformers (ViTs)\cite{Qwen2-VL,dehghani2023patch}, which significantly enhance fine-grained perception. Nonetheless, because standard vision encoders compress images at a fixed ratio, processing high-resolution inputs generates thousands of visual tokens, leading to substantial computational overhead. Moreover, minute visual details are often overwhelmed by a vast number of background tokens, impeding the LLM's ability to retrieve relevant information\cite{liu2025hide}. |
|
|
| \subsection{Grounding-Based Multimodal Perception and Importance-Reweighted Attention} |
| To address these challenges, recent works have proposed a ground-then-perceive framework~\cite{zhang2025mllms, jiang2025token, hrbench, shi2025scaling, shi2026sdrpn, shi2026q}. In this approach, small visual evidence is first localized using external models~\cite{jiang2025token}, sliding windows~\cite{hrbench}, or the MLLM's internal attention mechanisms~\cite{zhang2025mllms}. These grounded regions are then re-encoded to provide complementary inputs that bolster the model's perceptual accuracy. Beyond utilizing grounded crops as complementary inputs, some works treat saliency scores as a frozen preprocessing step to reweight attention inside the MLLM, thereby amplifying important visual tokens at every backbone layer through a hand-designed scalar bias~\cite{kang2025see}. Despite their effectiveness, these methods typically rely on handcrafted pipelines and multiple prefilling passes, which introduce significant efficiency bottlenecks and restrict their overall generality. |
| Another line of research~\cite{zheng2025deepeyes, zhang2025thyme, lai2025mini, wei2026zooming} draws inspiration from OpenAI's Thinking-with-Images paradigm~\cite{openai2025thinking}, proposing an interleaved text-and-image reasoning framework optimized via RL-based verified rewards~\cite{shao2024deepseekmath, guo2025deepseek}. However, this paradigm is computationally expensive during both training and inference. End-to-end optimization requires substantial GPU memory and carefully curated datasets, while the long chain-of-thought reasoning heavily burdens the decoding stage. |
| Departing from these computationally intensive and handcrafted pipelines, GroundFlow introduces a lightweight post-training procedure that jointly leverages grounded areas as complementary inputs and applies a trainable additive attention bias to the highlighted foreground regions. By avoiding redundant computations, this dual approach ensures a highly efficient visual grounding process without compromising the inherent generality of the MLLM. |
|
|
|
|
| \definecolor{selectcolor}{RGB}{200,50,0} |
| \definecolor{aligncolor}{RGB}{0,100,200} |
| \definecolor{reusecolor}{RGB}{0,150,50} |
| \definecolor{ourslavender}{RGB}{184,174,234} |
|
|
| \section{Method} |
| \label{sec:method} |
|
|
| \begin{figure*}[t] |
| \centering |
| \includegraphics[width=\linewidth]{figures/post1_.png} |
| \caption{ |
| \textbf{Overview of GroundFlow.} |
| GroundFlow attaches a middle-layer grounding predictor to a frozen multimodal backbone for query-relevant visual token retrieval. |
| The predictor is supervised by calibrated region-level attention maps and aligned with deep-layer generation through intermediate-to-deep KL distillation. |
| GroundFlow further injects predicted token importance $s_j$ into deep-layer attention via an importance-weighted attention (IWA) bias $\gamma_l\!\cdot\!s_j$. |
| During inference, the first forward retrieves query-relevant visual tokens, while the second forward reuses them as additional regional context for grounded generation. |
| } |
| \label{fig:main} |
| \end{figure*} |
|
|
| We aim to improve fine-grained visual grounding without full end-to-end retraining by retrieving query-relevant grounding regions from intermediate representations and reusing them for deep-layer generation. |
| As illustrated in Figure~\ref{fig:main}, GroundFlow consists of three coupled components. |
| First, a middle-layer grounding predictor estimates query-relevant visual tokens using calibrated region-level attention maps. |
| Second, intermediate-to-deep KL distillation aligns the predictor-side distribution with the deep backbone distribution. |
| Third, importance-weighted attention injects the predicted token importance into deep-layer attention as learnable biases. |
| During inference, GroundFlow performs two sequential forwards: the first forward retrieves query-relevant visual tokens, and the second forward reuses shallow-prefix KV caches while inserting the retrieved tokens as additional regional context. |
|
|
| |
| \subsection{Preliminary} |
| \label{subsec:preliminary} |
| |
|
|
| Let |
| $\mathbf{X}^{(0)}=[\mathbf{H}_{\mathrm{sys}},\mathbf{H}_{v},\mathbf{H}_{t}] |
| \in\mathbb{R}^{N\times d}$ |
| denote the multimodal input sequence, where |
| $\mathbf{H}_{\mathrm{sys}}$, $\mathbf{H}_{v}$, and $\mathbf{H}_{t}$ |
| are the system, visual, and text embeddings. |
| Here, $N$ is the sequence length and $d$ is the hidden dimension. |
| We denote the set of visual-token positions by |
| $\mathcal{I}_{v}\subseteq\{1,\ldots,N\}$ and the valid language-supervision positions by |
| $\mathcal{I}_{y}=\{t:y_{t+1}\neq -100\}$. |
| The backbone contains $L$ causal decoder layers |
| $\{f_l\}_{l=0}^{L-1}$, a final normalization layer |
| $\mathrm{Norm}(\cdot)$, and a language-model head |
| $W_{\mathrm{lm}}\in\mathbb{R}^{|\mathcal{Y}|\times d}$, |
| where $|\mathcal{Y}|$ is the vocabulary size. |
| GroundFlow attaches a grounding predictor at an intermediate layer $K$. |
| The first $K$ backbone layers form a shared prefix: |
| \begin{equation} |
| \mathbf{X}^{(K)} |
| = |
| f_{K-1}\circ\cdots\circ f_0(\mathbf{X}^{(0)}). |
| \end{equation} |
| From $\mathbf{X}^{(K)}$, the computation separates into a deep backbone path and a shallow grounding-predictor path. |
| The predictor contains $T$ decoder layers |
| $\{\hat f_k\}_{k=0}^{T-1}$, initialized from the corresponding backbone layers |
| $\{f_K,\ldots,f_{K+T-1}\}$: |
| \begin{equation} |
| \mathbf{H}_{\mathrm{bb}}^{(K)} |
| = |
| \mathbf{H}_{\mathrm{gp}}^{(0)} |
| = |
| \mathbf{X}^{(K)}, |
| \qquad |
| \mathbf{H}_{\mathrm{bb}}^{(l+1)} |
| = |
| f_l(\mathbf{H}_{\mathrm{bb}}^{(l)}), |
| \quad |
| l=K,\ldots,L-1, |
| \end{equation} |
| \begin{equation} |
| \mathbf{H}_{\mathrm{gp}}^{(k+1)} |
| = |
| \hat f_k(\mathbf{H}_{\mathrm{gp}}^{(k)}), |
| \quad |
| k=0,\ldots,T-1 . |
| \label{eq:dual_path} |
| \end{equation} |
| The deep backbone and grounding predictor share the same language-model head, producing |
| \begin{equation} |
| \mathbf{z}_{\mathrm{bb}} |
| = |
| W_{\mathrm{lm}}\mathrm{Norm}(\mathbf{H}_{\mathrm{bb}}^{(L)}), |
| \qquad |
| \mathbf{z}_{\mathrm{gp}} |
| = |
| W_{\mathrm{lm}}\mathrm{Norm}(\mathbf{H}_{\mathrm{gp}}^{(T)}), |
| \label{eq:logits} |
| \end{equation} |
| where |
| $\mathbf{z}_{\mathrm{bb}},\mathbf{z}_{\mathrm{gp}}\in\mathbb{R}^{N\times|\mathcal{Y}|}$ |
| are the backbone and predictor logits. |
|
|
| |
| \subsection{Grounding Predictor with Calibrated Supervision} |
| \label{subsec:grounding_predictor} |
| |
|
|
| The grounding predictor estimates which visual tokens correspond to query-relevant grounding regions. |
| Let |
| $\tilde{\mathbf{H}}_{\mathrm{gp}}=\mathrm{LayerNorm}(\mathbf{H}_{\mathrm{gp}}^{(T)})$ |
| be the normalized predictor output. |
| At the predictor's terminal layer, let |
| $\mathbf{Q}_{h,q}\in\mathbb{R}^{d_h}$ be the query vector of text position $q$ under head $h$, and |
| $\mathbf{K}_{h,j}\in\mathbb{R}^{d_h}$ be the key vector of visual position $j$. |
| Here, $h\in\{1,\ldots,N_h\}$ indexes attention heads, $N_h$ is the number of heads, and $d_h$ is the head dimension. |
| For a visual token $j\in\mathcal{I}_v$, the predictor computes its grounding score by aggregating text-to-visual attention logits: |
| \begin{equation} |
| S_j |
| = |
| \frac{1}{N_h|\mathcal{I}_{y}|} |
| \sum_{h=1}^{N_h} |
| \sum_{q\in\mathcal{I}_{y}} |
| \frac{\mathbf{Q}_{h,q}^{\top}\mathbf{K}_{h,j}}{\sqrt{d_h}}, |
| \qquad |
| s_j=\sigma(S_j), |
| \label{eq:saliency} |
| \end{equation} |
| where $S_j$ is the pre-sigmoid grounding logit, $s_j\in[0,1]$ is the predicted importance score, and $\sigma(\cdot)$ is the sigmoid function. |
| For non-visual positions $j\notin\mathcal{I}_v$, we set $s_j=0$. |
|
|
| The predictor is supervised by calibrated region-level attention maps. |
| Let |
| $\mathbf{S}=\{S_j:j\in\mathcal{I}_v\}\in\mathbb{R}^{|\mathcal{I}_v|}$ |
| be the pre-sigmoid grounding logits over visual tokens, and let |
| $\mathbf{Y}_{\mathrm{reg}}\in\{0,1\}^{|\mathcal{I}_v|}$ |
| be the calibrated pseudo region map. |
| The grounding supervision loss is |
| \begin{equation} |
| \mathcal{L}_{\mathrm{gp}} |
| = |
| \mathrm{BCEWithLogits} |
| \bigl(\mathbf{S},\mathbf{Y}_{\mathrm{reg}}\bigr). |
| \label{eq:grounding_supervision} |
| \end{equation} |
| This loss encourages high scores for visual tokens inside query-relevant grounding regions and low scores otherwise. |
|
|
| |
| \subsection{Intermediate-to-Deep KL Distillation} |
| \label{subsec:kl_distillation} |
| |
|
|
| Grounding supervision localizes query-relevant regions, but it does not by itself align intermediate region prediction with deep-layer generation. |
| GroundFlow therefore introduces intermediate-to-deep KL distillation between the predictor and backbone logits. |
|
|
| Let $\tau>0$ be the distillation temperature. |
| For a valid language-supervision position $t\in\mathcal{I}_{y}$ and vocabulary token $v\in\{1,\ldots,|\mathcal{Y}|\}$, we define |
| \begin{equation} |
| P_{\mathrm{bb},t}^{(\tau)}(v) |
| = |
| \frac{\exp(z_{\mathrm{bb},t,v}/\tau)} |
| {\sum_{v'=1}^{|\mathcal{Y}|}\exp(z_{\mathrm{bb},t,v'}/\tau)}, |
| \qquad |
| P_{\mathrm{gp},t}^{(\tau)}(v) |
| = |
| \frac{\exp(z_{\mathrm{gp},t,v}/\tau)} |
| {\sum_{v'=1}^{|\mathcal{Y}|}\exp(z_{\mathrm{gp},t,v'}/\tau)} . |
| \label{eq:softmax_tau} |
| \end{equation} |
| The intermediate-to-deep alignment loss is |
| \begin{equation} |
| \mathcal{L}_{\mathrm{kl}} |
| = |
| \frac{1}{|\mathcal{I}_{y}|} |
| \sum_{t\in\mathcal{I}_{y}} |
| D_{\mathrm{KL}} |
| \left( |
| P_{\mathrm{bb},t}^{(\tau)} |
| \,\Vert\, |
| P_{\mathrm{gp},t}^{(\tau)} |
| \right). |
| \label{eq:kl_align} |
| \end{equation} |
| During post-training, the backbone weights are frozen. |
| However, the backbone logits are not detached, so gradients through the deep path can update the IWA scalars and the grounding scores that enter the backbone attention. |
| This allows the grounding predictor and the importance-weighted attention pathway to be optimized jointly while preserving the pretrained backbone parameters. |
|
|
| |
| \subsection{Importance-Weighted Attention (IWA)} |
| \label{subsec:iwa} |
| |
|
|
| The predicted importance score $s_j$ identifies query-relevant visual tokens, but the deep backbone does not directly use this signal under standard causal attention. |
| GroundFlow introduces \emph{Importance-Weighted Attention} (IWA), which adds the predicted importance as a learnable bias to deep-layer attention logits. |
|
|
| For a deep backbone layer $l\in\{K,\ldots,L-1\}$, let |
| $\mathbf{Q}_{i}^{(l)},\mathbf{K}_{j}^{(l)}\in\mathbb{R}^{d_h}$ |
| be the query and key vectors for positions $i$ and $j$. |
| The standard causal attention logit is |
| \begin{equation} |
| A_{ij}^{(l)} |
| = |
| \frac{\mathbf{Q}_{i}^{(l)\top}\mathbf{K}_{j}^{(l)}}{\sqrt{d_h}} |
| + |
| M_{ij}^{\mathrm{causal}}, |
| \label{eq:base_attn} |
| \end{equation} |
| where $M_{ij}^{\mathrm{causal}}$ is the causal attention mask. |
| GroundFlow augments this logit as |
| \begin{equation} |
| \hat{A}_{ij}^{(l)} |
| = |
| \frac{\mathbf{Q}_{i}^{(l)\top}\mathbf{K}_{j}^{(l)}}{\sqrt{d_h}} |
| + |
| M_{ij}^{\mathrm{causal}} |
| + |
| \gamma_l s_j, |
| \qquad |
| l=K,\ldots,L-1 , |
| \label{eq:iwa_attn} |
| \end{equation} |
| where $\gamma_l\in\mathbb{R}$ is a learnable scalar for layer $l$ and is initialized as $\gamma_l=0$. |
| Since $s_j=0$ for non-visual positions, IWA only biases attention toward query-relevant visual tokens. |
| The bias is additive before softmax and does not change the causal mask or remove any token. |
|
|
| |
| \subsection{Training Objective} |
| \label{subsec:training_objective} |
| |
|
|
| GroundFlow optimizes the grounding predictor and the IWA scalars while keeping the pretrained backbone weights frozen. |
| The trainable parameters are |
| \begin{equation} |
| \boldsymbol{\Theta} |
| = |
| \left\{ |
| \{\hat f_k\}_{k=0}^{T-1}, |
| \{\gamma_l\}_{l=K}^{L-1} |
| \right\}. |
| \label{eq:trainable_parameters} |
| \end{equation} |
| Here, $\{\hat f_k\}$ denotes the grounding-predictor layers and $\{\gamma_l\}$ denotes the IWA scalars. |
| The loss is |
| \begin{equation} |
| \mathcal{L}_{\mathrm{GroundFlow}} |
| = |
| \alpha \mathcal{L}_{\mathrm{gp}} |
| + |
| (1-\alpha)\mathcal{L}_{\mathrm{kl}}, |
| \label{eq:groundflow_objective} |
| \end{equation} |
| where $\alpha\in[0,1]$ is a scalar trade-off, $\mathcal{L}_{\mathrm{gp}}$ is the grounding supervision loss in Equation~\ref{eq:grounding_supervision}, and $\mathcal{L}_{\mathrm{kl}}$ is the intermediate-to-deep KL loss in Equation~\ref{eq:kl_align}. |
| The temperature $\tau$ and the trade-off $\alpha$ are hyperparameters. |
| Although IWA has no separate loss term, its scalars $\gamma_l$ are learned through the gradient of $\mathcal{L}_{\mathrm{kl}}$, because the IWA bias affects the deep backbone logits used in the alignment loss. |
|
|
| |
| \subsection{Two-Forward Retrieval-and-Reuse Inference} |
| \label{subsec:two_forward_inference} |
| |
|
|
| After training, GroundFlow uses a deterministic two-forward inference procedure. |
|
|
| \textbf{First forward: region retrieval.} |
| The shared prefix $\{f_l\}_{l<K}$ processes the original multimodal input and the grounding predictor produces visual-token importance scores $\{s_j\}_{j\in\mathcal{I}_v}$. |
| These scores define a region mask by thresholding a smoothed grounding map: |
| \begin{equation} |
| \mathbf{M} |
| = |
| \mathbb{1} |
| \left[ |
| \mathrm{GaussBlur}(\mathbf{s})>\theta |
| \right], |
| \qquad |
| \mathbf{X}_{\mathrm{roi}} |
| = |
| \mathrm{ViT} |
| \left( |
| \mathrm{Crop}(\mathbf{I},\mathrm{bbox}(\mathbf{M})) |
| \right), |
| \label{eq:roi_extract} |
| \end{equation} |
| where $\mathbf{s}=\{s_j:j\in\mathcal{I}_v\}$ is the grounding map over visual tokens, $\theta$ is the retrieval threshold, $\mathbf{M}$ is the binary region mask, $\mathbf{I}$ is the input image, $\mathrm{bbox}(\mathbf{M})$ returns the bounding box of the selected region, and $\mathbf{X}_{\mathrm{roi}}$ denotes the re-encoded ROI features. |
|
|
| \textbf{Second forward: region reuse.} |
| The retrieved ROI features $\mathbf{X}_{\mathrm{roi}}$ are inserted as additional regional context between the original visual tokens and the text tokens. |
| The second forward reuses the shallow-prefix KV caches for unchanged prefix tokens from the first forward, while computing the retrieved ROI tokens and subsequent tokens under the updated sequence. |
| The deep backbone then runs from layer $K$ to $L-1$ with the IWA bias $\gamma_l s_j$ activated for final generation. |
|
|
| \paragraph{Resolution-invariant configuration.} |
| For each model scale, the same trained GroundFlow configuration is reused across different input resolutions without retraining. |
| Only the visual encoding resolution and the retrieval threshold $\theta$ are adjusted at evaluation time. |
| This allows GroundFlow to transfer its learned grounding predictor and IWA scalars across multiple input sizes. |
|
|
| |
| \subsection{Gradient Flow at the Bifurcation Point} |
| \label{subsec:gradient_flow} |
| |
|
|
| The three GroundFlow components are coupled at the shared prefix |
| $\mathbf{X}^{(K)}$, which receives gradient from both the backbone and the |
| predictor paths. |
| The gradient with respect to a shared-prefix parameter |
| $\theta_{f_l},\ l<K$ decomposes as |
| \begin{equation} |
| \frac{\partial\mathcal{L}_{\mathrm{GroundFlow}}}{\partial\theta_{f_l}} |
| \;=\; |
| \left( |
| \underbrace{\frac{\partial\mathcal{L}_{\mathrm{align}}}{\partial\mathbf{H}_{\mathrm{bb}}^{(K)}}}_{\text{from deep backbone}} |
| + |
| \underbrace{\frac{\partial(\mathcal{L}_{\mathrm{select}}+\mathcal{L}_{\mathrm{align}})}{\partial\mathbf{H}_{\mathrm{pr}}^{(0)}}}_{\text{from middle-layer predictor}} |
| \right) |
| \cdot |
| \prod_{m=l+1}^{K-1}\frac{\partial\mathbf{X}^{(m+1)}}{\partial\mathbf{X}^{(m)}} |
| \cdot |
| \frac{\partial\mathbf{X}^{(l+1)}}{\partial\theta_{f_l}}. |
| \label{eq:prefix_grad} |
| \end{equation} |
| The gradient with respect to a per-layer IWA scalar |
| $\textcolor{reusecolor}{\gamma_l}$ is simpler and acquires a closed form |
| because the bias enters the attention logit linearly: |
| \begin{equation} |
| \frac{\partial\mathcal{L}_{\mathrm{GroundFlow}}}{\partial\textcolor{reusecolor}{\gamma_l}} |
| \;=\; |
| \sum_{i,j} |
| \hat{\alpha}_{ij}^{(l)} |
| \!\left( |
| \frac{\partial\mathcal{L}_{\mathrm{align}}}{\partial o_i^{(l)}}\!\cdot\! v_j^{(l)} |
| - |
| \!\sum_{k}\hat{\alpha}_{ik}^{(l)} |
| \frac{\partial\mathcal{L}_{\mathrm{align}}}{\partial o_i^{(l)}}\!\cdot\! v_k^{(l)} |
| \!\right) |
| \textcolor{selectcolor}{s_j}, |
| \label{eq:gamma_grad} |
| \end{equation} |
| where $o_i^{(l)}=\sum_j\hat{\alpha}_{ij}^{(l)}v_j^{(l)}$. |
| Equation~\ref{eq:gamma_grad} shows that |
| $\textcolor{reusecolor}{\gamma_l}$ increases whenever attending more strongly |
| to high-saliency tokens reduces $\mathcal{L}_{\mathrm{align}}$, so the |
| language-modeling signal trains the per-layer importance bias end-to-end. |
|
|
| \section{Experiments} |
| \label{sec:experiments} |
|
|
| \subsection{Experimental Setup} |
| \label{subsec:experimental_setup} |
|
|
| \textbf{Models.} |
| We instantiate GroundFlow at two model scales, denoted as GroundFlow-$4$B and GroundFlow-$8$B, built on the corresponding Qwen3-VL-Instruct backbones. |
| For both scales, we set $K=24$ and $T=3$, where $K$ denotes the decoder layer at which the grounding predictor branches from the backbone, and $T$ denotes the number of predictor layers. The grounding predictor is a $3$-layer shallow path that bifurcates at decoder layer $24$ and shares the language-model head with the backbone. |
| |
| |
| |
| |
|
|
| \textbf{Baselines.} |
| We compare against three baselines that progressively ablate the components |
| of GroundFlow: |
| \textsc{Vanilla} (no auxiliary post-training), |
| \textsc{SD-RPN} (region-aware predictor trained on calibrated ROI BCE only, |
| with the backbone path detached and no IWA bias), and |
| \textsc{GroundFlow w/o IWA} (BCE supervision and KL alignment active with |
| $\alpha{=}2/3$, but |
| $\gamma_l{=}0$, i.e.\ retrieval and KL alignment are trained but the |
| retrieved importance is not injected back into the deep attention). |
| For a fair comparison, all methods are post-trained on the same $185$K |
| multimodal corpus, use the same shared prefix and predictor architecture, |
| and are evaluated under identical inference pipelines for the four input |
| resolutions. |
|
|
| \textbf{Benchmarks.} |
| We evaluate GroundFlow on a diverse set of grounding-related multimodal benchmarks, including OCRBench~\cite{liu2024ocrbench}, InfoVQA~\cite{mathew2022infographicvqa}, DocVQA~\cite{mathew2020docvqa}, ChartQA~\cite{masry2022chartqa}, TextVQA~\cite{singh2019towards}, V*Bench~\cite{vstar}, HRBench at 4K and 8K resolutions~\cite{hrbench}, RealWorldQA~\cite{}, and MME-RealWorld~\cite{zhang2024mme}. |
| These benchmarks cover document understanding, OCR-centric reasoning, chart interpretation, fine-grained visual grounding, and high-resolution perception. |
| We report official benchmark metrics and use the average score as the overall measure. |
|
|
| \textbf{Evaluation protocol.} |
| We evaluate GroundFlow under four progressively increased input resolutions, $\{576,1024,2048,4096\}$, corresponding to progressively higher-resolution visual encoding. |
| For each model scale, we use the same trained GroundFlow configuration across all resolutions without retraining. |
| During evaluation, the learned IWA scalars are kept fixed, and the retrieval threshold for the first forward is selected from $\theta\in\{0.10,0.15\}$. |
| The second forward then reuses the shared-prefix KV cache and incorporates the retrieved regional tokens for final generation. |
|
|
| \subsection{Main Results} |
| \label{subsec:main_results} |
|
|
| Table~\ref{tab:main_results} presents the main comparison across model |
| scales and benchmarks at the standard input resolution of $576$ visual |
| tokens. |
| GroundFlow consistently improves the multimodal foundation model on both |
| document/OCR and fine-grained-grounding/high-resolution groups. |
| The advantage is most pronounced on V*Bench, OCRBench, and HRBench, where |
| the calibrated region-level supervision and the IWA bias most directly |
| amplify the visual tokens that carry text or fine-grained spatial detail. |
|
|
| \begin{table*}[t] |
| \centering |
| \caption{ |
| \textbf{Main results at input resolution $576$.} |
| We compare GroundFlow with the vanilla backbone and related post-training variants under the same training data and inference setting. |
| GroundFlow consistently improves grounding-related performance across model scales. |
| We report official metrics for each benchmark and the average score for overall comparison. |
| |
| } |
| \label{tab:main_results} |
| \resizebox{\textwidth}{!}{ |
| \begin{tabular}{@{} l | c c c c c | c c c c c || c @{}} |
| \toprule |
| \multirow{2}{*}{\textbf{Method}} |
| & \multicolumn{5}{c|}{\textbf{Document \& OCR}} |
| & \multicolumn{5}{c||}{\textbf{Fine-grained grounding \& high-resolution}} |
| & \multirow{2}{*}{\textbf{Avg.}} \\ |
| \cmidrule(lr){2-6}\cmidrule(lr){7-11} |
| & OCR & Info & Doc & Chart & Text |
| & V* & HR4K & HR8K & RWQA & MME-RW & \\ |
| \midrule |
|
|
| \multicolumn{12}{@{}c}{\emph{4B-scale model}} \\ |
| \midrule |
| Qwen3-VL-4B-Instruct |
| & 839 & 68.0 & 91.2 & 83.9 & 79.3 |
| & 62.8 & 63.6 & 56.1 & 68.6 & 52.1 |
| & 70.5 \\ |
| SD-RPN |
| & 843 & 75.2 & 92.5 & 83.7 & 78.3 |
| & 79.6 & 75.2 & 67.4 & 71.5 & 55.3 |
| & 76.30 \\ |
| GroundFlow w/o IWA |
| & 847 & 74.9 & 92.6 & 83.6 & 78.2 |
| & 81.2 & 73.3 & 67.0 & 72.4 & 53.1 |
| & 76.10 \\ |
| \rowcolor{ourslavender!50} |
| \textbf{GroundFlow} |
| & \textbf{852} & \textbf{76.0} & \textbf{92.7} & 83.5 & \textbf{78.4} |
| & \textbf{83.2} & 74.7 & 66.6 & \textbf{72.8} & 54.9 |
| & \textbf{76.80} \\ |
|
|
| \midrule |
| \multicolumn{12}{@{}c}{\emph{8B-scale model}} \\ |
| \midrule |
| Qwen3-VL-8B-Instruct |
| & 849 & 72.9 & 93.0 & 84.8 & 81.5 |
| & 67.0 & 66.2 & 59.1 & 69.4 & 53.3 |
| & 72.0 \\ |
| SD-RPN |
| & 862 & 80.3 & 94.4 & 85.1 & 79.9 |
| & 82.2 & 74.1 & 68.9 & 71.6 & 56.7 |
| & 77.9 \\ |
| GroundFlow w/o IWA |
| & 862 & 80.3 & 94.4 & 85.2 & 79.7 |
| & 80.1 & \textbf{75.5} & \textbf{69.4} & \textbf{73.1} & 56.1 |
| & 78.0 \\ |
| \rowcolor{ourslavender!50} |
| \textbf{GroundFlow} |
| & \textbf{866} & \textbf{80.3} & \textbf{94.6} & \textbf{85.4} & \textbf{79.8} |
| & 81.2 & 75.0 & \textbf{69.5} & 72.7 & 55.8 |
| & \textbf{78.09} \\ |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| \bottomrule |
| \end{tabular} |
| } |
| \vspace{-0.5em} |
| \end{table*} |
|
|
| \paragraph{Analysis.} |
| On GroundFlow-$4$B, our method improves V*Bench from $62.8$ to $83.2$, OCRBench from $839$ to $852$, and InfoVQA from $68.0$ to $76.0$. |
| The largest gain appears on V*Bench, which suggests that retrieving query-relevant regions and reusing them through IWA is especially effective for small or cluttered visual details. |
| The GroundFlow w/o IWA variant already improves over the grounding-predictor baseline on OCRBench and V*Bench, showing that intermediate-to-deep KL alignment provides a useful training signal. |
| Activating IWA brings further gains on OCR-heavy and fine-grained grounding benchmarks. |
| On GroundFlow-$8$B, our method reaches $866$ on OCRBench and $94.6$ on DocVQA, outperforming the corresponding grounding-predictor baseline and vanilla backbone. |
| It also achieves the best average score among all compared variants. |
| For high-resolution evaluation, GroundFlow-$8$B matches or improves HRBench performance while maintaining strong OCR and document understanding results. |
|
|
| \paragraph{Resolution scaling.} |
| Table~\ref{tab:resolution_scaling} reports the same configurations |
| evaluated across input resolutions of $\{576,1024,2048,4096\}$ visual |
| tokens. |
| The trained $\boldsymbol{\Theta}^{\star}$ is reused without retraining; |
| only the Pass-$1$/Pass-$2$ routing threshold is selected from |
| $\theta\in\{0.10,0.15\}$. |
| On $4$B, V*Bench rises from $83.2$ at $576$ to $92.1$ at $4096$ tokens |
| under the same GroundFlow configuration, while HR8K rises from $66.6$ to |
| $76.9$. |
| On $8$B, MME-RealWorld improves from $55.8$ at $576$ to $63.0$ at $4096$ |
| tokens, and InfoVQA from $80.3$ to $83.6$. |
| The advantage of the retrieval-and-reuse recipe is preserved across all |
| four resolutions, indicating that the trained operator is |
| resolution-invariant. |
|
|
| \begin{table}[t] |
| \centering |
| \caption{ |
| \textbf{Resolution scaling of GroundFlow.} |
| We evaluate GroundFlow under increasing input resolutions across grounding-related multimodal benchmarks. |
| } |
| \label{tab:resolution_scaling} |
| \resizebox{0.95\linewidth}{!}{ |
| \begin{tabular}{@{}l|c|cccccccccc@{}} |
| \toprule |
| \textbf{Method} & \textbf{Res.} |
| & OCR & Info & Doc & Chart & Text |
| & V* & HR4K & HR8K & RWQA & MME-RW \\ |
| \midrule |
| \multirow{4}{*}{GroundFlow-4B} |
| & $576$ & 852 & 76.0 & 92.7 & 83.5 & 78.4 |
| & 83.2 & 74.7 & 66.6 & 72.8 & 54.9 \\ |
| & $1024$ & 854 & 80.3 & 93.9 & \underline{84.0} & 78.2 |
| & 84.8 & 78.6 & 73.5 & 73.1 & 60.1 \\ |
| & $2048$ & 853 & 81.9 & 94.6 & \underline{84.0} & \underline{80.4} |
| & 86.9 & 80.0 & 74.8 & \underline{73.6} & 62.8 \\ |
| & $4096$ & \underline{856} & \underline{83.3} & \underline{94.8} & \underline{84.0} & \underline{80.4} |
| & \underline{92.1} & \underline{82.0} & \underline{76.9} & \underline{73.6} & \underline{64.1} \\ |
| \midrule |
| \multirow{4}{*}{GroundFlow-8B} |
| & $576$ & \underline{866} & 80.3 & 94.6 & \underline{85.4} & \underline{79.8} |
| & 81.2 & 75.0 & 69.5 & 72.7 & 55.8 \\ |
| & $1024$ & 860 & 82.8 & 95.0 & 85.3 & \underline{79.8} |
| & 78.0 & \underline{79.1} & 70.5 & \underline{72.9} & 59.7 \\ |
| & $2048$ & 861 & 83.0 & 95.2 & 85.3 & \underline{79.8} |
| & 83.2 & 77.9 & 71.9 & 71.9 & 61.9 \\ |
| & $4096$ & 861 & \underline{83.6} & \underline{95.4} & 85.3 & \underline{79.8} |
| & \underline{86.9} & 78.6 & \underline{73.4} & 71.9 & \underline{63.0} \\ |
| \bottomrule |
| \end{tabular} |
| } |
| \end{table} |
|
|
|
|
| \section{Ablation Studies} |
| \label{sec:ablation} |
|
|
| We analyze GroundFlow from three complementary aspects: |
| \emph{(i)} the trade-off between grounding supervision and distribution alignment, controlled by the scalar $\alpha$ in Equation~\ref{eq:groundflow_objective}; |
| \emph{(ii)} the contribution of each component, including the grounding predictor, intermediate-to-deep KL alignment, and Importance-Weighted Attention (IWA); |
| and \emph{(iii)} the robustness of retrieval during inference under different alignment temperatures and retrieval thresholds. |
| Unless otherwise specified, all ablations use the same training data, backbone, and evaluation protocol, and vary only the factor under study. |
|
|
| |
| \subsection{Effect of the Grounding-Alignment Trade-off $\alpha$} |
| \label{sec:abl_alpha} |
| |
|
|
| The scalar $\alpha$ in Equation~\ref{eq:groundflow_objective} controls the relative strength of calibrated region-level supervision and intermediate-to-deep KL alignment. |
| We sweep $\alpha$ on Qwen3-VL-$4$B at input resolution $576$, while keeping the IWA scalars and the alignment temperature fixed. |
| Figure~\ref{fig:abl_alpha} shows that GroundFlow is broadly stable across a wide range of $\alpha$ values. |
| Document- and OCR-related benchmarks remain robust when $\alpha$ varies, while fine-grained grounding is more sensitive to the balance between region supervision and distribution alignment. |
| The best performance is obtained at an interior value of $\alpha$, rather than at either extreme. |
| This indicates that calibrated region-level supervision and intermediate-to-deep KL alignment are complementary: the former improves query-relevant region localization, while the latter preserves consistency with deep-layer generation. |
|
|
| \begin{figure*}[t] |
| \centering |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_ocr.pdf} |
| \caption{OCRBench} |
| \label{fig:abl_alpha_ocr} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_info.pdf} |
| \caption{InfoVQA} |
| \label{fig:abl_alpha_info} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_vstar.pdf} |
| \caption{V*Bench} |
| \label{fig:abl_alpha_vstar} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig3_alpha_hr4k.pdf} |
| \caption{HRBench-$4$K} |
| \label{fig:abl_alpha_hr4k} |
| \end{subfigure} |
| \caption{ |
| \textbf{Effect of the grounding-alignment trade-off $\alpha$.} |
| We sweep $\alpha$ on GroundFlow-$4$B at input resolution $576$. |
| GroundFlow achieves the best overall performance at an interior trade-off, showing that region-level supervision and intermediate-to-deep alignment are complementary. |
| } |
| \label{fig:abl_alpha} |
| \end{figure*} |
|
|
| \begin{figure*}[t] |
| \centering |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_ocr.pdf} |
| \caption{OCRBench} |
| \label{fig:abl_comp_ocr} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_info.pdf} |
| \caption{InfoVQA} |
| \label{fig:abl_comp_info} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_vstar.pdf} |
| \caption{V*Bench} |
| \label{fig:abl_comp_vstar} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig5_addcomp_hr4k.pdf} |
| \caption{HRBench-$4$K} |
| \label{fig:abl_comp_hr4k} |
| \end{subfigure} |
| \caption{ |
| \textbf{Component analysis of GroundFlow.} |
| We progressively add the grounding predictor, intermediate-to-deep KL alignment, and IWA to the vanilla backbone. |
| Each component contributes to the final performance, with IWA providing consistent additional gains after region retrieval and distribution alignment. |
| } |
| \label{fig:abl_components} |
| \end{figure*} |
|
|
| |
| \subsection{Component Analysis} |
| \label{sec:abl_components} |
| |
|
|
| We next study the contribution of each GroundFlow component on GroundFlow-$4$B at input resolution $576$. |
| Starting from the vanilla backbone, we progressively add: |
| \emph{(i)} the grounding predictor with calibrated region-level supervision, |
| \emph{(ii)} intermediate-to-deep KL alignment, |
| and \emph{(iii)} IWA. |
| Figure~\ref{fig:abl_components} reports the results on OCRBench, InfoVQA, V*Bench, and HRBench-$4$K. |
| The grounding predictor provides the main improvement on fine-grained grounding, showing the importance of explicitly retrieving query-relevant visual tokens from intermediate representations. |
| Adding intermediate-to-deep KL alignment further improves generation consistency, while IWA provides an additional gain by injecting predicted token importance into deep-layer attention. |
| These results show that GroundFlow benefits from the joint coupling of region retrieval, distribution alignment, and importance-guided deep attention. |
|
|
| |
| \subsection{Effect of the Alignment Temperature $\tau$} |
| \label{sec:abl_tau} |
| |
|
|
| We further study the effect of the alignment temperature $\tau$ in Equation~\ref{eq:kl_align}. |
| The temperature controls the softness of the intermediate-to-deep KL target. |
| We sweep $\tau$ on Qwen3-VL-$8$B at input resolution $576$, while keeping the grounding-alignment trade-off and IWA scalars fixed. |
| Figure~\ref{fig:abl_tau} shows that GroundFlow is robust to the choice of $\tau$. |
| The best performance is generally obtained around $\tau\in[1.5,2.0]$, and performance remains stable across a moderate temperature range. |
| This suggests that GroundFlow does not rely on a highly sensitive temperature setting, while still benefiting from softened distribution alignment. |
|
|
| \begin{figure*}[t] |
| \centering |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig4_tau_ocr.pdf} |
| \caption{OCRBench} |
| \label{fig:abl_tau_ocr} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig4_tau_info.pdf} |
| \caption{InfoVQA} |
| \label{fig:abl_tau_info} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig4_tau_vstar.pdf} |
| \caption{V*Bench} |
| \label{fig:abl_tau_vstar} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig4_tau_hr4k.pdf} |
| \caption{HRBench-$4$K} |
| \label{fig:abl_tau_hr4k} |
| \end{subfigure} |
| \caption{ |
| \textbf{Effect of the alignment temperature $\tau$.} |
| We sweep $\tau$ on Qwen3-VL-$8$B at input resolution $576$ with fixed IWA scalars and loss trade-off. |
| GroundFlow is robust to $\tau$, with the best performance around $\tau\in[1.5,2.0]$. |
| } |
| \label{fig:abl_tau} |
| \end{figure*} |
|
|
| |
| \subsection{Robustness to Retrieval Thresholds} |
| \label{sec:abl_threshold} |
| |
|
|
| Finally, we evaluate the robustness of the first-forward retrieval threshold $\theta$ in Equation~\ref{eq:roi_extract}. |
| The threshold controls the region mask produced from the predicted grounding map and therefore affects the regional tokens reused in the second forward. |
| We vary $\theta$ on Qwen3-VL-$4$B at input resolution $576$ while keeping the trained GroundFlow configuration unchanged. |
| Figure~\ref{fig:abl_threshold} shows that GroundFlow remains stable across a broad threshold range. |
| OCRBench maintains a stable plateau under moderate thresholds, while V*Bench favors a slightly lower threshold that retains more fine-grained regional information. |
| Performance degrades only when the threshold becomes overly restrictive, which may remove query-relevant visual tokens before the second forward. |
| Overall, the retrieval process is not sensitive to the precise threshold and does not require retraining for different threshold choices. |
|
|
| \begin{figure*}[t] |
| \centering |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_ocr.pdf} |
| \caption{OCRBench} |
| \label{fig:abl_thresh_ocr} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_info.pdf} |
| \caption{InfoVQA} |
| \label{fig:abl_thresh_info} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_vstar.pdf} |
| \caption{V*Bench} |
| \label{fig:abl_thresh_vstar} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.24\textwidth} |
| \centering\includegraphics[width=\linewidth]{figures/fig2_thresh_hr4k.pdf} |
| \caption{HRBench-$4$K} |
| \label{fig:abl_thresh_hr4k} |
| \end{subfigure} |
| \caption{ |
| \textbf{Robustness to retrieval thresholds $\theta$.} |
| We vary the first-forward retrieval threshold while keeping the trained GroundFlow configuration fixed. |
| |
| } |
| \label{fig:abl_threshold} |
| \vspace{-1em} |
| \end{figure*} |
|
|
|
|
| \section{Conclusion} |
| \label{sec:conclusion} |
|
|
| We presented \textbf{GroundFlow}, a lightweight post-training framework for fine-grained multimodal grounding. GroundFlow retrieves query-relevant visual tokens from intermediate representations, aligns intermediate prediction with deep-layer generation through KL distillation, and injects token importance into deep attention via IWA. During inference, the retrieved regional tokens are reused in a second forward with shared-prefix KV cache reuse, providing additional regional context for grounded generation. By connecting intermediate grounding prediction with deep language generation, GroundFlow offers a scalable alternative to efficient post-training for improving fine-grained visual grounding. |
|
|
|
|
|
|
|
|
|
|
|
|
| |
| |
|
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|
| |
| |
|
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
|
|
|
|
| |
| |
| |
| |
| |
|
|
| |
| |
|
|
| |
|
|
|
|
| |
| |
| |
| |
| |
| |
|
|
|
|
| |
| |
|
|
|
|
| |
| |
| |
| |
|
|
|
|
| |
| |
| |
|
|
|
|
| |
| |
| |
| |
|
|
|
|
| |
|
|
| |
|
|
| |
| |
|
|
| |
|
|
| \bibliographystyle{plainnat} |
| \bibliography{references} |
| |
|
|
|
|
|
|
| |
| \newpage |
|
|
| \appendix |
|
|
| \section{Technical appendices and supplementary material} |
|
|
| \subsection{Experimental Details} |
| \label{sec:exp_details} |
|
|
| \paragraph{Training data and spatial supervision.} |
| We post-train GroundFlow using $185$K image-text samples with pseudo spatial supervision. |
| For each sample, we construct a region-of-interest map from teacher attention patterns and use it as the supervision signal for intermediate visual evidence selection. |
| This avoids the need for manually annotated bounding boxes while still providing dense spatial guidance for text-rich, document-centric, and fine-grained visual inputs. |
|
|
| \paragraph{Model architecture.} |
| GroundFlow is built on Qwen3-VL backbones. |
| We attach a lightweight intermediate grounding branch after the $24$-th decoder layer. |
| The branch contains three auxiliary decoder layers initialized from the corresponding backbone layers. |
| Given the intermediate hidden states, the grounding branch estimates a spatial importance map over visual tokens by aggregating text-to-vision query-key interactions across attention heads. |
| This map is used for two purposes. |
| First, it supervises intermediate visual evidence selection during post-training. |
| Second, it identifies query-relevant regions for high-resolution re-encoding at inference time. |
|
|
| \paragraph{Post-training objective.} |
| GroundFlow combines spatial supervision with distributional alignment between intermediate and deep representations. |
| The training loss is defined as |
| \begin{equation} |
| \mathcal{L} |
| = |
| \alpha \mathcal{L}_{\mathrm{ROI}} |
| + |
| (1-\alpha)\mathcal{L}_{\mathrm{KL}}, |
| \end{equation} |
| where $\mathcal{L}_{\mathrm{ROI}}$ is the binary spatial supervision loss and $\mathcal{L}_{\mathrm{KL}}$ aligns the intermediate branch distribution with the deeper language-generating distribution using temperature-scaled KL distillation. |
| The coefficient $\alpha$ controls the balance between spatial evidence learning and distributional alignment. |
| For the 4B-scale model, we use $\alpha=0.667$ and temperature $\tau=2.0$. |
| For the 8B-scale model, we use $\alpha=0.667$ and temperature $\tau=1.5$. |
| We do not add a separate language-modeling cross-entropy term, which keeps the post-training objective focused on spatial evidence selection and representation alignment. |
|
|
| \paragraph{Intermediate-to-deep attention guidance.} |
| To propagate intermediate grounding cues into deeper language-generating layers, GroundFlow uses intermediate-to-deep attention guidance. |
| The grounding branch first predicts visual-token importance from intermediate hidden states. |
| The resulting importance scores are converted into a lightweight additive attention bias for the subsequent backbone layers. |
| This design encourages deep layers to attend more strongly to query-relevant visual evidence without adding extra visual tokens, external proposal modules, or additional inference-time model components. |
| The guidance is parameter-efficient and only introduces a small set of learnable layer-wise scaling factors. |
|
|
| \paragraph{Inference.} |
| At inference time, GroundFlow first predicts a region-of-interest map from the intermediate grounding branch. |
| The map is smoothed and thresholded to obtain a compact visual region, which is then cropped from the original image and re-encoded at high resolution. |
| The re-encoded regional features are inserted into the multimodal sequence after the original visual tokens. |
| To avoid redundant computation, the prefix key-value states before the grounding branch are reused when processing the enhanced sequence. |
| The later backbone layers then perform a full forward pass over the combined global and regional visual evidence. |
|
|
| \paragraph{Optimization.} |
| All models are post-trained for one epoch with AdamW, a learning rate of $1\times10^{-4}$, cosine decay, $3\%$ warmup, and gradient clipping at $1.0$. |
| We use bf16 precision and distributed data parallel training. |
| For the 4B-scale model, training uses an effective batch size of $128$ and runs for $1450$ optimization steps. |
| The same pseudo-ROI training set, branch design, and optimization recipe are used across model scales unless otherwise specified. |
|
|
| \paragraph{Evaluation.} |
| We evaluate all methods under the same training data and inference setting. |
| The main evaluation includes document and OCR-oriented benchmarks, including OCRBench, InfoVQA, DocVQA, ChartQA, and TextVQA. |
| We also evaluate fine-grained grounding and high-resolution perception on V*Bench, HRBench4K, HRBench8K, RealWorldQA, and MME-RealWorld. |
| All results are reported using the official metric of each benchmark. |
| For the average score, we normalize benchmark-specific metrics to comparable percentage-style scales, while OCRBench is divided by $10$ before averaging. |
|
|
| \begin{table*}[t] |
| \centering |
| \caption{ |
| \textbf{Post-training and inference details of GroundFlow.} |
| We report the main settings used for the 4B- and 8B-scale models. |
| } |
| \label{tab:training_details} |
| \resizebox{\textwidth}{!}{ |
| \begin{tabular}{@{}lcc@{}} |
| \toprule |
| \textbf{Configuration} & \textbf{4B-scale model} & \textbf{8B-scale model} \\ |
| \midrule |
| Base backbone & Qwen3-VL-4B-Instruct & Qwen3-VL-8B-Instruct \\ |
| Input resolution & $576$ visual tokens & $576$ visual tokens \\ |
| Post-training data & $185$K pseudo-ROI samples & $185$K pseudo-ROI samples \\ |
| Grounding branch position & Layer $24$ & Layer $24$ \\ |
| Grounding branch depth & $3$ decoder layers & $3$ decoder layers \\ |
| Spatial map construction & Text-to-vision QK aggregation & Text-to-vision QK aggregation \\ |
| Spatial supervision & Pseudo ROI map & Pseudo ROI map \\ |
| Spatial loss & Binary cross-entropy & Binary cross-entropy \\ |
| Distribution alignment & Temperature-scaled KL & Temperature-scaled KL \\ |
| ROI loss weight $\alpha$ & $0.667$ & $0.667$ \\ |
| Distillation temperature $\tau$ & $2.0$ & $1.5$ \\ |
| Language-modeling CE loss & $0.0$ & $0.0$ \\ |
| Intermediate-to-deep guidance & Additive attention bias & Additive attention bias \\ |
| Guidance parameters & Layer-wise scalars & Layer-wise scalars \\ |
| Regional enhancement & ROI crop re-encoding & ROI crop re-encoding \\ |
| Computation reuse & Prefix KV reuse & Prefix KV reuse \\ |
| Optimizer & AdamW & AdamW \\ |
| Learning rate & $1\times10^{-4}$ & $1\times10^{-4}$ \\ |
| Scheduler & Cosine decay & Cosine decay \\ |
| Warmup ratio & $0.03$ & $0.03$ \\ |
| Weight decay & $0.0$ & $0.0$ \\ |
| Max gradient norm & $1.0$ & $1.0$ \\ |
| Precision & bf16 & bf16 \\ |
| Training epochs & $1$ & $1$ \\ |
| Optimization steps & $1450$ & $1450$ \\ |
| Effective batch size & $128$ & $128$ \\ |
| Training parallelism & Distributed data parallel & Distributed data parallel \\ |
| \bottomrule |
| \end{tabular} |
| } |
| \vspace{-0.5em} |
| \end{table*} |
|
|
| \paragraph{Training dynamics.} |
| Figure~\ref{fig:app-train-dynamics} reports the training loss and gradient norm for |
| the $4$B and $8$B GroundFlow post-training runs. After a short transient the loss |
| settles at a non-zero plateau ($\approx 0.5$--$0.8$ nats): it is dominated by the |
| intermediate-to-deep KL term, and the two distributions being aligned ---the |
| intermediate grounding branch and the deeper language-generating distribution--- |
| remain meaningfully different throughout, which is exactly the residual signal the |
| alignment term transfers into the deep layers. Gradient norms are stable over the |
| full schedule at both scales (no spikes, no divergence). Hyperparameters are as in |
| Table~\ref{tab:training_details}. |
|
|
| \begin{figure*}[t] |
| \centering |
| \begin{subfigure}[t]{0.49\linewidth} |
| \centering\includegraphics[width=\linewidth]{figures/figA_loss_4b.pdf} |
| \caption{Qwen3-VL-$4$B --- training loss} |
| \label{fig:app-loss-4b} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.49\linewidth} |
| \centering\includegraphics[width=\linewidth]{figures/figA_loss_8b.pdf} |
| \caption{Qwen3-VL-$8$B --- training loss} |
| \label{fig:app-loss-8b} |
| \end{subfigure} |
|
|
| \vspace{0.4em} |
| \begin{subfigure}[t]{0.49\linewidth} |
| \centering\includegraphics[width=\linewidth]{figures/figA_gn_4b.pdf} |
| \caption{Qwen3-VL-$4$B --- gradient norm} |
| \label{fig:app-gn-4b} |
| \end{subfigure}\hfill |
| \begin{subfigure}[t]{0.49\linewidth} |
| \centering\includegraphics[width=\linewidth]{figures/figA_gn_8b.pdf} |
| \caption{Qwen3-VL-$8$B --- gradient norm} |
| \label{fig:app-gn-8b} |
| \end{subfigure} |
| \caption{\textbf{Training dynamics of GroundFlow.} Training loss (top) and |
| gradient norm (bottom) for the $4$B and $8$B post-training runs; curves are |
| EMA-smoothed with the raw trace shown faintly. After a brief transient the loss |
| settles at the irreducible intermediate-to-deep KL component |
| ($\approx 0.5$--$0.8$ nats; bold numbers mark the final value), and the gradient |
| norm stays stable over the full schedule. All runs: $185$K pseudo-ROI samples, |
| one epoch, $1450$ optimization steps (Table~\ref{tab:training_details}).} |
| \label{fig:app-train-dynamics} |
| \end{figure*} |
|
|
| \subsection{Limitations} |
| \label{sec:limitations} |
|
|
| GroundFlow demonstrates that intermediate spatial supervision and intermediate-to-deep guidance can improve fine-grained grounding, but several directions remain open. |
| First, the current spatial supervision is obtained from pseudo ROI maps derived from teacher attention patterns. |
| This makes the method scalable and annotation-efficient, but the quality of the supervision is still bounded by the teacher signal. |
| Cleaner pseudo-label filtering, human-verified region annotations, or instance-level spatial supervision may further improve the learned grounding maps. Second, our regional enhancement currently uses a simple crop-based strategy. |
| This keeps the inference pipeline lightweight and easy to reproduce, but a single compact region may be suboptimal when the relevant evidence is distributed across multiple distant areas. Future work may explore multi-region re-encoding, adaptive region selection, or stronger layout-aware region composition. |
|
|
| |
|
|
| \newpage |
| |
|
|
|
|
| \end{document} |
| |
| \subsection{Fine-Grained Multimodal Perception and Grounding} |
| Conventional Multimodal Large Language Models (MLLMs)~\cite{liu2023llava}, such as LLaVA, typically employ CLIP or its variants as vision encoders to align visual features with language representations. However, constrained by the limited input resolution of these encoders, such models struggle with tasks that demand fine-grained perception, including OCR, document understanding, and GUI localization. To mitigate this, subsequent research has introduced split-and-tile strategies and native-resolution Vision Transformers (ViTs), which significantly enhance fine-grained perception. Nonetheless, because standard vision encoders compress images at a fixed ratio, processing high-resolution inputs generates thousands of visual tokens, leading to substantial computational overhead. Moreover, minute visual details are often overwhelmed by a vast number of background tokens, impeding the LLM's ability to retrieve relevant information. To overcome these challenges, recent works have proposed a ground-then-perceive framework. In this approach, small visual evidence is first localized using external models, sliding windows, or the MLLM's internal attention mechanisms. These grounded regions are then re-encoded to provide complementary inputs that bolster the model's perceptual accuracy. Despite their effectiveness, these methods typically rely on handcrafted pipelines and multiple prefilling passes, which introduce significant efficiency bottlenecks and restrict their overall generality. |
| To overcome these limitations, GroundFlow introduces a lightweight post-training procedure that reuses retrieved visual tokens through an additive attention bias. By avoiding redundant computations, this approach ensures a highly efficient grounding process without compromising the inherent generality of the MLLM. |
| |
| \bibliography{references} |
|
|
