Evolution of Multimodal Representations in Vision-Language Models

1. Drawing directly from your background and lived experience, identify a critical blind spot, bias, or capability gap in current AI models that is rarely captured by standard benchmarking datasets and that you feel is underexplored and underdiscussed.

Premise : Evolution of the Multimodal Representation Space of VLMs

During my experiments with charting reasoning benchmarks, I found a very interesting observation: current chart-reasoning benchmarks which are constructed around retrieval-type QA (e.g. DVQA, or ChartX), and even some rudimentary compositional understanding of the data visualization all fail when all the textual hints (like axis ticks, labels, legend etc.) are removed from the test set. This notion has been consolidated in the 2025 NeurIPS paper ChartMuseum. Another intuitive observation is that VLMs construct some implicit summary of the visualization in the textual space in the early stages, before establishing compositional binding between visual attributes (e.g., before knowing how to compare 2 bars based on purely visual attributes like size and color). I wouldn't overstate my claim, and hence would leave this at intuition.

However, the broad notion of how exactly is visual evidence carried forward into the output is an open question. Firstly, whether relevant visual evidence gets registered by the model, and if yes, how is it carried forward/evolves during the forward pass of the model. I wanted to study this from a mechanistic/representational perspective, and found that existing approaches are based on activation patching/steering, that capture model behaviours across linear directions (e.g. : "Geometric Failure Models of Vision-Language Models").

2. Propose a path forward: what specific data curation strategy, fine-tuning paradigm, or architectural adjustment do you think might close this gap? This can be purely speculative.

My proposed path is to first identify where visual evidence is preserved, transformed, or lost during a model's forward pass using structured toolkits of geometry and topology, and then use that evidence to design targeted training data and interventions. Rather than treating benchmark accuracy as the complete explanation, I want to study whether correct visual attribute binding and compositional reasoning hold as representational invariants, and for that I have initially probed them mechanistically layer-by-layer in the notebook. I would recommend you to walk through the notebook where I have walked step-by-step over various considerations.

Diagnose the gap through controlled comparisons

I start with Winoground, an image-caption pairing dataset which contains 2 images, and the corresponding captions for each imge per example. I choose the test split, which has 400 examples in total. Interestingly, the captions are essentially permutations of each other. So, not only must the model associate each visual concept with its corresponding textual span, but also learn the variance between their different relationships in the 2 images. Illustrations are given in the notebook, as well as other explanations. Its four image-caption pairings make it possible to investigate sensitivity to image swaps, caption swaps, and their combination. These transformations provide a controlled way to ask whether the representations distinguish the correct bindings from the incorrect ones.

The initial models are Qwen3.5-4B, Qwen3.5-0.8B, and Gemma-3-4B-IT. The Qwen models provide a within-family capacity comparison, while Gemma introduces a SigLIP-based vision encoder and local/global attention instead of Qwen's hybrid linear/full-attention decoder. These are not controlled architectural ablations: the models also differ in training, visual preprocessing, and tokenization.

First I benchmark the models, on this 400-query task, with 3 gradings caption-correctness, image-correctness and common-correctness.

Here are the considerations (to be read after going through the notebook) -

1. Even though Winoground is a good dataset to study binding and compositional relationships, it is quite small, and by some considerations in recent works, not an excellent measure to characterize invariance in representations because even for the relationship between 2 visual concepts, those concepts themselves are not uniform (in terms of resolution quality, small or out of focus etc.) and I suspect most of the images would still overlap with what the model has already seen.

2. Within the scope of this topological analysis, there are several confounds -

• Difference in topological landscape itself is not a measure of anything. Gemma's landscape seems to be topologically poorer as compared to Qwen landscapes, but that can be an artifact of the specific training/optimization procedure or the training data itself. Viewing this landscape in an ablation would make more sense, where we control 1 variable at a time, such as by generating some counterfactuals.
• Despite the theoretical stability guarantees that PH provides to perturbations, which shields us from cross-run stochasticity somewhat,
• Currently the intended contribution is to behaviorally scope differences in how different models would organize their activation landscapes, i.e., first asking whether these behaviours do exist from a topological lens, and then asking whether we can reliably profile them with the tool of PH that I am using, (including all the choices of point cloud sampling, choice of filtration function, choice of thresholds etc.) So there is obviously a risk of learning something that does not exist (sets of spurious correlations or fitting over noise).
• The current plots show model-dependent geometry, but they do not establish a general signature of correctness. Large activation-coordinate outliers and differences in activation scale can dominate persistence features. Before drawing mechanistic conclusions, I would -
  • Perform a batch-wise analysis, contrasting batches of successes vs batches of failures by normalizing them within their own distributions initially, and then a mixed-batch analysis of both successes and failures.
  • Compare raw and scale-controlled results, add outlier-removal and point-count controls, and repeat the analysis across larger sample groups, datasets, and perturbations.

The intended outcome is a diagnostic framework that complements accuracy and activation-patching experiments: it should help identify candidate failures of visual grounding, guide targeted data curation or fine-tuning, and then test whether those interventions improve behavior (basically tell where something went wrong). I do not assume that making a persistence diagram look more like a successful example will itself improve reasoning.

• 06_qwen35_winoground_colab.ipynb: the experiment notebook, including saved outputs and the author's Markdown discussion.
• 22Oct_results/: all available model benchmark results, including model/case comparisons, per-model outcome and failure tables, summaries, and available raw JSONL scores. Gemma's raw JSONL is not present in the source results; its outcome, failure, and summary files are included. The duplicate Qwen4B summary is retained as supplied.
• topology_pilot/: the complete extracted pilot and local analyses.
  • *_last_tokens.npz: last-token activations for the four pairings, layer labels, captions, and observed/benchmark scores.
  • *_capture_metadata.json and *_architecture.csv: checkpoint revisions, inference settings, and decoder block types.
  • persistence/: H0/H1 diagrams for all 87 adjacent-layer transitions across the three models, plus their summaries.
  • plots/: selected coordinate clouds and finite persistence diagrams.
  • comparison/: early/middle/late barcodes, cross-model feature plots, and CSV exports.
  • all_layers/: all-layer total-persistence plots, raw and scale-controlled features, depth-band summaries, and a compressed export of every barcode interval.

The two examples are illustrative rather than statistically representative. Infinite bars are preserved in the saved diagrams and barcode exports, but excluded from finite lifetime and total-persistence summaries.