vlm-data-in-weights

Small neural generators that store a vision-language QA corpus in their weights and replay it without reading the original files. This is an experiment in synthetic data generation for the VLM backbone of a vision-language-action (VLA) model: how faithfully a QA corpus with images can be kept in compact model weights and regenerated offline.

What the weights contain

The generators were trained to reproduce existing public data, not to invent new data: questions, answer options, correct answers and images of the publicly released items of these VLM benchmarks — BLINK, CrossPoint-Bench, EmbSpatial-Bench, ERQA, MMMU (validation), MMMU-Pro (standard and vision) and MSMU; 12,288 items in total. The original datasets' licenses and terms apply to the regenerated content.

Architecture

• Text (generator/benchmark/text_models/): 16 token-counter MLPs (width 1024, depth 6, about 5.3M parameters each). A position counter with binary/Fourier features is mapped to the ID of the token at that position (SmolVLM2 tokenizer, tokenizer/), until a learned EOS. Trained with BCE-with-logits/AdamW until the full text stream and EOS were reproduced exactly. These models memorize; they are not general language models.
• Images (generator/benchmark/image_models/): 29 latent-counter models (up to 512 images each). An image index is mapped through fixed cosine features and a fixed orthogonal layer to a learned linear projection into a 4×64×64 latent of the frozen stabilityai/sd-vae-ft-mse decoder (MIT, included under codec/), decoded to 512×512. Stored in FP16.
• MMMU-Pro vision pages: learned_vision_typesetting_v2.py re-typesets 1,507 of the 1,730 pages from the regenerated question text (DejaVu Sans, included under fonts/ with its license).

Fidelity (all items, measured against the 512×512 targets the generators were fitted to)

Text: 12,288 / 12,288 prompts and answers byte-identical, number of images per item identical. Images, median PSNR dB / SSIM: BLINK 30.0 / .888, CrossPoint 34.3 / .938, EmbSpatial 34.1 / .943, ERQA 34.0 / .947, MMMU-Pro standard 32.2 / .985, MMMU 31.2 / .983, MSMU 35.5 / .958. MMMU-Pro vision 12.2 / .530 — pixel metrics do not measure content there, because the pages are re-typeset (different layout, background and font).

Usage

python generator/benchmark/generate.py --export generator/benchmark --out OUT --groups <group names>

A CUDA GPU is required. Group names are listed in generator/benchmark/text_registry.json. The script contains a file-access guard that refuses to read the original corpora during generation, so that the output provably comes from the weights. Training code, resolved configs and audits are in generator_training/.

License

Our code and the trained weights as our work are released under MIT. The content the weights reproduce is not ours: regenerated benchmark items remain under the original terms of each benchmark (BLINK, MMMU, MMMU-Pro: Apache-2.0; ERQA, EmbSpatial-Bench, CrossPoint-Bench, MSMU: see their pages). The bundled stabilityai/sd-vae-ft-mse codec is MIT; DejaVu fonts are under their own license (fonts/LICENSE.txt).