NeuroQformer

NeuroQformer contains the QFormer weights used by NeuroVLM brain-map-to-text generation.

Files

  • model.safetensors: tensor weights for the QFormer wrapper.
  • config.json: architecture config and canonical semantic metadata.

The weights include the QFormer, frozen image projection head, and frozen canonical semantic projection banks. The canonical bank tensors are registered as buffers, so they are part of the model state and never trainable.

Load with:

from neurovlm.retrieval_resources import _load_neuro_qformer

model = _load_neuro_qformer(device="cpu", projection_temp=0.05, canonical_basis="network")

Set projection_temp=None to bypass canonical projection.

Revision 2026-10

Weights of the revised NeuroVLM paper. What changed:

  • Deterministic training. Every stage runs with seeded data loaders and deterministic kernels. Two runs from scratch gave byte-identical weights on the same GPU model, driver and library versions.
  • MLP projector. The brain-to-text projectors (stages 13 and 14) are residual MLPs (28.3 M parameters) instead of Q-Formers. A package names its projector in config.json["projector"].
  • Stage-14 recipe. Each training text is projected onto the canonical bank of its own level, every training map is re-binarized per batch, and title tokens weigh 5 in the loss. Five seeds were trained and the one with the lowest validation loss (seed 3) is packaged.
  • Calibration. Text-to-brain maps are multiplied by one global factor per model, fitted on the validation split and clipped at 1 (map_calibration.json, head 19.16 and adapter 6.13).
  • Title decoding. After a [NETWORK] or [REGION] tag the decoder writes one of the canonical names in title_vocabulary.json (48 network and 83 region titles), so it never invents a title.

model.safetensors and config.json hold the stage-14 MLP projector with the frozen image head and the canonical banks, which are also in canonical_semantic_banks.pt. title_vocabulary.json lists the titles the decoder may write. Loading needs neurovlm with ResidualMLPProjector (October 2026, NeuroQFormer.from_state_dict_payload reads config.json["projector"]). load_projector in NeuroVLM's scripts/models/_common_models.py reads the package too.

file size sha256
model.safetensors 118.0 MB eaf5ee494e96e40f9c75f2b9bd4aed0edcbe4589b8fafe9d0365402a5515a5a9
config.json 1.5 kB 6e366e8e6fd5b254daaf13e1d75f0e805790f1fdcd988f82c71e37d4ceb99fcf
canonical_semantic_banks.pt 3.4 MB f28293a0595afab0a137ba8cc96b8023b5562e7b464f7071a3d5230da7f20771
title_vocabulary.json 41.3 kB 31acb78f7f5ef304cd6e69d037d4f91af069883859b2919b2c5c1a79056bdf85

Provenance: scripts/outputs/runs/final of NeuroVLM (scripts/pipeline/train_pipeline.py, finished 2026-10-10T19:56:25), training code at git 51bedf4 with 131 modified files, inputs pinned by scripts/config/hf_manifest.json (sha256 49f3e3a9825b). scripts/README.md records the later runs from scratch that gave the same bytes.

Earlier weights: revision 29b45da49a50edb4a130c5134d600e3aad66c668.

Downloads last month
27
Safetensors
Model size
29.5M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support