vgtfm — histology-guided autoencoder on Geneformer embeddings

Model for Slide-level batch structure limits histology-guided supervision of transcriptomic foundation models (bioRxiv) trained on Geneformer embeddings. Code and an example notebook: ratschlab/vision-guided-transcriptomics-fm-2026.

Model

An encoder–decoder trained to predict the Midnight H&E embedding of each Visium spot from its Geneformer embedding. At inference only the encoder is used:

z = LayerNorm(Encoder(geneformer_embedding))      # 1152-d -> 128-d

Encoder	1152 → 512 → 256 → 128, BatchNorm + GELU + dropout 0.1, LayerNorm on the output
Decoder	128 → 512 → 1024 → 3072 (not needed for inference)
Training data	231,610 Visium spots from 96 slides (De Zuani LUAD and LUSC; MOSAIC bladder, DLBCL, glioblastoma, mesothelioma, ovarian)

Files

File	Contents
model.safetensors	weights: encoder.*, decoder.*, encoder_norm.*
config.json	architecture and training configuration
history.json	training and validation losses
examples/pbmc3k_geneformer.parquet	Geneformer embeddings of the 2,700 PBMC 3k cells, for the example notebook

The PBMC 3k embeddings are derived from 3k PBMCs from a Healthy Donor (10x Genomics), licensed under CC BY 4.0.

Usage

Refer to the paper and code for more details.

The input is a Geneformer V2 cell embedding, computed from raw counts with python -m vgtfm.embed.gene_fm --model geneformer --h5ad <file> from the repository. The example notebook pbmc3k_embeddings.ipynb on Github shows the full workflow.

Citation

@article{McConnell2026.09.30.755585,
  author    = {McConnell, Ulysse and Nonchev, Kalin and Koelzer, Viktor H and R{\"a}tsch, Gunnar},
  title     = {Slide-level batch structure limits histology-guided supervision of transcriptomic foundation models},
  journal   = {bioRxiv},
  year      = {2026},
  elocation-id = {2026.09.30.755585},
  doi       = {10.64898/2026.09.30.755585},
  publisher = {Cold Spring Harbor Laboratory},
  URL       = {https://www.biorxiv.org/content/early/2026/10/05/2026.09.30.755585}
}