Hugging Face's logo

ratschlab
/
DeepSpotM

Image Feature Extraction
Safetensors
English
deepspotm
biology
medical
histology
pathology
spatial-transcriptomics
gene-expression
lora
computational-pathology
foundation-model
multimodal
virtual-spatial-transcriptomics
whole-slide-imaging
oncology
transcriptomics
deep-learning
cancer
Model card Files
xet
 Community
1
DeepSpotM / README.md
nonchev's picture
nonchev
Update README.md
48be27a verified
preview code
|
Raw
History Blame Contribute Delete
10.8 kB
metadata 
license: cc-by-nc-sa-4.0
library_name: deepspotm
pipeline_tag: image-feature-extraction
base_model: kaiko-ai/midnight
base_model_relation: adapter
language:
  - en
tags:
  - biology
  - medical
  - histology
  - pathology
  - spatial-transcriptomics
  - gene-expression
  - lora
  - computational-pathology
  - foundation-model
  - multimodal
  - virtual-spatial-transcriptomics
  - whole-slide-imaging
  - oncology
  - transcriptomics
  - deep-learning
  - cancer
datasets:
  - ratschlab/TCGA_virtual_spatial_transcriptomics_atlas
  - ratschlab/HEST_Xenium_virtual_spatial_transcriptomics
extra_gated_heading: Eligibility declarations required to request access
extra_gated_prompt: >-
  DeepSpotM is released under CC-BY-NC-SA-4.0 strictly for non-commercial
  academic research.

  ELIGIBILITY. Access is granted only to individuals whose affiliations are
  exclusively academic or public non-profit research institutions. Anyone
  holding any concurrent commercial affiliation — employment, consulting,
  advisory roles, internships, or founding roles at a company or startup — is
  not eligible, regardless of intended use.

  SCOPE. Research performed at, for, funded by, or in collaboration with a
  commercial entity is commercial use and is prohibited. This includes corporate
  R&D, internal evaluation, benchmarking, and proof-of-concept work, even if
  exploratory and never productized. "Research" conducted in a commercial
  setting does not qualify as non-commercial academic research.

  REVIEW. All requests are reviewed manually against these criteria. Vague,
  incomplete, or inaccurate submissions will be declined. False or incomplete
  declarations void any access granted and constitute a violation of the license
  terms. For commercial licensing, contact the maintainers.
extra_gated_fields:
  Full name (first and last): text
  Academic affiliation (full official name, no abbreviations): text
  Position at this institution:
    type: text
    help: E.g. PhD student, postdoc, research scientist, professor.
  Official institutional email:
    type: text
    help: >-
      Must match the primary email on your Hugging Face account and belong to a
      recognized academic or public research institution's domain. Personal
      domains (gmail, hotmail, qq, etc.) and corporate domains will be denied.
  Concurrent commercial affiliations (write "None" if none):
    type: text
    help: >-
      Disclose ALL current commercial ties: employment, consulting, advisory
      positions, internships, and founded or co-founded startups. Individuals
      with any concurrent commercial affiliation are not eligible. Failure to
      disclose is grounds for denial and revocation of access.
  Intended use (be specific):
    type: text
    help: >-
      Name your institution/lab, the broader research project or area, and
      concretely how the model will be used (e.g. model training, benchmarking,
      validation). Generic answers such as "for research purposes" will be
      declined.
  I declare that I hold no concurrent affiliation with any company, startup, or other commercial entity, and that I am requesting access solely in my academic capacity: checkbox
  I declare that this model, its weights, outputs, and any derivatives will be used exclusively for non-commercial academic research, and will not be used to train, fine-tune, or distill any model intended for commercial use or release under a different license: checkbox
  I declare that I will not share the model or its weights with any commercial entity or any individual who has not been granted access, and will not re-upload or mirror the weights in any location that bypasses this access process: checkbox
  I declare that if my status changes and I take on any commercial affiliation, or if my access is revoked, I will cease all use and delete all copies of the model and its weights: checkbox
  I declare that all information in this request is truthful and complete, and I understand that any misrepresentation voids my access and violates the license terms: checkbox
  I confirm that I have read the license terms and agree to retain attribution, license any derivatives under the same license, and cite the DeepSpotM publication in any work using this model:
    type: checkbox
    help: >-
      The license is CC-BY-NC-SA-4.0. Redistribution and derivatives are
      permitted only under this same license, with attribution and
      non-commercial use.
  I agree to receive news and updates about this technology: checkbox
extra_gated_button_content: Submit declarations and request access

DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology

Authors: Kalin Nonchev, Sebastian Dawo, Karina Silina, Viktor Hendrik Koelzer, and Gunnar Rätsch.

The preprint is available here.

News

  • [09.2026] Introducing Aurora - a no-code platform for virtual spatial transcriptomics from H&E images. Aurora makes virtual spatial transcriptomics accessible without requiring users to have their own GPUs, with a fully automated workflow designed to ensure that spatial transcriptomics predictions are performed correctly. Try the interactive Aurora demo.

DeepSpot-M is a multimodal foundation model that maps a histology image tile to spatial gene expression. It tokenises a 224x224 H&E tile with a LoRA-adapted pathology foundation backbone (Midnight) and lets each gene query attend to the patch tokens through a cross-attention gene decoder. A gene router hypernetwork generates gene-specific output projections from frozen biological embeddings drawn from DNA, RNA, protein, single-cell and text foundation models (Evo 2, Orthrus, ProtT5, scGPT, Apertus). Because genes are represented as queryable embeddings rather than fixed outputs, one model predicts transcriptome-wide expression and genes it never saw during training.

Code is available on GitHub.

DeepSpot-M model overview

Fig. DeepSpot-M predicts transcriptome-wide spatial gene expression from histology. A 224x224 H&E tile is tokenised into spatial patch embeddings by a LoRA-adapted pathology foundation model. A cross-attention gene decoder lets each gene query independently attend to patch tokens via multi-head attention, and a gene router hypernetwork generates gene-specific output projections from frozen biological embeddings drawn from DNA, RNA, protein, single-cell and text foundation models. This design enables zero-shot prediction of genes at inference time.

⚠️ Research use only. Not for clinical or diagnostic use.

Model description

DeepSpot-M adapts the Midnight pathology backbone with LoRA and feeds its patch tokens to a cross-attention gene decoder conditioned on biological gene embeddings. It takes 224x224 H&E tiles as input and outputs expression over the ~19k-gene panel in tokens.csv. Five embedding sources are available, namely evo2, orthrus, prott5, scgpt and apertus, selected at inference with source=.

Usage 

from deepspotm import DeepSpotM   # pip install git+https://github.com/ratschlab/DeepSpotM.git

model, image_processor = DeepSpotM.from_pretrained(
    "ratschlab/DeepSpotM",
    source="scgpt",   # one of evo2, orthrus, prott5, scgpt, apertus
)

import torch
tile = image_processor(my_pil_tile).unsqueeze(0)   # 224x224 H&E tile
with torch.no_grad():
    expression, _, _ = model(tile)                 # (1, 19338)

# Output column i corresponds to model.gene_names[i].
preds = dict(zip(model.gene_names, expression.squeeze(0).tolist()))
print(preds["EPCAM"])

The predicted vector is ordered by model.gene_names, the genes in tokens.csv, so model.gene_names[i] is the symbol for output column i.

Predict only specific genes (faster)

You don't have to predict all ~19k genes. Pass a gene or a list and only those are computed, because the cross-attention runs over just the requested gene queries.

vals = model.predict_genes(tile, ["EPCAM", "CD3D", "PTPRC"])   # (1, 3)
vals = model.predict_genes(tile, "EPCAM")                       # (1, 1)

Output columns follow the requested order. Unknown symbols raise KeyError.

The vision backbone is built offline from a bundled config and its weights are baked into model.safetensors, so loading needs no network access to the upstream backbone repo.

Tutorial

examples/predict_tcga_skcm.ipynb runs DeepSpot-M end to end on a whole-slide TCGA-SKCM H&E image. It tiles the slide, predicts BRAF, CD37 and COL1A1, and overlays the predictions on the tissue.

Resources

Limitations and biases

  • Trained on a finite set of cancer indications. Performance on unseen tissue types, stains, scanners or resolutions may degrade.
  • Predicts relative expression rather than absolute counts. Under-sequenced genes are predicted less reliably.
  • Trained on oncology cohorts, so it is not representative of healthy tissue or non-oncology contexts. Not for clinical or diagnostic use.

License

See WEIGHTS_LICENSE.md and THIRD_PARTY_LICENSES.md.

Citation

Paper: DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology (medRxiv, 2026).

@article{nonchev2026deepspotm,
  title   = {DeepSpot-M: a multimodal foundation model for transcriptome-wide virtual spatial transcriptomics from histology},
  author  = {Nonchev, Kalin and Dawo, Sebastian and Silina, Karina and Koelzer, Viktor H. and Raetsch, Gunnar},
  journal = {medRxiv},
  year    = {2026},
  doi     = {10.64898/2026.06.19.26356060},
  url     = {https://www.medrxiv.org/content/10.64898/2026.06.19.26356060v1}
}

See also CITATION.cff.