README.md

metadata

license: cc-by-nc-nd-4.0
library_name: pytorch
tags:
  - spatial-proteomics
  - multiplexed-immunofluorescence
  - histology
  - pathology
  - single-cell
  - virtual-staining
extra_gated_prompt: >-
  Access to the CellPlex weights is granted on request. The CELLO models contain
  fine-tuned copies of UNI2-h, Virchow2 or H-optimus-0 and are subject to the
  licences and terms of use of these models. The weights may only be used for
  non-commercial academic research.
extra_gated_fields:
  Name: text
  Affiliation: text
  Intended use: text
  I will use these models only for non-commercial academic research and will not redistribute them: checkbox
extra_gated_button_content: Request access

CellPlex

CellPlex is a benchmark for predicting single-cell spatial proteomics from H&E. This repository holds its trained models: each one predicts the protein levels of every cell in an H&E image from the image and the nucleus centroids, for the protein panel of the cohort it was trained on. Training code, data sources and processing scripts: github.com/zjgao02/CellPlex.

Models

Folder	Model
cello/{encoder}/{cohort}/fold{k}/	Pathology foundation model adapted with CELLO: its token map is sampled at each nucleus, refined by a distance-decay cross-attention over the tile and decoded into the panel; trained end to end, so the folder holds the fine-tuned encoder
linear_probe/{encoder}/{cohort}/fold{k}/	Ridge regression on the frozen encoder's token at each nucleus; the folder holds the regression only, and the encoder is loaded from its own repository
eva_ft/{cohort}/fold{k}/	Eva fine-tuned on the cohort with its own H&E-to-multiplex objective, every panel protein requested by its marker name; run with the code of the GitHub repository
gigatime_ft/{cohort}/fold{k}/	GigaTIME fine-tuned on the cohort with its own pixel-level objective, one output channel per panel protein; run with the code of the GitHub repository

Encoders: hoptimus0 (H-optimus-0), uni2 (UNI2-h), virchow2 (Virchow2). CELLO with H-optimus-0 is the most accurate model on every cohort.

Each cohort has one model per fold: fold0 to fold4 for the five-fold cohorts, and fold0, the fixed split, for Orion-CRC and glioma. The model of fold k was trained without the test units of fold k, so every sample of the benchmark is predicted by the model of the fold that holds it out. The splits are in data/splits/ of the GitHub repository. For new data, any fold can be used; fold 0 is the default in the examples below.

Cohorts

Cohort	Tissue	Platform	µm/px	Proteins	Folds	Test samples	PCC: CELLO H-optimus-0 / UNI2-h / Virchow2	Linear probe H-optimus-0	Eva fine-tuned	GigaTIME fine-tuned
orion_crc	Colorectal cancer	Orion	0.325	16	1	8	0.620 / 0.615 / 0.595	0.590	0.500	0.426
crc_tma	Colorectal cancer	CODEX	0.377	56	5	130	0.530 / 0.512 / 0.508	0.432	0.491	0.356
multitumor	Multiple tumor types	CODEX	0.377	57	5	66	0.443 / 0.426 / 0.431	0.339	0.419	0.297
bc_crc	Breast, colorectal cancer	CODEX	0.507	36	5	45	0.531 / 0.514 / 0.514	0.421	0.456	0.317
hnscc_ln	HNSCC lymph nodes	CODEX	0.377	57	5	388	0.479 / 0.466 / 0.455	0.389	0.441	0.333
bone_marrow	Bone marrow	CODEX	0.249	52	5	11	0.620 / 0.607 / 0.602	0.538	0.548	0.358
glioma	Glioma	CODEX	0.249	68	1	9	0.503 / 0.492 / 0.480	0.413	0.365	0.323

PCC: per test sample, the Pearson correlation between predicted and measured levels across the sample's cells, averaged over the proteins of the panel, then over the test samples of all folds.

Files

modeling_cellplex.py                      CELLO, LinearProbe, load_model
inference.py                              prediction for a whole image and a table of nucleus centroids
cello/{encoder}/{cohort}/fold{k}/
    model.safetensors                     weights, including the fine-tuned encoder
    config.json                           architecture, encoder, cohort, fold and resolution
    proteins.txt                          the panel's markers, in output order
linear_probe/{encoder}/{cohort}/fold{k}/
    probe.safetensors                     ridge weight (D, P) and bias (P,)
    config.json, proteins.txt
eva_ft/{cohort}/fold{k}/
    model.safetensors                     Eva state dict
    config.json, proteins.txt
gigatime_ft/{cohort}/fold{k}/
    model.safetensors                     GigaTIME state dict with one output channel per panel protein
    config.json, proteins.txt

Quick start

Access requires approval: request it on this page, then log in with huggingface-cli login.

pip install torch timm huggingface_hub safetensors numpy pandas pillow opencv-python-headless openslide-python openslide-bin

Predict every cell of a slide (only the chosen model is downloaded):

python inference.py --model cello/hoptimus0/orion_crc/fold0 --image slide.tif --mpp 0.325 --cells nuclei.csv --out pred.csv

nuclei.csv has one row per cell with its centroid x, y in level-0 pixels; the output adds one column per protein. From Python:

import sys
from huggingface_hub import snapshot_download

repo = snapshot_download("gaozijun/CellPlex", allow_patterns=["*.py", "cello/hoptimus0/orion_crc/fold0/*"])
sys.path.insert(0, repo)
from modeling_cellplex import CELLO
from inference import predict_cells

model = CELLO.from_pretrained(repo, subfolder="cello/hoptimus0/orion_crc/fold0", device="cuda")
pred = predict_cells(model, "slide.tif", cx, cy, image_mpp=0.325)   # (cells, proteins); columns: model.proteins

model.predict works on a batch of 224×224 tiles at the model's resolution and the cell positions inside each tile:

import torch
images = torch.rand(2, 3, 224, 224)                      # RGB in [0, 1]
cell_xy = [torch.tensor([[50.0, 60.0], [120.5, 33.0]]),  # (x, y) pixel positions per tile
           torch.tensor([[10.0, 200.0]])]
with torch.autocast("cuda", dtype=torch.float16):
    preds = model.predict(images.cuda(), cell_xy)        # list of (n_cells, n_proteins) tensors

A linear probe is loaded the same way with LinearProbe.from_pretrained(repo, subfolder="linear_probe/hoptimus0/orion_crc/fold0"); it downloads the original encoder from its own repository, whose terms have to be accepted there first.

Inputs and outputs

• Image: RGB H&E. Each model works at the resolution of its cohort (µm/px in the table); inference.py resamples the image when --mpp is given.
• Cells: nucleus centroids. CellPlex segments nuclei on the H&E with CellViT++; see the GitHub repository.
• Output: per cell and protein of the cohort's panel, the predicted log1p of the mean intensity over the nucleus and a 2 µm rim, on the intensity scale of that cohort's multiplexed images. Levels are not comparable across cohorts.

Accuracy drops on tissues, stains and scanners unlike the training cohort. Use the model whose cohort is closest to your data and, where possible, check it against paired measurements.

Licence

The weights are released for non-commercial academic research under CC BY-NC-ND 4.0. The CELLO models contain fine-tuned copies of their encoders and are also subject to the encoder's licence and terms of use: UNI2-h (CC BY-NC-ND 4.0), Virchow2 (CC BY-NC-ND 4.0) and H-optimus-0 (Apache 2.0). The linear probes use the original encoders, which are obtained from their own repositories under their terms. The gigatime_ft models are fine-tuned from GigaTIME (Apache 2.0, gigatime_ft/LICENSE): every layer was trained further on the cohort, and the output layer was replaced by one channel per panel protein. As for GigaTIME, they are intended for research only and not for clinical or other deployed use. The eva_ft models are fine-tuned from Eva (Eva_ft.ckpt; every parameter trained further on the cohort) and remain subject to the terms under which Eva is distributed.