---
library_name: pytorch
tags:
- biology
- spatial-transcriptomics
- image-to-expression
---
# Histology-Based Reconstruction of Single-Cell Transcriptomics through multimodal foundation models

Predict spatial single-cell gene expression from H&E images using an image encoder, a spatial decoder, and a reference bank of cells with measured expression.

## Resources

- **Code:** [GitHub](https://github.com/zchxhcz1/VOICE)
- **Paper:** [arXiv:2608.08366](https://arxiv.org/abs/2608.08366)

## Installation

Use Python 3.10 or later and install a PyTorch build suitable for your hardware.

```bash
git clone https://github.com/zchxhcz1/VOICE.git
cd VOICE
pip install -r requirements.txt
```

For SVS, NDPI, or MRXS images, install the OpenSlide system library and `openslide-python`. PNG, JPEG, and TIFF images are also supported. Large slide images should use an OpenSlide-readable format to avoid loading the entire image into memory.

## Download the model

Download the checkpoints, fusion weights, gene information, and reference bank from Hugging Face. UNI2-h requires separate access through [MahmoodLab/UNI2-h](https://huggingface.co/MahmoodLab/UNI2-h).

After receiving UNI2-h access, authenticate with your own Hugging Face account:

```bash
hf auth login
python - <<'PYTHON'
from huggingface_hub import snapshot_download, hf_hub_download

model_dir = "models/voice"
snapshot_download("zchx/VOICE", local_dir=model_dir)
hf_hub_download("MahmoodLab/UNI2-h", "pytorch_model.bin", local_dir=f"{model_dir}/uni2h")
PYTHON
```

See [model files](https://github.com/zchxhcz1/VOICE/blob/main/weights/README.md) for the directory structure.

## Prediction

Choose your own model, image and output locations in the commands below. Run prediction from a single H&E image:

```bash
python predict/predict.py \
  --image /path/to/slide.svs \
  --release models/voice \
  --out results/prediction.h5ad
```

The command segments nuclei, extracts cell-centered image crops, predicts expression, retrieves reference expression, and combines both predictions. It also saves the segmentation beside the output. The model download includes the fusion weights and reference bank used by this command. The first segmentation run downloads the Cellpose model; use `--segment-model` to provide local segmentation weights.

Spatial prediction uses up to 128 target cells and 64 neighboring context cells per block, with context selected within 137 µm. All cells in the block participate in attention; only target predictions are saved. Every cell is output once. Partitioning is deterministic for a given sample ID and fixed seed.

If the image does not contain physical resolution metadata, supply its actual microns per pixel:

```bash
python predict/predict.py --image /path/to/image.png --mpp 0.25 \
  --release models/voice --out results/prediction.h5ad
```

`0.25` is an example. Use the resolution of your image. Prediction requires no measured expression or precomputed image features for the target slide.

| Option | Purpose |
| --- | --- |
| `--cells cells.npz` | Use existing cell boundaries instead of automatic segmentation |
| `--prepared /path/to/slide` | Read prepared cell crops and boundaries instead of `--image` |
| `--tissue lung/wide` | Select organ and panel-width fusion weights; `--tissue lung` selects width automatically |
| `--genes all` | Predict all 18,580 model genes; this is the default |
| `--genes genes.txt` | Predict an ordered list of gene symbols, one per line |
| `--save-embeddings` | Save image and spatial features |
| `--workers 4` | Set CPU workers for loading cell crops |
| `--enc-chunk 32 --query-batch 64` | Reduce transient GPU memory use |
| `--sample-id SAMPLE --exclude RELATED_SAMPLE` | Exclude the target and related references when they occur in the bank |

CUDA is the default device. CPU inference is available with `--device cpu --amp fp32`. Prediction matrices are written to disk in batches; output size scales with the number of cells and requested genes. Automatic segmentation produces nuclei; use supplied cell boundaries when available.

## Fusion weights

All 15 reference organs and all 20 available organ/panel groups are included. The table links to individual gene-weight tables and shows the number of genes with fitted weights.

| Organ | narrow | wide |
| --- | ---: | ---: |
| bone | [469](beta/bone/narrow.json) | — |
| brain | [465](beta/brain/narrow.json) | — |
| breast | [769](beta/breast/narrow.json) | — |
| cervix | — | [5,030](beta/cervix/wide.json) |
| colon | [822](beta/colon/narrow.json) | [18,539](beta/colon/wide.json) |
| heart | [369](beta/heart/narrow.json) | — |
| kidney | [395](beta/kidney/narrow.json) | — |
| liver | [465](beta/liver/narrow.json) | [6,107](beta/liver/wide.json) |
| lung | [973](beta/lung/narrow.json) | [4,954](beta/lung/wide.json) |
| lymph_node | — | [4,577](beta/lymph_node/wide.json) |
| ovary | [721](beta/ovary/narrow.json) | [8,512](beta/ovary/wide.json) |
| pancreas | [465](beta/pancreas/narrow.json) | — |
| prostate | — | [4,958](beta/prostate/wide.json) |
| skin | [570](beta/skin/narrow.json) | [4,959](beta/skin/wide.json) |
| tonsil | [369](beta/tonsil/narrow.json) | — |

`narrow` means fewer than 1,000 measured model genes in the fitting reference; `wide` means at least 1,000. Each gene's weight is the equal mean of its fitted weights across contributing reference slides in that organ/panel group. Single-reference groups are included: fitting retrieves from other reference slides across tissues, excluding the query reference and its related donor/block samples. [beta/manifest.json](beta/manifest.json) lists reference counts, fitted-gene counts and checksums.

Use `--tissue lung/wide` to select an explicit group. With an organ alone, such as `--tissue lung`, width is selected from the number of requested output genes. Supply an explicit group to keep that choice fixed when changing the requested gene subset. A missing width uses another available width of the same organ; an omitted or unknown organ uses global weights.

The root `beta.json` combines all groups and is loaded automatically during prediction. Individual tables under `beta/ORGAN/WIDTH.json` contain the same weights. Global weights are fitted across the full reference set independently of organ grouping. Genes without reference coverage or a fitted weight use the direct prediction. No target expression is needed.

## Output

```python
import anndata as ad
prediction = ad.read_h5ad("results/prediction.h5ad")
```

| Field | Contents |
| --- | --- |
| `X` | Combined gene-expression scores |
| `layers['A']` | Direct predictions in `log1p(mu)` scale |
| `layers['R']` | Reference expression scores; NaN where reference coverage is unavailable |
| `var['beta']` | Applied fusion weight for each gene |
| `var['covered']` | Reference coverage for each gene |
| `obsm['spatial']` | Cell centroids as x,y coordinates in source-image pixels |

The combined score is `beta * A + (1 - beta) * R`. Genes without reference coverage or fitted weights use the direct prediction. Combined scores describe variation across cells within each gene; they are not molecule counts.

## Training

[Training instructions](https://github.com/zchxhcz1/VOICE/blob/main/docs/training.md) cover RNA embedding preparation, image–RNA alignment, expression training, reference-bank construction, and fitting fusion weights. Training requires paired H&E images and measured spatial expression.

[Input formats](https://github.com/zchxhcz1/VOICE/blob/main/docs/formats.md) describe image geometry, prepared-slide files, reference-bank inputs, and output coordinates.