scRM / README.md
mark-christiansen's picture
scRM weights
057d285 verified
|
Raw History Blame Contribute Delete
3.86 kB
---
license: mit
library_name: pytorch
pipeline_tag: feature-extraction
tags:
- single-cell
- genomics
- rna-seq
- jepa
- foundation-model
---
# scRM S-96
scRM is a single-cell foundation model trained directly on sequencing reads. Its input is raw
per-(cell barcode, UMI) sequence: there is no aligner, no reference genome, no gene annotation and no
count matrix anywhere on the input side. A cell embedding is read out through a hierarchy of
attention pools over bytes, contigs, UMIs and windows. The model holds 1,182,720 parameters.
Preprint: [He et al., *scRM: A single-cell foundation model trained on scRNA-seq reads*, bioRxiv
2026](https://www.biorxiv.org/content/10.64898/2026.10.24.761590v1).
Code: [github.com/altoslabs/scRM](https://github.com/altoslabs/scRM).
## Checkpoints
| name | trained on | steps | parameters |
| --- | --- | --- | --- |
| `scRM-S96` | 3,000 cells from four public mouse 10x 3′ datasets | 8,947 | 1,182,720 |
| `scRM-S96-sc_mixology` | `sc_mixology`'s own cells at the same 3,000-cell cap | 8,947 | 1,182,720 |
Both are the same architecture at the same recipe and differ only in what they read. `scRM-S96` is
the reported model, benchmarked zero-shot. `scRM-S96-sc_mixology` is the dataset-specific arm, used
only for the gene-detection comparison on `sc_mixology`.
## Files
Four files per checkpoint:
- `<name>.safetensors` — 103 weight tensors.
- `<name>.pt` — the same weights as a torch `state_dict`.
- `<name>.eval.pt` — that `state_dict` together with the `arch`, `readout` and `arch_sha` fields the
evaluation loader requires. This is the file to load.
- `<name>.config.json` — the `arch` and `readout` dicts, the step, the seed and the parameter count.
## Loading
`embed()` takes no architecture arguments; the architecture comes from the checkpoint, and
`Checkpoint.load` recomputes `arch_sha` over it and refuses a checkpoint whose weights and declared
shape disagree.
```python
from scrm_eval.embed import Checkpoint
from scrm_model.evalio import model_factory
ckpt = Checkpoint.load("scRM-S96.eval.pt", model_factory=model_factory)
```
Run with `PYTHONPATH=model:eval:target/py` from a checkout of the code repository.
## Architecture
A local trunk of 4 layers and a global trunk of 6 layers, both at width 96 with 4 heads, rotary
position embeddings at `theta = 10000` over a maximum of 1,600 positions, and a projector of hidden
width 256 onto 96 dimensions. The readout pools bytes into contigs, contigs into UMIs, UMIs into
windows, and two views into a cell, each pool a masked seed-attention pool. Both checkpoints carry
`arch_sha = 7c4e4ffc9d304902eaaaeacc4ebac858edb48d5f5078d7f7245d71e7f7359bba`, which gates
evaluation compatibility.
## Training
Trained as a JEPA. The pretraining corpora are four public mouse 10x 3′ datasets — `mouse_brain_5k`,
`neuron_1k_v3`, `neurons_5k` and `sod1_spinal_cord` — all downloadable without registration and
released under CC BY 4.0. The reported model draws its 8,947 steps from 3,000 of those cells. Seed
20260903, torch 2.11.0+cu128.
## Scope of the evidence
The reported numbers are zero-shot on held-out human and fly data; none of the four pretraining
datasets appears in any evaluation. A read-level representation retains nucleotide content and can
therefore carry germline variation, so these embeddings are better governed as genotype data than as
expression matrices.
## Citation
```bibtex
@article{he2026scrm,
title = {scRM: A single-cell foundation model trained on scRNA-seq reads},
author = {He, D. and Patro, R. and Nassar, M. and Wu, H. and Yan, Q. and
Xu, G. and Torkenczy, K. and Liu, J. and Tyanova, S.},
journal = {bioRxiv},
year = {2026},
doi = {10.64898/2026.10.24.761590},
url = {https://www.biorxiv.org/content/10.64898/2026.10.24.761590v1}
}
```