--- license: mit library_name: pytorch pipeline_tag: feature-extraction tags: - single-cell - genomics - rna-seq - jepa - foundation-model --- # scRM S-96 scRM is a single-cell foundation model trained directly on sequencing reads. Its input is raw per-(cell barcode, UMI) sequence: there is no aligner, no reference genome, no gene annotation and no count matrix anywhere on the input side. A cell embedding is read out through a hierarchy of attention pools over bytes, contigs, UMIs and windows. The model holds 1,182,720 parameters. Preprint: [He et al., *scRM: A single-cell foundation model trained on scRNA-seq reads*, bioRxiv 2026](https://www.biorxiv.org/content/10.64898/2026.10.24.761590v1). Code: [github.com/altoslabs/scRM](https://github.com/altoslabs/scRM). ## Checkpoints | name | trained on | steps | parameters | | --- | --- | --- | --- | | `scRM-S96` | 3,000 cells from four public mouse 10x 3′ datasets | 8,947 | 1,182,720 | | `scRM-S96-sc_mixology` | `sc_mixology`'s own cells at the same 3,000-cell cap | 8,947 | 1,182,720 | Both are the same architecture at the same recipe and differ only in what they read. `scRM-S96` is the reported model, benchmarked zero-shot. `scRM-S96-sc_mixology` is the dataset-specific arm, used only for the gene-detection comparison on `sc_mixology`. ## Files Four files per checkpoint: - `.safetensors` — 103 weight tensors. - `.pt` — the same weights as a torch `state_dict`. - `.eval.pt` — that `state_dict` together with the `arch`, `readout` and `arch_sha` fields the evaluation loader requires. This is the file to load. - `.config.json` — the `arch` and `readout` dicts, the step, the seed and the parameter count. ## Loading `embed()` takes no architecture arguments; the architecture comes from the checkpoint, and `Checkpoint.load` recomputes `arch_sha` over it and refuses a checkpoint whose weights and declared shape disagree. ```python from scrm_eval.embed import Checkpoint from scrm_model.evalio import model_factory ckpt = Checkpoint.load("scRM-S96.eval.pt", model_factory=model_factory) ``` Run with `PYTHONPATH=model:eval:target/py` from a checkout of the code repository. ## Architecture A local trunk of 4 layers and a global trunk of 6 layers, both at width 96 with 4 heads, rotary position embeddings at `theta = 10000` over a maximum of 1,600 positions, and a projector of hidden width 256 onto 96 dimensions. The readout pools bytes into contigs, contigs into UMIs, UMIs into windows, and two views into a cell, each pool a masked seed-attention pool. Both checkpoints carry `arch_sha = 7c4e4ffc9d304902eaaaeacc4ebac858edb48d5f5078d7f7245d71e7f7359bba`, which gates evaluation compatibility. ## Training Trained as a JEPA. The pretraining corpora are four public mouse 10x 3′ datasets — `mouse_brain_5k`, `neuron_1k_v3`, `neurons_5k` and `sod1_spinal_cord` — all downloadable without registration and released under CC BY 4.0. The reported model draws its 8,947 steps from 3,000 of those cells. Seed 20260903, torch 2.11.0+cu128. ## Scope of the evidence The reported numbers are zero-shot on held-out human and fly data; none of the four pretraining datasets appears in any evaluation. A read-level representation retains nucleotide content and can therefore carry germline variation, so these embeddings are better governed as genotype data than as expression matrices. ## Citation ```bibtex @article{he2026scrm, title = {scRM: A single-cell foundation model trained on scRNA-seq reads}, author = {He, D. and Patro, R. and Nassar, M. and Wu, H. and Yan, Q. and Xu, G. and Torkenczy, K. and Liu, J. and Tyanova, S.}, journal = {bioRxiv}, year = {2026}, doi = {10.64898/2026.10.24.761590}, url = {https://www.biorxiv.org/content/10.64898/2026.10.24.761590v1} } ```