takubo-lab/stemcell-geneformer

Fine-Tuned Geneformer Models for Murine HSPC Aging

This repository contains four fine-tuned checkpoints built on top of the original Geneformer foundation models and used in the manuscript "Geneformer-guided multi-omics integration identifies Pbx1 as a network hub of hematopoietic stem cell aging" by Hiroshi Kobayashi, Shintaro Watanuki, Yusuke Shiozawa, Motohiko Oshima, Shuhei Koide, Naoya Takayama, Takayuki Morikawa, Miho Haraguchi, Shinpei Tamaki, Takayoshi Asakura, Toshio Miyata, Atsushi Iwama, Seishi Ogawa, and Keiyo Takubo.

These are not original Geneformer pretraining checkpoints. They are task-specific fine-tuned derivatives of Geneformer for murine hematopoietic stem and progenitor cell (HSPC) classification and downstream analyses such as embedding comparison, in silico perturbation, and attention-based network interpretation. In the study, these fine-tuned models were used to distinguish young and old HSC-associated programs and to prioritize regulators such as Pbx1.

Credit to the Original Geneformer Model

The original Geneformer foundation model, pretraining strategy, tokenizer design, and broader modeling framework were developed by Christina V. Theodoris and colleagues. Full credit for the base Geneformer model belongs to the original authors.

This repository provides only our fine-tuned downstream checkpoints built from publicly available Geneformer base models:

• GF-6L-30M-i2048
• GF-12L-95M-i4096

For the original model and documentation, please refer to the Geneformer resources from Theodoris CV et al.

Model Overview

The repository includes one intermediate young bone marrow classifier and three manuscript fine-tuned models for age-aware HSPC classification.

Subfolder	Base checkpoint	Architecture	Max input genes	Labels	Role in study
Young_BM_finetuned_model	GF-6L-30M-i2048	BertForSequenceClassification	2048	13	Intermediate young bone marrow classifier used as the first stage of the 2-step workflow
6-layer-2-step_model	GF-6L-30M-i2048	BertForSequenceClassification	2048	10	Stepwise age-aware classifier obtained by additional fine-tuning of the young bone marrow model
6-layer-1-step_model	GF-6L-30M-i2048	BertForSequenceClassification	2048	10	Main manuscript model selected for downstream analyses because it balanced performance and computational cost
12-layer-1-step_model	GF-12L-95M-i4096	BertForSequenceClassification	4096	10	Larger single-step model with comparable classification performance and higher input capacity

Biological Scope

The models were trained to capture transcriptional differences among immature murine bone marrow and HSPC populations, with particular emphasis on aging-associated states. The manuscript shows that old HSCs carry two concurrent transcriptional programs: a primitive HSC-like program and a megakaryocyte-biased program. The fine-tuned Geneformer models were then used to identify regulatory hubs associated with these programs.

The checkpoints are intended for:

• classification of tokenized murine HSPC single-cell transcriptomes
• comparison of young and old HSC-related states
• reproduction or extension of the manuscript's Geneformer-based analyses
• hypothesis generation through Geneformer-compatible in silico perturbation workflows

They are not intended for:

• direct use on raw count matrices without Geneformer preprocessing
• human data without careful remapping and revalidation
• clinical or diagnostic use
• standalone causal inference without orthogonal validation

Training Data and Fine-Tuning Strategy

According to the manuscript, single-cell RNA-seq datasets from murine immature bone marrow and HSPC populations were integrated before transfer learning.

• 1-step 6-layer model: trained on 10 datasets totaling 160,005 cells with an 80% train and 20% evaluation split for 10-class prediction.
• 2-step 6-layer model: first trained on 4 young-only datasets totaling 112,124 cells for 13-class immature bone marrow classification, then further fine-tuned on 6 datasets containing both young and old cells totaling 40,382 cells for 10-class age-aware prediction.
• 1-step 12-layer model: trained on the same 10 datasets totaling 160,005 cells with a 70% train, 15% validation, and 15% test split for 10-class prediction.

Mouse gene identifiers were converted to human Ensembl identifiers before Geneformer tokenization, following the original Geneformer workflow used in the manuscript. Input data were converted to loom format and tokenized with the Geneformer TranscriptomeTokenizer.

All classifiers use the BertForSequenceClassification architecture with AdamW optimization. In the manuscript, the 6-layer 1-step model and the 12-layer 1-step model showed comparable performance for distinguishing young and old HSC states, whereas the 2-step model was less effective for the age-aware classification task. The 6-layer 1-step model was therefore chosen for downstream analyses.

Label Definitions

10-class age-aware models

The following label set applies to 6-layer-2-step_model, 6-layer-1-step_model, and 12-layer-1-step_model:

1. young_HSC
2. old_HSC
3. young_ST_HSC
4. old_ST_HSC
5. young_MPP
6. old_MPP
7. young_MkP
8. old_MkP
9. young_Other
10. old_Other

The exact integer-to-label order differs slightly between checkpoint folders, so users should always read config.json or id2label from the loaded model rather than assuming a shared numeric order.

13-class young bone marrow model

The Young_BM_finetuned_model corresponds to the first-stage immature bone marrow classifier described in the manuscript. Its 13 target classes are:

1. HSC
2. ST_HSC
3. MPP
4. MEP
5. MkP
6. Erythroid
7. CLP
8. B-Cell
9. GMP
10. Monocyte
11. DC
12. Neutrophil
13. Mast

Files in This Repository

• Young_BM_finetuned_model/: first-stage 13-class checkpoint stored with pytorch_model.bin
• 6-layer-2-step_model/: 10-class stepwise fine-tuned checkpoint stored with model.safetensors
• 6-layer-1-step_model/: 10-class single-step fine-tuned checkpoint stored with model.safetensors
• 12-layer-1-step_model/: 10-class larger single-step fine-tuned checkpoint stored with model.safetensors

Tokenizer files are not bundled here. These checkpoints are expected to be used with the Geneformer preprocessing pipeline.

How to Use

These checkpoints are compatible with the Hugging Face Transformers model loading interface, but the inputs are Geneformer token IDs derived from scRNA-seq data, not natural-language text tokens. They should be used as fine-tuned classifier heads on top of the Geneformer modeling framework rather than as replacements for the original pretrained Geneformer release.

import torch
from transformers import AutoConfig, AutoModelForSequenceClassification

model_id = "YOUR_HF_NAMESPACE/YOUR_REPOSITORY_NAME"
subfolder = "6-layer-1-step_model"

config = AutoConfig.from_pretrained(model_id, subfolder=subfolder)
model = AutoModelForSequenceClassification.from_pretrained(model_id, subfolder=subfolder)

# Example only: replace with Geneformer-tokenized gene IDs.
input_ids = torch.tensor([[1, 25, 316, 982, 1402]], dtype=torch.long)
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    outputs = model(input_ids=input_ids, attention_mask=attention_mask)

predicted_index = int(outputs.logits.argmax(dim=-1))
predicted_label = config.id2label[predicted_index]
print(predicted_label)

For actual use, preprocess data with the Geneformer pipeline:

1. start from single-cell RNA-seq count data
2. convert genes to the identifier space expected by Geneformer
3. tokenize cells with the Geneformer tokenizer
4. feed token IDs into one of the checkpoints in this repository

If you want to reproduce the manuscript analyses most closely, start with 6-layer-1-step_model.

Recommended Checkpoint Selection

• Use 6-layer-1-step_model for most reproductions of the manuscript.
• Use 12-layer-1-step_model if you want a larger model with longer supported input length.
• Use 6-layer-2-step_model if you specifically want the stepwise training strategy described in the manuscript.
• Use Young_BM_finetuned_model only if you need the intermediate 13-class young bone marrow classifier.

Limitations

• The models were trained on murine hematopoietic datasets and may not generalize to other tissues, species, or sequencing protocols without adaptation.
• Gene-to-human-Ensembl conversion and Geneformer tokenization are part of the expected input pipeline; skipping these steps will invalidate predictions.
• The Other classes aggregate multiple cell types and therefore provide coarser biological resolution than the named HSC, ST-HSC, MPP, and MkP classes.
• Model outputs are useful for prioritization and exploratory analysis, but the manuscript validates key findings experimentally and that level of validation remains necessary for new biological claims.

Manuscript Context

In the associated study, the fine-tuned models were integrated with transcriptomic profiling, ATAC-seq, transcription factor screening, and downstream Geneformer perturbation analyses. This framework identified Pbx1 as a regulator linked to age-associated HSC transcriptional features, delayed differentiation after cytokine stimulation, and reduced erythroid output through repression of Gata1.

Citation

If you use these checkpoints, please cite both the manuscript describing this fine-tuning study and the original Geneformer publication by Theodoris CV and colleagues.

@article{kobayashi_geneformer_guided_hsc_aging,
  title = {Geneformer-guided multi-omics integration identifies Pbx1 as a network hub of hematopoietic stem cell aging},
  author = {Kobayashi, Hiroshi and Watanuki, Shintaro and Shiozawa, Yusuke and Oshima, Motohiko and Koide, Shuhei and Takayama, Naoya and Morikawa, Takayuki and Haraguchi, Miho and Tamaki, Shinpei and Asakura, Takayoshi and Miyata, Toshio and Iwama, Atsushi and Ogawa, Seishi and Takubo, Keiyo},
  year = {2026},
  note = {Submitted manuscript}
}

@article{theodoris2024geneformer,
  title = {Geneformer: A foundation model pretrained on single-cell transcriptomes},
  author = {Theodoris, Christina V. and Xiao, Lu and Chopra, Anushka and Chaffin, Mark D. and Al Sayed, Zeina R. and Hill, Michael C. and Mantineo, Heather and Brydon, Eliza M. and Zeng, Zexian and Liu, Xin and others},
  journal = {Nature Methods},
  year = {2024}
}

License

No explicit license file is included in this repository snapshot. Please add the appropriate license metadata before public release on Hugging Face.