File size: 15,080 Bytes
5e449c6
 
 
 
 
 
 
 
 
 
 
 
e1f0461
 
 
 
5e449c6
e1f0461
 
5e449c6
e8c145d
e1f0461
 
5e449c6
e1f0461
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
07bb78a
e1f0461
 
 
e8c145d
e1f0461
e8c145d
 
 
e1f0461
 
 
e8c145d
 
e1f0461
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
999e5eb
5e449c6
 
e1f0461
 
999e5eb
 
e1f0461
07bb78a
e1f0461
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e8c145d
e1f0461
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5e449c6
 
 
e1f0461
 
 
 
 
 
5e449c6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
e8c145d
5e449c6
 
 
 
 
 
 
 
 
 
 
e1f0461
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
---
license: other
tags:
  - single-cell
  - transcriptomics
  - foundation-model
  - circulating-tumor-cells
library_name: pytorch
---

# BioMod

BioMod is a single-cell transcriptomics foundation model built around gene modules
and a Performer-based encoder. This repository provides pretrained model assets,
pretraining and fine-tuning code, downstream task tutorials, and interpretability
tools.

BioMod is released by **CTC-Atlas**, the joint laboratory of **Raymind Biotechnology
Co., Ltd** and the **Shenzhen Research Institute of Big Data (SRIBD)**.

[Installation](#installation) · [BioMod model weights](#biomod-model-weights) ·
[Tasks and examples](#tasks-and-examples) · [Usage notes](#usage-notes) ·
[Repository layout](#repository-layout)

## Installation

Download the project from [Hugging Face](https://huggingface.co/ctcatlas/BioMod).
Run the following commands from the parent of your intended project directory:

```bash
python -m pip install --upgrade huggingface_hub
hf auth login
hf download ctcatlas/BioMod --local-dir ./BioMod
cd BioMod
```

The download includes source code, model dependencies, the pretrained checkpoint,
and datasets. To omit datasets, add `--exclude "data/**"` to the download command.
If you already have the project, start from its root directory.

Create and activate the supplied Conda environment:

```bash
conda env create -f BioMod/environment.yml
conda activate performer_module
```

The environment specification uses Python 3.9, PyTorch 2.6, CUDA 12.4, and PyTorch
Geometric 2.6. See [environment.yml](BioMod/environment.yml) and
[requirements.txt](BioMod/requirements.txt) for the complete dependency lists.
Match the GPU packages to your system when adapting the environment.

Pretraining uses CUDA/NCCL. The zero-shot
runner selects CUDA when available and otherwise uses CPU. Native Windows
execution has not been validated for the supplied environment.

## BioMod model weights

This release includes the **BioMod model weights** and their matching
configuration. Use these files together; the configuration records the model
architecture and gene-module definitions.

| Asset | File |
| --- | --- |
| BioMod model weights | [mae_tost_70M_75mask_ec2_epoch0.pth](weight/0810model/mae_tost_70M_75mask_ec2_epoch0.pth) |
| Matching model configuration | [mae_tost_70M_75mask_ec2.pkl](weight/0810model/mae_tost_70M_75mask_ec2.pkl) |
| Model gene order | [selected_genes_27k.txt](model_dependencies/selected_genes_27k.txt) |
| Gene identifier mapping | [gene2ens.csv](model_dependencies/gene2ens.csv) |
| Gene2Vec embeddings | [selected_gene2vec_27k.npy](model_dependencies/selected_gene2vec_27k.npy) |
| Shared module definitions | [merged_modules.json](model_dependencies/module_data_all/merged_modules.json) |

The model components are bundled under
[`model_dependencies/performer_pytorch_cont/`](model_dependencies/performer_pytorch_cont/).
Task-specific checkpoint arguments are listed under
[Usage notes](#checkpoint-arguments).

## Tasks and examples

The following tasks have implementations in this repository. The links identify
available source code and examples; these workflows have not been validated end
to end with the bundled checkpoint.

| Task | Implementation | Notebook |
| --- | --- | --- |
| Cell-type annotation | [Fine-tuning script](BioMod/cell_annotation/performer_module_finetune.py) | [Notebook](BioMod/toturials/cell_annotation.ipynb) |
| Zero-shot cell embeddings and batch-effect evaluation | [Embedding and evaluation script](BioMod/zero_shot_batch/performer_module_zero_shot.py) | [Notebook](BioMod/toturials/zero_shot_batch_integration.ipynb) |
| Single-drug response regression (IC50) | [Training script](BioMod/drug_response/train_drug_performer_module.py) | [Notebook](BioMod/toturials/drug_response_prediction.ipynb) |
| Drug-pair synergy classification | [Training script](BioMod/drug_combination/train_drug_combination_performer_module.py) | [Notebook](BioMod/toturials/drug_combination.ipynb) |
| Gene perturbation response prediction | [Training script](BioMod/gene_perturbation/train_module.py) | [Notebook](BioMod/toturials/gene_perturbation.ipynb) |

### Pretraining and analysis scripts

The repository also includes the following supporting workflows.

| Workflow | Implementation | Documentation |
| --- | --- | --- |
| Pretraining and continued pretraining | [Pretraining script](BioMod/performer_module.py) | [Source guide](BioMod/README.md), [launcher reference](BioMod/command.txt) |
| Module interpretability analysis | [Module geometry](BioMod/interpretability/module_geometry.py), [module importance](BioMod/interpretability/module_importance/module_importance.py), [text alignment](BioMod/interpretability/module_text_alignment/module_text_alignment.py) | [Overview](BioMod/interpretability/README.md), [attention](BioMod/interpretability/attention/README.md), [module importance guide](BioMod/interpretability/module_importance/README.md) |

Tutorials and launcher references retain some historical checkpoint paths,
server paths, and arguments. Adapt them to the current assets above and check
the selected script's `--help` before running. Some notebook plotting steps
also refer to external scripts or reference results that are not bundled here.

## Usage notes

### Project paths and notebooks

The project root contains `BioMod/`, `model_dependencies/`, `weight/`, and
`data/`. The inner `BioMod/` directory contains the source code. Task commands
may assume different working directories; adjust relative paths when following
a guide.

Before running a notebook, update its configuration cells:

| Setting | Location |
| --- | --- |
| `BASE` or `ASSET_ROOT` | Project root |
| `PROJECT_ROOT` | Inner `BioMod/` source directory |
| `MODULE_MODEL` or `LEGACY_MODULE_MODEL` | `model_dependencies/` under the project root |
| Weight and configuration paths | The BioMod model weights and matching configuration listed above |
| `MODULE_DATA_DIR` | `BioMod/module_data_all/` under the project root |
| Data and output paths | Your task inputs and a writable output directory |

Checkpoint variable names differ by notebook: `CKPT_PATH` / `CONFIG_PKL`
for cell annotation, drug response, and zero-shot integration;
`SCFM_CKPT` / `SCFM_HYPER` for gene perturbation; and
`CHECKPOINT_PATH` / `HYPER_PARAMS_PATH` for drug combination.
Updating environment variables alone does not replace hard-coded notebook paths.

### Checkpoint arguments

| Task | Weights argument | Configuration argument |
| --- | --- | --- |
| Drug response / drug combination | `--checkpoint_path` | `--hyper_params_path` |
| Zero-shot / gene perturbation | `--scfm_ckpt_path` | `--scfm_hyper_params_path` |
| Cell annotation / backbone interpretation | `--performer_checkpoint` | `--config_pkl` |
| Continued pretraining | `--model_path` | `--model_config_path` or paired-file discovery |

### Data and evaluation

Datasets are stored under `data/`. Input formats, metadata fields, preprocessing,
and train/validation splits depend on the task; follow the linked implementation
and tutorial.

For zero-shot embeddings, provide an h5ad expression matrix and specify the gene,
batch, and cell-type metadata fields. The runner aligns genes to the model gene
list and fills missing genes with zeros. It does not normalize or log-transform
the input matrix, so prepare expression values according to the intended protocol.
Resolve duplicate gene identifiers and inspect gene overlap before evaluating
embeddings.

The drug-response and drug-combination workflows use validation metrics for
checkpoint selection. Their validation results should not be presented as
performance on an independent final test set.

For checks of the shared training utilities, see
[Training utility checks](BioMod/README.md#training-utility-checks).

## Repository layout

The tree below includes source files, model dependencies, and the pretrained
checkpoint. The top-level `data/` and `.cache/` directories are omitted for
readability. Folder descriptions explain where to find each component.

<details>
<summary>Expand the full repository structure</summary>

```text
BioMod/                                                   # Project root: source code, model dependencies and weights
├── BioMod/                                               # Task code: training, evaluation, interpretability and tutorials
│   ├── cell_annotation/                                  # Cell-type annotation and model fine-tuning
│   │   ├── module_data/                                  # Hallmark and Reactome modules with loading utilities
│   │   │   ├── load_module.py
│   │   │   ├── module1_hallmarks.json
│   │   │   └── module2_reactome_2022_filtered.json
│   │   ├── cmd_finetune.md
│   │   └── performer_module_finetune.py
│   ├── drug_combination/                                 # Drug-combination synergy prediction
│   │   ├── README.md
│   │   └── train_drug_combination_performer_module.py
│   ├── drug_response/                                    # Single-drug response prediction and training
│   │   ├── cmd.txt
│   │   └── train_drug_performer_module.py
│   ├── gene_perturbation/                                # Perturbation response prediction: processing, training and inference
│   │   ├── data_utils_local.py
│   │   ├── gears_reduce_local.py
│   │   ├── inference_local.py
│   │   ├── model_module.py
│   │   ├── pertdata_local.py
│   │   ├── README.md
│   │   ├── scfm_utils_local.py
│   │   ├── train_module.py
│   │   └── utils.py
│   ├── interpretability/                                 # Module representations and biological interpretability
│   │   ├── attention/                                    # Module crosstalk and within-module gene attention
│   │   │   ├── __init__.py
│   │   │   ├── module_crosstalk.py
│   │   │   ├── module_gene_attention.py
│   │   │   ├── README.md
│   │   │   └── README_gene_attention.md
│   │   ├── module_importance/                            # Module importance, ablation studies and marker-gene validation
│   │   │   ├── __init__.py
│   │   │   ├── cross_class_similarity.py
│   │   │   ├── family_ablation.py
│   │   │   ├── family_importance_analysis.py
│   │   │   ├── marker_gene_verification.py
│   │   │   ├── masking_ablation.py
│   │   │   ├── module_importance.py
│   │   │   ├── module_importance_publication.py
│   │   │   ├── pub_utils.py
│   │   │   ├── random_baseline.py
│   │   │   ├── README.md
│   │   │   ├── RESULTS.md
│   │   │   ├── tcell_reactome_analysis.py
│   │   │   └── TODO.md
│   │   ├── module_text_alignment/                        # Semantic alignment of module representations and pathway text
│   │   │   ├── __init__.py
│   │   │   ├── DESIGN.md
│   │   │   ├── fetch_reactome_descriptions.py
│   │   │   ├── MANUSCRIPT_SECTION.md
│   │   │   ├── module_text_alignment.py
│   │   │   ├── module_text_alignment_extended.py
│   │   │   ├── module_text_alignment_publication.py
│   │   │   └── reactome_descriptions.json
│   │   ├── __init__.py
│   │   ├── module_geometry.py
│   │   ├── README.md
│   │   └── utils.py
│   ├── module_data_all/                                  # Module definitions, merging and uncovered-gene tools
│   │   ├── create_uncovered_modules.py
│   │   ├── merge_modules.py
│   │   └── merged_modules.json
│   ├── tests/                                            # CPU tests for shared training utilities
│   │   └── test_training_utils.py
│   ├── toturials/                                        # Jupyter tutorials for five downstream tasks
│   │   ├── cell_annotation.ipynb
│   │   ├── drug_combination.ipynb
│   │   ├── drug_response_prediction.ipynb
│   │   ├── gene_perturbation.ipynb
│   │   └── zero_shot_batch_integration.ipynb
│   ├── zero_shot_batch/                                  # Zero-shot batch integration, evaluation and visualization
│   │   ├── sequencing_depth/                             # Low-depth data generation and zero-shot evaluation
│   │   │   ├── cmd.txt
│   │   │   ├── make_low_depth_h5ad.py
│   │   │   └── run_zero_shot_low_depth.py
│   │   ├── performer_module_zero_shot.py
│   │   ├── README.md
│   │   └── visualize_results.py
│   ├── .env.example
│   ├── .gitattributes
│   ├── .gitignore
│   ├── command.txt
│   ├── environment.yml
│   ├── models_performer_module.py
│   ├── module_dataset.py
│   ├── performer_module.py
│   ├── README.md
│   ├── requirements.txt
│   └── training_utils.py
├── model_dependencies/                                   # Shared gene mappings, gene lists, embeddings and model components
│   ├── module_data_all/                                  # Shared module definitions and construction tools
│   │   ├── create_uncovered_modules.py
│   │   ├── merge_modules.py
│   │   └── merged_modules.json
│   ├── performer_pytorch_cont/                           # Performer attention and model components
│   │   ├── performer_pytorch_cont/                       # Nested package containing copies of the model component source files
│   │   │   ├── __init__.py
│   │   │   ├── ding_models.py
│   │   │   ├── performer_pytorch.py
│   │   │   ├── performer_pytorch_old.py
│   │   │   └── reversible.py
│   │   ├── __init__.py
│   │   ├── ding_models.py
│   │   ├── performer_pytorch.py
│   │   ├── performer_pytorch_old.py
│   │   └── reversible.py
│   ├── gene2ens.csv
│   ├── selected_gene2vec_27k.npy
│   └── selected_genes_27k.txt
├── weight/                                               # Pretrained weights and matching model configurations
│   └── 0810model/                                        # BioMod model weights and matching configuration
│       ├── mae_tost_70M_75mask_ec2.pkl
│       └── mae_tost_70M_75mask_ec2_epoch0.pth
├── .gitattributes
├── .gitignore
└── README.md
```

</details>

## Partners

- [Raymind Biotechnology Co., Ltd](https://en.raymind.com/)
- [Shenzhen Research Institute of Big Data (SRIBD)](https://www.sribd.cn/en)