README.md

---
license: mit
tags:
  - rna-seq
  - bulk-rna
  - cancer
  - transcriptomics
  - graph-neural-network
  - transformer
  - performer
  - gcn
  - pytorch
model_size: 48M
pipeline_tag: feature-extraction
library_name: pytorch
---

# 🧬 CancerTranscriptome-Mini-48M  
*A compact, cancer-focused BulkFormer-style encoder for bulk RNA-seq*

**CancerTranscriptome-Mini-48M** is a lightweight derivative of **BulkFormer**, designed to learn cancer-specific transcriptomic structure from large-scale bulk RNA-seq.  
It combines **GCN-based gene graph propagation**, **Rotary Expression Embeddings (REE)**, **local bin-wise Performer attention**, and **global Performer attention** into a single unified encoder.

This model is a proof-of-concept designed for research, experimentation, and rapid iteration on BulkFormer-style architectures applied to cancer transcriptomes.

---

## 🔬 Origin & References

### **Primary Reference (BulkFormer)**
Boming Kang, Rui Fan, Meizheng Yi, Chunmei Cui, Qinghua Cui.  
**“A large-scale foundation model for bulk transcriptomes.”**  
bioRxiv (2025).  
doi: https://doi.org/10.1101/2025.06.11.659222

### **This Model (CancerTranscriptome-Mini-48M)**
A compact re-implementation based on the BulkFormer architecture, adapted for cancer-only bulk RNA-seq and simplified for accessibility and compute efficiency.  
Source Code: https://github.com/alwalt/BioFM

---

# 📊 Data Source

All training samples originate from the **ARCHS4 Human RNA-seq v2.5** public repository:

**ARCHS4 Reference:**  
Lachmann A., Torre D., Keenan A.B., Jagodnik K.M., et al.  
**“Massive mining of publicly available RNA-seq data from human and mouse.”**  
*Nature Communications* 9, 1366 (2018).  
Dataset: https://maayanlab.cloud/archs4/

### **Filtering Procedure**
- Loaded all human bulk RNA-seq metadata from ARCHS4 v2.5 HDF5  
- Selected samples matching:  
  `cancer | tumor | carcinoma | leukemia | lymphoma | melanoma | glioma`  
- Removed samples lacking clear disease annotations  
- Used ARCHS4 log-TPM matrices (gene × sample)  
- Final dataset: ~76k cancer samples, 19,357 genes

No private, clinical, controlled-access, or proprietary data were used.

---

# 🧠 Model Architecture (Summary)

CancerTranscriptome-Mini-48M includes:

### **1. Gene Identity Embeddings**
- Precomputed **ESM2 embeddings** for each protein-coding gene  
- Projected into model dimension (320)

### **2. Rotary Expression Embeddings (REE)**
- Deterministic sinusoidal continuous-value embedding  
- Masked positions zeroed (mask token = –10)

### **3. Graph Neural Network Layer**
- **GCNConv** (Kipf & Welling) applied on a curated gene-gene graph  
- Injects biological prior knowledge

### **4. Expression Binning**
- Learnable importance scores sort genes  
- Genes divided into 10 bins  
- Each bin receives its own **local Performer** attention

### **5. Global Performer Attention**
- 2 stacked Performer layers across all genes

### **6. Prediction Head**
- MLP → scalar value per gene  
- Used for masked-expression reconstruction

Total parameters: **48,336,162 (~48M)**

---

# 🎯 Intended Use

This model produces **context-aware gene embeddings** for downstream cancer transcriptomic tasks:

- Tumor subtype prediction  
- Drug response modeling  
- Immune infiltration scoring  
- Survival / risk modeling  
- Gene expression imputation  
- Dimensionality reduction  
- Transfer learning to TCGA, CCLE, DepMap, GEO tumor datasets  

---

# 🚀 How to Use

Download & run:

```python
import torch
from model import BulkFormer   # from this repo
import safetensors.torch as st

# Load model + weights
model = BulkFormer(
    dim=320,
    graph=torch.load("edge_index.pt"),   # provide your graph
    gene_emb=torch.load("esm2_gene_emb.pt"),
    gene_length=19357,
    bin_head=8,
    full_head=4,
    bins=10,
    gb_repeat=1,
    p_repeat=2
)

state = st.load_file("model.safetensors")
model.load_state_dict(state)
model.eval()

# Example input: 19,357-gene log-TPM vector
x = torch.randn(1, 19357)

with torch.no_grad():
    out = model(x)

print(out.shape)  # [1, 19357]