MethylLlama
Weights of MethylLlama, a Transformer encoder for DNA methylation array profiles pretrained with the Whole-CpG Encoder–Decoder (WCED) objective, and of the five chronological-age models fine-tuned from it. Code, tokenizer and instructions: github.com/netanelazran11/MethylLlama.
Files
| File | Content | sha256 (prefix) |
|---|---|---|
pretrained/methyllama_pretrained_epoch85.ckpt |
WCED-pretrained encoder, epoch 85 (lowest validation reconstruction loss) | c9e6202d3358b6cc |
finetuned_age/fold0.ckpt … fold4.ckpt |
age models of the five cross-validation folds | see SHA256SUMS |
resources/cpg_genomic_rank.npy |
genomic-rank position ids, 49,156-CpG pretraining panel | 082fe5a3141a14e8 |
resources/cpg_genomic_rank_finetune.npy |
genomic-rank position ids, 21,368-CpG downstream panel | fe0b03ba4f67cf6e |
tokenizer/cpg_sites/ |
the 49,161-entry CpG vocabulary | |
SHA256SUMS |
full checksums of the files above | |
ORIGINAL_SHA256SUMS |
checksums of the checkpoints as written by the training runs |
The checkpoints contain the weights and hyper-parameters of the training runs' checkpoint files
(PyTorch Lightning format); optimizer, scheduler and callback states were removed. Every tensor is
identical to the original file, and the models loaded from both are identical (checksums of the
originals in ORIGINAL_SHA256SUMS). The files hold Python objects besides tensors, so they are
loaded with weights_only=False; the package does this on import.
Model
6-layer pre-norm Transformer encoder (hidden size 256, 4 heads, SwiGLU 512, RMSNorm), rotary position embeddings driven by each CpG's genomic rank, 16.6 M parameters. Sample representation: pooled CLS output. Age head: 256 → 128 → 64 → 1.
Use
git clone https://github.com/netanelazran11/MethylLlama && cd MethylLlama
pip install -e ".[tutorials]"
bash scripts/download_checkpoint.sh # writes checkpoints/ in this layout
See the repository README and tutorials/ for encoding profiles and predicting age. Inputs must be
passed with genomic-rank position ids from the resources/ file that matches their CpG panel.
Citation
To be added with the preprint.