Calpha / README.md
liuyong9975's picture
Model card: technical report link pending
369b31d verified
|
Raw History Blame Contribute Delete
5.63 kB
---
license: other
license_name: mixed-see-model-card
license_link: https://github.com/yliust/Calpha/blob/main/NOTICE
base_model: Qwen/Qwen3.5-9B
language:
- en
tags:
- biology
- protein
- protein-structure-prediction
- protein-folding
library_name: calpha
---
# Calpha
**Toward native biomolecular intelligence in a general-purpose language model. Technical Report I: Protein Folding.**
[Homepage](https://calpha-ai.com) · [Code](https://github.com/yliust/Calpha) · Technical report (coming soon)
Calpha is a chat model that folds proteins. A protein sequence enters Qwen3.5-9B inside an
ordinary chat turn, together with per-residue features from frozen ESM2-3B. Asked for a
structure, the model answers with a `<structure>` token, and its hidden states from nine layers
at the residue positions pass through a small conditioning front-end into the frozen
SimpleFold-3B flow-matching head, which generates all-atom coordinates.
This repository holds the weights that reproduce the numbers of the report, the raw benchmark
outputs, and the training data.
## Files
| Path | Content | Size |
|---|---|---|
| `llm/` | Qwen3.5-9B after stage-1 training (bf16), tokenizer with amino-acid tokens and `<structure>` | 18 GB |
| `esm_adapter.safetensors` | adapter that adds ESM2-3B features to the amino-acid token embeddings (27M parameters) | 109 MB |
| `frontend.safetensors` | conditioning front-end of the structure head (8.4M parameters; stage-2 EMA, step 30,000) | 34 MB |
| `calpha_config.json` | layers read, canonical request, sampling settings | |
| `benchmark_outputs/` | alpha-carbon predictions of every system in the report on the three test sets, raw text of the chat models | 54 MB |
| `training_data/` | structure data and language data used for training (see its README) | 77 GB |
**Not included, downloaded automatically from their original sources:** the ESM2-3B encoder
(fair-esm) and the SimpleFold-3B head ([apple/ml-simplefold](https://github.com/apple/ml-simplefold)).
Every weight of the head other than the front-end is the public SimpleFold-3B checkpoint,
unchanged.
## Use
```bash
git clone https://github.com/yliust/Calpha.git && cd Calpha
pip install -e .
pip install git+https://github.com/apple/ml-simplefold.git@94e2cd704cf7b86ef8e7cb27a1e77154c4e35b34
calpha fold MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAVQVKVKALPDAQFEVV
```
```python
from calpha import Calpha
model = Calpha.from_pretrained("liuyong9975/Calpha")
out = model.fold("MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ", out_dir="predictions")
```
One GPU with about 48 GB of memory. `llm/` is a standard `transformers` checkpoint and can be
loaded on its own as a chat model; without the adapter and the code in the GitHub repository it
does not see ESM2 features and cannot produce structures.
## Results
TM-score on alpha carbons, one sample per target, failures counted as zero, all systems scored
by the same program.
| Model | FoldBench-334 | CASP16 | CAMEO-2025 |
|---|---|---|---|
| **Calpha** | **0.807** | **0.635** | **0.900** |
| Head-only control (ESM2 → SimpleFold-3B, no language model) | 0.822 | 0.682 | 0.917 |
| ESMFold | 0.869 | 0.762 | 0.905 |
| SimpleFold-3B | 0.842 | 0.704 | 0.938 |
| ESM3-open | 0.787 | 0.643 | 0.852 |
| DPLM-2 (650M) | 0.656 | 0.515 | 0.757 |
| Random walk (floor) | 0.140 | 0.139 | 0.136 |
Against the head-only control, no loss of accuracy is detected for chains of 100–300 residues
on any set, and the loss is significant for chains of 300–600 residues on all three.
## Training
- **Stage 1.** All of Qwen3.5-9B, the ESM2 adapter and a SimpleFold-100M head (initialised from
its public weights) trained jointly for 2B tokens: language cross-entropy, trigger
cross-entropy and the head's flow-matching loss. Language data: 75% protein tasks, 25% general
English text and reasoning. Structure data: 84,658 PDB chains cropped to 256 residues.
- **Stage 2.** Language model frozen, SimpleFold-3B frozen, only the front-end trained: 20,000
steps at a crop of 256 residues, then 10,000 steps at 512 (effective batch 64, learning rate
1e-4, EMA 0.999), about 11 hours on eight H200 GPUs.
## Limitations
- The language model does not make folding more accurate: Calpha is below the same head reading
ESM2 directly on two of three sets and below ESMFold on FoldBench-334 and CASP16.
- Above 300 residues the language model costs accuracy on every set.
- The system depends on ESM2 and on the SimpleFold weights.
- MMLU-Pro accuracy fell from 0.774 to 0.703 during protein training.
- Single chains, single sequence, no ligands or complexes. Accuracy was measured on alpha
carbons only, with one request phrasing; side chains, clashes and other phrasings were not
evaluated.
- Not for clinical or diagnostic use.
## License
- `llm/` is a fine-tuned derivative of Qwen3.5-9B and is released under the Apache License 2.0.
- `esm_adapter.safetensors`, `frontend.safetensors`, the benchmark outputs and the code are
released under the Apache License 2.0.
- Producing a structure requires the SimpleFold-3B weights, which Apple releases under the
**Apple Machine Learning Research Model License (research use only)**. Calpha as a whole can
therefore only be used under those terms.
- ESM2 is MIT-licensed. The training data has its own notes in `training_data/README.md`.
## Citation
```bibtex
@techreport{liu2026calpha,
title = {Calpha: Toward Native Biomolecular Intelligence in a General-Purpose Language Model.
Technical Report I: Protein Folding},
author = {Liu, Yong},
year = {2026},
url = {https://calpha-ai.com}
}
```