File size: 5,627 Bytes
c5368d8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
369b31d
c5368d8
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
---
license: other
license_name: mixed-see-model-card
license_link: https://github.com/yliust/Calpha/blob/main/NOTICE
base_model: Qwen/Qwen3.5-9B
language:
- en
tags:
- biology
- protein
- protein-structure-prediction
- protein-folding
library_name: calpha
---

# Calpha

**Toward native biomolecular intelligence in a general-purpose language model. Technical Report I: Protein Folding.**

[Homepage](https://calpha-ai.com) · [Code](https://github.com/yliust/Calpha) · Technical report (coming soon)

Calpha is a chat model that folds proteins. A protein sequence enters Qwen3.5-9B inside an
ordinary chat turn, together with per-residue features from frozen ESM2-3B. Asked for a
structure, the model answers with a `<structure>` token, and its hidden states from nine layers
at the residue positions pass through a small conditioning front-end into the frozen
SimpleFold-3B flow-matching head, which generates all-atom coordinates.

This repository holds the weights that reproduce the numbers of the report, the raw benchmark
outputs, and the training data.

## Files

| Path | Content | Size |
|---|---|---|
| `llm/` | Qwen3.5-9B after stage-1 training (bf16), tokenizer with amino-acid tokens and `<structure>` | 18 GB |
| `esm_adapter.safetensors` | adapter that adds ESM2-3B features to the amino-acid token embeddings (27M parameters) | 109 MB |
| `frontend.safetensors` | conditioning front-end of the structure head (8.4M parameters; stage-2 EMA, step 30,000) | 34 MB |
| `calpha_config.json` | layers read, canonical request, sampling settings | |
| `benchmark_outputs/` | alpha-carbon predictions of every system in the report on the three test sets, raw text of the chat models | 54 MB |
| `training_data/` | structure data and language data used for training (see its README) | 77 GB |

**Not included, downloaded automatically from their original sources:** the ESM2-3B encoder
(fair-esm) and the SimpleFold-3B head ([apple/ml-simplefold](https://github.com/apple/ml-simplefold)).
Every weight of the head other than the front-end is the public SimpleFold-3B checkpoint,
unchanged.

## Use

```bash
git clone https://github.com/yliust/Calpha.git && cd Calpha
pip install -e .
pip install git+https://github.com/apple/ml-simplefold.git@94e2cd704cf7b86ef8e7cb27a1e77154c4e35b34
calpha fold MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAVQVKVKALPDAQFEVV
```

```python
from calpha import Calpha
model = Calpha.from_pretrained("liuyong9975/Calpha")
out = model.fold("MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ", out_dir="predictions")
```

One GPU with about 48 GB of memory. `llm/` is a standard `transformers` checkpoint and can be
loaded on its own as a chat model; without the adapter and the code in the GitHub repository it
does not see ESM2 features and cannot produce structures.

## Results

TM-score on alpha carbons, one sample per target, failures counted as zero, all systems scored
by the same program.

| Model | FoldBench-334 | CASP16 | CAMEO-2025 |
|---|---|---|---|
| **Calpha** | **0.807** | **0.635** | **0.900** |
| Head-only control (ESM2 → SimpleFold-3B, no language model) | 0.822 | 0.682 | 0.917 |
| ESMFold | 0.869 | 0.762 | 0.905 |
| SimpleFold-3B | 0.842 | 0.704 | 0.938 |
| ESM3-open | 0.787 | 0.643 | 0.852 |
| DPLM-2 (650M) | 0.656 | 0.515 | 0.757 |
| Random walk (floor) | 0.140 | 0.139 | 0.136 |

Against the head-only control, no loss of accuracy is detected for chains of 100–300 residues
on any set, and the loss is significant for chains of 300–600 residues on all three.

## Training

- **Stage 1.** All of Qwen3.5-9B, the ESM2 adapter and a SimpleFold-100M head (initialised from
  its public weights) trained jointly for 2B tokens: language cross-entropy, trigger
  cross-entropy and the head's flow-matching loss. Language data: 75% protein tasks, 25% general
  English text and reasoning. Structure data: 84,658 PDB chains cropped to 256 residues.
- **Stage 2.** Language model frozen, SimpleFold-3B frozen, only the front-end trained: 20,000
  steps at a crop of 256 residues, then 10,000 steps at 512 (effective batch 64, learning rate
  1e-4, EMA 0.999), about 11 hours on eight H200 GPUs.

## Limitations

- The language model does not make folding more accurate: Calpha is below the same head reading
  ESM2 directly on two of three sets and below ESMFold on FoldBench-334 and CASP16.
- Above 300 residues the language model costs accuracy on every set.
- The system depends on ESM2 and on the SimpleFold weights.
- MMLU-Pro accuracy fell from 0.774 to 0.703 during protein training.
- Single chains, single sequence, no ligands or complexes. Accuracy was measured on alpha
  carbons only, with one request phrasing; side chains, clashes and other phrasings were not
  evaluated.
- Not for clinical or diagnostic use.

## License

- `llm/` is a fine-tuned derivative of Qwen3.5-9B and is released under the Apache License 2.0.
- `esm_adapter.safetensors`, `frontend.safetensors`, the benchmark outputs and the code are
  released under the Apache License 2.0.
- Producing a structure requires the SimpleFold-3B weights, which Apple releases under the
  **Apple Machine Learning Research Model License (research use only)**. Calpha as a whole can
  therefore only be used under those terms.
- ESM2 is MIT-licensed. The training data has its own notes in `training_data/README.md`.

## Citation

```bibtex
@techreport{liu2026calpha,
  title  = {Calpha: Toward Native Biomolecular Intelligence in a General-Purpose Language Model.
            Technical Report I: Protein Folding},
  author = {Liu, Yong},
  year   = {2026},
  url    = {https://calpha-ai.com}
}
```