--- license: other license_name: mixed-see-model-card license_link: https://github.com/yliust/Calpha/blob/main/NOTICE base_model: Qwen/Qwen3.5-9B language: - en tags: - biology - protein - protein-structure-prediction - protein-folding library_name: calpha --- # Calpha **Toward native biomolecular intelligence in a general-purpose language model. Technical Report I: Protein Folding.** [Homepage](https://calpha-ai.com) · [Code](https://github.com/yliust/Calpha) · Technical report (coming soon) Calpha is a chat model that folds proteins. A protein sequence enters Qwen3.5-9B inside an ordinary chat turn, together with per-residue features from frozen ESM2-3B. Asked for a structure, the model answers with a `` token, and its hidden states from nine layers at the residue positions pass through a small conditioning front-end into the frozen SimpleFold-3B flow-matching head, which generates all-atom coordinates. This repository holds the weights that reproduce the numbers of the report, the raw benchmark outputs, and the training data. ## Files | Path | Content | Size | |---|---|---| | `llm/` | Qwen3.5-9B after stage-1 training (bf16), tokenizer with amino-acid tokens and `` | 18 GB | | `esm_adapter.safetensors` | adapter that adds ESM2-3B features to the amino-acid token embeddings (27M parameters) | 109 MB | | `frontend.safetensors` | conditioning front-end of the structure head (8.4M parameters; stage-2 EMA, step 30,000) | 34 MB | | `calpha_config.json` | layers read, canonical request, sampling settings | | | `benchmark_outputs/` | alpha-carbon predictions of every system in the report on the three test sets, raw text of the chat models | 54 MB | | `training_data/` | structure data and language data used for training (see its README) | 77 GB | **Not included, downloaded automatically from their original sources:** the ESM2-3B encoder (fair-esm) and the SimpleFold-3B head ([apple/ml-simplefold](https://github.com/apple/ml-simplefold)). Every weight of the head other than the front-end is the public SimpleFold-3B checkpoint, unchanged. ## Use ```bash git clone https://github.com/yliust/Calpha.git && cd Calpha pip install -e . pip install git+https://github.com/apple/ml-simplefold.git@94e2cd704cf7b86ef8e7cb27a1e77154c4e35b34 calpha fold MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQAPILSRVGDGTQDNLSGAEKAVQVKVKALPDAQFEVV ``` ```python from calpha import Calpha model = Calpha.from_pretrained("liuyong9975/Calpha") out = model.fold("MKTAYIAKQRQISFVKSHFSRQLEERLGLIEVQ", out_dir="predictions") ``` One GPU with about 48 GB of memory. `llm/` is a standard `transformers` checkpoint and can be loaded on its own as a chat model; without the adapter and the code in the GitHub repository it does not see ESM2 features and cannot produce structures. ## Results TM-score on alpha carbons, one sample per target, failures counted as zero, all systems scored by the same program. | Model | FoldBench-334 | CASP16 | CAMEO-2025 | |---|---|---|---| | **Calpha** | **0.807** | **0.635** | **0.900** | | Head-only control (ESM2 → SimpleFold-3B, no language model) | 0.822 | 0.682 | 0.917 | | ESMFold | 0.869 | 0.762 | 0.905 | | SimpleFold-3B | 0.842 | 0.704 | 0.938 | | ESM3-open | 0.787 | 0.643 | 0.852 | | DPLM-2 (650M) | 0.656 | 0.515 | 0.757 | | Random walk (floor) | 0.140 | 0.139 | 0.136 | Against the head-only control, no loss of accuracy is detected for chains of 100–300 residues on any set, and the loss is significant for chains of 300–600 residues on all three. ## Training - **Stage 1.** All of Qwen3.5-9B, the ESM2 adapter and a SimpleFold-100M head (initialised from its public weights) trained jointly for 2B tokens: language cross-entropy, trigger cross-entropy and the head's flow-matching loss. Language data: 75% protein tasks, 25% general English text and reasoning. Structure data: 84,658 PDB chains cropped to 256 residues. - **Stage 2.** Language model frozen, SimpleFold-3B frozen, only the front-end trained: 20,000 steps at a crop of 256 residues, then 10,000 steps at 512 (effective batch 64, learning rate 1e-4, EMA 0.999), about 11 hours on eight H200 GPUs. ## Limitations - The language model does not make folding more accurate: Calpha is below the same head reading ESM2 directly on two of three sets and below ESMFold on FoldBench-334 and CASP16. - Above 300 residues the language model costs accuracy on every set. - The system depends on ESM2 and on the SimpleFold weights. - MMLU-Pro accuracy fell from 0.774 to 0.703 during protein training. - Single chains, single sequence, no ligands or complexes. Accuracy was measured on alpha carbons only, with one request phrasing; side chains, clashes and other phrasings were not evaluated. - Not for clinical or diagnostic use. ## License - `llm/` is a fine-tuned derivative of Qwen3.5-9B and is released under the Apache License 2.0. - `esm_adapter.safetensors`, `frontend.safetensors`, the benchmark outputs and the code are released under the Apache License 2.0. - Producing a structure requires the SimpleFold-3B weights, which Apple releases under the **Apple Machine Learning Research Model License (research use only)**. Calpha as a whole can therefore only be used under those terms. - ESM2 is MIT-licensed. The training data has its own notes in `training_data/README.md`. ## Citation ```bibtex @techreport{liu2026calpha, title = {Calpha: Toward Native Biomolecular Intelligence in a General-Purpose Language Model. Technical Report I: Protein Folding}, author = {Liu, Yong}, year = {2026}, url = {https://calpha-ai.com} } ```