P-Mistral-7B

Mistral-7B adapted to generate protein sequences

Mistral-7B fine-tuned on roughly 42,000 human protein sequences from UniRef50, with a tokenizer adapted to amino-acid sequences. The paper shows that general-purpose LLMs trained on this comparatively small dataset can match or exceed specialised protein models trained on hundreds of millions of sequences, judged by structural quality metrics.

From Design Proteins Using Large Language Models: Enhancements and Comparative Analyses (LangMol 2024, ACL workshop). Code: protein-design-LLMs.

Results

Structural evaluation of generated sequences (70–140 amino acids), compared with natural and random sequences and with specialised protein models (see the paper for the full table):

Model	pLDDT ↑	Intra-RMSD ↓	REU ↓	TM-Score ↑	Inter-RMSD ↓
Natural proteins	67.77	–	−153.06	–	4.40
Random	39.71	9.88	−197.22	0.41	6.81
P-Mistral-7B	72.03	5.42	−197.40	0.68	4.70
P-Llama2-7B	65.39	7.02	−153.31	0.63	4.76
P-Llama3-8B	62.99	7.38	−132.50	0.65	4.30
P-gemma-7B	62.24	5.80	−141.60	0.65	5.83

pLDDT is the predicted structure confidence, TM-Score the structural similarity to known folds, and REU the Rosetta energy; natural proteins give the reference range.

Usage

The generation script in the repository handles the tokenizer and decoding:

git clone https://github.com/KamyarZeinalipour/protein-design-LLMs && cd protein-design-LLMs
pip install -r requirements.txt
python main.py --help

Models from this paper

Model	Base	pLDDT
P-Mistral-7B (this model)	Mistral-7B	72.03
P-Llama2-7B	Llama-2-7B	65.39
P-Llama3-8B	Llama-3-8B	62.99
P-gemma-7B	Gemma-7B	62.24

Training data: UniRef50-HumanProteins.

Citation

@inproceedings{zeinalipour-etal-2024-design,
    title = "Design Proteins Using Large Language Models: Enhancements and Comparative Analyses",
    author = "Zeinalipour, Kamyar  and
      Jamshidi, Neda  and
      Bianchini, Monica  and
      Maggini, Marco  and
      Gori, Marco",
    editor = "Edwards, Carl  and
      Wang, Qingyun  and
      Li, Manling  and
      Zhao, Lawrence  and
      Hope, Tom  and
      Ji, Heng",
    booktitle = "Proceedings of the 1st Workshop on Language + Molecules ({L}+{M} 2024)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.langmol-1.5/",
    doi = "10.18653/v1/2024.langmol-1.5",
    pages = "34--47"
}

License

Released under the base model's license (apache-2.0). For research use.