README.md

---
license: mit
language: en
tags:
  - peptide
  - biology
  - drug-discovery
  - HELM
  - helm-notation
  - cyclic-peptide
  - peptide-language-model
pipeline_tag: fill-mask
widget:
  - text: "PEPTIDE1{[Abu].[Sar].[meL].V.[meL].A.[dA].[meL].[meL].[meV].[Me_Bmt(E)]}$PEPTIDE1,PEPTIDE1,1:R1-11:R2$$$"
---

# HELM-BERT

A peptide language model using **HELM (Hierarchical Editing Language for Macromolecules)** notation, compatible with Hugging Face Transformers.

[![GitHub](https://img.shields.io/badge/GitHub-clinfo%2FHELM--BERT-black?logo=github)](https://github.com/clinfo/HELM-BERT)

## Model Description

HELM-BERT is built upon the DeBERTa architecture, pre-trained on ~75k peptides from four databases (ChEMBL, CREMP, CycPeptMPDB, Propedia) using **Masked Language Modeling (MLM)** with a **Warmup-Stable-Decay (WSD)** learning rate schedule.

- **Disentangled Attention**: Decomposes attention into content-content and content-position terms
- **Enhanced Mask Decoder (EMD)**: Injects absolute position embeddings at the decoder stage
- **Span Masking**: Contiguous token masking with geometric distribution
- **nGiE**: n-gram Induced Encoding layer (1D convolution, kernel size 3)

<p align="center"><img src="assets/HELM-BERT.png" width="600"></p>

## Model Specifications

| Parameter | Value |
|-----------|-------|
| Parameters | 54.8M |
| Hidden size | 768 |
| Layers | 6 |
| Attention heads | 12 |
| Vocab size | 78 |
| Max token length | 512 |
| Pre-training data | ~75k peptides (ChEMBL, CREMP, CycPeptMPDB, Propedia) |
| Pre-training objective | MLM (span masking, p=0.15) |
| LR schedule | Warmup-Stable-Decay (WSD) |

## How to Use

```python
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Flansma/helm-bert", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Flansma/helm-bert", trust_remote_code=True)

# Cyclosporine A
inputs = tokenizer("PEPTIDE1{[Abu].[Sar].[meL].V.[meL].A.[dA].[meL].[meL].[meV].[Me_Bmt(E)]}$PEPTIDE1,PEPTIDE1,1:R1-11:R2$$$", return_tensors="pt")
outputs = model(**inputs)
embeddings = outputs.last_hidden_state
```

## Training Data

Pre-trained on deduplicated peptide sequences from:
- **ChEMBL**: Bioactive molecules database
- **CREMP**: Cyclic peptide conformational ensemble database
- **CycPeptMPDB**: Cyclic peptide membrane permeability database
- **Propedia**: Protein-peptide interaction database

## Downstream Performance

### Permeability Regression (CycPeptMPDB)

**Single-Assay** (mixed PAMPA/Caco-2 target):

| Split | R² | Pearson | RMSE | MAE |
|:-----:|:--:|:-------:|:----:|:---:|
| Random | 0.658 | 0.817 | 0.471 | 0.300 |
| Scaffold | 0.502 | 0.723 | 0.450 | 0.324 |

**Per-Assay** (separate models for PAMPA and Caco-2):

| Split | Assay | R² | Pearson | RMSE | MAE |
|:-----:|:-----:|:--:|:-------:|:----:|:---:|
| Random | PAMPA | 0.800 | 0.895 | 0.355 | 0.253 |
| Random | Caco-2 | 0.747 | 0.866 | 0.388 | 0.289 |
| Scaffold | PAMPA | 0.529 | 0.739 | 0.412 | 0.295 |
| Scaffold | Caco-2 | 0.637 | 0.874 | 0.405 | 0.334 |

Train/test 9:1, val 10% from train. Scaffold split by Murcko scaffolds.

<p align="center"><img src="assets/tsne_cycpeptmpdb_permeability_mix_random_scaffold.png" width="800"></p>

### PPI Classification (Propedia v2)

| Split | ROC-AUC | PR-AUC | F1 | MCC | Balanced Acc |
|:-----:|:-------:|:------:|:--:|:---:|:------------:|
| Random | 0.968 | 0.901 | 0.847 | 0.808 | 0.906 |
| aCSM | 0.862 | 0.683 | 0.587 | 0.522 | 0.722 |

Train/test 8:2, val 10% from train, 1:4 positive:negative ratio.
- **Random**: random split
- **aCSM**: clustering-based split on aCSM-ALL complex signatures with protein overlap pruning

<p align="center"><img src="assets/tsne_propedia_ppi_random_acsm.png" width="800"></p>

### PPI Classification (ChEMBL)

| Split | ROC-AUC | PR-AUC | F1 | MCC | Balanced Acc |
|:-----:|:-------:|:------:|:--:|:---:|:------------:|
| Random | 0.992 | 0.975 | 0.948 | 0.936 | 0.969 |
| Family | 0.786 | 0.449 | 0.267 | 0.222 | 0.570 |

Val 10% from train.

<p align="center"><img src="assets/tsne_chembl_ppi_random_family.png" width="800"></p>


## Citation

```bibtex
@article{lee2026helmbert,
  title={HELM-BERT: Topology-Aware Representations for Chemically Modified Peptides},
  author={Lee, Seungeon and Koyama, Takuto and Maeda, Itsuki and Matsumoto, Shigeyuki and Okuno, Yasushi},
  journal={Journal of Chemical Information and Modeling},
  year={2026},
  doi={10.1021/acs.jcim.6c00451},
  publisher={American Chemical Society},
  url={https://pubs.acs.org/doi/10.1021/acs.jcim.6c00451}
}
```

## License

MIT License