ProtSent-V2 ESM-2 35M
Contrastively fine-tuned ESM-2 35M producing fixed-length protein embeddings where biological similarity maps to embedding proximity. Intended for retrieval, clustering, and nearest-neighbour transfer.
Retrained on a corpus decontaminated against the benchmark test sets. Predecessor: oriel9p/protsent-esm2-35M. Other scale: GrimSqueaker/ProtSent-V2-150M.
Usage
from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim
model = SentenceTransformer("GrimSqueaker/ProtSent-V2-35M")
emb = model.encode([
"MKTLLLTLVVVTIVCLDLGYT",
"MKTLLLTLVVVTIVCLDLGYN",
"AGWYRSPQEGLKPVDTFKDIV",
])
print(cos_sim(emb[0], emb[1:]))
Embeddings are mean-pooled over the final layer, dimension 480. Matryoshka heads at 64/128/256 are available by truncating the embedding.
Training data
Three sources, all decontaminated (see below). No DMS/ProteinGym component — unlike the V1 release, which included ProteinGym DMS pairs under a CoSENT loss.
| source | pairs used |
|---|---|
| Pfam families | 777,306 |
| AlphaFold DB (Foldseek clusters) | 18,987,468 |
| STRING-DB v12 PPI | 15,000,000 |
| total | 34,764,774 |
Pfam and AFDB pairs are sampled within clusters; STRING is a fixed 15M-pair subsample (seed 42) of the filtered pair table.
Decontamination
Every source was searched against the benchmark test sequences with MMseqs2 easy-search
(corpus as query, 40% identity, 80% coverage, --cov-mode 1) and matching sequences
removed before training.
| corpus | rows before | rows after | removed |
|---|---|---|---|
| Pfam | 28,530,684 | 27,929,772 | 2.11% |
| AlphaFold DB | 135,404,259 | 126,301,607 | 6.72% |
| STRING | 76,070,154 | 71,891,417 | 5.49% |
Filter targets: biomap-research/fold_prediction (remote homology) and
Synthyra/bernett_gold_ppi (PPI) test splits. The result was verified by semi-joining each
training file against the removal lists: zero flagged sequences remained.
SCOPe-40 was not a filter target — it has no train/test split, so filtering against it would remove nearly all domain sequences from the corpus.
Training configuration
| setting | value |
|---|---|
| backbone | ESM-2 35M (480 hidden, 12 layers) |
| loss | CachedMultipleNegativesRankingLoss |
| contrastive batch | 1024 per device |
| gather across devices | off |
| synthetic hard negatives | off |
| multi-dataset sampler | proportional |
| Matryoshka dims | 64 / 128 / 256 |
| max sequence length | 512 |
| optimiser | AdamW, LR 2e-4, cosine_with_min_lr |
| precision / attention | bf16, flash-attention-2 |
| hardware | 7x NVIDIA B300 |
| steps | 4,850 (one epoch) |
| gradient-cache mini-batch | 256 |
| warmup | 1,000 steps |
| wall clock | 10 h 53 m |
Training code: github.com/oriel9p/ProtSent,
train_esm2_35m.sh.
Results
SCOPe-40 structural retrieval, test split, self excluded, no-hit queries counted as failures. Restricted to the 1,693 of 2,207 queries that have a non-self same-family protein in the gallery.
| method | R@1 | R@10 | MAP |
|---|---|---|---|
| ESM-2 35M | 0.4991 | 0.7614 | 0.4210 |
MMseqs2 (-s 7.5) |
0.6556 | 0.7348 | 0.4041 |
| ProtSent-V1 35M | 0.5854 | 0.8512 | 0.5509 |
| ProtSent-V2 35M | 0.6852 | 0.9220 | 0.6459 |
Paired bootstrap over queries (10,000 resamples), V2 − V1: R@1 +0.0986 [+0.0762, +0.1211], MAP +0.0943 [+0.0814, +0.1074]. Against a maximally sensitive HMMER (phmmer) this model is statistically tied at top-1 and ahead at R@10 and MAP.
Remote homology (the task the corpus was filtered against), test split:
| model | 3-NN accuracy | linear-probe accuracy |
|---|---|---|
| ESM-2 35M | 0.5835 | 0.6868 |
| ProtSent-V1 35M | 0.6587 | 0.6899 |
| ProtSent-V2 35M | 0.6668 | 0.7016 |
Limitations
- Under a trained linear probe on the final layer, this model is roughly neutral to slightly worse than the stock ESM-2 backbone across a 23-task suite. The advantage is in nearest-neighbour geometry, not in information a trained readout can extract.
- The final layer is not the best pooling layer for property prediction. In a layer sweep on remote homology, an intermediate layer (~2/3 depth) scored higher for every model tested, including the stock backbone.
- V2 differs from V1 in more than decontamination: no hard negatives, proportional sampling, no DMS source, larger effective batch. It is not a controlled ablation of filtering alone.
- Only the remote-homology and PPI test sets were decontamination targets; other benchmark test sets were not filtered against.
Citation
- Downloads last month
- -