AbLMs scaling laws

Antibody language models (AbLMs) help decode immune repertoires for therapeutic discovery, but optimal scaling strategies remain unclear. We trained ESM-2-based AbLMs across five sizes (8Mโ€“650M parameters) and three paired Ab sequences dataset scales (~1.6M total sequences). Results are described in our preprint on bioRxiv. Effective AbLM scaling requires balancing model size with data availability to avoid diminishing returns. The pre-trained models and the datasets used to train them are available on Zenodo and code is available on GitHub.

This model

  • Model size: 650M (33 transformer layers, hidden size 1280, 20 attention heads)
  • Training data: Half dataset (824,363 paired antibody sequences)
  • Checkpoint: training step 330,000 (as listed in Table S1 of the paper)

All 15 models of the study are in the AbLMs Scaling Laws collection.

Use

Load the model and tokenizer as follows:

from transformers import EsmTokenizer, EsmForMaskedLM

model = EsmForMaskedLM.from_pretrained("brineylab/650M_half_checkpoint-330000")
tokenizer = EsmTokenizer.from_pretrained("brineylab/650M_half_checkpoint-330000")  # the standard ESM-2 tokenizer

The models were trained on natively paired antibody sequences. The input is the heavy-chain amino acid sequence, two <cls> tokens as a separator, and the light-chain amino acid sequence (during training, inputs were padded or truncated to 320 tokens):

heavy = "EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLEWVARIYPTNGYTRYADSVKGRFTISADTSKNTAYLQMNSLRAEDTAVYYCSRWGGDGFYAMDYWGQGTLVTVSS"
light = "DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKLLIYSASFLYSGVPSRFSGSRSGTDFTLTISSLQPEDFATYYCQQHYTTPPTFGQGTKVEIK"

inputs = tokenizer(heavy + "<cls><cls>" + light, return_tensors="pt")
outputs = model(**inputs)

The model can be finetuned for classification tasks (such as specificity and pair classification in the paper) by loading the model with a sequence classification head:

from transformers import EsmForSequenceClassification

model = EsmForSequenceClassification.from_pretrained("brineylab/650M_half_checkpoint-330000")

# freeze the base model weights prior to finetuning
for param in model.base_model.parameters():
    param.requires_grad = False
Downloads last month
31
Safetensors
Model size
0.7B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including brineylab/650M_half_checkpoint-330000