Instructions to use brineylab/150M_half_checkpoint-330000 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use brineylab/150M_half_checkpoint-330000 with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("fill-mask", model="brineylab/150M_half_checkpoint-330000")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("brineylab/150M_half_checkpoint-330000") model = AutoModelForMaskedLM.from_pretrained("brineylab/150M_half_checkpoint-330000", device_map="auto") - Notebooks
- Google Colab
- Kaggle
AbLMs scaling laws
Antibody language models (AbLMs) help decode immune repertoires for therapeutic discovery, but optimal scaling strategies remain unclear. We trained ESM-2-based AbLMs across five sizes (8Mโ650M parameters) and three paired Ab sequences dataset scales (~1.6M total sequences). Results are described in our preprint on bioRxiv. Effective AbLM scaling requires balancing model size with data availability to avoid diminishing returns. The pre-trained models and the datasets used to train them are available on Zenodo and code is available on GitHub.
This model
- Model size: 150M (30 transformer layers, hidden size 640, 20 attention heads)
- Training data: Half dataset (824,363 paired antibody sequences)
- Checkpoint: training step 330,000 (as listed in Table S1 of the paper)
All 15 models of the study are in the AbLMs Scaling Laws collection.
Use
Load the model and tokenizer as follows:
from transformers import EsmTokenizer, EsmForMaskedLM
model = EsmForMaskedLM.from_pretrained("brineylab/150M_half_checkpoint-330000")
tokenizer = EsmTokenizer.from_pretrained("brineylab/150M_half_checkpoint-330000") # the standard ESM-2 tokenizer
The models were trained on natively paired antibody sequences. The input is the heavy-chain amino acid sequence, two <cls> tokens as a separator, and the light-chain amino acid sequence (during training, inputs were padded or truncated to 320 tokens):
heavy = "EVQLVESGGGLVQPGGSLRLSCAASGFNIKDTYIHWVRQAPGKGLEWVARIYPTNGYTRYADSVKGRFTISADTSKNTAYLQMNSLRAEDTAVYYCSRWGGDGFYAMDYWGQGTLVTVSS"
light = "DIQMTQSPSSLSASVGDRVTITCRASQDVNTAVAWYQQKPGKAPKLLIYSASFLYSGVPSRFSGSRSGTDFTLTISSLQPEDFATYYCQQHYTTPPTFGQGTKVEIK"
inputs = tokenizer(heavy + "<cls><cls>" + light, return_tensors="pt")
outputs = model(**inputs)
The model can be finetuned for classification tasks (such as specificity and pair classification in the paper) by loading the model with a sequence classification head:
from transformers import EsmForSequenceClassification
model = EsmForSequenceClassification.from_pretrained("brineylab/150M_half_checkpoint-330000")
# freeze the base model weights prior to finetuning
for param in model.base_model.parameters():
param.requires_grad = False
- Downloads last month
- 30