Instructions to use GleghornLab/DSM_650 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use GleghornLab/DSM_650 with Transformers:
# pip install -U transformers accelerate # Load model directly from transformers import ESM_Diff model = ESM_Diff.from_pretrained("GleghornLab/DSM_650", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Rewrite the model card from the project repository
Browse filesAdds what this checkpoint is, its architecture and file sizes from its own config, how to load and use it, the data and hyperparameters it was trained with, the preprint's numbers for it, links, and the citation. Keeps the shared project README below, and the license and library metadata. Nothing else in the repository changes.
README.md
CHANGED
|
@@ -1,506 +1,637 @@
|
|
| 1 |
-
---
|
| 2 |
-
library_name: transformers
|
| 3 |
-
|
| 4 |
-
--
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
-
|
| 11 |
-
|
| 12 |
-
|
| 13 |
-
|
| 14 |
-
-
|
| 15 |
-
|
| 16 |
-
|
| 17 |
-
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
##
|
| 28 |
-
|
| 29 |
-
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
| 35 |
-
|
| 36 |
-
|
| 37 |
-
|
| 38 |
-
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
| 62 |
-
|
| 63 |
-
|
| 64 |
-
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
``
|
| 68 |
-
|
| 69 |
-
``
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
|
| 73 |
-
|
| 74 |
-
|
| 75 |
-
|
| 76 |
-
|
| 77 |
-
|
| 78 |
-
|
| 79 |
-
|
| 80 |
-
|
| 81 |
-
|
| 82 |
-
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
|
| 90 |
-
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
|
| 98 |
-
|
| 99 |
-
|
| 100 |
-
|
| 101 |
-
|
| 102 |
-
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
|
| 112 |
-
|
| 113 |
-
|
| 114 |
-
|
| 115 |
-
|
| 116 |
-
|
| 117 |
-
|
| 118 |
-
|
| 119 |
-
|
| 120 |
-
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
| 124 |
-
|
| 125 |
-
|
| 126 |
-
|
| 127 |
-
|
| 128 |
-
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
|
| 134 |
-
|
| 135 |
-
|
| 136 |
-
|
| 137 |
-
|
| 138 |
-
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
|
| 154 |
-
|
| 155 |
-
|
| 156 |
-
|
| 157 |
-
|
| 158 |
-
|
| 159 |
-
|
| 160 |
-
|
| 161 |
-
|
| 162 |
-
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
|
| 166 |
-
|
| 167 |
-
|
| 168 |
-
|
| 169 |
-
|
| 170 |
-
|
| 171 |
-
|
| 172 |
-
|
| 173 |
-
|
| 174 |
-
|
| 175 |
-
|
| 176 |
-
|
| 177 |
-
|
| 178 |
-
|
| 179 |
-
|
| 180 |
-
|
| 181 |
-
|
| 182 |
-
|
| 183 |
-
|
| 184 |
-
|
| 185 |
-
|
| 186 |
-
|
| 187 |
-
|
| 188 |
-
|
| 189 |
-
|
| 190 |
-
|
| 191 |
-
|
| 192 |
-
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
|
| 196 |
-
|
| 197 |
-
|
| 198 |
-
|
| 199 |
-
#
|
| 200 |
-
|
| 201 |
-
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
-
|
| 206 |
-
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
|
| 210 |
-
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
|
| 216 |
-
|
| 217 |
-
|
| 218 |
-
|
| 219 |
-
|
| 220 |
-
|
| 221 |
-
|
| 222 |
-
|
| 223 |
-
|
| 224 |
-
|
| 225 |
-
|
| 226 |
-
|
| 227 |
-
|
| 228 |
-
|
| 229 |
-
|
| 230 |
-
|
| 231 |
-
|
| 232 |
-
|
| 233 |
-
|
| 234 |
-
|
| 235 |
-
|
| 236 |
-
|
| 237 |
-
|
| 238 |
-
#
|
| 239 |
-
|
| 240 |
-
|
| 241 |
-
|
| 242 |
-
|
| 243 |
-
|
| 244 |
-
|
| 245 |
-
|
| 246 |
-
|
| 247 |
-
|
| 248 |
-
|
| 249 |
-
|
| 250 |
-
|
| 251 |
-
|
| 252 |
-
|
| 253 |
-
|
| 254 |
-
|
| 255 |
-
|
| 256 |
-
|
| 257 |
-
|
| 258 |
-
|
| 259 |
-
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
| 271 |
-
|
| 272 |
-
|
| 273 |
-
|
| 274 |
-
|
| 275 |
-
|
| 276 |
-
|
| 277 |
-
|
| 278 |
-
|
| 279 |
-
|
| 280 |
-
|
| 281 |
-
|
| 282 |
-
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
```
|
| 288 |
-
|
| 289 |
-
|
| 290 |
-
|
| 291 |
-
|
| 292 |
-
|
| 293 |
-
|
| 294 |
-
|
| 295 |
-
|
| 296 |
-
|
| 297 |
-
|
| 298 |
-
|
| 299 |
-
|
| 300 |
-
|
| 301 |
-
|
| 302 |
-
|
| 303 |
-
|
| 304 |
-
|
| 305 |
-
|
| 306 |
-
|
| 307 |
-
|
| 308 |
-
|
| 309 |
-
|
| 310 |
-
|
| 311 |
-
|
| 312 |
-
|
| 313 |
-
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
|
| 317 |
-
|
| 318 |
-
|
| 319 |
-
|
| 320 |
-
|
| 321 |
-
|
| 322 |
-
|
| 323 |
-
|
| 324 |
-
```
|
| 325 |
-
|
| 326 |
-
|
| 327 |
-
|
| 328 |
-
|
| 329 |
-
|
| 330 |
-
|
| 331 |
-
|
| 332 |
-
|
| 333 |
-
|
| 334 |
-
|
| 335 |
-
|
| 336 |
-
|
| 337 |
-
|
| 338 |
-
|
| 339 |
-
|
| 340 |
-
|
| 341 |
-
|
| 342 |
-
|
| 343 |
-
|
| 344 |
-
|
| 345 |
-
|
| 346 |
-
|
| 347 |
-
|
| 348 |
-
|
| 349 |
-
|
| 350 |
-
|
| 351 |
-
|
| 352 |
-
|
| 353 |
-
|
| 354 |
-
|
| 355 |
-
|
| 356 |
-
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
|
| 360 |
-
|
| 361 |
-
|
| 362 |
-
|
| 363 |
-
|
| 364 |
-
|
| 365 |
-
|
| 366 |
-
|
| 367 |
-
|
| 368 |
-
|
| 369 |
-
|
| 370 |
-
|
| 371 |
-
|
| 372 |
-
|
| 373 |
-
|
| 374 |
-
|
| 375 |
-
|
| 376 |
-
|
| 377 |
-
|
| 378 |
-
|
| 379 |
-
|
| 380 |
-
|
| 381 |
-
|
| 382 |
-
|
| 383 |
-
|
| 384 |
-
|
| 385 |
-
|
| 386 |
-
|
| 387 |
-
|
| 388 |
-
|
| 389 |
-
|
| 390 |
-
|
| 391 |
-
|
| 392 |
-
|
| 393 |
-
|
| 394 |
-
|
| 395 |
-
|
| 396 |
-
|
| 397 |
-
|
| 398 |
-
|
| 399 |
-
|
| 400 |
-
|
| 401 |
-
|
| 402 |
-
|
| 403 |
-
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
|
| 407 |
-
|
| 408 |
-
|
| 409 |
-
|
| 410 |
-
|
| 411 |
-
|
| 412 |
-
|
| 413 |
-
|
| 414 |
-
|
| 415 |
-
|
| 416 |
-
|
| 417 |
-
|
| 418 |
-
|
| 419 |
-
|
| 420 |
-
|
| 421 |
-
|
| 422 |
-
|
| 423 |
-
|
| 424 |
-
|
| 425 |
-
|
| 426 |
-
|
| 427 |
-
|
| 428 |
-
- **
|
| 429 |
-
|
| 430 |
-
- **
|
| 431 |
-
-
|
| 432 |
-
-
|
| 433 |
-
|
| 434 |
-
- **
|
| 435 |
-
|
| 436 |
-
-
|
| 437 |
-
-
|
| 438 |
-
-
|
| 439 |
-
|
| 440 |
-
-
|
| 441 |
-
|
| 442 |
-
|
| 443 |
-
|
| 444 |
-
|
| 445 |
-
|
| 446 |
-
|
| 447 |
-
|
| 448 |
-
|
| 449 |
-
|
| 450 |
-
|
| 451 |
-
|
| 452 |
-
|
| 453 |
-
|
| 454 |
-
|
| 455 |
-
|
| 456 |
-
|
| 457 |
-
|
| 458 |
-
|
| 459 |
-
*
|
| 460 |
-
*
|
| 461 |
-
*
|
| 462 |
-
*
|
| 463 |
-
*
|
| 464 |
-
*
|
| 465 |
-
*
|
| 466 |
-
*
|
| 467 |
-
*
|
| 468 |
-
*
|
| 469 |
-
*
|
| 470 |
-
|
| 471 |
-
|
| 472 |
-
|
| 473 |
-
|
| 474 |
-
|
| 475 |
-
|
| 476 |
-
|
| 477 |
-
|
| 478 |
-
|
| 479 |
-
|
| 480 |
-
|
| 481 |
-
|
| 482 |
-
|
| 483 |
-
|
| 484 |
-
|
| 485 |
-
|
| 486 |
-
|
| 487 |
-
|
| 488 |
-
|
| 489 |
-
|
| 490 |
-
|
| 491 |
-
|
| 492 |
-
|
| 493 |
-
|
| 494 |
-
|
| 495 |
-
|
| 496 |
-
|
| 497 |
-
|
| 498 |
-
|
| 499 |
-
|
| 500 |
-
|
| 501 |
-
|
| 502 |
-
|
| 503 |
-
|
| 504 |
-
|
| 505 |
-
|
| 506 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
library_name: transformers
|
| 3 |
+
license: other
|
| 4 |
+
license_name: dsm-project-license
|
| 5 |
+
license_link: LICENSE
|
| 6 |
+
tags:
|
| 7 |
+
- protein-language-model
|
| 8 |
+
- protein
|
| 9 |
+
- biology
|
| 10 |
+
---
|
| 11 |
+
|
| 12 |
+
# DSM-650
|
| 13 |
+
|
| 14 |
+
DSM-650 is a 650M-parameter protein language model trained with masked diffusion, for both
|
| 15 |
+
representation learning and generative design. It extends
|
| 16 |
+
[facebook/esm2_t33_650M_UR50D](https://huggingface.co/facebook/esm2_t33_650M_UR50D) and is the
|
| 17 |
+
model the preprint reports for unconditional generation and representation quality.
|
| 18 |
+
|
| 19 |
+
| | |
|
| 20 |
+
|---|---|
|
| 21 |
+
| Class | `ESM_Diff`, model type `esm_diff` |
|
| 22 |
+
| Hidden size | 1,280, 33 layers, 20 attention heads, feed-forward width 5,120 |
|
| 23 |
+
| Positions | rotary, up to 1,026 tokens |
|
| 24 |
+
| Vocabulary | 33 tokens, the ESM2 alphabet, mask token id 32 |
|
| 25 |
+
| Weights | `model.safetensors`, 2.6 GB, float32 |
|
| 26 |
+
|
| 27 |
+
## Load it
|
| 28 |
+
|
| 29 |
+
```python
|
| 30 |
+
import torch
|
| 31 |
+
|
| 32 |
+
from models.modeling_dsm import DSM # from github.com/Gleghorn-Lab/DSM
|
| 33 |
+
|
| 34 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 35 |
+
model = DSM.from_pretrained("GleghornLab/DSM_650").to(device).eval()
|
| 36 |
+
tokenizer = model.tokenizer
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
```console
|
| 40 |
+
You are using a model of type esm_diff to instantiate a model of type dsm.
|
| 41 |
+
```
|
| 42 |
+
|
| 43 |
+
That warning is expected.
|
| 44 |
+
|
| 45 |
+
## Unconditional generation
|
| 46 |
+
|
| 47 |
+
```python
|
| 48 |
+
length = 100
|
| 49 |
+
mask_token = tokenizer.mask_token
|
| 50 |
+
input_tokens = tokenizer.encode(
|
| 51 |
+
'M' + ''.join([mask_token] * (length - 1)), add_special_tokens=True, return_tensors='pt'
|
| 52 |
+
).to(device)
|
| 53 |
+
output = model.mask_diffusion_generate(
|
| 54 |
+
tokenizer=tokenizer,
|
| 55 |
+
input_tokens=input_tokens,
|
| 56 |
+
step_divisor=100, # lower is slower but better
|
| 57 |
+
temperature=1.0,
|
| 58 |
+
remasking="random",
|
| 59 |
+
)
|
| 60 |
+
print(model.decode_output(output)[0])
|
| 61 |
+
```
|
| 62 |
+
|
| 63 |
+
## Training
|
| 64 |
+
|
| 65 |
+
DSM is further pretraining of an ESM2 checkpoint with a masked diffusion objective adapted from
|
| 66 |
+
LLaDA: a corruption level `t` is drawn uniformly, tokens are masked at that rate, every masked
|
| 67 |
+
token is predicted in one forward pass, and the loss is scaled by `1/(t + epsilon)`, which
|
| 68 |
+
penalizes errors more at low mask rates. The language modeling head adds a linear transformation
|
| 69 |
+
and a soft-logit cap (`tau=30`), with the output projection tied to the token embeddings.
|
| 70 |
+
|
| 71 |
+
Training data is [Synthyra/omg_prot50](https://huggingface.co/datasets/Synthyra/omg_prot50), the
|
| 72 |
+
Open MetaGenomic dataset clustered at 50% identity, 207M sequences. 10,000 sequences were held out
|
| 73 |
+
for validation and 10,000 for test, and 3,300 UniProt entries deposited after the OMG release, with
|
| 74 |
+
at least transcript-level evidence, were added to the test set.
|
| 75 |
+
|
| 76 |
+
DSM-650 started from ESM2-650M and ran 100,000 steps with a global batch size of 128, a maximum
|
| 77 |
+
sequence length of 2,048, and a learning rate of 1e-4 with AdamW and a cosine schedule after 1,000
|
| 78 |
+
warmup steps. It was the most expensive run of the study, 12 days on four A100 GPUs.
|
| 79 |
+
|
| 80 |
+
## Evaluation
|
| 81 |
+
|
| 82 |
+
Unconditionally generated sequences follow natural amino acid k-mer frequencies closely, with a
|
| 83 |
+
Jensen-Shannon divergence below 0.01 for 1-mers, 2-mers, and 3-mers, and predicted Annotation
|
| 84 |
+
Vocabulary terms at a divergence of 0.102. At 90% masking, reconstruction reaches an alignment
|
| 85 |
+
score of 0.2771 on validation and 0.2766 on test, against DPLM's 0.2721 and 0.2734. On linear
|
| 86 |
+
probing across ten supervised tasks, DSM-650 gives the strongest representations among the
|
| 87 |
+
similarly sized models tested.
|
| 88 |
+
|
| 89 |
+
Sequence reconstruction on the PPI test set at a 15% mask rate, from the preprint's supplementary
|
| 90 |
+
table. DSM-ppi models see SeqA and SeqB; every other model sees SeqB only.
|
| 91 |
+
|
| 92 |
+
| Model | Cross entropy | MCC | F1 | Accuracy |
|
| 93 |
+
|---|---:|---:|---:|---:|
|
| 94 |
+
| ESM2-150M | 2.202 | 0.276 | 0.318 | 32.2% |
|
| 95 |
+
| ESM2-650M | 2.407 | 0.260 | 0.307 | 30.7% |
|
| 96 |
+
| ESM2-3B | 2.234 | 0.286 | 0.334 | 33.1% |
|
| 97 |
+
| DSM-150 | 2.225 | 0.268 | 0.311 | 31.4% |
|
| 98 |
+
| DSM-650 | 2.046 | 0.325 | 0.365 | 36.7% |
|
| 99 |
+
| DSM-150-ppi-control | 2.199 | 0.274 | 0.315 | 31.9% |
|
| 100 |
+
| DSM-150-ppi | 2.165 | 0.285 | 0.326 | 33.0% |
|
| 101 |
+
| DSM-650-ppi | 1.989 | 0.342 | 0.382 | 38.3% |
|
| 102 |
+
|
| 103 |
+
## Links
|
| 104 |
+
|
| 105 |
+
- Code, demos, and the evaluation pipeline: [Gleghorn-Lab/DSM](https://github.com/Gleghorn-Lab/DSM)
|
| 106 |
+
- Preprint: [arXiv:2506.08293](https://arxiv.org/abs/2506.08293)
|
| 107 |
+
- The [Gleghorn Lab](https://www.gleghornlab.com/) and [Synthyra](https://synthyra.com/)
|
| 108 |
+
- Terms of use: the `LICENSE` file in this repository
|
| 109 |
+
|
| 110 |
+
## Cite
|
| 111 |
+
|
| 112 |
+
```bibtex
|
| 113 |
+
@misc{hallee2025diffusionsequencemodelsenhanced,
|
| 114 |
+
title={Diffusion Sequence Models for Enhanced Protein Representation and Generation},
|
| 115 |
+
author={Logan Hallee and Nikolaos Rafailidis and David B. Bichara and Jason P. Gleghorn},
|
| 116 |
+
year={2025},
|
| 117 |
+
eprint={2506.08293},
|
| 118 |
+
archivePrefix={arXiv},
|
| 119 |
+
primaryClass={q-bio.BM},
|
| 120 |
+
url={https://arxiv.org/abs/2506.08293},
|
| 121 |
+
}
|
| 122 |
+
```
|
| 123 |
+
|
| 124 |
+
---
|
| 125 |
+
|
| 126 |
+
# The DSM project
|
| 127 |
+
|
| 128 |
+
What follows is the project README, shared with [github.com/Gleghorn-Lab/DSM](https://github.com/Gleghorn-Lab/DSM).
|
| 129 |
+
|
| 130 |
+
## Table of Contents
|
| 131 |
+
- [Introduction](#introduction)
|
| 132 |
+
- [Models](#models)
|
| 133 |
+
- [Usage](#usage)
|
| 134 |
+
- [Demos](#usage)
|
| 135 |
+
- [Local installation](#installation)
|
| 136 |
+
- [Training](#training)
|
| 137 |
+
- [Evaluation](#evaluation)
|
| 138 |
+
- [Results](#results)
|
| 139 |
+
- [Experimental validation](#experimental-validation)
|
| 140 |
+
- [Cite](#cite)
|
| 141 |
+
|
| 142 |
+
## Introduction
|
| 143 |
+
|
| 144 |
+
DSM (Diffusion Sequence Model) is a novel Protein Language Model (pLM) developed in collaboration between the [Gleghorn Lab](https://www.gleghornlab.com/) and [Synthyra](https://synthyra.com/). It was trained with masked diffusion to enable both high-quality representation learning and generative protein design. This repository contains the code for training, evaluating, and applying DSM and its variants.
|
| 145 |
+
|
| 146 |
+
DSM is capable of generating diverse, biomimetic sequences that align with expected amino acid compositions, secondary structures, and predicted functions. Furthermore, DSM's learned representations match or exceed those of comparably sized pLMs on various downstream tasks. DSM is detailed extensively in our [preprint](https://arxiv.org/abs/2506.08293) (which is currently in review). Beyond the base and PPI variants, we are currently training versions to jointly diffuse over sequence and foldseek tokens, as well as [Annotation Vocabulary](https://www.biorxiv.org/content/10.1101/2024.07.30.605924v1) tokens. Since the preprint release, Synthyra has trained [Synthyra/DSM_ppi_full](https://huggingface.co/Synthyra/DSM_ppi_full) which neglects the LoRA PPI training in favor for full finetuning. Additionally, the sequences SeqA and SeqB are jointly masked instead of just SeqB in the original version. We plan on adding the **many** new results to the second version of the preprint and eventual journal article.
|
| 147 |
+
|
| 148 |
+
## Models
|
| 149 |
+
|
| 150 |
+
Relevant Huggingface hosted models and datasets
|
| 151 |
+
|
| 152 |
+
- **Base DSM Models**:
|
| 153 |
+
- [GleghornLab/DSM_150](https://huggingface.co/GleghornLab/DSM_150) - 150M parameter DSM model
|
| 154 |
+
- [GleghornLab/DSM_650](https://huggingface.co/GleghornLab/DSM_650) - 650M parameter DSM model
|
| 155 |
+
|
| 156 |
+
- **DSM-ppi Models**:
|
| 157 |
+
(LoRA versions - results reported in paper but not recommended for real use)
|
| 158 |
+
- [GleghornLab/DSM_150_ppi_lora](https://huggingface.co/GleghornLab/DSM_150_ppi_lora) - 150M parameter LoRA DSM-ppi model
|
| 159 |
+
- [GleghornLab/DSM_650_ppi_lora](https://huggingface.co/GleghornLab/DSM_650_ppi_lora) - 650M parameter LoRA DSM-ppi model
|
| 160 |
+
- [GleghornLab/DSM_150_ppi_control](https://huggingface.co/GleghornLab/DSM_150_ppi_control) - Control version of LoRA DSM-ppi
|
| 161 |
+
|
| 162 |
+
(Fully finetuned - recommended for real use)
|
| 163 |
+
- [Synthyra/DSM_ppi_full](https://huggingface.co/Synthyra/DSM_ppi_full) - 650M parameter DSM-ppi model
|
| 164 |
+
|
| 165 |
+
- **Datasets**:
|
| 166 |
+
- [Synthyra/omg_prot50](https://huggingface.co/datasets/Synthyra/omg_prot50) - Open MetaGenomic dataset clustered at 50% identity (207M sequences)
|
| 167 |
+
- [GleghornLab/stringv12_modelorgs_9090](https://huggingface.co/datasets/GleghornLab/stringv12_modelorgs_9090) - STRING database model organisms (653k sequences)
|
| 168 |
+
|
| 169 |
+
- **Utility Models**:
|
| 170 |
+
- [GleghornLab/production_ss4_model](https://huggingface.co/GleghornLab/production_ss4_model) - Secondary structure prediction (4-class)
|
| 171 |
+
- [GleghornLab/production_ss9_model](https://huggingface.co/GleghornLab/production_ss9_model) - Secondary structure prediction (9-class)
|
| 172 |
+
|
| 173 |
+
## Usage
|
| 174 |
+
|
| 175 |
+
This section outlines how to use a trained `DSM` model for common generation tasks. The core generation logic is provided by the `GenerateMixin` class, used by `DSM` models.
|
| 176 |
+
|
| 177 |
+
First, ensure you have a trained model (either one you trained or a pre-trained one from Hugging Face Hub) and the necessary environment set up.
|
| 178 |
+
|
| 179 |
+
```python
|
| 180 |
+
import torch
|
| 181 |
+
from models.modeling_dsm import DSM # Or DSM_ppi for binder generation
|
| 182 |
+
|
| 183 |
+
# Load a pre-trained model
|
| 184 |
+
model_name_or_path = "GleghornLab/DSM_650" # Replace with your model of choice
|
| 185 |
+
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
|
| 186 |
+
model = DSM.from_pretrained(model_name_or_path).to(device).eval()
|
| 187 |
+
tokenizer = model.tokenizer
|
| 188 |
+
```
|
| 189 |
+
|
| 190 |
+
```console
|
| 191 |
+
You are using a model of type esm_diff to instantiate a model of type dsm. This is not supported for all configurations of models and can yield errors.
|
| 192 |
+
```
|
| 193 |
+
This warning is normal - all good!
|
| 194 |
+
|
| 195 |
+
### 1. Unconditional Sequence Generation
|
| 196 |
+
To generate a novel sequence of a specific length. DSM uses a progressive denoising approach.
|
| 197 |
+
|
| 198 |
+
```python
|
| 199 |
+
### Unconditional generation
|
| 200 |
+
length = 100
|
| 201 |
+
mask_token = tokenizer.mask_token
|
| 202 |
+
# optionally, enforce starting with methionine
|
| 203 |
+
input_tokens = tokenizer.encode('M' + ''.join([mask_token] * (length - 1)), add_special_tokens=True, return_tensors='pt').to(device)
|
| 204 |
+
output = model.mask_diffusion_generate(
|
| 205 |
+
tokenizer=tokenizer,
|
| 206 |
+
input_tokens=input_tokens,
|
| 207 |
+
step_divisor=100, # lower is slower but better
|
| 208 |
+
temperature=1.0, # sampling temperature
|
| 209 |
+
remasking="random", # strategy for remasking tokens not kept
|
| 210 |
+
preview=False, # set this to True to watch the mask tokens get rilled in real time
|
| 211 |
+
slow=False, # adds a small delay to the real time filling (because it is usually very fast and watching carefully is hard!)
|
| 212 |
+
return_trajectory=False # set this to True to return the trajectory of the generation (what you watch in the preview)
|
| 213 |
+
) # Note: output will be a tuple if return_trajectory is True
|
| 214 |
+
|
| 215 |
+
generated_sequences = model.decode_output(output)
|
| 216 |
+
print(f"Generated sequence: {generated_sequences[0]}")
|
| 217 |
+
```
|
| 218 |
+
|
| 219 |
+
```console
|
| 220 |
+
Generated sequence: MFRVDALQVAQQETLAIGRSTAYDKQESPSMAQRQVLTQLAAYGGENDLRQICIPAERRNFLSIANGASYQFVEEDNEANGGYWSPHKAGLPESACKRFI
|
| 221 |
+
```
|
| 222 |
+
|
| 223 |
+
### 2. Mask Filling (Inpainting)
|
| 224 |
+
To fill in masked regions of a template sequence:
|
| 225 |
+
|
| 226 |
+
```python
|
| 227 |
+
# Mask Filling / Inpainting
|
| 228 |
+
template_sequence = "MA<mask><mask><mask>KEG<mask><mask>STL"
|
| 229 |
+
input_tokens = tokenizer.encode(template_sequence, add_special_tokens=True, return_tensors='pt').to(device)
|
| 230 |
+
|
| 231 |
+
output = model.mask_diffusion_generate(
|
| 232 |
+
tokenizer=tokenizer,
|
| 233 |
+
input_tokens=input_tokens,
|
| 234 |
+
step_divisor=100, # lower is slower but better
|
| 235 |
+
temperature=1.0, # sampling temperature
|
| 236 |
+
remasking="random", # strategy for remasking tokens not kept
|
| 237 |
+
preview=False, # set this to True to watch the mask tokens get rilled in real time
|
| 238 |
+
slow=False, # adds a small delay to the real time filling (because it is usually very fast and watching carefully is hard!)
|
| 239 |
+
return_trajectory=False # set this to True to return the trajectory of the generation (what you watch in the preview)
|
| 240 |
+
) # Note: output will be a tuple if return_trajectory is True
|
| 241 |
+
|
| 242 |
+
generated_sequences = model.decode_output(output)
|
| 243 |
+
print(f"Generated sequence: {generated_sequences[0]}")
|
| 244 |
+
```
|
| 245 |
+
|
| 246 |
+
```console
|
| 247 |
+
Generated sequence: MAVKFKEGGISTL
|
| 248 |
+
```
|
| 249 |
+
|
| 250 |
+
### 3. Conditional Generation (e.g., Binders - using DSM-ppi)
|
| 251 |
+
```python
|
| 252 |
+
# from models.modeling_dsm import DSM_ppi
|
| 253 |
+
# model_binder = DSM_ppi.from_pretrained("GleghornLab/DSM_650_ppi_lora").to(device).eval()
|
| 254 |
+
# The lora version from the paper leads to unreliable outputs
|
| 255 |
+
# Synthyra has generously trained a version through full fine tuning
|
| 256 |
+
|
| 257 |
+
model = DSM.from_pretrained("Synthyra/DSM_ppi_full").to(device).eval()
|
| 258 |
+
|
| 259 |
+
# BBF-14
|
| 260 |
+
target_seq = "MGTPLWALLGGPWRGTATYEDGTKVTLDYRYTRVSPDRLRADVTYTTPDGTTLEATVDLWKDANGVIRYHATYPDGTSADGTLTQLDADTLLATGTYDDGTKYTVTLTRVAPGSGWHHHHHH"
|
| 261 |
+
# For binder generation, the 'interactor' (SeqB) part is what gets generated/filled.
|
| 262 |
+
# Start with a fully masked interactor of desired length.
|
| 263 |
+
interactor_template_len = 256
|
| 264 |
+
interactor_template = ''.join([mask_token] * interactor_template_len)
|
| 265 |
+
|
| 266 |
+
combined_input_str = target_seq + '<eos>' + interactor_template
|
| 267 |
+
|
| 268 |
+
input_tokens = tokenizer.encode(combined_input_str, add_special_tokens=True, return_tensors='pt').to(device)
|
| 269 |
+
|
| 270 |
+
output = model.mask_diffusion_generate(
|
| 271 |
+
tokenizer=tokenizer,
|
| 272 |
+
input_tokens=input_tokens,
|
| 273 |
+
step_divisor=100, # lower is slower but better
|
| 274 |
+
temperature=1.0, # sampling temperature
|
| 275 |
+
remasking="random", # strategy for remasking tokens not kept
|
| 276 |
+
preview=False, # set this to True to watch the mask tokens get rilled in real time
|
| 277 |
+
slow=False, # adds a small delay to the real time filling (because it is usually very fast and watching carefully is hard!)
|
| 278 |
+
return_trajectory=False # set this to True to return the trajectory of the generation (what you watch in the preview)
|
| 279 |
+
) # Note: output will be a tuple if return_trajectory is True
|
| 280 |
+
|
| 281 |
+
target, binder = model.decode_dual_input(output, seperator='<eos>')
|
| 282 |
+
# Parse out the generated interactor part based on EOS tokens.
|
| 283 |
+
# Example: generated_full_seq_str.split(model_binder.tokenizer.eos_token)[1]
|
| 284 |
+
print(f"Generated binder {binder[0]}")
|
| 285 |
+
```
|
| 286 |
+
|
| 287 |
+
```console
|
| 288 |
+
Generated binder HRHHHRRPTHARETEWLARMRLGIAEHQRIAVPRSDLEPDQMRERAADNQRLVKEYDQVIDHQTEGSTERLFEVLRVWEQVNTEQAHHEASAALEFGRVGYPDDEGGRAFYTQANAHKKDLVEYIGGIDEDAKWDPRIAWLMPEGGQPVKATVIGVSEERINGLKVLDDHWGRERRLWLINLFTALQAYDDPTRPTQVTLTPATDQLTNDVQYLLLSTRYTPPGVTTAVKIRKLDGRTLKVLTTEAPYVVRGATLS
|
| 289 |
+
```
|
| 290 |
+
|
| 291 |
+
Folded with Chai1:
|
| 292 |
+
|
| 293 |
+

|
| 294 |
+
|
| 295 |
+
`Synthyra/DSM_ppi_full` was actually trained to fill masks from any part of SeqA and SeqB. That means you can fully hallucinate plausibly interacting protein pairs.
|
| 296 |
+
|
| 297 |
+
```python
|
| 298 |
+
seq_a_length = 128
|
| 299 |
+
seq_b_length = 128
|
| 300 |
+
|
| 301 |
+
seq_a_template = ''.join([mask_token] * seq_a_length)
|
| 302 |
+
seq_b_template = ''.join([mask_token] * seq_b_length)
|
| 303 |
+
|
| 304 |
+
combined_input_str = seq_a_template + '<eos>' + seq_b_template
|
| 305 |
+
|
| 306 |
+
input_tokens = tokenizer.encode(combined_input_str, add_special_tokens=True, return_tensors='pt').to(device)
|
| 307 |
+
|
| 308 |
+
output = model.mask_diffusion_generate(
|
| 309 |
+
tokenizer=tokenizer,
|
| 310 |
+
input_tokens=input_tokens,
|
| 311 |
+
step_divisor=10, # lower is slower but better
|
| 312 |
+
temperature=1.0, # sampling temperature
|
| 313 |
+
remasking="random", # strategy for remasking tokens not kept
|
| 314 |
+
preview=False, # set this to True to watch the mask tokens get rilled in real time
|
| 315 |
+
slow=False, # adds a small delay to the real time filling (because it is usually very fast and watching carefully is hard!)
|
| 316 |
+
return_trajectory=False # set this to True to return the trajectory of the generation (what you watch in the preview)
|
| 317 |
+
) # Note: output will be a tuple if return_trajectory is True
|
| 318 |
+
|
| 319 |
+
seqa, seqb = model.decode_dual_input(output, seperator='<eos>')
|
| 320 |
+
# Parse out the generated interactor part based on EOS tokens.
|
| 321 |
+
# Example: generated_full_seq_str.split(model_binder.tokenizer.eos_token)[1]
|
| 322 |
+
print(f"SeqA: {seqa[0][5:]}") # remove cls token
|
| 323 |
+
print(f"SeqB: {seqb[0]}")
|
| 324 |
+
```
|
| 325 |
+
|
| 326 |
+
```console
|
| 327 |
+
SeqA: MVNLAKMRQRTEQNLREVSSFVKILFHTVLKFPMKINIGIHVHINMQAAQNAAADQNMQATNVIDLHNFKMGKDIGVDNKASATAHIYDEAHHTFLQLGAIKLLHAIPMIAGPVRCRLPIGFGHRFRG
|
| 328 |
+
SeqB: HYKNPMHSLLDSNVLHKDVVEVRLPIKIGMELDVMASAMREFLMPGTQQGDLRVIAEKRPVNKLHTYRRDLVKLLLAGAKLGTEAKSVELDLYRTELGGLVVYIININIATWDIIFAKVKICRGNDKP
|
| 329 |
+
```
|
| 330 |
+
|
| 331 |
+
Folded with Chai1:
|
| 332 |
+
|
| 333 |
+

|
| 334 |
+
|
| 335 |
+
## Demos
|
| 336 |
+
There are various demos with many more to come. For example, in `demo_dsm_ppi_full.py` (run by `python -m demos.demo_dsm_ppi_full`) we perform a test on DSM-ppi.
|
| 337 |
+
We take 1000 protein pairs from BIOGRID (real protein-protein interactions) and 1000 from Negatome (non interacting protein pairs) and mask the second sequence (SeqB) by 50%.
|
| 338 |
+
This acts as a sanity check, as we expect the accuracy on reconstructing real positive PPIs to be higher than the accuracy on non-interacting proteins.
|
| 339 |
+
Indeed, this is the case:
|
| 340 |
+
|
| 341 |
+
```console
|
| 342 |
+
==================================================
|
| 343 |
+
RESULTS COMPARISON
|
| 344 |
+
==================================================
|
| 345 |
+
Positive examples:
|
| 346 |
+
Mean accuracy: 0.495 ± 0.322
|
| 347 |
+
Processed: 1000 examples
|
| 348 |
+
|
| 349 |
+
Negative examples:
|
| 350 |
+
Mean accuracy: 0.227 ± 0.231
|
| 351 |
+
Processed: 1000 examples
|
| 352 |
+
|
| 353 |
+
Difference (Positive - Negative): 0.267
|
| 354 |
+
T-test: t=21.331, p=0.000
|
| 355 |
+
Difference is statistically significant (p < 0.05)
|
| 356 |
+
```
|
| 357 |
+
|
| 358 |
+
|
| 359 |
+
## Installation
|
| 360 |
+
|
| 361 |
+
1. **Clone the repository:**
|
| 362 |
+
```bash
|
| 363 |
+
git clone https://github.com/Gleghorn-Lab/DSM.git
|
| 364 |
+
cd DSM
|
| 365 |
+
```
|
| 366 |
+
|
| 367 |
+
2. **Initialize the submodules:**
|
| 368 |
+
```bash
|
| 369 |
+
git submodule update --init --remote --recursive
|
| 370 |
+
```
|
| 371 |
+
|
| 372 |
+
3. **Set up the Python virtual environment:**
|
| 373 |
+
The `setup_bioenv.sh` script creates a virtual environment named `bioenv` in your home directory (`~/bioenv`), installs PyTorch with CUDA 12.6 support, and then installs all other dependencies from `requirements.txt`.
|
| 374 |
+
|
| 375 |
+
Make the script executable:
|
| 376 |
+
```bash
|
| 377 |
+
chmod +x setup_bioenv.sh
|
| 378 |
+
```
|
| 379 |
+
Run the script:
|
| 380 |
+
```bash
|
| 381 |
+
./setup_bioenv.sh
|
| 382 |
+
```
|
| 383 |
+
|
| 384 |
+
If you are not on a linux machine, you can install the requirements directly
|
| 385 |
+
```console
|
| 386 |
+
python -m pip install -r requirements.txt
|
| 387 |
+
```
|
| 388 |
+
|
| 389 |
+
4. **Activate the environment:**
|
| 390 |
+
Each time you want to work on this project, activate the virtual environment:
|
| 391 |
+
```bash
|
| 392 |
+
source ~/bioenv/bin/activate
|
| 393 |
+
```
|
| 394 |
+
|
| 395 |
+
5. **To deactivate the environment:**
|
| 396 |
+
```bash
|
| 397 |
+
deactivate
|
| 398 |
+
```
|
| 399 |
+
|
| 400 |
+
All together
|
| 401 |
+
```bash
|
| 402 |
+
git clone https://github.com/Gleghorn-Lab/DSM.git
|
| 403 |
+
cd DSM
|
| 404 |
+
git submodule update --init --remote --recursive
|
| 405 |
+
chmod +x setup_bioenv.sh
|
| 406 |
+
./setup_bioenv.sh
|
| 407 |
+
source ~/bioenv/bin/activate
|
| 408 |
+
```
|
| 409 |
+
|
| 410 |
+
### Logging in
|
| 411 |
+
|
| 412 |
+
Log in once, and the scripts find your credentials themselves:
|
| 413 |
+
|
| 414 |
+
```bash
|
| 415 |
+
hf auth login # Hugging Face; or set HF_TOKEN
|
| 416 |
+
wandb login # Weights & Biases; or set WANDB_API_KEY
|
| 417 |
+
```
|
| 418 |
+
|
| 419 |
+
Training pushes the trained model to the Hugging Face Hub and logs to Weights & Biases, so it needs both. Reading a private model or dataset needs the Hugging Face login.
|
| 420 |
+
|
| 421 |
+
## Training
|
| 422 |
+
|
| 423 |
+
The primary script for training models is `training/train_dsm.py`. This script further pretrains an ESM2 checkpoint using the DSM objective (masked diffusion based on LLaDA) on a large protein sequence dataset like [OMG-prot50](https://huggingface.co/datasets/Synthyra/omg_prot50).
|
| 424 |
+
|
| 425 |
+
### Main Training Script: `train_dsm.py`
|
| 426 |
+
|
| 427 |
+
- **Base Model**: DSM models are extended from pre-trained ESM2 checkpoints (e.g., ESM2-150M, ESM2-650M).
|
| 428 |
+
- **Training Objective**: Masked diffusion loss, where the model predicts masked tokens. The loss is scaled by `1/(t + epsilon)` where `t` is the corruption level, penalizing errors more at low mask rates.
|
| 429 |
+
- **Language Modeling Head**: Uses a modified head with a soft-logit cap (`tau=30`) and tied output projection weights to the token embeddings.
|
| 430 |
+
- **Data Handling**:
|
| 431 |
+
- Training data can be streamed from datasets like [Synthyra/omg_prot50](https://huggingface.co/datasets/Synthyra/omg_prot50) (a version of Open MetaGenomic dataset clustered at 50% identity).
|
| 432 |
+
- Uses `data.dataset_classes.SequenceDatasetFromList` for validation/test sets and `data.dataset_classes.IterableDatasetFromHF` for streaming training.
|
| 433 |
+
- `data.data_collators.SequenceCollator` is used for batching.
|
| 434 |
+
- **Training Process**:
|
| 435 |
+
- Utilizes Hugging Face `TrainingArguments`.
|
| 436 |
+
- A custom `IterableTrainer` (from `training.iterable_trainer.py`) handles iterable datasets.
|
| 437 |
+
- Uses AdamW optimizer and a cosine learning rate scheduler with linear warmup.
|
| 438 |
+
- Supports logging to Weights & Biases (wandb).
|
| 439 |
+
- The trained model can be pushed to Hugging Face Hub.
|
| 440 |
+
- Example checkpoints mentioned in the paper: [DSM-150](https://huggingface.co/GleghornLab/DSM_150) (from ESM2-150M, 100k steps, batch 32, seqlen 512, LR 1e-4) and [DSM-650](https://huggingface.co/GleghornLab/DSM_650) (from ESM2-650M, 100k steps, global batch 128, seqlen 2048, LR 1e-4).
|
| 441 |
+
|
| 442 |
+
**Usage Example:**
|
| 443 |
+
|
| 444 |
+
```bash
|
| 445 |
+
python -m training.train_dsm \
|
| 446 |
+
--model_path facebook/esm2_t33_650M_UR50D \
|
| 447 |
+
--save_path GleghornLab/DSM_650 \
|
| 448 |
+
--lr 1e-4 \
|
| 449 |
+
--batch_size 8 \
|
| 450 |
+
--grad_accum 16 \
|
| 451 |
+
--max_steps 100000 \
|
| 452 |
+
--save_every 1000 \
|
| 453 |
+
--fp16 \
|
| 454 |
+
--wandb_project "DSM_Training"
|
| 455 |
+
```
|
| 456 |
+
|
| 457 |
+
**Key Command-Line Arguments for `train_dsm.py`:**
|
| 458 |
+
|
| 459 |
+
* `--model_path`: Path to the base ESM2 model to start from.
|
| 460 |
+
* `--save_path`: Path to save the trained DSM model on Hugging Face Hub.
|
| 461 |
+
* `--lr`: Learning rate.
|
| 462 |
+
* `--batch_size`: Batch size per device.
|
| 463 |
+
* `--grad_accum`: Gradient accumulation steps.
|
| 464 |
+
* `--max_steps`: Maximum training steps.
|
| 465 |
+
* `--wandb_project`: Wandb project name (default: `DSM`).
|
| 466 |
+
* `--max_length`: Maximum sequence length.
|
| 467 |
+
* `--save_every`: Save model and evaluate every N steps.
|
| 468 |
+
* `--fp16`: Enable mixed-precision training.
|
| 469 |
+
* `--bugfix`: Use small batch size and max length for debugging.
|
| 470 |
+
|
| 471 |
+
### Other Training Scripts (e.g., for DSM-ppi)
|
| 472 |
+
|
| 473 |
+
The `training/` directory may also contain scripts like `train_dsm_bind.py`.
|
| 474 |
+
- DSM-ppi (e.g., [DSM-150-ppi](https://huggingface.co/GleghornLab/DSM_150_ppi_lora), [DSM-650-ppi](https://huggingface.co/GleghornLab/DSM_650_ppi_lora)) is fine-tuned on PPI datasets.
|
| 475 |
+
- Training involves conditioning on a target sequence (SeqA) to generate an interactor (SeqB) using the format `[CLS]--SeqA--[EOS]--[MASKED~SeqB]--[EOS]`.
|
| 476 |
+
- LoRA (Low-Rank Adaptation) can be applied to attention layers for efficient fine-tuning.
|
| 477 |
+
|
| 478 |
+
And `training/iterable_trainer.py` provides the `get_iterable_trainer` function used by `train_dsm.py` to enable training with iterable datasets.
|
| 479 |
+
|
| 480 |
+
## Evaluation
|
| 481 |
+
|
| 482 |
+
The repository includes a comprehensive suite for evaluating model performance, focusing on:
|
| 483 |
+
|
| 484 |
+
1. **Sequence Reconstruction (Mask Filling):**
|
| 485 |
+
* Evaluated by masking validation/test sets at various corruption rates (5% to 90%) and measuring cross-entropy loss, weighted F1 score, and Alignment Score (ASc) for the masked positions.
|
| 486 |
+
* The script `evaluation/mask_filling.py` is central to this.
|
| 487 |
+
|
| 488 |
+
2. **Unconditional Generation Quality:**
|
| 489 |
+
* Generate a corpus of sequences based on lengths from a reference set (e.g., validation data).
|
| 490 |
+
* Compare distributions (1-mers, 2-mers, 3-mers) of amino acids and predicted secondary structures between generated and natural sequences using χ² test and Jensen-Shannon (JS) divergence.
|
| 491 |
+
* Compare distributions of predicted functional annotations (e.g., using Annotation Vocabulary - AV terms).
|
| 492 |
+
* Scripts involved: `evaluation/unconditional_generation_tuning.py` (to find optimal generation parameters like temperature and step divisor `s`), `evaluation/unconditional_generation.py`, `evaluation/ss_pred.py` (using [production_ss4_model](https://huggingface.co/GleghornLab/production_ss4_model) or [production_ss9_model](https://huggingface.co/GleghornLab/production_ss9_model)), `evaluation/annotate_comparisons.py`, `evaluation/compare_distributions.py`, `evaluation/plot_distribution_comparisons.py`.
|
| 493 |
+
* The `run_eval_pipeline.py` script automates this workflow.
|
| 494 |
+
|
| 495 |
+
3. **Representation Quality (Model Probing):**
|
| 496 |
+
* Evaluate learned embeddings by training linear probes (or simple transformer blocks) on various downstream tasks (e.g., secondary structure prediction, localization prediction, etc.).
|
| 497 |
+
* Performance is compared against random vectors, randomized transformers, and other established pLMs.
|
| 498 |
+
* The assessment was done with [Protify](https://github.com/Synthyra/Protify), an open-source framework that can be used for pLM training and evaluation.
|
| 499 |
+
|
| 500 |
+
4. **Conditional Generation (Binder Design for DSM-ppi):**
|
| 501 |
+
* Evaluate DSM-ppi on benchmarks like BenchBB.
|
| 502 |
+
* Generate binders for target proteins using template-based masking strategies.
|
| 503 |
+
* Assess generated binders using *in-silico* tools like Synteract2 for predicted binding affinity (ppKd).
|
| 504 |
+
|
| 505 |
+
The `evaluation/` directory also contains a `readme.md` which provides further details on some evaluation workflows. Key metrics used include:
|
| 506 |
+
- **Alignment Score (ASc):** A normalized Needleman-Wunsch global alignment score (using BLOSUM62) to measure sequence similarity, robust to length variations. ASc(a, b) = l/(f(a, a) - f(a, b) + l).
|
| 507 |
+
- **Jensen-Shannon (JS) Divergence:** To compare distributions of k-mers and functional terms.
|
| 508 |
+
|
| 509 |
+
Functional annotations (`evaluation/annotate_comparisons.py`) and binding-affinity predictions (`evaluation/conditional_binder.py`, `evaluation/unconditional_binder.py`) come from the Synthyra API, and the scripts read its key from the `SYNTHYRA_API_KEY` environment variable. The pipeline below needs it for its annotation step. The binder scripts' `--test` flag substitutes random scores and needs no key.
|
| 510 |
+
|
| 511 |
+
**Running the Full Unconditional Evaluation Pipeline:**
|
| 512 |
+
|
| 513 |
+
```bash
|
| 514 |
+
python run_eval_pipeline.py --data_dir ./evaluation_results
|
| 515 |
+
```
|
| 516 |
+
|
| 517 |
+
Refer to `run_eval_pipeline.py --help` for more options, such as `--skip_tuning`.
|
| 518 |
+
|
| 519 |
+
### Mask Filling Evaluation
|
| 520 |
+
|
| 521 |
+
The script `evaluation/mask_filling.py` is used to evaluate models on their ability to predict masked tokens in a sequence across various masking rates.
|
| 522 |
+
|
| 523 |
+
- **Functionality:**
|
| 524 |
+
- Evaluates different models (DSM, DPLM, standard ESM models).
|
| 525 |
+
- Tests across multiple datasets ([Synthyra/omg_prot50](https://huggingface.co/datasets/Synthyra/omg_prot50), [GleghornLab/stringv12_modelorgs_9090](https://huggingface.co/datasets/GleghornLab/stringv12_modelorgs_9090)).
|
| 526 |
+
- Calculates metrics: loss, perplexity, precision, recall, F1, accuracy, MCC, and alignment score.
|
| 527 |
+
- Saves detailed results to CSV files.
|
| 528 |
+
- Can generate a summary plot comparing model performance across different mask rates using `evaluation/plot_mask_fill_results.py`.
|
| 529 |
+
|
| 530 |
+
- **Usage Example:**
|
| 531 |
+
```bash
|
| 532 |
+
python -m evaluation.mask_filling \
|
| 533 |
+
--batch_size 4 \
|
| 534 |
+
--mask_rates 0.15 0.30 0.50 \
|
| 535 |
+
--data_splits valid test \
|
| 536 |
+
--results_dir ./results/mask_fill_custom
|
| 537 |
+
```
|
| 538 |
+
To generate a comparison plot from existing results:
|
| 539 |
+
```bash
|
| 540 |
+
python -m evaluation.mask_filling --generate_comparison_plot --results_dir ./results/mask_fill_custom --plot_output ./results/mask_fill_custom/comparison.png
|
| 541 |
+
```
|
| 542 |
+
|
| 543 |
+
### Other Evaluation Scripts
|
| 544 |
+
|
| 545 |
+
The `evaluation/` directory contains additional scripts for more specific analyses. These are typically run independently:
|
| 546 |
+
|
| 547 |
+
- `evaluation/all_targets_uncond.py` and `evaluation/all_targets_cond.py`: Likely for evaluating generation towards specific targets, unconditionally and conditionally.
|
| 548 |
+
- `evaluation/conditional_binder.py` and `evaluation/unconditional_binder.py`: Suggest evaluation focused on generating protein binders.
|
| 549 |
+
- `evaluation/unconditional_by_length.py`: May evaluate unconditional generation focusing on sequence length distributions.
|
| 550 |
+
- `evaluation/utils.py`: Utility functions for evaluation scripts.
|
| 551 |
+
|
| 552 |
+
Users should refer to individual scripts (e.g., using `python -m evaluation.<script_name> --help`) for their specific usage and arguments.
|
| 553 |
+
The `evaluation/` directory also contains a `readme.md` which provides further details on the unconditional generation evaluation workflow.
|
| 554 |
+
|
| 555 |
+
## Results
|
| 556 |
+
|
| 557 |
+
DSM demonstrates strong performance in both protein sequence generation and representation learning, establishing masked diffusion as a powerful paradigm.
|
| 558 |
+
|
| 559 |
+
- **Biomimetic Sequence Generation**: Unconditionally generated DSM sequences closely mimic natural protein distributions in terms of amino acid k-mers, predicted secondary structures (JS divergence < 0.01 for AA k-mers), and predicted functional annotations (AV terms, JS divergence ~0.1). This suggests DSM captures underlying biological principles.
|
| 560 |
+
|
| 561 |
+
- **Superior Sequence Reconstruction**: DSM models significantly outperform MLM-based ESM2 models in reconstructing sequences from highly corrupted inputs (up to 90% masking).
|
| 562 |
+
- At 90% masking, DSM achieves an Alignment Score (ASc) of ~0.27, considerably higher than random.
|
| 563 |
+
- DSM models show higher F1 scores in reconstruction tasks compared to DPLM models, especially at high mask rates.
|
| 564 |
+
|
| 565 |
+
- **High-Quality Embeddings**: DSM embeddings match or exceed the quality of those from comparably sized pLMs (ESM2, DPLM) and even larger autoregressive models (ProtCLM 1B) on various downstream tasks evaluated by linear probing. [DSM-650](https://huggingface.co/GleghornLab/DSM_650) generally provides the best representations among tested models of similar size.
|
| 566 |
+
|
| 567 |
+
- **Effective Binder Design (DSM-ppi):**
|
| 568 |
+
- DSM-ppi fine-tuned on protein-protein interaction data, demonstrates the ability to generate protein binders conditioned on target sequences.
|
| 569 |
+
- On the BenchBB benchmark, DSM-generated binders (both unconditional DSM and conditional DSM-ppi) show promising predicted binding affinities, in some cases superior to known binders. For example, designs for EGFR showed high predicted pKd and good structural metrics (ipTM, pTM with AlphaFold3).
|
| 570 |
+
|
| 571 |
+
- **Efficiency**: DSM can generate realistic protein sequences from a single forward pass during reconstruction tasks at high mask rates, offering potential efficiency advantages over iterative AR or some discrete diffusion models.
|
| 572 |
+
|
| 573 |
+
These results highlight DSM's capability to unify high-quality protein representation learning and biologically coherent generative modeling within a single framework.
|
| 574 |
+
|
| 575 |
+
## Experimental validation
|
| 576 |
+
|
| 577 |
+
We validated various DSM generated binders using biolayer interferometry through [Adaptyv Bio](https://www.adaptyvbio.com/) - sending in 20 designs for EGFR and PD-L1. Sequences generated via unconditional generation were sorted hierarchically by predicted binding affinity (Synteract2), then ESMfold pLDDT, ESM2 PLL, and finally Chai1 iPTM.
|
| 578 |
+
|
| 579 |
+
### EGFR
|
| 580 |
+
|
| 581 |
+
Of the 13 expressed, 12 designs bound to EGFR with 11 of them binding strongly. Noteably, the top design, `dsm_egfr_10`, presented a mean KD in the picomolar range (861 pM), which is a ~30% increase in binding affinity vs. the winner (and our starting template) of the Adaptyv EGFR competition at 1.21 nM, and ~90% over the original starting scFV Cetuximab at 664 nM.
|
| 582 |
+
|
| 583 |
+
<img width="1593" height="527" alt="image" src="https://github.com/user-attachments/assets/f6f6a614-d5f4-4e9e-b7c9-fc0fdf9dc1e2" />
|
| 584 |
+
|
| 585 |
+
|
| 586 |
+
**dsm_egfr_10**
|
| 587 |
+
```
|
| 588 |
+
QVQLQQSGPGLVQPSQSLSITCTVSGFSLTNYGVHWVRQSPGKGLEWLGVIWSGGNTDYNTPFTSRLSISRDTSKSQVFFKMNSLQTDDTAVYYCARALTYYDYEFAYWGQGTLVTVSAGGGGSGGGGSGGGGSDILLTQSPVILSVSPGERVSFSCRASQSIGSNIHWYQQRTNGSPKLLIRYASESISGIPSRFSGSGSGTDFTLSINSVDPEDIADYYCQQNNNWPTTFGAGTKLEIK
|
| 589 |
+
```
|
| 590 |
+
* KD - 861 pM
|
| 591 |
+
* pKD - 9.06
|
| 592 |
+
* PPI probability (Synteract2) - 0.9991
|
| 593 |
+
* predicted pKD (Synteract2) - 8.94
|
| 594 |
+
* mask rate for DSM - 2%
|
| 595 |
+
* mutations from template - 3
|
| 596 |
+
* mutations - I92V, T165S, L240I
|
| 597 |
+
* average esm2 pll - 1.23
|
| 598 |
+
* esmfold plddt - 0.75
|
| 599 |
+
* pTM (AlphaFold3) - 0.81
|
| 600 |
+
* ipTM (AlphaFold3) - 0.91
|
| 601 |
+
|
| 602 |
+
<table>
|
| 603 |
+
<tr>
|
| 604 |
+
<td>
|
| 605 |
+
<img width="596" height="355" alt="image" src="https://github.com/user-attachments/assets/0c6f690d-3134-44d2-9540-12c277e187b3" />
|
| 606 |
+
</td>
|
| 607 |
+
<td>
|
| 608 |
+
<img src="https://github.com/Gleghorn-Lab/DSM/blob/main/wetlab_result_analysis/egfr/kinetics/dsm_egfr_10_2.png" width="400">
|
| 609 |
+
</td>
|
| 610 |
+
</tr>
|
| 611 |
+
</table>
|
| 612 |
+
|
| 613 |
+
This sequence would have won the EGFR competition by a wide margin! Of course, we piggybacked on the winning entry as our template. We have been using DSM-PPI-full to attempt to replicate competition winning binders from the Cetuximab starting point instead. Stay tuned!
|
| 614 |
+
|
| 615 |
+
<img width="1865" height="548" alt="image" src="https://github.com/user-attachments/assets/ce486326-f4ba-4604-af47-259f7bbe496f" />
|
| 616 |
+
|
| 617 |
+
|
| 618 |
+
### PD-L1
|
| 619 |
+
|
| 620 |
+
All 20 PD-L1 designs had high expression rates, with 15/20 binding. 1 weak, 10 medium, and 3 strong. The strongest presented with an average KD of 8.06 nM (pKD) , which is markedly less than the original template at 0.8 pM. We attribute the consistent binding but worse performance overall to the higher error between Synteract2 ppKD and true pKD of the template, implying it is not modeled well by our affinity system.
|
| 621 |
+
|
| 622 |
+
<img width="1592" height="516" alt="image" src="https://github.com/user-attachments/assets/6d2cde0e-75a4-4f29-999f-a8c601286845" />
|
| 623 |
+
|
| 624 |
+
<img src="https://github.com/Gleghorn-Lab/DSM/blob/main/wetlab_result_analysis/pdl1/kinetics/dsm_pdl1_7_1.png" width="400">
|
| 625 |
+
|
| 626 |
+
## Cite
|
| 627 |
+
```
|
| 628 |
+
@misc{hallee2025diffusionsequencemodelsenhanced,
|
| 629 |
+
title={Diffusion Sequence Models for Enhanced Protein Representation and Generation},
|
| 630 |
+
author={Logan Hallee and Nikolaos Rafailidis and David B. Bichara and Jason P. Gleghorn},
|
| 631 |
+
year={2025},
|
| 632 |
+
eprint={2506.08293},
|
| 633 |
+
archivePrefix={arXiv},
|
| 634 |
+
primaryClass={q-bio.BM},
|
| 635 |
+
url={https://arxiv.org/abs/2506.08293},
|
| 636 |
+
}
|
| 637 |
+
```
|