20eddibae commited on
Commit
54020f6
·
verified ·
1 Parent(s): 3cc8bf6

Update card

Browse files
Files changed (1) hide show
  1. README.md +127 -133
README.md CHANGED
@@ -13,156 +13,150 @@ pipeline_tag: feature-extraction
13
 
14
  # CLEST-TCRdist
15
 
16
- **C**ontrastive **LE**arning with **S**oft **T**argets from **TCRdist**.
17
- Checkpoint `SCEPTR6L_AC_soft_800k_20ep_bs1024`.
18
-
19
- A paired-chain αβ **T-cell receptor (TCR) language model** that maps a TCR to a
20
- 128-dimensional vector such that TCRs binding the same epitope land close
21
- together. It is built on
22
- [SCEPTR](https://github.com/yutanagano/sceptr) (Nagano et al. 2025, *Cell
23
- Systems*) and changes exactly one term of its training objective: the
24
- autocontrastive target, which treats every other receptor in the batch as a
25
- negative, is replaced with a graded target `exp(-d_TCRdist/τ)` computed in
26
- batch (τ = 50). Architecture, corpus, augmentation and compute are held fixed
27
- against the autocontrastive control, so the difference between them is
28
- attributable to that term.
29
-
30
- ### Two things to read before quoting a number
31
-
32
- - **This is one training run, not a mean.** The checkpoint predates seeding in
33
- our pipeline. Three seeded replicates of the identical configuration score
34
- 0.809 ± 0.006 prototype AUROC@50 and 0.774 ± 0.009 at nearest-neighbour, where
35
- this run scores 0.842 and 0.794 — it sits above all three in every cell. Treat
36
- the replicate means as the recipe's expected performance.
37
- - **What the objective is actually worth.** Seeding both arms (3 soft-label
38
- replicates, 2 autocontrastive controls, scored in one run on shared reference
39
- draws) puts the gap over the matched control at **+0.038 AUROC@50** at
40
- nearest-neighbour — six times the pooled seed spread, positive on five of six
41
- epitopes — down from the +0.058 the two published runs suggested. The control
42
- arm reproduces its published run almost exactly (0.7356/0.7366 vs 0.7361), so
43
- this is a real shrinkage and not environment drift.
44
- - **It does not beat TCRdist on accuracy, and we do not claim it does.** Over
45
- seeds the 1-NN margin changes sign. What the objective buys is measured
46
- against its own autocontrastive control, and a scoring-rule inversion that is
47
- itself conditional on the evaluation repertoire (see the paper).
48
-
49
- - **Architecture:** 6-layer Pre-LN / NormFormer transformer encoder,
50
- d_model = 128, 8 heads, `CdrSimpleEmbedder` (embedding_dim = 29), CLS-token
51
- readout. ~1.2 M parameters.
52
- - **Output:** L2-comparable 128-d embedding per TCR
53
- (`calc_vector_representations`); pairwise distances via
54
- `calc_cdist_matrix` / `calc_pdist_vector`.
55
- - **Inference is epitope-free:** the model sees only the TCR, never the epitope.
56
-
57
- ## How to use
58
 
59
  ```python
60
- from scaled_sceptr.hf_hub import load_champion
 
61
 
62
- model = load_champion() # downloads ~5 MB on first call
63
- # df: columns TRAV, CDR3A, TRAJ, TRBV, CDR3B, TRBJ
64
- vecs = model.calc_vector_representations(df) # (N, 128) float32
65
- cdist = model.calc_cdist_matrix(df_a, df_b) # (Na, Nb) distances
 
 
 
 
66
  ```
67
 
68
- Requires the study package installed (local editable `libtcrlm` + `sceptr`,
69
- see the repo README "Installation"). The encoder class is registered into
70
- `libtcrlm` automatically by `load_champion`.
71
-
72
- ## Application: few-shot epitope-specificity retrieval
73
-
74
- This is the model's headline use case. Give it **k known binders** for an
75
- epitope (the reference set) and it scores **any query TCR** for how strongly
76
- it is specific to that epitope — no epitope sequence and no classifier
77
- retraining required. It is the epitope-free, learned analogue of TCRMatch and
78
- tcrdist3 meta-clonotypes. On the paper's fourteen-method slate this
79
- checkpoint ranks first (NN AUROC@50 = 0.801, prototype AUROC@50 = 0.860), but
80
- that ranking should be read with two controls attached: the checkpoint was
81
- selected on the same epitope panel it is scored on (moving selection to six
82
- held-out epitopes costs 0.013), and it is a single unseeded run that sits above
83
- all three of its seeded replicates. The margin over TCRdist does not survive
84
- either control, and does not survive giving TCRdist its own published read-out
85
- either (Dash et al. score the nearest 10% of the reference set, not the nearest
86
- one; under each method's own best rule the comparison is a draw, +0.010,
87
- p = 0.42). The durable comparison is against the matched autocontrastive
88
- control. Note that the prototype-over-nearest-neighbour gain is **not** specific
89
- to this objective: once the control is reseeded it gains +0.041 from prototype
90
- scoring against this model's +0.034, so that inversion is a property of the
91
- evaluation repertoire rather than of the training target.
92
 
93
  ```python
94
- import numpy as np
95
- from scaled_sceptr.hf_hub import load_champion
96
 
97
- model = load_champion()
 
 
 
98
 
99
- # reference = k known binders for the epitope of interest
100
- # query = TCRs you want to score (a repertoire, a screen, etc.)
101
- # both are DataFrames with columns TRAV, CDR3A, TRAJ, TRBV, CDR3B, TRBJ
102
- cdist = model.calc_cdist_matrix(query, reference) # (n_query, k)
103
 
104
- # Prototype scoring (recommended — averages the whole reference cloud into a
105
- # centroid; this is where the soft-label geometry pays off). Higher = more
106
- # specific to the epitope.
107
- proto_score = -cdist.mean(axis=1) # (n_query,)
108
 
109
- # Nearest-neighbour scoring (single closest binder) — also strong, use when
110
- # the reference set is heterogeneous / multi-clonal.
111
- nn_score = -cdist.min(axis=1) # (n_query,)
112
 
113
- ranked = np.argsort(-proto_score) # most-specific queries first
 
 
 
114
  ```
115
 
116
- The score is a relative ranking, not a calibrated probability. To get a
117
- binary call, pick a threshold on a held-out labeled set, or rank a repertoire
118
- and take the top hits. Prototype scoring is the mode to prefer whenever the k
119
- reference binders are known to share the epitope; it is what produces the
120
- model's largest, statistically significant advantage over TCRdist.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
121
 
122
  ## Training
123
 
124
- | | |
125
- |---|---|
126
- | **Objective** | Autocontrastive (AC) + MLM. Positives = two augmented views of the *same* TCR (token dropout 0.2, chain dropout 0.5); the InfoNCE matrix is additionally weighted by **in-batch TCRdist soft labels** (`exp(−d_TCRdist/τ)`, τ = 50, distance threshold 200). |
127
- | **Contrastive loss** | dot-product InfoNCE, temperature 0.05 |
128
- | **MLM loss** | adjusted cross-entropy, label smoothing 0.1 |
129
- | **Data** | ~828k **unlabeled** paired αβ TCRs (Tanno et al. 2022, Zenodo 14288119). No epitope labels are used in pretraining. |
130
- | **Schedule** | batch size 1024, 20 epochs, Noam warmup 2000 steps |
131
- | **Checkpoint** | epoch 20 (lowest `valid_cont_loss` = 1.978); selected automatically |
132
-
133
- This is the **autocontrastive** regime — the protocol that matches the SCEPTR
134
- paper. It does **not** use epitope labels, so its benchmark AUROC is a genuine
135
- zero-shot number (unlike the supervised-contrastive "clean" variants elsewhere
136
- in the study, which see the benchmark epitopes during training and report
137
- inflated ~0.97 AUROC).
138
-
139
- ## Evaluation
140
-
141
- Few-shot epitope retrieval on the SCEPTR paper protocol (raw VDJdb, 6 epitopes,
142
- 1-NN scoring, `--max-neg 5000`, 100 repeats). AUROC, equal-weighted across
143
- epitopes:
144
-
145
- | Model | AUROC@50 | AUROC@200 |
146
- |---|---|---|
147
- | **CLEST-TCRdist (this model)** | **0.7945** | **0.8138** |
148
- | TCRdist (reference) | 0.7787 | 0.8020 |
149
- | SCEPTR (base AC, published) | 0.7360 | 0.7459 |
150
- | CDR3 Levenshtein | 0.6989 | — |
151
-
152
- The headline number reproduces exactly (`0.79448`) from the published
153
- checkpoint with the command in the repo README.
154
-
155
- **Scope / caveats.**
156
- - This is a **TCR-to-TCR similarity** model, not an epitope classifier. It
157
- predicts binding only by retrieval against a labeled reference set.
158
- - The scaling study found this recipe is **plateaued**: deeper (12L), wider
159
- (d=256), modern-norm (RMSNorm/SwiGLU/QK-Norm), and larger-corpus variants all
160
- tie or lose to this 6L Tanno-only recipe.
161
- - ~8.6% of Tanno CDR3β sequences also appear in VDJdb — the field-standard
162
- background contamination shared by TCRdist and the published SCEPTR baseline
163
- alike; a CDR3β-clean leak-free benchmark preserves the same small advantage.
164
 
165
  ## Citation
166
 
167
- Built on SCEPTR (Nagano et al. 2025, *Cell Systems*) and TCRdist
168
- (Dash et al. 2017, *Nature*). Pretraining data: Tanno et al. 2022.
 
 
 
 
13
 
14
  # CLEST-TCRdist
15
 
16
+ Contrastive LEarning with Soft Targets from TCRdist. Checkpoint
17
+ `SCEPTR6L_AC_soft_800k_20ep_bs1024`.
18
+
19
+ A paired-chain αβ T-cell receptor encoder. It maps a TCR to a 128-dimensional vector in
20
+ which receptors that recognise the same peptide sit close together, so retrieval and
21
+ clustering of specificity groups reduce to nearest-neighbour search. The model reads only
22
+ the receptor, never the epitope.
23
+
24
+ It is built on [SCEPTR](https://github.com/yutanagano/sceptr) (Nagano et al. 2025, *Cell
25
+ Systems*) and changes one term of its objective. Autocontrastive pretraining treats every
26
+ other receptor in a batch as a negative. Here each in-batch pair gets a graded target
27
+ `exp(-d_TCRdist / 50)`, zero beyond a TCRdist of 200, computed on the fly with tcrdist3.
28
+ No epitope labels are involved, and nothing is precomputed. Architecture, corpus,
29
+ augmentation and batch size are held fixed against an autocontrastive control trained the
30
+ same way, which is what makes the comparison between them a single-variable one.
31
+
32
+ ## Install
33
+
34
+ ```bash
35
+ pip install git+https://github.com/20eddibae/CLEST-TCRDist.git
36
+ pip install --no-deps sceptr==1.2.0
37
+ ```
38
+
39
+ sceptr goes in with `--no-deps` because its metadata asks for numpy 2 while tcrdist3 needs
40
+ 1.26. Everything it actually imports is already installed by the line above.
41
+
42
+ ## Use it
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
43
 
44
  ```python
45
+ import pandas as pd
46
+ from clest_tcrdist import load_pretrained
47
 
48
+ model = load_pretrained() # pulls ~5 MB here, then caches
49
+
50
+ tcrs = pd.DataFrame({
51
+ "TRAV": ["TRAV35*01"], "CDR3A": ["CAGQLGGGSQGNLIF"], "TRAJ": ["TRAJ42*01"],
52
+ "TRBV": ["TRBV19*01"], "CDR3B": ["CASSIRSSYEQYF"], "TRBJ": ["TRBJ2-7*01"],
53
+ })
54
+ vectors = model.calc_vector_representations(tcrs) # (N, 128)
55
+ distances = model.calc_cdist_matrix(tcrs, tcrs) # (N, N)
56
  ```
57
 
58
+ The returned object is a `sceptr.model.Sceptr`, so anything written against the SCEPTR API
59
+ works unchanged. Columns are IMGT symbols and amino-acid junctions. A missing chain or gene
60
+ can be left empty, and single-chain input is accepted.
61
+
62
+ ### Few-shot retrieval
63
+
64
+ Given a few known binders for one epitope, rank a repertoire:
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
65
 
66
  ```python
67
+ from clest_tcrdist import read_tcrs, retrieve
 
68
 
69
+ repertoire = read_tcrs("repertoire.csv") # any common column naming
70
+ seeds = read_tcrs("known_binders.csv")
71
+ ranked = retrieve(repertoire, seeds, model=model) # sorted best first
72
+ ```
73
 
74
+ or from the shell:
 
 
 
75
 
76
+ ```bash
77
+ python -m clest_tcrdist.retrieve repertoire.csv --seeds known_binders.csv -o ranked.csv
78
+ ```
 
79
 
80
+ Scoring is distance to the mean seed embedding by default, or to the nearest single seed
81
+ with `--scoring nearest`. Averaging over the seeds is worth about 0.05 AUROC to this model
82
+ and is what the headline number below uses.
83
 
84
+ ### Clustering
85
+
86
+ ```bash
87
+ python -m clest_tcrdist.cluster repertoire.csv -o clustered.csv
88
  ```
89
 
90
+ Pairs within a radius are linked and connected components become clusters, the same
91
+ construction GIANA and TCRdist use, with the radius measured in this embedding. The radius
92
+ is calibrated from the data unless you set `--threshold`, and it controls a purity against
93
+ retention tradeoff: on 120 TCRs across four epitopes, purity is 1.00 out to 20% retention,
94
+ then 0.88 at 30% and 0.74 at 40%. `--sweep` prints that curve for a labelled file. Compare
95
+ clustering methods at matched retention, never at matched radius.
96
+
97
+ ## How it scores
98
+
99
+ Few-shot retrieval on the six-epitope VDJdb backtest of the SCEPTR paper, 100 resamplings
100
+ of the reference set, mean AUROC over epitopes, nearest-neighbour scoring:
101
+
102
+ | Reference TCRs | CLEST-TCRdist | TCRdist | SCEPTR (released) | CDR3 Levenshtein |
103
+ |---|---|---|---|---|
104
+ | 1 | 0.701 | 0.684 | 0.643 | 0.621 |
105
+ | 10 | 0.765 | 0.756 | 0.710 | 0.687 |
106
+ | 50 | 0.794 | 0.779 | 0.737 | 0.699 |
107
+ | 200 | 0.814 | 0.802 | 0.754 | 0.708 |
108
+
109
+ ## Read this before quoting a number
110
+
111
+ **These weights are one training run, not an average.** The checkpoint predates seeding in
112
+ our pipeline. Five seeded replicates of the identical configuration average 0.7750 ± 0.0076
113
+ at nearest-neighbour and 0.8104 ± 0.0053 under average-distance scoring, where this run
114
+ gives 0.7945 and 0.8423. It sits above all five in every cell, 2.5 seed standard deviations
115
+ out at nearest-neighbour and 6.0 under averaging. Treat the replicate means as what the
116
+ recipe gives you, and this checkpoint as a lucky draw from it. All five replicates are in
117
+ the paper's supplement.
118
+
119
+ **What the objective is worth is measured against its own control, not against TCRdist.**
120
+ With the full set of replicates the soft-label target buys +0.039 AUROC@50 at
121
+ nearest-neighbour over the matched autocontrastive control (paired over six epitopes,
122
+ p = 0.043, positive on five) and +0.048 under averaging (p = 0.119). Across estimators the
123
+ range is +0.038 to +0.056. That single-variable comparison is the contribution.
124
+
125
+ **It does not reliably beat TCRdist, and we do not claim it does.** The margin in the table
126
+ above does not survive two controls: reselecting hyperparameters on held-out epitopes costs
127
+ about 0.013, and giving TCRdist the read-out from its own publication rather than the 1-NN
128
+ rule inherited here closes the rest. Over seeds the sign of the 1-NN margin changes. What
129
+ holds up is the gap over the control.
130
+
131
+ **It is not an epitope-binding classifier.** There is no peptide or MHC input. It tells you
132
+ which receptors resemble each other in a space shaped by specificity; it cannot tell you
133
+ that a receptor binds a peptide you have not given it examples of.
134
 
135
  ## Training
136
 
137
+ - 6-layer Pre-LN transformer with NormFormer norms, d_model 128, 8 heads, dropout 0.1,
138
+ `CdrSimpleEmbedder` (29-dim token embedding), CLS readout, 1,203,860 parameters.
139
+ - 827,802 unlabeled paired αβ TCRs from Tanno et al. 2022
140
+ ([Zenodo 14288119](https://doi.org/10.5281/zenodo.14288119)). No epitope labels anywhere
141
+ in pretraining.
142
+ - 20 epochs, batch 1024, Noam schedule with 2000 warmup steps, contrastive temperature
143
+ 0.05, masked-token auxiliary loss with 0.1 label smoothing.
144
+ - Shipped epoch is the one with the lowest validation contrastive loss, epoch 20. The
145
+ repository contains `config.json` and `log.csv` for the whole run.
146
+
147
+ ## Also here
148
+
149
+ - [Fine-tuned variant](https://huggingface.co/20eddibae/sceptr-ac-finetuned), for when you
150
+ have epitope labels to train on.
151
+ - [Benchmark data](https://huggingface.co/datasets/20eddibae/sceptr-tcr-benchmark), the
152
+ VDJdb snapshot these numbers come from.
153
+ - [Code](https://github.com/20eddibae/CLEST-TCRDist), including the training configs and
154
+ the benchmark that produced the table.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
155
 
156
  ## Citation
157
 
158
+ The manuscript is in preparation; cite the repository until it appears. Please also cite
159
+ SCEPTR (Nagano et al., *Cell Systems* 2025), TCRdist (Dash et al., *Nature* 2017), the
160
+ Tanno et al. 2022 paired repertoire, and VDJdb (Bagaev et al. 2020).
161
+
162
+ MIT licensed.