| # Supplementary Data — ProtTale |
|
|
| Source data for the figures and tables in the main text and Extended Data: |
| six CSV files (UTF-8, comma-separated, first row is the header), one Excel |
| workbook and one JSON metadata file. |
|
|
| The reliability score is written `p_JAY` throughout, the predicted probability |
| of a Judge-Assigned *Yes*, as in the paper. |
|
|
| Protein identifiers appear in two forms throughout: `UniProt_ID` is the |
| Swiss-Prot entry name (e.g. `ENG1_RHIMI`) and `UniProt_accession` is the |
| Swiss-Prot accession (e.g. `A0A023I7E1`). Entry names are unique across |
| Swiss-Prot, so the two map one to one. |
|
|
| | File | Rows | Covers | |
| |---|---|---| |
| | `Supplementary_Data_1_expert_curation.csv` | 232 | Blinded expert curation of ProtTale predictions | |
| | `Supplementary_Data_2_unseen_per_method.csv` | 6,186 | Unseen evaluation set, per protein, per method, per judge/extraction run | |
| | `Supplementary_Data_3_stage1_test_pairs.csv` | 534 | Stage 1 test pairs used to validate the LLM-as-judge | |
| | `Supplementary_Data_4_shuffled_baseline.csv` | 1,031 | Reliability scores for real versus residue-shuffled sequences | |
| | `Supplementary_Data_5_mips_validation.xlsx` | 1,000 | Sequence, structure and EC agreement within MIPS clusters | |
| | `Supplementary_Data_6_training_curves.csv` | 8,904 | Training and validation curves for both stages and the reliability head | |
| | `Supplementary_Data_7_ablation_no_stage1.csv` | 1,031 | ProtTale retrained without Stage 1, scored on the unseen set | |
|
|
| --- |
|
|
| ## Supplementary Data 1 — expert curation |
|
|
| One row per curated prediction, 232 in total, in the order the predictions were |
| shown to the curator. Sampling, blinding and the rating scale are described in |
| Methods, "Expert Curation". |
|
|
| | Column | Meaning | |
| |---|---| |
| | `Entry` | Identifier shown to the curator, 1–232 | |
| | `UniProt_ID` | Swiss-Prot entry name | |
| | `Length_aa` | Sequence length in residues | |
| | `p_JAY` | Reliability score from ProtTale's head | |
| | `Retained` | Whether the prediction passes the discrete filter `p_JAY > 0.537` | |
| | `Judge_label` | LLM-as-judge verdict, `Yes` or `not-Yes`, majority over five runs | |
| | `Curator_score` | Expert rating, 1 (unrelated or contradictory) to 5 (equivalent) | |
| | `Specificity` | Prediction more / equally / less specific than the Swiss-Prot description | |
| | `Information` | Prediction more / equally / less informative than the Swiss-Prot description | |
| | `Truncated` | Whether generation reached the 128-token cap without emitting an end-of-sequence token (82 of the 232 curated predictions; 233 of the 1,031 unseen predictions, 22.6%) | |
| | `SwissProt_description` | Reference text shown to the curator | |
| | `ProtTale_prediction` | Generated text shown to the curator | |
| | `Curator_notes` | Free-text justification, present for 175 of the 232 items | |
|
|
| Neither `p_JAY` nor `Judge_label` was visible to the curator at scoring time. |
| Both are included so that human and automated ratings can be compared directly. |
|
|
| Regenerates the expert-curation results in the main text and the Extended Data |
| judge-versus-curator agreement table. |
|
|
| --- |
|
|
| ## Supplementary Data 2 — unseen evaluation set, per method |
|
|
| One row per (protein, method), 1,031 proteins x 6 methods = 6,186 rows. The |
| sixth "method" is `Reference`, the curated Swiss-Prot description put through |
| the same GO-extraction pipeline; it is the curated-text reference line in the |
| GO-extraction figure and carries no judge labels. |
|
|
| | Column | Meaning | |
| |---|---| |
| | `UniProt_ID`, `UniProt_accession` | Protein identifiers | |
| | `Method` | `ESM2Text`, `Prot2Text`, `ProtT3`, `ProtTale`, `Protein2Text`, or `Reference` | |
| | `Kingdom` | Taxonomic group, `Eukaryota` / `Bacteria` / `Archaea` / `Viruses` | |
| | `Length_aa` | Sequence length. The short subset is `<= 1024` (n = 852), the long subset `> 1024` (n = 179) | |
| | `SwissProt_description` | The protein's curated Swiss-Prot `FUNCTION` description, the reference the judge compares against. Identical on all six rows of a protein | |
| | `Prediction` | The description this method generated for the protein, i.e. the text the judge scored; the curated description itself on `Reference` rows. Together with the judge prompt (Supplementary Note 2), this allows every judge label to be re-scored | |
| | `judge_run1` … `judge_run5` | LLM-as-judge label from each of the five independent runs: `Yes`, `No`, `Maybe`, or `Vague`. Blank for `Reference` | |
| | `judge_majority` | `Yes` when at least three of the five runs said `Yes`, otherwise `not-Yes`. This is the label used for every statistical comparison in the paper | |
| | `reference_GO` | The protein's curated, experimentally validated Molecular Function GO terms, semicolon separated. The target every extraction is scored against. Identical on all six rows of a protein | |
| | `extracted_GO_run1` … `extracted_GO_run5` | GO terms the extraction model returned from this method's text in each run, semicolon separated. **A blank cell is an empty extraction** | |
| | `wang_run1` … `wang_run5` | Wang semantic similarity between `extracted_GO_run*` and `reference_GO`, without ancestor expansion, as plotted in the main text. Blank where the extraction was empty | |
| | `n_nonempty_runs` | How many of the five runs returned at least one GO term, 0–5 | |
| | `wang_mean_nonempty` | Mean of the non-blank `wang_run*` values for this protein and method. Blank when all five runs were empty | |
| | `p_JAY` | ProtTale's reliability score for this protein. A property of the protein, so the same value repeats on all six rows of that protein | |
| | `Retained` | Whether `p_JAY > 0.537` | |
|
|
| Because the extracted terms themselves are included, the metrics reported in |
| Extended Data can be recomputed from this file as well: Wang similarity with |
| ancestor expansion, Jaccard similarity, and information accretion all take the |
| same two term sets as input. |
|
|
| ### Two aggregation conventions |
|
|
| The paper aggregates Wang similarity in two different ways, both derivable from |
| the per-run columns above. They are not interchangeable and the difference |
| matters when reproducing specific numbers. |
|
|
| - **Figures.** Within each run, average over the non-empty rows only, then |
| average those five per-run means. Empty extractions are excluded from the |
| average and reported separately as the empty rate printed above each bar. |
| - **Wilcoxon tests.** Empty extractions are scored zero and kept, so that all |
| 1,031 proteins remain paired across methods. Dropping rows per method would |
| break the pairing the test requires. |
|
|
| Reproduced under the figure convention, the per-method Wang means are |
| Reference 0.669, ESM2Text 0.569, Prot2Text 0.556, ProtT3 0.605, ProtTale 0.561, |
| Protein2Text 0.443, with empty rates of 4.79%, 31.81%, 22.00%, 10.03%, 14.28% |
| and 28.07% respectively. |
|
|
| Regenerates the GO-extraction figure, the method-comparison figure, the |
| reliability-filter figure, the per-taxonomic-group figure, the five-way Venn, |
| and the judge run-agreement panel. |
|
|
| --- |
|
|
| ## Supplementary Data 3 — Stage 1 test pairs |
|
|
| One row per pair, 534 in total. These pairs validate the LLM-as-judge against |
| an independent ground truth: the MIPS cluster label comes from embedding |
| clustering, the judge label from reading the two curated descriptions. Neither |
| is derived from the other, which is what makes their agreement informative. |
|
|
| | Column | Meaning | |
| |---|---| |
| | `Pair_index` | Row index, 0–533 | |
| | `UniProt_ID_1`, `UniProt_accession_1` | First protein of the pair | |
| | `UniProt_ID_2`, `UniProt_accession_2` | Second protein of the pair | |
| | `MIPS_pair` | `within` if both proteins fall in the same MIPS cluster, `across` otherwise. 267 of each | |
| | `Judge_label` | LLM-as-judge label for the two descriptions: `Yes`, `No`, `Maybe`, or `Vague` | |
| | `Description_1`, `Description_2` | The two curated Swiss-Prot function descriptions given to the judge. No model prediction is involved | |
|
|
| Identifiers for these pairs were recovered by exact sequence match against |
| Swiss-Prot, since the stored pairs carry sequences rather than accessions. 15 |
| of the 1,068 proteins have a sequence that is shared by more than one |
| Swiss-Prot entry; for those the identifier given here is one of several entries |
| with that exact sequence. The descriptions, labels and analysis are unaffected. |
|
|
| **The judge was run once on these pairs**, not five times. This analysis tests |
| the judge's accuracy against the cluster ground truth, not its run-to-run |
| reproducibility; the five-run reproducibility analysis was carried out on the |
| 1,031 unseen predictions and lives in Supplementary Data 2. |
|
|
| Regenerates the within-versus-across cluster panel of the Extended Data judge |
| figure and the accompanying Fisher exact test. |
|
|
| --- |
|
|
| ## Supplementary Data 4 — shuffled-sequence baseline |
|
|
| One row per unseen protein, 1,031 in total. The discrete reliability threshold |
| used throughout the paper is derived from this file. |
|
|
| | Column | Meaning | |
| |---|---| |
| | `UniProt_ID`, `UniProt_accession` | Protein identifiers | |
| | `Length_aa` | Sequence length | |
| | `p_JAY_real` | Reliability score for the real sequence | |
| | `p_JAY_shuffled` | Reliability score for the same sequence with its residues randomly permuted | |
|
|
| A Fisher exact sweep over thresholds on `p_JAY`, contrasting the real and |
| shuffled distributions, peaks at `tau* = 0.537`. That value is adopted as the |
| discrete reliability filter everywhere in the paper, so this file is what makes |
| the threshold checkable. |
|
|
| Regenerates both panels of the Extended Data shuffled-baseline figure. |
|
|
| --- |
|
|
| ## Supplementary Data 5 — MIPS cluster validation |
|
|
| Excel workbook, three sheets. Backs the opening result that embedding proximity |
| is a usable proxy for shared function. |
|
|
| - **README** — sampling, metric definitions, and how each confidence interval was computed. |
| - **Per-pair data** — 1,000 rows, one per protein pair: `cluster_id`, the two |
| UniProt accessions, `seq_identity_pct`, `tm_score`, `tm_score_ge_0.7`, the two |
| EC numbers, `ec_match_level` (0–4) and `ec_match_level4`. |
| - **Summary statistics** — the three reported figures with their intervals. The |
| TM-score and EC proportions recompute live from the per-pair sheet. |
|
|
| Pairs are same-cluster pairs drawn from reviewed UniProtKB entries that carry an |
| EC number and an AlphaFold structure, capped at five pairs per cluster, covering |
| 244 of the 2,204 eligible clusters. |
|
|
| **Note on the sequence-identity figure.** 37.5% is the **mean** pairwise |
| sequence identity over the 1,000 pairs, with a bootstrap confidence interval on |
| that mean. It is not a proportion, and no "sequence similar" threshold is used |
| or implied. The other two figures, 58.8% and 63.1%, are proportions with Wilson |
| intervals. |
|
|
| --- |
|
|
| ## Supplementary Data 6 — training curves |
|
|
| `Supplementary_Data_6_training_curves.csv`, 8,904 rows, long format: `panel`, |
| `series`, `step`, `epoch`, `x_epoch`, `value`. |
| `Supplementary_Data_6_training_curves_meta.json` records, for each panel, the |
| source metrics file, which logged columns were used, the number of epochs, |
| steps per epoch, and the best validation value with its epoch. |
|
|
| | Panel | Stage | Training series | Validation series | Epochs | |
| |---|---|---|---|---| |
| | A | Stage 1 alignment | `train_loss_step`, `train_loss_epoch` | `val_auc`, best 0.866 at epoch 4 | 10 | |
| | B | Stage 2 generation | `train_loss_step` | `val_bleu2`, best 17.51 at epoch 39 | 50 | |
| | C | Reliability head | `train_loss_step` | `val_pos_f1`, best 0.547 at epoch 12 | 20 | |
|
|
| Panel C's random baselines, 0.384 at q = 0.5 and 0.476 at q = 1.0, follow from a |
| positive prevalence of 312 in 1,000 on the head's validation split. |
|
|
| Regenerates the Extended Data training-dynamics figure. |
|
|
| --- |
|
|
| ## Supplementary Data 7 — ProtTale without Stage 1 |
|
|
| One row per unseen protein, 1,031 in total, in the same order and with the same |
| identifiers as the `ProtTale` rows of Supplementary Data 2, so the two files can |
| be joined row by row or on `UniProt_ID`. |
|
|
| ProtTale w/o Stage 1 was trained with Stage 2 only: its frozen encoder was |
| initialized from the original ESM-C weights rather than from the Stage 1 |
| student, and every other setting was unchanged. |
|
|
| | Column | Meaning | |
| |---|---| |
| | `UniProt_ID`, `UniProt_accession` | Protein identifiers | |
| | `Method` | `ProtTale w/o Stage 1` on every row | |
| | `Kingdom` | Taxonomic group, as in Supplementary Data 2 | |
| | `Length_aa` | Sequence length. The short subset is `<= 1024`, the long subset `> 1024` | |
| | `judge_run1` … `judge_run5` | LLM-as-judge label from each of five runs: `Yes`, `No`, `Maybe`, or `Vague`. A blank cell means the judge returned no label in that run, which happened for 21 runs across 5 proteins | |
| | `judge_majority` | `Yes` when at least three of the five runs said `Yes`, otherwise `not-Yes`. Blank runs count as not-`Yes` | |
| | `p_JAY` | ProtTale's reliability score for the protein, identical to Supplementary Data 2. The ablated model has no reliability head of its own, so the filter applied to it uses ProtTale's | |
| | `Retained` | Whether `p_JAY > 0.537` | |
| | `SwissProt_description` | Curated reference text given to the judge | |
| | `Prediction` | Description generated by ProtTale w/o Stage 1 | |
|
|
| By five-run majority, the *Yes* rate is 15.8% over all 1,031 proteins, 12.7% on |
| the short subset and 30.7% on the long subset, against 25.7%, 25.2% and 27.9% for |
| ProtTale. |
|
|
| Regenerates the Extended Data figure comparing ProtTale with ProtTale w/o |
| Stage 1, and the McNemar tests reported with it in the main text. |
|
|
| --- |
|
|
| ## Not covered here |
|
|
| Two display items have no tabular source data: the model architecture figure is |
| a schematic, and the two prompt figures reproduce the prompts themselves. |
|
|
| Training data, dataset splits and clustering files are in the sibling |
| `Data_availability` folder. |
|
|
