Embed evaluated confidence head and publish current training metrics

README.md

@@ -51,17 +51,79 @@ with torch.inference_mode():
 Path("complex.cif").write_text(model.result_to_cif(result), encoding="utf-8")
 ```
 
-Set `verbose=False` to silence the folding progress display. The pinned base has no confidence head. A Hub config with a published v2 head
-source loads the latest head and enables confidence by default; its evaluation
-status remains pending during training.
+Set `verbose=False` to silence the folding progress display. This variant has a Synthyra-adapted native confidence head and returns pLDDT,
+PAE, pTM, and iPTM fields. Confidence calculation is optional.
+
+## Confidence training and evaluation
+
+The packaged model includes the trained confidence head and enables it by default.
+
+The confidence head completed 780 training updates in
+18.1 hours on
+[AtlasFold-Data](https://huggingface.co/datasets/Synthyra/AtlasFold-Data). The backbone and folding
+model stayed frozen, and evaluation used the final exponential moving-average
+checkpoint. Training sampled from 475,969 eligible
+structures, including monomers, dimers, and larger complexes.
+
+The results below use 512 targets from the existing,
+already-used test split, with 5 predictions per target,
+3 recycling loops, and 50 diffusion steps.
+Intervals are 95% bootstrap intervals over targets. Longer sequences were evaluated
+separately and are not included in these tables.
+
+| Model | Standard targets evaluated | Long targets evaluated |
+| --- | ---: | ---: |
+| ESMFold2-300 | 512 | 64 |
+| Production ESMFold2 | 512 | 46 |
+
+| Measurement | ESMFold2-300 | 95% interval |
+| --- | ---: | ---: |
+| pLDDT against all-atom lDDT, Spearman | 0.88003 | 0.85090 to 0.90258 |
+| pTM against TM-score, Spearman | 0.84004 | 0.80621 to 0.86693 |
+| ipTM against DockQ, Spearman | 0.81403 | 0.76544 to 0.85119 |
+| Atom pLDDT mean absolute error | 0.07867 | 0.07617 to 0.08124 |
+| Calibration error, 10 bins | 0.00426 | 0.00241 to 0.00831 |
+| pLDDT cross-entropy | 2.72722 | 2.68388 to 2.77285 |
+| PAE cross-entropy | 2.81677 | 2.76512 to 2.86575 |
+| Within-target lDDT selection accuracy | 0.58505 | 0.50357 to 0.66201 |
+| Within-target ipTM against DockQ selection accuracy | 0.57173 | 0.50567 to 0.63475 |
+| Top-1 selection regret | 0.02312 | 0.01814 to 0.02899 |
+| Random-choice regret | 0.02556 |  |
+| Unresolved against resolved residue AUROC | 0.85591 | 0.83319 to 0.87786 |
+| Resolved residue mean pLDDT | 0.75550 | 0.74226 to 0.76848 |
+| Unresolved residue mean pLDDT | 0.49950 | 0.48104 to 0.51862 |
+| Resolved residues below pLDDT 0.5 | 0.11332 | 0.08916 to 0.13865 |
+| Unresolved residues below pLDDT 0.5 | 0.56934 | 0.52484 to 0.61523 |
+
+Confidence scores, errors, accuracies, and fractions use a 0–1 scale.
+Lower error, cross-entropy, and regret are better; higher correlation, ranking
+accuracy, and AUROC are better. Ranking accuracy compares predictions of the
+same target, with 0.5 representing chance. Regret is the quality lost by selecting
+a prediction instead of the best available one. Unresolved residues are a proxy
+for disorder, not definitive disorder labels.
+
+| Agreement with production ESMFold2 | Spearman correlation | Mean difference |
+| --- | ---: | ---: |
+| Mean pLDDT | 0.68423 | -0.07777 |
+| pTM | 0.79666 | -0.03602 |
+| ipTM | 0.73161 | +0.01022 |
+
+Production agreement compares each model's average confidence per target:
+512 targets for pLDDT and pTM, and
+320 multichain targets for ipTM. Differences are
+ESMFold2-300 minus production. Each model predicts its own structures, so these
+correlations do not measure folding accuracy or scores of identical structures.
+
+See the [FastPLMs confidence training guide](https://github.com/Synthyra/FastPLMs/blob/main/docs/confidence_training.md)
+for the training recipe, evaluation methods, and detailed records.
 
 ## Model overview
 
 `Synthyra/ESMFold2-300` packages the
 `biohub/ESMFold2-Experimental-Fast-base300M-step1500k` checkpoint with the
-FastPLMs runtime for Hugging Face Transformers. It accepts raw amino-acid
-sequences or typed molecular-complex specifications; low-level forward accepts
-prepared feature tensors.
+FastPLMs runtime and a Synthyra-adapted native confidence head for Hugging Face
+Transformers. It accepts raw amino-acid sequences or typed molecular-complex
+specifications; low-level forward accepts prepared feature tensors.
 
 The repository uses the standard Transformers loading interface with
 `trust_remote_code=True`. See Technical details for each registered class and
@@ -201,7 +263,7 @@ This checkpoint was trained without MSA conditioning. It rejects
 `ProteinInput.msa` and MSA-derived features. Typed multichain and multimolecule
 inputs remain supported without MSA conditioning.
 
-The pinned base has no confidence head. The Hub config can bind the latest published v2 head, which enables pLDDT, pTM, iPTM and PAE by default.
+The Synthyra-adapted native confidence head returns pLDDT, PAE, pTM, and iPTM. Confidence calculation is optional.
 The 300 and 600 suffixes describe backbone scale, not total model parameters.
 
 ## Folding speed settings
@@ -243,53 +305,13 @@ The learned projection maps `H: (b, l, 31, 960) -> Z: (b, l, 256)`.
 `embed_dataset` returns one `(l, 256)` residue representation per sequence.
 The experimental architecture does not expose folding TTT.
 
-## Current v2 confidence head
-
-The v2 reproduction publishes current EMA heads during training to the public
-[artifact dataset](https://huggingface.co/datasets/Synthyra/FastPLMs-artifacts/tree/main/confidence/v2/v2-reproduction-20260922/esmfold2_300).
-After the first trained checkpoint is uploaded, the Hub config binds this model
-to its latest published head. `AutoModel.from_pretrained` then loads that head
-and enables pLDDT, PAE, pTM and iPTM by default. These training checkpoints have
-pending evaluation; the historical results below do not validate them.
-
-Each load resolves an immutable dataset revision and verifies the head's hash
-and native state. `model.config.confidence_head_resolved` records the revision,
-training update and head identity. An already loaded model keeps its head;
-reload to obtain a newer publication. `save_pretrained` embeds the exact loaded
-head, so the saved model reloads without fetching a newer one.
-
-Pass `load_confidence_head=False` to load the unchanged base without its external
-head. This option does not remove a head already embedded in a saved model.
-External loading supports one resident model device and cached offline loads;
-split-device and disk-offloaded loading are unsupported. See the
-[confidence training guide](https://github.com/Synthyra/FastPLMs/blob/main/docs/confidence_training.md#ongoing-hugging-face-checkpoints).
-
-## Separately trained confidence head
-
-The pinned base checkpoint has its confidence head disabled. The historical
-GH200 confidence weights were not recovered and remain unpublished.
-
-The archived correlations, bootstrap intervals, and acceptance gates require
-recomputation after correcting tied ranks in Spearman correlation. Raw test
-predictions were not recovered from the closed GH200 workstation or W&B, so
-the historical numbers are withheld here pending raw prediction recovery.
-They do not establish corrected quality or acceptance results.
-
-The [W&B training history audit](https://wandb.ai/lhallee/fastplms-confidence/runs/d5587e06871a) inspected all
-780 update rows and found zero skipped training targets.
-The skipped-target gradient bug therefore did not affect this recorded run's
-training weights. This audit does not validate the archived correlations.
-
-The [confidence training guide](https://github.com/Synthyra/FastPLMs/blob/main/docs/confidence_training.md)
-preserves the historical results and their review status.
-
 ## Notes and limitations
 
-Experimental Fast checkpoint with a frozen 300M ESM++ backbone, tensor-exact in
-BF16 with the pinned step-1500000 source, 24 folding blocks, no MSA
-conditioning, and no confidence head. BF16 execution uses FP32 folding
-parameters with CUDA autocast. FP8 is unsupported. Docker BF16 inference
-validation passed on the compact Protein G case.
+Experimental Fast model with a frozen 300M ESM++ backbone, 24 folding blocks,
+no MSA conditioning, and a Synthyra-trained confidence head enabled by default.
+BF16 execution uses FP32 folding parameters with CUDA autocast; FP8 is
+unsupported. Confidence evaluation does not establish full structure-model
+equivalence to production ESMFold2.
 
 ## Technical details
 
@@ -322,10 +344,8 @@ identities and conversion details in `source-record.json`.
 - Release tiers: `check`, `compliance`, `structure`, `feature`, `artifact`, `benchmark`
 - Unresolved required file identities: `0`
 
-ESMFold2-300 passed a Docker BF16 reference comparison on one compact Protein G
-sequence. ESMFold2-600 is not inference-validated. Both have configuration,
-weight identity, and artifact loading checks. This is bounded checkpoint
-evidence, not a full structure benchmark result.
+The confidence evaluation above uses the existing test split. It does not
+establish full structure-model equivalence.
 
 Declared tiers compare configuration, tokenizer behavior, state, and
 representative inference with the pinned reference. A nonzero unresolved count

config.json

@@ -10,7 +10,7 @@
   },
   "confidence_head": {
     "distogram_bins": 128,
-    "enabled": false,
+    "enabled": true,
     "folding_trunk": {
       "dropout": 0.25,
       "n_heads": 8,
@@ -27,12 +27,10 @@
   "disable_msa_features": true,
   "dtype": "float32",
   "esmc_id": "Synthyra/ESMplusplus_small",
-  "fastplms_checkpoint_hash": "ea9ee0e05c199173d222df455ef1e03580ca5bd693e1c02783c3c3ab2163735a",
-  "fastplms_checkpoint_repo_id": "biohub/ESMFold2-Experimental-Fast-base300M-step1500k",
-  "fastplms_checkpoint_revision": "21531e59002c9205284715e28ee802dafb430637",
+  "fastplms_checkpoint_hash": "bb2eaf6389e1d58f5117ca3173a5ee73d73548cda5c89f95f85c4a2280ad0e7e",
+  "fastplms_checkpoint_repo_id": "Synthyra/ESMFold2-300",
   "fastplms_model_id": "esmfold2_300",
-  "fastplms_runtime_bundle_sha256": "843922e6065315ab840313fa684227a9c6db9b33e5c652297afbab1a8f38240d",
-  "fastplms_weights_revision": "21531e59002c9205284715e28ee802dafb430637",
+  "fastplms_runtime_bundle_sha256": "b865d946cb53efb12d954ebd1f1a9a523c06ffe5dabfc73f931c5422397db26d",
   "folding_trunk": {
     "dropout": 0.25,
     "n_heads": 8,
@@ -101,12 +99,12 @@
   },
   "transformers_version": "4.57.6",
   "type": "experimental",
-  "confidence_head_source": {
-    "repo_id": "Synthyra/FastPLMs-artifacts",
-    "repo_type": "dataset",
-    "latest_path": "confidence/v2/v2-reproduction-20260922/esmfold2_300/latest.json",
-    "revision": "main",
-    "model_id": "esmfold2_300",
-    "base_weight_sha256": "44d6797c5efebf24753d502b40950e0874871c96ceea14f2d7f7e39cebac67fd"
+  "confidence_head_release": {
+    "name": "v1",
+    "head_sha256": "40fd7f3d82fcefe8ad20ab2b32a37a68a84b54a527a4bce6eb9437bad2b77e31",
+    "base_weight_sha256": "44d6797c5efebf24753d502b40950e0874871c96ceea14f2d7f7e39cebac67fd",
+    "training_url": "https://wandb.ai/lhallee/fastplms-confidence/runs/9558b6d23daf",
+    "updates": 780,
+    "evaluation_request_sha256": "be5fb9a4b65a0c2abfa040069ffd4344a9d220ef8e595711c4849a62401e281a"
   }
 }

fastplms/models.toml

@@ -1254,6 +1254,7 @@ auto_map = { AutoConfig = "fastplms.models.esmfold2.configuration_esmfold2.ESMFo
 
 [[models]]
 id = "esmfold2_300"
+confidence_adaptation = { release = "v1", head_sha256 = "40fd7f3d82fcefe8ad20ab2b32a37a68a84b54a527a4bce6eb9437bad2b77e31", base_weight_sha256 = "44d6797c5efebf24753d502b40950e0874871c96ceea14f2d7f7e39cebac67fd", donor_repo = "biohub/ESMFold2-Experimental-Fast-Cutoff2025", donor_revision = "74b88548bf19688b8727432db0d698cb2e1d8783", donor_weight_sha256 = "4e903b740ad6ad704ec60881bfd593e0d6c874a630ffa0f0838276e0b665088f", training_url = "https://wandb.ai/lhallee/fastplms-confidence/runs/9558b6d23daf", evaluation_url = "https://huggingface.co/datasets/Synthyra/FastPLMs-artifacts/tree/309f353b07e0e46de4d77a5266d4eddd695538e3/confidence-v2/v2-reproduction-20260922/public/evaluation/esmfold2_300", evidence_path = "docs/evidence/confidence/esmfold2_300-v1.json", frozen_base = { repo = "Synthyra/ESMFold2-300", revision = "a38a62ae930d157484b331c2bf4241684573adba", files = ["config.json=git-sha1:47ec20cf8b234c3b41d6f3ae1bdfe95d4eb4849e", "model.safetensors=sha256:44d6797c5efebf24753d502b40950e0874871c96ceea14f2d7f7e39cebac67fd"] } }
 family = "esmfold2"
 size_category = "structure"
 generation_contract = "not_applicable"
@@ -1265,13 +1266,14 @@ fast_files = ["config.json=git-sha1:47ec20cf8b234c3b41d6f3ae1bdfe95d4eb4849e", "
 official_repo = "biohub/ESMFold2-Experimental-Fast-base300M-step1500k"
 official_revision = "21531e59002c9205284715e28ee802dafb430637"
 official_files = ["config.json=git-sha1:8c9a04fe22b0e5fca77bc4e2861a12c9494ef4d4", "model.safetensors=sha256:44d6797c5efebf24753d502b40950e0874871c96ceea14f2d7f7e39cebac67fd"]
-notes = "Experimental Fast checkpoint with a frozen 300M ESM++ backbone, tensor-exact in BF16 with the pinned step-1500000 source, 24 folding blocks, no MSA conditioning, and no confidence head. BF16 execution uses FP32 folding parameters with CUDA autocast. FP8 is unsupported. Docker BF16 inference validation passed on the compact Protein G case."
+notes = "Experimental Fast model with a frozen 300M ESM++ backbone, 24 folding blocks, no MSA conditioning, and a Synthyra-trained confidence head enabled by default. BF16 execution uses FP32 folding parameters with CUDA autocast; FP8 is unsupported. Confidence evaluation does not establish full structure-model equivalence to production ESMFold2."
 auto_map = { AutoConfig = "fastplms.models.esmfold2.configuration_esmfold2.ESMFold2Config", AutoModel = "fastplms.models.esmfold2.modeling_esmfold2_experimental.ESMFold2ExperimentalModel", AutoModelForSequenceClassification = "fastplms.models.esmfold2.modeling_esmfold2_classification.ESMFold2ExperimentalForSequenceClassification", AutoModelForTokenClassification = "fastplms.models.esmfold2.modeling_esmfold2_classification.ESMFold2ExperimentalForTokenClassification" }
 backbone_model = "esmc_small"
 backbone = { repo = "biohub/ESMC-300M-1500000", revision = "56803b6378b82e16c3b24aac49d1fce4445540b7", files = ["config.json=git-sha1:7fe728a0eb3fb81b24491d6cc1de816bf7797c27", "model.safetensors=sha256:8bd6cacf9b5a92d51954b64b20407f1f9f564a7e4849b8663470784d2a8b7ed2", "tokenizer.json=git-sha1:81c797f56768b22dec0301fa771f018b7e43e98c", "tokenizer_config.json=git-sha1:f49f57b24a1c93bd544974811e8ecbd61b7fae89", "special_tokens_map.json=git-sha1:c907ee1dc19b24241749b32d665c291c7e6e8e4b"] }
 
 [[models]]
 id = "esmfold2_600"
+confidence_adaptation = { release = "v1", head_sha256 = "e84726a050722e1b722712c87d17a5388bd3699e2520e4a59abb1d828dfb8de7", base_weight_sha256 = "11a53c1b4700b6c62a5a464fc3ec7076160c19e8584157f88e13275116cfd602", donor_repo = "biohub/ESMFold2-Experimental-Fast-Cutoff2025", donor_revision = "74b88548bf19688b8727432db0d698cb2e1d8783", donor_weight_sha256 = "4e903b740ad6ad704ec60881bfd593e0d6c874a630ffa0f0838276e0b665088f", training_url = "https://wandb.ai/lhallee/fastplms-confidence/runs/820d2cfa56c0", evaluation_url = "https://huggingface.co/datasets/Synthyra/FastPLMs-artifacts/tree/6e62186cd36b9047cc4691980076be9f76482192/confidence-v2/v2-reproduction-20260922/public/evaluation/esmfold2_600", evidence_path = "docs/evidence/confidence/esmfold2_600-v1.json", frozen_base = { repo = "Synthyra/ESMFold2-600", revision = "71c67d0b2b73dc245ea7c3cc0d0476439a882d08", files = ["config.json=git-sha1:8e271837cbdada96c4974c8e543f84065e0f06f1", "model.safetensors=sha256:11a53c1b4700b6c62a5a464fc3ec7076160c19e8584157f88e13275116cfd602"] } }
 family = "esmfold2"
 size_category = "structure"
 generation_contract = "not_applicable"
@@ -1283,7 +1285,7 @@ fast_files = ["config.json=git-sha1:8e271837cbdada96c4974c8e543f84065e0f06f1", "
 official_repo = "biohub/ESMFold2-Experimental-Fast-base600M-step1500k"
 official_revision = "15cf2d6648692f6c17cee1297d8a285476fffa9b"
 official_files = ["config.json=git-sha1:95517e555f17a1eca4b68866c89033a1f6916a5d", "model.safetensors=sha256:11a53c1b4700b6c62a5a464fc3ec7076160c19e8584157f88e13275116cfd602"]
-notes = "Experimental Fast checkpoint with a frozen 600M ESM++ backbone, tensor-exact in BF16 with the pinned step-1500000 source, 24 folding blocks, no MSA conditioning, and no confidence head. BF16 execution uses FP32 folding parameters with CUDA autocast. FP8 is unsupported. Configuration, weight identities, and artifact reload are verified; this model is not inference-validated."
+notes = "Experimental Fast model with a frozen 600M ESM++ backbone, 24 folding blocks, no MSA conditioning, and a Synthyra-trained confidence head enabled by default. BF16 execution uses FP32 folding parameters with CUDA autocast; FP8 is unsupported. Confidence evaluation does not establish full structure-model equivalence to production ESMFold2."
 auto_map = { AutoConfig = "fastplms.models.esmfold2.configuration_esmfold2.ESMFold2Config", AutoModel = "fastplms.models.esmfold2.modeling_esmfold2_experimental.ESMFold2ExperimentalModel", AutoModelForSequenceClassification = "fastplms.models.esmfold2.modeling_esmfold2_classification.ESMFold2ExperimentalForSequenceClassification", AutoModelForTokenClassification = "fastplms.models.esmfold2.modeling_esmfold2_classification.ESMFold2ExperimentalForTokenClassification" }
 backbone_model = "esmc_large"
 backbone = { repo = "biohub/ESMC-600M-1500000", revision = "21af9cc429af76ebda6c48074fb624db4735aaaf", files = ["config.json=git-sha1:ec29f6009b21d710f64bf1c058f3a9710833d692", "model.safetensors=sha256:d6869f5ae0f11e5dc829b195e062e87cfcc2f851a08a5edbaf5d1083ae7f76cc", "tokenizer.json=git-sha1:81c797f56768b22dec0301fa771f018b7e43e98c", "tokenizer_config.json=git-sha1:f49f57b24a1c93bd544974811e8ecbd61b7fae89", "special_tokens_map.json=git-sha1:c907ee1dc19b24241749b32d665c291c7e6e8e4b"] }

fastplms/registry.py

@@ -1344,6 +1344,9 @@ def _parse_confidence_adaptation(
             or weight.digest != digests["base_weight_sha256"]
         ):
             raise RegistryError(f"{base_context} model.safetensors must match base_weight_sha256.")
+        config = frozen_base.file_map.get("config.json")
+        if config is None or config.algorithm != "git-sha1":
+            raise RegistryError(f"{base_context} must pin config.json with git-sha1.")
     if release == "v1" and frozen_base is None:
         raise RegistryError(f"{adaptation_context}.frozen_base is required for v1.")
     donor_repo = _require_str(raw, "donor_repo", adaptation_context)

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:44d6797c5efebf24753d502b40950e0874871c96ceea14f2d7f7e39cebac67fd
-size 684606540
+oid sha256:bb2eaf6389e1d58f5117ca3173a5ee73d73548cda5c89f95f85c4a2280ad0e7e
+size 715690440

modeling_fastplms.py

@@ -13,7 +13,7 @@ from zipfile import ZIP_DEFLATED, ZipFile
 
 from .fastplms_bundle import RUNTIME_DATA, RUNTIME_HASH
 
-if RUNTIME_HASH != "5c466dfbd29ac7be773a338382f0e555f6bf595d0c4231f6deef6399e42b197f":
+if RUNTIME_HASH != "b865d946cb53efb12d954ebd1f1a9a523c06ffe5dabfc73f931c5422397db26d":
     raise RuntimeError("FastPLMs runtime identity differs from the bridge.")
 
 _RUNTIME_TEMPORARIES = []