| # Validation results |
|
|
| The original core-model validation is recorded first. The added atom14 API |
| and sequence-to-features example are covered separately at the end. |
|
|
| ## Result |
|
|
| All 66 paired forward cases passed **bitwise equality** (maximum absolute |
| difference **0.0**), comparing 1,034 returned tensors. Every compared tensor |
| was finite and retained its shape and dtype. All 17,150 checkpoint tensors |
| (3,430 per checkpoint) were preserved exactly during safetensors conversion. |
|
|
| ## Coverage |
|
|
| - CPU float32: model1, 6 cases. |
| - NVIDIA A100 80GB (GPU 2), float32 and bfloat16 autocast: all five checkpoints, |
| 60 cases. |
| - Seeded synthetic precomputed features: L=16/B=1 and L=31/B=2, three anchors, |
| final two residues masked. |
| - Three consecutive recycling passes, feeding each model's own previous state |
| back into itself. Tests include compressed recycling, auxiliary outputs |
| disabled/enabled, full/final-only structure outputs, pAE, distograms, |
| reduced/per-anchor pLDDT, and confidence anchor chunks of size two. |
| - Outputs include recycling tensors, translations, quaternions, confidence, |
| angles, PsiPhi, feature tensors, and optional outputs where enabled. |
| - cuDNN deterministic mode enabled, benchmark and TF32 disabled. Both sides use |
| the same precision, inputs, configuration and checkpoint. |
|
|
| ## Dependency isolation |
|
|
| Python 3.11, torch 2.5.1, transformers 4.45.2, huggingface-hub 0.36.0, |
| safetensors 0.4.5, NumPy 1.26.4. See environment.json for full dependency closure. |
|
|
| The exported model runs in a no-system-site-packages virtual environment with |
| only torch, transformers and their declared dependencies exposed. Existing |
| package files are symlinked to avoid downloading duplicate binaries. It is a |
| local test fixture, not an environment distributed to users. Extra packages |
| are absent; an additional import guard rejects them during the exported tests. |
| The original Cerebra_Seq package is also denied, proving that loading does not |
| silently fall back to the original source tree. |
| |
| The source Conda NumPy package emits an optional mkl-service warning in this |
| minimal environment. Tests still pass without installing that package. |
| |
| ## Boundaries |
| |
| These tests establish equality for the listed cases, not all possible inputs, |
| sequence lengths, CUDA versions or hardware. Features are synthetic; ESM3/ESMC, |
| FASTA-to-PDB output and relaxation are intentionally outside this core-model |
| release. No end-to-end protein prediction claim is made here. |
| |
| Random initialization consumes a different RNG sequence than SciPy; pretrained |
| weights are loaded strictly and exactly. Training-from-scratch equivalence is |
| not claimed. Optional FlashAttention/DeepSpeed paths are not supported by this |
| minimal release. No change was made to the original source model or checkpoints. |
| |
| Local AutoModel dynamic-code loading was verified. Uploading to a real Hub repo |
| and downloading from that remote repo has not yet been performed. |
| |
| ## Evidence |
| |
| - report_cpu.json: six CPU cases. |
| - report_cuda2.json: sixty A100 cases. |
| - weight_validation.json: exact weight comparisons and SHA-256 hashes. |
| - environment.json: exposed runtime distributions. |
| - ../hf_tools/: export scripts, templates, weight audit and regression runner. |
| |
| ## Additional atom14 API validation |
| |
| `report_atom14.json` records the added `extract_features(outputs, anchors, batch)` |
| interface. The original model forward, initialization and loading methods are |
| unchanged (AST comparison against the previously published wrapper). |
| |
| - The extraction function is identical to the function in the supplied |
| Cerebra-Epistasis script. Its reference execution uses this project's existing |
| `inference_utils.py` geometry helpers; the supplied script's separate |
| `model.utils.Cerebra_Seq_utils` module was not included in the attachment. |
| - All 26 copied helper definitions retain identical ASTs, and all 13 NumPy |
| residue-constant arrays equal those in `inference_utils.py`. |
| - Saved FP32 model outputs at L=16 and L=493, and saved CUDA BF16 model outputs |
| at L=16, give bitwise-identical four-field results after CPU postprocessing. |
| - A fresh four-pass CPU FP32 forward at L=16 produces four-field results |
| bitwise equal to the pre-change saved inference results with reference |
| postprocessing applied. |
| - The README's full GFP example (L=238) runs successfully in CPU FP32. Only the |
| core repository ID was replaced by the local export path, because the new |
| interface has not been published. ESM models use the documented cached Hub |
| revisions. All four resulting fields equal the reference extraction results |
| bitwise for the same final model outputs. |
| - Outputs have exactly the documented keys and shapes, are finite detached |
| CPU float32 tensors, and have binary masks and zero coordinates in unused |
| atom slots. The single-sequence API rejects B>1 rather than discarding items. |
| - Exported model loading and postprocessing succeed while imports of `esm`, |
| `einops`, `h5py`, `tree`, `Bio`, the original `Cerebra_Seq` package and local |
| inference scripts are blocked. The full ESM3 example still requires `einops`. |
|
|
| Additional-test runtime: PyTorch 2.13.0+cpu, Transformers 5.17.0. No new CUDA |
| inference or remote Hub download of the updated export was tested in this |
| run. These comparisons do not claim that HF ESM features equal historical SDK |
| features, or that FP32 results equal the supplied script's CUDA BF16 results. |
|
|
| ## Optional PDB output validation |
|
|
| `to_pdb(features, batch, outputs=None)` reuses the extracted atom14 coordinates |
| and the existing project's PDB formatting helpers. No new inference or relaxation |
| is performed. With `outputs`, confidence is reduced and formatted as in the |
| existing inference script. Without confidence, B-factors are explicitly zero. |
|
|
| `report_pdb.json` records bytewise agreement with the existing formatter on |
| saved L=16 and L=493 FP32 outputs and saved L=16 BF16 outputs, all processed on |
| CPU. Atom counts, atom names, residue identities, chain A, residue numbering |
| and coordinates (within PDB's three-decimal precision) were also checked. |
| The optional README PDB block was executed with saved GFP features and |
| `outputs=None`: it wrote 238 CA atoms with zero B-factors. Imports of the |
| original package, local inference helpers, BioPython, h5py, dm-tree, esm and |
| einops were blocked during exported model loading and PDB generation. |
| This test reused saved features; it did not rerun full GFP inference. |
|
|
