Hugging Face's logo

GongLab-THU
/
Cerebra-Seq

cerebra_seq
biology
single_sequence
protein_structure_prediction
custom_code
Model card Files
xet
 Community
Cerebra-Seq / VALIDATION.md
Gonglab's picture
Gonglab
Update inference example, feature extraction and PDB export
4aa18dc verified
preview code
|
Raw History Blame Contribute Delete
6.41 kB

Validation results

The original core-model validation is recorded first. The added atom14 API and sequence-to-features example are covered separately at the end.

Result

All 66 paired forward cases passed bitwise equality (maximum absolute difference 0.0), comparing 1,034 returned tensors. Every compared tensor was finite and retained its shape and dtype. All 17,150 checkpoint tensors (3,430 per checkpoint) were preserved exactly during safetensors conversion.

Coverage

  • CPU float32: model1, 6 cases.
  • NVIDIA A100 80GB (GPU 2), float32 and bfloat16 autocast: all five checkpoints, 60 cases.
  • Seeded synthetic precomputed features: L=16/B=1 and L=31/B=2, three anchors, final two residues masked.
  • Three consecutive recycling passes, feeding each model's own previous state back into itself. Tests include compressed recycling, auxiliary outputs disabled/enabled, full/final-only structure outputs, pAE, distograms, reduced/per-anchor pLDDT, and confidence anchor chunks of size two.
  • Outputs include recycling tensors, translations, quaternions, confidence, angles, PsiPhi, feature tensors, and optional outputs where enabled.
  • cuDNN deterministic mode enabled, benchmark and TF32 disabled. Both sides use the same precision, inputs, configuration and checkpoint.

Dependency isolation

Python 3.11, torch 2.5.1, transformers 4.45.2, huggingface-hub 0.36.0, safetensors 0.4.5, NumPy 1.26.4. See environment.json for full dependency closure.

The exported model runs in a no-system-site-packages virtual environment with only torch, transformers and their declared dependencies exposed. Existing package files are symlinked to avoid downloading duplicate binaries. It is a local test fixture, not an environment distributed to users. Extra packages are absent; an additional import guard rejects them during the exported tests. The original Cerebra_Seq package is also denied, proving that loading does not silently fall back to the original source tree.

The source Conda NumPy package emits an optional mkl-service warning in this minimal environment. Tests still pass without installing that package.

Boundaries

These tests establish equality for the listed cases, not all possible inputs, sequence lengths, CUDA versions or hardware. Features are synthetic; ESM3/ESMC, FASTA-to-PDB output and relaxation are intentionally outside this core-model release. No end-to-end protein prediction claim is made here.

Random initialization consumes a different RNG sequence than SciPy; pretrained weights are loaded strictly and exactly. Training-from-scratch equivalence is not claimed. Optional FlashAttention/DeepSpeed paths are not supported by this minimal release. No change was made to the original source model or checkpoints.

Local AutoModel dynamic-code loading was verified. Uploading to a real Hub repo and downloading from that remote repo has not yet been performed.

Evidence

  • report_cpu.json: six CPU cases.
  • report_cuda2.json: sixty A100 cases.
  • weight_validation.json: exact weight comparisons and SHA-256 hashes.
  • environment.json: exposed runtime distributions.
  • ../hf_tools/: export scripts, templates, weight audit and regression runner.

Additional atom14 API validation

report_atom14.json records the added extract_features(outputs, anchors, batch) interface. The original model forward, initialization and loading methods are unchanged (AST comparison against the previously published wrapper).

  • The extraction function is identical to the function in the supplied Cerebra-Epistasis script. Its reference execution uses this project's existing inference_utils.py geometry helpers; the supplied script's separate model.utils.Cerebra_Seq_utils module was not included in the attachment.
  • All 26 copied helper definitions retain identical ASTs, and all 13 NumPy residue-constant arrays equal those in inference_utils.py.
  • Saved FP32 model outputs at L=16 and L=493, and saved CUDA BF16 model outputs at L=16, give bitwise-identical four-field results after CPU postprocessing.
  • A fresh four-pass CPU FP32 forward at L=16 produces four-field results bitwise equal to the pre-change saved inference results with reference postprocessing applied.
  • The README's full GFP example (L=238) runs successfully in CPU FP32. Only the core repository ID was replaced by the local export path, because the new interface has not been published. ESM models use the documented cached Hub revisions. All four resulting fields equal the reference extraction results bitwise for the same final model outputs.
  • Outputs have exactly the documented keys and shapes, are finite detached CPU float32 tensors, and have binary masks and zero coordinates in unused atom slots. The single-sequence API rejects B>1 rather than discarding items.
  • Exported model loading and postprocessing succeed while imports of esm, einops, h5py, tree, Bio, the original Cerebra_Seq package and local inference scripts are blocked. The full ESM3 example still requires einops.

Additional-test runtime: PyTorch 2.13.0+cpu, Transformers 5.17.0. No new CUDA inference or remote Hub download of the updated export was tested in this run. These comparisons do not claim that HF ESM features equal historical SDK features, or that FP32 results equal the supplied script's CUDA BF16 results.

Optional PDB output validation

to_pdb(features, batch, outputs=None) reuses the extracted atom14 coordinates and the existing project's PDB formatting helpers. No new inference or relaxation is performed. With outputs, confidence is reduced and formatted as in the existing inference script. Without confidence, B-factors are explicitly zero.

report_pdb.json records bytewise agreement with the existing formatter on saved L=16 and L=493 FP32 outputs and saved L=16 BF16 outputs, all processed on CPU. Atom counts, atom names, residue identities, chain A, residue numbering and coordinates (within PDB's three-decimal precision) were also checked. The optional README PDB block was executed with saved GFP features and outputs=None: it wrote 238 CA atoms with zero B-factors. Imports of the original package, local inference helpers, BioPython, h5py, dm-tree, esm and einops were blocked during exported model loading and PDB generation. This test reused saved features; it did not rerun full GFP inference.