wasarou's picture
Add RGI manuscript evaluator rescoring
aefc53e verified
|
Raw History Blame Contribute Delete
7.98 kB
# Re-evaluation with the RGI manuscript method
Archived predictions were rescored; no model inference or weight editing was performed.
R/S uses the manuscript's legacy RDKit center assignment. E/Z includes only acyclic,
nonaromatic, heavy-atom-decidable defined bonds. Geometry uses the manuscript's
independent canonical-SMILES ETKDGv3/UFF/stereo-validated reference preparation.
The joint count is the conjunction of the manuscript's two per-system classifications.
It is an additional summary, not the original common-mapping full-stereochemistry gate.
The existing 55/50/104 split and previous validation/test exposure are unchanged.
The original full-stereochemistry repair requirement remains unmet.
All geometry entries below are medians of per-system RMS errors.
## development
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 55 | 21 | 41 | 20 | 0.038695 | 2.8723 |
| boltz2_wsp_absolute_vectors_distances_v17.ckpt | 55 | 24 | 41 | 23 | 0.038138 | 2.8443 |
| boltz2_wsp_absolute_vectors_double_bond_v15.ckpt | 55 | 21 | 42 | 20 | 0.038452 | 2.8680 |
| boltz2_wsp_absolute_vectors_fourfold_v16.ckpt | 55 | 22 | 42 | 21 | 0.039179 | 2.7212 |
| boltz2_wsp_absolute_vectors_only_v11.ckpt | 55 | 24 | 41 | 23 | 0.038121 | 2.8315 |
| boltz2_wsp_balanced_geometry_v9.ckpt | 55 | 21 | 42 | 20 | 0.038385 | 2.7390 |
| boltz2_wsp_both_digit_suppression_v19.ckpt | 55 | 19 | 42 | 17 | 0.044122 | 2.7141 |
| boltz2_wsp_denoiser_distances_only_v12.ckpt | 55 | 21 | 41 | 20 | 0.038440 | 2.8695 |
| boltz2_wsp_denoiser_name_projection_v7.ckpt | 55 | 21 | 40 | 19 | 0.074566 | 4.8459 |
| boltz2_wsp_direct_input_balance_v6.ckpt | 55 | 9 | 25 | 5 | 2.505421 | 46.2875 |
| boltz2_wsp_direct_input_conditioning_v5.ckpt | 55 | 22 | 43 | 21 | 0.044555 | 3.1729 |
| boltz2_wsp_double_bond_contrast_v14.ckpt | 55 | 21 | 41 | 19 | 0.038672 | 2.9020 |
| boltz2_wsp_input_center_preserving_v8.ckpt | 55 | 24 | 42 | 22 | 0.046064 | 3.1451 |
| boltz2_wsp_input_digit_suppression_v18.ckpt | 55 | 21 | 42 | 18 | 0.036351 | 2.9269 |
| boltz2_wsp_input_geometry_v1.ckpt | 55 | 23 | 40 | 22 | 0.038254 | 2.9360 |
| boltz2_wsp_input_geometry_v3.ckpt | 55 | 21 | 42 | 20 | 0.038556 | 2.7189 |
| boltz2_wsp_input_reference_mask_v20.ckpt | 55 | 20 | 41 | 19 | 0.038081 | 2.8679 |
| boltz2_wsp_ligand_name_projection_v4.ckpt | 55 | 20 | 41 | 19 | 0.038429 | 2.8946 |
| boltz2_wsp_neighbor_program_control_v26.ckpt | 55 | 21 | 41 | 20 | 0.037731 | 2.8868 |
| boltz2_wsp_orthogonal_atom_geometry_v22.ckpt | 55 | 21 | 41 | 20 | 0.038386 | 2.8960 |
| boltz2_wsp_orthogonal_input_geometry_v24.ckpt | 55 | 20 | 41 | 19 | 0.038285 | 2.9150 |
| boltz2_wsp_orthogonal_pair_geometry_v23.ckpt | 55 | 21 | 41 | 19 | 0.039263 | 2.8838 |
| boltz2_wsp_programmed_neighbor_prior_v25.ckpt | 55 | 20 | 41 | 19 | 0.038947 | 2.9092 |
| boltz2_wsp_protected_distance_direction_v13.ckpt | 55 | 22 | 41 | 20 | 0.038652 | 2.8792 |
| boltz2_wsp_reference_distances_v2.ckpt | 55 | 23 | 40 | 22 | 0.038295 | 2.9338 |
| boltz2_wsp_reference_mask_combined_v21.ckpt | 55 | 20 | 42 | 19 | 0.038563 | 2.8850 |
| boltz2_wsp_relative_vectors_only_v10.ckpt | 55 | 20 | 41 | 19 | 0.038319 | 2.8840 |
| boltz2_wsp_reserved_atom_channels_v28.ckpt | 55 | 22 | 39 | 20 | 0.098797 | 6.0517 |
| boltz2_wsp_reserved_denoiser_channels_v27.ckpt | 55 | 8 | 34 | 7 | 0.586642 | 34.4782 |
| boltz2_wsp_reserved_token_channels_v29.ckpt | 55 | 8 | 28 | 7 | 0.520425 | 23.8152 |
| boltz2_wsp_reserved_token_condition_bias_v30.ckpt | 55 | 8 | 25 | 1 | 0.640224 | 29.1628 |
## v2_validation_seed_1
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 50 | 27 | 43 | 22 | 0.030402 | 2.4558 |
| boltz2_wsp_reference_distances_v2.ckpt | 50 | 25 | 43 | 23 | 0.030114 | 2.5120 |
## v2_test_seed_1
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 104 | 41 | 79 | 35 | 0.037237 | 2.6254 |
| boltz2_wsp_reference_distances_v2.ckpt | 104 | 44 | 78 | 36 | 0.036704 | 2.6128 |
## v2_test_seed_2
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 104 | 43 | 79 | 34 | 0.036447 | 2.6106 |
| boltz2_wsp_reference_distances_v2.ckpt | 104 | 42 | 79 | 32 | 0.035559 | 2.6137 |
## v2_test_seed_3
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 104 | 41 | 78 | 34 | 0.036223 | 2.6611 |
| boltz2_wsp_reference_distances_v2.ckpt | 104 | 43 | 80 | 35 | 0.036433 | 2.5693 |
## v2_nondevelopment_seed_1
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 154 | 68 | 122 | 57 | 0.034383 | 2.5826 |
| boltz2_wsp_reference_distances_v2.ckpt | 154 | 69 | 121 | 59 | 0.034468 | 2.5866 |
## v11_validation_seed_1
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 50 | 27 | 43 | 22 | 0.030402 | 2.4558 |
| boltz2_wsp_absolute_vectors_only_v11.ckpt | 50 | 25 | 43 | 23 | 0.030142 | 2.4849 |
## v11_test_seed_1
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 104 | 41 | 79 | 35 | 0.037237 | 2.6254 |
| boltz2_wsp_absolute_vectors_only_v11.ckpt | 104 | 44 | 78 | 36 | 0.036368 | 2.6236 |
## v11_test_seed_2
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 104 | 43 | 79 | 34 | 0.036447 | 2.6106 |
| boltz2_wsp_absolute_vectors_only_v11.ckpt | 104 | 42 | 77 | 32 | 0.035828 | 2.6562 |
## v11_test_seed_3
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 104 | 41 | 78 | 34 | 0.036223 | 2.6611 |
| boltz2_wsp_absolute_vectors_only_v11.ckpt | 104 | 43 | 80 | 35 | 0.036183 | 2.5556 |
## v11_nondevelopment_seed_1
| Checkpoint | N | R/S | E/Z | Joint | Bond (A) | Angle (degrees) |
|---|---:|---:|---:|---:|---:|---:|
| boltz2_base.ckpt | 154 | 68 | 122 | 57 | 0.034383 | 2.5826 |
| boltz2_wsp_absolute_vectors_only_v11.ckpt | 154 | 69 | 121 | 59 | 0.033988 | 2.5827 |
## Paired effects
Average repeated seeds within each system, then bootstrap sequence/scaffold groups
10,000 times with resampling seed 20261103. These intervals are conditional on the
evaluated seeds and do not correct for candidate selection.
### v2: nondevelopment_seed_1
| Metric | Mean change | Cluster 95% CI |
|---|---:|---|
| match | 0.00649351 | [-0.03896358, 0.05625000] |
| ez_match | -0.00649351 | [-0.02255745, 0.00000000] |
| joint | 0.01298701 | [-0.02857143, 0.05194805] |
| emb_brms | 0.00046358 | [-0.00019102, 0.00120255] |
| emb_arms | 0.00892608 | [-0.06281499, 0.08946591] |
### v2: test_seeds_1_2_3
| Metric | Mean change | Cluster 95% CI |
|---|---:|---|
| match | 0.01282051 | [-0.02272889, 0.05246914] |
| ez_match | 0.00320513 | [-0.01980198, 0.02564103] |
| joint | 0.00000000 | [-0.03300330, 0.03205128] |
| emb_brms | 0.00011564 | [-0.00041220, 0.00068050] |
| emb_arms | -0.04241056 | [-0.12946040, 0.03628711] |
### v11: nondevelopment_seed_1
| Metric | Mean change | Cluster 95% CI |
|---|---:|---|
| match | 0.00649351 | [-0.03896358, 0.05625000] |
| ez_match | -0.00649351 | [-0.02255745, 0.00000000] |
| joint | 0.01298701 | [-0.02857143, 0.05194805] |
| emb_brms | 0.00064088 | [-0.00012454, 0.00129773] |
| emb_arms | 0.04754782 | [-0.03673725, 0.11713533] |
### v11: test_seeds_1_2_3
| Metric | Mean change | Cluster 95% CI |
|---|---:|---|
| match | 0.01282051 | [-0.02272889, 0.05246914] |
| ez_match | -0.00320513 | [-0.02380952, 0.01388889] |
| joint | 0.00000000 | [-0.03300330, 0.03205128] |
| emb_brms | 0.00000198 | [-0.00045463, 0.00047334] |
| emb_arms | -0.05296255 | [-0.14310073, 0.01931832] |