| --- |
| license: other |
| library_name: pytorch |
| tags: |
| - biology |
| - bioinformatics |
| - biosynthetic-gene-clusters |
| - retrieval |
| - protein-language-models |
| - pfam |
| - pytorch |
| pipeline_tag: feature-extraction |
| --- |
| |
| # BGC SetNet and Weighted Pfam Retrieval Models |
|
|
| This repository contains the final five-seed model artifacts for the manuscript **"Sequence-Derived Representations versus Pfam-Domain Content for Biosynthetic Gene Cluster Retrieval"**. |
|
|
| The release includes two model families: |
|
|
| - **Pfam-augmented BGC-SetNet** (`pfam_setnet`): a Set Transformer encoder over frozen ESM-2 gene embeddings, relative gene positions, padding masks, and a BGC-level Pfam inventory embedding. |
| - **Weighted Pfam Jaccard** (`weighted_pfam_jaccard`): learned non-negative Pfam-domain weights used in a weighted set-Jaccard retrieval baseline. |
|
|
| The manuscript's main result is conservative: Pfam-domain content remains the strongest signal for this silver-label retrieval benchmark, while the ESM + BGC-SetNet + Pfam ensemble gives small, statistically unsupported gains on some secondary metrics. |
|
|
| ## Repository Contents |
|
|
| ```text |
| checkpoints/ |
| setnet/seed-20260810..20260814/ |
| model.safetensors # verified model_state checkpoint |
| config.json |
| pfam_vocab.json |
| phase2_history.json |
| weighted_pfam/seed-20260810..20260814/ |
| model.safetensors |
| config.json |
| pfam_vocab.json |
| phase2_history.json |
| evaluation/ # per-seed metadata, summaries, group/query outputs, alpha searches |
| results/dgx_final/ # aggregate manuscript result tables |
| source/ # source snapshot used for training/evaluation/provenance |
| provenance/checkpoint_manifest.json |
| ``` |
|
|
| ## Evaluation Summary |
|
|
| Family means over five optimization seeds and 16 held-out MIBiG-reference groups: |
|
|
| | Method | Recall@50 | MRR | MAP | nDCG@50 | |
| |---|---:|---:|---:|---:| |
| | Raw ESM mean | 0.7946 | 0.2550 | 0.7251 | 0.8078 | |
| | BGC-SetNet + Pfam | 0.8472 | 0.2786 | 0.7771 | 0.8502 | |
| | Pfam Jaccard | 0.8788 | 0.3071 | 0.8480 | 0.9042 | |
| | ESM + BGC-SetNet + Pfam | 0.8769 | 0.3096 | 0.8503 | 0.9058 | |
| | Weighted Pfam Jaccard | 0.8789 | 0.3069 | 0.8477 | 0.9040 | |
|
|
| The weighted Pfam Recall@50 advantage over unweighted Pfam is numerically tiny (+0.00003) and should not be interpreted as a meaningful improvement. |
|
|
| ## Checkpoint Verification |
|
|
| All ten source checkpoints were verified by SHA-256 before conversion to clean `model.safetensors` files. The split file hash used by the final run is: |
|
|
| ```text |
| dc26fae17e54fd2ad41a9e10353b3da3e0aacf3144b64f6ee62e8341b4360555 |
| ``` |
|
|
| See `provenance/checkpoint_manifest.json` for per-seed original checkpoint hashes, safetensors hashes, parameter counts, selected validation ensemble alpha, model config hashes, and split hashes. |
|
|
| ## Minimal Loading Example |
|
|
| ```python |
| import json |
| import torch |
| from safetensors.torch import load_file |
| |
| from source.src.bgc_retrieval.model import ModelConfig, build_model |
| |
| seed_dir = "checkpoints/setnet/seed-20260810" |
| config = json.load(open(f"{seed_dir}/config.json"))["model"] |
| model = build_model(ModelConfig.from_dict(config)) |
| state = load_file(f"{seed_dir}/model.safetensors") |
| model.load_state_dict(state) |
| model.eval() |
| ``` |
|
|
| For `pfam_setnet`, inference expects: |
|
|
| - `gene_embeddings`: `[batch, genes, 1280]` frozen ESM-2 gene embeddings |
| - `relative_positions`: `[batch, genes]` normalized gene positions |
| - `padding_mask`: `[batch, genes]` boolean padding mask |
| - `pfam_tokens`: `[batch, domains]` Pfam IDs encoded with `pfam_vocab.json` (`0` = padding, `1` = unknown) |
|
|
| For `weighted_pfam_jaccard`, inference expects only `pfam_tokens` and returns learned domain weights; pairwise retrieval scores are computed with the model's `pairwise_jaccard` method. |
|
|
| ## Intended Use |
|
|
| These models are research artifacts for reproducing and auditing BGC retrieval experiments from the associated manuscript. They are not validated for clinical, diagnostic, regulatory, or production biological decision-making. |
|
|
| ## Limitations |
|
|
| - The benchmark labels are silver labels derived from shared MIBiG references, not a manually curated gold retrieval set. |
| - The split is group-disjoint by MIBiG-reference group, but it is not a complete sequence-homology separation benchmark. |
| - The strongest primary-metric methods are Pfam-based; learned sequence-derived representations did not show statistically supported superiority in this study. |
| - The large input atlas, frozen ESM-2 embeddings, and raw gene annotation tables are not bundled in this model repository. |
| - License terms are pending; contact the authors before reuse beyond inspection and reproducibility work. |
|
|
| ## Citation |
|
|
| Citation metadata will be updated after the preprint is posted. For now, cite the manuscript title above and this Hugging Face repository. |
|
|
