Spaces:
Sleeping
Sleeping
Commit History
-
Merge CRISPR lookup and unknown characterization
598817f -
Update CRISPR benchmark page
a3fb1da
-
Benchmark tab: add Cas-only Pfam-verified reference section
Adds a second benchmark block below the existing Cas+Acr mixed-reference results, using the 175-protein Cas-only reference built from UniProt Pfam-domain xrefs (scripts/analyses/crispr/build_cas_reference_pfam.py). Pfam-based fetch eliminates name-collision false positives that dogged the earlier reference (acriflavin resistance proteins matching AcrIF*, Type-III secretion HrpB matching broad PF09502, etc). Each family is pinned to a verified unique-to-that-family Pfam domain; coverage mixes SwissProt + TrEMBL annotation_score:5. Three panels: UMAP side-by-side, per-family silhouette, four-way aspect comparison + results table. Headline on Cas-only: - Twin-MF wins leave-one-out top-1 (0.874 vs ESM2 0.834, +0.04) and MRR (0.911 vs 0.887). Annotation-quality is better with Twin-MF. - Twin-BP leads 5-NN precision and AUC same-family by a hair. - ESM2 wins silhouette on this set because all 175 are Cas, so its narrow cosine range gives clean relative geometry; silhouette i…29d65cb -
Characterize: add ESM2-vs-Twin-MF signal-strength panel for unknowns
Before the 3-D UMAP gallery view, show a three-panel comparison of ESM2 and Twin-MF on the same 24 uncharacterised proteins: 1. Histogram of top-1 cosine similarities per method 2. Empirical "fraction of unknowns with top-1 cos >= x" (cumulative) 3. Verdict-confidence bucket counts (LOW/MEDIUM/HIGH) Answers directly how much more signal Twin-MF finds on real unknowns than ESM2 does. Plot generated by scripts/analyses/crispr/plot_ unknowns_distribution.py (reads data/unknown_proteins/results.json, re-runnable any time the gallery is rebuilt). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>9907e62 -
Benchmark: UMAP first, Twin-MF as primary, drop emojis, professional tone
- Reorder sections: UMAP now opens the tab (most visually compelling — AcrIF and other Acr families cluster clearly under Twin-MF where they overlap everything under ESM2). Then heatmap, then sub-cluster, then silhouette, then aspect comparison, then takeaways. - Switch the primary Twin variant shown in plots 1 (UMAP) and 2 (heatmap) and 3 (sub-cluster) from BP to MF, since MF is the more fitting aspect for CRISPR (Cas families = enzymatic function). The intro paragraph explains this choice up front. - Regenerate heatmap + sub-cluster + UMAP plots against Twin-MF. - Remove all emojis from the Benchmark tab. - Reword throughout for a more professional, less conversational tone (no "- ✅ ...", no **Surprising result**, no emoji-led bullets). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>4ecf210 -
Benchmark: replace ROC with silhouette; add URL-hash tab deep-linking
Silhouette score is a ratio-based clustering metric, unaffected by the compressed cosine-range issue that inflated ESM2's pairwise-cosine AUC. Replaces the ROC section with a silhouette analysis: model | mean silhouette ----------------+---------------- ESM2 baseline | +0.044 Twin-BP | +0.093 Twin-CC | +0.014 (*worse* than ESM2 — CC isn't useful for CRISPR) Twin-MF | +0.117 (best — ~3x over ESM2) Silhouette is the clearest single-number evidence that Twin-MF is the best aspect for CRISPR family-clustering, and exposes a useful negative result: training on cellular-component GO similarity actively degrades clustering for this task. Also adds a small JS hook on gr.Blocks that reads the URL hash or ?tab= query param on load and clicks the matching tab. Makes https://.../functional-distance#benchmark open directly on the Benchmark tab. Works for any tab name (distance, lookup, benchmark, api, about). Updates the takeaway bullets: drops the stal…cab3c9b -
Extend Benchmark tab: UMAP, ROC, and BP-vs-CC-vs-MF aspect comparison
Three new analyses packaged and wired into the Benchmark tab: 1. UMAP 2D projection (ESM2 left, Twin-BP right), coloured by CRISPR family, marker-shape by group (cas=circle, acr=triangle). Makes clustering visible at a glance. 2. ROC curves for two binary tasks: same-family pair prediction, and same-group (Cas vs Acr) prediction. Surfaces a counter-intuitive result honestly: ESM2 wins same-family AUC (0.85 vs 0.77) because its narrow cosine band avoids rank inversions; Twin's wider range does fine-grained discrimination that doesn't match coarse labels. 3. Compare all three Twin aspects (BP/CC/MF) + ESM2 on four metrics. Key finding: **Twin-MF wins on bulk structure** (within-vs-between separation = 0.37, best Cas-vs-Acr AUC = 0.68), while Twin-BP wins 5-NN retrieval purity. MF matches biological intuition (Cas families are defined by enzymatic function); BP matches the app's current default for retrieval use. Benchmark tab now explains the BP/CC/…019de53 -
Add Benchmark tab: CRISPR clustering comparison ESM2 vs Twin-BP
Package two plots + their CSVs (PNGs via git LFS) and wire a new Benchmark tab between Lookup and API: - data/benchmark/crispr_esm2_vs_twin_heatmap.png (all-vs-all heatmap) - data/benchmark/crispr_subcluster_comparison.png (per-family violins + kNN-purity bars) - data/benchmark/crispr_heatmap_stats.csv, crispr_knn_purity.csv Tab narrative walks through the two views, reports the +0.22 bulk cosine separation and the +0.05 global 5-NN purity, and interprets the AcrIIA "loss" as Twin correctly discriminating a heterogeneous label set. Takeaway section makes the case for using Twin in the app's CRISPR-annotation use case. Also adds *.png to LFS tracking (was missing from the default HF .gitattributes rules; HF Spaces requires binary files via LFS/xet). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>ac2224d