Spaces:
Running
Running
Replace the 2025 benchmark with the revised NTv3 benchmark, keeping the 2025 leaderboard as a page
#2
by ybornachot - opened
- .gitignore +3 -0
- README.md +72 -2
- data/initial_release/ntv3_benchmark_results.csv +0 -0
- data/ntv3_benchmark_results.csv +0 -0
- requirements.txt +1 -15
- src/initial_release.py +1078 -0
- src/streamlit_app.py +562 -893
.gitignore
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
__pycache__/
|
| 2 |
+
*.py[cod]
|
| 3 |
+
.DS_Store
|
README.md
CHANGED
|
@@ -1,5 +1,5 @@
|
|
| 1 |
---
|
| 2 |
-
title:
|
| 3 |
emoji: 🚀
|
| 4 |
colorFrom: red
|
| 5 |
colorTo: red
|
|
@@ -10,4 +10,74 @@ tags:
|
|
| 10 |
pinned: false
|
| 11 |
short_description: NTv3 Benchmark
|
| 12 |
license: apache-2.0
|
| 13 |
-
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
---
|
| 2 |
+
title: NTv3 Benchmark
|
| 3 |
emoji: 🚀
|
| 4 |
colorFrom: red
|
| 5 |
colorTo: red
|
|
|
|
| 10 |
pinned: false
|
| 11 |
short_description: NTv3 Benchmark
|
| 12 |
license: apache-2.0
|
| 13 |
+
---
|
| 14 |
+
|
| 15 |
+
# NTv3 Benchmark
|
| 16 |
+
|
| 17 |
+
A standardized comparison of sequence-to-function models on **155 functional-track
|
| 18 |
+
prediction tasks across 11 species**. Every model is given the same 32-kb input, the same
|
| 19 |
+
task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so
|
| 20 |
+
what is compared is the representation rather than the training recipe.
|
| 21 |
+
|
| 22 |
+
## What is measured
|
| 23 |
+
|
| 24 |
+
**Metric.** Pearson correlation on the held-out test set, per track, averaged over three
|
| 25 |
+
seeds where available.
|
| 26 |
+
|
| 27 |
+
**Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato.
|
| 28 |
+
Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*.
|
| 29 |
+
|
| 30 |
+
**Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription
|
| 31 |
+
factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene
|
| 32 |
+
expression (RNA-seq) and translation (Ribo-seq).
|
| 33 |
+
|
| 34 |
+
The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize,
|
| 35 |
+
which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent
|
| 36 |
+
from that post-training set. The remaining seven genomes were absent from post-training
|
| 37 |
+
entirely, so those species measure transfer to an unseen genome.
|
| 38 |
+
|
| 39 |
+
## Models
|
| 40 |
+
|
| 41 |
+
Ten models: two convolutional baselines trained from scratch, five pretrained genomic
|
| 42 |
+
foundation models, two pretrained sequence-to-function models, and NTv3.
|
| 43 |
+
|
| 44 |
+
| Model | Training |
|
| 45 |
+
| --- | --- |
|
| 46 |
+
| BPNet, Residual CNN | from scratch |
|
| 47 |
+
| HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained |
|
| 48 |
+
| Borzoi, AlphaGenome | pretrained sequence-to-function |
|
| 49 |
+
| **NTv3** | post-trained |
|
| 50 |
+
|
| 51 |
+
**AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds
|
| 52 |
+
checkpoint saw those test regions, and from its all-folds checkpoint on every other species.
|
| 53 |
+
|
| 54 |
+
## Reading the leaderboard
|
| 55 |
+
|
| 56 |
+
**Scores are never averaged across species.** Not every model was run on every species, so a
|
| 57 |
+
single cross-species number would compare models over different task sets. Pick a species in
|
| 58 |
+
the sidebar for its own leaderboard, where every model shown was evaluated on all of that
|
| 59 |
+
species' tracks.
|
| 60 |
+
|
| 61 |
+
## Contents
|
| 62 |
+
|
| 63 |
+
- `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying
|
| 64 |
+
the score and the training-cost columns the app reports. This is the data behind every
|
| 65 |
+
view; nothing is computed elsewhere.
|
| 66 |
+
- `src/streamlit_app.py`, the app.
|
| 67 |
+
|
| 68 |
+
## Links
|
| 69 |
+
|
| 70 |
+
- 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised),
|
| 71 |
+
the genomes, splits and tracks to reproduce or extend the benchmark.
|
| 72 |
+
- 🧬 **NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3).
|
| 73 |
+
- 🗄️ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space.
|
| 74 |
+
|
| 75 |
+
## Citation
|
| 76 |
+
|
| 77 |
+
`TBC`, to be added on publication of the NTv3 paper.
|
| 78 |
+
|
| 79 |
+
## License
|
| 80 |
+
|
| 81 |
+
The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.**
|
| 82 |
+
They are released under the InstaDeep Open Model Licence, which permits non-commercial use
|
| 83 |
+
only and also places conditions on model outputs. See each model repository for its terms.
|
data/initial_release/ntv3_benchmark_results.csv
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
data/ntv3_benchmark_results.csv
CHANGED
|
The diff for this file is too large to render.
See raw diff
|
|
|
requirements.txt
CHANGED
|
@@ -1,18 +1,4 @@
|
|
| 1 |
-
APScheduler
|
| 2 |
-
black
|
| 3 |
-
datasets
|
| 4 |
-
gradio
|
| 5 |
-
gradio[oauth]
|
| 6 |
-
gradio_leaderboard==0.0.9
|
| 7 |
-
gradio_client
|
| 8 |
-
huggingface-hub>=0.18.0
|
| 9 |
-
matplotlib
|
| 10 |
numpy
|
| 11 |
pandas
|
| 12 |
-
python-dateutil
|
| 13 |
-
tqdm
|
| 14 |
-
transformers
|
| 15 |
-
tokenizers>=0.15.0
|
| 16 |
-
sentencepiece
|
| 17 |
plotly
|
| 18 |
-
streamlit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
numpy
|
| 2 |
pandas
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
plotly
|
| 4 |
+
streamlit
|
src/initial_release.py
ADDED
|
@@ -0,0 +1,1078 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from typing import List
|
| 2 |
+
import os
|
| 3 |
+
|
| 4 |
+
import pandas as pd
|
| 5 |
+
import streamlit as st
|
| 6 |
+
import plotly.express as px
|
| 7 |
+
from plotly.subplots import make_subplots
|
| 8 |
+
import plotly.graph_objects as go
|
| 9 |
+
import numpy as np
|
| 10 |
+
|
| 11 |
+
# The leaderboard of the initial NTv3 release (December 2025), as it was published.
|
| 12 |
+
# streamlit_app.py runs main() as one page of the revised benchmark, which owns the page config.
|
| 13 |
+
|
| 14 |
+
# ---------------------------------------------------------------------
|
| 15 |
+
# Configuration
|
| 16 |
+
# ---------------------------------------------------------------------
|
| 17 |
+
COLORS = {
|
| 18 |
+
# Primary colors 1 (our models)
|
| 19 |
+
'blue_0': '#004697', # Darkest allowable blue
|
| 20 |
+
'blue_1': '#3973fc', # Main blue
|
| 21 |
+
'blue_2': '#7ea4fc', # Medium blue
|
| 22 |
+
'blue_3': '#c3d5fc', # Light blue (lightest allowable blue)
|
| 23 |
+
# Secondary colors 1
|
| 24 |
+
'red_1': '#ff554d', # Medium red
|
| 25 |
+
'red_2': '#ffe0de', # Light red
|
| 26 |
+
# Primary colors 2
|
| 27 |
+
'green_1': '#00b050', # Darkest green
|
| 28 |
+
'green_2': '#92d050', # Medium green
|
| 29 |
+
'green_3': '#c6e0b4', # Light green (lightest allowable green)
|
| 30 |
+
# Secondary colors 2
|
| 31 |
+
'gold_1': '#fdb932',
|
| 32 |
+
# Tertiary colors
|
| 33 |
+
'orange_1': '#ff975e',
|
| 34 |
+
'purple_1': '#9a6ce4',
|
| 35 |
+
'purple_2': '#bb9aef', # Medium purple
|
| 36 |
+
'purple_3': '#ceb5f5', # Light purple (lightest allowable purple)
|
| 37 |
+
# Grays (other models)
|
| 38 |
+
'gray_1': '#808080', # Darkest gray (use as a last resort)
|
| 39 |
+
'gray_2': '#b3b3b3', # Medium gray (start with this as the darkest when possible)
|
| 40 |
+
'gray_3': '#e6e6e6', # Lightest gray
|
| 41 |
+
'gray_4': '#ffffff', # It's actually just white (use as a last resort)
|
| 42 |
+
# If all other options are exhausted
|
| 43 |
+
'cyan_1': '#0096b4', # Darkest teal
|
| 44 |
+
'cyan_2': '#28bed2', # Medium cyan
|
| 45 |
+
'cyan_3': '#8cdceb', # Lightest cyan
|
| 46 |
+
'magenta_1': '#b428a0', # Darkest magenta
|
| 47 |
+
'magenta_2': '#dc50be', # Medium pink
|
| 48 |
+
'magenta_3': '#f5a0dc', # Lightest pink
|
| 49 |
+
'yellow_1': '#c8aa00', # Darkest yellow
|
| 50 |
+
'yellow_2': '#ffd200', # Medium yellow
|
| 51 |
+
'yellow_3': '#fff08c', # Lightest yellow
|
| 52 |
+
}
|
| 53 |
+
|
| 54 |
+
|
| 55 |
+
ASSAY_TYPE_MAPPING = {
|
| 56 |
+
'ATAC-seq': 'chromatin accessibility',
|
| 57 |
+
'DNase-seq': 'chromatin accessibility',
|
| 58 |
+
'Histone ChIP-seq': 'histone modifications',
|
| 59 |
+
'TF ChIP-seq': 'chromatin accessibility',
|
| 60 |
+
'PRO-cap': 'transcription initiation',
|
| 61 |
+
'eCLIP': 'RNA binding sites',
|
| 62 |
+
'RNA-seq': 'gene expression',
|
| 63 |
+
'ribo-seq': 'mRNA translation',
|
| 64 |
+
'Annotation': 'genome annotation',
|
| 65 |
+
"Exon": "exon",
|
| 66 |
+
"Intron": "intron",
|
| 67 |
+
"Splice acceptor": "splice acceptor",
|
| 68 |
+
"Start codon": "start codon",
|
| 69 |
+
|
| 70 |
+
}
|
| 71 |
+
|
| 72 |
+
ASSAY_COLORS = {
|
| 73 |
+
'chromatin accessibility': '#004697',
|
| 74 |
+
'histone modifications': '#cc0000',
|
| 75 |
+
'transcription initiation': '#ff9900',
|
| 76 |
+
'RNA binding sites': '#9933cc',
|
| 77 |
+
'gene expression': '#009900',
|
| 78 |
+
'mRNA translation': '#ff6699',
|
| 79 |
+
'genome annotation': '#ffcc00',
|
| 80 |
+
"intron": '#004697',
|
| 81 |
+
"exon": '#cc0000',
|
| 82 |
+
"splice acceptor": '#ff9900',
|
| 83 |
+
"start codon": '#9933cc',
|
| 84 |
+
}
|
| 85 |
+
ASSAY_COLORS["other"] = "#808080"
|
| 86 |
+
|
| 87 |
+
MODEL_COLORS = {
|
| 88 |
+
"NTv3 650M (pos)": COLORS['blue_0'],
|
| 89 |
+
'NTv3 650M (pre)': COLORS['blue_1'], # #3973fc (Darkest blue)
|
| 90 |
+
'NTv3 100M (pre)': COLORS['blue_2'], # #7ea4fc (Medium blue)
|
| 91 |
+
'NTv3 8M (pre)': COLORS['blue_3'], # #c3d5fc (Light blue)
|
| 92 |
+
'Evo2 1B': COLORS['green_3'], # #b3b3b3 (Medium gray)
|
| 93 |
+
"NTv2 500M": COLORS['gray_1'],
|
| 94 |
+
"BPNet arch. 6M": COLORS['cyan_1'],
|
| 95 |
+
"Residual CNN 44M": COLORS['magenta_1'],
|
| 96 |
+
"PlantCAD2 88M": COLORS["purple_1"],
|
| 97 |
+
"Caduceus 7M": COLORS["purple_2"],
|
| 98 |
+
"HyenaDNA 7M": COLORS["yellow_2"]
|
| 99 |
+
}
|
| 100 |
+
|
| 101 |
+
MODEL_TRAINING_STATUS = {
|
| 102 |
+
"NTv3 650M (pos)": "POS",
|
| 103 |
+
"NTv3 650M (pre)": "PRE",
|
| 104 |
+
"NTv3 100M (pre)": "PRE",
|
| 105 |
+
"NTv3 8M (pre)": "PRE",
|
| 106 |
+
"Residual CNN 44M": "SCRATCH",
|
| 107 |
+
"Caduceus 7M": "PRE",
|
| 108 |
+
"Evo2 1B": "PRE",
|
| 109 |
+
"NTv2 500M": "PRE",
|
| 110 |
+
"BPNet arch. 6M": "SCRATCH",
|
| 111 |
+
"PlantCAD2 88M": "PRE",
|
| 112 |
+
"HyenaDNA 7M": "PRE"
|
| 113 |
+
}
|
| 114 |
+
|
| 115 |
+
MODEL_GPU_MULTIPLIER = {
|
| 116 |
+
"Evo2 1B": 8, # trained on 8 GPUs
|
| 117 |
+
}
|
| 118 |
+
|
| 119 |
+
|
| 120 |
+
MODEL_NAMES = list(MODEL_COLORS.keys())
|
| 121 |
+
|
| 122 |
+
PLANT_SPECIES = ["tomato", "rice", "maize", "arabidopsis"]
|
| 123 |
+
ANIMAL_SPECIES = ["human", "chicken", "cattle"]
|
| 124 |
+
|
| 125 |
+
SPECIES_GROUPS = {
|
| 126 |
+
"Plants": PLANT_SPECIES,
|
| 127 |
+
"Animals": ANIMAL_SPECIES, # (your code calls these HUMAN_SPECIES, but they’re the “animal” set)
|
| 128 |
+
}
|
| 129 |
+
|
| 130 |
+
|
| 131 |
+
_LAST_UPDATED = "Dec 10, 2025"
|
| 132 |
+
_INTRO = """
|
| 133 |
+
The **NTv3 Benchmark** is a curated benchmark of 106 long-range genomic datasets
|
| 134 |
+
designed to evaluate models under realistic 32 kb input, single-base-pair output settings.
|
| 135 |
+
The dataset spans two complementary task families: genome annotation (exon, intron, splice acceptor, start codon)
|
| 136 |
+
and functional-regulatory prediction, which includes diverse experimental tracks such as chromatin accessibility,
|
| 137 |
+
histone modifications, transcription initiation (PRO-cap), RNA binding (eCLIP), gene expression (RNA-seq),
|
| 138 |
+
and translation (Ribo-seq).
|
| 139 |
+
|
| 140 |
+
Data are drawn from a phylogenetically diverse set of species, including organisms seen during post-training
|
| 141 |
+
(human, chicken, arabidopsis, rice, maize) and entirely unseen species (cattle, tomato), with careful curation
|
| 142 |
+
to avoid data leakage. This design allows the dataset to probe long-range sequence-to-function mapping,
|
| 143 |
+
cross-species generalization, and transfer across heterogeneous regulatory modalities,
|
| 144 |
+
including assays not present in prior multispecies training corpora. By standardizing sequence length,
|
| 145 |
+
resolution, and evaluation metrics across all tracks, the NTv3 Benchmark provides a controlled dataset
|
| 146 |
+
for comparing representation quality across genomic foundation models.
|
| 147 |
+
|
| 148 |
+
The metrics used are:
|
| 149 |
+
- **Pearson correlations (multi-assay)**: per-dataset scores across species and models for functional tracks.
|
| 150 |
+
- **MCC (bed tracks)**: per-track MCC values across species and models for gene annotation tracks.
|
| 151 |
+
"""
|
| 152 |
+
|
| 153 |
+
HERE = os.path.dirname(os.path.abspath(__file__)) # /app/src
|
| 154 |
+
PROJECT_ROOT = os.path.dirname(HERE) # /app
|
| 155 |
+
DATA_DIR = os.path.join(PROJECT_ROOT, "data")
|
| 156 |
+
|
| 157 |
+
SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "initial_release", "ntv3_benchmark_results.csv")
|
| 158 |
+
|
| 159 |
+
# ---------------------------------------------------------------------
|
| 160 |
+
# Data loading & preprocessing
|
| 161 |
+
# ---------------------------------------------------------------------
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
@st.cache_data
|
| 165 |
+
def load_raw_data():
|
| 166 |
+
df = pd.read_csv(SINGLE_TABLE_PATH)
|
| 167 |
+
df.columns = [c.strip() for c in df.columns]
|
| 168 |
+
return df
|
| 169 |
+
|
| 170 |
+
|
| 171 |
+
def _normalize_training_time_to_gpu_hours(df: pd.DataFrame) -> pd.DataFrame:
|
| 172 |
+
"""
|
| 173 |
+
Your new column is `running_time`. In your sample it looks like seconds
|
| 174 |
+
(e.g. 317034 ~= 88 hours). We'll convert to hours if values look like seconds.
|
| 175 |
+
"""
|
| 176 |
+
if "running_time" not in df.columns:
|
| 177 |
+
return df
|
| 178 |
+
|
| 179 |
+
rt = pd.to_numeric(df["running_time"], errors="coerce")
|
| 180 |
+
# Heuristic: if median is huge, it's probably seconds -> convert to hours
|
| 181 |
+
# (88 hours = 316800 seconds is a typical-looking value in your sample)
|
| 182 |
+
if rt.dropna().median() > 10_000:
|
| 183 |
+
df["GPU hours"] = rt / 3600.0
|
| 184 |
+
else:
|
| 185 |
+
df["GPU hours"] = rt.astype(float)
|
| 186 |
+
|
| 187 |
+
return df
|
| 188 |
+
|
| 189 |
+
|
| 190 |
+
def _best_step_time_to_hours(s: pd.Series) -> pd.Series:
|
| 191 |
+
"""
|
| 192 |
+
Converts strings like '3 days 04:26:26.467000' to hours (float).
|
| 193 |
+
Works with pandas Timedelta parsing.
|
| 194 |
+
"""
|
| 195 |
+
td = pd.to_timedelta(s, errors="coerce")
|
| 196 |
+
return td.dt.total_seconds() / 3600.0
|
| 197 |
+
|
| 198 |
+
|
| 199 |
+
|
| 200 |
+
@st.cache_data
|
| 201 |
+
def load_expanded_data():
|
| 202 |
+
df = load_raw_data().copy()
|
| 203 |
+
|
| 204 |
+
df = df.rename(columns={"Metric": "Score", "model_name": "Model"})
|
| 205 |
+
df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
|
| 206 |
+
|
| 207 |
+
if "best_step" in df.columns:
|
| 208 |
+
df["best_step"] = pd.to_numeric(df["best_step"], errors="coerce")
|
| 209 |
+
|
| 210 |
+
if "best_step_time" in df.columns:
|
| 211 |
+
df["best_step_time_hours"] = _best_step_time_to_hours(df["best_step_time"])
|
| 212 |
+
else:
|
| 213 |
+
df["best_step_time_hours"] = np.nan
|
| 214 |
+
|
| 215 |
+
is_annot = df.get("assay_type", "").astype(str).eq("Annotation")
|
| 216 |
+
pearson_raw = df[~is_annot].copy()
|
| 217 |
+
mcc_raw = df[is_annot].copy()
|
| 218 |
+
|
| 219 |
+
# -------------------------
|
| 220 |
+
# Functional Tracks (Pearson)
|
| 221 |
+
# -------------------------
|
| 222 |
+
pearson_group_cols = ["species", "datasets", "Model"]
|
| 223 |
+
if "assay_type" in pearson_raw.columns:
|
| 224 |
+
pearson_group_cols.append("assay_type")
|
| 225 |
+
|
| 226 |
+
pearson_df = (
|
| 227 |
+
pearson_raw
|
| 228 |
+
.groupby(pearson_group_cols, as_index=False, dropna=False)
|
| 229 |
+
.agg({
|
| 230 |
+
"Score": "mean",
|
| 231 |
+
"best_step": "mean",
|
| 232 |
+
"best_step_time_hours": "mean",
|
| 233 |
+
})
|
| 234 |
+
)
|
| 235 |
+
|
| 236 |
+
# ✅ merge track_name_clean WHILE assay_type is still raw
|
| 237 |
+
if "track_name_clean" in pearson_raw.columns:
|
| 238 |
+
map_keys = ["species", "datasets"]
|
| 239 |
+
if "assay_type" in pearson_raw.columns:
|
| 240 |
+
map_keys.append("assay_type")
|
| 241 |
+
|
| 242 |
+
track_map = (
|
| 243 |
+
pearson_raw[map_keys + ["track_name_clean"]]
|
| 244 |
+
.dropna(subset=["track_name_clean"])
|
| 245 |
+
.drop_duplicates()
|
| 246 |
+
)
|
| 247 |
+
pearson_df = pearson_df.merge(track_map, on=map_keys, how="left")
|
| 248 |
+
|
| 249 |
+
# ✅ now it’s safe to map assay_type to categories
|
| 250 |
+
if "assay_type" in pearson_df.columns:
|
| 251 |
+
pearson_df["assay_type"] = (
|
| 252 |
+
pearson_df["assay_type"].astype(str).map(ASSAY_TYPE_MAPPING).fillna("Other")
|
| 253 |
+
)
|
| 254 |
+
|
| 255 |
+
# -------------------------
|
| 256 |
+
# Genome Annotation (MCC)
|
| 257 |
+
# -------------------------
|
| 258 |
+
mcc_df = (
|
| 259 |
+
mcc_raw
|
| 260 |
+
.groupby(["species", "datasets", "Model"], as_index=False, dropna=False)
|
| 261 |
+
.agg({
|
| 262 |
+
"Score": "mean",
|
| 263 |
+
"best_step": "mean",
|
| 264 |
+
"best_step_time_hours": "mean",
|
| 265 |
+
})
|
| 266 |
+
)
|
| 267 |
+
|
| 268 |
+
return pearson_df, mcc_df
|
| 269 |
+
|
| 270 |
+
|
| 271 |
+
_PEARSON_DF, _MCC_DF = load_expanded_data()
|
| 272 |
+
|
| 273 |
+
# Global sets (we'll further filter per-benchmark below)
|
| 274 |
+
_ALL_SPECIES = sorted(
|
| 275 |
+
set(_PEARSON_DF["species"].unique()).union(_MCC_DF["species"].unique())
|
| 276 |
+
)
|
| 277 |
+
|
| 278 |
+
_ALL_ASSAYS = (
|
| 279 |
+
sorted(_PEARSON_DF["assay_type"].dropna().unique())
|
| 280 |
+
if "assay_type" in _PEARSON_DF.columns
|
| 281 |
+
else []
|
| 282 |
+
)
|
| 283 |
+
|
| 284 |
+
_ALL_MODELS = MODEL_NAMES[:]
|
| 285 |
+
|
| 286 |
+
_BENCHMARKS = {
|
| 287 |
+
"Functional Tracks": {
|
| 288 |
+
"df": _PEARSON_DF,
|
| 289 |
+
"metric_label": "Pearson correlation",
|
| 290 |
+
"has_assay_type": True,
|
| 291 |
+
},
|
| 292 |
+
"Genome Annotation": {
|
| 293 |
+
"df": _MCC_DF,
|
| 294 |
+
"metric_label": "MCC",
|
| 295 |
+
"has_assay_type": False,
|
| 296 |
+
},
|
| 297 |
+
}
|
| 298 |
+
|
| 299 |
+
|
| 300 |
+
# ---------------------------------------------------------------------
|
| 301 |
+
# Computation helpers
|
| 302 |
+
# ---------------------------------------------------------------------
|
| 303 |
+
|
| 304 |
+
|
| 305 |
+
def filter_base_df(
|
| 306 |
+
benchmark_name: str,
|
| 307 |
+
selected_species: List[str],
|
| 308 |
+
selected_assays: List[str],
|
| 309 |
+
selected_models: List[str],
|
| 310 |
+
selected_datasets: List[str],
|
| 311 |
+
) -> pd.DataFrame:
|
| 312 |
+
cfg = _BENCHMARKS[benchmark_name]
|
| 313 |
+
df = cfg["df"].copy()
|
| 314 |
+
|
| 315 |
+
# Species filter
|
| 316 |
+
if selected_species:
|
| 317 |
+
df = df[df["species"].isin(selected_species)]
|
| 318 |
+
|
| 319 |
+
# Assay type filter (Pearson only)
|
| 320 |
+
if cfg.get("has_assay_type", False) and selected_assays and "assay_type" in df.columns:
|
| 321 |
+
df = df[df["assay_type"].isin(selected_assays)]
|
| 322 |
+
|
| 323 |
+
# Dataset / bed track filter (for MCC, but safe to apply generally)
|
| 324 |
+
if selected_datasets and "datasets" in df.columns:
|
| 325 |
+
df = df[df["datasets"].isin(selected_datasets)]
|
| 326 |
+
|
| 327 |
+
# Model filter
|
| 328 |
+
if selected_models:
|
| 329 |
+
df = df[df["Model"].isin(selected_models)]
|
| 330 |
+
|
| 331 |
+
return df
|
| 332 |
+
|
| 333 |
+
|
| 334 |
+
def build_leaderboard(
|
| 335 |
+
benchmark_name: str,
|
| 336 |
+
selected_species: List[str],
|
| 337 |
+
selected_assays: List[str],
|
| 338 |
+
selected_models: List[str],
|
| 339 |
+
selected_datasets: List[str],
|
| 340 |
+
) -> pd.DataFrame:
|
| 341 |
+
df = filter_base_df(
|
| 342 |
+
benchmark_name,
|
| 343 |
+
selected_species,
|
| 344 |
+
selected_assays,
|
| 345 |
+
selected_models,
|
| 346 |
+
selected_datasets,
|
| 347 |
+
)
|
| 348 |
+
|
| 349 |
+
if df.empty:
|
| 350 |
+
return pd.DataFrame(columns=["Model", "Model Type", "Num entries", "Mean score"])
|
| 351 |
+
|
| 352 |
+
agg = (
|
| 353 |
+
df.groupby("Model")["Score"]
|
| 354 |
+
.mean()
|
| 355 |
+
.reset_index()
|
| 356 |
+
.rename(columns={"Score": "Mean score"})
|
| 357 |
+
)
|
| 358 |
+
agg["Mean score"] = agg["Mean score"].round(3)
|
| 359 |
+
|
| 360 |
+
agg["Num entries"] = (
|
| 361 |
+
df.groupby("Model")["Score"].count().reindex(agg["Model"]).values
|
| 362 |
+
)
|
| 363 |
+
|
| 364 |
+
# 👇 Add training regime column
|
| 365 |
+
agg["Training"] = agg["Model"].map(MODEL_TRAINING_STATUS).fillna("UNKNOWN")
|
| 366 |
+
|
| 367 |
+
# Sort by performance
|
| 368 |
+
agg = agg.sort_values("Mean score", ascending=False).reset_index(drop=True)
|
| 369 |
+
|
| 370 |
+
# Column order
|
| 371 |
+
agg = agg[["Model", "Training", "Num entries", "Mean score"]]
|
| 372 |
+
|
| 373 |
+
# Ensure the index starts with 1
|
| 374 |
+
agg.index += 1
|
| 375 |
+
|
| 376 |
+
return agg
|
| 377 |
+
|
| 378 |
+
|
| 379 |
+
|
| 380 |
+
def build_bar_df(
|
| 381 |
+
benchmark_name: str,
|
| 382 |
+
selected_species: List[str],
|
| 383 |
+
selected_assays: List[str],
|
| 384 |
+
selected_models: List[str],
|
| 385 |
+
selected_datasets: List[str],
|
| 386 |
+
) -> pd.DataFrame:
|
| 387 |
+
"""For now, just one bar per model (same as leaderboard)."""
|
| 388 |
+
return build_leaderboard(
|
| 389 |
+
benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
|
| 390 |
+
)
|
| 391 |
+
|
| 392 |
+
|
| 393 |
+
def build_category_model_df(
|
| 394 |
+
benchmark_name: str,
|
| 395 |
+
selected_species: List[str],
|
| 396 |
+
selected_assays: List[str],
|
| 397 |
+
selected_models: List[str],
|
| 398 |
+
selected_datasets: List[str],
|
| 399 |
+
) -> pd.DataFrame:
|
| 400 |
+
"""
|
| 401 |
+
Mean score per (category, Model) after applying the same filters.
|
| 402 |
+
Category = assay_type (Functional Tracks) or datasets (Genome Annotation).
|
| 403 |
+
"""
|
| 404 |
+
cfg = _BENCHMARKS[benchmark_name]
|
| 405 |
+
df = filter_base_df(
|
| 406 |
+
benchmark_name,
|
| 407 |
+
selected_species,
|
| 408 |
+
selected_assays,
|
| 409 |
+
selected_models,
|
| 410 |
+
selected_datasets,
|
| 411 |
+
)
|
| 412 |
+
if df.empty:
|
| 413 |
+
return pd.DataFrame(columns=["Category", "Model", "Mean score"])
|
| 414 |
+
|
| 415 |
+
# Pick the right breakdown column
|
| 416 |
+
if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
|
| 417 |
+
category_col = "assay_type"
|
| 418 |
+
category_label = "Assay type"
|
| 419 |
+
else:
|
| 420 |
+
category_col = "datasets"
|
| 421 |
+
category_label = "Dataset"
|
| 422 |
+
|
| 423 |
+
if category_col not in df.columns:
|
| 424 |
+
return pd.DataFrame(columns=["Category", "Model", "Mean score"])
|
| 425 |
+
|
| 426 |
+
out = (
|
| 427 |
+
df.groupby([category_col, "Model"], as_index=False)["Score"]
|
| 428 |
+
.mean()
|
| 429 |
+
.rename(columns={category_col: "Category", "Score": "Mean score"})
|
| 430 |
+
)
|
| 431 |
+
out["Mean score"] = out["Mean score"].round(3)
|
| 432 |
+
out.attrs["category_label"] = category_label # for nicer axis title
|
| 433 |
+
return out
|
| 434 |
+
|
| 435 |
+
|
| 436 |
+
def plot_breakdown_facets_sorted_models(
|
| 437 |
+
breakdown_df: pd.DataFrame,
|
| 438 |
+
metric_label: str,
|
| 439 |
+
height: int = 420,
|
| 440 |
+
):
|
| 441 |
+
categories = list(breakdown_df["Category"].dropna().unique())
|
| 442 |
+
categories = sorted(categories)
|
| 443 |
+
|
| 444 |
+
n = len(categories)
|
| 445 |
+
if n == 0:
|
| 446 |
+
return None
|
| 447 |
+
|
| 448 |
+
rows = 1
|
| 449 |
+
cols = n # 👈 everything in one row
|
| 450 |
+
|
| 451 |
+
# Global y-range (consistent scale)
|
| 452 |
+
y_min = breakdown_df["Mean score"].min()
|
| 453 |
+
y_max = breakdown_df["Mean score"].max()
|
| 454 |
+
pad = 0.05 * (y_max - y_min if y_max > y_min else 1.0)
|
| 455 |
+
y_range = [y_min - pad, y_max + pad]
|
| 456 |
+
|
| 457 |
+
fig = make_subplots(
|
| 458 |
+
rows=rows,
|
| 459 |
+
cols=cols,
|
| 460 |
+
subplot_titles=categories,
|
| 461 |
+
shared_yaxes=True,
|
| 462 |
+
horizontal_spacing=0.04, # tighter spacing
|
| 463 |
+
)
|
| 464 |
+
|
| 465 |
+
for i, cat in enumerate(categories):
|
| 466 |
+
r = (i // cols) + 1
|
| 467 |
+
c = (i % cols) + 1
|
| 468 |
+
|
| 469 |
+
sub = (
|
| 470 |
+
breakdown_df[breakdown_df["Category"] == cat]
|
| 471 |
+
.sort_values("Mean score", ascending=True)
|
| 472 |
+
)
|
| 473 |
+
|
| 474 |
+
fig.add_trace(
|
| 475 |
+
go.Bar(
|
| 476 |
+
x=sub["Model"],
|
| 477 |
+
y=sub["Mean score"],
|
| 478 |
+
marker_color=[MODEL_COLORS.get(m, "#808080") for m in sub["Model"]],
|
| 479 |
+
showlegend=False,
|
| 480 |
+
),
|
| 481 |
+
row=r,
|
| 482 |
+
col=c,
|
| 483 |
+
)
|
| 484 |
+
|
| 485 |
+
fig.update_xaxes(showticklabels=False, title_text="", row=r, col=c)
|
| 486 |
+
fig.update_yaxes(range=y_range, title_text="", row=r, col=c) # 👈 apply range
|
| 487 |
+
|
| 488 |
+
fig.update_layout(
|
| 489 |
+
height=height,
|
| 490 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 491 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 492 |
+
margin=dict(t=60, l=10, r=10, b=10),
|
| 493 |
+
)
|
| 494 |
+
|
| 495 |
+
# Single y-axis label on the leftmost panel
|
| 496 |
+
fig.update_yaxes(title_text=metric_label, row=1, col=1)
|
| 497 |
+
|
| 498 |
+
return fig
|
| 499 |
+
|
| 500 |
+
def build_pairwise_scatter_df(
|
| 501 |
+
benchmark_name: str,
|
| 502 |
+
selected_species: List[str],
|
| 503 |
+
selected_assays: List[str],
|
| 504 |
+
selected_models: List[str],
|
| 505 |
+
selected_datasets: List[str],
|
| 506 |
+
model_a: str,
|
| 507 |
+
model_b: str,
|
| 508 |
+
) -> pd.DataFrame:
|
| 509 |
+
cfg = _BENCHMARKS[benchmark_name]
|
| 510 |
+
|
| 511 |
+
models_for_filter = (
|
| 512 |
+
list(set(selected_models + [model_a, model_b]))
|
| 513 |
+
if selected_models else [model_a, model_b]
|
| 514 |
+
)
|
| 515 |
+
|
| 516 |
+
df = filter_base_df(
|
| 517 |
+
benchmark_name,
|
| 518 |
+
selected_species,
|
| 519 |
+
selected_assays,
|
| 520 |
+
models_for_filter,
|
| 521 |
+
selected_datasets,
|
| 522 |
+
)
|
| 523 |
+
if df.empty:
|
| 524 |
+
return pd.DataFrame()
|
| 525 |
+
|
| 526 |
+
# ---- define "track identity" for head-to-head ----
|
| 527 |
+
# Always use datasets for the identity (x/y points)
|
| 528 |
+
track_cols = ["datasets"]
|
| 529 |
+
if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
|
| 530 |
+
track_cols = ["assay_type", "datasets"]
|
| 531 |
+
|
| 532 |
+
keep_species = "species" in df.columns and (selected_species is None or len(selected_species) != 1)
|
| 533 |
+
id_cols = (["species"] if keep_species else []) + track_cols
|
| 534 |
+
|
| 535 |
+
wide = (
|
| 536 |
+
df[df["Model"].isin([model_a, model_b])]
|
| 537 |
+
.pivot_table(index=id_cols, columns="Model", values="Score", aggfunc="mean")
|
| 538 |
+
.reset_index()
|
| 539 |
+
)
|
| 540 |
+
|
| 541 |
+
if model_a not in wide.columns or model_b not in wide.columns:
|
| 542 |
+
return pd.DataFrame()
|
| 543 |
+
|
| 544 |
+
wide = wide.dropna(subset=[model_a, model_b])
|
| 545 |
+
|
| 546 |
+
# Nice display label: use datasets (not track_name_clean)
|
| 547 |
+
if "assay_type" in wide.columns:
|
| 548 |
+
wide["Track"] = wide["assay_type"].astype(str) + " / " + wide["datasets"].astype(str)
|
| 549 |
+
else:
|
| 550 |
+
wide["Track"] = wide["datasets"].astype(str)
|
| 551 |
+
|
| 552 |
+
wide = wide.rename(columns={model_a: "Model A", model_b: "Model B"})
|
| 553 |
+
|
| 554 |
+
# ---- Pearson-only: merge track_name_clean for hover ----
|
| 555 |
+
if benchmark_name == "Functional Tracks" and "track_name_clean" in df.columns:
|
| 556 |
+
merge_keys = id_cols.copy() # species? + assay_type? + datasets
|
| 557 |
+
track_map = (
|
| 558 |
+
df[merge_keys + ["track_name_clean"]]
|
| 559 |
+
.dropna(subset=["track_name_clean"])
|
| 560 |
+
.drop_duplicates()
|
| 561 |
+
)
|
| 562 |
+
wide = wide.merge(track_map, on=merge_keys, how="left")
|
| 563 |
+
|
| 564 |
+
return wide
|
| 565 |
+
|
| 566 |
+
|
| 567 |
+
|
| 568 |
+
def build_violin_df(
|
| 569 |
+
benchmark_name: str,
|
| 570 |
+
selected_species: List[str],
|
| 571 |
+
selected_assays: List[str],
|
| 572 |
+
selected_models: List[str],
|
| 573 |
+
selected_datasets: List[str],
|
| 574 |
+
) -> pd.DataFrame:
|
| 575 |
+
# Use the same base filtering, but keep all per-track rows
|
| 576 |
+
df = filter_base_df(
|
| 577 |
+
benchmark_name,
|
| 578 |
+
selected_species,
|
| 579 |
+
selected_assays,
|
| 580 |
+
selected_models,
|
| 581 |
+
selected_datasets,
|
| 582 |
+
)
|
| 583 |
+
# Keep only needed columns
|
| 584 |
+
keep = ["Model", "Score"]
|
| 585 |
+
for col in ["species", "assay_type", "datasets"]:
|
| 586 |
+
if col in df.columns:
|
| 587 |
+
keep.append(col)
|
| 588 |
+
return df[keep].copy()
|
| 589 |
+
|
| 590 |
+
|
| 591 |
+
def build_convergence_df(
|
| 592 |
+
benchmark_name: str,
|
| 593 |
+
selected_species: List[str],
|
| 594 |
+
selected_assays: List[str],
|
| 595 |
+
selected_models: List[str],
|
| 596 |
+
selected_datasets: List[str],
|
| 597 |
+
x_mode: str = "best_step", # "best_step" | "best_step_time"
|
| 598 |
+
) -> pd.DataFrame:
|
| 599 |
+
|
| 600 |
+
df = filter_base_df(
|
| 601 |
+
benchmark_name,
|
| 602 |
+
selected_species,
|
| 603 |
+
selected_assays,
|
| 604 |
+
selected_models,
|
| 605 |
+
selected_datasets,
|
| 606 |
+
)
|
| 607 |
+
|
| 608 |
+
if df.empty:
|
| 609 |
+
return pd.DataFrame(columns=["Model", "X", "Performance"])
|
| 610 |
+
|
| 611 |
+
# Mean performance per model
|
| 612 |
+
out = (
|
| 613 |
+
df.groupby("Model", as_index=False)
|
| 614 |
+
.agg({"Score": "mean"})
|
| 615 |
+
.rename(columns={"Score": "Performance"})
|
| 616 |
+
)
|
| 617 |
+
|
| 618 |
+
# -------------------------
|
| 619 |
+
# X axis selection
|
| 620 |
+
# -------------------------
|
| 621 |
+
if x_mode == "Steps (billions)":
|
| 622 |
+
if "best_step" not in df.columns:
|
| 623 |
+
return pd.DataFrame(columns=["Model", "X", "Performance"])
|
| 624 |
+
|
| 625 |
+
x = (
|
| 626 |
+
df.groupby("Model", as_index=False)["best_step"]
|
| 627 |
+
.mean()
|
| 628 |
+
.rename(columns={"best_step": "X"})
|
| 629 |
+
)
|
| 630 |
+
|
| 631 |
+
else: # best_step_time (GPU hours)
|
| 632 |
+
if "best_step_time_hours" not in df.columns:
|
| 633 |
+
return pd.DataFrame(columns=["Model", "X", "Performance"])
|
| 634 |
+
|
| 635 |
+
x = (
|
| 636 |
+
df.groupby("Model", as_index=False)["best_step_time_hours"]
|
| 637 |
+
.mean()
|
| 638 |
+
.rename(columns={"best_step_time_hours": "X"})
|
| 639 |
+
)
|
| 640 |
+
|
| 641 |
+
# 👇 Apply GPU multiplier (Evo2 uses 8 GPUs)
|
| 642 |
+
gpu_multiplier = {
|
| 643 |
+
"Evo2 1B": 8,
|
| 644 |
+
}
|
| 645 |
+
|
| 646 |
+
x["X"] = x.apply(
|
| 647 |
+
lambda r: r["X"] * gpu_multiplier.get(r["Model"], 1),
|
| 648 |
+
axis=1,
|
| 649 |
+
)
|
| 650 |
+
|
| 651 |
+
# Merge + clean
|
| 652 |
+
out = out.merge(x, on="Model", how="left")
|
| 653 |
+
out = out.dropna(subset=["X", "Performance"])
|
| 654 |
+
|
| 655 |
+
out["Performance"] = out["Performance"].round(3)
|
| 656 |
+
|
| 657 |
+
return out
|
| 658 |
+
|
| 659 |
+
|
| 660 |
+
# ---------------------------------------------------------------------
|
| 661 |
+
# UI helpers
|
| 662 |
+
# ---------------------------------------------------------------------
|
| 663 |
+
|
| 664 |
+
|
| 665 |
+
def sidebar_toggle(label: str, value: bool = False, key: str | None = None) -> bool:
|
| 666 |
+
"""
|
| 667 |
+
Wrapper to use st.sidebar.toggle when available, otherwise fall back to checkbox.
|
| 668 |
+
This makes the app compatible with older Streamlit versions on Hugging Face.
|
| 669 |
+
"""
|
| 670 |
+
toggle_fn = getattr(st.sidebar, "toggle", None)
|
| 671 |
+
if toggle_fn is not None:
|
| 672 |
+
return toggle_fn(label, value=value, key=key)
|
| 673 |
+
# Fallback for older Streamlit versions
|
| 674 |
+
return st.sidebar.checkbox(label, value=value, key=key)
|
| 675 |
+
|
| 676 |
+
# ---------------------------------------------------------------------
|
| 677 |
+
# Streamlit UI
|
| 678 |
+
# ---------------------------------------------------------------------
|
| 679 |
+
def main():
|
| 680 |
+
st.title("🗄️ NTv3 Benchmark: initial release (2025)")
|
| 681 |
+
st.info(
|
| 682 |
+
"The leaderboard of the initial NTv3 release, as published in December 2025. "
|
| 683 |
+
"The other pages show the revised release."
|
| 684 |
+
)
|
| 685 |
+
st.markdown(_INTRO)
|
| 686 |
+
st.markdown(f"_Last updated: **{_LAST_UPDATED}**_")
|
| 687 |
+
|
| 688 |
+
# --- Sidebar filters ---
|
| 689 |
+
st.sidebar.header("Filters")
|
| 690 |
+
|
| 691 |
+
# Benchmark
|
| 692 |
+
benchmark_name = st.sidebar.selectbox(
|
| 693 |
+
"Benchmark",
|
| 694 |
+
options=list(_BENCHMARKS.keys()),
|
| 695 |
+
index=0,
|
| 696 |
+
)
|
| 697 |
+
|
| 698 |
+
cfg = _BENCHMARKS[benchmark_name]
|
| 699 |
+
df_bench = cfg["df"]
|
| 700 |
+
|
| 701 |
+
# Species toggles, but only for species present in this benchmark
|
| 702 |
+
st.sidebar.subheader("Species")
|
| 703 |
+
|
| 704 |
+
# Toggle: Plants vs Animals
|
| 705 |
+
species_group = st.sidebar.radio(
|
| 706 |
+
"Group",
|
| 707 |
+
options=["Animals", "Plants"],
|
| 708 |
+
index=0,
|
| 709 |
+
horizontal=True,
|
| 710 |
+
key=f"species_group_{benchmark_name}",
|
| 711 |
+
)
|
| 712 |
+
|
| 713 |
+
available_species_all = sorted(df_bench["species"].unique())
|
| 714 |
+
allowed_species = set(SPECIES_GROUPS[species_group]).intersection(available_species_all)
|
| 715 |
+
available_species = sorted(allowed_species)
|
| 716 |
+
|
| 717 |
+
selected_species: List[str] = []
|
| 718 |
+
for sp in available_species:
|
| 719 |
+
if sidebar_toggle(sp, value=True, key=f"species_{benchmark_name}_{species_group}_{sp}"):
|
| 720 |
+
selected_species.append(sp)
|
| 721 |
+
|
| 722 |
+
# (Optional) If no species exist for that group in this benchmark
|
| 723 |
+
if not available_species:
|
| 724 |
+
st.sidebar.info(f"No {species_group.lower()} species available for this benchmark.")
|
| 725 |
+
|
| 726 |
+
# Assay toggles (Pearson only), based on filtered species
|
| 727 |
+
if cfg.get("has_assay_type", False):
|
| 728 |
+
st.sidebar.subheader("Assay types")
|
| 729 |
+
if selected_species:
|
| 730 |
+
df_for_assays = df_bench[df_bench["species"].isin(selected_species)]
|
| 731 |
+
else:
|
| 732 |
+
df_for_assays = df_bench
|
| 733 |
+
available_assays = (
|
| 734 |
+
sorted(df_for_assays["assay_type"].dropna().unique())
|
| 735 |
+
if "assay_type" in df_for_assays.columns
|
| 736 |
+
else []
|
| 737 |
+
)
|
| 738 |
+
selected_assays: List[str] = []
|
| 739 |
+
for assay in available_assays:
|
| 740 |
+
if sidebar_toggle(assay, value=True, key=f"assay_{benchmark_name}_{assay}"):
|
| 741 |
+
selected_assays.append(assay)
|
| 742 |
+
else:
|
| 743 |
+
selected_assays = []
|
| 744 |
+
|
| 745 |
+
# Bed track / dataset toggles (MCC only), based on species selection
|
| 746 |
+
selected_datasets: List[str] = []
|
| 747 |
+
if benchmark_name == "Genome Annotation":
|
| 748 |
+
st.sidebar.subheader("Genome annotations")
|
| 749 |
+
if selected_species:
|
| 750 |
+
df_for_tracks = df_bench[df_bench["species"].isin(selected_species)]
|
| 751 |
+
else:
|
| 752 |
+
df_for_tracks = df_bench
|
| 753 |
+
available_datasets = sorted(df_for_tracks["datasets"].unique())
|
| 754 |
+
for ds in available_datasets:
|
| 755 |
+
if sidebar_toggle(ds, value=True, key=f"dataset_{benchmark_name}_{ds}"):
|
| 756 |
+
selected_datasets.append(ds)
|
| 757 |
+
else:
|
| 758 |
+
selected_datasets = []
|
| 759 |
+
|
| 760 |
+
# Model toggles (we keep all models in MODEL_NAMES; filters + data will prune)
|
| 761 |
+
st.sidebar.subheader("Models")
|
| 762 |
+
selected_models: List[str] = []
|
| 763 |
+
for model in _ALL_MODELS:
|
| 764 |
+
if sidebar_toggle(model, value=True, key=f"model_{model}"):
|
| 765 |
+
selected_models.append(model)
|
| 766 |
+
|
| 767 |
+
# -------------------------
|
| 768 |
+
# ✅ Validation: require ≥1 selection per relevant category
|
| 769 |
+
# -------------------------
|
| 770 |
+
missing = []
|
| 771 |
+
|
| 772 |
+
# Always required
|
| 773 |
+
if not selected_species:
|
| 774 |
+
missing.append("Species")
|
| 775 |
+
if not selected_models:
|
| 776 |
+
missing.append("Models")
|
| 777 |
+
|
| 778 |
+
# Required depending on benchmark
|
| 779 |
+
if cfg.get("has_assay_type", False) and not selected_assays:
|
| 780 |
+
missing.append("Assay types")
|
| 781 |
+
|
| 782 |
+
if benchmark_name == "Genome Annotation" and not selected_datasets:
|
| 783 |
+
missing.append("Genome annotations")
|
| 784 |
+
|
| 785 |
+
if missing:
|
| 786 |
+
# Show a single message and prevent *any* further display
|
| 787 |
+
st.error(
|
| 788 |
+
"Please select at least one item in each category. Currently missing: "
|
| 789 |
+
+ ", ".join(missing)
|
| 790 |
+
+ "."
|
| 791 |
+
)
|
| 792 |
+
st.stop()
|
| 793 |
+
|
| 794 |
+
# --- Main content ---
|
| 795 |
+
leaderboard_df = build_leaderboard(
|
| 796 |
+
benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
|
| 797 |
+
)
|
| 798 |
+
bar_df = build_bar_df(
|
| 799 |
+
benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
|
| 800 |
+
)
|
| 801 |
+
|
| 802 |
+
col1, col2 = st.columns([1, 1])
|
| 803 |
+
|
| 804 |
+
with col1:
|
| 805 |
+
st.subheader("🏅 Leaderboard")
|
| 806 |
+
st.write("\n") # spacer to match plotly padding
|
| 807 |
+
st.write("\n")
|
| 808 |
+
st.write("\n")
|
| 809 |
+
if leaderboard_df.empty:
|
| 810 |
+
st.info("No data for the selected filters.")
|
| 811 |
+
else:
|
| 812 |
+
st.dataframe(leaderboard_df, use_container_width=True)
|
| 813 |
+
|
| 814 |
+
with col2:
|
| 815 |
+
st.subheader("📈 Mean score per model")
|
| 816 |
+
if bar_df.empty:
|
| 817 |
+
st.info("No data for the selected filters.")
|
| 818 |
+
else:
|
| 819 |
+
# Order models by performance (least -> most)
|
| 820 |
+
bar_df = bar_df.sort_values("Mean score", ascending=True)
|
| 821 |
+
model_order = bar_df["Model"].tolist()
|
| 822 |
+
|
| 823 |
+
fig = px.bar(
|
| 824 |
+
bar_df,
|
| 825 |
+
x="Model",
|
| 826 |
+
y="Mean score",
|
| 827 |
+
color="Model",
|
| 828 |
+
color_discrete_map=MODEL_COLORS,
|
| 829 |
+
category_orders={"Model": model_order},
|
| 830 |
+
)
|
| 831 |
+
fig.update_layout(
|
| 832 |
+
barmode="group",
|
| 833 |
+
height=500,
|
| 834 |
+
xaxis_title="",
|
| 835 |
+
yaxis_title=cfg["metric_label"],
|
| 836 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 837 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 838 |
+
bargap=0.08,
|
| 839 |
+
)
|
| 840 |
+
fig.update_xaxes(showticklabels=False)
|
| 841 |
+
st.plotly_chart(fig, use_container_width=True)
|
| 842 |
+
|
| 843 |
+
# --- Breakdown plot: assay_type (Functional Tracks) OR datasets (Genome Annotation) ---
|
| 844 |
+
breakdown_df = build_category_model_df(
|
| 845 |
+
benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
|
| 846 |
+
)
|
| 847 |
+
|
| 848 |
+
type_of_data = "assay type" if benchmark_name == "Functional Tracks" else "gene annotation"
|
| 849 |
+
|
| 850 |
+
st.subheader(f"🧪 Mean score by {type_of_data}")
|
| 851 |
+
if breakdown_df.empty:
|
| 852 |
+
st.info("No data for the selected filters.")
|
| 853 |
+
else:
|
| 854 |
+
fig_breakdown = plot_breakdown_facets_sorted_models(
|
| 855 |
+
breakdown_df,
|
| 856 |
+
metric_label=cfg["metric_label"],
|
| 857 |
+
height=300,
|
| 858 |
+
)
|
| 859 |
+
st.plotly_chart(fig_breakdown, use_container_width=True)
|
| 860 |
+
|
| 861 |
+
# ------------------------------------------------------------------
|
| 862 |
+
# Model comparison: Head-to-head (left) + Convergence (right)
|
| 863 |
+
# ------------------------------------------------------------------
|
| 864 |
+
left, right = st.columns([1, 1], gap="large")
|
| 865 |
+
|
| 866 |
+
with left:
|
| 867 |
+
st.markdown("#### ⚖️ Head-to-head (per track)")
|
| 868 |
+
|
| 869 |
+
model_picker_options = selected_models if selected_models else _ALL_MODELS
|
| 870 |
+
default_a = model_picker_options[0] if model_picker_options else _ALL_MODELS[0]
|
| 871 |
+
default_b = model_picker_options[1] if len(model_picker_options) > 1 else (
|
| 872 |
+
_ALL_MODELS[1] if len(_ALL_MODELS) > 1 else default_a
|
| 873 |
+
)
|
| 874 |
+
|
| 875 |
+
cA, cB = st.columns([1, 1])
|
| 876 |
+
with cA:
|
| 877 |
+
model_a = st.selectbox(
|
| 878 |
+
"Model A (y-axis)",
|
| 879 |
+
options=model_picker_options,
|
| 880 |
+
index=model_picker_options.index(default_a) if default_a in model_picker_options else 0,
|
| 881 |
+
key=f"pair_model_a_{benchmark_name}",
|
| 882 |
+
)
|
| 883 |
+
with cB:
|
| 884 |
+
b_options = [m for m in model_picker_options if m != model_a] or model_picker_options
|
| 885 |
+
model_b = st.selectbox(
|
| 886 |
+
"Model B (x-axis)",
|
| 887 |
+
options=b_options,
|
| 888 |
+
index=0,
|
| 889 |
+
key=f"pair_model_b_{benchmark_name}",
|
| 890 |
+
)
|
| 891 |
+
|
| 892 |
+
scatter_df = build_pairwise_scatter_df(
|
| 893 |
+
benchmark_name,
|
| 894 |
+
selected_species,
|
| 895 |
+
selected_assays,
|
| 896 |
+
selected_models,
|
| 897 |
+
selected_datasets,
|
| 898 |
+
model_a,
|
| 899 |
+
model_b,
|
| 900 |
+
)
|
| 901 |
+
|
| 902 |
+
if scatter_df.empty:
|
| 903 |
+
st.info("No overlapping tracks for the selected filters (or one model is missing values).")
|
| 904 |
+
else:
|
| 905 |
+
min_v = float(min(scatter_df["Model A"].min(), scatter_df["Model B"].min()))
|
| 906 |
+
max_v = float(max(scatter_df["Model A"].max(), scatter_df["Model B"].max()))
|
| 907 |
+
pad = 0.05 * (max_v - min_v if max_v > min_v else 1.0)
|
| 908 |
+
axis_range = [min_v - pad, max_v + pad]
|
| 909 |
+
tick_step = (axis_range[1] - axis_range[0]) / 5
|
| 910 |
+
|
| 911 |
+
hover_cols = ["datasets"]
|
| 912 |
+
if benchmark_name == "Functional Tracks" and "track_name_clean" in scatter_df.columns:
|
| 913 |
+
hover_cols.append("track_name_clean")
|
| 914 |
+
else:
|
| 915 |
+
hover_cols.append("datasets")
|
| 916 |
+
|
| 917 |
+
color_col = "assay_type" if "assay_type" in scatter_df.columns else "datasets"
|
| 918 |
+
|
| 919 |
+
fig_scatter = px.scatter(
|
| 920 |
+
scatter_df,
|
| 921 |
+
x="Model B",
|
| 922 |
+
y="Model A",
|
| 923 |
+
color=color_col,
|
| 924 |
+
color_discrete_map=ASSAY_COLORS,
|
| 925 |
+
hover_name="Track",
|
| 926 |
+
hover_data=hover_cols,
|
| 927 |
+
)
|
| 928 |
+
|
| 929 |
+
fig_scatter.add_shape(
|
| 930 |
+
type="line",
|
| 931 |
+
x0=axis_range[0], y0=axis_range[0],
|
| 932 |
+
x1=axis_range[1], y1=axis_range[1],
|
| 933 |
+
xref="x", yref="y",
|
| 934 |
+
line=dict(color="red", dash="dot", width=2),
|
| 935 |
+
)
|
| 936 |
+
|
| 937 |
+
fig_scatter.update_layout(
|
| 938 |
+
height=550,
|
| 939 |
+
margin=dict(l=60, r=20, t=20, b=60),
|
| 940 |
+
xaxis=dict(
|
| 941 |
+
title=f"{model_b} — {cfg['metric_label']}",
|
| 942 |
+
range=axis_range,
|
| 943 |
+
dtick=tick_step,
|
| 944 |
+
constrain="domain",
|
| 945 |
+
),
|
| 946 |
+
yaxis=dict(
|
| 947 |
+
title=f"{model_a} — {cfg['metric_label']}",
|
| 948 |
+
range=axis_range,
|
| 949 |
+
dtick=tick_step,
|
| 950 |
+
scaleanchor="x",
|
| 951 |
+
scaleratio=1,
|
| 952 |
+
constrain="domain",
|
| 953 |
+
),
|
| 954 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 955 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 956 |
+
)
|
| 957 |
+
|
| 958 |
+
fig_scatter.update_layout(
|
| 959 |
+
legend=dict(
|
| 960 |
+
title="Assay type" if benchmark_name == "Functional Tracks" else "Genome Annotation",
|
| 961 |
+
x=0.98,
|
| 962 |
+
y=0.1,
|
| 963 |
+
xanchor="right",
|
| 964 |
+
yanchor="bottom",
|
| 965 |
+
bgcolor="rgba(255,255,255,0.2)", # semi-transparent white
|
| 966 |
+
bordercolor="rgba(0,0,0,0.2)",
|
| 967 |
+
borderwidth=1,
|
| 968 |
+
)
|
| 969 |
+
)
|
| 970 |
+
|
| 971 |
+
st.plotly_chart(fig_scatter, use_container_width=True)
|
| 972 |
+
|
| 973 |
+
with right:
|
| 974 |
+
st.markdown("#### ⏱️ Time to convergence")
|
| 975 |
+
|
| 976 |
+
x_mode = st.selectbox(
|
| 977 |
+
"X-axis",
|
| 978 |
+
options=["GPU (hours)", "Steps (billions)"],
|
| 979 |
+
index=0,
|
| 980 |
+
key=f"conv_x_mode_{benchmark_name}",
|
| 981 |
+
)
|
| 982 |
+
|
| 983 |
+
|
| 984 |
+
conv_df = build_convergence_df(
|
| 985 |
+
benchmark_name,
|
| 986 |
+
selected_species,
|
| 987 |
+
selected_assays,
|
| 988 |
+
selected_models,
|
| 989 |
+
selected_datasets,
|
| 990 |
+
x_mode=x_mode,
|
| 991 |
+
)
|
| 992 |
+
|
| 993 |
+
if conv_df.empty:
|
| 994 |
+
st.info("No convergence data found for the selected filters / x-axis mode.")
|
| 995 |
+
else:
|
| 996 |
+
fig_conv = px.scatter(
|
| 997 |
+
conv_df,
|
| 998 |
+
x="X",
|
| 999 |
+
y="Performance",
|
| 1000 |
+
text="Model",
|
| 1001 |
+
color="Model",
|
| 1002 |
+
color_discrete_map=MODEL_COLORS,
|
| 1003 |
+
hover_data=["Model", "X", "Performance"],
|
| 1004 |
+
)
|
| 1005 |
+
fig_conv.update_layout(
|
| 1006 |
+
height=550,
|
| 1007 |
+
xaxis_title=("GPU (hours)" if x_mode == "GPU (hours)" else x_mode),
|
| 1008 |
+
yaxis_title=cfg["metric_label"],
|
| 1009 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 1010 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 1011 |
+
showlegend=False, # ✅ no legend
|
| 1012 |
+
)
|
| 1013 |
+
fig_conv.update_traces(
|
| 1014 |
+
marker=dict(size=14), # 👈 bigger dots
|
| 1015 |
+
textposition="top center",
|
| 1016 |
+
)
|
| 1017 |
+
|
| 1018 |
+
# Log scale only makes sense for hours (and sometimes best_step)
|
| 1019 |
+
if x_mode in ["GPU (hours)"]:
|
| 1020 |
+
fig_conv.update_xaxes(
|
| 1021 |
+
type="log",
|
| 1022 |
+
dtick=1,
|
| 1023 |
+
minor=dict(ticks="", showgrid=False),
|
| 1024 |
+
)
|
| 1025 |
+
st.plotly_chart(fig_conv, use_container_width=True)
|
| 1026 |
+
|
| 1027 |
+
# ------------------------------------------------------------------
|
| 1028 |
+
# Violin (full width, below)
|
| 1029 |
+
# ------------------------------------------------------------------
|
| 1030 |
+
st.subheader("🎻 Performance comparison across tracks")
|
| 1031 |
+
|
| 1032 |
+
violin_df = build_violin_df(
|
| 1033 |
+
benchmark_name,
|
| 1034 |
+
selected_species,
|
| 1035 |
+
selected_assays,
|
| 1036 |
+
selected_models,
|
| 1037 |
+
selected_datasets,
|
| 1038 |
+
)
|
| 1039 |
+
|
| 1040 |
+
if violin_df.empty:
|
| 1041 |
+
st.info("No data for the selected filters.")
|
| 1042 |
+
else:
|
| 1043 |
+
model_order = (
|
| 1044 |
+
violin_df
|
| 1045 |
+
.groupby("Model")["Score"]
|
| 1046 |
+
.median()
|
| 1047 |
+
.sort_values(ascending=True)
|
| 1048 |
+
.index
|
| 1049 |
+
.tolist()
|
| 1050 |
+
)
|
| 1051 |
+
|
| 1052 |
+
fig_violin = px.violin(
|
| 1053 |
+
violin_df,
|
| 1054 |
+
x="Model",
|
| 1055 |
+
y="Score",
|
| 1056 |
+
color="Model",
|
| 1057 |
+
color_discrete_map=MODEL_COLORS,
|
| 1058 |
+
box=True,
|
| 1059 |
+
points=False,
|
| 1060 |
+
category_orders={"Model": model_order},
|
| 1061 |
+
)
|
| 1062 |
+
|
| 1063 |
+
fig_violin.update_layout(
|
| 1064 |
+
height=650,
|
| 1065 |
+
xaxis_title="",
|
| 1066 |
+
yaxis_title=cfg["metric_label"],
|
| 1067 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 1068 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 1069 |
+
showlegend=False,
|
| 1070 |
+
)
|
| 1071 |
+
|
| 1072 |
+
fig_violin.update_traces(
|
| 1073 |
+
box_visible=True,
|
| 1074 |
+
meanline_visible=False,
|
| 1075 |
+
)
|
| 1076 |
+
|
| 1077 |
+
st.plotly_chart(fig_violin, use_container_width=True)
|
| 1078 |
+
|
src/streamlit_app.py
CHANGED
|
@@ -1,12 +1,11 @@
|
|
| 1 |
-
from typing import List
|
| 2 |
import os
|
| 3 |
|
|
|
|
| 4 |
import pandas as pd
|
| 5 |
-
import streamlit as st
|
| 6 |
import plotly.express as px
|
| 7 |
-
from plotly.subplots import make_subplots
|
| 8 |
import plotly.graph_objects as go
|
| 9 |
-
import
|
|
|
|
| 10 |
|
| 11 |
# ---------------------------------------------------------------------
|
| 12 |
# Page config (must be the first Streamlit command)
|
|
@@ -16,150 +15,163 @@ st.set_page_config(
|
|
| 16 |
layout="wide",
|
| 17 |
)
|
| 18 |
|
|
|
|
|
|
|
| 19 |
# ---------------------------------------------------------------------
|
| 20 |
# Configuration
|
| 21 |
# ---------------------------------------------------------------------
|
| 22 |
COLORS = {
|
| 23 |
-
|
| 24 |
-
'
|
| 25 |
-
'
|
| 26 |
-
'
|
| 27 |
-
'
|
| 28 |
-
|
| 29 |
-
'
|
| 30 |
-
'red_2': '#ffe0de', # Light red
|
| 31 |
-
# Primary colors 2
|
| 32 |
-
'green_1': '#00b050', # Darkest green
|
| 33 |
-
'green_2': '#92d050', # Medium green
|
| 34 |
-
'green_3': '#c6e0b4', # Light green (lightest allowable green)
|
| 35 |
-
# Secondary colors 2
|
| 36 |
'gold_1': '#fdb932',
|
| 37 |
-
# Tertiary colors
|
| 38 |
'orange_1': '#ff975e',
|
| 39 |
'purple_1': '#9a6ce4',
|
| 40 |
-
'purple_2': '#bb9aef',
|
| 41 |
-
'
|
| 42 |
-
|
| 43 |
-
'
|
| 44 |
-
'
|
| 45 |
-
'gray_3': '#e6e6e6', # Lightest gray
|
| 46 |
-
'gray_4': '#ffffff', # It's actually just white (use as a last resort)
|
| 47 |
-
# If all other options are exhausted
|
| 48 |
-
'cyan_1': '#0096b4', # Darkest teal
|
| 49 |
-
'cyan_2': '#28bed2', # Medium cyan
|
| 50 |
-
'cyan_3': '#8cdceb', # Lightest cyan
|
| 51 |
-
'magenta_1': '#b428a0', # Darkest magenta
|
| 52 |
-
'magenta_2': '#dc50be', # Medium pink
|
| 53 |
-
'magenta_3': '#f5a0dc', # Lightest pink
|
| 54 |
-
'yellow_1': '#c8aa00', # Darkest yellow
|
| 55 |
-
'yellow_2': '#ffd200', # Medium yellow
|
| 56 |
-
'yellow_3': '#fff08c', # Lightest yellow
|
| 57 |
}
|
| 58 |
|
| 59 |
-
|
| 60 |
ASSAY_TYPE_MAPPING = {
|
| 61 |
'ATAC-seq': 'chromatin accessibility',
|
| 62 |
'DNase-seq': 'chromatin accessibility',
|
| 63 |
'Histone ChIP-seq': 'histone modifications',
|
| 64 |
-
'TF ChIP-seq': '
|
| 65 |
'PRO-cap': 'transcription initiation',
|
| 66 |
'eCLIP': 'RNA binding sites',
|
| 67 |
'RNA-seq': 'gene expression',
|
| 68 |
'ribo-seq': 'mRNA translation',
|
| 69 |
-
'Annotation': 'genome annotation',
|
| 70 |
-
"Exon": "exon",
|
| 71 |
-
"Intron": "intron",
|
| 72 |
-
"Splice acceptor": "splice acceptor",
|
| 73 |
-
"Start codon": "start codon",
|
| 74 |
-
|
| 75 |
}
|
| 76 |
|
| 77 |
ASSAY_COLORS = {
|
| 78 |
'chromatin accessibility': '#004697',
|
| 79 |
'histone modifications': '#cc0000',
|
|
|
|
| 80 |
'transcription initiation': '#ff9900',
|
| 81 |
'RNA binding sites': '#9933cc',
|
| 82 |
'gene expression': '#009900',
|
| 83 |
'mRNA translation': '#ff6699',
|
| 84 |
-
'
|
| 85 |
-
"intron": '#004697',
|
| 86 |
-
"exon": '#cc0000',
|
| 87 |
-
"splice acceptor": '#ff9900',
|
| 88 |
-
"start codon": '#9933cc',
|
| 89 |
}
|
| 90 |
-
ASSAY_COLORS["other"] = "#808080"
|
| 91 |
|
| 92 |
-
|
| 93 |
-
"
|
| 94 |
-
|
| 95 |
-
|
| 96 |
-
|
| 97 |
-
'Evo2 1B': COLORS['green_3'], # #b3b3b3 (Medium gray)
|
| 98 |
-
"NTv2 500M": COLORS['gray_1'],
|
| 99 |
-
"BPNet arch. 6M": COLORS['cyan_1'],
|
| 100 |
-
"Residual CNN 44M": COLORS['magenta_1'],
|
| 101 |
-
"PlantCAD2 88M": COLORS["purple_1"],
|
| 102 |
-
"Caduceus 7M": COLORS["purple_2"],
|
| 103 |
-
"HyenaDNA 7M": COLORS["yellow_2"]
|
| 104 |
}
|
| 105 |
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
"
|
| 110 |
-
"
|
| 111 |
-
"
|
| 112 |
-
"Caduceus 7M": "
|
| 113 |
-
"
|
| 114 |
-
"
|
| 115 |
-
"
|
| 116 |
-
"
|
| 117 |
-
"
|
|
|
|
| 118 |
}
|
|
|
|
| 119 |
|
| 120 |
-
|
| 121 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
}
|
| 123 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
|
| 125 |
MODEL_NAMES = list(MODEL_COLORS.keys())
|
| 126 |
|
| 127 |
-
|
| 128 |
-
ANIMAL_SPECIES = ["human", "chicken", "cattle"]
|
| 129 |
-
|
| 130 |
SPECIES_GROUPS = {
|
| 131 |
-
"
|
| 132 |
-
"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 133 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
|
| 135 |
-
|
| 136 |
-
_LAST_UPDATED = "Dec 10, 2025"
|
| 137 |
_INTRO = """
|
| 138 |
-
The **NTv3 Benchmark** is
|
| 139 |
-
|
| 140 |
-
|
| 141 |
-
|
| 142 |
-
|
| 143 |
-
|
| 144 |
-
|
| 145 |
-
|
| 146 |
-
|
| 147 |
-
|
| 148 |
-
|
| 149 |
-
|
| 150 |
-
|
| 151 |
-
|
| 152 |
-
|
| 153 |
-
The metrics used are:
|
| 154 |
-
- **Pearson correlations (multi-assay)**: per-dataset scores across species and models for functional tracks.
|
| 155 |
-
- **MCC (bed tracks)**: per-track MCC values across species and models for gene annotation tracks.
|
| 156 |
"""
|
| 157 |
|
| 158 |
HERE = os.path.dirname(os.path.abspath(__file__)) # /app/src
|
| 159 |
PROJECT_ROOT = os.path.dirname(HERE) # /app
|
| 160 |
DATA_DIR = os.path.join(PROJECT_ROOT, "data")
|
| 161 |
-
|
| 162 |
-
SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "ntv3_benchmark_results.csv")
|
| 163 |
|
| 164 |
# ---------------------------------------------------------------------
|
| 165 |
# Data loading & preprocessing
|
|
@@ -167,139 +179,54 @@ SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "ntv3_benchmark_results.csv")
|
|
| 167 |
|
| 168 |
|
| 169 |
@st.cache_data
|
| 170 |
-
def load_raw_data():
|
| 171 |
df = pd.read_csv(SINGLE_TABLE_PATH)
|
| 172 |
df.columns = [c.strip() for c in df.columns]
|
| 173 |
return df
|
| 174 |
|
| 175 |
|
| 176 |
-
def _normalize_training_time_to_gpu_hours(df: pd.DataFrame) -> pd.DataFrame:
|
| 177 |
-
"""
|
| 178 |
-
Your new column is `running_time`. In your sample it looks like seconds
|
| 179 |
-
(e.g. 317034 ~= 88 hours). We'll convert to hours if values look like seconds.
|
| 180 |
-
"""
|
| 181 |
-
if "running_time" not in df.columns:
|
| 182 |
-
return df
|
| 183 |
-
|
| 184 |
-
rt = pd.to_numeric(df["running_time"], errors="coerce")
|
| 185 |
-
# Heuristic: if median is huge, it's probably seconds -> convert to hours
|
| 186 |
-
# (88 hours = 316800 seconds is a typical-looking value in your sample)
|
| 187 |
-
if rt.dropna().median() > 10_000:
|
| 188 |
-
df["GPU hours"] = rt / 3600.0
|
| 189 |
-
else:
|
| 190 |
-
df["GPU hours"] = rt.astype(float)
|
| 191 |
-
|
| 192 |
-
return df
|
| 193 |
-
|
| 194 |
-
|
| 195 |
-
def _best_step_time_to_hours(s: pd.Series) -> pd.Series:
|
| 196 |
-
"""
|
| 197 |
-
Converts strings like '3 days 04:26:26.467000' to hours (float).
|
| 198 |
-
Works with pandas Timedelta parsing.
|
| 199 |
-
"""
|
| 200 |
-
td = pd.to_timedelta(s, errors="coerce")
|
| 201 |
-
return td.dt.total_seconds() / 3600.0
|
| 202 |
-
|
| 203 |
-
|
| 204 |
-
|
| 205 |
@st.cache_data
|
| 206 |
-
def
|
| 207 |
-
|
| 208 |
-
|
| 209 |
-
df = df
|
|
|
|
| 210 |
df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
|
| 211 |
-
|
| 212 |
-
|
| 213 |
-
|
| 214 |
-
|
| 215 |
-
if "best_step_time" in df.columns:
|
| 216 |
-
df["best_step_time_hours"] = _best_step_time_to_hours(df["best_step_time"])
|
| 217 |
-
else:
|
| 218 |
-
df["best_step_time_hours"] = np.nan
|
| 219 |
-
|
| 220 |
-
is_annot = df.get("assay_type", "").astype(str).eq("Annotation")
|
| 221 |
-
pearson_raw = df[~is_annot].copy()
|
| 222 |
-
mcc_raw = df[is_annot].copy()
|
| 223 |
-
|
| 224 |
-
# -------------------------
|
| 225 |
-
# Functional Tracks (Pearson)
|
| 226 |
-
# -------------------------
|
| 227 |
-
pearson_group_cols = ["species", "datasets", "Model"]
|
| 228 |
-
if "assay_type" in pearson_raw.columns:
|
| 229 |
-
pearson_group_cols.append("assay_type")
|
| 230 |
-
|
| 231 |
-
pearson_df = (
|
| 232 |
-
pearson_raw
|
| 233 |
-
.groupby(pearson_group_cols, as_index=False, dropna=False)
|
| 234 |
-
.agg({
|
| 235 |
-
"Score": "mean",
|
| 236 |
-
"best_step": "mean",
|
| 237 |
-
"best_step_time_hours": "mean",
|
| 238 |
-
})
|
| 239 |
)
|
|
|
|
|
|
|
| 240 |
|
| 241 |
-
# ✅ merge track_name_clean WHILE assay_type is still raw
|
| 242 |
-
if "track_name_clean" in pearson_raw.columns:
|
| 243 |
-
map_keys = ["species", "datasets"]
|
| 244 |
-
if "assay_type" in pearson_raw.columns:
|
| 245 |
-
map_keys.append("assay_type")
|
| 246 |
-
|
| 247 |
-
track_map = (
|
| 248 |
-
pearson_raw[map_keys + ["track_name_clean"]]
|
| 249 |
-
.dropna(subset=["track_name_clean"])
|
| 250 |
-
.drop_duplicates()
|
| 251 |
-
)
|
| 252 |
-
pearson_df = pearson_df.merge(track_map, on=map_keys, how="left")
|
| 253 |
-
|
| 254 |
-
# ✅ now it’s safe to map assay_type to categories
|
| 255 |
-
if "assay_type" in pearson_df.columns:
|
| 256 |
-
pearson_df["assay_type"] = (
|
| 257 |
-
pearson_df["assay_type"].astype(str).map(ASSAY_TYPE_MAPPING).fillna("Other")
|
| 258 |
-
)
|
| 259 |
|
| 260 |
-
|
| 261 |
-
|
| 262 |
-
|
| 263 |
-
|
| 264 |
-
|
| 265 |
-
|
| 266 |
-
|
| 267 |
-
|
| 268 |
-
|
| 269 |
-
|
| 270 |
-
|
|
|
|
|
|
|
| 271 |
)
|
| 272 |
-
|
| 273 |
-
|
|
|
|
| 274 |
|
| 275 |
|
| 276 |
-
|
|
|
|
| 277 |
|
| 278 |
-
# Global sets (we'll further filter per-benchmark below)
|
| 279 |
-
_ALL_SPECIES = sorted(
|
| 280 |
-
set(_PEARSON_DF["species"].unique()).union(_MCC_DF["species"].unique())
|
| 281 |
-
)
|
| 282 |
|
| 283 |
-
|
| 284 |
-
|
| 285 |
-
|
| 286 |
-
|
| 287 |
-
)
|
| 288 |
-
|
| 289 |
-
_ALL_MODELS = MODEL_NAMES[:]
|
| 290 |
-
|
| 291 |
-
_BENCHMARKS = {
|
| 292 |
-
"Functional Tracks": {
|
| 293 |
-
"df": _PEARSON_DF,
|
| 294 |
-
"metric_label": "Pearson correlation",
|
| 295 |
-
"has_assay_type": True,
|
| 296 |
-
},
|
| 297 |
-
"Genome Annotation": {
|
| 298 |
-
"df": _MCC_DF,
|
| 299 |
-
"metric_label": "MCC",
|
| 300 |
-
"has_assay_type": False,
|
| 301 |
-
},
|
| 302 |
-
}
|
| 303 |
|
| 304 |
|
| 305 |
# ---------------------------------------------------------------------
|
|
@@ -307,775 +234,517 @@ _BENCHMARKS = {
|
|
| 307 |
# ---------------------------------------------------------------------
|
| 308 |
|
| 309 |
|
| 310 |
-
def
|
| 311 |
-
|
| 312 |
-
|
| 313 |
-
|
| 314 |
-
|
| 315 |
-
|
| 316 |
-
) -> pd.DataFrame:
|
| 317 |
-
cfg = _BENCHMARKS[benchmark_name]
|
| 318 |
-
df = cfg["df"].copy()
|
| 319 |
-
|
| 320 |
-
# Species filter
|
| 321 |
-
if selected_species:
|
| 322 |
-
df = df[df["species"].isin(selected_species)]
|
| 323 |
-
|
| 324 |
-
# Assay type filter (Pearson only)
|
| 325 |
-
if cfg.get("has_assay_type", False) and selected_assays and "assay_type" in df.columns:
|
| 326 |
-
df = df[df["assay_type"].isin(selected_assays)]
|
| 327 |
-
|
| 328 |
-
# Dataset / bed track filter (for MCC, but safe to apply generally)
|
| 329 |
-
if selected_datasets and "datasets" in df.columns:
|
| 330 |
-
df = df[df["datasets"].isin(selected_datasets)]
|
| 331 |
-
|
| 332 |
-
# Model filter
|
| 333 |
-
if selected_models:
|
| 334 |
-
df = df[df["Model"].isin(selected_models)]
|
| 335 |
-
|
| 336 |
-
return df
|
| 337 |
-
|
| 338 |
-
|
| 339 |
-
def build_leaderboard(
|
| 340 |
-
benchmark_name: str,
|
| 341 |
-
selected_species: List[str],
|
| 342 |
-
selected_assays: List[str],
|
| 343 |
-
selected_models: List[str],
|
| 344 |
-
selected_datasets: List[str],
|
| 345 |
-
) -> pd.DataFrame:
|
| 346 |
-
df = filter_base_df(
|
| 347 |
-
benchmark_name,
|
| 348 |
-
selected_species,
|
| 349 |
-
selected_assays,
|
| 350 |
-
selected_models,
|
| 351 |
-
selected_datasets,
|
| 352 |
)
|
| 353 |
|
| 354 |
-
if df.empty:
|
| 355 |
-
return pd.DataFrame(columns=["Model", "Model Type", "Num entries", "Mean score"])
|
| 356 |
|
| 357 |
-
|
| 358 |
-
|
| 359 |
-
.
|
| 360 |
-
|
| 361 |
-
|
| 362 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 363 |
agg["Mean score"] = agg["Mean score"].round(3)
|
| 364 |
-
|
| 365 |
-
agg["Num entries"] = (
|
| 366 |
-
df.groupby("Model")["Score"].count().reindex(agg["Model"]).values
|
| 367 |
-
)
|
| 368 |
-
|
| 369 |
-
# 👇 Add training regime column
|
| 370 |
agg["Training"] = agg["Model"].map(MODEL_TRAINING_STATUS).fillna("UNKNOWN")
|
| 371 |
-
|
| 372 |
-
# Sort by performance
|
| 373 |
agg = agg.sort_values("Mean score", ascending=False).reset_index(drop=True)
|
| 374 |
-
|
| 375 |
-
# Column order
|
| 376 |
-
agg = agg[["Model", "Training", "Num entries", "Mean score"]]
|
| 377 |
-
|
| 378 |
-
# Ensure the index starts with 1
|
| 379 |
agg.index += 1
|
| 380 |
-
|
| 381 |
return agg
|
| 382 |
|
| 383 |
|
| 384 |
-
|
| 385 |
-
|
| 386 |
-
|
| 387 |
-
|
| 388 |
-
|
| 389 |
-
|
| 390 |
-
|
| 391 |
-
) -> pd.DataFrame:
|
| 392 |
-
"""For now, just one bar per model (same as leaderboard)."""
|
| 393 |
-
return build_leaderboard(
|
| 394 |
-
benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
|
| 395 |
)
|
|
|
|
|
|
|
| 396 |
|
| 397 |
|
| 398 |
-
def
|
| 399 |
-
|
| 400 |
-
|
| 401 |
-
|
| 402 |
-
|
| 403 |
-
|
| 404 |
-
|
| 405 |
-
|
| 406 |
-
|
| 407 |
-
|
| 408 |
-
|
| 409 |
-
|
| 410 |
-
|
| 411 |
-
|
| 412 |
-
|
| 413 |
-
|
| 414 |
-
|
| 415 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 416 |
)
|
| 417 |
-
|
| 418 |
-
|
|
|
|
| 419 |
|
| 420 |
-
# Pick the right breakdown column
|
| 421 |
-
if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
|
| 422 |
-
category_col = "assay_type"
|
| 423 |
-
category_label = "Assay type"
|
| 424 |
-
else:
|
| 425 |
-
category_col = "datasets"
|
| 426 |
-
category_label = "Dataset"
|
| 427 |
|
| 428 |
-
|
| 429 |
-
|
|
|
|
| 430 |
|
| 431 |
-
out = (
|
| 432 |
-
df.groupby([category_col, "Model"], as_index=False)["Score"]
|
| 433 |
-
.mean()
|
| 434 |
-
.rename(columns={category_col: "Category", "Score": "Mean score"})
|
| 435 |
-
)
|
| 436 |
-
out["Mean score"] = out["Mean score"].round(3)
|
| 437 |
-
out.attrs["category_label"] = category_label # for nicer axis title
|
| 438 |
-
return out
|
| 439 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 440 |
|
| 441 |
-
def plot_breakdown_facets_sorted_models(
|
| 442 |
-
breakdown_df: pd.DataFrame,
|
| 443 |
-
metric_label: str,
|
| 444 |
-
height: int = 420,
|
| 445 |
-
):
|
| 446 |
-
categories = list(breakdown_df["Category"].dropna().unique())
|
| 447 |
-
categories = sorted(categories)
|
| 448 |
|
| 449 |
-
|
| 450 |
-
|
| 451 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 452 |
|
| 453 |
-
rows = 1
|
| 454 |
-
cols = n # 👈 everything in one row
|
| 455 |
|
| 456 |
-
|
|
|
|
|
|
|
|
|
|
| 457 |
y_min = breakdown_df["Mean score"].min()
|
| 458 |
y_max = breakdown_df["Mean score"].max()
|
| 459 |
pad = 0.05 * (y_max - y_min if y_max > y_min else 1.0)
|
| 460 |
-
y_range = [y_min - pad, y_max + pad]
|
| 461 |
|
| 462 |
fig = make_subplots(
|
| 463 |
-
rows=
|
| 464 |
-
cols=
|
| 465 |
subplot_titles=categories,
|
| 466 |
shared_yaxes=True,
|
| 467 |
-
horizontal_spacing=0.04,
|
| 468 |
)
|
| 469 |
-
|
| 470 |
for i, cat in enumerate(categories):
|
| 471 |
-
|
| 472 |
-
c = (i % cols) + 1
|
| 473 |
-
|
| 474 |
-
sub = (
|
| 475 |
-
breakdown_df[breakdown_df["Category"] == cat]
|
| 476 |
-
.sort_values("Mean score", ascending=True)
|
| 477 |
-
)
|
| 478 |
-
|
| 479 |
fig.add_trace(
|
| 480 |
go.Bar(
|
| 481 |
x=sub["Model"],
|
| 482 |
y=sub["Mean score"],
|
| 483 |
marker_color=[MODEL_COLORS.get(m, "#808080") for m in sub["Model"]],
|
| 484 |
showlegend=False,
|
|
|
|
| 485 |
),
|
| 486 |
-
row=
|
| 487 |
-
col=
|
| 488 |
)
|
| 489 |
-
|
| 490 |
-
fig.
|
| 491 |
-
fig.update_yaxes(range=y_range, title_text="", row=r, col=c) # 👈 apply range
|
| 492 |
-
|
| 493 |
fig.update_layout(
|
| 494 |
height=height,
|
| 495 |
plot_bgcolor="rgba(0,0,0,0)",
|
| 496 |
paper_bgcolor="rgba(0,0,0,0)",
|
| 497 |
margin=dict(t=60, l=10, r=10, b=10),
|
| 498 |
)
|
| 499 |
-
|
| 500 |
-
# Single y-axis label on the leftmost panel
|
| 501 |
-
fig.update_yaxes(title_text=metric_label, row=1, col=1)
|
| 502 |
-
|
| 503 |
return fig
|
| 504 |
|
| 505 |
-
def build_pairwise_scatter_df(
|
| 506 |
-
benchmark_name: str,
|
| 507 |
-
selected_species: List[str],
|
| 508 |
-
selected_assays: List[str],
|
| 509 |
-
selected_models: List[str],
|
| 510 |
-
selected_datasets: List[str],
|
| 511 |
-
model_a: str,
|
| 512 |
-
model_b: str,
|
| 513 |
-
) -> pd.DataFrame:
|
| 514 |
-
cfg = _BENCHMARKS[benchmark_name]
|
| 515 |
-
|
| 516 |
-
models_for_filter = (
|
| 517 |
-
list(set(selected_models + [model_a, model_b]))
|
| 518 |
-
if selected_models else [model_a, model_b]
|
| 519 |
-
)
|
| 520 |
-
|
| 521 |
-
df = filter_base_df(
|
| 522 |
-
benchmark_name,
|
| 523 |
-
selected_species,
|
| 524 |
-
selected_assays,
|
| 525 |
-
models_for_filter,
|
| 526 |
-
selected_datasets,
|
| 527 |
-
)
|
| 528 |
-
if df.empty:
|
| 529 |
-
return pd.DataFrame()
|
| 530 |
-
|
| 531 |
-
# ---- define "track identity" for head-to-head ----
|
| 532 |
-
# Always use datasets for the identity (x/y points)
|
| 533 |
-
track_cols = ["datasets"]
|
| 534 |
-
if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
|
| 535 |
-
track_cols = ["assay_type", "datasets"]
|
| 536 |
|
| 537 |
-
|
| 538 |
-
|
| 539 |
-
|
| 540 |
-
|
| 541 |
-
df[df["Model"].isin([model_a, model_b])]
|
| 542 |
-
.pivot_table(index=id_cols, columns="Model", values="Score", aggfunc="mean")
|
| 543 |
-
.reset_index()
|
| 544 |
-
)
|
| 545 |
-
|
| 546 |
-
if model_a not in wide.columns or model_b not in wide.columns:
|
| 547 |
-
return pd.DataFrame()
|
| 548 |
|
| 549 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 550 |
|
| 551 |
-
|
| 552 |
-
if
|
| 553 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 554 |
else:
|
| 555 |
-
|
| 556 |
-
|
| 557 |
-
|
| 558 |
-
|
| 559 |
-
# ---- Pearson-only: merge track_name_clean for hover ----
|
| 560 |
-
if benchmark_name == "Functional Tracks" and "track_name_clean" in df.columns:
|
| 561 |
-
merge_keys = id_cols.copy() # species? + assay_type? + datasets
|
| 562 |
-
track_map = (
|
| 563 |
-
df[merge_keys + ["track_name_clean"]]
|
| 564 |
-
.dropna(subset=["track_name_clean"])
|
| 565 |
-
.drop_duplicates()
|
| 566 |
)
|
| 567 |
-
|
| 568 |
-
|
| 569 |
-
|
| 570 |
-
|
| 571 |
-
|
| 572 |
-
|
| 573 |
-
|
| 574 |
-
|
| 575 |
-
|
| 576 |
-
|
| 577 |
-
|
| 578 |
-
|
| 579 |
-
|
| 580 |
-
|
| 581 |
-
|
| 582 |
-
|
| 583 |
-
|
| 584 |
-
|
| 585 |
-
selected_models,
|
| 586 |
-
selected_datasets,
|
| 587 |
)
|
| 588 |
-
|
| 589 |
-
|
| 590 |
-
|
| 591 |
-
|
| 592 |
-
|
| 593 |
-
|
| 594 |
-
|
| 595 |
-
|
| 596 |
-
def build_convergence_df(
|
| 597 |
-
benchmark_name: str,
|
| 598 |
-
selected_species: List[str],
|
| 599 |
-
selected_assays: List[str],
|
| 600 |
-
selected_models: List[str],
|
| 601 |
-
selected_datasets: List[str],
|
| 602 |
-
x_mode: str = "best_step", # "best_step" | "best_step_time"
|
| 603 |
-
) -> pd.DataFrame:
|
| 604 |
-
|
| 605 |
-
df = filter_base_df(
|
| 606 |
-
benchmark_name,
|
| 607 |
-
selected_species,
|
| 608 |
-
selected_assays,
|
| 609 |
-
selected_models,
|
| 610 |
-
selected_datasets,
|
| 611 |
)
|
| 612 |
-
|
| 613 |
-
|
| 614 |
-
|
| 615 |
-
|
| 616 |
-
|
| 617 |
-
|
| 618 |
-
|
| 619 |
-
|
| 620 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 621 |
)
|
|
|
|
| 622 |
|
| 623 |
-
# -------------------------
|
| 624 |
-
# X axis selection
|
| 625 |
-
# -------------------------
|
| 626 |
-
if x_mode == "Steps (billions)":
|
| 627 |
-
if "best_step" not in df.columns:
|
| 628 |
-
return pd.DataFrame(columns=["Model", "X", "Performance"])
|
| 629 |
-
|
| 630 |
-
x = (
|
| 631 |
-
df.groupby("Model", as_index=False)["best_step"]
|
| 632 |
-
.mean()
|
| 633 |
-
.rename(columns={"best_step": "X"})
|
| 634 |
-
)
|
| 635 |
|
| 636 |
-
|
| 637 |
-
|
| 638 |
-
|
| 639 |
-
|
| 640 |
-
|
| 641 |
-
|
| 642 |
-
|
| 643 |
-
.rename(columns={"best_step_time_hours": "X"})
|
| 644 |
-
)
|
| 645 |
|
| 646 |
-
# 👇 Apply GPU multiplier (Evo2 uses 8 GPUs)
|
| 647 |
-
gpu_multiplier = {
|
| 648 |
-
"Evo2 1B": 8,
|
| 649 |
-
}
|
| 650 |
|
| 651 |
-
|
| 652 |
-
|
| 653 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 654 |
)
|
| 655 |
-
|
| 656 |
-
|
| 657 |
-
|
| 658 |
-
|
| 659 |
-
|
| 660 |
-
|
| 661 |
-
|
| 662 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 663 |
|
| 664 |
|
| 665 |
# ---------------------------------------------------------------------
|
| 666 |
-
#
|
| 667 |
# ---------------------------------------------------------------------
|
| 668 |
|
| 669 |
|
| 670 |
-
|
| 671 |
-
|
| 672 |
-
|
| 673 |
-
|
| 674 |
-
|
| 675 |
-
|
| 676 |
-
|
| 677 |
-
|
| 678 |
-
|
| 679 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 680 |
|
| 681 |
-
|
| 682 |
-
|
| 683 |
-
|
| 684 |
-
def main():
|
| 685 |
st.title("🧬 NTv3 Benchmark")
|
| 686 |
st.markdown(_INTRO)
|
| 687 |
st.markdown(f"_Last updated: **{_LAST_UPDATED}**_")
|
|
|
|
| 688 |
|
| 689 |
-
|
| 690 |
-
st.
|
| 691 |
-
|
| 692 |
-
|
| 693 |
-
benchmark_name = st.sidebar.selectbox(
|
| 694 |
-
"Benchmark",
|
| 695 |
-
options=list(_BENCHMARKS.keys()),
|
| 696 |
-
index=0,
|
| 697 |
-
)
|
| 698 |
-
|
| 699 |
-
cfg = _BENCHMARKS[benchmark_name]
|
| 700 |
-
df_bench = cfg["df"]
|
| 701 |
-
|
| 702 |
-
# Species toggles, but only for species present in this benchmark
|
| 703 |
-
st.sidebar.subheader("Species")
|
| 704 |
-
|
| 705 |
-
# Toggle: Plants vs Animals
|
| 706 |
-
species_group = st.sidebar.radio(
|
| 707 |
-
"Group",
|
| 708 |
-
options=["Animals", "Plants"],
|
| 709 |
-
index=0,
|
| 710 |
-
horizontal=True,
|
| 711 |
-
key=f"species_group_{benchmark_name}",
|
| 712 |
)
|
|
|
|
|
|
|
| 713 |
|
| 714 |
-
|
| 715 |
-
|
| 716 |
-
|
| 717 |
-
|
| 718 |
-
|
| 719 |
-
|
| 720 |
-
|
| 721 |
-
|
| 722 |
-
|
| 723 |
-
# (Optional) If no species exist for that group in this benchmark
|
| 724 |
-
if not available_species:
|
| 725 |
-
st.sidebar.info(f"No {species_group.lower()} species available for this benchmark.")
|
| 726 |
-
|
| 727 |
-
# Assay toggles (Pearson only), based on filtered species
|
| 728 |
-
if cfg.get("has_assay_type", False):
|
| 729 |
-
st.sidebar.subheader("Assay types")
|
| 730 |
-
if selected_species:
|
| 731 |
-
df_for_assays = df_bench[df_bench["species"].isin(selected_species)]
|
| 732 |
-
else:
|
| 733 |
-
df_for_assays = df_bench
|
| 734 |
-
available_assays = (
|
| 735 |
-
sorted(df_for_assays["assay_type"].dropna().unique())
|
| 736 |
-
if "assay_type" in df_for_assays.columns
|
| 737 |
-
else []
|
| 738 |
-
)
|
| 739 |
-
selected_assays: List[str] = []
|
| 740 |
-
for assay in available_assays:
|
| 741 |
-
if sidebar_toggle(assay, value=True, key=f"assay_{benchmark_name}_{assay}"):
|
| 742 |
-
selected_assays.append(assay)
|
| 743 |
-
else:
|
| 744 |
-
selected_assays = []
|
| 745 |
-
|
| 746 |
-
# Bed track / dataset toggles (MCC only), based on species selection
|
| 747 |
-
selected_datasets: List[str] = []
|
| 748 |
-
if benchmark_name == "Genome Annotation":
|
| 749 |
-
st.sidebar.subheader("Genome annotations")
|
| 750 |
-
if selected_species:
|
| 751 |
-
df_for_tracks = df_bench[df_bench["species"].isin(selected_species)]
|
| 752 |
-
else:
|
| 753 |
-
df_for_tracks = df_bench
|
| 754 |
-
available_datasets = sorted(df_for_tracks["datasets"].unique())
|
| 755 |
-
for ds in available_datasets:
|
| 756 |
-
if sidebar_toggle(ds, value=True, key=f"dataset_{benchmark_name}_{ds}"):
|
| 757 |
-
selected_datasets.append(ds)
|
| 758 |
-
else:
|
| 759 |
-
selected_datasets = []
|
| 760 |
-
|
| 761 |
-
# Model toggles (we keep all models in MODEL_NAMES; filters + data will prune)
|
| 762 |
-
st.sidebar.subheader("Models")
|
| 763 |
-
selected_models: List[str] = []
|
| 764 |
-
for model in _ALL_MODELS:
|
| 765 |
-
if sidebar_toggle(model, value=True, key=f"model_{model}"):
|
| 766 |
-
selected_models.append(model)
|
| 767 |
-
|
| 768 |
-
# -------------------------
|
| 769 |
-
# ✅ Validation: require ≥1 selection per relevant category
|
| 770 |
-
# -------------------------
|
| 771 |
-
missing = []
|
| 772 |
-
|
| 773 |
-
# Always required
|
| 774 |
-
if not selected_species:
|
| 775 |
-
missing.append("Species")
|
| 776 |
-
if not selected_models:
|
| 777 |
-
missing.append("Models")
|
| 778 |
-
|
| 779 |
-
# Required depending on benchmark
|
| 780 |
-
if cfg.get("has_assay_type", False) and not selected_assays:
|
| 781 |
-
missing.append("Assay types")
|
| 782 |
-
|
| 783 |
-
if benchmark_name == "Genome Annotation" and not selected_datasets:
|
| 784 |
-
missing.append("Genome annotations")
|
| 785 |
-
|
| 786 |
-
if missing:
|
| 787 |
-
# Show a single message and prevent *any* further display
|
| 788 |
-
st.error(
|
| 789 |
-
"Please select at least one item in each category. Currently missing: "
|
| 790 |
-
+ ", ".join(missing)
|
| 791 |
-
+ "."
|
| 792 |
)
|
| 793 |
-
st.stop()
|
| 794 |
|
| 795 |
-
|
| 796 |
-
|
| 797 |
-
|
| 798 |
-
)
|
| 799 |
-
|
| 800 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 801 |
)
|
| 802 |
|
|
|
|
| 803 |
col1, col2 = st.columns([1, 1])
|
| 804 |
-
|
| 805 |
with col1:
|
| 806 |
st.subheader("🏅 Leaderboard")
|
| 807 |
-
st.write("\n") # spacer to match plotly padding
|
| 808 |
st.write("\n")
|
| 809 |
st.write("\n")
|
| 810 |
-
|
| 811 |
-
|
| 812 |
-
else:
|
| 813 |
-
st.dataframe(leaderboard_df, use_container_width=True)
|
| 814 |
-
|
| 815 |
with col2:
|
| 816 |
st.subheader("📈 Mean score per model")
|
| 817 |
-
|
| 818 |
-
st.info("No data for the selected filters.")
|
| 819 |
-
else:
|
| 820 |
-
# Order models by performance (least -> most)
|
| 821 |
-
bar_df = bar_df.sort_values("Mean score", ascending=True)
|
| 822 |
-
model_order = bar_df["Model"].tolist()
|
| 823 |
-
|
| 824 |
-
fig = px.bar(
|
| 825 |
-
bar_df,
|
| 826 |
-
x="Model",
|
| 827 |
-
y="Mean score",
|
| 828 |
-
color="Model",
|
| 829 |
-
color_discrete_map=MODEL_COLORS,
|
| 830 |
-
category_orders={"Model": model_order},
|
| 831 |
-
)
|
| 832 |
-
fig.update_layout(
|
| 833 |
-
barmode="group",
|
| 834 |
-
height=500,
|
| 835 |
-
xaxis_title="",
|
| 836 |
-
yaxis_title=cfg["metric_label"],
|
| 837 |
-
plot_bgcolor="rgba(0,0,0,0)",
|
| 838 |
-
paper_bgcolor="rgba(0,0,0,0)",
|
| 839 |
-
bargap=0.08,
|
| 840 |
-
)
|
| 841 |
-
fig.update_xaxes(showticklabels=False)
|
| 842 |
-
st.plotly_chart(fig, use_container_width=True)
|
| 843 |
-
|
| 844 |
-
# --- Breakdown plot: assay_type (Functional Tracks) OR datasets (Genome Annotation) ---
|
| 845 |
-
breakdown_df = build_category_model_df(
|
| 846 |
-
benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
|
| 847 |
-
)
|
| 848 |
|
| 849 |
-
|
|
|
|
|
|
|
|
|
|
| 850 |
|
| 851 |
-
st.subheader(f"🧪 Mean score by {type_of_data}")
|
| 852 |
-
if breakdown_df.empty:
|
| 853 |
-
st.info("No data for the selected filters.")
|
| 854 |
-
else:
|
| 855 |
-
fig_breakdown = plot_breakdown_facets_sorted_models(
|
| 856 |
-
breakdown_df,
|
| 857 |
-
metric_label=cfg["metric_label"],
|
| 858 |
-
height=300,
|
| 859 |
-
)
|
| 860 |
-
st.plotly_chart(fig_breakdown, use_container_width=True)
|
| 861 |
-
|
| 862 |
-
# ------------------------------------------------------------------
|
| 863 |
-
# Model comparison: Head-to-head (left) + Convergence (right)
|
| 864 |
-
# ------------------------------------------------------------------
|
| 865 |
left, right = st.columns([1, 1], gap="large")
|
| 866 |
-
|
| 867 |
with left:
|
| 868 |
-
|
| 869 |
-
|
| 870 |
-
model_picker_options = selected_models if selected_models else _ALL_MODELS
|
| 871 |
-
default_a = model_picker_options[0] if model_picker_options else _ALL_MODELS[0]
|
| 872 |
-
default_b = model_picker_options[1] if len(model_picker_options) > 1 else (
|
| 873 |
-
_ALL_MODELS[1] if len(_ALL_MODELS) > 1 else default_a
|
| 874 |
-
)
|
| 875 |
-
|
| 876 |
-
cA, cB = st.columns([1, 1])
|
| 877 |
-
with cA:
|
| 878 |
-
model_a = st.selectbox(
|
| 879 |
-
"Model A (y-axis)",
|
| 880 |
-
options=model_picker_options,
|
| 881 |
-
index=model_picker_options.index(default_a) if default_a in model_picker_options else 0,
|
| 882 |
-
key=f"pair_model_a_{benchmark_name}",
|
| 883 |
-
)
|
| 884 |
-
with cB:
|
| 885 |
-
b_options = [m for m in model_picker_options if m != model_a] or model_picker_options
|
| 886 |
-
model_b = st.selectbox(
|
| 887 |
-
"Model B (x-axis)",
|
| 888 |
-
options=b_options,
|
| 889 |
-
index=0,
|
| 890 |
-
key=f"pair_model_b_{benchmark_name}",
|
| 891 |
-
)
|
| 892 |
-
|
| 893 |
-
scatter_df = build_pairwise_scatter_df(
|
| 894 |
-
benchmark_name,
|
| 895 |
-
selected_species,
|
| 896 |
-
selected_assays,
|
| 897 |
-
selected_models,
|
| 898 |
-
selected_datasets,
|
| 899 |
-
model_a,
|
| 900 |
-
model_b,
|
| 901 |
-
)
|
| 902 |
-
|
| 903 |
-
if scatter_df.empty:
|
| 904 |
-
st.info("No overlapping tracks for the selected filters (or one model is missing values).")
|
| 905 |
-
else:
|
| 906 |
-
min_v = float(min(scatter_df["Model A"].min(), scatter_df["Model B"].min()))
|
| 907 |
-
max_v = float(max(scatter_df["Model A"].max(), scatter_df["Model B"].max()))
|
| 908 |
-
pad = 0.05 * (max_v - min_v if max_v > min_v else 1.0)
|
| 909 |
-
axis_range = [min_v - pad, max_v + pad]
|
| 910 |
-
tick_step = (axis_range[1] - axis_range[0]) / 5
|
| 911 |
-
|
| 912 |
-
hover_cols = ["datasets"]
|
| 913 |
-
if benchmark_name == "Functional Tracks" and "track_name_clean" in scatter_df.columns:
|
| 914 |
-
hover_cols.append("track_name_clean")
|
| 915 |
-
else:
|
| 916 |
-
hover_cols.append("datasets")
|
| 917 |
-
|
| 918 |
-
color_col = "assay_type" if "assay_type" in scatter_df.columns else "datasets"
|
| 919 |
-
|
| 920 |
-
fig_scatter = px.scatter(
|
| 921 |
-
scatter_df,
|
| 922 |
-
x="Model B",
|
| 923 |
-
y="Model A",
|
| 924 |
-
color=color_col,
|
| 925 |
-
color_discrete_map=ASSAY_COLORS,
|
| 926 |
-
hover_name="Track",
|
| 927 |
-
hover_data=hover_cols,
|
| 928 |
-
)
|
| 929 |
-
|
| 930 |
-
fig_scatter.add_shape(
|
| 931 |
-
type="line",
|
| 932 |
-
x0=axis_range[0], y0=axis_range[0],
|
| 933 |
-
x1=axis_range[1], y1=axis_range[1],
|
| 934 |
-
xref="x", yref="y",
|
| 935 |
-
line=dict(color="red", dash="dot", width=2),
|
| 936 |
-
)
|
| 937 |
-
|
| 938 |
-
fig_scatter.update_layout(
|
| 939 |
-
height=550,
|
| 940 |
-
margin=dict(l=60, r=20, t=20, b=60),
|
| 941 |
-
xaxis=dict(
|
| 942 |
-
title=f"{model_b} — {cfg['metric_label']}",
|
| 943 |
-
range=axis_range,
|
| 944 |
-
dtick=tick_step,
|
| 945 |
-
constrain="domain",
|
| 946 |
-
),
|
| 947 |
-
yaxis=dict(
|
| 948 |
-
title=f"{model_a} — {cfg['metric_label']}",
|
| 949 |
-
range=axis_range,
|
| 950 |
-
dtick=tick_step,
|
| 951 |
-
scaleanchor="x",
|
| 952 |
-
scaleratio=1,
|
| 953 |
-
constrain="domain",
|
| 954 |
-
),
|
| 955 |
-
plot_bgcolor="rgba(0,0,0,0)",
|
| 956 |
-
paper_bgcolor="rgba(0,0,0,0)",
|
| 957 |
-
)
|
| 958 |
-
|
| 959 |
-
fig_scatter.update_layout(
|
| 960 |
-
legend=dict(
|
| 961 |
-
title="Assay type" if benchmark_name == "Functional Tracks" else "Genome Annotation",
|
| 962 |
-
x=0.98,
|
| 963 |
-
y=0.1,
|
| 964 |
-
xanchor="right",
|
| 965 |
-
yanchor="bottom",
|
| 966 |
-
bgcolor="rgba(255,255,255,0.2)", # semi-transparent white
|
| 967 |
-
bordercolor="rgba(0,0,0,0.2)",
|
| 968 |
-
borderwidth=1,
|
| 969 |
-
)
|
| 970 |
-
)
|
| 971 |
-
|
| 972 |
-
st.plotly_chart(fig_scatter, use_container_width=True)
|
| 973 |
-
|
| 974 |
with right:
|
| 975 |
-
|
| 976 |
|
| 977 |
-
|
| 978 |
-
|
| 979 |
-
options=["GPU (hours)", "Steps (billions)"],
|
| 980 |
-
index=0,
|
| 981 |
-
key=f"conv_x_mode_{benchmark_name}",
|
| 982 |
-
)
|
| 983 |
|
| 984 |
|
| 985 |
-
|
| 986 |
-
|
| 987 |
-
|
| 988 |
-
selected_assays,
|
| 989 |
-
selected_models,
|
| 990 |
-
selected_datasets,
|
| 991 |
-
x_mode=x_mode,
|
| 992 |
-
)
|
| 993 |
|
| 994 |
-
|
| 995 |
-
|
| 996 |
-
|
| 997 |
-
|
| 998 |
-
|
| 999 |
-
|
| 1000 |
-
y="Performance",
|
| 1001 |
-
text="Model",
|
| 1002 |
-
color="Model",
|
| 1003 |
-
color_discrete_map=MODEL_COLORS,
|
| 1004 |
-
hover_data=["Model", "X", "Performance"],
|
| 1005 |
-
)
|
| 1006 |
-
fig_conv.update_layout(
|
| 1007 |
-
height=550,
|
| 1008 |
-
xaxis_title=("GPU (hours)" if x_mode == "GPU (hours)" else x_mode),
|
| 1009 |
-
yaxis_title=cfg["metric_label"],
|
| 1010 |
-
plot_bgcolor="rgba(0,0,0,0)",
|
| 1011 |
-
paper_bgcolor="rgba(0,0,0,0)",
|
| 1012 |
-
showlegend=False, # ✅ no legend
|
| 1013 |
-
)
|
| 1014 |
-
fig_conv.update_traces(
|
| 1015 |
-
marker=dict(size=14), # 👈 bigger dots
|
| 1016 |
-
textposition="top center",
|
| 1017 |
-
)
|
| 1018 |
-
|
| 1019 |
-
# Log scale only makes sense for hours (and sometimes best_step)
|
| 1020 |
-
if x_mode in ["GPU (hours)"]:
|
| 1021 |
-
fig_conv.update_xaxes(
|
| 1022 |
-
type="log",
|
| 1023 |
-
dtick=1,
|
| 1024 |
-
minor=dict(ticks="", showgrid=False),
|
| 1025 |
-
)
|
| 1026 |
-
st.plotly_chart(fig_conv, use_container_width=True)
|
| 1027 |
-
|
| 1028 |
-
# ------------------------------------------------------------------
|
| 1029 |
-
# Violin (full width, below)
|
| 1030 |
-
# ------------------------------------------------------------------
|
| 1031 |
-
st.subheader("🎻 Performance comparison across tracks")
|
| 1032 |
-
|
| 1033 |
-
violin_df = build_violin_df(
|
| 1034 |
-
benchmark_name,
|
| 1035 |
-
selected_species,
|
| 1036 |
-
selected_assays,
|
| 1037 |
-
selected_models,
|
| 1038 |
-
selected_datasets,
|
| 1039 |
)
|
| 1040 |
|
| 1041 |
-
if violin_df.empty:
|
| 1042 |
-
st.info("No data for the selected filters.")
|
| 1043 |
-
else:
|
| 1044 |
-
model_order = (
|
| 1045 |
-
violin_df
|
| 1046 |
-
.groupby("Model")["Score"]
|
| 1047 |
-
.median()
|
| 1048 |
-
.sort_values(ascending=True)
|
| 1049 |
-
.index
|
| 1050 |
-
.tolist()
|
| 1051 |
-
)
|
| 1052 |
-
|
| 1053 |
-
fig_violin = px.violin(
|
| 1054 |
-
violin_df,
|
| 1055 |
-
x="Model",
|
| 1056 |
-
y="Score",
|
| 1057 |
-
color="Model",
|
| 1058 |
-
color_discrete_map=MODEL_COLORS,
|
| 1059 |
-
box=True,
|
| 1060 |
-
points=False,
|
| 1061 |
-
category_orders={"Model": model_order},
|
| 1062 |
-
)
|
| 1063 |
|
| 1064 |
-
|
| 1065 |
-
|
| 1066 |
-
|
| 1067 |
-
|
| 1068 |
-
|
| 1069 |
-
|
| 1070 |
-
|
| 1071 |
-
)
|
| 1072 |
-
|
| 1073 |
-
|
| 1074 |
-
|
| 1075 |
-
|
| 1076 |
-
|
| 1077 |
-
|
| 1078 |
-
st.
|
|
|
|
|
|
|
| 1079 |
|
| 1080 |
|
| 1081 |
if __name__ == "__main__":
|
|
|
|
|
|
|
| 1 |
import os
|
| 2 |
|
| 3 |
+
import numpy as np
|
| 4 |
import pandas as pd
|
|
|
|
| 5 |
import plotly.express as px
|
|
|
|
| 6 |
import plotly.graph_objects as go
|
| 7 |
+
import streamlit as st
|
| 8 |
+
from plotly.subplots import make_subplots
|
| 9 |
|
| 10 |
# ---------------------------------------------------------------------
|
| 11 |
# Page config (must be the first Streamlit command)
|
|
|
|
| 15 |
layout="wide",
|
| 16 |
)
|
| 17 |
|
| 18 |
+
import initial_release # noqa: E402 (reads its CSV on import, so after the page config)
|
| 19 |
+
|
| 20 |
# ---------------------------------------------------------------------
|
| 21 |
# Configuration
|
| 22 |
# ---------------------------------------------------------------------
|
| 23 |
COLORS = {
|
| 24 |
+
'blue_0': '#004697',
|
| 25 |
+
'blue_1': '#3973fc',
|
| 26 |
+
'blue_2': '#7ea4fc',
|
| 27 |
+
'blue_3': '#c3d5fc',
|
| 28 |
+
'red_1': '#ff554d',
|
| 29 |
+
'green_1': '#00b050',
|
| 30 |
+
'green_3': '#c6e0b4',
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 31 |
'gold_1': '#fdb932',
|
|
|
|
| 32 |
'orange_1': '#ff975e',
|
| 33 |
'purple_1': '#9a6ce4',
|
| 34 |
+
'purple_2': '#bb9aef',
|
| 35 |
+
'gray_1': '#808080',
|
| 36 |
+
'gray_2': '#b3b3b3',
|
| 37 |
+
'cyan_1': '#0096b4',
|
| 38 |
+
'magenta_1': '#b428a0',
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
}
|
| 40 |
|
|
|
|
| 41 |
ASSAY_TYPE_MAPPING = {
|
| 42 |
'ATAC-seq': 'chromatin accessibility',
|
| 43 |
'DNase-seq': 'chromatin accessibility',
|
| 44 |
'Histone ChIP-seq': 'histone modifications',
|
| 45 |
+
'TF ChIP-seq': 'TF binding',
|
| 46 |
'PRO-cap': 'transcription initiation',
|
| 47 |
'eCLIP': 'RNA binding sites',
|
| 48 |
'RNA-seq': 'gene expression',
|
| 49 |
'ribo-seq': 'mRNA translation',
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
}
|
| 51 |
|
| 52 |
ASSAY_COLORS = {
|
| 53 |
'chromatin accessibility': '#004697',
|
| 54 |
'histone modifications': '#cc0000',
|
| 55 |
+
'TF binding': '#00b050',
|
| 56 |
'transcription initiation': '#ff9900',
|
| 57 |
'RNA binding sites': '#9933cc',
|
| 58 |
'gene expression': '#009900',
|
| 59 |
'mRNA translation': '#ff6699',
|
| 60 |
+
'other': '#808080',
|
|
|
|
|
|
|
|
|
|
|
|
|
| 61 |
}
|
|
|
|
| 62 |
|
| 63 |
+
CLADE_COLORS = {
|
| 64 |
+
"Animals": COLORS['blue_1'],
|
| 65 |
+
"Plants": COLORS['green_1'],
|
| 66 |
+
"Fungi": COLORS['gold_1'],
|
| 67 |
+
"Protists": COLORS['purple_1'],
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
}
|
| 69 |
|
| 70 |
+
# CSV model name -> the label the paper figure uses. Other models in the CSV (v1 NTv3
|
| 71 |
+
# checkpoints, small BPNet / Residual CNN) are kept in the data but not shown.
|
| 72 |
+
MODEL_LABELS = {
|
| 73 |
+
"BPNet arch. 6M": "BPNet",
|
| 74 |
+
"Residual CNN 44M": "Residual CNN",
|
| 75 |
+
"HyenaDNA 7M": "HyenaDNA",
|
| 76 |
+
"Caduceus 7M": "Caduceus",
|
| 77 |
+
"NTv2 500M": "NTv2",
|
| 78 |
+
"Evo2 1B": "Evo2",
|
| 79 |
+
"PlantCAD2 88M": "PlantCAD2",
|
| 80 |
+
"Borzoi": "Borzoi",
|
| 81 |
+
"AlphaGenome": "AlphaGenome",
|
| 82 |
+
"NTv3 700M (tracks)": "NTv3",
|
| 83 |
}
|
| 84 |
+
NTV3 = "NTv3"
|
| 85 |
|
| 86 |
+
# In the paper figure's architecture blocks, with its colours.
|
| 87 |
+
MODEL_COLORS = {
|
| 88 |
+
"BPNet": COLORS['cyan_1'],
|
| 89 |
+
"Residual CNN": COLORS['magenta_1'],
|
| 90 |
+
"HyenaDNA": COLORS['gold_1'],
|
| 91 |
+
"Caduceus": COLORS['purple_2'],
|
| 92 |
+
"NTv2": COLORS['gray_1'],
|
| 93 |
+
"Evo2": COLORS['green_3'],
|
| 94 |
+
"PlantCAD2": COLORS['purple_1'],
|
| 95 |
+
"Borzoi": COLORS['gray_2'],
|
| 96 |
+
"AlphaGenome": COLORS['orange_1'],
|
| 97 |
+
NTV3: COLORS['blue_0'],
|
| 98 |
}
|
| 99 |
|
| 100 |
+
MODEL_TRAINING_STATUS = {
|
| 101 |
+
"BPNet": "SCRATCH",
|
| 102 |
+
"Residual CNN": "SCRATCH",
|
| 103 |
+
"HyenaDNA": "PRE",
|
| 104 |
+
"Caduceus": "PRE",
|
| 105 |
+
"NTv2": "PRE",
|
| 106 |
+
"Evo2": "PRE",
|
| 107 |
+
"PlantCAD2": "PRE",
|
| 108 |
+
"Borzoi": "PRE",
|
| 109 |
+
"AlphaGenome": "PRE",
|
| 110 |
+
NTV3: "POS",
|
| 111 |
+
}
|
| 112 |
|
| 113 |
MODEL_NAMES = list(MODEL_COLORS.keys())
|
| 114 |
|
| 115 |
+
# CSV species key -> label, grouped by clade, in the paper's order.
|
|
|
|
|
|
|
| 116 |
SPECIES_GROUPS = {
|
| 117 |
+
"Animals": {"human": "Human", "cattle": "Cattle", "rat": "Rat"},
|
| 118 |
+
"Plants": {
|
| 119 |
+
"arabidopsis": "A. thaliana",
|
| 120 |
+
"rice": "Rice",
|
| 121 |
+
"maize": "Maize",
|
| 122 |
+
"tomato": "Tomato",
|
| 123 |
+
},
|
| 124 |
+
"Fungi": {"yeast": "S. cerevisiae", "fission yeast": "S. pombe"},
|
| 125 |
+
"Protists": {"plasmodium": "P. falciparum", "dictyostelium": "D. discoideum"},
|
| 126 |
}
|
| 127 |
+
SPECIES_LABEL = {k: v for group in SPECIES_GROUPS.values() for k, v in group.items()}
|
| 128 |
+
SPECIES_ORDER = list(SPECIES_LABEL)
|
| 129 |
+
SPECIES_CLADE = {s: name for name, group in SPECIES_GROUPS.items() for s in group}
|
| 130 |
+
SPECIES_ICON = {
|
| 131 |
+
"human": "🧍",
|
| 132 |
+
"cattle": "🐄",
|
| 133 |
+
"rat": "🐀",
|
| 134 |
+
"arabidopsis": "🌱",
|
| 135 |
+
"rice": "🌾",
|
| 136 |
+
"maize": "🌽",
|
| 137 |
+
"tomato": "🍅",
|
| 138 |
+
"yeast": "🍞",
|
| 139 |
+
"fission yeast": "🍺",
|
| 140 |
+
"plasmodium": "🦟",
|
| 141 |
+
"dictyostelium": "🦠",
|
| 142 |
+
}
|
| 143 |
+
SPECIES_COMMON_NAME = {
|
| 144 |
+
"arabidopsis": "thale cress",
|
| 145 |
+
"yeast": "baker's yeast",
|
| 146 |
+
"fission yeast": "fission yeast",
|
| 147 |
+
"plasmodium": "malaria parasite",
|
| 148 |
+
"dictyostelium": "social amoeba",
|
| 149 |
+
}
|
| 150 |
+
METRIC_LABEL = "Pearson correlation"
|
| 151 |
|
| 152 |
+
_LAST_UPDATED = "Sep 24, 2026"
|
|
|
|
| 153 |
_INTRO = """
|
| 154 |
+
The **NTv3 Benchmark** is 155 long-range functional genomics tracks for evaluating models
|
| 155 |
+
under realistic 32 kb input, single-base-pair output settings. The tracks span chromatin accessibility (ATAC-seq),
|
| 156 |
+
histone modifications and transcription factor binding (ChIP-seq), transcription initiation (PRO-cap),
|
| 157 |
+
RNA binding (eCLIP), gene expression (RNA-seq) and translation (Ribo-seq).
|
| 158 |
+
|
| 159 |
+
The benchmark covers 11 phylogenetically diverse species:
|
| 160 |
+
**animals** (human, cattle, rat), **plants** (*A. thaliana*, rice, maize, tomato),
|
| 161 |
+
**fungi** (*S. cerevisiae*, *S. pombe*) and **protists** (*P. falciparum*, *D. discoideum*).
|
| 162 |
+
|
| 163 |
+
The metric is the **Pearson correlation** on the test set, per track, averaged over three
|
| 164 |
+
seeds where available. Not every model was run on every species, so scores are never
|
| 165 |
+
averaged across species: pick a species in the sidebar for its own leaderboard, where every
|
| 166 |
+
model shown was evaluated on all of that species' tracks. AlphaGenome is fine-tuned from its
|
| 167 |
+
fold-1 checkpoint on human, whose test regions the all-folds checkpoint saw, and from its
|
| 168 |
+
all-folds checkpoint on every other species.
|
|
|
|
|
|
|
|
|
|
| 169 |
"""
|
| 170 |
|
| 171 |
HERE = os.path.dirname(os.path.abspath(__file__)) # /app/src
|
| 172 |
PROJECT_ROOT = os.path.dirname(HERE) # /app
|
| 173 |
DATA_DIR = os.path.join(PROJECT_ROOT, "data")
|
| 174 |
+
SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "ntv3_benchmark_results.csv")
|
|
|
|
| 175 |
|
| 176 |
# ---------------------------------------------------------------------
|
| 177 |
# Data loading & preprocessing
|
|
|
|
| 179 |
|
| 180 |
|
| 181 |
@st.cache_data
|
| 182 |
+
def load_raw_data() -> pd.DataFrame:
|
| 183 |
df = pd.read_csv(SINGLE_TABLE_PATH)
|
| 184 |
df.columns = [c.strip() for c in df.columns]
|
| 185 |
return df
|
| 186 |
|
| 187 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 188 |
@st.cache_data
|
| 189 |
+
def load_track_data() -> pd.DataFrame:
|
| 190 |
+
"""One row per (species, track, model): the score averaged over seeds."""
|
| 191 |
+
df = load_raw_data().rename(columns={"Metric": "Score", "model_name": "Model"})
|
| 192 |
+
df["Model"] = df["Model"].map(MODEL_LABELS)
|
| 193 |
+
df = df[df["Model"].notna() & df["species"].isin(SPECIES_ORDER)]
|
| 194 |
df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
|
| 195 |
+
keys = ["species", "datasets", "Model", "assay_type", "track_name_clean"]
|
| 196 |
+
out = df.groupby(keys, as_index=False, dropna=False).agg(
|
| 197 |
+
Score=("Score", "mean"), Seeds=("Score", "size")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 198 |
)
|
| 199 |
+
out["assay_type"] = out["assay_type"].map(ASSAY_TYPE_MAPPING).fillna("other")
|
| 200 |
+
return out
|
| 201 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 202 |
|
| 203 |
+
@st.cache_data
|
| 204 |
+
def load_run_data() -> pd.DataFrame:
|
| 205 |
+
"""One row per training run: its best checkpoint's step and GPU hours."""
|
| 206 |
+
df = load_raw_data().rename(columns={"model_name": "Model"})
|
| 207 |
+
df["Model"] = df["Model"].map(MODEL_LABELS)
|
| 208 |
+
df = df[df["Model"].notna() & df["species"].isin(SPECIES_ORDER)]
|
| 209 |
+
run_cols = ["species", "Model", "best_step", "best_step_time", "running_time"]
|
| 210 |
+
runs = df.drop_duplicates(run_cols).copy()
|
| 211 |
+
# Training steps only: excludes job setup and validation, whose cadence differs
|
| 212 |
+
# between runs of the benchmark.
|
| 213 |
+
# Parsed one by one: pandas 3 misparses a column mixing fraction lengths.
|
| 214 |
+
hours = runs["best_step_train_time"].map(
|
| 215 |
+
lambda v: pd.Timedelta(v).total_seconds() / 3600 if pd.notna(v) else np.nan
|
| 216 |
)
|
| 217 |
+
runs["GPU hours"] = hours * pd.to_numeric(runs["num_gpus"], errors="coerce")
|
| 218 |
+
runs["Tokens (billions)"] = pd.to_numeric(runs["best_step"], errors="coerce") / 1e9
|
| 219 |
+
return runs
|
| 220 |
|
| 221 |
|
| 222 |
+
_TRACKS = load_track_data()
|
| 223 |
+
_RUNS = load_run_data()
|
| 224 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 225 |
|
| 226 |
+
def models_in(df: pd.DataFrame) -> list[str]:
|
| 227 |
+
"""Models present in `df`, in the paper's block order."""
|
| 228 |
+
present = set(df["Model"])
|
| 229 |
+
return [m for m in MODEL_NAMES if m in present]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 230 |
|
| 231 |
|
| 232 |
# ---------------------------------------------------------------------
|
|
|
|
| 234 |
# ---------------------------------------------------------------------
|
| 235 |
|
| 236 |
|
| 237 |
+
def build_species_scores(df: pd.DataFrame) -> pd.DataFrame:
|
| 238 |
+
"""Mean over tracks per (species, model): the paper figure's points."""
|
| 239 |
+
return df.groupby(["species", "Model"], as_index=False).agg(
|
| 240 |
+
Score=("Score", "mean"),
|
| 241 |
+
Tracks=("datasets", "nunique"),
|
| 242 |
+
Seeds=("Seeds", "max"),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
)
|
| 244 |
|
|
|
|
|
|
|
| 245 |
|
| 246 |
+
def build_leaderboard(df: pd.DataFrame) -> pd.DataFrame:
|
| 247 |
+
if df.empty:
|
| 248 |
+
return pd.DataFrame(columns=["Model", "Training", "Num entries", "Mean score"])
|
| 249 |
+
agg = df.groupby("Model").agg(
|
| 250 |
+
**{
|
| 251 |
+
"Num entries": ("Score", "count"),
|
| 252 |
+
"Seeds": ("Seeds", "max"),
|
| 253 |
+
"Mean score": ("Score", "mean"),
|
| 254 |
+
}
|
| 255 |
+
).reset_index()
|
| 256 |
agg["Mean score"] = agg["Mean score"].round(3)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 257 |
agg["Training"] = agg["Model"].map(MODEL_TRAINING_STATUS).fillna("UNKNOWN")
|
|
|
|
|
|
|
| 258 |
agg = agg.sort_values("Mean score", ascending=False).reset_index(drop=True)
|
| 259 |
+
agg = agg[["Model", "Training", "Num entries", "Seeds", "Mean score"]]
|
|
|
|
|
|
|
|
|
|
|
|
|
| 260 |
agg.index += 1
|
|
|
|
| 261 |
return agg
|
| 262 |
|
| 263 |
|
| 264 |
+
def build_category_model_df(df: pd.DataFrame) -> pd.DataFrame:
|
| 265 |
+
"""Mean score per (assay category, model)."""
|
| 266 |
+
if df.empty:
|
| 267 |
+
return pd.DataFrame(columns=["Category", "Model", "Mean score"])
|
| 268 |
+
out = (
|
| 269 |
+
df.groupby(["assay_type", "Model"], as_index=False)["Score"].mean()
|
| 270 |
+
.rename(columns={"assay_type": "Category", "Score": "Mean score"})
|
|
|
|
|
|
|
|
|
|
|
|
|
| 271 |
)
|
| 272 |
+
out["Mean score"] = out["Mean score"].round(3)
|
| 273 |
+
return out
|
| 274 |
|
| 275 |
|
| 276 |
+
def build_pairwise_scatter_df(df: pd.DataFrame, model_a: str, model_b: str) -> pd.DataFrame:
|
| 277 |
+
"""Scores of two models on the tracks both were evaluated on."""
|
| 278 |
+
sub = df[df["Model"].isin([model_a, model_b])]
|
| 279 |
+
wide = sub.pivot_table(
|
| 280 |
+
index=["species", "assay_type", "datasets", "track_name_clean"],
|
| 281 |
+
columns="Model",
|
| 282 |
+
values="Score",
|
| 283 |
+
).reset_index()
|
| 284 |
+
if model_a not in wide.columns or model_b not in wide.columns:
|
| 285 |
+
return pd.DataFrame()
|
| 286 |
+
wide = wide.dropna(subset=[model_a, model_b])
|
| 287 |
+
wide["Species"] = wide["species"].map(SPECIES_LABEL)
|
| 288 |
+
wide["Track"] = wide["Species"] + " / " + wide["track_name_clean"].astype(str)
|
| 289 |
+
return wide.rename(columns={model_a: "Model A", model_b: "Model B"})
|
| 290 |
+
|
| 291 |
+
|
| 292 |
+
def aggregate_pairs_by_species(pairs: pd.DataFrame) -> pd.DataFrame:
|
| 293 |
+
"""One point per species: both models' means over the tracks they share there."""
|
| 294 |
+
clade = {s: name for name, group in SPECIES_GROUPS.items() for s in group}
|
| 295 |
+
out = pairs.groupby(["species", "Species"], as_index=False).agg(
|
| 296 |
+
**{
|
| 297 |
+
"Model A": ("Model A", "mean"),
|
| 298 |
+
"Model B": ("Model B", "mean"),
|
| 299 |
+
"Shared tracks": ("datasets", "size"),
|
| 300 |
+
}
|
| 301 |
)
|
| 302 |
+
out["Clade"] = out["species"].map(clade)
|
| 303 |
+
out["Track"] = out["Species"]
|
| 304 |
+
return out
|
| 305 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 306 |
|
| 307 |
+
# ---------------------------------------------------------------------
|
| 308 |
+
# Plots
|
| 309 |
+
# ---------------------------------------------------------------------
|
| 310 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 311 |
|
| 312 |
+
def plot_species_overview(scores: pd.DataFrame, models: list[str]) -> go.Figure:
|
| 313 |
+
"""The paper figure: one column per species, one point per model evaluated on it."""
|
| 314 |
+
cols = [s for s in SPECIES_ORDER if s in set(scores["species"])]
|
| 315 |
+
labels = [SPECIES_LABEL[s] for s in cols]
|
| 316 |
+
fig = go.Figure()
|
| 317 |
+
# Thin guide line per species, from the weakest to the strongest model.
|
| 318 |
+
span = scores.groupby("species")["Score"].agg(["min", "max"]).reindex(cols)
|
| 319 |
+
for label, (lo, hi) in zip(labels, span.itertuples(index=False)):
|
| 320 |
+
fig.add_shape(
|
| 321 |
+
type="line", x0=label, x1=label, y0=lo, y1=hi, layer="below",
|
| 322 |
+
line=dict(color="rgba(128,128,128,0.35)", width=2),
|
| 323 |
+
)
|
| 324 |
+
# AlphaGenome is drawn last so it stays visible where it sits on NTv3's marker;
|
| 325 |
+
# `legendrank` keeps the legend in the paper's block order.
|
| 326 |
+
draw_order = [m for m in models if m != "AlphaGenome"] + (
|
| 327 |
+
["AlphaGenome"] if "AlphaGenome" in models else []
|
| 328 |
+
)
|
| 329 |
+
for model in draw_order:
|
| 330 |
+
sub = scores[scores["Model"] == model].set_index("species").reindex(cols)
|
| 331 |
+
is_ntv3 = model == NTV3
|
| 332 |
+
fig.add_trace(
|
| 333 |
+
go.Scatter(
|
| 334 |
+
x=labels,
|
| 335 |
+
y=sub["Score"],
|
| 336 |
+
mode="markers",
|
| 337 |
+
name=model,
|
| 338 |
+
legendrank=models.index(model),
|
| 339 |
+
marker=dict(
|
| 340 |
+
color=MODEL_COLORS[model],
|
| 341 |
+
size=16 if is_ntv3 else 11,
|
| 342 |
+
symbol="diamond" if is_ntv3 else "circle",
|
| 343 |
+
line=dict(color="rgba(0,0,0,0.5)", width=1),
|
| 344 |
+
),
|
| 345 |
+
customdata=np.stack([sub["Tracks"], sub["Seeds"]], axis=-1),
|
| 346 |
+
hovertemplate=(
|
| 347 |
+
f"<b>{model}</b><br>%{{x}}<br>{METRIC_LABEL}: %{{y:.3f}}"
|
| 348 |
+
"<br>%{customdata[0]} tracks, %{customdata[1]} seed(s)<extra></extra>"
|
| 349 |
+
),
|
| 350 |
+
)
|
| 351 |
+
)
|
| 352 |
+
# Clade separators and clade names above the plot.
|
| 353 |
+
x = 0
|
| 354 |
+
for name, group in SPECIES_GROUPS.items():
|
| 355 |
+
n = sum(1 for s in group if s in cols)
|
| 356 |
+
if n == 0:
|
| 357 |
+
continue
|
| 358 |
+
fig.add_annotation(
|
| 359 |
+
x=x + (n - 1) / 2, y=1.0, xref="x", yref="paper", yanchor="bottom",
|
| 360 |
+
text=f"<b>{name}</b>", showarrow=False,
|
| 361 |
+
)
|
| 362 |
+
x += n
|
| 363 |
+
if x < len(cols):
|
| 364 |
+
fig.add_vline(x=x - 0.5, line=dict(color="rgba(128,128,128,0.4)", dash="dot"))
|
| 365 |
+
fig.update_xaxes(categoryorder="array", categoryarray=labels, title_text="")
|
| 366 |
+
fig.update_yaxes(
|
| 367 |
+
title_text=f"{METRIC_LABEL} on test set",
|
| 368 |
+
showgrid=True,
|
| 369 |
+
gridcolor="rgba(128,128,128,0.15)",
|
| 370 |
+
automargin=True,
|
| 371 |
+
tickformat=".1f",
|
| 372 |
+
)
|
| 373 |
+
fig.update_layout(
|
| 374 |
+
height=600,
|
| 375 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 376 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 377 |
+
legend=dict(orientation="h", yanchor="top", y=-0.1, x=0.5, xanchor="center"),
|
| 378 |
+
margin=dict(t=40, l=80, r=10, b=40),
|
| 379 |
+
)
|
| 380 |
+
return fig
|
| 381 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 382 |
|
| 383 |
+
def plot_mean_bars(leaderboard: pd.DataFrame) -> go.Figure:
|
| 384 |
+
bar_df = leaderboard.sort_values("Mean score", ascending=True)
|
| 385 |
+
fig = px.bar(
|
| 386 |
+
bar_df,
|
| 387 |
+
x="Model",
|
| 388 |
+
y="Mean score",
|
| 389 |
+
color="Model",
|
| 390 |
+
color_discrete_map=MODEL_COLORS,
|
| 391 |
+
category_orders={"Model": bar_df["Model"].tolist()},
|
| 392 |
+
)
|
| 393 |
+
fig.update_layout(
|
| 394 |
+
height=500,
|
| 395 |
+
xaxis_title="",
|
| 396 |
+
yaxis_title=METRIC_LABEL,
|
| 397 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 398 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 399 |
+
bargap=0.08,
|
| 400 |
+
)
|
| 401 |
+
fig.update_xaxes(showticklabels=False)
|
| 402 |
+
return fig
|
| 403 |
|
|
|
|
|
|
|
| 404 |
|
| 405 |
+
def plot_breakdown_facets_sorted_models(breakdown_df: pd.DataFrame, height: int = 300):
|
| 406 |
+
categories = sorted(breakdown_df["Category"].dropna().unique())
|
| 407 |
+
if not categories:
|
| 408 |
+
return None
|
| 409 |
y_min = breakdown_df["Mean score"].min()
|
| 410 |
y_max = breakdown_df["Mean score"].max()
|
| 411 |
pad = 0.05 * (y_max - y_min if y_max > y_min else 1.0)
|
| 412 |
+
y_range = [min(0, y_min - pad), y_max + pad]
|
| 413 |
|
| 414 |
fig = make_subplots(
|
| 415 |
+
rows=1,
|
| 416 |
+
cols=len(categories),
|
| 417 |
subplot_titles=categories,
|
| 418 |
shared_yaxes=True,
|
| 419 |
+
horizontal_spacing=0.04,
|
| 420 |
)
|
|
|
|
| 421 |
for i, cat in enumerate(categories):
|
| 422 |
+
sub = breakdown_df[breakdown_df["Category"] == cat].sort_values("Mean score")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 423 |
fig.add_trace(
|
| 424 |
go.Bar(
|
| 425 |
x=sub["Model"],
|
| 426 |
y=sub["Mean score"],
|
| 427 |
marker_color=[MODEL_COLORS.get(m, "#808080") for m in sub["Model"]],
|
| 428 |
showlegend=False,
|
| 429 |
+
hovertemplate="%{x}: %{y:.3f}<extra></extra>",
|
| 430 |
),
|
| 431 |
+
row=1,
|
| 432 |
+
col=i + 1,
|
| 433 |
)
|
| 434 |
+
fig.update_xaxes(showticklabels=False, title_text="", row=1, col=i + 1)
|
| 435 |
+
fig.update_yaxes(range=y_range, title_text="", row=1, col=i + 1)
|
|
|
|
|
|
|
| 436 |
fig.update_layout(
|
| 437 |
height=height,
|
| 438 |
plot_bgcolor="rgba(0,0,0,0)",
|
| 439 |
paper_bgcolor="rgba(0,0,0,0)",
|
| 440 |
margin=dict(t=60, l=10, r=10, b=10),
|
| 441 |
)
|
| 442 |
+
fig.update_yaxes(title_text=METRIC_LABEL, row=1, col=1)
|
|
|
|
|
|
|
|
|
|
| 443 |
return fig
|
| 444 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 445 |
|
| 446 |
+
def show_head_to_head(
|
| 447 |
+
df: pd.DataFrame, key: str, per_species: bool = False, title: str | None = None
|
| 448 |
+
) -> None:
|
| 449 |
+
"""Two models on their shared tracks only, with the number of tracks compared.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 450 |
|
| 451 |
+
Per track by default, one point per track coloured by assay; `per_species` shows
|
| 452 |
+
one point per species instead, each the mean over that species' shared tracks.
|
| 453 |
+
"""
|
| 454 |
+
st.markdown(title or "#### ⚖️ Head-to-head (per track)")
|
| 455 |
+
options = models_in(df)
|
| 456 |
+
if len(options) < 2:
|
| 457 |
+
st.info("Fewer than two models were evaluated here.")
|
| 458 |
+
return
|
| 459 |
+
default_a = options.index(NTV3) if NTV3 in options else 0
|
| 460 |
+
c_a, c_b = st.columns([1, 1])
|
| 461 |
+
with c_a:
|
| 462 |
+
model_a = st.selectbox(
|
| 463 |
+
"Model A (y-axis)", options=options, index=default_a, key=f"a_{key}"
|
| 464 |
+
)
|
| 465 |
+
with c_b:
|
| 466 |
+
b_options = [m for m in options if m != model_a]
|
| 467 |
+
default_b = b_options.index("AlphaGenome") if "AlphaGenome" in b_options else 0
|
| 468 |
+
model_b = st.selectbox(
|
| 469 |
+
"Model B (x-axis)", options=b_options, index=default_b, key=f"b_{key}"
|
| 470 |
+
)
|
| 471 |
|
| 472 |
+
scatter_df = build_pairwise_scatter_df(df, model_a, model_b)
|
| 473 |
+
if scatter_df.empty:
|
| 474 |
+
st.info(f"{model_a} and {model_b} share no evaluated track.")
|
| 475 |
+
return
|
| 476 |
+
|
| 477 |
+
n_tracks = len(scatter_df)
|
| 478 |
+
n_species = scatter_df["species"].nunique()
|
| 479 |
+
if per_species:
|
| 480 |
+
scatter_df = aggregate_pairs_by_species(scatter_df)
|
| 481 |
+
wins = int((scatter_df["Model A"] > scatter_df["Model B"]).sum())
|
| 482 |
+
st.caption(
|
| 483 |
+
f"Compared on **n = {n_species} shared species** ({n_tracks} shared "
|
| 484 |
+
f"tracks). {model_a} is higher on {wins} of them."
|
| 485 |
+
)
|
| 486 |
+
color_col, hover = "Clade", {"Shared tracks": True}
|
| 487 |
else:
|
| 488 |
+
wins = int((scatter_df["Model A"] > scatter_df["Model B"]).sum())
|
| 489 |
+
st.caption(
|
| 490 |
+
f"Compared on **n = {n_tracks} shared tracks** over {n_species} "
|
| 491 |
+
f"species. {model_a} is higher on {wins} of them."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 492 |
)
|
| 493 |
+
color_col, hover = "assay_type", {"Species": True, "datasets": True}
|
| 494 |
+
|
| 495 |
+
min_v = float(min(scatter_df["Model A"].min(), scatter_df["Model B"].min()))
|
| 496 |
+
max_v = float(max(scatter_df["Model A"].max(), scatter_df["Model B"].max()))
|
| 497 |
+
pad = 0.05 * (max_v - min_v if max_v > min_v else 1.0)
|
| 498 |
+
axis_range = [min_v - pad, max_v + pad]
|
| 499 |
+
|
| 500 |
+
color_map = ASSAY_COLORS if color_col == "assay_type" else CLADE_COLORS
|
| 501 |
+
fig = px.scatter(
|
| 502 |
+
scatter_df,
|
| 503 |
+
x="Model B",
|
| 504 |
+
y="Model A",
|
| 505 |
+
color=color_col,
|
| 506 |
+
color_discrete_map=color_map,
|
| 507 |
+
category_orders={"Clade": list(SPECIES_GROUPS)},
|
| 508 |
+
hover_name="Track",
|
| 509 |
+
hover_data={**hover, "Model A": ":.3f", "Model B": ":.3f"},
|
| 510 |
+
text="Species" if per_species else None,
|
|
|
|
|
|
|
| 511 |
)
|
| 512 |
+
if per_species:
|
| 513 |
+
fig.update_traces(marker=dict(size=13), textposition="top center")
|
| 514 |
+
fig.add_shape(
|
| 515 |
+
type="line",
|
| 516 |
+
x0=axis_range[0], y0=axis_range[0], x1=axis_range[1], y1=axis_range[1],
|
| 517 |
+
xref="x", yref="y",
|
| 518 |
+
line=dict(color="red", dash="dot", width=2),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 519 |
)
|
| 520 |
+
fig.update_layout(
|
| 521 |
+
height=620,
|
| 522 |
+
margin=dict(l=60, r=20, t=20, b=110),
|
| 523 |
+
xaxis=dict(
|
| 524 |
+
title=f"{model_b} — {METRIC_LABEL}",
|
| 525 |
+
range=axis_range, dtick=0.1, tickformat=".1f", constrain="domain",
|
| 526 |
+
),
|
| 527 |
+
yaxis=dict(
|
| 528 |
+
title=f"{model_a} — {METRIC_LABEL}",
|
| 529 |
+
range=axis_range, dtick=0.1, tickformat=".1f",
|
| 530 |
+
scaleanchor="x", scaleratio=1, constrain="domain",
|
| 531 |
+
),
|
| 532 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 533 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 534 |
+
legend=dict(
|
| 535 |
+
title="",
|
| 536 |
+
orientation="h",
|
| 537 |
+
x=0.5, xanchor="center", y=-0.15, yanchor="top",
|
| 538 |
+
),
|
| 539 |
)
|
| 540 |
+
st.plotly_chart(fig, width="stretch")
|
| 541 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 542 |
|
| 543 |
+
def build_convergence_df(tracks: pd.DataFrame, runs: pd.DataFrame, x_col: str) -> pd.DataFrame:
|
| 544 |
+
"""Per model: mean score and the mean cost of reaching its best checkpoint."""
|
| 545 |
+
perf = tracks.groupby("Model", as_index=False)["Score"].mean()
|
| 546 |
+
cost = runs.groupby("Model", as_index=False).agg(X=(x_col, "mean"), Runs=(x_col, "count"))
|
| 547 |
+
out = perf.merge(cost, on="Model").dropna(subset=["X"])
|
| 548 |
+
out["Score"] = out["Score"].round(3)
|
| 549 |
+
return out
|
|
|
|
|
|
|
| 550 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 551 |
|
| 552 |
+
def show_convergence(tracks: pd.DataFrame, runs: pd.DataFrame, key: str) -> None:
|
| 553 |
+
st.markdown("#### ⏱️ Time to convergence")
|
| 554 |
+
x_mode = st.selectbox(
|
| 555 |
+
"X-axis",
|
| 556 |
+
options=["GPU hours", "Tokens (billions)"],
|
| 557 |
+
index=0,
|
| 558 |
+
key=f"conv_{key}",
|
| 559 |
+
)
|
| 560 |
+
conv_df = build_convergence_df(tracks, runs, x_mode)
|
| 561 |
+
if conv_df.empty:
|
| 562 |
+
st.info("No convergence data for this species.")
|
| 563 |
+
return
|
| 564 |
+
if x_mode == "GPU hours":
|
| 565 |
+
caption = (
|
| 566 |
+
"GPU hours spent in training steps until the best validation checkpoint "
|
| 567 |
+
"(the one scored on the test set), × GPUs used, averaged over seeds. Job "
|
| 568 |
+
"setup and validation are excluded. All runs on H100s."
|
| 569 |
)
|
| 570 |
+
else:
|
| 571 |
+
caption = (
|
| 572 |
+
"Training tokens seen at the best validation checkpoint, averaged over "
|
| 573 |
+
"seeds. Validation ran every 1.05B tokens on most runs, so this is "
|
| 574 |
+
"resolved to about 1B tokens."
|
| 575 |
+
)
|
| 576 |
+
st.caption(caption)
|
| 577 |
+
fig = px.scatter(
|
| 578 |
+
conv_df,
|
| 579 |
+
x="X",
|
| 580 |
+
y="Score",
|
| 581 |
+
text="Model",
|
| 582 |
+
color="Model",
|
| 583 |
+
color_discrete_map=MODEL_COLORS,
|
| 584 |
+
hover_data={"Model": True, "X": ":.2f", "Score": ":.3f", "Runs": True},
|
| 585 |
+
)
|
| 586 |
+
fig.update_traces(marker=dict(size=14), textposition="middle right")
|
| 587 |
+
fig.update_layout(
|
| 588 |
+
height=620,
|
| 589 |
+
xaxis_title=x_mode + (" (log scale)" if x_mode == "GPU hours" else ""),
|
| 590 |
+
yaxis_title=METRIC_LABEL,
|
| 591 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 592 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 593 |
+
showlegend=False,
|
| 594 |
+
)
|
| 595 |
+
if x_mode == "GPU hours":
|
| 596 |
+
lo, hi = np.log10(conv_df["X"].min()), np.log10(conv_df["X"].max())
|
| 597 |
+
fig.update_xaxes(
|
| 598 |
+
type="log", dtick=1, range=[lo - 0.15, hi + 0.45],
|
| 599 |
+
minor=dict(ticks="", showgrid=False),
|
| 600 |
+
)
|
| 601 |
+
else:
|
| 602 |
+
lo, hi = conv_df["X"].min(), conv_df["X"].max()
|
| 603 |
+
fig.update_xaxes(range=[lo - 0.05 * (hi - lo), hi + 0.3 * (hi - lo)])
|
| 604 |
+
st.plotly_chart(fig, width="stretch")
|
| 605 |
+
|
| 606 |
+
|
| 607 |
+
def plot_violin(df: pd.DataFrame) -> go.Figure:
|
| 608 |
+
order = df.groupby("Model")["Score"].median().sort_values().index.tolist()
|
| 609 |
+
fig = px.violin(
|
| 610 |
+
df,
|
| 611 |
+
x="Model",
|
| 612 |
+
y="Score",
|
| 613 |
+
color="Model",
|
| 614 |
+
color_discrete_map=MODEL_COLORS,
|
| 615 |
+
box=True,
|
| 616 |
+
points=False,
|
| 617 |
+
category_orders={"Model": order},
|
| 618 |
+
hover_data=["track_name_clean"],
|
| 619 |
+
)
|
| 620 |
+
fig.update_layout(
|
| 621 |
+
height=650,
|
| 622 |
+
xaxis_title="",
|
| 623 |
+
yaxis_title=METRIC_LABEL,
|
| 624 |
+
plot_bgcolor="rgba(0,0,0,0)",
|
| 625 |
+
paper_bgcolor="rgba(0,0,0,0)",
|
| 626 |
+
showlegend=False,
|
| 627 |
+
)
|
| 628 |
+
fig.update_traces(box_visible=True, meanline_visible=False)
|
| 629 |
+
return fig
|
| 630 |
|
| 631 |
|
| 632 |
# ---------------------------------------------------------------------
|
| 633 |
+
# Pages
|
| 634 |
# ---------------------------------------------------------------------
|
| 635 |
|
| 636 |
|
| 637 |
+
SIDEBAR_CSS = """
|
| 638 |
+
<style>
|
| 639 |
+
[data-testid="stSidebarNav"] a, [data-testid="stSidebarNavLink"] {
|
| 640 |
+
padding-top: 0.35rem; padding-bottom: 0.35rem; border-radius: 0.5rem;
|
| 641 |
+
}
|
| 642 |
+
[data-testid="stSidebarNav"] a span, [data-testid="stSidebarNavLink"] span {
|
| 643 |
+
font-size: 1.08rem;
|
| 644 |
+
}
|
| 645 |
+
[data-testid="stNavSectionHeader"], [data-testid="stSidebarNav"] header {
|
| 646 |
+
font-size: 0.85rem; font-weight: 700; letter-spacing: 0.06em;
|
| 647 |
+
text-transform: uppercase; opacity: 0.7; margin-top: 0.8rem;
|
| 648 |
+
}
|
| 649 |
+
</style>
|
| 650 |
+
"""
|
| 651 |
|
| 652 |
+
|
| 653 |
+
def overview_page() -> None:
|
| 654 |
+
df = _TRACKS
|
|
|
|
| 655 |
st.title("🧬 NTv3 Benchmark")
|
| 656 |
st.markdown(_INTRO)
|
| 657 |
st.markdown(f"_Last updated: **{_LAST_UPDATED}**_")
|
| 658 |
+
st.divider()
|
| 659 |
|
| 660 |
+
st.subheader("📊 Every model on every species")
|
| 661 |
+
st.caption(
|
| 662 |
+
"One point per model and species: the mean over that species' tracks. "
|
| 663 |
+
"A model with no point on a species was not evaluated on it. Hover for values."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 664 |
)
|
| 665 |
+
scores = build_species_scores(df)
|
| 666 |
+
st.plotly_chart(plot_species_overview(scores, models_in(df)), width="stretch")
|
| 667 |
|
| 668 |
+
left, right = st.columns([1, 1], gap="large")
|
| 669 |
+
with left:
|
| 670 |
+
show_head_to_head(df, key="overview_track")
|
| 671 |
+
with right:
|
| 672 |
+
show_head_to_head(
|
| 673 |
+
df,
|
| 674 |
+
key="overview_species",
|
| 675 |
+
per_species=True,
|
| 676 |
+
title="#### ⚖️ Head-to-head (per species)",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 677 |
)
|
|
|
|
| 678 |
|
| 679 |
+
|
| 680 |
+
def species_page(species: str) -> None:
|
| 681 |
+
sub = _TRACKS[_TRACKS["species"] == species]
|
| 682 |
+
runs = _RUNS[(_RUNS["species"] == species) & _RUNS["Model"].isin(set(sub["Model"]))]
|
| 683 |
+
n_tracks = sub["datasets"].nunique()
|
| 684 |
+
common = SPECIES_COMMON_NAME.get(species)
|
| 685 |
+
st.title(f"{SPECIES_ICON[species]} {SPECIES_LABEL[species]}")
|
| 686 |
+
st.caption(
|
| 687 |
+
f"{SPECIES_CLADE[species]}{f' · {common}' if common else ''} · {n_tracks} tracks · "
|
| 688 |
+
f"{sub['Model'].nunique()} models, each evaluated on all {n_tracks} tracks."
|
| 689 |
)
|
| 690 |
|
| 691 |
+
leaderboard = build_leaderboard(sub)
|
| 692 |
col1, col2 = st.columns([1, 1])
|
|
|
|
| 693 |
with col1:
|
| 694 |
st.subheader("🏅 Leaderboard")
|
|
|
|
| 695 |
st.write("\n")
|
| 696 |
st.write("\n")
|
| 697 |
+
st.write("\n")
|
| 698 |
+
st.dataframe(leaderboard, width="stretch")
|
|
|
|
|
|
|
|
|
|
| 699 |
with col2:
|
| 700 |
st.subheader("📈 Mean score per model")
|
| 701 |
+
st.plotly_chart(plot_mean_bars(leaderboard), width="stretch")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 702 |
|
| 703 |
+
breakdown_df = build_category_model_df(sub)
|
| 704 |
+
if breakdown_df["Category"].nunique() > 1:
|
| 705 |
+
st.subheader("🧪 Mean score by assay type")
|
| 706 |
+
st.plotly_chart(plot_breakdown_facets_sorted_models(breakdown_df), width="stretch")
|
| 707 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 708 |
left, right = st.columns([1, 1], gap="large")
|
|
|
|
| 709 |
with left:
|
| 710 |
+
show_head_to_head(sub, key=species)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 711 |
with right:
|
| 712 |
+
show_convergence(sub, runs, key=species)
|
| 713 |
|
| 714 |
+
st.subheader("🎻 Performance across tracks")
|
| 715 |
+
st.plotly_chart(plot_violin(sub), width="stretch")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 716 |
|
| 717 |
|
| 718 |
+
def make_species_page(species: str) -> st.Page:
|
| 719 |
+
def render() -> None:
|
| 720 |
+
species_page(species)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 721 |
|
| 722 |
+
render.__name__ = f"page_{species.replace(' ', '_')}"
|
| 723 |
+
return st.Page(
|
| 724 |
+
render,
|
| 725 |
+
title=SPECIES_LABEL[species],
|
| 726 |
+
icon=SPECIES_ICON[species],
|
| 727 |
+
url_path=species.replace(" ", "-"),
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 728 |
)
|
| 729 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 730 |
|
| 731 |
+
# ---------------------------------------------------------------------
|
| 732 |
+
# Streamlit UI
|
| 733 |
+
# ---------------------------------------------------------------------
|
| 734 |
+
def main():
|
| 735 |
+
st.markdown(SIDEBAR_CSS, unsafe_allow_html=True)
|
| 736 |
+
available = set(_TRACKS["species"])
|
| 737 |
+
sections = {
|
| 738 |
+
"": [st.Page(overview_page, title="Overview", icon="🌍", default=True)],
|
| 739 |
+
}
|
| 740 |
+
for clade, group in SPECIES_GROUPS.items():
|
| 741 |
+
pages = [make_species_page(s) for s in group if s in available]
|
| 742 |
+
if pages:
|
| 743 |
+
sections[clade] = pages
|
| 744 |
+
sections["Initial release (2025)"] = [
|
| 745 |
+
st.Page(initial_release.main, title="2025 leaderboard", icon="🗄️", url_path="initial-release")
|
| 746 |
+
]
|
| 747 |
+
st.navigation(sections, position="sidebar").run()
|
| 748 |
|
| 749 |
|
| 750 |
if __name__ == "__main__":
|