Replace the 2025 benchmark with the revised NTv3 benchmark, keeping the 2025 leaderboard as a page

#2
by ybornachot - opened
.gitignore ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ __pycache__/
2
+ *.py[cod]
3
+ .DS_Store
README.md CHANGED
@@ -1,5 +1,5 @@
1
  ---
2
- title: Ntv3 Benchmark
3
  emoji: 🚀
4
  colorFrom: red
5
  colorTo: red
@@ -10,4 +10,74 @@ tags:
10
  pinned: false
11
  short_description: NTv3 Benchmark
12
  license: apache-2.0
13
- ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ title: NTv3 Benchmark
3
  emoji: 🚀
4
  colorFrom: red
5
  colorTo: red
 
10
  pinned: false
11
  short_description: NTv3 Benchmark
12
  license: apache-2.0
13
+ ---
14
+
15
+ # NTv3 Benchmark
16
+
17
+ A standardized comparison of sequence-to-function models on **155 functional-track
18
+ prediction tasks across 11 species**. Every model is given the same 32-kb input, the same
19
+ task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so
20
+ what is compared is the representation rather than the training recipe.
21
+
22
+ ## What is measured
23
+
24
+ **Metric.** Pearson correlation on the held-out test set, per track, averaged over three
25
+ seeds where available.
26
+
27
+ **Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato.
28
+ Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*.
29
+
30
+ **Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription
31
+ factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene
32
+ expression (RNA-seq) and translation (Ribo-seq).
33
+
34
+ The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize,
35
+ which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent
36
+ from that post-training set. The remaining seven genomes were absent from post-training
37
+ entirely, so those species measure transfer to an unseen genome.
38
+
39
+ ## Models
40
+
41
+ Ten models: two convolutional baselines trained from scratch, five pretrained genomic
42
+ foundation models, two pretrained sequence-to-function models, and NTv3.
43
+
44
+ | Model | Training |
45
+ | --- | --- |
46
+ | BPNet, Residual CNN | from scratch |
47
+ | HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained |
48
+ | Borzoi, AlphaGenome | pretrained sequence-to-function |
49
+ | **NTv3** | post-trained |
50
+
51
+ **AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds
52
+ checkpoint saw those test regions, and from its all-folds checkpoint on every other species.
53
+
54
+ ## Reading the leaderboard
55
+
56
+ **Scores are never averaged across species.** Not every model was run on every species, so a
57
+ single cross-species number would compare models over different task sets. Pick a species in
58
+ the sidebar for its own leaderboard, where every model shown was evaluated on all of that
59
+ species' tracks.
60
+
61
+ ## Contents
62
+
63
+ - `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying
64
+ the score and the training-cost columns the app reports. This is the data behind every
65
+ view; nothing is computed elsewhere.
66
+ - `src/streamlit_app.py`, the app.
67
+
68
+ ## Links
69
+
70
+ - 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised),
71
+ the genomes, splits and tracks to reproduce or extend the benchmark.
72
+ - 🧬 **NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3).
73
+ - 🗄️ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space.
74
+
75
+ ## Citation
76
+
77
+ `TBC`, to be added on publication of the NTv3 paper.
78
+
79
+ ## License
80
+
81
+ The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.**
82
+ They are released under the InstaDeep Open Model Licence, which permits non-commercial use
83
+ only and also places conditions on model outputs. See each model repository for its terms.
data/initial_release/ntv3_benchmark_results.csv ADDED
The diff for this file is too large to render. See raw diff
 
data/ntv3_benchmark_results.csv CHANGED
The diff for this file is too large to render. See raw diff
 
requirements.txt CHANGED
@@ -1,18 +1,4 @@
1
- APScheduler
2
- black
3
- datasets
4
- gradio
5
- gradio[oauth]
6
- gradio_leaderboard==0.0.9
7
- gradio_client
8
- huggingface-hub>=0.18.0
9
- matplotlib
10
  numpy
11
  pandas
12
- python-dateutil
13
- tqdm
14
- transformers
15
- tokenizers>=0.15.0
16
- sentencepiece
17
  plotly
18
- streamlit
 
 
 
 
 
 
 
 
 
 
1
  numpy
2
  pandas
 
 
 
 
 
3
  plotly
4
+ streamlit
src/initial_release.py ADDED
@@ -0,0 +1,1078 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from typing import List
2
+ import os
3
+
4
+ import pandas as pd
5
+ import streamlit as st
6
+ import plotly.express as px
7
+ from plotly.subplots import make_subplots
8
+ import plotly.graph_objects as go
9
+ import numpy as np
10
+
11
+ # The leaderboard of the initial NTv3 release (December 2025), as it was published.
12
+ # streamlit_app.py runs main() as one page of the revised benchmark, which owns the page config.
13
+
14
+ # ---------------------------------------------------------------------
15
+ # Configuration
16
+ # ---------------------------------------------------------------------
17
+ COLORS = {
18
+ # Primary colors 1 (our models)
19
+ 'blue_0': '#004697', # Darkest allowable blue
20
+ 'blue_1': '#3973fc', # Main blue
21
+ 'blue_2': '#7ea4fc', # Medium blue
22
+ 'blue_3': '#c3d5fc', # Light blue (lightest allowable blue)
23
+ # Secondary colors 1
24
+ 'red_1': '#ff554d', # Medium red
25
+ 'red_2': '#ffe0de', # Light red
26
+ # Primary colors 2
27
+ 'green_1': '#00b050', # Darkest green
28
+ 'green_2': '#92d050', # Medium green
29
+ 'green_3': '#c6e0b4', # Light green (lightest allowable green)
30
+ # Secondary colors 2
31
+ 'gold_1': '#fdb932',
32
+ # Tertiary colors
33
+ 'orange_1': '#ff975e',
34
+ 'purple_1': '#9a6ce4',
35
+ 'purple_2': '#bb9aef', # Medium purple
36
+ 'purple_3': '#ceb5f5', # Light purple (lightest allowable purple)
37
+ # Grays (other models)
38
+ 'gray_1': '#808080', # Darkest gray (use as a last resort)
39
+ 'gray_2': '#b3b3b3', # Medium gray (start with this as the darkest when possible)
40
+ 'gray_3': '#e6e6e6', # Lightest gray
41
+ 'gray_4': '#ffffff', # It's actually just white (use as a last resort)
42
+ # If all other options are exhausted
43
+ 'cyan_1': '#0096b4', # Darkest teal
44
+ 'cyan_2': '#28bed2', # Medium cyan
45
+ 'cyan_3': '#8cdceb', # Lightest cyan
46
+ 'magenta_1': '#b428a0', # Darkest magenta
47
+ 'magenta_2': '#dc50be', # Medium pink
48
+ 'magenta_3': '#f5a0dc', # Lightest pink
49
+ 'yellow_1': '#c8aa00', # Darkest yellow
50
+ 'yellow_2': '#ffd200', # Medium yellow
51
+ 'yellow_3': '#fff08c', # Lightest yellow
52
+ }
53
+
54
+
55
+ ASSAY_TYPE_MAPPING = {
56
+ 'ATAC-seq': 'chromatin accessibility',
57
+ 'DNase-seq': 'chromatin accessibility',
58
+ 'Histone ChIP-seq': 'histone modifications',
59
+ 'TF ChIP-seq': 'chromatin accessibility',
60
+ 'PRO-cap': 'transcription initiation',
61
+ 'eCLIP': 'RNA binding sites',
62
+ 'RNA-seq': 'gene expression',
63
+ 'ribo-seq': 'mRNA translation',
64
+ 'Annotation': 'genome annotation',
65
+ "Exon": "exon",
66
+ "Intron": "intron",
67
+ "Splice acceptor": "splice acceptor",
68
+ "Start codon": "start codon",
69
+
70
+ }
71
+
72
+ ASSAY_COLORS = {
73
+ 'chromatin accessibility': '#004697',
74
+ 'histone modifications': '#cc0000',
75
+ 'transcription initiation': '#ff9900',
76
+ 'RNA binding sites': '#9933cc',
77
+ 'gene expression': '#009900',
78
+ 'mRNA translation': '#ff6699',
79
+ 'genome annotation': '#ffcc00',
80
+ "intron": '#004697',
81
+ "exon": '#cc0000',
82
+ "splice acceptor": '#ff9900',
83
+ "start codon": '#9933cc',
84
+ }
85
+ ASSAY_COLORS["other"] = "#808080"
86
+
87
+ MODEL_COLORS = {
88
+ "NTv3 650M (pos)": COLORS['blue_0'],
89
+ 'NTv3 650M (pre)': COLORS['blue_1'], # #3973fc (Darkest blue)
90
+ 'NTv3 100M (pre)': COLORS['blue_2'], # #7ea4fc (Medium blue)
91
+ 'NTv3 8M (pre)': COLORS['blue_3'], # #c3d5fc (Light blue)
92
+ 'Evo2 1B': COLORS['green_3'], # #b3b3b3 (Medium gray)
93
+ "NTv2 500M": COLORS['gray_1'],
94
+ "BPNet arch. 6M": COLORS['cyan_1'],
95
+ "Residual CNN 44M": COLORS['magenta_1'],
96
+ "PlantCAD2 88M": COLORS["purple_1"],
97
+ "Caduceus 7M": COLORS["purple_2"],
98
+ "HyenaDNA 7M": COLORS["yellow_2"]
99
+ }
100
+
101
+ MODEL_TRAINING_STATUS = {
102
+ "NTv3 650M (pos)": "POS",
103
+ "NTv3 650M (pre)": "PRE",
104
+ "NTv3 100M (pre)": "PRE",
105
+ "NTv3 8M (pre)": "PRE",
106
+ "Residual CNN 44M": "SCRATCH",
107
+ "Caduceus 7M": "PRE",
108
+ "Evo2 1B": "PRE",
109
+ "NTv2 500M": "PRE",
110
+ "BPNet arch. 6M": "SCRATCH",
111
+ "PlantCAD2 88M": "PRE",
112
+ "HyenaDNA 7M": "PRE"
113
+ }
114
+
115
+ MODEL_GPU_MULTIPLIER = {
116
+ "Evo2 1B": 8, # trained on 8 GPUs
117
+ }
118
+
119
+
120
+ MODEL_NAMES = list(MODEL_COLORS.keys())
121
+
122
+ PLANT_SPECIES = ["tomato", "rice", "maize", "arabidopsis"]
123
+ ANIMAL_SPECIES = ["human", "chicken", "cattle"]
124
+
125
+ SPECIES_GROUPS = {
126
+ "Plants": PLANT_SPECIES,
127
+ "Animals": ANIMAL_SPECIES, # (your code calls these HUMAN_SPECIES, but they’re the “animal” set)
128
+ }
129
+
130
+
131
+ _LAST_UPDATED = "Dec 10, 2025"
132
+ _INTRO = """
133
+ The **NTv3 Benchmark** is a curated benchmark of 106 long-range genomic datasets
134
+ designed to evaluate models under realistic 32 kb input, single-base-pair output settings.
135
+ The dataset spans two complementary task families: genome annotation (exon, intron, splice acceptor, start codon)
136
+ and functional-regulatory prediction, which includes diverse experimental tracks such as chromatin accessibility,
137
+ histone modifications, transcription initiation (PRO-cap), RNA binding (eCLIP), gene expression (RNA-seq),
138
+ and translation (Ribo-seq).
139
+
140
+ Data are drawn from a phylogenetically diverse set of species, including organisms seen during post-training
141
+ (human, chicken, arabidopsis, rice, maize) and entirely unseen species (cattle, tomato), with careful curation
142
+ to avoid data leakage. This design allows the dataset to probe long-range sequence-to-function mapping,
143
+ cross-species generalization, and transfer across heterogeneous regulatory modalities,
144
+ including assays not present in prior multispecies training corpora. By standardizing sequence length,
145
+ resolution, and evaluation metrics across all tracks, the NTv3 Benchmark provides a controlled dataset
146
+ for comparing representation quality across genomic foundation models.
147
+
148
+ The metrics used are:
149
+ - **Pearson correlations (multi-assay)**: per-dataset scores across species and models for functional tracks.
150
+ - **MCC (bed tracks)**: per-track MCC values across species and models for gene annotation tracks.
151
+ """
152
+
153
+ HERE = os.path.dirname(os.path.abspath(__file__)) # /app/src
154
+ PROJECT_ROOT = os.path.dirname(HERE) # /app
155
+ DATA_DIR = os.path.join(PROJECT_ROOT, "data")
156
+
157
+ SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "initial_release", "ntv3_benchmark_results.csv")
158
+
159
+ # ---------------------------------------------------------------------
160
+ # Data loading & preprocessing
161
+ # ---------------------------------------------------------------------
162
+
163
+
164
+ @st.cache_data
165
+ def load_raw_data():
166
+ df = pd.read_csv(SINGLE_TABLE_PATH)
167
+ df.columns = [c.strip() for c in df.columns]
168
+ return df
169
+
170
+
171
+ def _normalize_training_time_to_gpu_hours(df: pd.DataFrame) -> pd.DataFrame:
172
+ """
173
+ Your new column is `running_time`. In your sample it looks like seconds
174
+ (e.g. 317034 ~= 88 hours). We'll convert to hours if values look like seconds.
175
+ """
176
+ if "running_time" not in df.columns:
177
+ return df
178
+
179
+ rt = pd.to_numeric(df["running_time"], errors="coerce")
180
+ # Heuristic: if median is huge, it's probably seconds -> convert to hours
181
+ # (88 hours = 316800 seconds is a typical-looking value in your sample)
182
+ if rt.dropna().median() > 10_000:
183
+ df["GPU hours"] = rt / 3600.0
184
+ else:
185
+ df["GPU hours"] = rt.astype(float)
186
+
187
+ return df
188
+
189
+
190
+ def _best_step_time_to_hours(s: pd.Series) -> pd.Series:
191
+ """
192
+ Converts strings like '3 days 04:26:26.467000' to hours (float).
193
+ Works with pandas Timedelta parsing.
194
+ """
195
+ td = pd.to_timedelta(s, errors="coerce")
196
+ return td.dt.total_seconds() / 3600.0
197
+
198
+
199
+
200
+ @st.cache_data
201
+ def load_expanded_data():
202
+ df = load_raw_data().copy()
203
+
204
+ df = df.rename(columns={"Metric": "Score", "model_name": "Model"})
205
+ df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
206
+
207
+ if "best_step" in df.columns:
208
+ df["best_step"] = pd.to_numeric(df["best_step"], errors="coerce")
209
+
210
+ if "best_step_time" in df.columns:
211
+ df["best_step_time_hours"] = _best_step_time_to_hours(df["best_step_time"])
212
+ else:
213
+ df["best_step_time_hours"] = np.nan
214
+
215
+ is_annot = df.get("assay_type", "").astype(str).eq("Annotation")
216
+ pearson_raw = df[~is_annot].copy()
217
+ mcc_raw = df[is_annot].copy()
218
+
219
+ # -------------------------
220
+ # Functional Tracks (Pearson)
221
+ # -------------------------
222
+ pearson_group_cols = ["species", "datasets", "Model"]
223
+ if "assay_type" in pearson_raw.columns:
224
+ pearson_group_cols.append("assay_type")
225
+
226
+ pearson_df = (
227
+ pearson_raw
228
+ .groupby(pearson_group_cols, as_index=False, dropna=False)
229
+ .agg({
230
+ "Score": "mean",
231
+ "best_step": "mean",
232
+ "best_step_time_hours": "mean",
233
+ })
234
+ )
235
+
236
+ # ✅ merge track_name_clean WHILE assay_type is still raw
237
+ if "track_name_clean" in pearson_raw.columns:
238
+ map_keys = ["species", "datasets"]
239
+ if "assay_type" in pearson_raw.columns:
240
+ map_keys.append("assay_type")
241
+
242
+ track_map = (
243
+ pearson_raw[map_keys + ["track_name_clean"]]
244
+ .dropna(subset=["track_name_clean"])
245
+ .drop_duplicates()
246
+ )
247
+ pearson_df = pearson_df.merge(track_map, on=map_keys, how="left")
248
+
249
+ # ✅ now it’s safe to map assay_type to categories
250
+ if "assay_type" in pearson_df.columns:
251
+ pearson_df["assay_type"] = (
252
+ pearson_df["assay_type"].astype(str).map(ASSAY_TYPE_MAPPING).fillna("Other")
253
+ )
254
+
255
+ # -------------------------
256
+ # Genome Annotation (MCC)
257
+ # -------------------------
258
+ mcc_df = (
259
+ mcc_raw
260
+ .groupby(["species", "datasets", "Model"], as_index=False, dropna=False)
261
+ .agg({
262
+ "Score": "mean",
263
+ "best_step": "mean",
264
+ "best_step_time_hours": "mean",
265
+ })
266
+ )
267
+
268
+ return pearson_df, mcc_df
269
+
270
+
271
+ _PEARSON_DF, _MCC_DF = load_expanded_data()
272
+
273
+ # Global sets (we'll further filter per-benchmark below)
274
+ _ALL_SPECIES = sorted(
275
+ set(_PEARSON_DF["species"].unique()).union(_MCC_DF["species"].unique())
276
+ )
277
+
278
+ _ALL_ASSAYS = (
279
+ sorted(_PEARSON_DF["assay_type"].dropna().unique())
280
+ if "assay_type" in _PEARSON_DF.columns
281
+ else []
282
+ )
283
+
284
+ _ALL_MODELS = MODEL_NAMES[:]
285
+
286
+ _BENCHMARKS = {
287
+ "Functional Tracks": {
288
+ "df": _PEARSON_DF,
289
+ "metric_label": "Pearson correlation",
290
+ "has_assay_type": True,
291
+ },
292
+ "Genome Annotation": {
293
+ "df": _MCC_DF,
294
+ "metric_label": "MCC",
295
+ "has_assay_type": False,
296
+ },
297
+ }
298
+
299
+
300
+ # ---------------------------------------------------------------------
301
+ # Computation helpers
302
+ # ---------------------------------------------------------------------
303
+
304
+
305
+ def filter_base_df(
306
+ benchmark_name: str,
307
+ selected_species: List[str],
308
+ selected_assays: List[str],
309
+ selected_models: List[str],
310
+ selected_datasets: List[str],
311
+ ) -> pd.DataFrame:
312
+ cfg = _BENCHMARKS[benchmark_name]
313
+ df = cfg["df"].copy()
314
+
315
+ # Species filter
316
+ if selected_species:
317
+ df = df[df["species"].isin(selected_species)]
318
+
319
+ # Assay type filter (Pearson only)
320
+ if cfg.get("has_assay_type", False) and selected_assays and "assay_type" in df.columns:
321
+ df = df[df["assay_type"].isin(selected_assays)]
322
+
323
+ # Dataset / bed track filter (for MCC, but safe to apply generally)
324
+ if selected_datasets and "datasets" in df.columns:
325
+ df = df[df["datasets"].isin(selected_datasets)]
326
+
327
+ # Model filter
328
+ if selected_models:
329
+ df = df[df["Model"].isin(selected_models)]
330
+
331
+ return df
332
+
333
+
334
+ def build_leaderboard(
335
+ benchmark_name: str,
336
+ selected_species: List[str],
337
+ selected_assays: List[str],
338
+ selected_models: List[str],
339
+ selected_datasets: List[str],
340
+ ) -> pd.DataFrame:
341
+ df = filter_base_df(
342
+ benchmark_name,
343
+ selected_species,
344
+ selected_assays,
345
+ selected_models,
346
+ selected_datasets,
347
+ )
348
+
349
+ if df.empty:
350
+ return pd.DataFrame(columns=["Model", "Model Type", "Num entries", "Mean score"])
351
+
352
+ agg = (
353
+ df.groupby("Model")["Score"]
354
+ .mean()
355
+ .reset_index()
356
+ .rename(columns={"Score": "Mean score"})
357
+ )
358
+ agg["Mean score"] = agg["Mean score"].round(3)
359
+
360
+ agg["Num entries"] = (
361
+ df.groupby("Model")["Score"].count().reindex(agg["Model"]).values
362
+ )
363
+
364
+ # 👇 Add training regime column
365
+ agg["Training"] = agg["Model"].map(MODEL_TRAINING_STATUS).fillna("UNKNOWN")
366
+
367
+ # Sort by performance
368
+ agg = agg.sort_values("Mean score", ascending=False).reset_index(drop=True)
369
+
370
+ # Column order
371
+ agg = agg[["Model", "Training", "Num entries", "Mean score"]]
372
+
373
+ # Ensure the index starts with 1
374
+ agg.index += 1
375
+
376
+ return agg
377
+
378
+
379
+
380
+ def build_bar_df(
381
+ benchmark_name: str,
382
+ selected_species: List[str],
383
+ selected_assays: List[str],
384
+ selected_models: List[str],
385
+ selected_datasets: List[str],
386
+ ) -> pd.DataFrame:
387
+ """For now, just one bar per model (same as leaderboard)."""
388
+ return build_leaderboard(
389
+ benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
390
+ )
391
+
392
+
393
+ def build_category_model_df(
394
+ benchmark_name: str,
395
+ selected_species: List[str],
396
+ selected_assays: List[str],
397
+ selected_models: List[str],
398
+ selected_datasets: List[str],
399
+ ) -> pd.DataFrame:
400
+ """
401
+ Mean score per (category, Model) after applying the same filters.
402
+ Category = assay_type (Functional Tracks) or datasets (Genome Annotation).
403
+ """
404
+ cfg = _BENCHMARKS[benchmark_name]
405
+ df = filter_base_df(
406
+ benchmark_name,
407
+ selected_species,
408
+ selected_assays,
409
+ selected_models,
410
+ selected_datasets,
411
+ )
412
+ if df.empty:
413
+ return pd.DataFrame(columns=["Category", "Model", "Mean score"])
414
+
415
+ # Pick the right breakdown column
416
+ if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
417
+ category_col = "assay_type"
418
+ category_label = "Assay type"
419
+ else:
420
+ category_col = "datasets"
421
+ category_label = "Dataset"
422
+
423
+ if category_col not in df.columns:
424
+ return pd.DataFrame(columns=["Category", "Model", "Mean score"])
425
+
426
+ out = (
427
+ df.groupby([category_col, "Model"], as_index=False)["Score"]
428
+ .mean()
429
+ .rename(columns={category_col: "Category", "Score": "Mean score"})
430
+ )
431
+ out["Mean score"] = out["Mean score"].round(3)
432
+ out.attrs["category_label"] = category_label # for nicer axis title
433
+ return out
434
+
435
+
436
+ def plot_breakdown_facets_sorted_models(
437
+ breakdown_df: pd.DataFrame,
438
+ metric_label: str,
439
+ height: int = 420,
440
+ ):
441
+ categories = list(breakdown_df["Category"].dropna().unique())
442
+ categories = sorted(categories)
443
+
444
+ n = len(categories)
445
+ if n == 0:
446
+ return None
447
+
448
+ rows = 1
449
+ cols = n # 👈 everything in one row
450
+
451
+ # Global y-range (consistent scale)
452
+ y_min = breakdown_df["Mean score"].min()
453
+ y_max = breakdown_df["Mean score"].max()
454
+ pad = 0.05 * (y_max - y_min if y_max > y_min else 1.0)
455
+ y_range = [y_min - pad, y_max + pad]
456
+
457
+ fig = make_subplots(
458
+ rows=rows,
459
+ cols=cols,
460
+ subplot_titles=categories,
461
+ shared_yaxes=True,
462
+ horizontal_spacing=0.04, # tighter spacing
463
+ )
464
+
465
+ for i, cat in enumerate(categories):
466
+ r = (i // cols) + 1
467
+ c = (i % cols) + 1
468
+
469
+ sub = (
470
+ breakdown_df[breakdown_df["Category"] == cat]
471
+ .sort_values("Mean score", ascending=True)
472
+ )
473
+
474
+ fig.add_trace(
475
+ go.Bar(
476
+ x=sub["Model"],
477
+ y=sub["Mean score"],
478
+ marker_color=[MODEL_COLORS.get(m, "#808080") for m in sub["Model"]],
479
+ showlegend=False,
480
+ ),
481
+ row=r,
482
+ col=c,
483
+ )
484
+
485
+ fig.update_xaxes(showticklabels=False, title_text="", row=r, col=c)
486
+ fig.update_yaxes(range=y_range, title_text="", row=r, col=c) # 👈 apply range
487
+
488
+ fig.update_layout(
489
+ height=height,
490
+ plot_bgcolor="rgba(0,0,0,0)",
491
+ paper_bgcolor="rgba(0,0,0,0)",
492
+ margin=dict(t=60, l=10, r=10, b=10),
493
+ )
494
+
495
+ # Single y-axis label on the leftmost panel
496
+ fig.update_yaxes(title_text=metric_label, row=1, col=1)
497
+
498
+ return fig
499
+
500
+ def build_pairwise_scatter_df(
501
+ benchmark_name: str,
502
+ selected_species: List[str],
503
+ selected_assays: List[str],
504
+ selected_models: List[str],
505
+ selected_datasets: List[str],
506
+ model_a: str,
507
+ model_b: str,
508
+ ) -> pd.DataFrame:
509
+ cfg = _BENCHMARKS[benchmark_name]
510
+
511
+ models_for_filter = (
512
+ list(set(selected_models + [model_a, model_b]))
513
+ if selected_models else [model_a, model_b]
514
+ )
515
+
516
+ df = filter_base_df(
517
+ benchmark_name,
518
+ selected_species,
519
+ selected_assays,
520
+ models_for_filter,
521
+ selected_datasets,
522
+ )
523
+ if df.empty:
524
+ return pd.DataFrame()
525
+
526
+ # ---- define "track identity" for head-to-head ----
527
+ # Always use datasets for the identity (x/y points)
528
+ track_cols = ["datasets"]
529
+ if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
530
+ track_cols = ["assay_type", "datasets"]
531
+
532
+ keep_species = "species" in df.columns and (selected_species is None or len(selected_species) != 1)
533
+ id_cols = (["species"] if keep_species else []) + track_cols
534
+
535
+ wide = (
536
+ df[df["Model"].isin([model_a, model_b])]
537
+ .pivot_table(index=id_cols, columns="Model", values="Score", aggfunc="mean")
538
+ .reset_index()
539
+ )
540
+
541
+ if model_a not in wide.columns or model_b not in wide.columns:
542
+ return pd.DataFrame()
543
+
544
+ wide = wide.dropna(subset=[model_a, model_b])
545
+
546
+ # Nice display label: use datasets (not track_name_clean)
547
+ if "assay_type" in wide.columns:
548
+ wide["Track"] = wide["assay_type"].astype(str) + " / " + wide["datasets"].astype(str)
549
+ else:
550
+ wide["Track"] = wide["datasets"].astype(str)
551
+
552
+ wide = wide.rename(columns={model_a: "Model A", model_b: "Model B"})
553
+
554
+ # ---- Pearson-only: merge track_name_clean for hover ----
555
+ if benchmark_name == "Functional Tracks" and "track_name_clean" in df.columns:
556
+ merge_keys = id_cols.copy() # species? + assay_type? + datasets
557
+ track_map = (
558
+ df[merge_keys + ["track_name_clean"]]
559
+ .dropna(subset=["track_name_clean"])
560
+ .drop_duplicates()
561
+ )
562
+ wide = wide.merge(track_map, on=merge_keys, how="left")
563
+
564
+ return wide
565
+
566
+
567
+
568
+ def build_violin_df(
569
+ benchmark_name: str,
570
+ selected_species: List[str],
571
+ selected_assays: List[str],
572
+ selected_models: List[str],
573
+ selected_datasets: List[str],
574
+ ) -> pd.DataFrame:
575
+ # Use the same base filtering, but keep all per-track rows
576
+ df = filter_base_df(
577
+ benchmark_name,
578
+ selected_species,
579
+ selected_assays,
580
+ selected_models,
581
+ selected_datasets,
582
+ )
583
+ # Keep only needed columns
584
+ keep = ["Model", "Score"]
585
+ for col in ["species", "assay_type", "datasets"]:
586
+ if col in df.columns:
587
+ keep.append(col)
588
+ return df[keep].copy()
589
+
590
+
591
+ def build_convergence_df(
592
+ benchmark_name: str,
593
+ selected_species: List[str],
594
+ selected_assays: List[str],
595
+ selected_models: List[str],
596
+ selected_datasets: List[str],
597
+ x_mode: str = "best_step", # "best_step" | "best_step_time"
598
+ ) -> pd.DataFrame:
599
+
600
+ df = filter_base_df(
601
+ benchmark_name,
602
+ selected_species,
603
+ selected_assays,
604
+ selected_models,
605
+ selected_datasets,
606
+ )
607
+
608
+ if df.empty:
609
+ return pd.DataFrame(columns=["Model", "X", "Performance"])
610
+
611
+ # Mean performance per model
612
+ out = (
613
+ df.groupby("Model", as_index=False)
614
+ .agg({"Score": "mean"})
615
+ .rename(columns={"Score": "Performance"})
616
+ )
617
+
618
+ # -------------------------
619
+ # X axis selection
620
+ # -------------------------
621
+ if x_mode == "Steps (billions)":
622
+ if "best_step" not in df.columns:
623
+ return pd.DataFrame(columns=["Model", "X", "Performance"])
624
+
625
+ x = (
626
+ df.groupby("Model", as_index=False)["best_step"]
627
+ .mean()
628
+ .rename(columns={"best_step": "X"})
629
+ )
630
+
631
+ else: # best_step_time (GPU hours)
632
+ if "best_step_time_hours" not in df.columns:
633
+ return pd.DataFrame(columns=["Model", "X", "Performance"])
634
+
635
+ x = (
636
+ df.groupby("Model", as_index=False)["best_step_time_hours"]
637
+ .mean()
638
+ .rename(columns={"best_step_time_hours": "X"})
639
+ )
640
+
641
+ # 👇 Apply GPU multiplier (Evo2 uses 8 GPUs)
642
+ gpu_multiplier = {
643
+ "Evo2 1B": 8,
644
+ }
645
+
646
+ x["X"] = x.apply(
647
+ lambda r: r["X"] * gpu_multiplier.get(r["Model"], 1),
648
+ axis=1,
649
+ )
650
+
651
+ # Merge + clean
652
+ out = out.merge(x, on="Model", how="left")
653
+ out = out.dropna(subset=["X", "Performance"])
654
+
655
+ out["Performance"] = out["Performance"].round(3)
656
+
657
+ return out
658
+
659
+
660
+ # ---------------------------------------------------------------------
661
+ # UI helpers
662
+ # ---------------------------------------------------------------------
663
+
664
+
665
+ def sidebar_toggle(label: str, value: bool = False, key: str | None = None) -> bool:
666
+ """
667
+ Wrapper to use st.sidebar.toggle when available, otherwise fall back to checkbox.
668
+ This makes the app compatible with older Streamlit versions on Hugging Face.
669
+ """
670
+ toggle_fn = getattr(st.sidebar, "toggle", None)
671
+ if toggle_fn is not None:
672
+ return toggle_fn(label, value=value, key=key)
673
+ # Fallback for older Streamlit versions
674
+ return st.sidebar.checkbox(label, value=value, key=key)
675
+
676
+ # ---------------------------------------------------------------------
677
+ # Streamlit UI
678
+ # ---------------------------------------------------------------------
679
+ def main():
680
+ st.title("🗄️ NTv3 Benchmark: initial release (2025)")
681
+ st.info(
682
+ "The leaderboard of the initial NTv3 release, as published in December 2025. "
683
+ "The other pages show the revised release."
684
+ )
685
+ st.markdown(_INTRO)
686
+ st.markdown(f"_Last updated: **{_LAST_UPDATED}**_")
687
+
688
+ # --- Sidebar filters ---
689
+ st.sidebar.header("Filters")
690
+
691
+ # Benchmark
692
+ benchmark_name = st.sidebar.selectbox(
693
+ "Benchmark",
694
+ options=list(_BENCHMARKS.keys()),
695
+ index=0,
696
+ )
697
+
698
+ cfg = _BENCHMARKS[benchmark_name]
699
+ df_bench = cfg["df"]
700
+
701
+ # Species toggles, but only for species present in this benchmark
702
+ st.sidebar.subheader("Species")
703
+
704
+ # Toggle: Plants vs Animals
705
+ species_group = st.sidebar.radio(
706
+ "Group",
707
+ options=["Animals", "Plants"],
708
+ index=0,
709
+ horizontal=True,
710
+ key=f"species_group_{benchmark_name}",
711
+ )
712
+
713
+ available_species_all = sorted(df_bench["species"].unique())
714
+ allowed_species = set(SPECIES_GROUPS[species_group]).intersection(available_species_all)
715
+ available_species = sorted(allowed_species)
716
+
717
+ selected_species: List[str] = []
718
+ for sp in available_species:
719
+ if sidebar_toggle(sp, value=True, key=f"species_{benchmark_name}_{species_group}_{sp}"):
720
+ selected_species.append(sp)
721
+
722
+ # (Optional) If no species exist for that group in this benchmark
723
+ if not available_species:
724
+ st.sidebar.info(f"No {species_group.lower()} species available for this benchmark.")
725
+
726
+ # Assay toggles (Pearson only), based on filtered species
727
+ if cfg.get("has_assay_type", False):
728
+ st.sidebar.subheader("Assay types")
729
+ if selected_species:
730
+ df_for_assays = df_bench[df_bench["species"].isin(selected_species)]
731
+ else:
732
+ df_for_assays = df_bench
733
+ available_assays = (
734
+ sorted(df_for_assays["assay_type"].dropna().unique())
735
+ if "assay_type" in df_for_assays.columns
736
+ else []
737
+ )
738
+ selected_assays: List[str] = []
739
+ for assay in available_assays:
740
+ if sidebar_toggle(assay, value=True, key=f"assay_{benchmark_name}_{assay}"):
741
+ selected_assays.append(assay)
742
+ else:
743
+ selected_assays = []
744
+
745
+ # Bed track / dataset toggles (MCC only), based on species selection
746
+ selected_datasets: List[str] = []
747
+ if benchmark_name == "Genome Annotation":
748
+ st.sidebar.subheader("Genome annotations")
749
+ if selected_species:
750
+ df_for_tracks = df_bench[df_bench["species"].isin(selected_species)]
751
+ else:
752
+ df_for_tracks = df_bench
753
+ available_datasets = sorted(df_for_tracks["datasets"].unique())
754
+ for ds in available_datasets:
755
+ if sidebar_toggle(ds, value=True, key=f"dataset_{benchmark_name}_{ds}"):
756
+ selected_datasets.append(ds)
757
+ else:
758
+ selected_datasets = []
759
+
760
+ # Model toggles (we keep all models in MODEL_NAMES; filters + data will prune)
761
+ st.sidebar.subheader("Models")
762
+ selected_models: List[str] = []
763
+ for model in _ALL_MODELS:
764
+ if sidebar_toggle(model, value=True, key=f"model_{model}"):
765
+ selected_models.append(model)
766
+
767
+ # -------------------------
768
+ # ✅ Validation: require ≥1 selection per relevant category
769
+ # -------------------------
770
+ missing = []
771
+
772
+ # Always required
773
+ if not selected_species:
774
+ missing.append("Species")
775
+ if not selected_models:
776
+ missing.append("Models")
777
+
778
+ # Required depending on benchmark
779
+ if cfg.get("has_assay_type", False) and not selected_assays:
780
+ missing.append("Assay types")
781
+
782
+ if benchmark_name == "Genome Annotation" and not selected_datasets:
783
+ missing.append("Genome annotations")
784
+
785
+ if missing:
786
+ # Show a single message and prevent *any* further display
787
+ st.error(
788
+ "Please select at least one item in each category. Currently missing: "
789
+ + ", ".join(missing)
790
+ + "."
791
+ )
792
+ st.stop()
793
+
794
+ # --- Main content ---
795
+ leaderboard_df = build_leaderboard(
796
+ benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
797
+ )
798
+ bar_df = build_bar_df(
799
+ benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
800
+ )
801
+
802
+ col1, col2 = st.columns([1, 1])
803
+
804
+ with col1:
805
+ st.subheader("🏅 Leaderboard")
806
+ st.write("\n") # spacer to match plotly padding
807
+ st.write("\n")
808
+ st.write("\n")
809
+ if leaderboard_df.empty:
810
+ st.info("No data for the selected filters.")
811
+ else:
812
+ st.dataframe(leaderboard_df, use_container_width=True)
813
+
814
+ with col2:
815
+ st.subheader("📈 Mean score per model")
816
+ if bar_df.empty:
817
+ st.info("No data for the selected filters.")
818
+ else:
819
+ # Order models by performance (least -> most)
820
+ bar_df = bar_df.sort_values("Mean score", ascending=True)
821
+ model_order = bar_df["Model"].tolist()
822
+
823
+ fig = px.bar(
824
+ bar_df,
825
+ x="Model",
826
+ y="Mean score",
827
+ color="Model",
828
+ color_discrete_map=MODEL_COLORS,
829
+ category_orders={"Model": model_order},
830
+ )
831
+ fig.update_layout(
832
+ barmode="group",
833
+ height=500,
834
+ xaxis_title="",
835
+ yaxis_title=cfg["metric_label"],
836
+ plot_bgcolor="rgba(0,0,0,0)",
837
+ paper_bgcolor="rgba(0,0,0,0)",
838
+ bargap=0.08,
839
+ )
840
+ fig.update_xaxes(showticklabels=False)
841
+ st.plotly_chart(fig, use_container_width=True)
842
+
843
+ # --- Breakdown plot: assay_type (Functional Tracks) OR datasets (Genome Annotation) ---
844
+ breakdown_df = build_category_model_df(
845
+ benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
846
+ )
847
+
848
+ type_of_data = "assay type" if benchmark_name == "Functional Tracks" else "gene annotation"
849
+
850
+ st.subheader(f"🧪 Mean score by {type_of_data}")
851
+ if breakdown_df.empty:
852
+ st.info("No data for the selected filters.")
853
+ else:
854
+ fig_breakdown = plot_breakdown_facets_sorted_models(
855
+ breakdown_df,
856
+ metric_label=cfg["metric_label"],
857
+ height=300,
858
+ )
859
+ st.plotly_chart(fig_breakdown, use_container_width=True)
860
+
861
+ # ------------------------------------------------------------------
862
+ # Model comparison: Head-to-head (left) + Convergence (right)
863
+ # ------------------------------------------------------------------
864
+ left, right = st.columns([1, 1], gap="large")
865
+
866
+ with left:
867
+ st.markdown("#### ⚖️ Head-to-head (per track)")
868
+
869
+ model_picker_options = selected_models if selected_models else _ALL_MODELS
870
+ default_a = model_picker_options[0] if model_picker_options else _ALL_MODELS[0]
871
+ default_b = model_picker_options[1] if len(model_picker_options) > 1 else (
872
+ _ALL_MODELS[1] if len(_ALL_MODELS) > 1 else default_a
873
+ )
874
+
875
+ cA, cB = st.columns([1, 1])
876
+ with cA:
877
+ model_a = st.selectbox(
878
+ "Model A (y-axis)",
879
+ options=model_picker_options,
880
+ index=model_picker_options.index(default_a) if default_a in model_picker_options else 0,
881
+ key=f"pair_model_a_{benchmark_name}",
882
+ )
883
+ with cB:
884
+ b_options = [m for m in model_picker_options if m != model_a] or model_picker_options
885
+ model_b = st.selectbox(
886
+ "Model B (x-axis)",
887
+ options=b_options,
888
+ index=0,
889
+ key=f"pair_model_b_{benchmark_name}",
890
+ )
891
+
892
+ scatter_df = build_pairwise_scatter_df(
893
+ benchmark_name,
894
+ selected_species,
895
+ selected_assays,
896
+ selected_models,
897
+ selected_datasets,
898
+ model_a,
899
+ model_b,
900
+ )
901
+
902
+ if scatter_df.empty:
903
+ st.info("No overlapping tracks for the selected filters (or one model is missing values).")
904
+ else:
905
+ min_v = float(min(scatter_df["Model A"].min(), scatter_df["Model B"].min()))
906
+ max_v = float(max(scatter_df["Model A"].max(), scatter_df["Model B"].max()))
907
+ pad = 0.05 * (max_v - min_v if max_v > min_v else 1.0)
908
+ axis_range = [min_v - pad, max_v + pad]
909
+ tick_step = (axis_range[1] - axis_range[0]) / 5
910
+
911
+ hover_cols = ["datasets"]
912
+ if benchmark_name == "Functional Tracks" and "track_name_clean" in scatter_df.columns:
913
+ hover_cols.append("track_name_clean")
914
+ else:
915
+ hover_cols.append("datasets")
916
+
917
+ color_col = "assay_type" if "assay_type" in scatter_df.columns else "datasets"
918
+
919
+ fig_scatter = px.scatter(
920
+ scatter_df,
921
+ x="Model B",
922
+ y="Model A",
923
+ color=color_col,
924
+ color_discrete_map=ASSAY_COLORS,
925
+ hover_name="Track",
926
+ hover_data=hover_cols,
927
+ )
928
+
929
+ fig_scatter.add_shape(
930
+ type="line",
931
+ x0=axis_range[0], y0=axis_range[0],
932
+ x1=axis_range[1], y1=axis_range[1],
933
+ xref="x", yref="y",
934
+ line=dict(color="red", dash="dot", width=2),
935
+ )
936
+
937
+ fig_scatter.update_layout(
938
+ height=550,
939
+ margin=dict(l=60, r=20, t=20, b=60),
940
+ xaxis=dict(
941
+ title=f"{model_b} — {cfg['metric_label']}",
942
+ range=axis_range,
943
+ dtick=tick_step,
944
+ constrain="domain",
945
+ ),
946
+ yaxis=dict(
947
+ title=f"{model_a} — {cfg['metric_label']}",
948
+ range=axis_range,
949
+ dtick=tick_step,
950
+ scaleanchor="x",
951
+ scaleratio=1,
952
+ constrain="domain",
953
+ ),
954
+ plot_bgcolor="rgba(0,0,0,0)",
955
+ paper_bgcolor="rgba(0,0,0,0)",
956
+ )
957
+
958
+ fig_scatter.update_layout(
959
+ legend=dict(
960
+ title="Assay type" if benchmark_name == "Functional Tracks" else "Genome Annotation",
961
+ x=0.98,
962
+ y=0.1,
963
+ xanchor="right",
964
+ yanchor="bottom",
965
+ bgcolor="rgba(255,255,255,0.2)", # semi-transparent white
966
+ bordercolor="rgba(0,0,0,0.2)",
967
+ borderwidth=1,
968
+ )
969
+ )
970
+
971
+ st.plotly_chart(fig_scatter, use_container_width=True)
972
+
973
+ with right:
974
+ st.markdown("#### ⏱️ Time to convergence")
975
+
976
+ x_mode = st.selectbox(
977
+ "X-axis",
978
+ options=["GPU (hours)", "Steps (billions)"],
979
+ index=0,
980
+ key=f"conv_x_mode_{benchmark_name}",
981
+ )
982
+
983
+
984
+ conv_df = build_convergence_df(
985
+ benchmark_name,
986
+ selected_species,
987
+ selected_assays,
988
+ selected_models,
989
+ selected_datasets,
990
+ x_mode=x_mode,
991
+ )
992
+
993
+ if conv_df.empty:
994
+ st.info("No convergence data found for the selected filters / x-axis mode.")
995
+ else:
996
+ fig_conv = px.scatter(
997
+ conv_df,
998
+ x="X",
999
+ y="Performance",
1000
+ text="Model",
1001
+ color="Model",
1002
+ color_discrete_map=MODEL_COLORS,
1003
+ hover_data=["Model", "X", "Performance"],
1004
+ )
1005
+ fig_conv.update_layout(
1006
+ height=550,
1007
+ xaxis_title=("GPU (hours)" if x_mode == "GPU (hours)" else x_mode),
1008
+ yaxis_title=cfg["metric_label"],
1009
+ plot_bgcolor="rgba(0,0,0,0)",
1010
+ paper_bgcolor="rgba(0,0,0,0)",
1011
+ showlegend=False, # ✅ no legend
1012
+ )
1013
+ fig_conv.update_traces(
1014
+ marker=dict(size=14), # 👈 bigger dots
1015
+ textposition="top center",
1016
+ )
1017
+
1018
+ # Log scale only makes sense for hours (and sometimes best_step)
1019
+ if x_mode in ["GPU (hours)"]:
1020
+ fig_conv.update_xaxes(
1021
+ type="log",
1022
+ dtick=1,
1023
+ minor=dict(ticks="", showgrid=False),
1024
+ )
1025
+ st.plotly_chart(fig_conv, use_container_width=True)
1026
+
1027
+ # ------------------------------------------------------------------
1028
+ # Violin (full width, below)
1029
+ # ------------------------------------------------------------------
1030
+ st.subheader("🎻 Performance comparison across tracks")
1031
+
1032
+ violin_df = build_violin_df(
1033
+ benchmark_name,
1034
+ selected_species,
1035
+ selected_assays,
1036
+ selected_models,
1037
+ selected_datasets,
1038
+ )
1039
+
1040
+ if violin_df.empty:
1041
+ st.info("No data for the selected filters.")
1042
+ else:
1043
+ model_order = (
1044
+ violin_df
1045
+ .groupby("Model")["Score"]
1046
+ .median()
1047
+ .sort_values(ascending=True)
1048
+ .index
1049
+ .tolist()
1050
+ )
1051
+
1052
+ fig_violin = px.violin(
1053
+ violin_df,
1054
+ x="Model",
1055
+ y="Score",
1056
+ color="Model",
1057
+ color_discrete_map=MODEL_COLORS,
1058
+ box=True,
1059
+ points=False,
1060
+ category_orders={"Model": model_order},
1061
+ )
1062
+
1063
+ fig_violin.update_layout(
1064
+ height=650,
1065
+ xaxis_title="",
1066
+ yaxis_title=cfg["metric_label"],
1067
+ plot_bgcolor="rgba(0,0,0,0)",
1068
+ paper_bgcolor="rgba(0,0,0,0)",
1069
+ showlegend=False,
1070
+ )
1071
+
1072
+ fig_violin.update_traces(
1073
+ box_visible=True,
1074
+ meanline_visible=False,
1075
+ )
1076
+
1077
+ st.plotly_chart(fig_violin, use_container_width=True)
1078
+
src/streamlit_app.py CHANGED
@@ -1,12 +1,11 @@
1
- from typing import List
2
  import os
3
 
 
4
  import pandas as pd
5
- import streamlit as st
6
  import plotly.express as px
7
- from plotly.subplots import make_subplots
8
  import plotly.graph_objects as go
9
- import numpy as np
 
10
 
11
  # ---------------------------------------------------------------------
12
  # Page config (must be the first Streamlit command)
@@ -16,150 +15,163 @@ st.set_page_config(
16
  layout="wide",
17
  )
18
 
 
 
19
  # ---------------------------------------------------------------------
20
  # Configuration
21
  # ---------------------------------------------------------------------
22
  COLORS = {
23
- # Primary colors 1 (our models)
24
- 'blue_0': '#004697', # Darkest allowable blue
25
- 'blue_1': '#3973fc', # Main blue
26
- 'blue_2': '#7ea4fc', # Medium blue
27
- 'blue_3': '#c3d5fc', # Light blue (lightest allowable blue)
28
- # Secondary colors 1
29
- 'red_1': '#ff554d', # Medium red
30
- 'red_2': '#ffe0de', # Light red
31
- # Primary colors 2
32
- 'green_1': '#00b050', # Darkest green
33
- 'green_2': '#92d050', # Medium green
34
- 'green_3': '#c6e0b4', # Light green (lightest allowable green)
35
- # Secondary colors 2
36
  'gold_1': '#fdb932',
37
- # Tertiary colors
38
  'orange_1': '#ff975e',
39
  'purple_1': '#9a6ce4',
40
- 'purple_2': '#bb9aef', # Medium purple
41
- 'purple_3': '#ceb5f5', # Light purple (lightest allowable purple)
42
- # Grays (other models)
43
- 'gray_1': '#808080', # Darkest gray (use as a last resort)
44
- 'gray_2': '#b3b3b3', # Medium gray (start with this as the darkest when possible)
45
- 'gray_3': '#e6e6e6', # Lightest gray
46
- 'gray_4': '#ffffff', # It's actually just white (use as a last resort)
47
- # If all other options are exhausted
48
- 'cyan_1': '#0096b4', # Darkest teal
49
- 'cyan_2': '#28bed2', # Medium cyan
50
- 'cyan_3': '#8cdceb', # Lightest cyan
51
- 'magenta_1': '#b428a0', # Darkest magenta
52
- 'magenta_2': '#dc50be', # Medium pink
53
- 'magenta_3': '#f5a0dc', # Lightest pink
54
- 'yellow_1': '#c8aa00', # Darkest yellow
55
- 'yellow_2': '#ffd200', # Medium yellow
56
- 'yellow_3': '#fff08c', # Lightest yellow
57
  }
58
 
59
-
60
  ASSAY_TYPE_MAPPING = {
61
  'ATAC-seq': 'chromatin accessibility',
62
  'DNase-seq': 'chromatin accessibility',
63
  'Histone ChIP-seq': 'histone modifications',
64
- 'TF ChIP-seq': 'chromatin accessibility',
65
  'PRO-cap': 'transcription initiation',
66
  'eCLIP': 'RNA binding sites',
67
  'RNA-seq': 'gene expression',
68
  'ribo-seq': 'mRNA translation',
69
- 'Annotation': 'genome annotation',
70
- "Exon": "exon",
71
- "Intron": "intron",
72
- "Splice acceptor": "splice acceptor",
73
- "Start codon": "start codon",
74
-
75
  }
76
 
77
  ASSAY_COLORS = {
78
  'chromatin accessibility': '#004697',
79
  'histone modifications': '#cc0000',
 
80
  'transcription initiation': '#ff9900',
81
  'RNA binding sites': '#9933cc',
82
  'gene expression': '#009900',
83
  'mRNA translation': '#ff6699',
84
- 'genome annotation': '#ffcc00',
85
- "intron": '#004697',
86
- "exon": '#cc0000',
87
- "splice acceptor": '#ff9900',
88
- "start codon": '#9933cc',
89
  }
90
- ASSAY_COLORS["other"] = "#808080"
91
 
92
- MODEL_COLORS = {
93
- "NTv3 650M (pos)": COLORS['blue_0'],
94
- 'NTv3 650M (pre)': COLORS['blue_1'], # #3973fc (Darkest blue)
95
- 'NTv3 100M (pre)': COLORS['blue_2'], # #7ea4fc (Medium blue)
96
- 'NTv3 8M (pre)': COLORS['blue_3'], # #c3d5fc (Light blue)
97
- 'Evo2 1B': COLORS['green_3'], # #b3b3b3 (Medium gray)
98
- "NTv2 500M": COLORS['gray_1'],
99
- "BPNet arch. 6M": COLORS['cyan_1'],
100
- "Residual CNN 44M": COLORS['magenta_1'],
101
- "PlantCAD2 88M": COLORS["purple_1"],
102
- "Caduceus 7M": COLORS["purple_2"],
103
- "HyenaDNA 7M": COLORS["yellow_2"]
104
  }
105
 
106
- MODEL_TRAINING_STATUS = {
107
- "NTv3 650M (pos)": "POS",
108
- "NTv3 650M (pre)": "PRE",
109
- "NTv3 100M (pre)": "PRE",
110
- "NTv3 8M (pre)": "PRE",
111
- "Residual CNN 44M": "SCRATCH",
112
- "Caduceus 7M": "PRE",
113
- "Evo2 1B": "PRE",
114
- "NTv2 500M": "PRE",
115
- "BPNet arch. 6M": "SCRATCH",
116
- "PlantCAD2 88M": "PRE",
117
- "HyenaDNA 7M": "PRE"
 
118
  }
 
119
 
120
- MODEL_GPU_MULTIPLIER = {
121
- "Evo2 1B": 8, # trained on 8 GPUs
 
 
 
 
 
 
 
 
 
 
122
  }
123
 
 
 
 
 
 
 
 
 
 
 
 
 
124
 
125
  MODEL_NAMES = list(MODEL_COLORS.keys())
126
 
127
- PLANT_SPECIES = ["tomato", "rice", "maize", "arabidopsis"]
128
- ANIMAL_SPECIES = ["human", "chicken", "cattle"]
129
-
130
  SPECIES_GROUPS = {
131
- "Plants": PLANT_SPECIES,
132
- "Animals": ANIMAL_SPECIES, # (your code calls these HUMAN_SPECIES, but they’re the “animal” set)
 
 
 
 
 
 
 
133
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
 
135
-
136
- _LAST_UPDATED = "Dec 10, 2025"
137
  _INTRO = """
138
- The **NTv3 Benchmark** is a curated benchmark of 106 long-range genomic datasets
139
- designed to evaluate models under realistic 32 kb input, single-base-pair output settings.
140
- The dataset spans two complementary task families: genome annotation (exon, intron, splice acceptor, start codon)
141
- and functional-regulatory prediction, which includes diverse experimental tracks such as chromatin accessibility,
142
- histone modifications, transcription initiation (PRO-cap), RNA binding (eCLIP), gene expression (RNA-seq),
143
- and translation (Ribo-seq).
144
-
145
- Data are drawn from a phylogenetically diverse set of species, including organisms seen during post-training
146
- (human, chicken, arabidopsis, rice, maize) and entirely unseen species (cattle, tomato), with careful curation
147
- to avoid data leakage. This design allows the dataset to probe long-range sequence-to-function mapping,
148
- cross-species generalization, and transfer across heterogeneous regulatory modalities,
149
- including assays not present in prior multispecies training corpora. By standardizing sequence length,
150
- resolution, and evaluation metrics across all tracks, the NTv3 Benchmark provides a controlled dataset
151
- for comparing representation quality across genomic foundation models.
152
-
153
- The metrics used are:
154
- - **Pearson correlations (multi-assay)**: per-dataset scores across species and models for functional tracks.
155
- - **MCC (bed tracks)**: per-track MCC values across species and models for gene annotation tracks.
156
  """
157
 
158
  HERE = os.path.dirname(os.path.abspath(__file__)) # /app/src
159
  PROJECT_ROOT = os.path.dirname(HERE) # /app
160
  DATA_DIR = os.path.join(PROJECT_ROOT, "data")
161
-
162
- SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "ntv3_benchmark_results.csv")
163
 
164
  # ---------------------------------------------------------------------
165
  # Data loading & preprocessing
@@ -167,139 +179,54 @@ SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "ntv3_benchmark_results.csv")
167
 
168
 
169
  @st.cache_data
170
- def load_raw_data():
171
  df = pd.read_csv(SINGLE_TABLE_PATH)
172
  df.columns = [c.strip() for c in df.columns]
173
  return df
174
 
175
 
176
- def _normalize_training_time_to_gpu_hours(df: pd.DataFrame) -> pd.DataFrame:
177
- """
178
- Your new column is `running_time`. In your sample it looks like seconds
179
- (e.g. 317034 ~= 88 hours). We'll convert to hours if values look like seconds.
180
- """
181
- if "running_time" not in df.columns:
182
- return df
183
-
184
- rt = pd.to_numeric(df["running_time"], errors="coerce")
185
- # Heuristic: if median is huge, it's probably seconds -> convert to hours
186
- # (88 hours = 316800 seconds is a typical-looking value in your sample)
187
- if rt.dropna().median() > 10_000:
188
- df["GPU hours"] = rt / 3600.0
189
- else:
190
- df["GPU hours"] = rt.astype(float)
191
-
192
- return df
193
-
194
-
195
- def _best_step_time_to_hours(s: pd.Series) -> pd.Series:
196
- """
197
- Converts strings like '3 days 04:26:26.467000' to hours (float).
198
- Works with pandas Timedelta parsing.
199
- """
200
- td = pd.to_timedelta(s, errors="coerce")
201
- return td.dt.total_seconds() / 3600.0
202
-
203
-
204
-
205
  @st.cache_data
206
- def load_expanded_data():
207
- df = load_raw_data().copy()
208
-
209
- df = df.rename(columns={"Metric": "Score", "model_name": "Model"})
 
210
  df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
211
-
212
- if "best_step" in df.columns:
213
- df["best_step"] = pd.to_numeric(df["best_step"], errors="coerce")
214
-
215
- if "best_step_time" in df.columns:
216
- df["best_step_time_hours"] = _best_step_time_to_hours(df["best_step_time"])
217
- else:
218
- df["best_step_time_hours"] = np.nan
219
-
220
- is_annot = df.get("assay_type", "").astype(str).eq("Annotation")
221
- pearson_raw = df[~is_annot].copy()
222
- mcc_raw = df[is_annot].copy()
223
-
224
- # -------------------------
225
- # Functional Tracks (Pearson)
226
- # -------------------------
227
- pearson_group_cols = ["species", "datasets", "Model"]
228
- if "assay_type" in pearson_raw.columns:
229
- pearson_group_cols.append("assay_type")
230
-
231
- pearson_df = (
232
- pearson_raw
233
- .groupby(pearson_group_cols, as_index=False, dropna=False)
234
- .agg({
235
- "Score": "mean",
236
- "best_step": "mean",
237
- "best_step_time_hours": "mean",
238
- })
239
  )
 
 
240
 
241
- # ✅ merge track_name_clean WHILE assay_type is still raw
242
- if "track_name_clean" in pearson_raw.columns:
243
- map_keys = ["species", "datasets"]
244
- if "assay_type" in pearson_raw.columns:
245
- map_keys.append("assay_type")
246
-
247
- track_map = (
248
- pearson_raw[map_keys + ["track_name_clean"]]
249
- .dropna(subset=["track_name_clean"])
250
- .drop_duplicates()
251
- )
252
- pearson_df = pearson_df.merge(track_map, on=map_keys, how="left")
253
-
254
- # ✅ now it’s safe to map assay_type to categories
255
- if "assay_type" in pearson_df.columns:
256
- pearson_df["assay_type"] = (
257
- pearson_df["assay_type"].astype(str).map(ASSAY_TYPE_MAPPING).fillna("Other")
258
- )
259
 
260
- # -------------------------
261
- # Genome Annotation (MCC)
262
- # -------------------------
263
- mcc_df = (
264
- mcc_raw
265
- .groupby(["species", "datasets", "Model"], as_index=False, dropna=False)
266
- .agg({
267
- "Score": "mean",
268
- "best_step": "mean",
269
- "best_step_time_hours": "mean",
270
- })
 
 
271
  )
272
-
273
- return pearson_df, mcc_df
 
274
 
275
 
276
- _PEARSON_DF, _MCC_DF = load_expanded_data()
 
277
 
278
- # Global sets (we'll further filter per-benchmark below)
279
- _ALL_SPECIES = sorted(
280
- set(_PEARSON_DF["species"].unique()).union(_MCC_DF["species"].unique())
281
- )
282
 
283
- _ALL_ASSAYS = (
284
- sorted(_PEARSON_DF["assay_type"].dropna().unique())
285
- if "assay_type" in _PEARSON_DF.columns
286
- else []
287
- )
288
-
289
- _ALL_MODELS = MODEL_NAMES[:]
290
-
291
- _BENCHMARKS = {
292
- "Functional Tracks": {
293
- "df": _PEARSON_DF,
294
- "metric_label": "Pearson correlation",
295
- "has_assay_type": True,
296
- },
297
- "Genome Annotation": {
298
- "df": _MCC_DF,
299
- "metric_label": "MCC",
300
- "has_assay_type": False,
301
- },
302
- }
303
 
304
 
305
  # ---------------------------------------------------------------------
@@ -307,775 +234,517 @@ _BENCHMARKS = {
307
  # ---------------------------------------------------------------------
308
 
309
 
310
- def filter_base_df(
311
- benchmark_name: str,
312
- selected_species: List[str],
313
- selected_assays: List[str],
314
- selected_models: List[str],
315
- selected_datasets: List[str],
316
- ) -> pd.DataFrame:
317
- cfg = _BENCHMARKS[benchmark_name]
318
- df = cfg["df"].copy()
319
-
320
- # Species filter
321
- if selected_species:
322
- df = df[df["species"].isin(selected_species)]
323
-
324
- # Assay type filter (Pearson only)
325
- if cfg.get("has_assay_type", False) and selected_assays and "assay_type" in df.columns:
326
- df = df[df["assay_type"].isin(selected_assays)]
327
-
328
- # Dataset / bed track filter (for MCC, but safe to apply generally)
329
- if selected_datasets and "datasets" in df.columns:
330
- df = df[df["datasets"].isin(selected_datasets)]
331
-
332
- # Model filter
333
- if selected_models:
334
- df = df[df["Model"].isin(selected_models)]
335
-
336
- return df
337
-
338
-
339
- def build_leaderboard(
340
- benchmark_name: str,
341
- selected_species: List[str],
342
- selected_assays: List[str],
343
- selected_models: List[str],
344
- selected_datasets: List[str],
345
- ) -> pd.DataFrame:
346
- df = filter_base_df(
347
- benchmark_name,
348
- selected_species,
349
- selected_assays,
350
- selected_models,
351
- selected_datasets,
352
  )
353
 
354
- if df.empty:
355
- return pd.DataFrame(columns=["Model", "Model Type", "Num entries", "Mean score"])
356
 
357
- agg = (
358
- df.groupby("Model")["Score"]
359
- .mean()
360
- .reset_index()
361
- .rename(columns={"Score": "Mean score"})
362
- )
 
 
 
 
363
  agg["Mean score"] = agg["Mean score"].round(3)
364
-
365
- agg["Num entries"] = (
366
- df.groupby("Model")["Score"].count().reindex(agg["Model"]).values
367
- )
368
-
369
- # 👇 Add training regime column
370
  agg["Training"] = agg["Model"].map(MODEL_TRAINING_STATUS).fillna("UNKNOWN")
371
-
372
- # Sort by performance
373
  agg = agg.sort_values("Mean score", ascending=False).reset_index(drop=True)
374
-
375
- # Column order
376
- agg = agg[["Model", "Training", "Num entries", "Mean score"]]
377
-
378
- # Ensure the index starts with 1
379
  agg.index += 1
380
-
381
  return agg
382
 
383
 
384
-
385
- def build_bar_df(
386
- benchmark_name: str,
387
- selected_species: List[str],
388
- selected_assays: List[str],
389
- selected_models: List[str],
390
- selected_datasets: List[str],
391
- ) -> pd.DataFrame:
392
- """For now, just one bar per model (same as leaderboard)."""
393
- return build_leaderboard(
394
- benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
395
  )
 
 
396
 
397
 
398
- def build_category_model_df(
399
- benchmark_name: str,
400
- selected_species: List[str],
401
- selected_assays: List[str],
402
- selected_models: List[str],
403
- selected_datasets: List[str],
404
- ) -> pd.DataFrame:
405
- """
406
- Mean score per (category, Model) after applying the same filters.
407
- Category = assay_type (Functional Tracks) or datasets (Genome Annotation).
408
- """
409
- cfg = _BENCHMARKS[benchmark_name]
410
- df = filter_base_df(
411
- benchmark_name,
412
- selected_species,
413
- selected_assays,
414
- selected_models,
415
- selected_datasets,
 
 
 
 
 
 
 
416
  )
417
- if df.empty:
418
- return pd.DataFrame(columns=["Category", "Model", "Mean score"])
 
419
 
420
- # Pick the right breakdown column
421
- if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
422
- category_col = "assay_type"
423
- category_label = "Assay type"
424
- else:
425
- category_col = "datasets"
426
- category_label = "Dataset"
427
 
428
- if category_col not in df.columns:
429
- return pd.DataFrame(columns=["Category", "Model", "Mean score"])
 
430
 
431
- out = (
432
- df.groupby([category_col, "Model"], as_index=False)["Score"]
433
- .mean()
434
- .rename(columns={category_col: "Category", "Score": "Mean score"})
435
- )
436
- out["Mean score"] = out["Mean score"].round(3)
437
- out.attrs["category_label"] = category_label # for nicer axis title
438
- return out
439
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
440
 
441
- def plot_breakdown_facets_sorted_models(
442
- breakdown_df: pd.DataFrame,
443
- metric_label: str,
444
- height: int = 420,
445
- ):
446
- categories = list(breakdown_df["Category"].dropna().unique())
447
- categories = sorted(categories)
448
 
449
- n = len(categories)
450
- if n == 0:
451
- return None
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
452
 
453
- rows = 1
454
- cols = n # 👈 everything in one row
455
 
456
- # Global y-range (consistent scale)
 
 
 
457
  y_min = breakdown_df["Mean score"].min()
458
  y_max = breakdown_df["Mean score"].max()
459
  pad = 0.05 * (y_max - y_min if y_max > y_min else 1.0)
460
- y_range = [y_min - pad, y_max + pad]
461
 
462
  fig = make_subplots(
463
- rows=rows,
464
- cols=cols,
465
  subplot_titles=categories,
466
  shared_yaxes=True,
467
- horizontal_spacing=0.04, # tighter spacing
468
  )
469
-
470
  for i, cat in enumerate(categories):
471
- r = (i // cols) + 1
472
- c = (i % cols) + 1
473
-
474
- sub = (
475
- breakdown_df[breakdown_df["Category"] == cat]
476
- .sort_values("Mean score", ascending=True)
477
- )
478
-
479
  fig.add_trace(
480
  go.Bar(
481
  x=sub["Model"],
482
  y=sub["Mean score"],
483
  marker_color=[MODEL_COLORS.get(m, "#808080") for m in sub["Model"]],
484
  showlegend=False,
 
485
  ),
486
- row=r,
487
- col=c,
488
  )
489
-
490
- fig.update_xaxes(showticklabels=False, title_text="", row=r, col=c)
491
- fig.update_yaxes(range=y_range, title_text="", row=r, col=c) # 👈 apply range
492
-
493
  fig.update_layout(
494
  height=height,
495
  plot_bgcolor="rgba(0,0,0,0)",
496
  paper_bgcolor="rgba(0,0,0,0)",
497
  margin=dict(t=60, l=10, r=10, b=10),
498
  )
499
-
500
- # Single y-axis label on the leftmost panel
501
- fig.update_yaxes(title_text=metric_label, row=1, col=1)
502
-
503
  return fig
504
 
505
- def build_pairwise_scatter_df(
506
- benchmark_name: str,
507
- selected_species: List[str],
508
- selected_assays: List[str],
509
- selected_models: List[str],
510
- selected_datasets: List[str],
511
- model_a: str,
512
- model_b: str,
513
- ) -> pd.DataFrame:
514
- cfg = _BENCHMARKS[benchmark_name]
515
-
516
- models_for_filter = (
517
- list(set(selected_models + [model_a, model_b]))
518
- if selected_models else [model_a, model_b]
519
- )
520
-
521
- df = filter_base_df(
522
- benchmark_name,
523
- selected_species,
524
- selected_assays,
525
- models_for_filter,
526
- selected_datasets,
527
- )
528
- if df.empty:
529
- return pd.DataFrame()
530
-
531
- # ---- define "track identity" for head-to-head ----
532
- # Always use datasets for the identity (x/y points)
533
- track_cols = ["datasets"]
534
- if cfg.get("has_assay_type", False) and "assay_type" in df.columns:
535
- track_cols = ["assay_type", "datasets"]
536
 
537
- keep_species = "species" in df.columns and (selected_species is None or len(selected_species) != 1)
538
- id_cols = (["species"] if keep_species else []) + track_cols
539
-
540
- wide = (
541
- df[df["Model"].isin([model_a, model_b])]
542
- .pivot_table(index=id_cols, columns="Model", values="Score", aggfunc="mean")
543
- .reset_index()
544
- )
545
-
546
- if model_a not in wide.columns or model_b not in wide.columns:
547
- return pd.DataFrame()
548
 
549
- wide = wide.dropna(subset=[model_a, model_b])
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
550
 
551
- # Nice display label: use datasets (not track_name_clean)
552
- if "assay_type" in wide.columns:
553
- wide["Track"] = wide["assay_type"].astype(str) + " / " + wide["datasets"].astype(str)
 
 
 
 
 
 
 
 
 
 
 
 
554
  else:
555
- wide["Track"] = wide["datasets"].astype(str)
556
-
557
- wide = wide.rename(columns={model_a: "Model A", model_b: "Model B"})
558
-
559
- # ---- Pearson-only: merge track_name_clean for hover ----
560
- if benchmark_name == "Functional Tracks" and "track_name_clean" in df.columns:
561
- merge_keys = id_cols.copy() # species? + assay_type? + datasets
562
- track_map = (
563
- df[merge_keys + ["track_name_clean"]]
564
- .dropna(subset=["track_name_clean"])
565
- .drop_duplicates()
566
  )
567
- wide = wide.merge(track_map, on=merge_keys, how="left")
568
-
569
- return wide
570
-
571
-
572
-
573
- def build_violin_df(
574
- benchmark_name: str,
575
- selected_species: List[str],
576
- selected_assays: List[str],
577
- selected_models: List[str],
578
- selected_datasets: List[str],
579
- ) -> pd.DataFrame:
580
- # Use the same base filtering, but keep all per-track rows
581
- df = filter_base_df(
582
- benchmark_name,
583
- selected_species,
584
- selected_assays,
585
- selected_models,
586
- selected_datasets,
587
  )
588
- # Keep only needed columns
589
- keep = ["Model", "Score"]
590
- for col in ["species", "assay_type", "datasets"]:
591
- if col in df.columns:
592
- keep.append(col)
593
- return df[keep].copy()
594
-
595
-
596
- def build_convergence_df(
597
- benchmark_name: str,
598
- selected_species: List[str],
599
- selected_assays: List[str],
600
- selected_models: List[str],
601
- selected_datasets: List[str],
602
- x_mode: str = "best_step", # "best_step" | "best_step_time"
603
- ) -> pd.DataFrame:
604
-
605
- df = filter_base_df(
606
- benchmark_name,
607
- selected_species,
608
- selected_assays,
609
- selected_models,
610
- selected_datasets,
611
  )
612
-
613
- if df.empty:
614
- return pd.DataFrame(columns=["Model", "X", "Performance"])
615
-
616
- # Mean performance per model
617
- out = (
618
- df.groupby("Model", as_index=False)
619
- .agg({"Score": "mean"})
620
- .rename(columns={"Score": "Performance"})
 
 
 
 
 
 
 
 
 
 
621
  )
 
622
 
623
- # -------------------------
624
- # X axis selection
625
- # -------------------------
626
- if x_mode == "Steps (billions)":
627
- if "best_step" not in df.columns:
628
- return pd.DataFrame(columns=["Model", "X", "Performance"])
629
-
630
- x = (
631
- df.groupby("Model", as_index=False)["best_step"]
632
- .mean()
633
- .rename(columns={"best_step": "X"})
634
- )
635
 
636
- else: # best_step_time (GPU hours)
637
- if "best_step_time_hours" not in df.columns:
638
- return pd.DataFrame(columns=["Model", "X", "Performance"])
639
-
640
- x = (
641
- df.groupby("Model", as_index=False)["best_step_time_hours"]
642
- .mean()
643
- .rename(columns={"best_step_time_hours": "X"})
644
- )
645
 
646
- # 👇 Apply GPU multiplier (Evo2 uses 8 GPUs)
647
- gpu_multiplier = {
648
- "Evo2 1B": 8,
649
- }
650
 
651
- x["X"] = x.apply(
652
- lambda r: r["X"] * gpu_multiplier.get(r["Model"], 1),
653
- axis=1,
 
 
 
 
 
 
 
 
 
 
 
 
 
 
654
  )
655
-
656
- # Merge + clean
657
- out = out.merge(x, on="Model", how="left")
658
- out = out.dropna(subset=["X", "Performance"])
659
-
660
- out["Performance"] = out["Performance"].round(3)
661
-
662
- return out
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
663
 
664
 
665
  # ---------------------------------------------------------------------
666
- # UI helpers
667
  # ---------------------------------------------------------------------
668
 
669
 
670
- def sidebar_toggle(label: str, value: bool = False, key: str | None = None) -> bool:
671
- """
672
- Wrapper to use st.sidebar.toggle when available, otherwise fall back to checkbox.
673
- This makes the app compatible with older Streamlit versions on Hugging Face.
674
- """
675
- toggle_fn = getattr(st.sidebar, "toggle", None)
676
- if toggle_fn is not None:
677
- return toggle_fn(label, value=value, key=key)
678
- # Fallback for older Streamlit versions
679
- return st.sidebar.checkbox(label, value=value, key=key)
 
 
 
 
680
 
681
- # ---------------------------------------------------------------------
682
- # Streamlit UI
683
- # ---------------------------------------------------------------------
684
- def main():
685
  st.title("🧬 NTv3 Benchmark")
686
  st.markdown(_INTRO)
687
  st.markdown(f"_Last updated: **{_LAST_UPDATED}**_")
 
688
 
689
- # --- Sidebar filters ---
690
- st.sidebar.header("Filters")
691
-
692
- # Benchmark
693
- benchmark_name = st.sidebar.selectbox(
694
- "Benchmark",
695
- options=list(_BENCHMARKS.keys()),
696
- index=0,
697
- )
698
-
699
- cfg = _BENCHMARKS[benchmark_name]
700
- df_bench = cfg["df"]
701
-
702
- # Species toggles, but only for species present in this benchmark
703
- st.sidebar.subheader("Species")
704
-
705
- # Toggle: Plants vs Animals
706
- species_group = st.sidebar.radio(
707
- "Group",
708
- options=["Animals", "Plants"],
709
- index=0,
710
- horizontal=True,
711
- key=f"species_group_{benchmark_name}",
712
  )
 
 
713
 
714
- available_species_all = sorted(df_bench["species"].unique())
715
- allowed_species = set(SPECIES_GROUPS[species_group]).intersection(available_species_all)
716
- available_species = sorted(allowed_species)
717
-
718
- selected_species: List[str] = []
719
- for sp in available_species:
720
- if sidebar_toggle(sp, value=True, key=f"species_{benchmark_name}_{species_group}_{sp}"):
721
- selected_species.append(sp)
722
-
723
- # (Optional) If no species exist for that group in this benchmark
724
- if not available_species:
725
- st.sidebar.info(f"No {species_group.lower()} species available for this benchmark.")
726
-
727
- # Assay toggles (Pearson only), based on filtered species
728
- if cfg.get("has_assay_type", False):
729
- st.sidebar.subheader("Assay types")
730
- if selected_species:
731
- df_for_assays = df_bench[df_bench["species"].isin(selected_species)]
732
- else:
733
- df_for_assays = df_bench
734
- available_assays = (
735
- sorted(df_for_assays["assay_type"].dropna().unique())
736
- if "assay_type" in df_for_assays.columns
737
- else []
738
- )
739
- selected_assays: List[str] = []
740
- for assay in available_assays:
741
- if sidebar_toggle(assay, value=True, key=f"assay_{benchmark_name}_{assay}"):
742
- selected_assays.append(assay)
743
- else:
744
- selected_assays = []
745
-
746
- # Bed track / dataset toggles (MCC only), based on species selection
747
- selected_datasets: List[str] = []
748
- if benchmark_name == "Genome Annotation":
749
- st.sidebar.subheader("Genome annotations")
750
- if selected_species:
751
- df_for_tracks = df_bench[df_bench["species"].isin(selected_species)]
752
- else:
753
- df_for_tracks = df_bench
754
- available_datasets = sorted(df_for_tracks["datasets"].unique())
755
- for ds in available_datasets:
756
- if sidebar_toggle(ds, value=True, key=f"dataset_{benchmark_name}_{ds}"):
757
- selected_datasets.append(ds)
758
- else:
759
- selected_datasets = []
760
-
761
- # Model toggles (we keep all models in MODEL_NAMES; filters + data will prune)
762
- st.sidebar.subheader("Models")
763
- selected_models: List[str] = []
764
- for model in _ALL_MODELS:
765
- if sidebar_toggle(model, value=True, key=f"model_{model}"):
766
- selected_models.append(model)
767
-
768
- # -------------------------
769
- # ✅ Validation: require ≥1 selection per relevant category
770
- # -------------------------
771
- missing = []
772
-
773
- # Always required
774
- if not selected_species:
775
- missing.append("Species")
776
- if not selected_models:
777
- missing.append("Models")
778
-
779
- # Required depending on benchmark
780
- if cfg.get("has_assay_type", False) and not selected_assays:
781
- missing.append("Assay types")
782
-
783
- if benchmark_name == "Genome Annotation" and not selected_datasets:
784
- missing.append("Genome annotations")
785
-
786
- if missing:
787
- # Show a single message and prevent *any* further display
788
- st.error(
789
- "Please select at least one item in each category. Currently missing: "
790
- + ", ".join(missing)
791
- + "."
792
  )
793
- st.stop()
794
 
795
- # --- Main content ---
796
- leaderboard_df = build_leaderboard(
797
- benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
798
- )
799
- bar_df = build_bar_df(
800
- benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
 
 
 
 
801
  )
802
 
 
803
  col1, col2 = st.columns([1, 1])
804
-
805
  with col1:
806
  st.subheader("🏅 Leaderboard")
807
- st.write("\n") # spacer to match plotly padding
808
  st.write("\n")
809
  st.write("\n")
810
- if leaderboard_df.empty:
811
- st.info("No data for the selected filters.")
812
- else:
813
- st.dataframe(leaderboard_df, use_container_width=True)
814
-
815
  with col2:
816
  st.subheader("📈 Mean score per model")
817
- if bar_df.empty:
818
- st.info("No data for the selected filters.")
819
- else:
820
- # Order models by performance (least -> most)
821
- bar_df = bar_df.sort_values("Mean score", ascending=True)
822
- model_order = bar_df["Model"].tolist()
823
-
824
- fig = px.bar(
825
- bar_df,
826
- x="Model",
827
- y="Mean score",
828
- color="Model",
829
- color_discrete_map=MODEL_COLORS,
830
- category_orders={"Model": model_order},
831
- )
832
- fig.update_layout(
833
- barmode="group",
834
- height=500,
835
- xaxis_title="",
836
- yaxis_title=cfg["metric_label"],
837
- plot_bgcolor="rgba(0,0,0,0)",
838
- paper_bgcolor="rgba(0,0,0,0)",
839
- bargap=0.08,
840
- )
841
- fig.update_xaxes(showticklabels=False)
842
- st.plotly_chart(fig, use_container_width=True)
843
-
844
- # --- Breakdown plot: assay_type (Functional Tracks) OR datasets (Genome Annotation) ---
845
- breakdown_df = build_category_model_df(
846
- benchmark_name, selected_species, selected_assays, selected_models, selected_datasets
847
- )
848
 
849
- type_of_data = "assay type" if benchmark_name == "Functional Tracks" else "gene annotation"
 
 
 
850
 
851
- st.subheader(f"🧪 Mean score by {type_of_data}")
852
- if breakdown_df.empty:
853
- st.info("No data for the selected filters.")
854
- else:
855
- fig_breakdown = plot_breakdown_facets_sorted_models(
856
- breakdown_df,
857
- metric_label=cfg["metric_label"],
858
- height=300,
859
- )
860
- st.plotly_chart(fig_breakdown, use_container_width=True)
861
-
862
- # ------------------------------------------------------------------
863
- # Model comparison: Head-to-head (left) + Convergence (right)
864
- # ------------------------------------------------------------------
865
  left, right = st.columns([1, 1], gap="large")
866
-
867
  with left:
868
- st.markdown("#### ⚖️ Head-to-head (per track)")
869
-
870
- model_picker_options = selected_models if selected_models else _ALL_MODELS
871
- default_a = model_picker_options[0] if model_picker_options else _ALL_MODELS[0]
872
- default_b = model_picker_options[1] if len(model_picker_options) > 1 else (
873
- _ALL_MODELS[1] if len(_ALL_MODELS) > 1 else default_a
874
- )
875
-
876
- cA, cB = st.columns([1, 1])
877
- with cA:
878
- model_a = st.selectbox(
879
- "Model A (y-axis)",
880
- options=model_picker_options,
881
- index=model_picker_options.index(default_a) if default_a in model_picker_options else 0,
882
- key=f"pair_model_a_{benchmark_name}",
883
- )
884
- with cB:
885
- b_options = [m for m in model_picker_options if m != model_a] or model_picker_options
886
- model_b = st.selectbox(
887
- "Model B (x-axis)",
888
- options=b_options,
889
- index=0,
890
- key=f"pair_model_b_{benchmark_name}",
891
- )
892
-
893
- scatter_df = build_pairwise_scatter_df(
894
- benchmark_name,
895
- selected_species,
896
- selected_assays,
897
- selected_models,
898
- selected_datasets,
899
- model_a,
900
- model_b,
901
- )
902
-
903
- if scatter_df.empty:
904
- st.info("No overlapping tracks for the selected filters (or one model is missing values).")
905
- else:
906
- min_v = float(min(scatter_df["Model A"].min(), scatter_df["Model B"].min()))
907
- max_v = float(max(scatter_df["Model A"].max(), scatter_df["Model B"].max()))
908
- pad = 0.05 * (max_v - min_v if max_v > min_v else 1.0)
909
- axis_range = [min_v - pad, max_v + pad]
910
- tick_step = (axis_range[1] - axis_range[0]) / 5
911
-
912
- hover_cols = ["datasets"]
913
- if benchmark_name == "Functional Tracks" and "track_name_clean" in scatter_df.columns:
914
- hover_cols.append("track_name_clean")
915
- else:
916
- hover_cols.append("datasets")
917
-
918
- color_col = "assay_type" if "assay_type" in scatter_df.columns else "datasets"
919
-
920
- fig_scatter = px.scatter(
921
- scatter_df,
922
- x="Model B",
923
- y="Model A",
924
- color=color_col,
925
- color_discrete_map=ASSAY_COLORS,
926
- hover_name="Track",
927
- hover_data=hover_cols,
928
- )
929
-
930
- fig_scatter.add_shape(
931
- type="line",
932
- x0=axis_range[0], y0=axis_range[0],
933
- x1=axis_range[1], y1=axis_range[1],
934
- xref="x", yref="y",
935
- line=dict(color="red", dash="dot", width=2),
936
- )
937
-
938
- fig_scatter.update_layout(
939
- height=550,
940
- margin=dict(l=60, r=20, t=20, b=60),
941
- xaxis=dict(
942
- title=f"{model_b} — {cfg['metric_label']}",
943
- range=axis_range,
944
- dtick=tick_step,
945
- constrain="domain",
946
- ),
947
- yaxis=dict(
948
- title=f"{model_a} — {cfg['metric_label']}",
949
- range=axis_range,
950
- dtick=tick_step,
951
- scaleanchor="x",
952
- scaleratio=1,
953
- constrain="domain",
954
- ),
955
- plot_bgcolor="rgba(0,0,0,0)",
956
- paper_bgcolor="rgba(0,0,0,0)",
957
- )
958
-
959
- fig_scatter.update_layout(
960
- legend=dict(
961
- title="Assay type" if benchmark_name == "Functional Tracks" else "Genome Annotation",
962
- x=0.98,
963
- y=0.1,
964
- xanchor="right",
965
- yanchor="bottom",
966
- bgcolor="rgba(255,255,255,0.2)", # semi-transparent white
967
- bordercolor="rgba(0,0,0,0.2)",
968
- borderwidth=1,
969
- )
970
- )
971
-
972
- st.plotly_chart(fig_scatter, use_container_width=True)
973
-
974
  with right:
975
- st.markdown("#### ⏱️ Time to convergence")
976
 
977
- x_mode = st.selectbox(
978
- "X-axis",
979
- options=["GPU (hours)", "Steps (billions)"],
980
- index=0,
981
- key=f"conv_x_mode_{benchmark_name}",
982
- )
983
 
984
 
985
- conv_df = build_convergence_df(
986
- benchmark_name,
987
- selected_species,
988
- selected_assays,
989
- selected_models,
990
- selected_datasets,
991
- x_mode=x_mode,
992
- )
993
 
994
- if conv_df.empty:
995
- st.info("No convergence data found for the selected filters / x-axis mode.")
996
- else:
997
- fig_conv = px.scatter(
998
- conv_df,
999
- x="X",
1000
- y="Performance",
1001
- text="Model",
1002
- color="Model",
1003
- color_discrete_map=MODEL_COLORS,
1004
- hover_data=["Model", "X", "Performance"],
1005
- )
1006
- fig_conv.update_layout(
1007
- height=550,
1008
- xaxis_title=("GPU (hours)" if x_mode == "GPU (hours)" else x_mode),
1009
- yaxis_title=cfg["metric_label"],
1010
- plot_bgcolor="rgba(0,0,0,0)",
1011
- paper_bgcolor="rgba(0,0,0,0)",
1012
- showlegend=False, # ✅ no legend
1013
- )
1014
- fig_conv.update_traces(
1015
- marker=dict(size=14), # 👈 bigger dots
1016
- textposition="top center",
1017
- )
1018
-
1019
- # Log scale only makes sense for hours (and sometimes best_step)
1020
- if x_mode in ["GPU (hours)"]:
1021
- fig_conv.update_xaxes(
1022
- type="log",
1023
- dtick=1,
1024
- minor=dict(ticks="", showgrid=False),
1025
- )
1026
- st.plotly_chart(fig_conv, use_container_width=True)
1027
-
1028
- # ------------------------------------------------------------------
1029
- # Violin (full width, below)
1030
- # ------------------------------------------------------------------
1031
- st.subheader("🎻 Performance comparison across tracks")
1032
-
1033
- violin_df = build_violin_df(
1034
- benchmark_name,
1035
- selected_species,
1036
- selected_assays,
1037
- selected_models,
1038
- selected_datasets,
1039
  )
1040
 
1041
- if violin_df.empty:
1042
- st.info("No data for the selected filters.")
1043
- else:
1044
- model_order = (
1045
- violin_df
1046
- .groupby("Model")["Score"]
1047
- .median()
1048
- .sort_values(ascending=True)
1049
- .index
1050
- .tolist()
1051
- )
1052
-
1053
- fig_violin = px.violin(
1054
- violin_df,
1055
- x="Model",
1056
- y="Score",
1057
- color="Model",
1058
- color_discrete_map=MODEL_COLORS,
1059
- box=True,
1060
- points=False,
1061
- category_orders={"Model": model_order},
1062
- )
1063
 
1064
- fig_violin.update_layout(
1065
- height=650,
1066
- xaxis_title="",
1067
- yaxis_title=cfg["metric_label"],
1068
- plot_bgcolor="rgba(0,0,0,0)",
1069
- paper_bgcolor="rgba(0,0,0,0)",
1070
- showlegend=False,
1071
- )
1072
-
1073
- fig_violin.update_traces(
1074
- box_visible=True,
1075
- meanline_visible=False,
1076
- )
1077
-
1078
- st.plotly_chart(fig_violin, use_container_width=True)
 
 
1079
 
1080
 
1081
  if __name__ == "__main__":
 
 
1
  import os
2
 
3
+ import numpy as np
4
  import pandas as pd
 
5
  import plotly.express as px
 
6
  import plotly.graph_objects as go
7
+ import streamlit as st
8
+ from plotly.subplots import make_subplots
9
 
10
  # ---------------------------------------------------------------------
11
  # Page config (must be the first Streamlit command)
 
15
  layout="wide",
16
  )
17
 
18
+ import initial_release # noqa: E402 (reads its CSV on import, so after the page config)
19
+
20
  # ---------------------------------------------------------------------
21
  # Configuration
22
  # ---------------------------------------------------------------------
23
  COLORS = {
24
+ 'blue_0': '#004697',
25
+ 'blue_1': '#3973fc',
26
+ 'blue_2': '#7ea4fc',
27
+ 'blue_3': '#c3d5fc',
28
+ 'red_1': '#ff554d',
29
+ 'green_1': '#00b050',
30
+ 'green_3': '#c6e0b4',
 
 
 
 
 
 
31
  'gold_1': '#fdb932',
 
32
  'orange_1': '#ff975e',
33
  'purple_1': '#9a6ce4',
34
+ 'purple_2': '#bb9aef',
35
+ 'gray_1': '#808080',
36
+ 'gray_2': '#b3b3b3',
37
+ 'cyan_1': '#0096b4',
38
+ 'magenta_1': '#b428a0',
 
 
 
 
 
 
 
 
 
 
 
 
39
  }
40
 
 
41
  ASSAY_TYPE_MAPPING = {
42
  'ATAC-seq': 'chromatin accessibility',
43
  'DNase-seq': 'chromatin accessibility',
44
  'Histone ChIP-seq': 'histone modifications',
45
+ 'TF ChIP-seq': 'TF binding',
46
  'PRO-cap': 'transcription initiation',
47
  'eCLIP': 'RNA binding sites',
48
  'RNA-seq': 'gene expression',
49
  'ribo-seq': 'mRNA translation',
 
 
 
 
 
 
50
  }
51
 
52
  ASSAY_COLORS = {
53
  'chromatin accessibility': '#004697',
54
  'histone modifications': '#cc0000',
55
+ 'TF binding': '#00b050',
56
  'transcription initiation': '#ff9900',
57
  'RNA binding sites': '#9933cc',
58
  'gene expression': '#009900',
59
  'mRNA translation': '#ff6699',
60
+ 'other': '#808080',
 
 
 
 
61
  }
 
62
 
63
+ CLADE_COLORS = {
64
+ "Animals": COLORS['blue_1'],
65
+ "Plants": COLORS['green_1'],
66
+ "Fungi": COLORS['gold_1'],
67
+ "Protists": COLORS['purple_1'],
 
 
 
 
 
 
 
68
  }
69
 
70
+ # CSV model name -> the label the paper figure uses. Other models in the CSV (v1 NTv3
71
+ # checkpoints, small BPNet / Residual CNN) are kept in the data but not shown.
72
+ MODEL_LABELS = {
73
+ "BPNet arch. 6M": "BPNet",
74
+ "Residual CNN 44M": "Residual CNN",
75
+ "HyenaDNA 7M": "HyenaDNA",
76
+ "Caduceus 7M": "Caduceus",
77
+ "NTv2 500M": "NTv2",
78
+ "Evo2 1B": "Evo2",
79
+ "PlantCAD2 88M": "PlantCAD2",
80
+ "Borzoi": "Borzoi",
81
+ "AlphaGenome": "AlphaGenome",
82
+ "NTv3 700M (tracks)": "NTv3",
83
  }
84
+ NTV3 = "NTv3"
85
 
86
+ # In the paper figure's architecture blocks, with its colours.
87
+ MODEL_COLORS = {
88
+ "BPNet": COLORS['cyan_1'],
89
+ "Residual CNN": COLORS['magenta_1'],
90
+ "HyenaDNA": COLORS['gold_1'],
91
+ "Caduceus": COLORS['purple_2'],
92
+ "NTv2": COLORS['gray_1'],
93
+ "Evo2": COLORS['green_3'],
94
+ "PlantCAD2": COLORS['purple_1'],
95
+ "Borzoi": COLORS['gray_2'],
96
+ "AlphaGenome": COLORS['orange_1'],
97
+ NTV3: COLORS['blue_0'],
98
  }
99
 
100
+ MODEL_TRAINING_STATUS = {
101
+ "BPNet": "SCRATCH",
102
+ "Residual CNN": "SCRATCH",
103
+ "HyenaDNA": "PRE",
104
+ "Caduceus": "PRE",
105
+ "NTv2": "PRE",
106
+ "Evo2": "PRE",
107
+ "PlantCAD2": "PRE",
108
+ "Borzoi": "PRE",
109
+ "AlphaGenome": "PRE",
110
+ NTV3: "POS",
111
+ }
112
 
113
  MODEL_NAMES = list(MODEL_COLORS.keys())
114
 
115
+ # CSV species key -> label, grouped by clade, in the paper's order.
 
 
116
  SPECIES_GROUPS = {
117
+ "Animals": {"human": "Human", "cattle": "Cattle", "rat": "Rat"},
118
+ "Plants": {
119
+ "arabidopsis": "A. thaliana",
120
+ "rice": "Rice",
121
+ "maize": "Maize",
122
+ "tomato": "Tomato",
123
+ },
124
+ "Fungi": {"yeast": "S. cerevisiae", "fission yeast": "S. pombe"},
125
+ "Protists": {"plasmodium": "P. falciparum", "dictyostelium": "D. discoideum"},
126
  }
127
+ SPECIES_LABEL = {k: v for group in SPECIES_GROUPS.values() for k, v in group.items()}
128
+ SPECIES_ORDER = list(SPECIES_LABEL)
129
+ SPECIES_CLADE = {s: name for name, group in SPECIES_GROUPS.items() for s in group}
130
+ SPECIES_ICON = {
131
+ "human": "🧍",
132
+ "cattle": "🐄",
133
+ "rat": "🐀",
134
+ "arabidopsis": "🌱",
135
+ "rice": "🌾",
136
+ "maize": "🌽",
137
+ "tomato": "🍅",
138
+ "yeast": "🍞",
139
+ "fission yeast": "🍺",
140
+ "plasmodium": "🦟",
141
+ "dictyostelium": "🦠",
142
+ }
143
+ SPECIES_COMMON_NAME = {
144
+ "arabidopsis": "thale cress",
145
+ "yeast": "baker's yeast",
146
+ "fission yeast": "fission yeast",
147
+ "plasmodium": "malaria parasite",
148
+ "dictyostelium": "social amoeba",
149
+ }
150
+ METRIC_LABEL = "Pearson correlation"
151
 
152
+ _LAST_UPDATED = "Sep 24, 2026"
 
153
  _INTRO = """
154
+ The **NTv3 Benchmark** is 155 long-range functional genomics tracks for evaluating models
155
+ under realistic 32 kb input, single-base-pair output settings. The tracks span chromatin accessibility (ATAC-seq),
156
+ histone modifications and transcription factor binding (ChIP-seq), transcription initiation (PRO-cap),
157
+ RNA binding (eCLIP), gene expression (RNA-seq) and translation (Ribo-seq).
158
+
159
+ The benchmark covers 11 phylogenetically diverse species:
160
+ **animals** (human, cattle, rat), **plants** (*A. thaliana*, rice, maize, tomato),
161
+ **fungi** (*S. cerevisiae*, *S. pombe*) and **protists** (*P. falciparum*, *D. discoideum*).
162
+
163
+ The metric is the **Pearson correlation** on the test set, per track, averaged over three
164
+ seeds where available. Not every model was run on every species, so scores are never
165
+ averaged across species: pick a species in the sidebar for its own leaderboard, where every
166
+ model shown was evaluated on all of that species' tracks. AlphaGenome is fine-tuned from its
167
+ fold-1 checkpoint on human, whose test regions the all-folds checkpoint saw, and from its
168
+ all-folds checkpoint on every other species.
 
 
 
169
  """
170
 
171
  HERE = os.path.dirname(os.path.abspath(__file__)) # /app/src
172
  PROJECT_ROOT = os.path.dirname(HERE) # /app
173
  DATA_DIR = os.path.join(PROJECT_ROOT, "data")
174
+ SINGLE_TABLE_PATH = os.path.join(DATA_DIR, "ntv3_benchmark_results.csv")
 
175
 
176
  # ---------------------------------------------------------------------
177
  # Data loading & preprocessing
 
179
 
180
 
181
  @st.cache_data
182
+ def load_raw_data() -> pd.DataFrame:
183
  df = pd.read_csv(SINGLE_TABLE_PATH)
184
  df.columns = [c.strip() for c in df.columns]
185
  return df
186
 
187
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
188
  @st.cache_data
189
+ def load_track_data() -> pd.DataFrame:
190
+ """One row per (species, track, model): the score averaged over seeds."""
191
+ df = load_raw_data().rename(columns={"Metric": "Score", "model_name": "Model"})
192
+ df["Model"] = df["Model"].map(MODEL_LABELS)
193
+ df = df[df["Model"].notna() & df["species"].isin(SPECIES_ORDER)]
194
  df["Score"] = pd.to_numeric(df["Score"], errors="coerce")
195
+ keys = ["species", "datasets", "Model", "assay_type", "track_name_clean"]
196
+ out = df.groupby(keys, as_index=False, dropna=False).agg(
197
+ Score=("Score", "mean"), Seeds=("Score", "size")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
198
  )
199
+ out["assay_type"] = out["assay_type"].map(ASSAY_TYPE_MAPPING).fillna("other")
200
+ return out
201
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
202
 
203
+ @st.cache_data
204
+ def load_run_data() -> pd.DataFrame:
205
+ """One row per training run: its best checkpoint's step and GPU hours."""
206
+ df = load_raw_data().rename(columns={"model_name": "Model"})
207
+ df["Model"] = df["Model"].map(MODEL_LABELS)
208
+ df = df[df["Model"].notna() & df["species"].isin(SPECIES_ORDER)]
209
+ run_cols = ["species", "Model", "best_step", "best_step_time", "running_time"]
210
+ runs = df.drop_duplicates(run_cols).copy()
211
+ # Training steps only: excludes job setup and validation, whose cadence differs
212
+ # between runs of the benchmark.
213
+ # Parsed one by one: pandas 3 misparses a column mixing fraction lengths.
214
+ hours = runs["best_step_train_time"].map(
215
+ lambda v: pd.Timedelta(v).total_seconds() / 3600 if pd.notna(v) else np.nan
216
  )
217
+ runs["GPU hours"] = hours * pd.to_numeric(runs["num_gpus"], errors="coerce")
218
+ runs["Tokens (billions)"] = pd.to_numeric(runs["best_step"], errors="coerce") / 1e9
219
+ return runs
220
 
221
 
222
+ _TRACKS = load_track_data()
223
+ _RUNS = load_run_data()
224
 
 
 
 
 
225
 
226
+ def models_in(df: pd.DataFrame) -> list[str]:
227
+ """Models present in `df`, in the paper's block order."""
228
+ present = set(df["Model"])
229
+ return [m for m in MODEL_NAMES if m in present]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
230
 
231
 
232
  # ---------------------------------------------------------------------
 
234
  # ---------------------------------------------------------------------
235
 
236
 
237
+ def build_species_scores(df: pd.DataFrame) -> pd.DataFrame:
238
+ """Mean over tracks per (species, model): the paper figure's points."""
239
+ return df.groupby(["species", "Model"], as_index=False).agg(
240
+ Score=("Score", "mean"),
241
+ Tracks=("datasets", "nunique"),
242
+ Seeds=("Seeds", "max"),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
243
  )
244
 
 
 
245
 
246
+ def build_leaderboard(df: pd.DataFrame) -> pd.DataFrame:
247
+ if df.empty:
248
+ return pd.DataFrame(columns=["Model", "Training", "Num entries", "Mean score"])
249
+ agg = df.groupby("Model").agg(
250
+ **{
251
+ "Num entries": ("Score", "count"),
252
+ "Seeds": ("Seeds", "max"),
253
+ "Mean score": ("Score", "mean"),
254
+ }
255
+ ).reset_index()
256
  agg["Mean score"] = agg["Mean score"].round(3)
 
 
 
 
 
 
257
  agg["Training"] = agg["Model"].map(MODEL_TRAINING_STATUS).fillna("UNKNOWN")
 
 
258
  agg = agg.sort_values("Mean score", ascending=False).reset_index(drop=True)
259
+ agg = agg[["Model", "Training", "Num entries", "Seeds", "Mean score"]]
 
 
 
 
260
  agg.index += 1
 
261
  return agg
262
 
263
 
264
+ def build_category_model_df(df: pd.DataFrame) -> pd.DataFrame:
265
+ """Mean score per (assay category, model)."""
266
+ if df.empty:
267
+ return pd.DataFrame(columns=["Category", "Model", "Mean score"])
268
+ out = (
269
+ df.groupby(["assay_type", "Model"], as_index=False)["Score"].mean()
270
+ .rename(columns={"assay_type": "Category", "Score": "Mean score"})
 
 
 
 
271
  )
272
+ out["Mean score"] = out["Mean score"].round(3)
273
+ return out
274
 
275
 
276
+ def build_pairwise_scatter_df(df: pd.DataFrame, model_a: str, model_b: str) -> pd.DataFrame:
277
+ """Scores of two models on the tracks both were evaluated on."""
278
+ sub = df[df["Model"].isin([model_a, model_b])]
279
+ wide = sub.pivot_table(
280
+ index=["species", "assay_type", "datasets", "track_name_clean"],
281
+ columns="Model",
282
+ values="Score",
283
+ ).reset_index()
284
+ if model_a not in wide.columns or model_b not in wide.columns:
285
+ return pd.DataFrame()
286
+ wide = wide.dropna(subset=[model_a, model_b])
287
+ wide["Species"] = wide["species"].map(SPECIES_LABEL)
288
+ wide["Track"] = wide["Species"] + " / " + wide["track_name_clean"].astype(str)
289
+ return wide.rename(columns={model_a: "Model A", model_b: "Model B"})
290
+
291
+
292
+ def aggregate_pairs_by_species(pairs: pd.DataFrame) -> pd.DataFrame:
293
+ """One point per species: both models' means over the tracks they share there."""
294
+ clade = {s: name for name, group in SPECIES_GROUPS.items() for s in group}
295
+ out = pairs.groupby(["species", "Species"], as_index=False).agg(
296
+ **{
297
+ "Model A": ("Model A", "mean"),
298
+ "Model B": ("Model B", "mean"),
299
+ "Shared tracks": ("datasets", "size"),
300
+ }
301
  )
302
+ out["Clade"] = out["species"].map(clade)
303
+ out["Track"] = out["Species"]
304
+ return out
305
 
 
 
 
 
 
 
 
306
 
307
+ # ---------------------------------------------------------------------
308
+ # Plots
309
+ # ---------------------------------------------------------------------
310
 
 
 
 
 
 
 
 
 
311
 
312
+ def plot_species_overview(scores: pd.DataFrame, models: list[str]) -> go.Figure:
313
+ """The paper figure: one column per species, one point per model evaluated on it."""
314
+ cols = [s for s in SPECIES_ORDER if s in set(scores["species"])]
315
+ labels = [SPECIES_LABEL[s] for s in cols]
316
+ fig = go.Figure()
317
+ # Thin guide line per species, from the weakest to the strongest model.
318
+ span = scores.groupby("species")["Score"].agg(["min", "max"]).reindex(cols)
319
+ for label, (lo, hi) in zip(labels, span.itertuples(index=False)):
320
+ fig.add_shape(
321
+ type="line", x0=label, x1=label, y0=lo, y1=hi, layer="below",
322
+ line=dict(color="rgba(128,128,128,0.35)", width=2),
323
+ )
324
+ # AlphaGenome is drawn last so it stays visible where it sits on NTv3's marker;
325
+ # `legendrank` keeps the legend in the paper's block order.
326
+ draw_order = [m for m in models if m != "AlphaGenome"] + (
327
+ ["AlphaGenome"] if "AlphaGenome" in models else []
328
+ )
329
+ for model in draw_order:
330
+ sub = scores[scores["Model"] == model].set_index("species").reindex(cols)
331
+ is_ntv3 = model == NTV3
332
+ fig.add_trace(
333
+ go.Scatter(
334
+ x=labels,
335
+ y=sub["Score"],
336
+ mode="markers",
337
+ name=model,
338
+ legendrank=models.index(model),
339
+ marker=dict(
340
+ color=MODEL_COLORS[model],
341
+ size=16 if is_ntv3 else 11,
342
+ symbol="diamond" if is_ntv3 else "circle",
343
+ line=dict(color="rgba(0,0,0,0.5)", width=1),
344
+ ),
345
+ customdata=np.stack([sub["Tracks"], sub["Seeds"]], axis=-1),
346
+ hovertemplate=(
347
+ f"<b>{model}</b><br>%{{x}}<br>{METRIC_LABEL}: %{{y:.3f}}"
348
+ "<br>%{customdata[0]} tracks, %{customdata[1]} seed(s)<extra></extra>"
349
+ ),
350
+ )
351
+ )
352
+ # Clade separators and clade names above the plot.
353
+ x = 0
354
+ for name, group in SPECIES_GROUPS.items():
355
+ n = sum(1 for s in group if s in cols)
356
+ if n == 0:
357
+ continue
358
+ fig.add_annotation(
359
+ x=x + (n - 1) / 2, y=1.0, xref="x", yref="paper", yanchor="bottom",
360
+ text=f"<b>{name}</b>", showarrow=False,
361
+ )
362
+ x += n
363
+ if x < len(cols):
364
+ fig.add_vline(x=x - 0.5, line=dict(color="rgba(128,128,128,0.4)", dash="dot"))
365
+ fig.update_xaxes(categoryorder="array", categoryarray=labels, title_text="")
366
+ fig.update_yaxes(
367
+ title_text=f"{METRIC_LABEL} on test set",
368
+ showgrid=True,
369
+ gridcolor="rgba(128,128,128,0.15)",
370
+ automargin=True,
371
+ tickformat=".1f",
372
+ )
373
+ fig.update_layout(
374
+ height=600,
375
+ plot_bgcolor="rgba(0,0,0,0)",
376
+ paper_bgcolor="rgba(0,0,0,0)",
377
+ legend=dict(orientation="h", yanchor="top", y=-0.1, x=0.5, xanchor="center"),
378
+ margin=dict(t=40, l=80, r=10, b=40),
379
+ )
380
+ return fig
381
 
 
 
 
 
 
 
 
382
 
383
+ def plot_mean_bars(leaderboard: pd.DataFrame) -> go.Figure:
384
+ bar_df = leaderboard.sort_values("Mean score", ascending=True)
385
+ fig = px.bar(
386
+ bar_df,
387
+ x="Model",
388
+ y="Mean score",
389
+ color="Model",
390
+ color_discrete_map=MODEL_COLORS,
391
+ category_orders={"Model": bar_df["Model"].tolist()},
392
+ )
393
+ fig.update_layout(
394
+ height=500,
395
+ xaxis_title="",
396
+ yaxis_title=METRIC_LABEL,
397
+ plot_bgcolor="rgba(0,0,0,0)",
398
+ paper_bgcolor="rgba(0,0,0,0)",
399
+ bargap=0.08,
400
+ )
401
+ fig.update_xaxes(showticklabels=False)
402
+ return fig
403
 
 
 
404
 
405
+ def plot_breakdown_facets_sorted_models(breakdown_df: pd.DataFrame, height: int = 300):
406
+ categories = sorted(breakdown_df["Category"].dropna().unique())
407
+ if not categories:
408
+ return None
409
  y_min = breakdown_df["Mean score"].min()
410
  y_max = breakdown_df["Mean score"].max()
411
  pad = 0.05 * (y_max - y_min if y_max > y_min else 1.0)
412
+ y_range = [min(0, y_min - pad), y_max + pad]
413
 
414
  fig = make_subplots(
415
+ rows=1,
416
+ cols=len(categories),
417
  subplot_titles=categories,
418
  shared_yaxes=True,
419
+ horizontal_spacing=0.04,
420
  )
 
421
  for i, cat in enumerate(categories):
422
+ sub = breakdown_df[breakdown_df["Category"] == cat].sort_values("Mean score")
 
 
 
 
 
 
 
423
  fig.add_trace(
424
  go.Bar(
425
  x=sub["Model"],
426
  y=sub["Mean score"],
427
  marker_color=[MODEL_COLORS.get(m, "#808080") for m in sub["Model"]],
428
  showlegend=False,
429
+ hovertemplate="%{x}: %{y:.3f}<extra></extra>",
430
  ),
431
+ row=1,
432
+ col=i + 1,
433
  )
434
+ fig.update_xaxes(showticklabels=False, title_text="", row=1, col=i + 1)
435
+ fig.update_yaxes(range=y_range, title_text="", row=1, col=i + 1)
 
 
436
  fig.update_layout(
437
  height=height,
438
  plot_bgcolor="rgba(0,0,0,0)",
439
  paper_bgcolor="rgba(0,0,0,0)",
440
  margin=dict(t=60, l=10, r=10, b=10),
441
  )
442
+ fig.update_yaxes(title_text=METRIC_LABEL, row=1, col=1)
 
 
 
443
  return fig
444
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
445
 
446
+ def show_head_to_head(
447
+ df: pd.DataFrame, key: str, per_species: bool = False, title: str | None = None
448
+ ) -> None:
449
+ """Two models on their shared tracks only, with the number of tracks compared.
 
 
 
 
 
 
 
450
 
451
+ Per track by default, one point per track coloured by assay; `per_species` shows
452
+ one point per species instead, each the mean over that species' shared tracks.
453
+ """
454
+ st.markdown(title or "#### ⚖️ Head-to-head (per track)")
455
+ options = models_in(df)
456
+ if len(options) < 2:
457
+ st.info("Fewer than two models were evaluated here.")
458
+ return
459
+ default_a = options.index(NTV3) if NTV3 in options else 0
460
+ c_a, c_b = st.columns([1, 1])
461
+ with c_a:
462
+ model_a = st.selectbox(
463
+ "Model A (y-axis)", options=options, index=default_a, key=f"a_{key}"
464
+ )
465
+ with c_b:
466
+ b_options = [m for m in options if m != model_a]
467
+ default_b = b_options.index("AlphaGenome") if "AlphaGenome" in b_options else 0
468
+ model_b = st.selectbox(
469
+ "Model B (x-axis)", options=b_options, index=default_b, key=f"b_{key}"
470
+ )
471
 
472
+ scatter_df = build_pairwise_scatter_df(df, model_a, model_b)
473
+ if scatter_df.empty:
474
+ st.info(f"{model_a} and {model_b} share no evaluated track.")
475
+ return
476
+
477
+ n_tracks = len(scatter_df)
478
+ n_species = scatter_df["species"].nunique()
479
+ if per_species:
480
+ scatter_df = aggregate_pairs_by_species(scatter_df)
481
+ wins = int((scatter_df["Model A"] > scatter_df["Model B"]).sum())
482
+ st.caption(
483
+ f"Compared on **n = {n_species} shared species** ({n_tracks} shared "
484
+ f"tracks). {model_a} is higher on {wins} of them."
485
+ )
486
+ color_col, hover = "Clade", {"Shared tracks": True}
487
  else:
488
+ wins = int((scatter_df["Model A"] > scatter_df["Model B"]).sum())
489
+ st.caption(
490
+ f"Compared on **n = {n_tracks} shared tracks** over {n_species} "
491
+ f"species. {model_a} is higher on {wins} of them."
 
 
 
 
 
 
 
492
  )
493
+ color_col, hover = "assay_type", {"Species": True, "datasets": True}
494
+
495
+ min_v = float(min(scatter_df["Model A"].min(), scatter_df["Model B"].min()))
496
+ max_v = float(max(scatter_df["Model A"].max(), scatter_df["Model B"].max()))
497
+ pad = 0.05 * (max_v - min_v if max_v > min_v else 1.0)
498
+ axis_range = [min_v - pad, max_v + pad]
499
+
500
+ color_map = ASSAY_COLORS if color_col == "assay_type" else CLADE_COLORS
501
+ fig = px.scatter(
502
+ scatter_df,
503
+ x="Model B",
504
+ y="Model A",
505
+ color=color_col,
506
+ color_discrete_map=color_map,
507
+ category_orders={"Clade": list(SPECIES_GROUPS)},
508
+ hover_name="Track",
509
+ hover_data={**hover, "Model A": ":.3f", "Model B": ":.3f"},
510
+ text="Species" if per_species else None,
 
 
511
  )
512
+ if per_species:
513
+ fig.update_traces(marker=dict(size=13), textposition="top center")
514
+ fig.add_shape(
515
+ type="line",
516
+ x0=axis_range[0], y0=axis_range[0], x1=axis_range[1], y1=axis_range[1],
517
+ xref="x", yref="y",
518
+ line=dict(color="red", dash="dot", width=2),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
519
  )
520
+ fig.update_layout(
521
+ height=620,
522
+ margin=dict(l=60, r=20, t=20, b=110),
523
+ xaxis=dict(
524
+ title=f"{model_b} — {METRIC_LABEL}",
525
+ range=axis_range, dtick=0.1, tickformat=".1f", constrain="domain",
526
+ ),
527
+ yaxis=dict(
528
+ title=f"{model_a} — {METRIC_LABEL}",
529
+ range=axis_range, dtick=0.1, tickformat=".1f",
530
+ scaleanchor="x", scaleratio=1, constrain="domain",
531
+ ),
532
+ plot_bgcolor="rgba(0,0,0,0)",
533
+ paper_bgcolor="rgba(0,0,0,0)",
534
+ legend=dict(
535
+ title="",
536
+ orientation="h",
537
+ x=0.5, xanchor="center", y=-0.15, yanchor="top",
538
+ ),
539
  )
540
+ st.plotly_chart(fig, width="stretch")
541
 
 
 
 
 
 
 
 
 
 
 
 
 
542
 
543
+ def build_convergence_df(tracks: pd.DataFrame, runs: pd.DataFrame, x_col: str) -> pd.DataFrame:
544
+ """Per model: mean score and the mean cost of reaching its best checkpoint."""
545
+ perf = tracks.groupby("Model", as_index=False)["Score"].mean()
546
+ cost = runs.groupby("Model", as_index=False).agg(X=(x_col, "mean"), Runs=(x_col, "count"))
547
+ out = perf.merge(cost, on="Model").dropna(subset=["X"])
548
+ out["Score"] = out["Score"].round(3)
549
+ return out
 
 
550
 
 
 
 
 
551
 
552
+ def show_convergence(tracks: pd.DataFrame, runs: pd.DataFrame, key: str) -> None:
553
+ st.markdown("#### ⏱️ Time to convergence")
554
+ x_mode = st.selectbox(
555
+ "X-axis",
556
+ options=["GPU hours", "Tokens (billions)"],
557
+ index=0,
558
+ key=f"conv_{key}",
559
+ )
560
+ conv_df = build_convergence_df(tracks, runs, x_mode)
561
+ if conv_df.empty:
562
+ st.info("No convergence data for this species.")
563
+ return
564
+ if x_mode == "GPU hours":
565
+ caption = (
566
+ "GPU hours spent in training steps until the best validation checkpoint "
567
+ "(the one scored on the test set), × GPUs used, averaged over seeds. Job "
568
+ "setup and validation are excluded. All runs on H100s."
569
  )
570
+ else:
571
+ caption = (
572
+ "Training tokens seen at the best validation checkpoint, averaged over "
573
+ "seeds. Validation ran every 1.05B tokens on most runs, so this is "
574
+ "resolved to about 1B tokens."
575
+ )
576
+ st.caption(caption)
577
+ fig = px.scatter(
578
+ conv_df,
579
+ x="X",
580
+ y="Score",
581
+ text="Model",
582
+ color="Model",
583
+ color_discrete_map=MODEL_COLORS,
584
+ hover_data={"Model": True, "X": ":.2f", "Score": ":.3f", "Runs": True},
585
+ )
586
+ fig.update_traces(marker=dict(size=14), textposition="middle right")
587
+ fig.update_layout(
588
+ height=620,
589
+ xaxis_title=x_mode + (" (log scale)" if x_mode == "GPU hours" else ""),
590
+ yaxis_title=METRIC_LABEL,
591
+ plot_bgcolor="rgba(0,0,0,0)",
592
+ paper_bgcolor="rgba(0,0,0,0)",
593
+ showlegend=False,
594
+ )
595
+ if x_mode == "GPU hours":
596
+ lo, hi = np.log10(conv_df["X"].min()), np.log10(conv_df["X"].max())
597
+ fig.update_xaxes(
598
+ type="log", dtick=1, range=[lo - 0.15, hi + 0.45],
599
+ minor=dict(ticks="", showgrid=False),
600
+ )
601
+ else:
602
+ lo, hi = conv_df["X"].min(), conv_df["X"].max()
603
+ fig.update_xaxes(range=[lo - 0.05 * (hi - lo), hi + 0.3 * (hi - lo)])
604
+ st.plotly_chart(fig, width="stretch")
605
+
606
+
607
+ def plot_violin(df: pd.DataFrame) -> go.Figure:
608
+ order = df.groupby("Model")["Score"].median().sort_values().index.tolist()
609
+ fig = px.violin(
610
+ df,
611
+ x="Model",
612
+ y="Score",
613
+ color="Model",
614
+ color_discrete_map=MODEL_COLORS,
615
+ box=True,
616
+ points=False,
617
+ category_orders={"Model": order},
618
+ hover_data=["track_name_clean"],
619
+ )
620
+ fig.update_layout(
621
+ height=650,
622
+ xaxis_title="",
623
+ yaxis_title=METRIC_LABEL,
624
+ plot_bgcolor="rgba(0,0,0,0)",
625
+ paper_bgcolor="rgba(0,0,0,0)",
626
+ showlegend=False,
627
+ )
628
+ fig.update_traces(box_visible=True, meanline_visible=False)
629
+ return fig
630
 
631
 
632
  # ---------------------------------------------------------------------
633
+ # Pages
634
  # ---------------------------------------------------------------------
635
 
636
 
637
+ SIDEBAR_CSS = """
638
+ <style>
639
+ [data-testid="stSidebarNav"] a, [data-testid="stSidebarNavLink"] {
640
+ padding-top: 0.35rem; padding-bottom: 0.35rem; border-radius: 0.5rem;
641
+ }
642
+ [data-testid="stSidebarNav"] a span, [data-testid="stSidebarNavLink"] span {
643
+ font-size: 1.08rem;
644
+ }
645
+ [data-testid="stNavSectionHeader"], [data-testid="stSidebarNav"] header {
646
+ font-size: 0.85rem; font-weight: 700; letter-spacing: 0.06em;
647
+ text-transform: uppercase; opacity: 0.7; margin-top: 0.8rem;
648
+ }
649
+ </style>
650
+ """
651
 
652
+
653
+ def overview_page() -> None:
654
+ df = _TRACKS
 
655
  st.title("🧬 NTv3 Benchmark")
656
  st.markdown(_INTRO)
657
  st.markdown(f"_Last updated: **{_LAST_UPDATED}**_")
658
+ st.divider()
659
 
660
+ st.subheader("📊 Every model on every species")
661
+ st.caption(
662
+ "One point per model and species: the mean over that species' tracks. "
663
+ "A model with no point on a species was not evaluated on it. Hover for values."
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
664
  )
665
+ scores = build_species_scores(df)
666
+ st.plotly_chart(plot_species_overview(scores, models_in(df)), width="stretch")
667
 
668
+ left, right = st.columns([1, 1], gap="large")
669
+ with left:
670
+ show_head_to_head(df, key="overview_track")
671
+ with right:
672
+ show_head_to_head(
673
+ df,
674
+ key="overview_species",
675
+ per_species=True,
676
+ title="#### ⚖️ Head-to-head (per species)",
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
677
  )
 
678
 
679
+
680
+ def species_page(species: str) -> None:
681
+ sub = _TRACKS[_TRACKS["species"] == species]
682
+ runs = _RUNS[(_RUNS["species"] == species) & _RUNS["Model"].isin(set(sub["Model"]))]
683
+ n_tracks = sub["datasets"].nunique()
684
+ common = SPECIES_COMMON_NAME.get(species)
685
+ st.title(f"{SPECIES_ICON[species]} {SPECIES_LABEL[species]}")
686
+ st.caption(
687
+ f"{SPECIES_CLADE[species]}{f' · {common}' if common else ''} · {n_tracks} tracks · "
688
+ f"{sub['Model'].nunique()} models, each evaluated on all {n_tracks} tracks."
689
  )
690
 
691
+ leaderboard = build_leaderboard(sub)
692
  col1, col2 = st.columns([1, 1])
 
693
  with col1:
694
  st.subheader("🏅 Leaderboard")
 
695
  st.write("\n")
696
  st.write("\n")
697
+ st.write("\n")
698
+ st.dataframe(leaderboard, width="stretch")
 
 
 
699
  with col2:
700
  st.subheader("📈 Mean score per model")
701
+ st.plotly_chart(plot_mean_bars(leaderboard), width="stretch")
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
702
 
703
+ breakdown_df = build_category_model_df(sub)
704
+ if breakdown_df["Category"].nunique() > 1:
705
+ st.subheader("🧪 Mean score by assay type")
706
+ st.plotly_chart(plot_breakdown_facets_sorted_models(breakdown_df), width="stretch")
707
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
708
  left, right = st.columns([1, 1], gap="large")
 
709
  with left:
710
+ show_head_to_head(sub, key=species)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
711
  with right:
712
+ show_convergence(sub, runs, key=species)
713
 
714
+ st.subheader("🎻 Performance across tracks")
715
+ st.plotly_chart(plot_violin(sub), width="stretch")
 
 
 
 
716
 
717
 
718
+ def make_species_page(species: str) -> st.Page:
719
+ def render() -> None:
720
+ species_page(species)
 
 
 
 
 
721
 
722
+ render.__name__ = f"page_{species.replace(' ', '_')}"
723
+ return st.Page(
724
+ render,
725
+ title=SPECIES_LABEL[species],
726
+ icon=SPECIES_ICON[species],
727
+ url_path=species.replace(" ", "-"),
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
728
  )
729
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
730
 
731
+ # ---------------------------------------------------------------------
732
+ # Streamlit UI
733
+ # ---------------------------------------------------------------------
734
+ def main():
735
+ st.markdown(SIDEBAR_CSS, unsafe_allow_html=True)
736
+ available = set(_TRACKS["species"])
737
+ sections = {
738
+ "": [st.Page(overview_page, title="Overview", icon="🌍", default=True)],
739
+ }
740
+ for clade, group in SPECIES_GROUPS.items():
741
+ pages = [make_species_page(s) for s in group if s in available]
742
+ if pages:
743
+ sections[clade] = pages
744
+ sections["Initial release (2025)"] = [
745
+ st.Page(initial_release.main, title="2025 leaderboard", icon="🗄️", url_path="initial-release")
746
+ ]
747
+ st.navigation(sections, position="sidebar").run()
748
 
749
 
750
  if __name__ == "__main__":