Spaces:
Running
Running
Replace the 2025 benchmark with the revised NTv3 benchmark, keeping the 2025 leaderboard as a page
326d57c verified | title: NTv3 Benchmark | |
| emoji: 🚀 | |
| colorFrom: red | |
| colorTo: red | |
| sdk: docker | |
| app_port: 8501 | |
| tags: | |
| - streamlit | |
| pinned: false | |
| short_description: NTv3 Benchmark | |
| license: apache-2.0 | |
| # NTv3 Benchmark | |
| A standardized comparison of sequence-to-function models on **155 functional-track | |
| prediction tasks across 11 species**. Every model is given the same 32-kb input, the same | |
| task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so | |
| what is compared is the representation rather than the training recipe. | |
| ## What is measured | |
| **Metric.** Pearson correlation on the held-out test set, per track, averaged over three | |
| seeds where available. | |
| **Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato. | |
| Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*. | |
| **Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription | |
| factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene | |
| expression (RNA-seq) and translation (Ribo-seq). | |
| The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize, | |
| which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent | |
| from that post-training set. The remaining seven genomes were absent from post-training | |
| entirely, so those species measure transfer to an unseen genome. | |
| ## Models | |
| Ten models: two convolutional baselines trained from scratch, five pretrained genomic | |
| foundation models, two pretrained sequence-to-function models, and NTv3. | |
| | Model | Training | | |
| | --- | --- | | |
| | BPNet, Residual CNN | from scratch | | |
| | HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained | | |
| | Borzoi, AlphaGenome | pretrained sequence-to-function | | |
| | **NTv3** | post-trained | | |
| **AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds | |
| checkpoint saw those test regions, and from its all-folds checkpoint on every other species. | |
| ## Reading the leaderboard | |
| **Scores are never averaged across species.** Not every model was run on every species, so a | |
| single cross-species number would compare models over different task sets. Pick a species in | |
| the sidebar for its own leaderboard, where every model shown was evaluated on all of that | |
| species' tracks. | |
| ## Contents | |
| - `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying | |
| the score and the training-cost columns the app reports. This is the data behind every | |
| view; nothing is computed elsewhere. | |
| - `src/streamlit_app.py`, the app. | |
| ## Links | |
| - 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised), | |
| the genomes, splits and tracks to reproduce or extend the benchmark. | |
| - 🧬 **NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3). | |
| - 🗄️ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space. | |
| ## Citation | |
| `TBC`, to be added on publication of the NTv3 paper. | |
| ## License | |
| The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.** | |
| They are released under the InstaDeep Open Model Licence, which permits non-commercial use | |
| only and also places conditions on model outputs. See each model repository for its terms. | |