Spaces:
Running
Running
| title: NTv3 Benchmark | |
| emoji: 🚀 | |
| colorFrom: red | |
| colorTo: red | |
| sdk: docker | |
| app_port: 8501 | |
| tags: | |
| - streamlit | |
| pinned: false | |
| short_description: NTv3 Benchmark | |
| license: apache-2.0 | |
| # NTv3 Benchmark | |
| A standardized comparison of sequence-to-function models on **155 functional-track | |
| prediction tasks across 11 species**. Every model is given the same 32-kb input, the same | |
| task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so | |
| what is compared is the representation rather than the training recipe. | |
| ## What is measured | |
| **Metric.** Pearson correlation on the held-out test set, per track, averaged over three | |
| seeds where available. | |
| **Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato. | |
| Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*. | |
| **Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription | |
| factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene | |
| expression (RNA-seq) and translation (Ribo-seq). | |
| The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize, | |
| which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent | |
| from that post-training set. The remaining seven genomes were absent from post-training | |
| entirely, so those species measure transfer to an unseen genome. | |
| ## Models | |
| Ten models: two convolutional baselines trained from scratch, five pretrained genomic | |
| foundation models, two pretrained sequence-to-function models, and NTv3. | |
| | Model | Training | | |
| | --- | --- | | |
| | BPNet, Residual CNN | from scratch | | |
| | HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained | | |
| | Borzoi, AlphaGenome | pretrained sequence-to-function | | |
| | **NTv3** | post-trained | | |
| **AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds | |
| checkpoint saw those test regions, and from its all-folds checkpoint on every other species. | |
| ## Reading the leaderboard | |
| **Scores are never averaged across species.** Not every model was run on every species, so a | |
| single cross-species number would compare models over different task sets. Pick a species in | |
| the sidebar for its own leaderboard, where every model shown was evaluated on all of that | |
| species' tracks. | |
| ## Contents | |
| - `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying | |
| the score and the training-cost columns the app reports. This is the data behind every | |
| view; nothing is computed elsewhere. | |
| - `src/streamlit_app.py`, the app. | |
| ## Links | |
| - 📄 **Paper:** [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2025.12.22.695963). | |
| - 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised), | |
| the genomes, splits and tracks to reproduce or extend the benchmark. | |
| - 🧬 **NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3). | |
| - 🗄️ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space. | |
| ## Citation | |
| ```bibtex | |
| @article{ntv3, | |
| title = {A multispecies foundation model for genome function prediction, gene annotation and sequence design}, | |
| author = {Boshar, Sam and Evans, Benjamin and Tang, Ziqi and Adel, Yanis and Bornachot, Yohan and Sidbon, Shawn and Lorbeer, Franziska K. and Picard, Armand and Rajesh, Chandana and Seitz, Evan and Emms, David and Mendoza-Revilla, Javier and Hernandez, Ariana and Al-Ani, Fatimah and Karch, Tristan and Schiff, Yair and Lopez, Marie and Laterre, Alexandre and Beguir, Karim and Koo, Peter and Kuleshov, Volodymyr and Stark, Alexander and de Almeida, Bernardo P. and Pierrot, Thomas}, | |
| journal = {bioRxiv}, | |
| year = {2026}, | |
| doi = {10.64898/2025.12.22.695963}, | |
| url = {https://www.biorxiv.org/content/10.64898/2025.12.22.695963} | |
| } | |
| ``` | |
| ## License | |
| The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.** | |
| They are released under the InstaDeep Open Model Licence, which permits non-commercial use | |
| only and also places conditions on model outputs. See each model repository for its terms. | |