Spaces:
Running
Running
File size: 3,387 Bytes
4188d2d 326d57c 4188d2d dea7853 4188d2d dea7853 326d57c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 | ---
title: NTv3 Benchmark
emoji: π
colorFrom: red
colorTo: red
sdk: docker
app_port: 8501
tags:
- streamlit
pinned: false
short_description: NTv3 Benchmark
license: apache-2.0
---
# NTv3 Benchmark
A standardized comparison of sequence-to-function models on **155 functional-track
prediction tasks across 11 species**. Every model is given the same 32-kb input, the same
task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so
what is compared is the representation rather than the training recipe.
## What is measured
**Metric.** Pearson correlation on the held-out test set, per track, averaged over three
seeds where available.
**Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato.
Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*.
**Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription
factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene
expression (RNA-seq) and translation (Ribo-seq).
The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize,
which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent
from that post-training set. The remaining seven genomes were absent from post-training
entirely, so those species measure transfer to an unseen genome.
## Models
Ten models: two convolutional baselines trained from scratch, five pretrained genomic
foundation models, two pretrained sequence-to-function models, and NTv3.
| Model | Training |
| --- | --- |
| BPNet, Residual CNN | from scratch |
| HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained |
| Borzoi, AlphaGenome | pretrained sequence-to-function |
| **NTv3** | post-trained |
**AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds
checkpoint saw those test regions, and from its all-folds checkpoint on every other species.
## Reading the leaderboard
**Scores are never averaged across species.** Not every model was run on every species, so a
single cross-species number would compare models over different task sets. Pick a species in
the sidebar for its own leaderboard, where every model shown was evaluated on all of that
species' tracks.
## Contents
- `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying
the score and the training-cost columns the app reports. This is the data behind every
view; nothing is computed elsewhere.
- `src/streamlit_app.py`, the app.
## Links
- π **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised),
the genomes, splits and tracks to reproduce or extend the benchmark.
- 𧬠**NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3).
- ποΈ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space.
## Citation
`TBC`, to be added on publication of the NTv3 paper.
## License
The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.**
They are released under the InstaDeep Open Model Licence, which permits non-commercial use
only and also places conditions on model outputs. See each model repository for its terms.
|