---
title: NTv3 Benchmark
emoji: 🚀
colorFrom: red
colorTo: red
sdk: docker
app_port: 8501
tags:
- streamlit
pinned: false
short_description: NTv3 Benchmark
license: apache-2.0
---

# NTv3 Benchmark

A standardized comparison of sequence-to-function models on **155 functional-track
prediction tasks across 11 species**. Every model is given the same 32-kb input, the same
task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so
what is compared is the representation rather than the training recipe.

## What is measured

**Metric.** Pearson correlation on the held-out test set, per track, averaged over three
seeds where available.

**Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato.
Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*.

**Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription
factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene
expression (RNA-seq) and translation (Ribo-seq).

The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize,
which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent
from that post-training set. The remaining seven genomes were absent from post-training
entirely, so those species measure transfer to an unseen genome.

## Models

Ten models: two convolutional baselines trained from scratch, five pretrained genomic
foundation models, two pretrained sequence-to-function models, and NTv3.

| Model | Training |
| --- | --- |
| BPNet, Residual CNN | from scratch |
| HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained |
| Borzoi, AlphaGenome | pretrained sequence-to-function |
| **NTv3** | post-trained |

**AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds
checkpoint saw those test regions, and from its all-folds checkpoint on every other species.

## Reading the leaderboard

**Scores are never averaged across species.** Not every model was run on every species, so a
single cross-species number would compare models over different task sets. Pick a species in
the sidebar for its own leaderboard, where every model shown was evaluated on all of that
species' tracks.

## Contents

- `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying
  the score and the training-cost columns the app reports. This is the data behind every
  view; nothing is computed elsewhere.
- `src/streamlit_app.py`, the app.

## Links

- 📚 **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised),
  the genomes, splits and tracks to reproduce or extend the benchmark.
- 🧬 **NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3).
- 🗄️ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space.

## Citation

`TBC`, to be added on publication of the NTv3 paper.

## License

The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.**
They are released under the InstaDeep Open Model Licence, which permits non-commercial use
only and also places conditions on model outputs. See each model repository for its terms.