File size: 3,387 Bytes
4188d2d
326d57c
4188d2d
 
 
 
 
 
dea7853
4188d2d
dea7853
 
326d57c
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
---
title: NTv3 Benchmark
emoji: πŸš€
colorFrom: red
colorTo: red
sdk: docker
app_port: 8501
tags:
- streamlit
pinned: false
short_description: NTv3 Benchmark
license: apache-2.0
---

# NTv3 Benchmark

A standardized comparison of sequence-to-function models on **155 functional-track
prediction tasks across 11 species**. Every model is given the same 32-kb input, the same
task-specific 1-bp prediction head and the same per-nucleotide Poisson-multinomial loss, so
what is compared is the representation rather than the training recipe.

## What is measured

**Metric.** Pearson correlation on the held-out test set, per track, averaged over three
seeds where available.

**Species (11).** Animals: human, cattle, rat. Plants: *A. thaliana*, rice, maize, tomato.
Fungi: *S. cerevisiae*, *S. pombe*. Protists: *P. falciparum*, *D. discoideum*.

**Assays (7).** Chromatin accessibility (ATAC-seq), histone modifications and transcription
factor binding (ChIP-seq), transcription initiation (PRO-cap), RNA binding (eCLIP), gene
expression (RNA-seq) and translation (Ribo-seq).

The benchmark tests two kinds of generalization. For human, *A. thaliana*, rice and maize,
which NTv3-tracks saw during post-training, the assays and tracks were chosen to be absent
from that post-training set. The remaining seven genomes were absent from post-training
entirely, so those species measure transfer to an unseen genome.

## Models

Ten models: two convolutional baselines trained from scratch, five pretrained genomic
foundation models, two pretrained sequence-to-function models, and NTv3.

| Model | Training |
| --- | --- |
| BPNet, Residual CNN | from scratch |
| HyenaDNA, Caduceus, NTv2, Evo2, PlantCAD2 | pretrained |
| Borzoi, AlphaGenome | pretrained sequence-to-function |
| **NTv3** | post-trained |

**AlphaGenome** is fine-tuned from its fold-1 checkpoint on human, because the all-folds
checkpoint saw those test regions, and from its all-folds checkpoint on every other species.

## Reading the leaderboard

**Scores are never averaged across species.** Not every model was run on every species, so a
single cross-species number would compare models over different task sets. Pick a species in
the sidebar for its own leaderboard, where every model shown was evaluated on all of that
species' tracks.

## Contents

- `data/ntv3_benchmark_results.csv`, one row per model, species, track and seed, carrying
  the score and the training-cost columns the app reports. This is the data behind every
  view; nothing is computed elsewhere.
- `src/streamlit_app.py`, the app.

## Links

- πŸ“š **Benchmark dataset:** [`InstaDeepAI/NTv3_benchmark_dataset_revised`](https://huggingface.co/datasets/InstaDeepAI/NTv3_benchmark_dataset_revised),
  the genomes, splits and tracks to reproduce or extend the benchmark.
- 🧬 **NTv3 tutorials and models:** [`InstaDeepAI/ntv3`](https://huggingface.co/spaces/InstaDeepAI/ntv3).
- πŸ—„οΈ **Initial release (2025):** the leaderboard of the initial NTv3 release is the last page of this Space.

## Citation

`TBC`, to be added on publication of the NTv3 paper.

## License

The app code and the results table here are Apache-2.0. **The NTv3 model weights are not.**
They are released under the InstaDeep Open Model Licence, which permits non-commercial use
only and also places conditions on model outputs. See each model repository for its terms.