Request access to the NTv3 models

One approval grants access to every gated NTv3 model. Requests are reviewed manually.

The NTv3 weights are released under the InstaDeep Open Model Licence for non-commercial use only, and its terms also cover the models' outputs. For commercial use, contact legal@instadeep.com.

NTv3 pretrained

The pretrained backbone of Nucleotide Transformer v3 (NTv3): a genomic foundation model that reads DNA at single-base resolution across long context windows. Use it for sequence embeddings, for masked-token prediction, and as the starting point for your own fine-tuning.

This checkpoint is the backbone the other NTv3 capabilities are built on. The task-adapted models — functional-track prediction and gene annotation — start from these weights and resolve their modelling code from this repository.

Architecture

A U-Net around a transformer bottleneck. A convolutional tower reads the sequence and downsamples it by 128, a transformer tower carries long-range context at that reduced resolution, and a deconvolutional tower returns predictions at base resolution, with skip connections from the encoder at every scale.

The transformer uses grouped-query attention with per-head query, key and value normalisation, rotary position embeddings on a frequency schedule chosen for genomic distances, and a GeGLU feed-forward block.


Parameters	703,349,146
Resolution	1 bp
Input length	any multiple of 128
Vocabulary	11 tokens
Precision	float32

Usage

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo = "InstaDeepAI/NTv3_pretrained"
model = AutoModelForMaskedLM.from_pretrained(repo, trust_remote_code=True).eval()
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)

# The input length must be a multiple of 128.
sequence = "ACGT" * 256

# add_special_tokens=False: the model reads a genomic window, so one base is one
# token and nothing is prepended.
batch = tokenizer([sequence], return_tensors="pt", add_special_tokens=False)

with torch.no_grad():
    out = model(**batch)

out.embedding.shape  # (1, 1024, 1024)  one vector per base
out.logits.shape     # (1, 1024, 11)    masked-token prediction

AutoModel works too, and outputs are reachable both as attributes (out.logits) and as keys (out["logits"]).

trust_remote_code=True is required because the modelling code ships with the repository rather than in transformers.

An attention_mask is accepted and ignored: the model derives its own mask from the pad token at the resolution of its 128 bp blocks, so a per-token mask has nowhere to apply. Pad with the pad token rather than masking.

Outputs

Key	Shape	What it is
`embedding`	`(batch, length, 1024)`	Decoder output, one vector per base
`logits`	`(batch, length, 11)`	Masked-token prediction over the vocabulary
`after_transformer_embedding`	`(batch, length / 128, 2048)`	Trunk output, one vector per 128 bp block
`before_transformer_embedding`	`(batch, length / 128, 2048)`	Encoder output, before the transformer

Use skip_lm_head=True to skip the language-modelling head when you only need embeddings.

Input requirements

Two constraints, both of which fail quietly rather than loudly.

Length must be a multiple of 128. That is the total downsampling factor of the trunk. Pad or crop your window before calling the model.

The alphabet is uppercase. This checkpoint's vocabulary is A C G T N plus the special tokens; lowercase is not in it. Most reference assemblies are distributed soft-masked, with repeat regions in lowercase, and feeding one straight in turns nearly every base into <unk>. The model will still return numbers. Upper-case your sequence first, or use InstaDeepAI/NTv3_tracks, which reads a 16-symbol soft-masked alphabet.

Characters outside the vocabulary become <unk> and the sequence keeps its length.

Intended use

Embeddings and masked-token prediction on genomic DNA, and fine-tuning on your own task. The model was pretrained across a broad panel of species, so it is not specific to one genome.

Out of scope

Not a clinical or diagnostic tool, and not validated for variant interpretation in a medical setting. Not designed for RNA, protein, or non-genomic sequence. Predictions on sequence unlike anything in the pretraining distribution — synthetic constructs, assemblies from very distant clades — should be treated as unvalidated rather than as evidence.

Limitations

The model sees sequence only. It has no access to cell type, developmental stage, experimental condition, or any annotation, so nothing it outputs is conditioned on that context.

Masked-token prediction reflects the statistics of the training genomes. High confidence on a masked base means the surrounding sequence is predictive, not that the base is functionally important.

Accuracy varies by species and by genomic region, and is not uniform across the genome. Repetitive and low-complexity regions are harder, and this checkpoint cannot see the soft-masking that would tell it where they are.

Related models

Model	For
`NTv3_tracks`	Functional-track prediction, species-conditioned, from 16 kb to 1 Mb inputs
`NTv3_tracks_1mb`	The same tracks from 1 Mb inputs only: the specialist behind the NTv3 paper's main results
NTv3_annot: `_mammals` / `_plants` / `_insects`	Ab initio gene annotation, one model per clade

Licence

Released under the InstaDeep Open Model Licence. The licence restricts output as well as weights, so predictions made with this model carry the same terms. Read it before building on the model commercially.