| --- |
| library_name: transformers |
| pipeline_tag: fill-mask |
| tags: |
| - genomics |
| - dna |
| - masked-lm |
| - ntv3 |
| - long-range |
| - base-model |
| license: other |
| language: |
| - code |
| model_parameter_count: 7692475 |
| --- |
| |
| > [!NOTE] |
| > **This repo holds the code for the initial NTv3 release (2025).** The 2025 checkpoints load their modelling code from here. The [revised NTv3 models](https://huggingface.co/collections/InstaDeepAI/nucleotide-transformer-v3-6ab15d5afd21c2a62b45117e) ship their own code in each repo. |
|
|
| # InstaDeepAI/ntv3_base_model |
|
|
| **Unified base model repository for NTv3 models.** |
|
|
| This repository contains shared modeling code used by both: |
| - **Pre-trained models** (masked language models) |
| - **Post-trained models** (conditioned multi-species models with functional genomics heads) |
|
|
| **Note:** This repo should not be used standalone. It provides modeling code that is referenced by individual model checkpoints via `trust_remote_code=True`. |
|
|
| ## Contents |
|
|
| | File | Purpose | |
| |------|---------| |
| | `configuration_ntv3_pretrained.py` | Config class: `Ntv3PreTrainedConfig` | |
| | `configuration_ntv3_posttrained.py` | Config classes: `DiscreteConditionedNTv3Config`, `NTv3PostTrainedConfig` | |
| | `modeling_ntv3_pretrained.py` | Pre-trained model: `NTv3PreTrained` | |
| | `modeling_ntv3_posttrained.py` | Post-trained model: `NTv3PostTrained` with conditioned towers and heads | |
| | `tokenization_ntv3.py` | Tokenizer: `NTv3Tokenizer` (DNA) | |
|
|
| ## Architecture |
|
|
| - U-Net style conv tower → Transformer stack → deconv tower → LM head |
| - Post-trained models add adaptive layer norms and multi-species prediction heads |
| - Tokenizer: character-level over A T C G N + specials (`<unk>` `<pad>` `<mask>` `<cls>` `<eos>` `<bos>`) |
|
|
