Feature Extraction
Transformers
Safetensors
code
ntv3_annotate
genomics
dna
gene-annotation
ntv3
long-range
custom_code
Instructions to use InstaDeepAI/NTv3_annot_mammals with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use InstaDeepAI/NTv3_annot_mammals with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="InstaDeepAI/NTv3_annot_mammals", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("InstaDeepAI/NTv3_annot_mammals", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
|
Download README.md from InstaDeepAI/NTv3_annot_mammals: direct link, hf CLI and curl.
- Browser
- Download file 6.4 kB
-
https://huggingface.co/InstaDeepAI/NTv3_annot_mammals/resolve/refs%2Fpr%2F1/README.md
- Command line
-
hf download hf://InstaDeepAI/NTv3_annot_mammals@refs/pr/1/README.md
-
curl -L -H "Authorization: Bearer $HF_TOKEN" -o README.md https://huggingface.co/InstaDeepAI/NTv3_annot_mammals/resolve/refs%2Fpr%2F1/README.md
6.4 kB
| library_name: transformers | |
| tags: | |
| - genomics | |
| - dna | |
| - gene-annotation | |
| - ntv3 | |
| - long-range | |
| license: other | |
| language: | |
| - code | |
| > [!WARNING] | |
| > **TEMPORARY FILE β DELETE BEFORE RELEASE.** | |
| > This README is a working record of the JAX β PyTorch port, written for the NTv3 | |
| > release team while the repository is being assembled. It is not the model card. | |
| > Replace it in full with the public model card before this repository is made | |
| > public. | |
| # NTv3 Annotate β mammals (staging) | |
| PyTorch conversion of `new_ntv3_700M_annotate_mammals_1mb`, the released mammalian | |
| gene-annotation model. 703,499,416 parameters across 511 tensors. Modelling code is | |
| not in this repository: `config.json` resolves it from | |
| [`InstaDeepAI/NTv3_pretrained`](https://huggingface.co/InstaDeepAI/NTv3_pretrained) | |
| through `auto_map`, so loading requires `trust_remote_code=True`. | |
| Source checkpoint: `s3://genomix-research-3e09d/checkpoints/ntv3_annotate/new_ntv3_700M_annotate_mammals_1mb` | |
| (MLflow run `958c8d60db6c46ad99c4492ef9b9deaf`, step 925089, β970 B tokens). | |
| ## Verification against the JAX reference | |
| Every number below is measured on the 1024 bp fixture the checkpoint bundle ships in | |
| its own `test_inputs` item (`sequence = "ACGT" * 256`), comparing the annotation head's | |
| `bed_tracks_logits` of shape `(1, 1024, 1, 15)`. | |
| | comparison | max abs | mean abs | decoded-class agreement | | |
| | --- | --- | --- | --- | | |
| | **torch fp32 vs JAX fp32** | 2.81e-05 | 4.16e-06 | **100%** | | |
| | JAX bf16 vs JAX fp32 β *the intrinsic cost of bf16* | 6.28e-02 | 1.47e-02 | 100% | | |
| | torch bf16 autocast vs JAX bf16 | 6.82e-02 | 1.01e-02 | 100% | | |
| | torch bf16 autocast + fp32 annotation head vs JAX bf16 | 6.22e-02 | 8.78e-03 | 100% | | |
| **Conclusion: the port is correct.** In fp32 the two implementations agree to 2.8e-05, | |
| which is fp32 accumulation noise across a 703M-parameter network. Under bf16 the port | |
| disagrees with JAX by 6.2β6.8e-02 β the *same magnitude* as bf16's own disagreement | |
| with fp32 (6.28e-02), so the conversion contributes no error of its own; the two | |
| frameworks simply round at different points. In all four settings the decoded class is | |
| identical at every one of the 1024 positions, and since a Viterbi decode consumes the | |
| argmax, **no setting above can change the emitted GFF3**. | |
| Intermediate activations were compared as well, and the trunk is exact: | |
| | tensor | max abs | relative | | |
| | --- | --- | --- | | |
| | `before_transformer_embedding` | 6.9e-06 | 0.00% | | |
| | `after_transformer_embedding` | 1.4e-05 | 0.00% | | |
| ### Precision guidance | |
| Weights ship in **fp32**. bf16 is a *compute* dtype, matching the JAX | |
| `build_bf16_compute_model` path, which lowers five trunk stages | |
| (`stem`, `down_convolution`, `up_convolution`, `transformer_qkvo`, `transformer_ffn`) | |
| and leaves parameters, norms, embeddings, the LM head and the annotation head in fp32: | |
| ```python | |
| with torch.autocast(device_type="cuda", dtype=torch.bfloat16): | |
| out = model(tokens, conditions_tokens=conds) | |
| ``` | |
| Do **not** use `torch_dtype="bfloat16"` or `model.to(torch.bfloat16)`. A whole-module | |
| cast raises `expected m1 and m2 to have the same dtype`: `RMSBatchNorm.forward` | |
| promotes its input to β₯fp32 and never casts back, `StandardizedConv1D` then follows | |
| the input dtype, and the first bare `nn.Linear` (`GQABlock.q_layer`) still holds bf16 | |
| weights. | |
| ## How the weights were converted | |
| 511 Flax variables β 477 `nnx.Param` plus 34 `nnx.BatchStat` (`RMSBatchNorm.var_ema`) β | |
| map one-to-one onto the torch `state_dict`, loaded with `strict=True`. Leaf renames are | |
| `kernel`/`embedding`/`scale` β `weight`, under a `model.` prefix. Layout rules: | |
| | rule | count | | |
| | --- | --- | | |
| | copy | 348 | | |
| | `transpose(1, 0)` | 108 | | |
| | `transpose(2, 1, 0)` | 55 | | |
| **The rule is keyed on the owning torch module, never on the tensor shape.** Seven | |
| kernels β `decoder_blocks.{0..6}.pointwise_unet_skip.proj.kernel` β are square (2048Β² | |
| down to 1024Β²) because each maps `C β C`. A shape-based rule cannot tell a transpose | |
| from a copy there, and copying them produced a model that loaded cleanly under | |
| `strict=True`, carried the correct parameter count, and *still agreed 100% on the | |
| annotation argmax* while the trunk was silently wrong from the first decoder block. | |
| Only a layer-by-layer comparison caught it. `embed_layer.weight` is square too and a | |
| copy is correct there, so the heuristic cannot be patched, only replaced. Census check: | |
| the model has 108 `Linear.weight` tensors; a converter that transposes 101 has this bug. | |
| ## Known gaps in this staging repository | |
| - **Tokenizer.** `vocab.json` here is the 16-token soft-masked vocabulary these models | |
| were trained on (`A T C G N` plus lowercase `a t c g n`), which differs from the | |
| 11-token vocabulary of the initial NTv3 release. `tokenizer_config.json` resolves the | |
| tokenizer class from `InstaDeepAI/NTv3_pretrained` through `auto_map`, so | |
| `AutoTokenizer.from_pretrained(..., trust_remote_code=True)` returns a 16-token | |
| tokenizer reading this repository's own `vocab.json`. | |
| - **No decoding code.** `config.json` carries the full HMM specification under | |
| `annotation` β 15 states, the transition grammar, motif rules, the U12 lane, the | |
| served crop of 0.75 and the 1,048,576 bp window β but the Viterbi decoder and GFF3 | |
| writer are not published here yet. Logits alone are not a gene annotation. | |
| - **Inputs are soft-masked.** Lowercase repeat-masked bases carry their own token ids; | |
| an uppercase-only assembly loses the repeat signal without raising. | |
| - **Window geometry is not recoverable from the weights.** `ntv3.keep_target_center_fraction` | |
| is set to the validated serving crop of **0.75**, so a default forward reproduces | |
| `jax_model/roundtrip_fixture.npz` exactly. Running at a different window produces a | |
| plausible but wrong annotation rather than an error. Note that | |
| `annotation.window.trained_keep_target_center_fraction` records 1.0 while | |
| `jax_model/hyperparameters.json` records 0.75 for the same field; that disagreement is | |
| unresolved and does not affect serving, since both permit 0.75. | |
| - **The model is unconditioned.** It carries no condition table, so `forward` takes an **empty** `conditions_tokens` list, the same as the other two annotation repositories. | |
| - **transformers 5.** `residual_scale` is stored as a scalar, as the model declares it; it was a one-element list before, which transformers 5 refused to load. | |