NTv3_annot_mammals / README.md
Yanisadel's picture
fix: store residual_scale as a scalar and drop the unused conditioning
38a6919 verified
|
Raw History Blame
6.4 kB
---
library_name: transformers
tags:
- genomics
- dna
- gene-annotation
- ntv3
- long-range
license: other
language:
- code
---
> [!WARNING]
> **TEMPORARY FILE β€” DELETE BEFORE RELEASE.**
> This README is a working record of the JAX β†’ PyTorch port, written for the NTv3
> release team while the repository is being assembled. It is not the model card.
> Replace it in full with the public model card before this repository is made
> public.
# NTv3 Annotate β€” mammals (staging)
PyTorch conversion of `new_ntv3_700M_annotate_mammals_1mb`, the released mammalian
gene-annotation model. 703,499,416 parameters across 511 tensors. Modelling code is
not in this repository: `config.json` resolves it from
[`InstaDeepAI/NTv3_pretrained`](https://huggingface.co/InstaDeepAI/NTv3_pretrained)
through `auto_map`, so loading requires `trust_remote_code=True`.
Source checkpoint: `s3://genomix-research-3e09d/checkpoints/ntv3_annotate/new_ntv3_700M_annotate_mammals_1mb`
(MLflow run `958c8d60db6c46ad99c4492ef9b9deaf`, step 925089, β‰ˆ970 B tokens).
## Verification against the JAX reference
Every number below is measured on the 1024 bp fixture the checkpoint bundle ships in
its own `test_inputs` item (`sequence = "ACGT" * 256`), comparing the annotation head's
`bed_tracks_logits` of shape `(1, 1024, 1, 15)`.
| comparison | max abs | mean abs | decoded-class agreement |
| --- | --- | --- | --- |
| **torch fp32 vs JAX fp32** | 2.81e-05 | 4.16e-06 | **100%** |
| JAX bf16 vs JAX fp32 β€” *the intrinsic cost of bf16* | 6.28e-02 | 1.47e-02 | 100% |
| torch bf16 autocast vs JAX bf16 | 6.82e-02 | 1.01e-02 | 100% |
| torch bf16 autocast + fp32 annotation head vs JAX bf16 | 6.22e-02 | 8.78e-03 | 100% |
**Conclusion: the port is correct.** In fp32 the two implementations agree to 2.8e-05,
which is fp32 accumulation noise across a 703M-parameter network. Under bf16 the port
disagrees with JAX by 6.2–6.8e-02 β€” the *same magnitude* as bf16's own disagreement
with fp32 (6.28e-02), so the conversion contributes no error of its own; the two
frameworks simply round at different points. In all four settings the decoded class is
identical at every one of the 1024 positions, and since a Viterbi decode consumes the
argmax, **no setting above can change the emitted GFF3**.
Intermediate activations were compared as well, and the trunk is exact:
| tensor | max abs | relative |
| --- | --- | --- |
| `before_transformer_embedding` | 6.9e-06 | 0.00% |
| `after_transformer_embedding` | 1.4e-05 | 0.00% |
### Precision guidance
Weights ship in **fp32**. bf16 is a *compute* dtype, matching the JAX
`build_bf16_compute_model` path, which lowers five trunk stages
(`stem`, `down_convolution`, `up_convolution`, `transformer_qkvo`, `transformer_ffn`)
and leaves parameters, norms, embeddings, the LM head and the annotation head in fp32:
```python
with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
out = model(tokens, conditions_tokens=conds)
```
Do **not** use `torch_dtype="bfloat16"` or `model.to(torch.bfloat16)`. A whole-module
cast raises `expected m1 and m2 to have the same dtype`: `RMSBatchNorm.forward`
promotes its input to β‰₯fp32 and never casts back, `StandardizedConv1D` then follows
the input dtype, and the first bare `nn.Linear` (`GQABlock.q_layer`) still holds bf16
weights.
## How the weights were converted
511 Flax variables β€” 477 `nnx.Param` plus 34 `nnx.BatchStat` (`RMSBatchNorm.var_ema`) β€”
map one-to-one onto the torch `state_dict`, loaded with `strict=True`. Leaf renames are
`kernel`/`embedding`/`scale` β†’ `weight`, under a `model.` prefix. Layout rules:
| rule | count |
| --- | --- |
| copy | 348 |
| `transpose(1, 0)` | 108 |
| `transpose(2, 1, 0)` | 55 |
**The rule is keyed on the owning torch module, never on the tensor shape.** Seven
kernels β€” `decoder_blocks.{0..6}.pointwise_unet_skip.proj.kernel` β€” are square (2048Β²
down to 1024Β²) because each maps `C β†’ C`. A shape-based rule cannot tell a transpose
from a copy there, and copying them produced a model that loaded cleanly under
`strict=True`, carried the correct parameter count, and *still agreed 100% on the
annotation argmax* while the trunk was silently wrong from the first decoder block.
Only a layer-by-layer comparison caught it. `embed_layer.weight` is square too and a
copy is correct there, so the heuristic cannot be patched, only replaced. Census check:
the model has 108 `Linear.weight` tensors; a converter that transposes 101 has this bug.
## Known gaps in this staging repository
- **Tokenizer.** `vocab.json` here is the 16-token soft-masked vocabulary these models
were trained on (`A T C G N` plus lowercase `a t c g n`), which differs from the
11-token vocabulary of the initial NTv3 release. `tokenizer_config.json` resolves the
tokenizer class from `InstaDeepAI/NTv3_pretrained` through `auto_map`, so
`AutoTokenizer.from_pretrained(..., trust_remote_code=True)` returns a 16-token
tokenizer reading this repository's own `vocab.json`.
- **No decoding code.** `config.json` carries the full HMM specification under
`annotation` β€” 15 states, the transition grammar, motif rules, the U12 lane, the
served crop of 0.75 and the 1,048,576 bp window β€” but the Viterbi decoder and GFF3
writer are not published here yet. Logits alone are not a gene annotation.
- **Inputs are soft-masked.** Lowercase repeat-masked bases carry their own token ids;
an uppercase-only assembly loses the repeat signal without raising.
- **Window geometry is not recoverable from the weights.** `ntv3.keep_target_center_fraction`
is set to the validated serving crop of **0.75**, so a default forward reproduces
`jax_model/roundtrip_fixture.npz` exactly. Running at a different window produces a
plausible but wrong annotation rather than an error. Note that
`annotation.window.trained_keep_target_center_fraction` records 1.0 while
`jax_model/hyperparameters.json` records 0.75 for the same field; that disagreement is
unresolved and does not affect serving, since both permit 0.75.
- **The model is unconditioned.** It carries no condition table, so `forward` takes an **empty** `conditions_tokens` list, the same as the other two annotation repositories.
- **transformers 5.** `residual_scale` is stored as a scalar, as the model declares it; it was a one-element list before, which transformers 5 refused to load.