NTv3_annot_mammals / README.md
Yanisadel's picture
fix: store residual_scale as a scalar and drop the unused conditioning
38a6919 verified
|
Raw History Blame
6.4 kB
metadata
library_name: transformers
tags:
  - genomics
  - dna
  - gene-annotation
  - ntv3
  - long-range
license: other
language:
  - code

TEMPORARY FILE β€” DELETE BEFORE RELEASE. This README is a working record of the JAX β†’ PyTorch port, written for the NTv3 release team while the repository is being assembled. It is not the model card. Replace it in full with the public model card before this repository is made public.

NTv3 Annotate β€” mammals (staging)

PyTorch conversion of new_ntv3_700M_annotate_mammals_1mb, the released mammalian gene-annotation model. 703,499,416 parameters across 511 tensors. Modelling code is not in this repository: config.json resolves it from InstaDeepAI/NTv3_pretrained through auto_map, so loading requires trust_remote_code=True.

Source checkpoint: s3://genomix-research-3e09d/checkpoints/ntv3_annotate/new_ntv3_700M_annotate_mammals_1mb (MLflow run 958c8d60db6c46ad99c4492ef9b9deaf, step 925089, β‰ˆ970 B tokens).

Verification against the JAX reference

Every number below is measured on the 1024 bp fixture the checkpoint bundle ships in its own test_inputs item (sequence = "ACGT" * 256), comparing the annotation head's bed_tracks_logits of shape (1, 1024, 1, 15).

comparison max abs mean abs decoded-class agreement
torch fp32 vs JAX fp32 2.81e-05 4.16e-06 100%
JAX bf16 vs JAX fp32 β€” the intrinsic cost of bf16 6.28e-02 1.47e-02 100%
torch bf16 autocast vs JAX bf16 6.82e-02 1.01e-02 100%
torch bf16 autocast + fp32 annotation head vs JAX bf16 6.22e-02 8.78e-03 100%

Conclusion: the port is correct. In fp32 the two implementations agree to 2.8e-05, which is fp32 accumulation noise across a 703M-parameter network. Under bf16 the port disagrees with JAX by 6.2–6.8e-02 β€” the same magnitude as bf16's own disagreement with fp32 (6.28e-02), so the conversion contributes no error of its own; the two frameworks simply round at different points. In all four settings the decoded class is identical at every one of the 1024 positions, and since a Viterbi decode consumes the argmax, no setting above can change the emitted GFF3.

Intermediate activations were compared as well, and the trunk is exact:

tensor max abs relative
before_transformer_embedding 6.9e-06 0.00%
after_transformer_embedding 1.4e-05 0.00%

Precision guidance

Weights ship in fp32. bf16 is a compute dtype, matching the JAX build_bf16_compute_model path, which lowers five trunk stages (stem, down_convolution, up_convolution, transformer_qkvo, transformer_ffn) and leaves parameters, norms, embeddings, the LM head and the annotation head in fp32:

with torch.autocast(device_type="cuda", dtype=torch.bfloat16):
    out = model(tokens, conditions_tokens=conds)

Do not use torch_dtype="bfloat16" or model.to(torch.bfloat16). A whole-module cast raises expected m1 and m2 to have the same dtype: RMSBatchNorm.forward promotes its input to β‰₯fp32 and never casts back, StandardizedConv1D then follows the input dtype, and the first bare nn.Linear (GQABlock.q_layer) still holds bf16 weights.

How the weights were converted

511 Flax variables β€” 477 nnx.Param plus 34 nnx.BatchStat (RMSBatchNorm.var_ema) β€” map one-to-one onto the torch state_dict, loaded with strict=True. Leaf renames are kernel/embedding/scale β†’ weight, under a model. prefix. Layout rules:

rule count
copy 348
transpose(1, 0) 108
transpose(2, 1, 0) 55

The rule is keyed on the owning torch module, never on the tensor shape. Seven kernels β€” decoder_blocks.{0..6}.pointwise_unet_skip.proj.kernel β€” are square (2048Β² down to 1024Β²) because each maps C β†’ C. A shape-based rule cannot tell a transpose from a copy there, and copying them produced a model that loaded cleanly under strict=True, carried the correct parameter count, and still agreed 100% on the annotation argmax while the trunk was silently wrong from the first decoder block. Only a layer-by-layer comparison caught it. embed_layer.weight is square too and a copy is correct there, so the heuristic cannot be patched, only replaced. Census check: the model has 108 Linear.weight tensors; a converter that transposes 101 has this bug.

Known gaps in this staging repository

  • Tokenizer. vocab.json here is the 16-token soft-masked vocabulary these models were trained on (A T C G N plus lowercase a t c g n), which differs from the 11-token vocabulary of the initial NTv3 release. tokenizer_config.json resolves the tokenizer class from InstaDeepAI/NTv3_pretrained through auto_map, so AutoTokenizer.from_pretrained(..., trust_remote_code=True) returns a 16-token tokenizer reading this repository's own vocab.json.
  • No decoding code. config.json carries the full HMM specification under annotation β€” 15 states, the transition grammar, motif rules, the U12 lane, the served crop of 0.75 and the 1,048,576 bp window β€” but the Viterbi decoder and GFF3 writer are not published here yet. Logits alone are not a gene annotation.
  • Inputs are soft-masked. Lowercase repeat-masked bases carry their own token ids; an uppercase-only assembly loses the repeat signal without raising.
  • Window geometry is not recoverable from the weights. ntv3.keep_target_center_fraction is set to the validated serving crop of 0.75, so a default forward reproduces jax_model/roundtrip_fixture.npz exactly. Running at a different window produces a plausible but wrong annotation rather than an error. Note that annotation.window.trained_keep_target_center_fraction records 1.0 while jax_model/hyperparameters.json records 0.75 for the same field; that disagreement is unresolved and does not affect serving, since both permit 0.75.
  • The model is unconditioned. It carries no condition table, so forward takes an empty conditions_tokens list, the same as the other two annotation repositories.
  • transformers 5. residual_scale is stored as a scalar, as the model declares it; it was a one-element list before, which transformers 5 refused to load.