Support Transformers 5

INFERENCE.md

@@ -75,7 +75,7 @@ Predicted calls are `-1` at excluded special-token positions; references use `-1
 
 `load_data.py` reads the already-tokenized inputs without adding BOS/EOS tokens. Raw signed labels are converted to binary with `!= 0`, concatenated forward then reverse, and masked at special token positions. Its token IDs match the model tokenizer. To tokenize raw DNA separately, use uppercase sequence and `add_special_tokens=False`; this script itself accepts packed HDF5 only.
 
-The model requires `trust_remote_code=True` to load its custom architecture. The requirements pin Transformers to the tested version, 4.56.0. Apache-2.0 notices in the model code are retained.
+The model requires `trust_remote_code=True` to load its custom architecture. It supports Transformers 4.56 and later, including 5.x. Apache-2.0 notices in the model code are retained.
 
 ## Inference check
 

README.md

@@ -27,18 +27,48 @@ A DNA annotation model from the **Carbon** family.
 - **Inputs:** genomic DNA. The examples support GenBank, FASTA, and packed HDF5 files.
 - **Outputs:** per-base probabilities for non-coding and CDS on each strand; binary calls use a configurable threshold, default **0.5**.
 - **Training data:** RefSeq GCF assemblies from mammals, other vertebrates, invertebrates, plants, fungi, and protozoa.
-- **Transformers interface:** `AutoModelForTokenClassification` with `trust_remote_code=True`.
+- **Transformers interface:** `AutoModelForTokenClassification` with `trust_remote_code=True`; Transformers 4.56 or later, including 5.x.
 - **Inference:** PyTorch SDPA, with BF16 on supported GPUs. A separate FlashAttention installation is not required.
 
 ## How to use
 
-Download this repository, install its requirements in a Python 3.10+ environment with CUDA-enabled PyTorch, and authenticate with your Hugging Face account:
+Use a Python 3.10+ environment with CUDA-enabled PyTorch. Authenticate with your Hugging Face account, download the example scripts, and install their requirements:
 
 ```bash
-pip install -r requirements.txt
+pip install -U "huggingface_hub>=0.34"
 hf auth login
+hf download HuggingFaceBio/Carbon-A-1.2B --exclude model.safetensors --local-dir Carbon-A-1.2B
+cd Carbon-A-1.2B
+pip install -r requirements.txt
 ```
 
+The scripts load the weights through `from_pretrained`, which downloads them (4.6 GB) to the Hugging Face cache on first use, so they are excluded above.
+
+Full-length 98,304-base windows rely on PyTorch's fused SDPA kernels (FlashAttention or memory-efficient attention), which recent NVIDIA GPUs provide. Without them, PyTorch falls back to a kernel that materializes 16,384 × 16,384 attention matrices and needs more than 32 GB of additional GPU memory.
+
+### Use from Python
+
+```python
+import torch
+from transformers import AutoModelForTokenClassification, AutoTokenizer
+
+model_id = "HuggingFaceBio/Carbon-A-1.2B"
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForTokenClassification.from_pretrained(
+    model_id, trust_remote_code=True, dtype=torch.bfloat16, attn_implementation="sdpa"
+).to("cuda").eval()
+
+sequence = "ATGTCTGCTACACAAAAGCAACACTAA"  # up to 98,304 bases
+sequence = sequence.upper()
+sequence += "N" * (-len(sequence) % 6)  # complete the final 6-mer
+inputs = tokenizer(sequence, add_special_tokens=False, return_tensors="pt").to("cuda")
+with torch.inference_mode():
+    logits = model(**inputs).logits  # (1, 2 * len(sequence), 2)
+cds_plus, cds_minus = logits.float().softmax(-1)[0, :, 1].reshape(2, -1).cpu()
+```
+
+Logits are per base: all forward-strand positions, then all reverse-strand positions, both in input coordinates. Positions covered by a 6-mer containing `N` or another ambiguous base are not meaningful; the example scripts exclude them. Pass `attention_mask`, which the tokenizer returns, because the model requires it. Uppercase the sequence and use `add_special_tokens=False`: the tokenizer adds a BOS token by default, and lowercase (soft-masked) bases are not grouped into 6-mers, which shifts every later position.
+
 ### Annotate a GenBank or FASTA file
 
 The included [infer_genbank.py](infer_genbank.py) script reads the DNA sequence from each record and writes per-base predictions:

infer_packed.py

@@ -70,7 +70,7 @@ def main():
         parser.error('Output already exists; choose a new output directory or row range.')
     model = AutoModelForTokenClassification.from_pretrained(
         args.model, revision=args.revision, trust_remote_code=True,
-        torch_dtype=dtype, attn_implementation='sdpa',
+        dtype=dtype, attn_implementation='sdpa',
     ).to(args.device).eval()
     if (model.config.k, model.config.num_prediction_heads, model.config.num_labels) != (6, 2, 2):
         raise ValueError('Expected a 6-mer, two-strand binary CDS model.')

modeling_generanno.py

@@ -126,13 +126,34 @@ class GenerannoRotaryEmbedding(nn.Module):
             self.original_max_seq_len = config.max_position_embeddings
 
         self.config = config
-        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+        # Transformers v5 removed "default" from ROPE_INIT_FUNCTIONS; define it here as Llama does.
+        self.rope_init_fn = self.compute_default_rope_parameters
+        if self.rope_type != "default":
+            self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
 
         inv_freq, self.attention_scaling = self.rope_init_fn(
             self.config, device, **self.rope_kwargs
         )
         self.register_buffer("inv_freq", inv_freq, persistent=False)
-        self.original_inv_freq = self.inv_freq
+        self.register_buffer("original_inv_freq", inv_freq.clone(), persistent=False)
+
+    @staticmethod
+    def compute_default_rope_parameters(config=None, device=None, seq_len=None, **rope_kwargs):
+        """Unscaled RoPE inverse frequencies and attention scaling (unused, 1.0)."""
+        if config is not None:
+            base = config.rope_theta
+            dim = getattr(config, "head_dim", None) or config.hidden_size // config.num_attention_heads
+        else:
+            base, dim = rope_kwargs["base"], rope_kwargs["dim"]
+        inv_freq = 1.0 / (
+            base ** (torch.arange(0, dim, 2, dtype=torch.int64).to(device=device, dtype=torch.float) / dim)
+        )
+        return inv_freq, 1.0
+
+    def compute_inv_freq(self, device=None):
+        """Recompute inverse frequencies (non-persistent, so not restored from checkpoints)."""
+        inv_freq, _ = self.rope_init_fn(self.config, device, **self.rope_kwargs)
+        return inv_freq
 
     def _dynamic_frequency_update(self, position_ids, device):
         """
@@ -823,6 +844,13 @@ class GenerannoPreTrainedModel(PreTrainedModel):
             module.weight.data.normal_(mean=0.0, std=std)
             if module.padding_idx is not None:
                 module.weight.data[module.padding_idx].zero_()
+        elif isinstance(module, GenerannoRotaryEmbedding):
+            # Transformers v5 builds the model on the meta device and only loads checkpoint
+            # tensors, so non-persistent RoPE buffers must be recomputed here.
+            with torch.no_grad():
+                inv_freq = module.compute_inv_freq(device=module.inv_freq.device)
+                module.inv_freq.copy_(inv_freq)
+                module.original_inv_freq.copy_(inv_freq)
 
 
 class GenerannoModel(GenerannoPreTrainedModel):

provenance.json

@@ -10,10 +10,6 @@
       "sha256": "7a979aed9842232514880ffcf6d62192119ed94fde2cb53b9f9c9c5cb3b5b83b",
       "bytes": 10670
     },
-    "modeling_generanno.py": {
-      "sha256": "55433fb07920df4a18ba393f7276eba7859faca605deda97ac2dc8087f53a2b2",
-      "bytes": 54670
-    },
     "model.safetensors": {
       "sha256": "5e0a58184ac94be5c729b3d94464bb5fd400340c6ab4d2c63f4a3b97a850f71c",
       "bytes": 4614654504
@@ -47,5 +43,14 @@
     "inference-check.json",
     "genbank-inference-check.json",
     "INFERENCE.md"
-  ]
+  ],
+  "modified_source_files": {
+    "modeling_generanno.py": {
+      "source_sha256": "55433fb07920df4a18ba393f7276eba7859faca605deda97ac2dc8087f53a2b2",
+      "source_bytes": 54670,
+      "sha256": "d4ee34258c1a3bdad43ea3c3d687e4f210c758790859e0d671033b181809e0ef",
+      "bytes": 56240,
+      "change": "Transformers 5.x compatibility: default RoPE parameters and initialization of non-persistent RoPE buffers."
+    }
+  }
 }

requirements.txt

@@ -1,6 +1,6 @@
 torch>=2.7,<3
-transformers==4.56.0
-huggingface_hub>=0.34,<1
+transformers>=4.56,<6
+huggingface_hub>=0.34
 safetensors>=0.4.3
 numpy>=1.24
 h5py>=3.10