Replace weights with the paper's final model (run 073017)

- pytorch_model.bin: 30-epoch from-scratch pretraining run reported in the
paper (global attention in every layer), fp32. Replaces the earlier upload
of the windowed local-global ablation variant.
- config.json: matches the training architecture (global_attn_every_n_layers=1,
local_attention=512); token-id fields corrected to match vocab.json.
- tokenizer_config.json: add missing 'k' (from_pretrained crashed without it).
- tokenizer.py: map U->T in _tokenize so RNA-alphabet input tokenizes
correctly (previously produced [UNK]); T-alphabet encoding unchanged.
- README: architecture note and weight-version note.

README.md

@@ -24,7 +24,9 @@ RNAElectra is a nucleotide-resolution RNA language model trained using an ELECTR
 - **Training Objective**: ELECTRA-style replaced-token detection  
 - **Resolution**: Single-nucleotide  
 - **Domain**: RNA and transcriptomic sequences  
-- **Architecture**: ModernBERT-style backbone adapted for nucleotide sequences  
+- **Architecture**: ModernBERT-style backbone adapted for nucleotide sequences (22 layers, hidden size 512, 16 heads, global self-attention in every layer)  
+
+> **Note (2026-08)**: the released weights correspond to the final model reported in the RNAElectra paper (30-epoch pretraining on the RNAcentral-derived corpus, full global attention in every layer). They replace an earlier upload of an ablation variant that used windowed local-global attention.
 
 RNAElectra focuses on efficient pre-training by learning to discriminate corrupted tokens rather than reconstruct them, leading to strong representations with improved training efficiency.
 

config.json

@@ -4,7 +4,7 @@
   ],
   "attention_bias": false,
   "attention_dropout": 0.0,
-  "bos_token_id": 50281,
+  "bos_token_id": 5,
   "classifier_activation": "gelu",
   "classifier_bias": false,
   "classifier_dropout": 0.0,
@@ -12,10 +12,9 @@
   "cls_token_id": 2,
   "decoder_bias": true,
   "deterministic_flash_attn": false,
-  "dtype": "float32",
   "embedding_dropout": 0.0,
-  "eos_token_id": 50282,
-  "global_attn_every_n_layers": 3,
+  "eos_token_id": 6,
+  "global_attn_every_n_layers": 1,
   "global_rope_theta": 10000,
   "hidden_activation": "gelu",
   "hidden_size": 512,
@@ -23,9 +22,9 @@
   "initializer_range": 0.02,
   "intermediate_size": 2048,
   "layer_norm_eps": 1e-12,
-  "local_attention": 128,
+  "local_attention": 512,
   "local_rope_theta": 1000,
-  "mask_token_id": 3,
+  "mask_token_id": 4,
   "max_position_embeddings": 8192,
   "mlp_bias": false,
   "mlp_dropout": 0.0,
@@ -34,13 +33,14 @@
   "norm_eps": 1e-12,
   "num_attention_heads": 16,
   "num_hidden_layers": 22,
-  "pad_token_id": 1,
+  "pad_token_id": 0,
   "repad_logits_with_grad": false,
-  "sep_token_id": 50282,
+  "sep_token_id": 3,
   "sparse_pred_ignore_index": -100,
   "sparse_prediction": false,
   "tie_word_embeddings": false,
-  "transformers_version": "4.57.3",
-  "unknown_token_id": 0,
+  "torch_dtype": "float32",
+  "transformers_version": "4.48.2",
+  "unknown_token_id": 1,
   "vocab_size": 27
-}
+}

pytorch_model.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:0c477cad751b23b02b49fbc1dd7e4339fc74191ca082bd5f05eb20d71bf385dc
-size 369289915
+oid sha256:1a969702e2fa804ade947ab3c221030653a0accbf7a9d1bec72585a9a0bf44e4
+size 369288059

tokenizer.py

@@ -110,7 +110,7 @@ class NucEL_Tokenizer(PreTrainedTokenizer):
         Returns:
             List of tokens.
         """
-        text = text.upper().strip()
+        text = text.upper().strip().replace('U', 'T')  # RNA U -> T (paper: {A,C,G,U} mapped to {A,C,G,T})
         tokens = [self.cls_token]
         i = 0
         

tokenizer_config.json

@@ -62,6 +62,7 @@
   "cls_token": "[CLS]",
   "eos_token": "[EOS]",
   "extra_special_tokens": {},
+  "k": 1,
   "mask_token": "[MASK]",
   "model_max_length": 1025,
   "pad_token": "[PAD]",