Update README.md

README.md

@@ -15,57 +15,13 @@ tags:
 ModernGENA is a DNA foundation model based on a modernized BERT-style encoder architecture adapted for genomic sequence modeling.  
 **ModernGENA base** is the 135M-parameter version introduced in the paper *Back to BERT in 2026: ModernGENA as a Strong, Efficient Baseline for DNA Foundation Models*.
 
-Pretrained models:
-- [AIRI-Institute/moderngena-base](https://huggingface.co/AIRI-Institute/moderngena-base)
-- [AIRI-Institute/moderngena-large](https://huggingface.co/AIRI-Institute/moderngena-large)
 
-Training and fine-tuning code:
+Fine-tuning code:
 - [GENA_LM repository](https://github.com/AIRI-Institute/GENA_LM/tree/main/examples/modernGENA)
 
-## Key features
-
-Compared to earlier DNA BERT-style models, ModernGENA incorporates several modern Transformer improvements:
-
-- **RoPE positional embeddings** instead of absolute positional embeddings;
-- **Hybrid attention pattern** with alternating local sliding-window and global attention;
-- **Pre-norm Transformer blocks** for improved optimization stability;
-- **GeGLU feed-forward layers** for more expressive nonlinear transformations;
-- **Reduced bias parameterization** in most linear and normalization layers;
-- **Efficient long-context implementation** with unpadding, FlashAttention-compatible kernels, and `torch.compile`.
-
-These changes make ModernGENA a practical and reproducible baseline for studying DNA language modeling under modern training and inference setups.
-
-## Model description
-
-**ModernGENA base** is an encoder-only Transformer trained with masked language modeling (MLM) on genomic sequences.
-
-### Architecture
-
-- **Parameters:** 135M
-- **Number of layers:** 22
-- **Hidden size:** 768
-- **Intermediate size:** 1152
-- **Attention heads:** 12
-- **Vocabulary size:** 32k BPE
-- **Maximum sequence length during training:** up to 1024 tokens with dynamic packing
-- **Sliding window size:** 128
-- **Global attention every:** 3 layers
-
 ### Tokenization
 
-ModernGENA uses a **32k BPE vocabulary** built over DNA alphabet symbols `A/T/G/C/N`, with special tokens:
-
-- `[CLS]`
-- `[SEP]`
-- `[PAD]`
-- `[UNK]`
-- `[MASK]`
-
-As a preprocessing step, long runs of `N` are collapsed into a single token.
-
-## Training data
-
-The model was pretrained on a large multispecies vertebrate genomic corpus derived from **NCBI RefSeq** annotations available as of **December 2024**.
+ModernGENA uses a **32k BPE vocabulary** built over DNA alphabet symbols `A/T/G/C/N`, with special tokens (`[CLS]`, `[SEP]`, `[PAD]`, `[UNK]`, `[MASK]`)
 
 ### Pretraining corpus
 
@@ -80,57 +36,6 @@ To reduce overrepresentation of simple repeats and enrich biologically informati
 - overlapping intervals merged with BEDTools
 - both strands included for each resulting region
 
-## Training setup
-
-ModernGENA base was trained with masked language modeling on:
-
-- **8 × NVIDIA A100 80GB GPUs**
-- **Global batch size:** 4096 sequences
-- **Sequence lengths sampled dynamically from 10 to 1024 tokens**
-- **Average sequence length:** ~700 tokens
-- **Optimizer:** AdamW
-- **Learning rate:** `4e-4`
-- **Betas:** `(0.9, 0.98)`
-- **Epsilon:** `1e-6`
-- **Weight decay:** `1e-5`
-- **Warmup:** first `3e9` tokens, then constant LR
-
-### Training scale
-
-- **158 epochs**
-- **1,510B processed tokens**
-- **Observed pretraining throughput:** **1.33M tokens/s**
-
-## Evaluation
-
-ModernGENA was evaluated on the **Nucleotide Transformer benchmark** consisting of 18 genomic downstream tasks.
-
-Key results reported in the paper:
-
-- **ModernGENA base** ranks **first within the primary baseline set** among encoder-only Transformer models of comparable size;
-- it shows a strong **efficiency–quality trade-off**;
-- it achieves **higher inference throughput** than the main comparable baselines under FlashAttention-based inference settings.
-
-The primary comparison group includes:
-
-- DNABERT-2
-- GENA-LM
-- Nucleotide Transformer v2
-
-## Intended use
-
-ModernGENA base is intended for:
-
-- masked language modeling on DNA sequences;
-- extracting contextual embeddings for genomic regions;
-- fine-tuning on downstream genomics classification tasks such as:
-  - promoter prediction,
-  - enhancer prediction,
-  - histone mark prediction,
-  - splice-site related tasks.
-
-## Example usage
-
 ### Load pretrained model
 
 ```python