---
library_name: keras-hub
---
### Model Overview
### Model Overview
ModernBERT is a family of encoder-only Transformer models published by [Answer.AI](https://www.answer.ai/) and [LightOn](https://www.lighton.ai/). It is a modernized drop-in replacement for BERT: pre-trained on 2 trillion tokens of English and code with a native 8,192-token context window, it uses Rotary Position Embeddings (RoPE), alternating local and global attention, and GeGLU feed-forward layers. ModernBERT is intended for classification, retrieval, and embedding of text, not for text-generation. See the model card below for benchmarks, data sources, and intended use cases.

Weights are released under the [Apache 2 License](https://github.com/AnswerDotAI/ModernBERT/blob/main/LICENSE). Keras model code is released under the [Apache 2 License](https://github.com/keras-team/keras-hub/blob/master/LICENSE).

## Links

* [ModernBERT Quickstart Notebook](https://www.kaggle.com/code/ruchithaxwf/modernbert-quickstart)
* [ModernBERT Model Card](https://huggingface.co/answerdotai/ModernBERT-base)
* [ModernBERT Model Paper](https://arxiv.org/abs/2412.13663)
* [KerasHub Beginner Guide](https://keras.io/guides/keras_hub/getting_started/)
* [KerasHub Model Publishing Guide](https://keras.io/guides/keras_hub/upload/)

## Installation

Keras and KerasHub can be installed with:

```
pip install -U -q keras-hub
pip install -U -q keras
```

Jax, TensorFlow, and Torch come preinstalled in Kaggle Notebooks. For instructions on installing them in another environment see the [Keras Getting Started](https://keras.io/getting_started/) page.

## Presets

The following model checkpoints are provided by the Keras team. Full code examples for each are available below.

| Preset name            | Parameters | Description                                                                                                                                            |
|------------------------|------------|--------------------------------------------------------------------------------------------------------------------------------------------------------|
| `modern_bert_base_en`  | 149.01M    | 22-layer ModernBERT Base encoder model pretrained on English for masked language modeling. Uses RoPE, alternating local and global attention, and GeGLU feedforward layers.  |
| `modern_bert_large_en` | 394.78M    | 28-layer ModernBERT Large encoder model pretrained on English for masked language modeling. Uses RoPE, alternating local and global attention, and GeGLU feedforward layers. |

## Example Usage
```python
import keras
import keras_hub
import numpy as np
```

Raw string data.

```python
features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]

# Pretrained classifier.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modern_bert_base_en",
    num_classes=4,
)
classifier.fit(x=features, y=labels, batch_size=2)
classifier.predict(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
classifier.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
classifier.backbone.trainable = False
# Fit again.
classifier.fit(x=features, y=labels, batch_size=2)
```

Preprocessed integer data.

```python
features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}
labels = [0, 3]

# Pretrained classifier without preprocessing.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modernbert_base_en",
    num_classes=4,
    preprocessor=None,
)
classifier.fit(x=features, y=labels, batch_size=2)
```

Masked language modeling.

```python
masked_lm = keras_hub.models.ModernBertMaskedLM.from_preset("modern_bert_base_en")
tokenizer = masked_lm.preprocessor.tokenizer

text = "The capital of France is [MASK]."
token_ids = (
    [tokenizer.cls_token_id]
    + list(np.asarray(tokenizer(text)))
    + [tokenizer.sep_token_id]
)
mask_positions = [
    i for i, t in enumerate(token_ids) if t == tokenizer.mask_token_id
]
logits = masked_lm(
    {
        "token_ids": np.array([token_ids], dtype="int32"),
        "padding_mask": np.ones((1, len(token_ids)), dtype="bool"),
        "mask_positions": np.array([mask_positions], dtype="int32"),
    }
)
top5 = np.argsort(-np.asarray(logits)[0, 0])[:5]
print([tokenizer.detokenize([int(t)]).strip() for t in top5])
```

## Example Usage with Hugging Face URI

```python
import keras
import keras_hub
import numpy as np
```

Raw string data.

```python
features = ["The quick brown fox jumped.", "I forgot my homework."]
labels = [0, 3]

# Pretrained classifier.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "hf://keras/modern_bert_base_en",
    num_classes=4,
)
classifier.fit(x=features, y=labels, batch_size=2)
classifier.predict(x=features, batch_size=2)

# Re-compile (e.g., with a new learning rate).
classifier.compile(
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
    optimizer=keras.optimizers.Adam(5e-5),
    jit_compile=True,
)
# Access backbone programmatically (e.g., to change `trainable`).
classifier.backbone.trainable = False
# Fit again.
classifier.fit(x=features, y=labels, batch_size=2)
```

Preprocessed integer data.

```python
features = {
    "token_ids": np.ones(shape=(2, 12), dtype="int32"),
    "padding_mask": np.array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0]] * 2),
}
labels = [0, 3]

# Pretrained classifier without preprocessing.
classifier = keras_hub.models.ModernBertTextClassifier.from_preset(
    "modernbert_base_en",
    num_classes=4,
    preprocessor=None,
)
classifier.fit(x=features, y=labels, batch_size=2)
```

Masked language modeling.

```python
masked_lm = keras_hub.models.ModernBertMaskedLM.from_preset("hf://keras/modern_bert_base_en")
tokenizer = masked_lm.preprocessor.tokenizer

text = "The capital of France is [MASK]."
token_ids = (
    [tokenizer.cls_token_id]
    + list(np.asarray(tokenizer(text)))
    + [tokenizer.sep_token_id]
)
mask_positions = [
    i for i, t in enumerate(token_ids) if t == tokenizer.mask_token_id
]
logits = masked_lm(
    {
        "token_ids": np.array([token_ids], dtype="int32"),
        "padding_mask": np.ones((1, len(token_ids)), dtype="bool"),
        "mask_positions": np.array([mask_positions], dtype="int32"),
    }
)
top5 = np.argsort(-np.asarray(logits)[0, 0])[:5]
print([tokenizer.detokenize([int(t)]).strip() for t in top5])
```