File size: 4,325 Bytes
f31bc5d 608474e f31bc5d 608474e f31bc5d 8341b1e f31bc5d 8341b1e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d 608474e f31bc5d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | ---
pipeline_tag: fill-mask
license: apache-2.0
base_model: google-bert/bert-base-cased
library_name: zeromodels
tags:
- keras
- zeromodels
- bert
- cased
- fill-mask
- text-encoder
- arxiv:1810.04805
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/bert-6a8eae4f9ce86ca44e03bcc5) for all versions of BERT.***
# Run BERT with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/bert/) [](https://huggingface.co/collections/zeromodels/bert-6a8eae4f9ce86ca44e03bcc5)
# zeromodels/bert_base_cased
Paper: [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805)](https://arxiv.org/abs/1810.04805) · [HF Papers](https://huggingface.co/papers/1810.04805)
BERT is Google's bidirectional transformer text encoder, pretrained with masked LM and next-sentence prediction. WordPiece tokenizer; mask token `[MASK]`. Uncased variants lower-case the input; cased variants preserve case.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google-bert/bert-base-cased).
Pure-**Keras 3** conversion of [`google-bert/bert-base-cased`](https://huggingface.co/google-bert/bert-base-cased) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **fill-mask / encoder** checkpoint (`BertMaskedLM`, base cased). Task heads (sequence/token classify, QA, NSP, …) load via `hf:` fine-tunes.
## ✨ Quick start (fill-mask)
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.bert import BertMaskedLM, BertTokenizer
mlm = BertMaskedLM.from_weights("zeromodels/bert_base_cased")
tokenizer = BertTokenizer.from_weights("zeromodels/bert_base_cased")
inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.ids_to_tokens[int(logits[0, mask].argmax())])
```
Load any BERT variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Casing |
|---|---|---|
| `bert_base_uncased` | [`zeromodels/bert_base_uncased`](https://huggingface.co/zeromodels/bert_base_uncased) | uncased |
| `bert_large_uncased` | [`zeromodels/bert_large_uncased`](https://huggingface.co/zeromodels/bert_large_uncased) | uncased |
| `bert_base_cased` | [`zeromodels/bert_base_cased`](https://huggingface.co/zeromodels/bert_base_cased) | cased |
| `bert_large_cased` | [`zeromodels/bert_large_cased`](https://huggingface.co/zeromodels/bert_large_cased) | cased |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/bert_base_cased")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `BertModel` | Encoder backbone |
| `BertMaskedLM` | Masked language modeling (fill-mask) |
| `BertSequenceClassify` | Sequence classification |
| `BertTokenClassify` | Token classification (NER / POS) |
| `BertNextSentencePredict` | Next-sentence prediction |
| `BertQnA` | Extractive question answering |
| `BertMultipleChoice` | Multiple choice |
```python
from zeromodels.models.bert import BertSequenceClassify
model = BertSequenceClassify.from_weights("zeromodels/bert_base_cased")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `BertTokenizer.from_weights(...)` so WordPiece casing matches.
- Use `[MASK]` (not `<mask>`).
- See [BERT docs](https://imvision12.github.io/ZeroModels/bert/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `BertMaskedLM.from_weights("hf:google-bert/bert-base-cased")`.
## Special Thanks
A huge thank you to the Google BERT authors for creating and releasing these models.
License: Apache 2.0.
|