| --- |
| pipeline_tag: fill-mask |
| license: apache-2.0 |
| base_model: google-bert/bert-base-uncased |
| library_name: zeromodels |
| tags: |
| - keras |
| - zeromodels |
| - bert |
| - uncased |
| - fill-mask |
| - text-encoder |
| - arxiv:1810.04805 |
| - pytorch |
| - jax |
| - tf |
| --- |
| |
| ## ***See [our collection](https://huggingface.co/collections/zeromodels/bert-6a8eae4f9ce86ca44e03bcc5) for all versions of BERT.*** |
|
|
| # Run BERT with Keras 3: JAX, PyTorch, or TensorFlow |
|
|
| [](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/bert/) [](https://huggingface.co/collections/zeromodels/bert-6a8eae4f9ce86ca44e03bcc5) |
|
|
| # zeromodels/bert_base_uncased |
|
|
| Paper: [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805)](https://arxiv.org/abs/1810.04805) · [HF Papers](https://huggingface.co/papers/1810.04805) |
|
|
| BERT is Google's bidirectional transformer text encoder, pretrained with masked LM and next-sentence prediction. WordPiece tokenizer; mask token `[MASK]`. Uncased variants lower-case the input; cased variants preserve case. |
|
|
| For more details on the model, please go to the upstream [model card](https://huggingface.co/google-bert/bert-base-uncased). |
|
|
| Pure-**Keras 3** conversion of [`google-bert/bert-base-uncased`](https://huggingface.co/google-bert/bert-base-uncased) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**. |
|
|
| This is a **fill-mask / encoder** checkpoint (`BertMaskedLM`, base uncased). Task heads (sequence/token classify, QA, NSP, …) load via `hf:` fine-tunes. |
|
|
| ## ✨ Quick start (fill-mask) |
|
|
| ```python |
| import os |
| os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow" |
| |
| from zeromodels.models.bert import BertMaskedLM, BertTokenizer |
| |
| mlm = BertMaskedLM.from_weights("zeromodels/bert_base_uncased") |
| tokenizer = BertTokenizer.from_weights("zeromodels/bert_base_uncased") |
| |
| inputs = tokenizer("the capital of france is [MASK].") |
| logits = mlm(inputs) # (1, L, vocab_size) |
| mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax()) |
| print(tokenizer.ids_to_tokens[int(logits[0, mask].argmax())]) |
| ``` |
|
|
| Load any BERT variant the same way with `from_weights("zeromodels/<variant>")`: |
|
|
| | Variant | Hub | Casing | |
| |---|---|---| |
| | `bert_base_uncased` | [`zeromodels/bert_base_uncased`](https://huggingface.co/zeromodels/bert_base_uncased) | uncased | |
| | `bert_large_uncased` | [`zeromodels/bert_large_uncased`](https://huggingface.co/zeromodels/bert_large_uncased) | uncased | |
| | `bert_base_cased` | [`zeromodels/bert_base_cased`](https://huggingface.co/zeromodels/bert_base_cased) | cased | |
| | `bert_large_cased` | [`zeromodels/bert_large_cased`](https://huggingface.co/zeromodels/bert_large_cased) | cased | |
|
|
| ## Available classes |
|
|
| Load any of these from this repo with `from_weights("zeromodels/bert_base_uncased")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune). |
|
|
| | Class | Task | |
| |---|---| |
| | `BertModel` | Encoder backbone | |
| | `BertMaskedLM` | Masked language modeling (fill-mask) | |
| | `BertSequenceClassify` | Sequence classification | |
| | `BertTokenClassify` | Token classification (NER / POS) | |
| | `BertNextSentencePredict` | Next-sentence prediction | |
| | `BertQnA` | Extractive question answering | |
| | `BertMultipleChoice` | Multiple choice | |
|
|
| ```python |
| from zeromodels.models.bert import BertSequenceClassify |
| model = BertSequenceClassify.from_weights("zeromodels/bert_base_uncased") |
| ``` |
|
|
| ## Tips |
|
|
| - Set `KERAS_BACKEND` **before** importing Keras / zeromodels. |
| - Prefer `BertTokenizer.from_weights(...)` so WordPiece casing matches. |
| - Use `[MASK]` (not `<mask>`). |
| - See [BERT docs](https://imvision12.github.io/ZeroModels/bert/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/). |
| - Community / upstream safetensors still work via the `hf:` prefix, e.g. `BertMaskedLM.from_weights("hf:google-bert/bert-base-uncased")`. |
|
|
| ## Special Thanks |
|
|
| A huge thank you to the Google BERT authors for creating and releasing these models. |
|
|
| License: Apache 2.0. |
|
|
