File size: 4,360 Bytes
a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa a0584f1 a6f3afa a0584f1 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa c7c6c32 a6f3afa | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 | ---
pipeline_tag: fill-mask
license: apache-2.0
base_model: google-bert/bert-large-uncased
library_name: zeromodels
tags:
- keras
- zeromodels
- bert
- uncased
- fill-mask
- text-encoder
- arxiv:1810.04805
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/bert-6a8eae4f9ce86ca44e03bcc5) for all versions of BERT.***
# Run BERT with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/bert/) [](https://huggingface.co/collections/zeromodels/bert-6a8eae4f9ce86ca44e03bcc5)
# zeromodels/bert_large_uncased
Paper: [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (arXiv:1810.04805)](https://arxiv.org/abs/1810.04805) · [HF Papers](https://huggingface.co/papers/1810.04805)
BERT is Google's bidirectional transformer text encoder, pretrained with masked LM and next-sentence prediction. WordPiece tokenizer; mask token `[MASK]`. Uncased variants lower-case the input; cased variants preserve case.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google-bert/bert-large-uncased).
Pure-**Keras 3** conversion of [`google-bert/bert-large-uncased`](https://huggingface.co/google-bert/bert-large-uncased) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **fill-mask / encoder** checkpoint (`BertMaskedLM`, large uncased). Task heads (sequence/token classify, QA, NSP, …) load via `hf:` fine-tunes.
## ✨ Quick start (fill-mask)
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.bert import BertMaskedLM, BertTokenizer
mlm = BertMaskedLM.from_weights("zeromodels/bert_large_uncased")
tokenizer = BertTokenizer.from_weights("zeromodels/bert_large_uncased")
inputs = tokenizer("the capital of france is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.ids_to_tokens[int(logits[0, mask].argmax())])
```
Load any BERT variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Casing |
|---|---|---|
| `bert_base_uncased` | [`zeromodels/bert_base_uncased`](https://huggingface.co/zeromodels/bert_base_uncased) | uncased |
| `bert_large_uncased` | [`zeromodels/bert_large_uncased`](https://huggingface.co/zeromodels/bert_large_uncased) | uncased |
| `bert_base_cased` | [`zeromodels/bert_base_cased`](https://huggingface.co/zeromodels/bert_base_cased) | cased |
| `bert_large_cased` | [`zeromodels/bert_large_cased`](https://huggingface.co/zeromodels/bert_large_cased) | cased |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/bert_large_uncased")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `BertModel` | Encoder backbone |
| `BertMaskedLM` | Masked language modeling (fill-mask) |
| `BertSequenceClassify` | Sequence classification |
| `BertTokenClassify` | Token classification (NER / POS) |
| `BertNextSentencePredict` | Next-sentence prediction |
| `BertQnA` | Extractive question answering |
| `BertMultipleChoice` | Multiple choice |
```python
from zeromodels.models.bert import BertSequenceClassify
model = BertSequenceClassify.from_weights("zeromodels/bert_large_uncased")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `BertTokenizer.from_weights(...)` so WordPiece casing matches.
- Use `[MASK]` (not `<mask>`).
- See [BERT docs](https://imvision12.github.io/ZeroModels/bert/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `BertMaskedLM.from_weights("hf:google-bert/bert-large-uncased")`.
## Special Thanks
A huge thank you to the Google BERT authors for creating and releasing these models.
License: Apache 2.0.
|