File size: 4,339 Bytes
8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab 697075e 8431fab 697075e 8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab 858ac4c 3d9fc6f 858ac4c 3d9fc6f 858ac4c 8431fab 3d9fc6f 8431fab 3d9fc6f 8431fab | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
pipeline_tag: fill-mask
license: apache-2.0
base_model: google/electra-base-generator
library_name: zeromodels
tags:
- keras
- zeromodels
- electra
- generator
- text-encoder
- fill-mask
- arxiv:2003.10555
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba) for all versions of ELECTRA.***
# Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/electra/) [](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba)
# zeromodels/electra_base_generator
Paper: [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)](https://arxiv.org/abs/2003.10555) · [HF Papers](https://huggingface.co/papers/2003.10555)
ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token **discriminator** (with a smaller **generator** producing the corrupted tokens). This repo is the **masked-LM (fill-mask)** checkpoint. WordPiece tokenizer; mask token `[MASK]`.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google/electra-base-generator).
Pure-**Keras 3** conversion of [`google/electra-base-generator`](https://huggingface.co/google/electra-base-generator) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
## ✨ Quick start (masked-LM (fill-mask))
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.electra import ElectraMaskedLM, ElectraTokenizer
mlm = ElectraMaskedLM.from_weights("zeromodels/electra_base_generator")
tokenizer = ElectraTokenizer.from_weights("zeromodels/electra_base_generator")
inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))
```
Load any ELECTRA variant the same way with `from_weights("zeromodels/<variant>")`:
| Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
|---|---|---|
| small | [`zeromodels/electra_small_discriminator`](https://huggingface.co/zeromodels/electra_small_discriminator) | [`zeromodels/electra_small_generator`](https://huggingface.co/zeromodels/electra_small_generator) |
| base | [`zeromodels/electra_base_discriminator`](https://huggingface.co/zeromodels/electra_base_discriminator) | [`zeromodels/electra_base_generator`](https://huggingface.co/zeromodels/electra_base_generator) |
| large | [`zeromodels/electra_large_discriminator`](https://huggingface.co/zeromodels/electra_large_discriminator) | [`zeromodels/electra_large_generator`](https://huggingface.co/zeromodels/electra_large_generator) |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/electra_base_generator")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `ElectraMaskedLM` | Masked language modeling (fill-mask) |
```python
from zeromodels.models.electra import ElectraMaskedLM
model = ElectraMaskedLM.from_weights("zeromodels/electra_base_generator")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `ElectraTokenizer.from_weights(...)` so WordPiece tokenization matches.
- Downstream tasks (classification / QA / NER) use the **discriminator** repos; the **generator** repos are the masked-LM.
- See [ELECTRA docs](https://imvision12.github.io/ZeroModels/electra/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `ElectraModel.from_weights("hf:google/electra-base-generator")`.
## Special Thanks
A huge thank you to the Google ELECTRA authors for creating and releasing these models.
License: Apache 2.0.
|