File size: 4,349 Bytes
2302a23 4168574 2302a23 4168574 2302a23 4a971dc 2302a23 4a971dc 2302a23 4168574 2302a23 4168574 2302a23 4168574 2302a23 4168574 2302a23 4168574 2302a23 4168574 2302a23 f4382b1 4168574 f4382b1 4168574 f4382b1 2302a23 4168574 2302a23 4168574 2302a23 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
pipeline_tag: fill-mask
license: apache-2.0
base_model: google/electra-large-generator
library_name: zeromodels
tags:
- keras
- zeromodels
- electra
- generator
- text-encoder
- fill-mask
- arxiv:2003.10555
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba) for all versions of ELECTRA.***
# Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/electra/) [](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba)
# zeromodels/electra_large_generator
Paper: [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)](https://arxiv.org/abs/2003.10555) · [HF Papers](https://huggingface.co/papers/2003.10555)
ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token **discriminator** (with a smaller **generator** producing the corrupted tokens). This repo is the **masked-LM (fill-mask)** checkpoint. WordPiece tokenizer; mask token `[MASK]`.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google/electra-large-generator).
Pure-**Keras 3** conversion of [`google/electra-large-generator`](https://huggingface.co/google/electra-large-generator) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
## ✨ Quick start (masked-LM (fill-mask))
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.electra import ElectraMaskedLM, ElectraTokenizer
mlm = ElectraMaskedLM.from_weights("zeromodels/electra_large_generator")
tokenizer = ElectraTokenizer.from_weights("zeromodels/electra_large_generator")
inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))
```
Load any ELECTRA variant the same way with `from_weights("zeromodels/<variant>")`:
| Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
|---|---|---|
| small | [`zeromodels/electra_small_discriminator`](https://huggingface.co/zeromodels/electra_small_discriminator) | [`zeromodels/electra_small_generator`](https://huggingface.co/zeromodels/electra_small_generator) |
| base | [`zeromodels/electra_base_discriminator`](https://huggingface.co/zeromodels/electra_base_discriminator) | [`zeromodels/electra_base_generator`](https://huggingface.co/zeromodels/electra_base_generator) |
| large | [`zeromodels/electra_large_discriminator`](https://huggingface.co/zeromodels/electra_large_discriminator) | [`zeromodels/electra_large_generator`](https://huggingface.co/zeromodels/electra_large_generator) |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/electra_large_generator")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `ElectraMaskedLM` | Masked language modeling (fill-mask) |
```python
from zeromodels.models.electra import ElectraMaskedLM
model = ElectraMaskedLM.from_weights("zeromodels/electra_large_generator")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `ElectraTokenizer.from_weights(...)` so WordPiece tokenization matches.
- Downstream tasks (classification / QA / NER) use the **discriminator** repos; the **generator** repos are the masked-LM.
- See [ELECTRA docs](https://imvision12.github.io/ZeroModels/electra/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `ElectraModel.from_weights("hf:google/electra-large-generator")`.
## Special Thanks
A huge thank you to the Google ELECTRA authors for creating and releasing these models.
License: Apache 2.0.
|