File size: 4,349 Bytes
8f7efd8 887b484 8f7efd8 887b484 8f7efd8 3e4494b 8f7efd8 3e4494b 8f7efd8 887b484 8f7efd8 887b484 8f7efd8 887b484 8f7efd8 887b484 8f7efd8 887b484 8f7efd8 887b484 8f7efd8 76773f9 887b484 76773f9 887b484 76773f9 8f7efd8 887b484 8f7efd8 887b484 8f7efd8 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 | ---
pipeline_tag: fill-mask
license: apache-2.0
base_model: google/electra-small-generator
library_name: zeromodels
tags:
- keras
- zeromodels
- electra
- generator
- text-encoder
- fill-mask
- arxiv:2003.10555
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba) for all versions of ELECTRA.***
# Run ELECTRA with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/electra/) [](https://huggingface.co/collections/zeromodels/electra-6a8eadf9dc472c12a679ebba)
# zeromodels/electra_small_generator
Paper: [ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators (arXiv:2003.10555)](https://arxiv.org/abs/2003.10555) · [HF Papers](https://huggingface.co/papers/2003.10555)
ELECTRA is Google's BERT-style bidirectional text encoder, pre-trained as a replaced-token **discriminator** (with a smaller **generator** producing the corrupted tokens). This repo is the **masked-LM (fill-mask)** checkpoint. WordPiece tokenizer; mask token `[MASK]`.
For more details on the model, please go to the upstream [model card](https://huggingface.co/google/electra-small-generator).
Pure-**Keras 3** conversion of [`google/electra-small-generator`](https://huggingface.co/google/electra-small-generator) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
## ✨ Quick start (masked-LM (fill-mask))
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.electra import ElectraMaskedLM, ElectraTokenizer
mlm = ElectraMaskedLM.from_weights("zeromodels/electra_small_generator")
tokenizer = ElectraTokenizer.from_weights("zeromodels/electra_small_generator")
inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))
```
Load any ELECTRA variant the same way with `from_weights("zeromodels/<variant>")`:
| Size | Discriminator (encoder / downstream) | Generator (masked-LM) |
|---|---|---|
| small | [`zeromodels/electra_small_discriminator`](https://huggingface.co/zeromodels/electra_small_discriminator) | [`zeromodels/electra_small_generator`](https://huggingface.co/zeromodels/electra_small_generator) |
| base | [`zeromodels/electra_base_discriminator`](https://huggingface.co/zeromodels/electra_base_discriminator) | [`zeromodels/electra_base_generator`](https://huggingface.co/zeromodels/electra_base_generator) |
| large | [`zeromodels/electra_large_discriminator`](https://huggingface.co/zeromodels/electra_large_discriminator) | [`zeromodels/electra_large_generator`](https://huggingface.co/zeromodels/electra_large_generator) |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/electra_small_generator")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `ElectraMaskedLM` | Masked language modeling (fill-mask) |
```python
from zeromodels.models.electra import ElectraMaskedLM
model = ElectraMaskedLM.from_weights("zeromodels/electra_small_generator")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `ElectraTokenizer.from_weights(...)` so WordPiece tokenization matches.
- Downstream tasks (classification / QA / NER) use the **discriminator** repos; the **generator** repos are the masked-LM.
- See [ELECTRA docs](https://imvision12.github.io/ZeroModels/electra/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `ElectraModel.from_weights("hf:google/electra-small-generator")`.
## Special Thanks
A huge thank you to the Google ELECTRA authors for creating and releasing these models.
License: Apache 2.0.
|