File size: 4,886 Bytes
08d783d 24db011 08d783d 24db011 08d783d 1ff5738 08d783d 1ff5738 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d 24db011 08d783d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | ---
pipeline_tag: fill-mask
license: mit
base_model: microsoft/deberta-base
library_name: zeromodels
tags:
- keras
- zeromodels
- deberta
- deberta-v1
- fill-mask
- text-encoder
- arxiv:2006.03654
- arxiv:2111.09543
- pytorch
- jax
- tf
---
## ***See [our collection](https://huggingface.co/collections/zeromodels/deberta-v1-v2-v3-6a8eae49464403784b9d6cd0) for all versions of DeBERTa (v1 / v2 / v3).***
# Run DeBERTa with Keras 3: JAX, PyTorch, or TensorFlow
[](https://github.com/IMvision12/ZeroModels) [](https://imvision12.github.io/ZeroModels/deberta/) [](https://huggingface.co/collections/zeromodels/deberta-v1-v2-v3-6a8eae49464403784b9d6cd0)
# zeromodels/deberta_base
Papers: [DeBERTa: Decoding-enhanced BERT with Disentangled Attention (arXiv:2006.03654)](https://arxiv.org/abs/2006.03654) · [DeBERTaV3 (arXiv:2111.09543)](https://arxiv.org/abs/2111.09543) · [HF Papers](https://huggingface.co/papers/2006.03654)
DeBERTa is Microsoft's disentangled-attention text encoder (content + relative position). v1 uses byte-level BPE; v2/v3 use SentencePiece. v3 adds ELECTRA-style pretraining with gradient-disentangled embedding sharing. Import from `deberta` / `deberta_v2` / `deberta_v3` to match the generation.
For more details on the model, please go to the upstream [model card](https://huggingface.co/microsoft/deberta-base).
Pure-**Keras 3** conversion of [`microsoft/deberta-base`](https://huggingface.co/microsoft/deberta-base) for [zeromodels](https://github.com/IMvision12/ZeroModels). One implementation runs unmodified on **TensorFlow / Torch / JAX**.
This is a **fill-mask / encoder** checkpoint (`DebertaMaskedLM`, v1 base). Task heads (sequence/token classify, QA, …) load via `hf:` fine-tunes.
## ✨ Quick start (fill-mask)
```python
import os
os.environ["KERAS_BACKEND"] = "torch" # or "jax" / "tensorflow"
from zeromodels.models.deberta import (
DebertaMaskedLM,
DebertaTokenizer,
)
mlm = DebertaMaskedLM.from_weights("zeromodels/deberta_base")
tokenizer = DebertaTokenizer.from_weights("zeromodels/deberta_base")
inputs = tokenizer("The capital of France is [MASK].")
logits = mlm(inputs) # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))
```
Load any DeBERTa variant the same way with `from_weights("zeromodels/<variant>")`:
| Variant | Hub | Generation |
|---|---|---|
| `deberta_base` | [`zeromodels/deberta_base`](https://huggingface.co/zeromodels/deberta_base) | v1 |
| `deberta_large` | [`zeromodels/deberta_large`](https://huggingface.co/zeromodels/deberta_large) | v1 |
| `deberta_v2_xlarge` | [`zeromodels/deberta_v2_xlarge`](https://huggingface.co/zeromodels/deberta_v2_xlarge) | v2 |
| `deberta_v2_xxlarge` | [`zeromodels/deberta_v2_xxlarge`](https://huggingface.co/zeromodels/deberta_v2_xxlarge) | v2 |
| `deberta_v3_xsmall` | [`zeromodels/deberta_v3_xsmall`](https://huggingface.co/zeromodels/deberta_v3_xsmall) | v3 |
| `deberta_v3_small` | [`zeromodels/deberta_v3_small`](https://huggingface.co/zeromodels/deberta_v3_small) | v3 |
| `deberta_v3_base` | [`zeromodels/deberta_v3_base`](https://huggingface.co/zeromodels/deberta_v3_base) | v3 |
| `deberta_v3_large` | [`zeromodels/deberta_v3_large`](https://huggingface.co/zeromodels/deberta_v3_large) | v3 |
## Available classes
Load any of these from this repo with `from_weights("zeromodels/deberta_base")` (or on the fly via the `hf:` prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a `hf:` fine-tune).
| Class | Task |
|---|---|
| `DebertaModel` | Encoder backbone |
| `DebertaMaskedLM` | Masked language modeling (fill-mask) |
| `DebertaSequenceClassify` | Sequence classification |
| `DebertaTokenClassify` | Token classification (NER / POS) |
| `DebertaQnA` | Extractive question answering |
```python
from zeromodels.models.deberta import DebertaSequenceClassify
model = DebertaSequenceClassify.from_weights("zeromodels/deberta_base")
```
## Tips
- Set `KERAS_BACKEND` **before** importing Keras / zeromodels.
- Prefer `Tokenizer.from_weights(...)` so vocab and mask token match.
- Do not mix packages across generations (v1 ≠ v2 ≠ v3).
- See [DeBERTa docs](https://imvision12.github.io/ZeroModels/deberta/) and [Loading Weights](https://imvision12.github.io/ZeroModels/loading_weights/).
- Community / upstream safetensors still work via the `hf:` prefix, e.g. `DebertaMaskedLM.from_weights("hf:microsoft/deberta-base")`.
## Special Thanks
A huge thank you to the Microsoft DeBERTa authors for creating and releasing these models.
License: MIT.
|