Fix Collection badge link to the current zeromodels collection slug

2bdb51b verified about 1 month ago

3.86 kB

pipeline_tag: fill-mask
license: mit
base_model: FacebookAI/roberta-base
library_name: zeromodels
tags:
  - keras
  - zeromodels
  - roberta
  - fill-mask
  - text-encoder
  - arxiv:1907.11692
  - pytorch
  - jax
  - tf

See our collection for all versions of RoBERTa.

Run RoBERTa with Keras 3: JAX, PyTorch, or TensorFlow

zeromodels/roberta_base

Paper: RoBERTa: A Robustly Optimized BERT Pretraining Approach (arXiv:1907.11692) · HF Papers

RoBERTa is a robustly optimized BERT encoder: more data/steps, no NSP, dynamic masking, byte-level BPE (mask token <mask>), and padding-offset position ids.

For more details on the model, please go to the upstream model card.

Pure-Keras 3 conversion of FacebookAI/roberta-base for zeromodels. One implementation runs unmodified on TensorFlow / Torch / JAX.

This is a fill-mask / encoder checkpoint (RobertaMaskedLM, base). Task heads load via hf: fine-tunes.

✨ Quick start (fill-mask)

import os
os.environ["KERAS_BACKEND"] = "torch"  # or "jax" / "tensorflow"

from zeromodels.models.roberta import RobertaMaskedLM, RobertaTokenizer

mlm = RobertaMaskedLM.from_weights("zeromodels/roberta_base")
tokenizer = RobertaTokenizer.from_weights("zeromodels/roberta_base")

inputs = tokenizer("The capital of France is <mask>.")
logits = mlm(inputs)  # (1, L, vocab_size)
mask = int((inputs["input_ids"][0] == tokenizer.mask_token_id).argmax())
print(tokenizer.decode([int(logits[0, mask].argmax())]))

Load any RoBERTa variant the same way with from_weights("zeromodels/<variant>"):

Variant	Hub
`roberta_base`	`zeromodels/roberta_base`
`roberta_large`	`zeromodels/roberta_large`

Available classes

Load any of these from this repo with from_weights("zeromodels/roberta_base") (or on the fly via the hf: prefix). The pretrained backbone is shared; task heads not stored in this checkpoint start randomly initialized, ready for fine-tuning (or load a hf: fine-tune).

Class	Task
`RobertaModel`	Encoder backbone
`RobertaMaskedLM`	Masked language modeling (fill-mask)
`RobertaSequenceClassify`	Sequence classification
`RobertaTokenClassify`	Token classification (NER / POS)
`RobertaQnA`	Extractive question answering
`RobertaMultipleChoice`	Multiple choice

from zeromodels.models.roberta import RobertaSequenceClassify
model = RobertaSequenceClassify.from_weights("zeromodels/roberta_base")

Tips

Set KERAS_BACKEND before importing Keras / zeromodels.
Prefer RobertaTokenizer.from_weights(...) so BPE vocab matches.
Use <mask> (not [MASK]).
See RoBERTa docs and Loading Weights.
Community / upstream safetensors still work via the hf: prefix, e.g. RobertaMaskedLM.from_weights("hf:FacebookAI/roberta-base").

Special Thanks

A huge thank you to the Facebook AI RoBERTa authors for creating and releasing these models.

License: MIT.