EBFM-OWT-160M

OpenEBFM 160M model (variant d12) trained on OWT. This repository contains a Hugging Face Transformers-compatible export for the pretraining checkpoint.

How to Use

Install the runtime dependencies:

pip install torch transformers

Load the tokenizer and model with the custom OpenEBFM code enabled:

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "guan-wang/EBFM-OWT-160M"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()
model.requires_grad_(False)
inputs = tokenizer("Hello world", return_tensors="pt").to(device)
outputs = model(**inputs)

print(outputs.logits.shape)

The first load may prompt you to review and allow the repository's custom model code. Loading remote Python code requires trust_remote_code=True.

Model Details

Detail	Value
Model family	OpenEBFM energy-based language model
Architecture	Custom OpenEBFM implementation (d12)
Parameter scale	160M
Training dataset	OWT
Training stage	pretraining
Transformer blocks	12
Embedding dimension	768
Attention heads	6
Context length	2048
Vocabulary size	32768

The parameter scale is the label used for this model in the OpenEBFM model configuration. The architecture and sequence settings above are read from the exported checkpoint configuration.

Files

• config.json: model configuration.
• model*.safetensors: model weights in the standard Transformers format.
• modeling_ebfm.py: remote model and tokenizer implementation.
• configuration_ebfm.py: remote configuration implementation.
• tokenizer.pkl: serialized EBFM tokenizer.
• tokenizer_config.json: tokenizer auto-loading configuration.
• token_bytes.pt: token byte table used by OpenEBFM metrics.

The code is maintained in the OpenEBFM repository.