EBFM-ClimbMix-520M

OpenEBFM 520M model (variant d20) trained on ClimbMix. This repository contains a Hugging Face Transformers-compatible export for the pretraining checkpoint.

How to Use

Install the runtime dependencies:

pip install torch transformers tiktoken

Load the tokenizer and model with the custom OpenEBFM code enabled:

import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer

repo_id = "guan-wang/EBFM-ClimbMix-520M"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained(
    repo_id,
    trust_remote_code=True,
)

device = "cuda" if torch.cuda.is_available() else "cpu"
model.to(device)
model.eval()
model.requires_grad_(False)
inputs = tokenizer("Hello world", return_tensors="pt").to(device)
outputs = model(**inputs)

print(outputs.logits.shape)

The first load may prompt you to review and allow the repository's custom model code. Loading remote Python code requires trust_remote_code=True.

Model Details

Detail	Value
Model family	OpenEBFM energy-based language model
Architecture	Custom OpenEBFM implementation (d20)
Parameter scale	520M
Training dataset	ClimbMix
Training stage	pretraining
Transformer blocks	20
Embedding dimension	1280
Attention heads	10
Context length	2048
Vocabulary size	32768

The parameter scale is the label used for this model in the OpenEBFM model configuration. The architecture and sequence settings above are read from the exported checkpoint configuration.

Files

• config.json: model configuration.
• model.safetensors: model weights in the standard Transformers format.
• modeling_ebfm.py: remote model and tokenizer implementation.
• configuration_ebfm.py: remote configuration implementation.
• tokenizer.pkl: serialized EBFM tokenizer.
• tokenizer_config.json: tokenizer auto-loading configuration.
• token_bytes.pt: token byte table used by OpenEBFM metrics.

The code is maintained in the OpenEBFM repository.