ModernBERT-CJE-large

ModernBERT-CJE-large is a text encoder for Chinese, Japanese, and English, pretrained from scratch using the ModernBERT architecture. It provides contextual token representations for downstream NLP tasks, text-to-speech (TTS) systems, and other applications that use text features.

This repository contains the large checkpoint, including its masked language modeling head and tokenizer. A distilled ModernBERT-CJE-tiny variant is also available.

Source code, pretraining, distillation, evaluation details, and more examples: KdaiP/ModernBERT-CJE on GitHub.

Quick start

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "KdaiP/ModernBERT-CJE-large"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).to(device).eval()

texts = [
    "你好，世界！",
    "こんにちは、世界！",
    "Hello, world!",
]
inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=4096,
    return_tensors="pt",
).to(device)

with torch.inference_mode():
    outputs = model(**inputs, return_dict=True)

features = outputs.last_hidden_state
print(features.shape)  # [batch_size, padded_sequence_length, 1024]

License

This model is released under the MIT License.