ModernBERT-CJE-tiny

ModernBERT-CJE-tiny is a lightweight text encoder for Chinese, Japanese, and English, distilled from ModernBERT-CJE-large using activation-based initialization and relation distillation. It provides contextual token representations for downstream NLP tasks, text-to-speech (TTS) systems, and other applications that use text features.

This repository contains the tiny checkpoint, including its masked language modeling head and tokenizer. The model uses the ModernBERT architecture with 6 transformer layers, a hidden size of 256, and 4.525M backbone parameters.

Source code, pretraining, distillation, evaluation details, and more examples: KdaiP/ModernBERT-CJE on GitHub.

Quick start

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "KdaiP/ModernBERT-CJE-tiny"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id).to(device).eval()

texts = [
    "你好，世界！",
    "こんにちは、世界！",
    "Hello, world!",
]
inputs = tokenizer(
    texts,
    padding=True,
    truncation=True,
    max_length=512,
    return_tensors="pt",
).to(device)

with torch.inference_mode():
    outputs = model(**inputs, return_dict=True)

features = outputs.last_hidden_state
print(features.shape)  # [batch_size, padded_sequence_length, 256]

License

This model is released under the MIT License.