BabyLM-community/BabyLM-2026-Strict-Small
Viewer β’ Updated β’ 1.1M β’ 1.28k β’ 3
How to use JimK28/gpt-bert-entmax-graded with Transformers:
# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("fill-mask", model="JimK28/gpt-bert-entmax-graded", trust_remote_code=True) # pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True, device_map="auto")BabyLM 2026 Strict-Small submission (NTUA). A GPT-BERT hybrid language model: gpt_bert_mixed β hidden 384, 12 layers, 6 heads, alpha-entmax attention with per-layer alpha schedule [1.25, 1.25, 1.25, 1.25, 1.75, 1.75, 1.75, 1.75, 1.5, 1.5, 1.5, 1.5].
BabyLM-community/BabyLM-2026-Strict-Small (β€100M words, counting
repeated exposures, per CfP Β§4.2). Uniform ~10 passes via bin-packing (each segment
once per epoch).mntp backend.chck_1M β¦ chck_100M
(words seen), per the BabyLM checkpoint convention.from transformers import AutoTokenizer, AutoModelForMaskedLM
tok = AutoTokenizer.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True)
model = AutoModelForMaskedLM.from_pretrained("JimK28/gpt-bert-entmax-graded", trust_remote_code=True)
Custom architecture β trust_remote_code=True is required (config auto_map points at
modeling_gpt_bert).
Requires pip install entmax (entmax attention). Trained with the NTUA BabyLM pipeline
(https://github.com/emhadzi/BabyLM).