SzegedAI/ModernBERT-LSM

This repository contains the models that were used in the paper No Masking Needed: Predicting Latent Semantics for Efficient Pre-training of Encoder-based Language Models.

Besides the final pre-trained model, this repository also comes with 3 additional internal checkpoints corresponding to 10k, 25k and 50k update steps (out of the total 100k updates of the final model).

As such, when loading the model make sure to specify a subfolder, corresponding to one of the checkpoints 10000, 25000, 50000 or 100000.

Since the output space of the model is no longer the vocabulary space (for MLSM and LSM models), use the ignore_mismatched_sizes=True flag when instantializing an AutoModelForMaskedLM.
When instantializing other model types (e.g. AutoModelForSequenceClassification), no special action is needed.

Example code usage, which loads the checkpoint after 25k update steps:

from transformers import AutoModelForMaskedLM, AutoModelForSequenceClassification, AutoTokenizer

model_name = 'SzegedAI/ModernBERT-LSM'
tokenizer = AutoTokenizer.from_pretrained(model_name)
model_to_finetune = AutoModelForSequenceClassification.from_pretrained(model_name, subfolder='25000', num_labels=2)
model = AutoModelForMaskedLM.from_pretrained(model_name, subfolder='25000', ignore_mismatched_sizes=True)