| This repository contains the models that were used in the paper [No Masking Needed: Predicting Latent Semantics for Efficient Pre-training of Encoder-based Language Models](https://aclanthology.org/2026.findings-aacl.217/). |
|
|
| Besides the final pre-trained model, this repository also comes with 3 additional internal checkpoints corresponding to 10k, 25k and 50k update steps (out of the total 100k updates of the final model). |
|
|
| As such, when loading the model make sure to specify a subfolder, corresponding to one of the checkpoints 10000, 25000, 50000 or 100000. |
|
|
| Since the output space of the model is no longer the vocabulary space (for MLSM and LSM models), use the `ignore_mismatched_sizes=True` flag when instantializing an `AutoModelForMaskedLM`. |
| When instantializing other model types (e.g. `AutoModelForSequenceClassification`), no special action is needed. |
|
|
| Example code usage, which loads the checkpoint after 25k update steps: |
| ``` |
| from transformers import AutoModelForMaskedLM, AutoModelForSequenceClassification, AutoTokenizer |
| |
| model_name = 'SzegedAI/ModernBERT-LSM' |
| tokenizer = AutoTokenizer.from_pretrained(model_name) |
| model_to_finetune = AutoModelForSequenceClassification.from_pretrained(model_name, subfolder='25000', num_labels=2) |
| model = AutoModelForMaskedLM.from_pretrained(model_name, subfolder='25000', ignore_mismatched_sizes=True) |
| ``` |
