| --- |
| library_name: transformers |
| license: apache-2.0 |
| --- |
| |
| # BERnaT: Basque Encoders for Representing Natural Textual Diversity |
|
|
| Submitted to LREC 2026 |
|
|
| ## Model Description |
|
|
| BERnaT is a family of monolingual Basque encoder-only language models trained to better represent linguistic variation—including standard, dialectal, historical, and informal Basque—rather than focusing solely on standard textual corpora. Models were trained on corpora that combine high-quality standard Basque with varied sources such as social media and historical texts, aiming to enhance robustness and generalization across natural language understanding (NLU) tasks. |
|
|
| - **Developed by:** HiTZ Research Center & IXA Research group (University of the Basque Country UPV/EHU) |
| - **Funded by:** Ikergaitu and ALIA projects (Basque and Spanish Government) |
| - **License:** Apache 2.0 |
| - **Model Type**: Encoder-only Transformer models (RoBERTa-style) |
| - **Languages**: Basque (Euskara) |
|
|
|
|
| ## Getting Started |
|
|
| You can either use this model directly as the example below, or fine-tune it to your task of interest. |
|
|
| ```python |
| >>> from transformers import pipeline |
| >>> pipe = pipeline("fill-mask", model='HiTZ/BERnaT-base') |
| >>> pipe("Kaixo! Ni <mask> naiz!") |
| [{'score': 0.022003261372447014, |
| 'token': 7497, |
| 'token_str': ' euskalduna', |
| 'sequence': 'Kaixo! Ni euskalduna naiz!'}, |
| {'score': 0.016429167240858078, |
| 'token': 14067, |
| 'token_str': ' Olentzero', |
| 'sequence': 'Kaixo! Ni Olentzero naiz!'}, |
| {'score': 0.012804778292775154, |
| 'token': 31087, |
| 'token_str': ' ahobizi', |
| 'sequence': 'Kaixo! Ni ahobizi naiz!'}, |
| {'score': 0.01173020526766777, |
| 'token': 331, |
| 'token_str': ' ez', |
| 'sequence': 'Kaixo! Ni ez naiz!'}, |
| {'score': 0.010091394186019897, |
| 'token': 7618, |
| 'token_str': ' irakaslea', |
| 'sequence': 'Kaixo! Ni irakaslea naiz!'}] |
| ``` |
|
|
| ## Training Data |
|
|
| The BERnaT family was pre-trained on a combination of: |
| - Standard Basque corpora (e.g., Wikipedia, Egunkaria, EusCrawl). |
| - Diverse corpora including Basque social media text and historical Basque books. |
| - Combined corpora for the unified BERnaT models. |
|
|
| Training objective is masked language modeling (MLM) on encoder-only architectures across medium (51M), base (124M), and large (355M) sizes. |
|
|
| ## Evaluation |
|
|
| | | **AVG standard tasks** | **AVG diverse tasks** | **AVG overall** | |
| |---------------------|:----------------------:|:---------------------:|:---------------:| |
| | **BERnaT_standard** | | | | |
| | medium | 74.10 | 70.30 | 72.58 | |
| | base | 75.33 | 71.26 | 73.70 | |
| | large | 76.83 | 73.13 | 75.35 | |
| | **BERnaT_diverse** | | | | |
| | medium | 71.66 | 69.91 | 70.96 | |
| | base | 72.44 | 71.43 | 72.04 | |
| | large | 74.48 | 71.87 | 73.43 | |
| | **BERnaT** | | | | |
| | medium | 73.56 | 70.59 | 72.37 | |
| | base | 75.42 | 71.28 | 73.76 | |
| | large | **77.88** | **73.77** | **76.24** | |
|
|
| ## Acknowledgments |
|
|
| This work has been partially supported by the Basque Government (Research group funding IT1570-22 and IKER-GAITU project), the Spanish Ministry for Digital Transformation and Civil Service, and the EU-funded NextGenerationEU Recovery, Transformation and Resilience Plan (ILENIA project, 2022/TL22/00215335; and ALIA project). The project also received funding from the European Union’s Horizon Europe research and innovation program under Grant Agreement No 101135724, Topic HORIZON-CL4-2023-HUMAN-01-21 and DeepKnowledge (PID2021-127777OB-C21) founded by MCIN/AEI/10.13039/501100011033 and FEDER. Jaione Bengoetxea, Julen Etxaniz and Ekhi Azurmendi hold a PhD grant from the Basque Government (PRE_2024_1_0028, PRE_2024_2_0028 and PRE_2024_1_0035, respectively). Maite Heredia and Mikel Zubillaga hold a PhD grant from the University of the Basque Country UPV/EHU (PIF23/218 and PIF24/04, respectively). The models were trained on the Leonardo supercomputer at CINECA under the EuroHPC Joint Undertaking, project EHPC-EXT-2024E01-042. |
| |
| ## Citation: |
| |
| To cite our work, please use: |
| |
| ```bibtex |
| @misc{azurmendi2025bernatbasqueencodersrepresenting, |
| title={BERnaT: Basque Encoders for Representing Natural Textual Diversity}, |
| author={Ekhi Azurmendi and Joseba Fernandez de Landa and Jaione Bengoetxea and Maite Heredia and Julen Etxaniz and Mikel Zubillaga and Ander Soraluze and Aitor Soroa}, |
| year={2025}, |
| eprint={2512.03903}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CL}, |
| url={https://arxiv.org/abs/2512.03903}, |
| } |
| ``` |
