Commit History
-
Update readme file
Verified signature988424f
-
Create README.md
Verified signatureMeMo-BERT-3 MeMo-BERT-3 is a Danish language model built for advanced natural language understanding tasks. It is part of the MeMo-BERT series and has been further pre-trained on top of DanskBERT, leveraging large-scale Danish corpora for improved language representation in a contemporary context. Model Description Base model: DanskBERT (Snæbjarnarson et al., 2023) Original foundation: XLM-RoBERTa (24 layers, 1024 hidden size, 16 attention heads) Tokenizer: Subword tokenizer with a vocabulary size of 250,000 Language: Danish 🇩🇰 Type: Transformer-based masked language model (MLM) Pretraining objective: Continued masked language modeling (MLM) Architecture MeMo-BERT-3 inherits the architecture from XLM-RoBERTa-large: 24 Transformer layers (encoder blocks) Hidden size: 1024 Attention heads: 16 Total vocabulary size: 250,000 (subword-based) Pretraining Details MeMo-BERT-3 continues pretraining from DanskBERT, which was itself further pre-trained from XLM-RoBERTa using the Con…c1492c3
-
Upload 7 files
04cad87 -
initial commit
066c41e