hafsteinn/icelandic-modernbert-s

Icelandic ModernBERT, size s

Question this run answers. How good does a ModernBERT-architecture encoder trained only on Icelandic get at this size when it sees 4x the Chinchilla-optimal number of tokens (80 tokens per parameter)? It is one rung of a size ladder (xs, s, m, base, large) trained with one recipe, so the rungs can be compared with each other and with existing Icelandic encoders (IceBERT, ELECTRA/ConvBERT-IGC, XLM-R, mmBERT, ScandiBERT) under one fine-tuning protocol.

Release. Released under the MIT licence with the paper, as one rung of its scratch-trained size ladder (xs to large, at 80 and 160 tokens per parameter); the training corpus is not released, because not every source states a licence that permits redistribution. Parts of the training data carry licences that require attribution (CC BY 4.0: IGC; CC BY-SA: Wikipedia, Wikisource) or come from web crawls with mixed rights (FineWeb-2, FinePDFs, HPLT 3.0, MaCoCu, MADLAD-400; IC3-v2 states no licence). The IGC restricted-licence parts (News2, Books) were not used. MIM-GOLD-NER, used only for evaluation, has a research-only licence, so no model fine-tuned on it is released.

Model

Architecture	ModernBERT (Warner et al. 2024): pre-norm, GeGLU, RoPE (global theta 160k, local 10k), alternating attention (global every 3rd layer, local window 128), no biases, tied embeddings
Shape	10 layers, hidden 384, GeGLU inner 576, 6 heads (Ettin/ModernBERT ladder)
Parameters	25.3M total = 12.6M embedding + 12.7M non-embedding
Vocabulary	32,768 byte-level BPE trained on our Icelandic corpus (NFC, digits split)
Context	trained at 1,024 tokens, last 10% of tokens at 8,192
Loads with	transformers AutoModelForMaskedLM (model_type modernbert)

Training

Tokens seen	2.02B = 80.0 tokens/parameter (4x Chinchilla, counting all parameters as Hoffmann et al.); 159 per non-embedding parameter
Steps	7,723 at 262,144 tokens per step
Objective	MLM, 30% masking (80/10/10), 15% in the decay phase; loss on masked tokens only
Optimiser	AdamW (0.9, 0.98), eps 1e-6, decoupled weight decay 0.0003, grad clip 1.0, bf16 autocast
Schedule	warmup 61M tokens, constant peak LR 0.002, 1-sqrt decay to 0 over the last 15%
Hardware	MareNostrum 5 accelerated partition (4x H100 64 GB per node), DDP, FlexAttention with document masks (no cross-document attention)

Data

Deduplicated (exact + MinHash 0.75 across all sources, curated sources kept first), quality- and language-filtered, and decontaminated against every evaluation set (13-gram and whole-sentence matching). Per-source epochs below are for the 1,024-token phase.

source	docs	tokens (unique, after filtering)	epochs
dynaword	63,034	0.059B	0.40
finepdfs	230,722	1.862B	0.20
fineweb2	1,977,502	1.614B	0.20
hplt3	1,715,238	1.376B	0.20
ic3v2	964,100	0.364B	0.20
igc	2,012,883	2.012B	0.20
macocu	2,083,595	1.173B	0.20
madlad	604,825	0.559B	0.20
wiki	8,843	0.001B	0.40
all	9,660,742	9.02B

Sources: IGC-2024 open part (arnastofnun/IGC-2024, CC BY 4.0); the non-IGC subsets of danish-foundation-models/icelandic-dynaword (government gazette and Rafbókavefur under Icelandic copyright law's exemptions; Wikipedia, Wikisource, Wikibooks, wiki comments, radio talks CC BY-SA; sagas CC BY; Gutenberg); Icelandic Wikipedia (wikimedia/wikipedia 20231101.is); MaCoCu-is 2.0 (CC0); FineWeb-2 and FinePDFs isl_Latn (ODC-By); HPLT 3.0 isl_Latn; MADLAD-400 is, clean split (ODC-By); IC3-v2 (Miðeind; licence not stated, Common Crawl derived). IcePaHC excluded (evaluation source); restricted IGC parts (News2, Books) not used.

Results

Held-out MLM (15% masking) loss per source at the end of training: {"dynaword": 1.119, "finepdfs": 2.127, "fineweb2": 2.022, "hplt3": 2.079, "ic3v2": 1.595, "igc": 1.571, "macocu": 1.942, "madlad": 1.732, "wiki": 1.425}.

Fine-tuning (same protocol for every model: learning rate picked on dev from a 3-value grid with seed 0, then 5 seeds at that rate; best-dev epoch; 95% CI from a hierarchical bootstrap over seeds and test items):

task (metric)	this model	95% CI	seeds	IceBERT (reference)	difference vs IceBERT [95% CI]
iceec (mcc)	41.2	38.7 to 43.7	5	47.1	-6.0 [-8.4, -3.7]
ner (f1)	90.3	89.4 to 91.1	5	94.1	-3.8 [-4.5, -3.2]
pos (acc)	94.6	94.2 to 95.0	5	95.4	-0.8 [-1.1, -0.5]
qa (f1)	51.3	49.9 to 52.6	5	57.5	-6.2 [-7.2, -5.2]
scala (mcc)	51.2	46.6 to 56.0	5	73.6	-22.3 [-27.1, -18.0]

Tasks: NER = MIM-GOLD-NER (IceBERT split, entity micro F1); POS = UD Icelandic-Modern + GC (UPOS accuracy); ScaLA = acceptability built from the same UD splits (MCC); IceEC = error vs no error (MCC); QA = NQiI + RUQuAD-1 (token F1).

Caveats

• One pretraining seed per size; differences between sizes include pretraining-seed noise that the CIs do not cover.
• The 4x Chinchilla budget is a decoder-derived heuristic; no compute-optimal law for MLM encoders is established.
• Larger sizes repeat the data several times (see epochs); returns from repetition fall off beyond about 4 epochs (Muennighoff et al. 2023).
• IGC JSONL documents were not used for the 8,192-token phase because their paragraph order is not guaranteed.
• AdamW with gradient clipping replaces ModernBERT's StableAdamW; global RoPE theta is 160k from the start.
• The fine-tuning learning rate comes from a 3-value grid per size class. Where the best value sits at the top of the grid (e.g. large: NER, POS, QA at 4e-5), a wider grid might raise those scores a little.
• Fine-tuning used bf16 autocast; ScandiBERT gave non-finite losses in bf16 and was fine-tuned in fp32 (flagged per run).

Full numbers: the accompanying paper (in preparation). The fine-tuning results above are the first round; the paper extends every learning-rate grid until the chosen rate is not at an edge, so its numbers differ slightly. Export parity check (ours vs transformers logits): see export_check.json.