jonpojonpo/Grasmere-E4B

Grasmere

Two variants:

• Grasmere-E4B: the strongest benchmark scorer.
• Grasmere-E4B-FTPO: the same model with a targeted anti-repetition fix (FTPO). We recommend it for everyday use.

A small (Gemma 4 E4B) all-purpose chat model for Welsh, Irish, Scottish Gaelic and Scots, with English. It chats, follows precise instructions, cites and quotes sources, and calls tools in all four Celtic languages. On the AIsteddfod CymraegBench it averages 50.3 over three 100-case samples (stock Gemma 4 E4B-it: 40.2), the highest we know of among small models. We did not tune for any benchmark. The gains come from better Welsh text and general skills (see "How we kept it honest").

Results

CymraegBench (generative Welsh; leaderboard settings: temperature 0, 512-token floor, thinking off, Q8_0, llama.cpp)

Model	Overall	Reasoning	Practical Welsh	Translation BLEU
Grasmere, mean of 3 samples (seeds 1–3)	50.33	53.4	47.3	45.1
seed 1 / 2 / 3	50.56 / 50.36 / 50.06
EryriLabs gemma-4-e4b-cymraeg-v4 (leaderboard)	49.70
EryriLabs v4, our local run (seed 1)	49.88	51.55	48.22	40.67
Grasmere-E4B-FTPO (seed 1)	50.14	52.83	47.44	41.81
Gemma 4 E4B-it (stock, seed 1)	40.22	39.43	41.00	26.67

The released Q5_K_M and Q4_K_M quants score within ~2 points of Q8_0 (see "Files"). They are calibrated with a Celtic-text importance matrix; without it, 4-bit lost over 3 points. For the best quality, use bf16 or Q8_0.

Each 100-case suite moves by several points between samples, so we report the mean of three different case samples rather than one run.

Skills (our script-scored eval, skills_eval.py; no judge model)

	Gemma 4 E4B-it	Grasmere
Tool calls, correct tool + arguments (cy / ga / gd / sco, 8 each)	75 / 75 / 38 / 50 %	100 / 88 / 100 / 88 %
Verbatim quote + exact-length summary, JSON (cy / ga / gd)	50 / 20 / 27 %	87 / 87 / 90 %
Exactly-3-bullets; JSON entity extraction	0–100 %	100 %
FLORES+ chrF en→cy / cy→en	40.2 / 51.1	49.9 / 64.9
FLORES+ chrF ga→en / gd→en	46.0 / 39.1	52.2 / 44.2
FLORES+ chrF en→ga / en→gd	34.6 / 31.5	31.9 / 28.6

Tool requests are hand-written (e.g. "Chwilia am drenau o Fangor i Gaerdydd ddydd Gwener nesaf") with 8 tools on offer. Precision tasks use held-out FLORES+ passages.

Known weaknesses (please read)

• Writing Irish and Scottish Gaelic is 2–3 chrF below stock Gemma when translating into them, though understanding them improved. Continued pretraining was 76% Welsh, only about 21% Irish and Gaelic, and contained no English text, which may also weaken translating into those languages. The next version will rebalance this.
• Repetition loops in long answers (Grasmere-E4B): at temperature 0, 39% of long free-form answers end in a loop (Welsh/Irish/Gaelic/Scots prompts, 512 tokens), and about 9% with normal sampling (T 0.3, top-p 0.9, repeat penalty 1.05). Grasmere-E4B-FTPO cuts this to 8% greedy and 0% with sampling. It costs a little on some tasks (citation −8, legislation translation BLEU −5, maths −4 on single 100-case suites) and gains on others (TruthfulQA +4, ARC +4, PIQA +3). Short benchmark-style answers are hardly affected by loops (0.3% of CymraegBench answers loop).
• Dates: relative dates ("the morn" = tomorrow in Scots, "20 Samhain") are occasionally resolved to today.
• Scots data is thin, so Scots output can drift towards English.

Built in under four days on one gaming PC

One person, a single RTX 4090 (24 GB), and Claude Code sessions doing the data engineering, evaluation and training runs. No cluster and no cloud GPUs. API spend for synthetic data and judging was about $30.

When (2026)	Step	CymraegBench
4 Nov	Project starts
5 Nov	Gemma 4 E4B chosen as the base (BritEval comparison); Welsh book OCR pilot (1.0% character error rate); first continued pretraining (89M tokens, 7 h)	stock -it: 40.2
6 Nov	SFT iterations; FineWeb-2 + FinePDFs quality classifier; grounded synthetic SFT	47.6 (sft7)
6–7 Nov (overnight)	Continued pretraining on 39M tokens of quality-filtered Welsh, Irish, Gaelic and Scots, with trained embeddings (4 h)	48.9 (sft8, mean of 3)
7 Nov	Instruction-precision and tool-calling SFT (6.9k rows, script-checked and Haiku-gated)	50.3 (mean of 3)

For comparison, the previous best small model was trained on a 4×RTX 3090 workstation. What made the difference here was the data, not the compute: careful filtering, deduplication, contamination gates, and synthetic data checked by script. We expect others to beat this soon, and we hope this shows how far one GPU and good data can go.

How it was made

1. Continued pretraining (CPT2) on Gemma 4 E4B-it (4-bit QLoRA via Unsloth, one RTX 4090):
   • Adapter and embeddings: rsLoRA r128 α32 on all language-layer attention and MLP at lr 1e-4, plus the tied input/output embedding matrix at lr 1e-5 (the EryriLabs v4 recipe).
   • Run: micro-batch 2 × gradient accumulation 16, sequence length 2,048, 601 steps = 39.3M tokens, 253 minutes, final loss 2.19.
   • Mix (Mtok, from the corpus manifest): fw2-cy 10.8, pdf-cy 7.5, twe-cy 3.6, llyw2 3.0, senedd 1.8, ws-cy 1.5, banc 1.3 (spoken Welsh, 3× weight); fw2-ga 3.1, pdf-ga 1.7; fw2-gd 2.6, pdf-gd 1.4; fw2-sco 0.6, pdf-sco 0.3.
   • That's about 75% Welsh, with no English (English ability comes from the -it base and SFT replay).
   • Web and PDF text was quality-filtered:
     • a Gemma-4-12B-it teacher scored 33k documents (0–5, probability-weighted, fluency plus value);
     • this was distilled into an mmBERT-base classifier (held-out Spearman 0.93 against the teacher);
     • 3.5M documents and chunks were scored, and only the top tier is kept.
   • FinePDFs: non-target pages were removed from bilingual PDFs, and line-level deduplication dropped boilerplate (~10% of Welsh chunks).
2. SFT:
   • Recipe: LoRA r64 α64 on the merged -it + CPT2, lr 5e-5 cosine (3% warmup), adamw_8bit, micro-batch 4 × gradient accumulation 4, sequence length 2,048, 1 epoch = 2,556 steps (139 minutes). Loss is on assistant turns only.
   • Data: 38,934 conversations (18.7M tokens), plus 4.0M tokens of raw CPT-corpus replay in the first 80% of batches; the last 20% is chat-only. It includes:
   • Senedd record dialogues; Welsh Government and Wikipedia tasks; back-translation; translation segments; poetry; Carmina Gadelica.
   • Maths with worked answers (English and Welsh).
   • Grounded synthetic SFT (3.9k): each answer is built from a real passage in the language, written by DeepSeek V4.1 Flash, and judged.
   • Instruction-precision SFT (6.9k: citing, verbatim quoting, format rules, correcting and explaining injected grammar errors, entity extraction, and 1,016 native tool-call conversations, all in cy/ga/gd/sco):
     • every row is script-verified (valid JSON, verbatim quotes, citation ids, counts, tool schema);
     • it is gated by Claude Haiku 5.5 as a pass/fail judge;
     • a Haiku audit pass caught fabricated tool replies and English-syntax "Scots", and both were filtered out. No judge-written text is in the data.
   • Programmatic grammar-injection and grounded fact/quote/cite rows (5.7k, no teacher model).
   • English replay: our own synthetic prompts and SmolTalk Smol-Magpie-Ultra user turns, answered by stock Gemma 4 E4B-it.
3. Merge: a single pass of -it + ΔCPT2 + ΔSFT at α 1.0, with the trained embeddings swapped in. Then a Q8_0 GGUF (llama.cpp).
4. FTPO (Grasmere-E4B-FTPO only; LiquidAI antidoom recipe, Apache-2.0):
   • 1,961 loop pairs mined from the previous model's own generations. Prompts come from SFT user turns and quality-filtered Celtic passages, never the loop-check prompts.
   • LoRA r64, lr 2e-5, early-stopped at step 75/117; tether λ_mse 0.4, λ_tgt 0.05, τ 0.5.
   • Merged as -it + ΔCPT2 + ΔSFT + ΔFTPO.

How we kept it honest

• Contamination gates: every training document and SFT row is checked for word 8-gram overlap against FLORES+ dev/devtest (cy/ga/gd/en), and our held-out evaluation sets. Thousands of CPT documents were dropped (e.g. 2,674 FineWeb-2 Welsh docs).
• No benchmark tuning: no CymraegBench test data is used. Some SFT rows come from the training/validation splits of public tasks (ARC-cy 800, COPA-cy 600, GSM8K maths with working). Test items were removed by exact match for ARC/COPA, and an 8-gram gate (against all CymraegBench files and logged prompts) covers the newer SFT data.
  • A post-hoc 8-gram audit of the full SFT mix flagged 172 of 38,965 rows (0.44%), almost all legal boilerplate or multiple-choice stems.
  • It found 2 ARC-train items that paraphrase ARC test items (an upstream quirk). The impact is ≤ 0.1 points, and they will be removed next round.
• Each model is selected on three seeds plus loop rate, translation and tool-use checks, not on one benchmark number.

Use

Files (each model): bf16 safetensors (Grasmere-E4B, Grasmere-E4B-FTPO) and GGUF (-GGUF repos). Q5_K_M and Q4_K_M are quantised from the bf16 GGUF with a Celtic-text imatrix (400 × 512-token chunks: ~1.4M characters of CPT text in cy/ga/gd/sco plus ~0.6M characters of SFT conversations).

GGUF	Size	Grasmere-E4B CymraegBench (seed 1: overall / reasoning / practical / BLEU)	Grasmere-E4B-FTPO (seed 1)
Q8_0	8.0 GB	50.56 / 53.68 / 47.44 / 46.57	50.14 / 52.83 / 47.44 / 41.81
Q5_K_M	5.8 GB	49.34 / 52.46 / 46.22 / 47.45	49.28 / 51.34 / 47.22 / 41.39
Q4_K_M	5.3 GB	48.86 / 52.61 / 45.11 / 44.42	48.51 / 51.81 / 45.22 / 39.75

Single 100-case runs vary by about ±1 point (a test Q5_K_M scored 50.30), so the differences between quants are partly noise. Our Q6_K scored ≈50.0–50.3 for both models; mradermacher's community quants (below) include a Q6_K. For llama.cpp, use --jinja for native tool calling. Recommended sampling: temperature 0.3, top-p 0.9, repeat penalty 1.05. Disable thinking.

Community quants (thanks, mradermacher!)

mradermacher has published full sets of quants for both models: static quants, plus weighted/imatrix (i1) quants that range from IQ3 up to Q6_K. They're a good choice for smaller GPUs and laptops (llama.cpp, LM Studio, Ollama). Their imatrix is a general one, not our Celtic-text imatrix, and we haven't run CymraegBench on these files.

Model	Static quants	Weighted/imatrix quants
Grasmere-E4B	mradermacher/Grasmere-E4B-GGUF	mradermacher/Grasmere-E4B-i1-GGUF
Grasmere-E4B-FTPO	mradermacher/Grasmere-E4B-FTPO-GGUF	mradermacher/Grasmere-E4B-FTPO-i1-GGUF

Licences

Base model: Gemma 4 (its licence and terms carry through). Data:

• Open government: Senedd Record (techiaith/cofnodycynulliad via locailabs) and BangorAI hysbysiadau-llyw-cymru-2, OGL v3.
• Crawls: FineWeb-2, FinePDFs and TransWebEdu (ODC-By compilations).
• Spoken Welsh: banc-trawsgrifiadau-bangor, CC0.
• Encyclopedic: Wikipedia (CC BY-SA 4.0) and Wikisource (PD / CC BY-SA).
• Public domain: Gutenberg and Carmina Gadelica.
• Task splits: ARC-Welsh train/val (CC BY-SA), COPA-cy train/val (BSD-2-Clause), GSM8K train (MIT).
• SmolTalk Smol-Magpie-Ultra prompts (Apache-2.0; generated with Llama 3.1, so "Built with Llama").
• Synthetic answers: DeepSeek V4.1 Flash (MIT). Some prompts were written by a local Qwen model.
• ⚠ Web-crawl caveat: some SFT answers (bt_sft, 2.5k) are native web text from news and culture publishers (e.g. BBC Cymru Fyw, Golwg360, S4C, Tuairisc, RTÉ, NÓS, LearnGaelic), obtained through the FineWeb-2 public crawl. The crawl is ODC-By as a compilation, but the underlying articles remain © their publishers.

CorCenCC, XNLI, no_robots, duchas.ie, Dolly and other NC/SA-incompatible sources are not used. FLORES+ is evaluation-only.

Thanks (credit where it's due)

• EryriLabs (Dwain Barnes) for openly publishing the v4 recipe this builds on: raw-text replay, trained embeddings, an instruction-only finish. Also for the bar to beat.
• BangorAI / Canolfan Bedwyr for CymraegBench, hysbysiadau-llyw-cymru-2, the spoken-Welsh transcripts (banc-trawsgrifiadau-bangor), and years of open Welsh language technology. AIsteddfod for the leaderboard.
• Hugging Face for FineWeb-2 and FinePDFs. The TransWebEdu authors. OLDI for FLORES+. Google for Gemma 4.
• The Wikipedia and Wikisource communities. The Internet Archive and the University of Toronto for the scanned books.
• jhu-clsp for mmBERT. LiquidAI for the antidoom prompt mix. Sam Paech et al. for FTPO/Antislop. Huang et al. for chat vectors.
• Built by Jonathan Poole on one RTX 4090, with Claude Code sessions doing the data engineering, evaluation and training runs.

Future work

• Rebalance continued pretraining towards Irish and Gaelic. More OCR'd Welsh books (69 queued).
• Better ways of blending chat and Welsh ability: chat-vector scaling per layer, model soups of SFT runs, and merging the CPT and chat deltas with different weights.
• Possibly training the per-layer embeddings.