jonpojonpo/Llandudno-9B

Llandudno

Llandudno-4B and Llandudno-9B are Welsh-first chat models from Britain, fine-tuned from Qwen3.5 in the UK on a single RTX 4090. They keep everything that makes Qwen3.5 modern: thinking mode, native tool calling, 64K+ context, vision (OCR) and the multi-token-prediction (MTP) head for fast speculative decoding. On top of that they add much better Welsh: reasoning, translation, formal and informal register, and an awareness of North (Gog) and South (Hwntw) Welsh.

Named after the seaside town in North Wales. Small, and well, still quite small.

Results

CymraegBench (AIsteddfod leaderboard harness)

Generative Welsh, run the way the leaderboard runs local models: BF16 weights on vLLM, 4,096-token window, 4 concurrent requests, thinking off, temperature 0, seed 1, up to 100 cases per suite. Overall = ½ reasoning + ½ practical Welsh. Legislation-translation BLEU is reported separately.

Model	Overall	Reasoning	Practical Welsh	Translation BLEU
Llandudno-9B	58.07	63.91	52.22	49.72
Llandudno-4B	55.36	53.72	57.00	47.71
Qwen3.5-9B stock (our rerun, same setup)	55.20	58.29	52.11	37.04
Qwen3.5-9B stock (official leaderboard)	55.10	57.98	52.22	37.06
EryriLabs Gemma 4 12B Cymraeg v6 (leaderboard)	52.70
Qwen3.5-4B stock (Q8_0, mean of 3 seeds)	52.39	48.32	56.45	28.10
EryriLabs Gemma 4 E4B Cymraeg v4 (leaderboard)	49.70
Qwen2.5-7B-Instruct (Q8_0)	41.26	32.19	50.33	14.54

Q8_0 GGUF runs (llama.cpp, seed 1) land within about a point of these: Llandudno-9B 57.84, Llandudno-4B 56.36. Single 100-case runs move by about ±1 point.

Our setup reproduces the official stock Qwen3.5-9B score to within 0.1 points (55.20 vs 55.10).

In short: Llandudno-9B is +2.9 over stock Qwen3.5-9B, and Llandudno-4B matches stock Qwen3.5-9B at less than half the size (and beats EryriLabs' 12B Welsh model). Largest gains over stock (9B): source citation 33→65, XNLI 47→69, formal/informal register 50→71, legal translation +12 BLEU, grammar +5, MMLU-cy +5.

A note on MGSM (Welsh maths): that suite tells the model to "answer with the final number only, without explanation". Llandudno follows the instruction, so it solves word problems without visible working, and the maths score is lower than it would be if the model ignored the format. We chose not to train it to disobey format instructions to win those points. With thinking mode on, it reasons first and then gives the bare number.

Modern features kept (our features_eval, Q8_0)

	Qwen3.5-4B stock	Llandudno-4B	Qwen3.5-9B stock	Llandudno-9B
Long-context needle (8K / 32K / 64K)	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00	1.00 / 1.00 / 1.00
Tool calls, thinking off / on	0.88 / 1.00	0.88 / 0.94	0.75 / 1.00	1.00 / 1.00
Thinking answers / closes </think>	0.88 / 1.00	0.62 / 0.62	0.50 / 0.62	0.88 / 0.88
Vision OCR character error rate	0.01	0.01	0.01	0.01
MTP draft acceptance	0.50	0.50	0.40	0.47

Dialect mirroring (our dialect_eval, 40 informal Gog/Hwntw chat openers)

CymraegBench doesn't test this, so we built our own eval. Does the model answer a North Walian in Gog and a South Walian in Hwntw, without mixing them? Replies are labelled by a marker plus mmBERT dialect classifier (marker words copied from the user are not counted).

Model	Mirrors user's dialect	Gog	Hwntw
Claude Opus 5 / Sonnet 5.5 / Gemini 3.8 Flash	52%	10–12/20	9–11/20
GLM-5.3	40%	7/20	9/20
EryriLabs E4B v4	22%	7/20	2/20
Qwen3.5-4B / -9B stock	5–8%
Llandudno-4B	8%	1/20	2/20
Llandudno-9B	12%	1/20	4/20

Even the frontier models mirror only about half the time. Llandudno identifies Gog vs Hwntw and continues real dialect text, but doesn't yet reply in the user's dialect; that is the next round's main dialect target.

Known weaknesses (please read)

• Welsh swear-word spotting (9B): on CymraegBench's obscenities suite (bare Y/N, "is this word offensive?") Llandudno-9B scores 15 (stock 9B: 33). The same training data took Llandudno-4B from 54 to 83, close to stock 4B's 86. The 9B isn't refusing: on unfamiliar or mutated forms it defaults to "N". Both models explain slang and swearing frankly when asked, but don't use the 9B as a profanity filter yet.
• Thinking mode (4B): Llandudno-4B closes its reasoning cleanly only 62% of the time in our test, against 100% for stock 4B. For production use, keep thinking off or cap the thinking budget; the 9B is fine (88%).
• Irish, Scottish Gaelic and Scots were trained on but not separately evaluated this round.
• Dialect: replies are mostly standard Welsh even when you write in Gog or Hwntw, and North/South forms are occasionally mixed.
• MGSM (Welsh maths) is low because the model obeys the suite's "number only" instruction (see above).

How it was made

One person, one RTX 4090 (24 GB), about two days for this release, with Claude Code sessions doing the data engineering, evaluation and training runs.

1. Continued pretraining (CPT) on Qwen3.5-{4B,9B}-Base:
   • Adapter: bf16 LoRA r64 on the full-attention and Gated-DeltaNet projections and the MLPs (vision tower and MTP head untouched).
   • Run: lr 1e-4 cosine, 2,048-token sequences, 1,043 steps = 68.4M tokens (4B 5.4 h, 9B 8.1 h).
   • Corpus "q1" (characters):
     • quality-filtered Welsh/Irish/Gaelic/Scots web, PDF and spoken text, 104.8M (the Grasmere v2 mix)
     • Senedd and Welsh Government cy↔en parallel text, 60.0M
     • British English web, 25.0M
     • Wikipedia cy/ga/gd/sco, 12.0M
2. Chat vector: the CPT delta is added to the instruct model at 0.5 (Qwen3.5 instruct + 0.5 × ΔCPT). This keeps Qwen's own instruction, tool and thinking training instead of redoing it.
3. SFT on that merge:
   • Adapter: LoRA r64, lr 5e-5 cosine, one epoch, loss on assistant turns only, plus 4M tokens of raw-text replay.
   • 43,402 conversations:
     • the Grasmere SFT mix (38,965): Senedd dialogues, grounded and instruction-precision tasks, native tool calls in cy/ga/gd/sco, translation, maths with working, English replay
     • Welsh yes/no grammar (604, programmatic: answering with the right echoed verb, Ydy / Oedd / Bydd / Do / Oes…)
     • Welsh worked maths (1,200: GSM8K-train problems translated to Welsh and solved by the model itself, kept only when the answer is correct and shows working)
     • register, slang, swearing and moderation (1,629: explaining informal Welsh and swearing "like a mate", labelling messages from polite to abusive, formal↔informal rewrites; generated by DeepSeek V4.1 Flash and kept only when a judge rated the Welsh flawless; no slurs against protected groups)
     • North/South dialect (914: identifying Gog vs Hwntw and continuing real dialect passages, all from authentic FineWeb-2 text)
     • identity (90: "I'm Llandudno…", plus rows that let a system prompt rename it)
4. Merge: the SFT delta is added at 0.6, which scored best in a sweep on the 4B (1.0: 54.2, 0.6: 55.3, 0.4: 53.6, seed 1).
5. Embedding training was tried and dropped: it cost 0.9 points and weakened thinking and MTP.

How we kept it honest

• No benchmark data in training. Every new SFT source was 8-gram-checked against all CymraegBench files: 0 of 4,437 new rows overlap. The inherited Grasmere mix was audited in the same way (0.44% flagged, almost all legal boilerplate). Maths uses GSM8K train; MGSM comes from the test split.
• Inherited rows that ignore format: the Grasmere maths rows include some that show working even when asked for "number only". We kept them for this release but disclose them here; Llandudno still mostly follows the instruction (see the MGSM note).
• We report what the model actually does, including the regressions above, and we didn't tune for any benchmark item.

Use

• Weights: bf16 safetensors with the MTP head and vision tower included. Works with transformers and vLLM.
• GGUF (llama.cpp, LM Studio, Ollama): quantised by mradermacher, thank you!
  • Static quants: mradermacher/Llandudno-9B-GGUF (Q2_K to Q8_0 and f16)
  • Weighted/imatrix quants: mradermacher/Llandudno-9B-i1-GGUF
  • For vision, also load the matching mmproj file from the static GGUF repo. Q4_K_M is a good default; our GGUF scores above used Q8_0.
• Thinking: off by default for chat (enable_thinking: false); turn it on for harder reasoning.
• Sampling: temperature 0.6, top-p 0.95 for chat; temperature 0 for extraction tasks.
• Tools: use the standard Qwen3.5 chat template (--jinja in llama.cpp).
• Identity: it introduces itself as Llandudno unless your system prompt gives it another name.

Licences

• Base model: Qwen3.5 (Apache-2.0).
• Data: as Grasmere (OGL v3 government text, ODC-By crawls, CC0 spoken Welsh, CC BY-SA Wikipedia, public-domain books, ARC/COPA train splits, GSM8K train MIT, SmolTalk prompts), plus DeepSeek V4.1 Flash (MIT) outputs for the register/slang rows.
• ⚠ Web-crawl caveat: some SFT targets are native text from Welsh news and culture publishers (e.g. BBC Cymru Fyw, Golwg360, S4C), obtained via the FineWeb-2 public crawl. The crawl is ODC-By as a compilation, but the underlying articles remain © their publishers.
• CorCenCC and other NC/SA-incompatible sources are not used.

Thanks

• The Qwen team for Qwen3.5.
• BangorAI / Canolfan Bedwyr for CymraegBench and Welsh open data, and AIsteddfod for the leaderboard.
• mradermacher for the GGUF quants.
• EryriLabs for the open recipe and the bar to beat.
• Hugging Face for FineWeb-2 and FinePDFs; the Wikipedia communities; the Internet Archive.
• Huang et al. for chat vectors.
• Built by Jonathan Poole, with Claude Code.

Future work

• Fix swearing recognition: word-form and lexicon data, and calibration of yes/no answers.
• Dialect: real conversational Gog/Hwntw chats, native-speaker review of our marker set and classifier, and a dialect steering vector.
• Personality: on-policy self-distillation with persona prompts and vectors.
• More pretraining data: the 65 OCR'd 19th-century Welsh books (57M characters, not yet used).
• Training sweeps: learning-rate and schedule sweeps (WSD), and per-layer merge weights.