smallm-wolof: 16.5M sparse-MoE Wolof LM trained from scratch

ce9e1f6 verified 3 months ago

6.16 kB

---
language:
- wo
license: mit
tags:
- wolof
- low-resource
- african-languages
- moe
- mixture-of-experts
- text-generation
- from-scratch
pipeline_tag: text-generation
---
# smallm-wolof

A **16.5M-parameter sparse Mixture-of-Experts language model trained from
scratch on Wolof**, a Senegalese language with very limited digital
resources. This is a research/proof-of-concept project, not a production
model — see [Limitations](#limitations) before using it for anything beyond
experimentation.

## Why this exists

Wolof has no dedicated open reasoning/generative LLM despite being spoken by
~10M+ people across Senegal, Mauritania, and Gambia. Existing multilingual
LLMs (Gemini, GPT-4o, Gemma, Qwen) are all weak in Wolof because it
represents a tiny fraction of their training data. This project explores
what a **from-scratch, Wolof-first** small model can achieve when the whole
pipeline — tokenizer, architecture, data, training recipe — is designed
around the constraints of a genuinely low-resource language, instead of
Wolof being an afterthought in a 100+-language mix.

## Architecture

| | |
|---|---|
| Parameters | 16.5M |
| Layers | 8 |
| Hidden size | 320 |
| Attention | Grouped-Query Attention (8 heads, 2 KV heads) + RoPE |
| Feed-forward | Sparse MoE, 4 experts, top-2 routing, SwiGLU |
| Vocab size | 8,192 (dedicated BPE tokenizer, not reused from another model) |
| Context length | 512 tokens |
| Embeddings | Tied input/output |

**Design choices, and why:**

- **Dedicated tokenizer.** Reusing Qwen's tokenizer on this corpus measured
  a fertility of only 2.46 chars/token (vs ~4.3 for French) — Wolof text was
  being fragmented far more than necessary. A tokenizer trained on Wolof
  itself improves this to 2.92 chars/token with a 6x smaller vocab (8,192 vs
  ~150K), which matters a lot when the whole model is 16.5M parameters.
- **Sparse MoE, not dense.** Data-constrained scaling research
  ([Muennighoff et al., 2023](https://arxiv.org/abs/2305.16264); ["When Data
  Is Scarce"](https://arxiv.org/abs/2606.01155)) shows sparse architectures
  tolerate multi-epoch training on repeated data better than dense ones —
  directly relevant here since this model saw its ~9.7M-token corpus 18
  times, far below the Chinchilla-optimal token budget for this parameter
  count.
- **Explicit load-balancing loss on the router.** Recent work on
  cross-lingual MoE routing ([Bandarkar et al.,
  2026](https://arxiv.org/html/2605.17598v1)) found that low-resource
  language tokens tend to collapse onto a narrow subset of experts in
  naively-trained MoE models. The auxiliary load-balancing loss here forces
  the router to spread Wolof tokens across all 4 experts instead of letting
  a narrow pathway emerge.

## Training data

~9.7M tokens, unified from multiple sources and deduplicated (exact +
near-duplicate via SimHash): Wikipedia (Wolof), FineWeb2, MADLAD-400,
SMOL-doc, plus smaller lexicon/instruction sources (Jolof, Aya, Gatitos).
GSM8K (7,473 grade-school math problems), machine-translated to Wolof and
quality-checked, is folded directly into the pretraining mix rather than
kept as a separate SFT stage — at this parameter budget, a strict
CPT-then-SFT split wastes capacity; mixing lets the model see the reasoning
pattern throughout training.

**Recipe:** 18 epochs (chosen per data-constrained scaling literature:
repeating data up to ~4 epochs is nearly free, meaningful gains extend to
~16, returns vanish around 40), AdamW, cosine LR schedule with warmup,
bf16 mixed precision. Final validation loss: 3.48 (perplexity ≈32.5).
Train/val loss divergence after epoch ~13 indicates mild overfitting,
consistent with the token budget being well below what this parameter count
would ideally need.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("Fallovski/smallm-wolof", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Fallovski/smallm-wolof")
inputs = tokenizer("Sama xarit", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.85,
    top_k=40,
    repetition_penalty=1.3,  # important: without this the model loops on short/common prompts
)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

`trust_remote_code=True` is required — this is a custom architecture, not a
standard `transformers` model class.

## Limitations

Read this before drawing conclusions from the model's output.

- **~9.7M training tokens is very small.** For comparison, Chinchilla-optimal
  scaling would want ~330M tokens for a 16.5M-parameter model. The model
  has learned Wolof surface grammar and the *style* of the corpus (including
  GSM8K-style "step-by-step, then `####` answer" formatting) convincingly,
  but arithmetic in generated reasoning chains is frequently **wrong** — it
  has learned the form of reasoning, not reliable calculation.
- **Repetition loops.** Short or generic prompts (e.g. single common nouns)
  can cause the model to loop on a phrase. Use `repetition_penalty >= 1.3`;
  even then, some prompts still degrade after a few dozen tokens.
- **No independent native-speaker validation of the training corpus.** Part
  of the pretraining data (FineWeb2-derived) mixes standard Wolof
  orthography with phonetic/social-media spelling, and a portion of the
  GSM8K translation was produced by Gemini 2.5 Flash rather than human
  translators — spot checks looked reasonable but no systematic quality
  audit has been done.
- **512-token context**, no long-context capability.
- Not intended for factual reliance, translation accuracy, or any
  production use case.

## Files

- `modeling_smallm_wolof.py` / `configuration_smallm_wolof.py` — custom
  architecture code (loaded via `trust_remote_code=True`)
- `model.safetensors` — trained weights
- `tokenizer.json` — dedicated BPE tokenizer

## Citation

If you build on this, please note it is a research/educational project, not
a peer-reviewed publication. No formal citation is required, but a link back
to this repository is appreciated.

	---
	language:
	- wo
	license: mit
	tags:
	- wolof
	- low-resource
	- african-languages
	- moe
	- mixture-of-experts
	- text-generation
	- from-scratch
	pipeline_tag: text-generation
	---

	# smallm-wolof

	A **16.5M-parameter sparse Mixture-of-Experts language model trained from
	scratch on Wolof**, a Senegalese language with very limited digital
	resources. This is a research/proof-of-concept project, not a production
	model — see [Limitations](#limitations) before using it for anything beyond
	experimentation.

	## Why this exists

	Wolof has no dedicated open reasoning/generative LLM despite being spoken by
	~10M+ people across Senegal, Mauritania, and Gambia. Existing multilingual
	LLMs (Gemini, GPT-4o, Gemma, Qwen) are all weak in Wolof because it
	represents a tiny fraction of their training data. This project explores
	what a from-scratch, Wolof-first small model can achieve when the whole
	pipeline — tokenizer, architecture, data, training recipe — is designed
	around the constraints of a genuinely low-resource language, instead of
	Wolof being an afterthought in a 100+-language mix.

	## Architecture

	\| \| \|
	\|---\|---\|
	\| Parameters \| 16.5M \|
	\| Layers \| 8 \|
	\| Hidden size \| 320 \|
	\| Attention \| Grouped-Query Attention (8 heads, 2 KV heads) + RoPE \|
	\| Feed-forward \| Sparse MoE, 4 experts, top-2 routing, SwiGLU \|
	\| Vocab size \| 8,192 (dedicated BPE tokenizer, not reused from another model) \|
	\| Context length \| 512 tokens \|
	\| Embeddings \| Tied input/output \|

	Design choices, and why:

	- Dedicated tokenizer. Reusing Qwen's tokenizer on this corpus measured
	a fertility of only 2.46 chars/token (vs ~4.3 for French) — Wolof text was
	being fragmented far more than necessary. A tokenizer trained on Wolof
	itself improves this to 2.92 chars/token with a 6x smaller vocab (8,192 vs
	~150K), which matters a lot when the whole model is 16.5M parameters.
	- Sparse MoE, not dense. Data-constrained scaling research
	([Muennighoff et al., 2023](https://arxiv.org/abs/2305.16264); ["When Data
	Is Scarce"](https://arxiv.org/abs/2606.01155)) shows sparse architectures
	tolerate multi-epoch training on repeated data better than dense ones —
	directly relevant here since this model saw its ~9.7M-token corpus 18
	times, far below the Chinchilla-optimal token budget for this parameter
	count.
	- Explicit load-balancing loss on the router. Recent work on
	cross-lingual MoE routing ([Bandarkar et al.,
	2026](https://arxiv.org/html/2605.17598v1)) found that low-resource
	language tokens tend to collapse onto a narrow subset of experts in
	naively-trained MoE models. The auxiliary load-balancing loss here forces
	the router to spread Wolof tokens across all 4 experts instead of letting
	a narrow pathway emerge.

	## Training data

	~9.7M tokens, unified from multiple sources and deduplicated (exact +
	near-duplicate via SimHash): Wikipedia (Wolof), FineWeb2, MADLAD-400,
	SMOL-doc, plus smaller lexicon/instruction sources (Jolof, Aya, Gatitos).
	GSM8K (7,473 grade-school math problems), machine-translated to Wolof and
	quality-checked, is folded directly into the pretraining mix rather than
	kept as a separate SFT stage — at this parameter budget, a strict
	CPT-then-SFT split wastes capacity; mixing lets the model see the reasoning
	pattern throughout training.

	Recipe: 18 epochs (chosen per data-constrained scaling literature:
	repeating data up to ~4 epochs is nearly free, meaningful gains extend to
	~16, returns vanish around 40), AdamW, cosine LR schedule with warmup,
	bf16 mixed precision. Final validation loss: 3.48 (perplexity ≈32.5).
	Train/val loss divergence after epoch ~13 indicates mild overfitting,
	consistent with the token budget being well below what this parameter count
	would ideally need.

	## Usage

	```python
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model = AutoModelForCausalLM.from_pretrained("Fallovski/smallm-wolof", trust_remote_code=True)
	tokenizer = AutoTokenizer.from_pretrained("Fallovski/smallm-wolof")

	inputs = tokenizer("Sama xarit", return_tensors="pt")
	output = model.generate(
	**inputs,
	max_new_tokens=100,
	do_sample=True,
	temperature=0.85,
	top_k=40,
	repetition_penalty=1.3, # important: without this the model loops on short/common prompts
	)
	print(tokenizer.decode(output[0], skip_special_tokens=True))
	```

	`trust_remote_code=True` is required — this is a custom architecture, not a
	standard `transformers` model class.

	## Limitations

	Read this before drawing conclusions from the model's output.

	- ~9.7M training tokens is very small. For comparison, Chinchilla-optimal
	scaling would want ~330M tokens for a 16.5M-parameter model. The model
	has learned Wolof surface grammar and the style of the corpus (including
	GSM8K-style "step-by-step, then `####` answer" formatting) convincingly,
	but arithmetic in generated reasoning chains is frequently wrong — it
	has learned the form of reasoning, not reliable calculation.
	- Repetition loops. Short or generic prompts (e.g. single common nouns)
	can cause the model to loop on a phrase. Use `repetition_penalty >= 1.3`;
	even then, some prompts still degrade after a few dozen tokens.
	- No independent native-speaker validation of the training corpus. Part
	of the pretraining data (FineWeb2-derived) mixes standard Wolof
	orthography with phonetic/social-media spelling, and a portion of the
	GSM8K translation was produced by Gemini 2.5 Flash rather than human
	translators — spot checks looked reasonable but no systematic quality
	audit has been done.
	- 512-token context, no long-context capability.
	- Not intended for factual reliance, translation accuracy, or any
	production use case.

	## Files

	- `modeling_smallm_wolof.py` / `configuration_smallm_wolof.py` — custom
	architecture code (loaded via `trust_remote_code=True`)
	- `model.safetensors` — trained weights
	- `tokenizer.json` — dedicated BPE tokenizer

	## Citation

	If you build on this, please note it is a research/educational project, not
	a peer-reviewed publication. No formal citation is required, but a link back
	to this repository is appreciated.