README.md

---
library_name: transformers
license: apache-2.0
language:
  - eu
  - en
  - es
tags:
  - mamba-2
  - basque
  - autocomplete
  - on-device
  - low-resource
pipeline_tag: text-generation
---

# Morpheus (Mamba-2) — Basque Autocomplete

A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autocompletion. Trained from scratch on a curated Basque corpus and deployed as a 55 MB quantized model running on consumer CPUs.

## Model Details

- **Architecture:** Mamba-2 (State Space Model, Structured State Space Duality)
- **Parameters:** 91M (only 3.4% in embeddings thanks to the 4K vocab)
- **Embedding vocab:** 4,000 (Unigram SentencePiece)
- **Hidden dimension:** 768
- **Layers:** 24
- **State dimension:** 64
- **Head dimension:** 64
- **Inner dimension:** 1,536
- **Sequence length:** 1,024
- **Training:** ~10B tokens seen (~2.16 epochs over a 4.62B-token unique corpus)
- **Best checkpoint:** step 74,000 (held-out PPL 7.13)
- **Trained without BOS token** (`add_bos_token=false`)

## Tokenizer

A 4K Unigram SentencePiece tokenizer trained on the cleaned Basque corpus. The small vocabulary size was chosen based on a vocabulary-size ablation (paper §4.4) motivated by the **fertility paradox** in agglutinative languages:

| Vocab | Fertility (tok/word) | Morpheme Boundary Accuracy |
|------:|---------------------:|---------------------------:|
| 32,000 | 1.85 | 28.6% |
| 4,000 | 2.58 | **66.7%** |

Larger vocabularies fuse Basque roots and suffixes into opaque atomic tokens (e.g. `▁etxetik`, `▁etxera`), destroying the morphological boundaries the model needs to productively generate unseen inflections. The 4K vocabulary keeps individual case suffixes and pluralizers as reusable subwords (`etxe+tik` → `▁etxe tik`), which is decisive for an agglutinative language where a single verb can encode subject, object, indirect object, tense, mood, and aspect through suffix chains. This mirrors the QuechuaTok finding for Quechua.

- `add_bos_token: false` (the model was trained without a BOS token)
- EOS token: `</s>` (id=2)
- UNK token: `<unk>` (id=0)

## Usage (transformers)

```python
import torch
from transformers import AutoModelForCausalLM, LlamaTokenizer

model = AutoModelForCausalLM.from_pretrained("itzune/morpheus", torch_dtype=torch.float32)
tokenizer = LlamaTokenizer.from_pretrained("itzune/morpheus")
# tokenizer_config.json already sets add_bos_token=False — matches no-BOS training

prompt = "Kaixo, zer moduz"
inputs = tokenizer(prompt, return_tensors="pt")   # NO BOS is prepended (correct)
output = model.generate(**inputs, max_new_tokens=5, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=True))
```

> With the `transformers` tokenizer, **string prompts are fine** — `LlamaTokenizer` uses the reference `sentencepiece` library, so tokenization matches training exactly. The token-ID caveat below applies only to the **GGUF / `llama.cpp`** deployment path.

## ⚠️ Deploying with llama.cpp? Use token-ID prompts

If you quantize this model to GGUF and serve it with `llama-server`/`llama-cli`, **do not send string prompts**. `llama.cpp`'s built-in SentencePiece tokenizer diverges from the reference library on this 4K vocabulary, and `llama-server` may auto-prepend a BOS token the model never saw in training. Combined, these drop character-saving rate from **~28% to ~4%** — a 7× degradation (paper §5.1).

The fix is to load `tokenizer.model` with the `sentencepiece` library, encode to token IDs (no BOS), and send the ID list to `llama-server`'s `/completion` endpoint. Pre-quantized GGUF models and a full token-ID usage example are at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf).

## Training Data

Trained on a curated subset of the publicly available **Latxa Corpus v2** (`HiTZ/latxa-corpus-v2`; Etxaniz et al., 2024):

- **4.62 billion subword tokens** (~9 GB tokenized; ~15 GB raw text), of which **~10B tokens were seen** during training (~2.16 epochs).
- 11 of the 14 Latxa Corpus v2 sub-corpora were retained; 3 were omitted for quality reasons (`hplt-v1`: 83.8% duplicates; `BOG`: sentence-splitting destroyed legal text; `Aldizkariak`: 35% boilerplate). An LLM-based audit rated the retained sources 4.6/5 on average.
- Four-phase cleaning pipeline: document re-parsing, form regularity, content filtering (incl. validation-leakage removal), and MinHash-LSH deduplication. See paper §4.2.

## Intended Use

On-device Basque text autocomplete and predictive keyboard input. The model is small enough to run on CPU via `llama.cpp` — see the GGUF quantized versions at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf) (55 MB Q4_K_M). On a 2017 consumer laptop CPU (Intel i7-8550U) it achieves 318 tok/s decode and 97 ms end-to-end autocomplete latency (paper §5.2).

## Citation

```bibtex
@misc{morpheus_mamba,
  author       = {Xabier Ezpeleta},
  title        = {Morpheus: On-Device Basque Autocompletion with Mamba-2},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/itzune/morpheus}},
}
```

See the accompanying paper (`morpheus-on-device-basque-autocompletion`) for full architecture, training, evaluation, and deployment details.