| --- |
| library_name: transformers |
| license: apache-2.0 |
| language: |
| - eu |
| - en |
| - es |
| tags: |
| - mamba-2 |
| - basque |
| - autocomplete |
| - on-device |
| - low-resource |
| pipeline_tag: text-generation |
| --- |
| |
| # Morpheus (Mamba-2) — Basque Autocomplete |
|
|
| A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autocompletion. Trained from scratch on a curated Basque corpus and deployed as a 55 MB quantized model running on consumer CPUs. |
|
|
| ## Model Details |
|
|
| - **Architecture:** Mamba-2 (State Space Model, Structured State Space Duality) |
| - **Parameters:** 91M (only 3.4% in embeddings thanks to the 4K vocab) |
| - **Embedding vocab:** 4,000 (Unigram SentencePiece) |
| - **Hidden dimension:** 768 |
| - **Layers:** 24 |
| - **State dimension:** 64 |
| - **Head dimension:** 64 |
| - **Inner dimension:** 1,536 |
| - **Sequence length:** 1,024 |
| - **Training:** ~10B tokens seen (~2.16 epochs over a 4.62B-token unique corpus) |
| - **Best checkpoint:** step 74,000 (held-out PPL 7.13) |
| - **Trained without BOS token** (`add_bos_token=false`) |
|
|
| ## Tokenizer |
|
|
| A 4K Unigram SentencePiece tokenizer trained on the cleaned Basque corpus. The small vocabulary size was chosen based on a vocabulary-size ablation (paper §4.4) motivated by the **fertility paradox** in agglutinative languages: |
|
|
| | Vocab | Fertility (tok/word) | Morpheme Boundary Accuracy | |
| |------:|---------------------:|---------------------------:| |
| | 32,000 | 1.85 | 28.6% | |
| | 4,000 | 2.58 | **66.7%** | |
|
|
| Larger vocabularies fuse Basque roots and suffixes into opaque atomic tokens (e.g. `▁etxetik`, `▁etxera`), destroying the morphological boundaries the model needs to productively generate unseen inflections. The 4K vocabulary keeps individual case suffixes and pluralizers as reusable subwords (`etxe+tik` → `▁etxe tik`), which is decisive for an agglutinative language where a single verb can encode subject, object, indirect object, tense, mood, and aspect through suffix chains. This mirrors the QuechuaTok finding for Quechua. |
|
|
| - `add_bos_token: false` (the model was trained without a BOS token) |
| - EOS token: `</s>` (id=2) |
| - UNK token: `<unk>` (id=0) |
|
|
| ## Usage (transformers) |
|
|
| ```python |
| import torch |
| from transformers import AutoModelForCausalLM, LlamaTokenizer |
| |
| model = AutoModelForCausalLM.from_pretrained("itzune/morpheus", torch_dtype=torch.float32) |
| tokenizer = LlamaTokenizer.from_pretrained("itzune/morpheus") |
| # tokenizer_config.json already sets add_bos_token=False — matches no-BOS training |
| |
| prompt = "Kaixo, zer moduz" |
| inputs = tokenizer(prompt, return_tensors="pt") # NO BOS is prepended (correct) |
| output = model.generate(**inputs, max_new_tokens=5, do_sample=False) |
| print(tokenizer.decode(output[0], skip_special_tokens=True)) |
| ``` |
|
|
| > With the `transformers` tokenizer, **string prompts are fine** — `LlamaTokenizer` uses the reference `sentencepiece` library, so tokenization matches training exactly. The token-ID caveat below applies only to the **GGUF / `llama.cpp`** deployment path. |
|
|
| ## ⚠️ Deploying with llama.cpp? Use token-ID prompts |
|
|
| If you quantize this model to GGUF and serve it with `llama-server`/`llama-cli`, **do not send string prompts**. `llama.cpp`'s built-in SentencePiece tokenizer diverges from the reference library on this 4K vocabulary, and `llama-server` may auto-prepend a BOS token the model never saw in training. Combined, these drop character-saving rate from **~28% to ~4%** — a 7× degradation (paper §5.1). |
|
|
| The fix is to load `tokenizer.model` with the `sentencepiece` library, encode to token IDs (no BOS), and send the ID list to `llama-server`'s `/completion` endpoint. Pre-quantized GGUF models and a full token-ID usage example are at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf). |
|
|
| ## Training Data |
|
|
| Trained on a curated subset of the publicly available **Latxa Corpus v2** (`HiTZ/latxa-corpus-v2`; Etxaniz et al., 2024): |
|
|
| - **4.62 billion subword tokens** (~9 GB tokenized; ~15 GB raw text), of which **~10B tokens were seen** during training (~2.16 epochs). |
| - 11 of the 14 Latxa Corpus v2 sub-corpora were retained; 3 were omitted for quality reasons (`hplt-v1`: 83.8% duplicates; `BOG`: sentence-splitting destroyed legal text; `Aldizkariak`: 35% boilerplate). An LLM-based audit rated the retained sources 4.6/5 on average. |
| - Four-phase cleaning pipeline: document re-parsing, form regularity, content filtering (incl. validation-leakage removal), and MinHash-LSH deduplication. See paper §4.2. |
|
|
| ## Intended Use |
|
|
| On-device Basque text autocomplete and predictive keyboard input. The model is small enough to run on CPU via `llama.cpp` — see the GGUF quantized versions at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf) (55 MB Q4_K_M). On a 2017 consumer laptop CPU (Intel i7-8550U) it achieves 318 tok/s decode and 97 ms end-to-end autocomplete latency (paper §5.2). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{morpheus_mamba, |
| author = {Xabier Ezpeleta}, |
| title = {Morpheus: On-Device Basque Autocompletion with Mamba-2}, |
| year = {2026}, |
| howpublished = {\url{https://huggingface.co/itzune/morpheus}}, |
| } |
| ``` |
|
|
| See the accompanying paper (`morpheus-on-device-basque-autocompletion`) for full architecture, training, evaluation, and deployment details. |
|
|
