Upload README.md with huggingface_hub

README.md

@@ -14,14 +14,14 @@ tags:
 pipeline_tag: text-generation
 ---
 
-# Morpheus v2 (Mamba-2) — Basque Autocomplete
+# Morpheus (Mamba-2) — Basque Autocomplete
 
-A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autocompletion.
+A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autocompletion. Trained from scratch on a curated Basque corpus and deployed as a 55 MB quantized model running on consumer CPUs.
 
 ## Model Details
 
-- **Architecture:** Mamba-2 (State Space Model)
-- **Parameters:** 91M
+- **Architecture:** Mamba-2 (State Space Model, Structured State Space Duality)
+- **Parameters:** 91M (only 3.4% in embeddings thanks to the 4K vocab)
 - **Embedding vocab:** 4,000 (Unigram SentencePiece)
 - **Hidden dimension:** 768
 - **Layers:** 24
@@ -29,38 +29,70 @@ A 91M-parameter Mamba-2 language model for on-device Basque (Euskara) text autoc
 - **Head dimension:** 64
 - **Inner dimension:** 1,536
 - **Sequence length:** 1,024
-- **Training tokens:** ~10 billion
-- **Training steps:** 76,000 (best checkpoint at 74,000)
-- **Held-out PPL:** 7.13
-- **Trained without BOS token**
+- **Training:** ~10B tokens seen (~2.16 epochs over a 4.62B-token unique corpus)
+- **Best checkpoint:** step 74,000 (held-out PPL 7.13)
+- **Trained without BOS token** (`add_bos_token=false`)
 
 ## Tokenizer
 
-A 4K Unigram SentencePiece tokenizer trained on the cleaned Basque corpus. The small vocabulary size was chosen based on evidence that lower vocab sizes achieve lower downstream perplexity for agglutinative low-resource languages (cf. QuechuaTok).
+A 4K Unigram SentencePiece tokenizer trained on the cleaned Basque corpus. The small vocabulary size was chosen based on a vocabulary-size ablation (paper §4.4) motivated by the **fertility paradox** in agglutinative languages:
+
+| Vocab | Fertility (tok/word) | Morpheme Boundary Accuracy |
+|------:|---------------------:|---------------------------:|
+| 32,000 | 1.85 | 28.6% |
+| 4,000 | 2.58 | **66.7%** |
+
+Larger vocabularies fuse Basque roots and suffixes into opaque atomic tokens (e.g. `▁etxetik`, `▁etxera`), destroying the morphological boundaries the model needs to productively generate unseen inflections. The 4K vocabulary keeps individual case suffixes and pluralizers as reusable subwords (`etxe+tik` → `▁etxe tik`), which is decisive for an agglutinative language where a single verb can encode subject, object, indirect object, tense, mood, and aspect through suffix chains. This mirrors the QuechuaTok finding for Quechua.
 
 - `add_bos_token: false` (the model was trained without a BOS token)
 - EOS token: `</s>` (id=2)
 - UNK token: `<unk>` (id=0)
 
-## Intended Use
+## Usage (transformers)
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, LlamaTokenizer
+
+model = AutoModelForCausalLM.from_pretrained("itzune/morpheus", torch_dtype=torch.float32)
+tokenizer = LlamaTokenizer.from_pretrained("itzune/morpheus")
+# tokenizer_config.json already sets add_bos_token=False — matches no-BOS training
+
+prompt = "Kaixo, zer moduz"
+inputs = tokenizer(prompt, return_tensors="pt")   # NO BOS is prepended (correct)
+output = model.generate(**inputs, max_new_tokens=5, do_sample=False)
+print(tokenizer.decode(output[0], skip_special_tokens=True))
+```
 
-On-device Basque text autocomplete and predictive keyboard input. The model is small enough to run on CPU via llama.cpp (see the GGUF quantized versions at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf)).
+> With the `transformers` tokenizer, **string prompts are fine** — `LlamaTokenizer` uses the reference `sentencepiece` library, so tokenization matches training exactly. The token-ID caveat below applies only to the **GGUF / `llama.cpp`** deployment path.
+
+## ⚠️ Deploying with llama.cpp? Use token-ID prompts
+
+If you quantize this model to GGUF and serve it with `llama-server`/`llama-cli`, **do not send string prompts**. `llama.cpp`'s built-in SentencePiece tokenizer diverges from the reference library on this 4K vocabulary, and `llama-server` may auto-prepend a BOS token the model never saw in training. Combined, these drop character-saving rate from **~28% to ~4%** — a 7× degradation (paper §5.1).
+
+The fix is to load `tokenizer.model` with the `sentencepiece` library, encode to token IDs (no BOS), and send the ID list to `llama-server`'s `/completion` endpoint. Pre-quantized GGUF models and a full token-ID usage example are at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf).
 
 ## Training Data
 
-Trained on a ~22 GB cleaned Basque text corpus comprising Wikipedia, news (Berria), literature, and other web-crawled sources. The corpus underwent a multi-stage cleaning pipeline (deduplication, language filtering, quality auditing).
+Trained on a curated subset of the publicly available **Latxa Corpus v2** (`HiTZ/latxa-corpus-v2`; Etxaniz et al., 2024):
 
-## Quantized Versions
+- **4.62 billion subword tokens** (~9 GB tokenized; ~15 GB raw text), of which **~10B tokens were seen** during training (~2.16 epochs).
+- 11 of the 14 Latxa Corpus v2 sub-corpora were retained; 3 were omitted for quality reasons (`hplt-v1`: 83.8% duplicates; `BOG`: sentence-splitting destroyed legal text; `Aldizkariak`: 35% boilerplate). An LLM-based audit rated the retained sources 4.6/5 on average.
+- Four-phase cleaning pipeline: document re-parsing, form regularity, content filtering (incl. validation-leakage removal), and MinHash-LSH deduplication. See paper §4.2.
 
-GGUF quantized models (Q4_K_M, Q5_K_M) for llama.cpp inference are available at:
-[itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf)
+## Intended Use
+
+On-device Basque text autocomplete and predictive keyboard input. The model is small enough to run on CPU via `llama.cpp` — see the GGUF quantized versions at [itzune/morpheus-gguf](https://huggingface.co/itzune/morpheus-gguf) (55 MB Q4_K_M). On a 2017 consumer laptop CPU (Intel i7-8550U) it achieves 318 tok/s decode and 97 ms end-to-end autocomplete latency (paper §5.2).
 
 ## Citation
 
 ```bibtex
-@misc{morpheus_v2_mamba,
-  author = {Xabier Ezpeleta},
-  title = {Morpheus v2: On-Device Basque Autocompletion with Mamba-2},
-  year = {2026},
+@misc{morpheus_mamba,
+  author       = {Xabier Ezpeleta},
+  title        = {Morpheus: On-Device Basque Autocompletion with Mamba-2},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/itzune/morpheus}},
 }
 ```
+
+See the accompanying paper (`morpheus-on-device-basque-autocompletion`) for full architecture, training, evaluation, and deployment details.