--- base_model: unsloth/gemma-4-E4B-it-unsloth-bnb-4bit library_name: peft pipeline_tag: text-generation license: apache-2.0 language: - ca tags: - maia - andorra - lora - qlora - unsloth - sft --- # gemma-maia-e4b LoRA adapter of **maia**, a model specialised in Andorra built from Gemma 4. Experimental: **not a final model yet**. This card describes the latest version, **`20261102-1437-e4b-k-canon`**. Every training run is a tagged version of this repo: - `20261102-1437-e4b-k-canon` - `20261028-2208-e4b-k` - `20261027-2015-e4b-k` Load a specific version with `revision=""`. - Base model: `unsloth/gemma-4-E4B-it-unsloth-bnb-4bit` - Data: variant **K-canon** of the `training-data` export (conversations in Andorran Catalan) - Code: [github.com/ericrisco/maia](https://github.com/ericrisco/maia), with a [training guide](https://github.com/ericrisco/maia/blob/main/training/README.md) - The whole process, including what failed: [how maia was trained](https://ericrisco.github.io/maia-bench/training.html) ## Evaluation Both models were run as Q4_K_M GGUF through Ollama, at temperature 0, under the same conditions. **Fact recall** on 338 held-out questions about facts of the augmented canon (asked in wording never seen in training; a correct answer must contain the expected keywords): | | Gemma 4 E4B (base) | maia (`20261102-1437-e4b-k-canon`) | |---|---:|---:| | Facts recalled | 25 / 338 (7 %) | **121 / 338 (36 %)** | maia recalls 103 facts the base model misses and misses 7 the base model knew. **[maia-bench](https://github.com/ericrisco/maia-bench)** (926 questions about Andorra, judge Gemini 3.8 Flash; the benchmark questions and the articles it reserves are never trained on): | Question type | Gemma 4 E4B (base) | maia | Difference (95 % CI, paired bootstrap) | |---|---:|---:|---| | All | 30.3 % | **32.9 %** | +2.6 pts [−0.6, +5.9] | | Multiple choice (604) | 42.4 % | 46.2 % | +3.8 pts [−1.0, +8.6] | | Open (231) | 9.1 % | 9.3 % | +0.2 pts [−3.2, +3.7] | | Trap (91) | 3.8 % | 5.0 % | +1.1 pts [−3.8, +6.0] | maia improves on every question type and regresses on none, but the benchmark gain alone is not statistically significant (p ≈ 0.05): the benchmark deliberately asks about facts that are not in the training data, so most of the gain shows up in fact recall. ## Training QLoRA with Unsloth on 1 NVIDIA L4. The loss is computed on the model's responses only. | Parameter | Value | |---|---| | LoRA r / alpha | 32 / 32 | | Epochs | 2 | | Learning rate | 0.0002 (cosine, 3 % warmup) | | Effective batch | 16 (4 x 4) | | Max length | 2048 tokens | | Chat template | `gemma-4` | | Seed | 3407 | Final validation loss: **1.6060** | Step | Epoch | Validation loss | |---:|---:|---:| | 200 | 0.11 | 1.8957 | | 400 | 0.22 | 1.7958 | | 600 | 0.33 | 1.7506 | | 800 | 0.45 | 1.7123 | | 1000 | 0.56 | 1.6821 | | 1200 | 0.67 | 1.6637 | | 1400 | 0.78 | 1.6394 | | 1600 | 0.89 | 1.6271 | | 1800 | 1.00 | 1.6266 | | 2000 | 1.12 | 1.6423 | | 2200 | 1.23 | 1.6382 | | 2400 | 1.34 | 1.6218 | | 2600 | 1.45 | 1.6203 | | 2800 | 1.56 | 1.6143 | | 3000 | 1.67 | 1.6076 | | 3200 | 1.78 | 1.6036 | | 3400 | 1.90 | 1.6059 | | 3586 | 2.00 | 1.6060 | ## System message The model was always trained with this system message (in Catalan). Use it at inference time too: > Ets maia, un assistent que coneix a fons Andorra: la seva història, les institucions, la geografia, la cultura i la gent. Respons en català, amb naturalitat i amb la parla d'Andorra. ## Samples Generated at the end of training (greedy decoding). Unedited and not fact-checked: **some of these answers are wrong or invented**, so do not take them as facts about Andorra. ## Què és el Consell de la Terra? És el nom que el Consell General tenia abans de la Constitució del 1993. ## Quines són les set parròquies d'Andorra? Canillo, Encamp, Ordino, la Massana, Andorra la Vella, Sant Julià de Lòria i Escaldes-Engordany. ## Explica'm què era la Querimònia. Era un acte de gràcia que el Consell General concedia a un veí que havia fet una feina útil per a la comunitat. No era un salari: el Consell pagava una quantitat simbòlica, i el beneficiari podia fer servir aquella quantitat per pagar el seu propi deute. Per això, la recompensa ajudava a alliberar-lo d’una obligació, no pas a guanyar diners per a ell. ## Què es menja per Sant Jordi a Andorra? Per Sant Jordi es menja la coca massegada, una coca dolça amb llet, farina, sucre, ous i mantega. ## Com va anar la reforma constitucional del 1993? El Consell General va aprovar la Constitució el 2 de febrer de 1993, i el poble la va ratificar en referèndum el 14 de març. ## Usage ```python from unsloth import FastModel model, tokenizer = FastModel.from_pretrained("ericrisco/gemma-maia-e4b", revision="20261102-1437-e4b-k-canon", load_in_4bit=True) ``` Quantized GGUF (Q4_K_M) for llama.cpp or Ollama: `maia-e4b-Q4_K_M.gguf`. With Ollama, use a Modelfile with `FROM ./maia-e4b-Q4_K_M.gguf`, `RENDERER gemma4` and `PARSER gemma4`, and keep the system message above.