gemma-maia-e4b / README.md
ericrisco's picture
Card: link the training process and warn about the samples
2ef539b verified
|
Raw History Blame Contribute Delete
5.08 kB
---
base_model: unsloth/gemma-4-E4B-it-unsloth-bnb-4bit
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
- ca
tags:
- maia
- andorra
- lora
- qlora
- unsloth
- sft
---
# gemma-maia-e4b
LoRA adapter of **maia**, a model specialised in Andorra built from Gemma 4.
Experimental: **not a final model yet**.
This card describes the latest version, **`20261102-1437-e4b-k-canon`**.
Every training run is a tagged version of this repo:
- `20261102-1437-e4b-k-canon`
- `20261028-2208-e4b-k`
- `20261027-2015-e4b-k`
Load a specific version with `revision="<tag>"`.
- Base model: `unsloth/gemma-4-E4B-it-unsloth-bnb-4bit`
- Data: variant **K-canon** of the `training-data` export
(conversations in Andorran Catalan)
- Code: [github.com/ericrisco/maia](https://github.com/ericrisco/maia), with a
[training guide](https://github.com/ericrisco/maia/blob/main/training/README.md)
- The whole process, including what failed:
[how maia was trained](https://ericrisco.github.io/maia-bench/training.html)
## Evaluation
Both models were run as Q4_K_M GGUF through Ollama, at temperature 0, under the same conditions.
**Fact recall** on 338 held-out questions about facts of the augmented canon (asked in wording
never seen in training; a correct answer must contain the expected keywords):
| | Gemma 4 E4B (base) | maia (`20261102-1437-e4b-k-canon`) |
|---|---:|---:|
| Facts recalled | 25 / 338 (7 %) | **121 / 338 (36 %)** |
maia recalls 103 facts the base model misses and misses 7 the base model knew.
**[maia-bench](https://github.com/ericrisco/maia-bench)** (926 questions about Andorra, judge
Gemini 3.8 Flash; the benchmark questions and the articles it reserves are never trained on):
| Question type | Gemma 4 E4B (base) | maia | Difference (95 % CI, paired bootstrap) |
|---|---:|---:|---|
| All | 30.3 % | **32.9 %** | +2.6 pts [−0.6, +5.9] |
| Multiple choice (604) | 42.4 % | 46.2 % | +3.8 pts [−1.0, +8.6] |
| Open (231) | 9.1 % | 9.3 % | +0.2 pts [−3.2, +3.7] |
| Trap (91) | 3.8 % | 5.0 % | +1.1 pts [−3.8, +6.0] |
maia improves on every question type and regresses on none, but the benchmark gain alone is not
statistically significant (p ≈ 0.05): the benchmark deliberately asks about facts that are not in
the training data, so most of the gain shows up in fact recall.
## Training
QLoRA with Unsloth on 1 NVIDIA L4. The loss is computed on the model's responses only.
| Parameter | Value |
|---|---|
| LoRA r / alpha | 32 / 32 |
| Epochs | 2 |
| Learning rate | 0.0002 (cosine, 3 % warmup) |
| Effective batch | 16 (4 x 4) |
| Max length | 2048 tokens |
| Chat template | `gemma-4` |
| Seed | 3407 |
Final validation loss: **1.6060**
| Step | Epoch | Validation loss |
|---:|---:|---:|
| 200 | 0.11 | 1.8957 |
| 400 | 0.22 | 1.7958 |
| 600 | 0.33 | 1.7506 |
| 800 | 0.45 | 1.7123 |
| 1000 | 0.56 | 1.6821 |
| 1200 | 0.67 | 1.6637 |
| 1400 | 0.78 | 1.6394 |
| 1600 | 0.89 | 1.6271 |
| 1800 | 1.00 | 1.6266 |
| 2000 | 1.12 | 1.6423 |
| 2200 | 1.23 | 1.6382 |
| 2400 | 1.34 | 1.6218 |
| 2600 | 1.45 | 1.6203 |
| 2800 | 1.56 | 1.6143 |
| 3000 | 1.67 | 1.6076 |
| 3200 | 1.78 | 1.6036 |
| 3400 | 1.90 | 1.6059 |
| 3586 | 2.00 | 1.6060 |
## System message
The model was always trained with this system message (in Catalan). Use it at inference time too:
> Ets maia, un assistent que coneix a fons Andorra: la seva història, les institucions, la geografia, la cultura i la gent. Respons en català, amb naturalitat i amb la parla d'Andorra.
## Samples
Generated at the end of training (greedy decoding). Unedited and not fact-checked: **some of these
answers are wrong or invented**, so do not take them as facts about Andorra.
## Què és el Consell de la Terra?
És el nom que el Consell General tenia abans de la Constitució del 1993.
## Quines són les set parròquies d'Andorra?
Canillo, Encamp, Ordino, la Massana, Andorra la Vella, Sant Julià de Lòria i Escaldes-Engordany.
## Explica'm què era la Querimònia.
Era un acte de gràcia que el Consell General concedia a un veí que havia fet una feina útil per a la comunitat. No era un salari: el Consell pagava una quantitat simbòlica, i el beneficiari podia fer servir aquella quantitat per pagar el seu propi deute. Per això, la recompensa ajudava a alliberar-lo d’una obligació, no pas a guanyar diners per a ell.
## Què es menja per Sant Jordi a Andorra?
Per Sant Jordi es menja la coca massegada, una coca dolça amb llet, farina, sucre, ous i mantega.
## Com va anar la reforma constitucional del 1993?
El Consell General va aprovar la Constitució el 2 de febrer de 1993, i el poble la va ratificar en referèndum el 14 de març.
## Usage
```python
from unsloth import FastModel
model, tokenizer = FastModel.from_pretrained("ericrisco/gemma-maia-e4b", revision="20261102-1437-e4b-k-canon", load_in_4bit=True)
```
Quantized GGUF (Q4_K_M) for llama.cpp or Ollama: `maia-e4b-Q4_K_M.gguf`. With Ollama, use a
Modelfile with `FROM ./maia-e4b-Q4_K_M.gguf`, `RENDERER gemma4` and `PARSER gemma4`, and keep the
system message above.