gemma-maia-e4b / README.md
ericrisco's picture
Card: link the training process and warn about the samples
2ef539b verified
|
Raw History Blame Contribute Delete
5.08 kB
metadata
base_model: unsloth/gemma-4-E4B-it-unsloth-bnb-4bit
library_name: peft
pipeline_tag: text-generation
license: apache-2.0
language:
  - ca
tags:
  - maia
  - andorra
  - lora
  - qlora
  - unsloth
  - sft

gemma-maia-e4b

LoRA adapter of maia, a model specialised in Andorra built from Gemma 4. Experimental: not a final model yet.

This card describes the latest version, 20261102-1437-e4b-k-canon. Every training run is a tagged version of this repo:

  • 20261102-1437-e4b-k-canon
  • 20261028-2208-e4b-k
  • 20261027-2015-e4b-k

Load a specific version with revision="<tag>".

Evaluation

Both models were run as Q4_K_M GGUF through Ollama, at temperature 0, under the same conditions.

Fact recall on 338 held-out questions about facts of the augmented canon (asked in wording never seen in training; a correct answer must contain the expected keywords):

Gemma 4 E4B (base) maia (20261102-1437-e4b-k-canon)
Facts recalled 25 / 338 (7 %) 121 / 338 (36 %)

maia recalls 103 facts the base model misses and misses 7 the base model knew.

maia-bench (926 questions about Andorra, judge Gemini 3.8 Flash; the benchmark questions and the articles it reserves are never trained on):

Question type Gemma 4 E4B (base) maia Difference (95 % CI, paired bootstrap)
All 30.3 % 32.9 % +2.6 pts [−0.6, +5.9]
Multiple choice (604) 42.4 % 46.2 % +3.8 pts [−1.0, +8.6]
Open (231) 9.1 % 9.3 % +0.2 pts [−3.2, +3.7]
Trap (91) 3.8 % 5.0 % +1.1 pts [−3.8, +6.0]

maia improves on every question type and regresses on none, but the benchmark gain alone is not statistically significant (p ≈ 0.05): the benchmark deliberately asks about facts that are not in the training data, so most of the gain shows up in fact recall.

Training

QLoRA with Unsloth on 1 NVIDIA L4. The loss is computed on the model's responses only.

Parameter Value
LoRA r / alpha 32 / 32
Epochs 2
Learning rate 0.0002 (cosine, 3 % warmup)
Effective batch 16 (4 x 4)
Max length 2048 tokens
Chat template gemma-4
Seed 3407

Final validation loss: 1.6060

Step Epoch Validation loss
200 0.11 1.8957
400 0.22 1.7958
600 0.33 1.7506
800 0.45 1.7123
1000 0.56 1.6821
1200 0.67 1.6637
1400 0.78 1.6394
1600 0.89 1.6271
1800 1.00 1.6266
2000 1.12 1.6423
2200 1.23 1.6382
2400 1.34 1.6218
2600 1.45 1.6203
2800 1.56 1.6143
3000 1.67 1.6076
3200 1.78 1.6036
3400 1.90 1.6059
3586 2.00 1.6060

System message

The model was always trained with this system message (in Catalan). Use it at inference time too:

Ets maia, un assistent que coneix a fons Andorra: la seva història, les institucions, la geografia, la cultura i la gent. Respons en català, amb naturalitat i amb la parla d'Andorra.

Samples

Generated at the end of training (greedy decoding). Unedited and not fact-checked: some of these answers are wrong or invented, so do not take them as facts about Andorra.

Què és el Consell de la Terra?

És el nom que el Consell General tenia abans de la Constitució del 1993.

Quines són les set parròquies d'Andorra?

Canillo, Encamp, Ordino, la Massana, Andorra la Vella, Sant Julià de Lòria i Escaldes-Engordany.

Explica'm què era la Querimònia.

Era un acte de gràcia que el Consell General concedia a un veí que havia fet una feina útil per a la comunitat. No era un salari: el Consell pagava una quantitat simbòlica, i el beneficiari podia fer servir aquella quantitat per pagar el seu propi deute. Per això, la recompensa ajudava a alliberar-lo d’una obligació, no pas a guanyar diners per a ell.

Què es menja per Sant Jordi a Andorra?

Per Sant Jordi es menja la coca massegada, una coca dolça amb llet, farina, sucre, ous i mantega.

Com va anar la reforma constitucional del 1993?

El Consell General va aprovar la Constitució el 2 de febrer de 1993, i el poble la va ratificar en referèndum el 14 de març.

Usage

from unsloth import FastModel

model, tokenizer = FastModel.from_pretrained("ericrisco/gemma-maia-e4b", revision="20261102-1437-e4b-k-canon", load_in_4bit=True)

Quantized GGUF (Q4_K_M) for llama.cpp or Ollama: maia-e4b-Q4_K_M.gguf. With Ollama, use a Modelfile with FROM ./maia-e4b-Q4_K_M.gguf, RENDERER gemma4 and PARSER gemma4, and keep the system message above.