README.md

---
license: gemma
language:
  - hy
  - en
base_model: google/gemma-4-e4b
pipeline_tag: text-generation
tags:
  - arxiv:2609.03350
  - armenian
  - continued-pretraining
datasets:
  - COPA-AI/armweb
  - COPA-AI/armstem
---

# arm-gemma-e4b

**arm-gemma-e4b** is Gemma-4-E4B adapted to Armenian by continued pretraining.
To our knowledge, it is the **first open Armenian LLM released together with
its complete training corpus and recipe** — every training token is either
public ([ArmWeb](https://huggingface.co/datasets/COPA-AI/armweb),
[ArmSTEM](https://huggingface.co/datasets/COPA-AI/armstem), FineWeb-Edu,
Stack-smol) or reproducible from the released pipeline.

This is a **base model**: no instruction tuning, no chat template. Use it for
Armenian text continuation, likelihood scoring, or as a starting point for
Armenian SFT.

## Recipe

- **Base**: Gemma-4-E4B, stock tokenizer (best Armenian fertility among CPT
  candidates: 4.15 tokens/word; vocabulary extension was sharply harmful in
  our ablation at this budget).
- **Continued pretraining**: 10B tokens, sequence-packed at length 4096,
  LR 3×10⁻⁵ cosine. The architectural context window is unchanged from the
  base (128K, `max_position_embeddings=131072`); CPT only exercised
  positions up to 4096, and long-context behavior beyond that is inherited
  from the base and not specifically evaluated.
- **Mixture (epoch-capped)**: 69% ArmWeb, 4% ArmSTEM-HY, 2% ArmSTEM-EN,
  20% English web replay (FineWeb-Edu), 5% code (Stack-smol) — every ArmWeb
  token seen ~2× and every ArmSTEM token 7–9× (Gemma-tokenizer accounting).
  The ArmSTEM stream is a 109,885-item subset of the corpus (~30%, sampled
  at random with balanced stratification across the math and science
  pools); its manifest ships with ArmSTEM (`stats/cpt_training_subset_ids.txt`,
  104,630 items verbatim in the release, 5,255 superseded before release).
  A control run trained on the full 373K corpus (~1.3 epochs instead of
  7–9) scores within noise of the released model (paper §5), so the
  repetition is costless.
- **Details**: global batch 512 sequences (~2.1M tokens/step), 4,770 steps,
  100 warmup steps, AdamW (β₂=0.95), weight decay 0.1, bf16, seed 42.

The mixture is the headline finding of the accompanying paper: news-only CPT
catastrophically forgets (−21.2pp Belebele at LR 10⁻⁴); a gentler LR recovers
two-thirds; **6% verified translated STEM data reverses forgetting entirely**,
ending +2.2pp above the unadapted base while keeping the fluency gains.

## Results

Six-task Armenian likelihood suite (accuracy; harness: lm-eval):

| Task | Gemma-4-E4B (base) | **arm-gemma-e4b** |
|---|---|---|
| Belebele-hye | 0.619 | **0.716** |
| INCLUDE-Armenian | 0.416 | **0.456** |
| m-MMLU-hy | 0.343 | 0.337 |
| ARC-hy | 0.227 | **0.229** |
| HellaSwag-hy | 0.266 | 0.262 |
| MultiBLiMP-hye | 0.989 | **0.992** |
| **Mean** | 0.477 | **0.499** |

This is the highest six-task mean among all open Armenian models we
evaluated — the best prior models score 0.471 (ArmenianGPT-1.0-3B) and
0.436 (HyGPT-10b), both *below* the unadapted base.

ArmBench-LLM generative tasks (base-model-appropriate metrics; full
results in the paper appendix):

| Task | base | **arm-gemma-e4b** |
|---|---|---|
| SynDARin (EM) | 0.04 | **0.92** |
| Hartak (EM) | 0.02 | **0.82** |
| DREAM (EM) | 0.48 | **0.84** |
| Belebele gen. (EM) | 0.66 | **0.90** |
| Scientific MCQA (EM) | 0.86 | **1.00**\* |
| MMLU-Pro-Hy | 0.154 | **0.251** |
| Topic (14-class) | 0.004 | **0.482** |

\*Audited: zero shared 8-grams between the benchmark items and the ArmSTEM
training corpus (paper, Appendix).

**Honest negatives**: POS tagging regresses under CPT (0.18→0.01); exam
mathematics is flat for the released model (1.75 points), and the
full-corpus control run reaching 2.75 suggests data diversity rather than
difficulty is the binding factor; instruction-dependent
ArmBench tasks (judged generation, BLEU QA) are low for all base-style models
including this one — they measure formatting, and we defer them to an
instruction-tuned variant.

## Training data & contamination

Training data was decontaminated by 13-gram overlap against ten Armenian
evaluation sets and all ArmBench items, on both the English and Armenian
side for translated data. The two remaining reported benchmarks, m-MMLU-hy
and ARC-hy, were scanned post hoc: zero of 4.31M ArmWeb training
documents and 4 of 372,907 ArmSTEM pairs share any 13-gram with their
items. Existing public Armenian corpora overlap these benchmarks at
7.9–17.4% (see the [ArmWeb card](https://huggingface.co/datasets/COPA-AI/armweb)).

## Limitations

Inherits Gemma-4's biases and terms of use. News-domain-concentrated Armenian
exposure. Machine-translated STEM data verified for answer preservation and
language identity, not stylistic fluency (though a two-annotator
native-speaker audit rated 299 of 300 sampled problems valid, Cohen's
κ = 1.0). No safety tuning.

## Citation

```bibtex
@article{arakelyan2026armweb,
  title  = {From Zero to Hero: An Open LLM Ecosystem for Armenian},
  author = {Arakelyan, Erik and Avetisyan, Khatun and Davtyan, Meri and Grigoryan, Heghine and Khachatryan, Nane and Shahsuvaryan, Hayk and Sergoyan, Henrik and Martirosyan, Vahan},
  year   = {2026},
  journal = {arXiv preprint arXiv:2609.03350}
}
```