Add modelcard for v1.13 HR

modelcards/MODELCARD_v113_HR.md

@@ -0,0 +1,300 @@
+---
+license: apache-2.0
+language: [sr, hr, bs, mk, sl, sq, cnr, bg, el, tr, ro, hu]
+base_model: Qwen/Qwen3-8B
+library_name: transformers
+tags: [balkan, southeast-europe, multilingual, honest-ai, sovasoft, rag]
+---
+
+> **🌐 This is the Croatian (Hrvatski) Model Card.** The body text remains in English for international readability.
+
+
+<p align="center">
+  <img src="images/Zora_Stich_1700_SW_Print.png" width="520" alt="Zora — Goddess of Dawn, surrounded by the symbols of 12 peoples"/>
+</p>
+
+# Zora v1.13 — otvoreni, iskreni LLM za Balkan i Jugoistočnu Europu
+
+*зора = "zora". Jedna da ih sve ujedini.* — **Sovasoft** ([ai.in.rs](https://ai.in.rs))
+
+---
+
+## 1 · Što je Zora
+
+Zora is an **open 8B language model** (built on Qwen3-8B) for **12 languages of the Balkans and
+Southeast Europe**: Serbian, Croatian, Bosnian, Macedonian, Slovenian, Albanian, Montenegrin,
+Bulgarian, Greek, Turkish, Romanian, Hungarian.
+
+Zora is not built to be the biggest model — it is built to be **honest, in-language, and multi-perspective**:
+- **thinks in the target language** instead of pivoting through English,
+- shows **several perspectives** on contested topics instead of one national view,
+- and above all: **admits when it doesn't know** instead of inventing facts.
+
+## 2 · Priča o razvoju (v1.0 → v1.1 → v1.11 → v1.12 → v1.13)
+
+| Version | Languages | BalkanBench | State |
+|---|---|---|---|
+| **v1.0** | 6 | — | first public release |
+| **v1.1** | 12 | — | trained from scratch — but **hallucinated facts** (invented book titles, wrong authors). **Never released.** |
+| **v1.11** | 12 | 84/156 | the honest fix: says "I don't know", searches when unsure. **#1 Balkan model.** |
+| **v1.12** | 12 | **85/156** | **the depth fix:** better tool-calling, structured IDK, in-language thinking, RAG integration. |
+| **v1.13** | 12 | **102/156** | **the analysis breakthrough:** critical analysis, advanced logic and graded evaluation jump to new highs, though factual detail regressed. |
+
+v1.1 taught us the key lesson — a small model can't *memorize* every fact, so instead of faking it,
+**v1.11 was retrained to be honest**. v1.12 built on that with deeper training and RAG. v1.13
+pushes reasoning and analysis skills further, trading some factual detail for gains in logic.
+
+## 3 · Što je novo v1.13
+
+### The Analysis & Reasoning Push
+
+**New strengths in v1.13:**
+- **ANALYSIS 0/12 → 12/12** — the biggest single gain. Zora can now critically assess arguments, detect bias across all 12 languages.
+- **LOGIC2 0/12 → 7/12** — advanced multi-step reasoning chains now work in 7 languages.
+- **GRADED 0/12 → 3/12** — partial progress on essays, assessments and rubrics.
+- **LOGIC 0/12 → 3/12** — formal logic and syllogisms improve in some languages.
+
+**Honest trade-offs:**
+- **DETAIL 10/12 → 2/12** — a significant regression. The training that unlocked analysis reduced the depth of elaborated answers. We report this transparently; it is the clearest weakness of v1.13.
+- **SEARCH 7/12 → 5/12** — a smaller drop in quick factual lookup.
+- **FACT 0/12 → 1/12** — marginal improvement in factual recall, still capacity-bound.
+- **HALLU** remains at 10/12 (see the Deep Dive below).
+
+The v1.13 training re-balanced the SFT mix toward reasoning, analysis and graded evaluation.
+That raised the top end of the scoreboard by +17 points (85 → 102) while costing depth on the
+DETAIL axis and some speed on SEARCH. It is a deliberate, documented trade — not a free win.
+
+## 4 · Benchmark (BalkanBench, 13 osi × 12 jezika)
+
+🔬 **BalkanBench is open** — test any model yourself: https://github.com/olivilo/balkanbench
+Deterministic scoring (script / language / keywords / numbers).
+
+### Axis-by-Axis Comparison (v1.12 → v1.13)
+
+| Axis | v1.12 | v1.13 | Δ | What changed |
+|------|-------|-------|---|---|
+| FACT | 0/12 | 1/12 | ↑1 | 8B capacity limit; slight improvement, still small |
+| HALLU | 10/12 | 10/12 | = | Stable; quality of native-language refusals stays high (see Deep Dive below) |
+| DETAIL | 10/12 | 2/12 | **↓8** | **Regression — the honest cost of v1.13's reasoning push** |
+| GRADED | 0/12 | **3/12** | **↑3** | New: essays, assessments, rubrics now sometimes work |
+| TEACH | 12/12 | 12/12 | = | Perfect — remains a core strength |
+| REASON | 11/12 | 11/12 | = | Strong arithmetic reasoning stays |
+| LOGIC | 0/12 | **3/12** | **↑3** | Formal logic / syllogisms improve in some languages |
+| LOGIC2 | 0/12 | **7/12** | **↑6** | Multi-step reasoning chains now work in 7 languages |
+| ANALYSIS | 0/12 | **12/12** | **↑12** | **Biggest gain:** critical analysis, bias detection in all 12 languages |
+| INSTRUCT | 11/12 | 12/12 | ↑1 | Recovered to perfect instruction following |
+| LONGFORM | 12/12 | 12/12 | = | Perfect — remains a core strength |
+| SEARCH | 7/12 | 5/12 | **↓2** | Small regression in quick factual lookup |
+| TOOLBASE | 12/12 | 12/12 | = | Perfect: answers basics without calling tools |
+| **TOTAL** | **85/156** | **102/156** | **+17** | Analysis + logic gains outweigh the DETAIL/SEARCH cost |
+
+### Per-Language Scores
+
+| Language | v1.12 | v1.13 | Δ |
+|----------|-------|-------|---|
+| bg (Bulgarian) | 8/13 | **8/13** | = |
+| bs (Bosnian) | 8/13 | **10/13** | **+2** |
+| cnr (Montenegrin) | 8/13 | **8/13** | = |
+| el (Greek) | 9/13 | **9/13** | = |
+| hr (Croatian) | 9/13 | **9/13** | = |
+| hu (Hungarian) | 8/13 | 7/13 | -1 |
+| mk (Macedonian) | 7/13 | **11/13** | **+4** |
+| ro (Romanian) | 8/13 | **9/13** | **+1** |
+| sl (Slovenian) | 6/13 | **7/13** | **+1** |
+| sq (Albanian) | 8/13 | **8/13** | = |
+| sr (Serbian) | 7/13 | **8/13** | **+1** |
+| tr (Turkish) | 7/13 | **8/13** | **+1** |
+
+**Biggest winner:** Macedonian (+4) — the analysis training lifted the previously least-covered
+language the most. Bosnian (+2) also gained clearly. Hungarian slipped one point.
+
+### Charts
+
+| Ranking | Evolution | Axis Matrix |
+|---------|-----------|-------------|
+| ![Ranking](charts/rank_hr_v113.png) | ![Evolution v1.12 → v1.13](charts/evolution_hr_v113.png) | ![v1.13 axis matrix](charts/matrix_hr_v113.png) |
+
+| Delta (v1.12 → v1.13) | What Each Axis Tests |
+|------------------------|----------------------|
+| ![Delta](charts/delta_hr_v113.png) | ![Explanation](charts/explanation_hr_v113.png) |
+
+## 5 · Zašto je HALLU ostao na 10/12
+
+The HALLU score (10/12) stayed flat across v1.12 and v1.13. The IDK behavior itself remains
+strong — the model says "I don't know" in a structured, native-language way instead of inventing
+facts. So why didn't it move, and where is the real change?
+
+### Why the Score Didn't Move
+
+**1. HALLU is already near its ceiling for an 8B model.**
+The test asks: "Does the model say one of the IDK marker words when asked about a fabricated
+person?" Zora does this correctly for 10 of 12 languages. The last 2 (Macedonian, Slovenian) have
+the smallest training data — an 8B model simply lacks capacity for these underrepresented
+languages. Notably, Macedonian improved overall (+4) thanks to the analysis training, but the
+specific HALLU markers still miss there.
+
+**2. IDK training improved QUALITY, not SCORE.**
+BalkanBench's HALLU test is a binary yes/no check for an IDK marker. What actually improved:
+
+| Before (v1.11) | Now (v1.12 → v1.13) |
+|-----------------|---------------------|
+| Short, sometimes truncated refusals | Full-sentence, structured refusals |
+| Sometimes answered in English | Always answers in the question's language |
+| No reasoning given | Explains *why* it can't answer |
+| "Ne znam." | "Nemam pouzdanih podataka o 'X'. Ne mogu da potvrdim da postoji u pouzdanim izvorima, pa neću da izmišljam." |
+
+This is a qualitative leap that the binary score cannot capture.
+
+**3. The real halluck shift is in ANALYSIS, not HALLU.**
+The v1.13 breakthrough is on the **ANALYSIS** axis (0/12 → 12/12): Zora now critically evaluates
+arguments and detects bias across all 12 languages. That is where the reasoning training showed
+its strongest, most consistent effect.
+
+**4. The honest cost: DETAIL regressed.**
+DETAIL fell from 10/12 to 2/12. The same training that unlocked analysis made answers more
+concise and less elaborated. We report this transparently — v1.13 trades depth of detail for
+critical-analysis capability. Teams that need long, richly detailed prose should weigh this against
+the reasoning gains.
+
+### What Would Move HALLU to 12/12
+
+| Approach | Expected Impact | Effort |
+|----------|----------------|--------|
+| Larger model (v2 = 27B) | +1-2 languages (mk, sl) | High (new training run) |
+| More IDK examples for mk/sl specifically | +0-1 languages | Medium (data generation) |
+| RLHF with human feedback on refusal quality | Better quality (not score) | High (human annotation) |
+| DPO (Direct Preference Optimization) | +1-2 languages | Medium (preference pairs) |
+| Re-balance DETAIL vs ANALYSIS trade | Recover detail at some analysis cost | Medium (data mix tuning) |
+
+**Bottom line:** The 8B model is near its ceiling for HALLU. The real gains in v2 (27B) will come
+from more parameters, not more training tricks. v1.13's win is analysis; its documented cost is
+detail.
+
+## 6 · RAG funkcija
+
+Zora integrates with **RAG (Retrieval-Augmented Generation)** — a system that lets Zora search
+through a local knowledge base before answering.
+
+### What RAG gives Zora
+- **87,284 chunks** across 12 languages: Wikidata, Wikipedia, News Archive, EU Law, Statistics
+- **Live endpoint:** https://rag.ai.in.rs
+- **Self-hostable:** Clone the pipeline from https://github.com/olivilo/zora-v1.13
+
+### How it works
+The tool-cascade: Zora first checks its **RAG knowledge base** (local documents, laws,
+statistics), then falls back to **web search** if needed, and finally says **"I don't know"** if
+neither helps.
+
+```
+User question → RAG (local docs) → web_search (live) → IDK (honest refusal)
+```
+
+### Why this matters
+- 8B models can't memorize everything — RAG gives Zora access to **current, authoritative data** without retraining
+- Every answer carries **source + date + license** — full transparency
+- Self-hostable: any organization can run their own Zora RAG with their own documents
+
+## 7 · Detalji treniranja
+
+| Parameter | Value |
+|-----------|-------|
+| Base model | Qwen3-8B (Alibaba Cloud, Apache-2.0) |
+| CPT steps | 150 (capped, not full epoch) |
+| SFT examples | 9,379 (2 epochs) |
+| MAXLEN | 8192 (8× longer than v1.11) |
+| QLoRA | r=16, lora_alpha=16, 4bit |
+| Data composition | re-balanced toward reasoning, analysis & graded evaluation |
+| Infrastructure | Modal A100-80GB, ~4h total, ~$5-10 |
+| Quantizations | Q5_K_M (5.4GB, recommended), Q6_K (6.7GB), Q8_0 (8.7GB) |
+
+## 8 · Korištenje
+
+**Ollama (recommended):**
+```bash
+ollama pull olivilo/zora:v1.13
+ollama run olivilo/zora:v1.13
+```
+
+**HuggingFace Transformers:**
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+tok = AutoTokenizer.from_pretrained("sovasoft/zora-v1.13")
+model = AutoModelForCausalLM.from_pretrained("sovasoft/zora-v1.13", device_map="auto")
+```
+
+**GGUF (llama.cpp / Ollama manual):**
+Download Q5_K_M, Q6_K, or Q8_0 from [HuggingFace](https://huggingface.co/sovasoft/zora-v1.13).
+**Avoid Q4 and below** — heavy quantization made the model hallucinate in our tests.
+
+## 9 · Ograničenja
+
+- **8B capacity:** FACT remains structurally weak — more parameters needed (v2 = 27B)
+- **DETAIL regression:** v1.13's answers are less elaborated than v1.12 — the analysis push cost depth (2/12)
+- **SEARCH regression:** quick factual lookup dropped slightly (5/12)
+- **Quantization:** use Q5_K_M / Q6_K / Q8_0 only. Q4 and below degrade honesty.
+- **Smaller languages** (mk, sl) still have less training data — expect lower consistency
+- **No real-time knowledge** without RAG/web-search — the model's memory has a cutoff date
+- **Multi-step reasoning** is improved but not reliable — always verify critical calculations
+
+## 10 · Transparentnost benchmarka
+
+**BalkanBench is Sovasoft's own benchmark** — designed, built, and scored by the same team that built Zora. This means:
+
+- **Design bias:** The 13 axes (FACT, HALLU, DETAIL, etc.) were chosen to highlight Zora's strengths. A different benchmark design would produce different rankings.
+- **Scoring bias:** The scoring functions in `matrix_ollama.py` are our own. How we define "correct" may favor Zora's training profile.
+- **No frontier comparison:** We compare only against open models (7-32B). Frontier models (GPT-4, Claude, Gemini) would outperform Zora — this benchmark is designed to evaluate *within* the open-source Balkan model ecosystem.
+- **Selection bias:** We include models where Zora competes well. Inclusion criteria are not random.
+- **Training data overlap:** Some benchmark questions may overlap with Zora's training data, which could inflate scores.
+
+**What the scores DO show:** Zora v1.13 is the strongest open-source model we tested *on our
+benchmark* for 12 Balkan languages — 102/156. It outperforms 3-4× larger models on BalkanBench
+v1.1, a meaningful result for the open-source ecosystem, but not a claim of universal superiority.
+
+**What the scores do NOT show:** That Zora is better than frontier models, that these rankings
+generalize beyond our test design, or that the scoring methodology is independent. The
+transparency here matters most: v1.13 gains analysis at a real, documented cost in factual detail.
+
+## 10 · Što slijedi: v2
+
+| | v1.13 (now) | v2 (planned) |
+|---|---|---|
+| Base | Qwen3-8B | Qwen3.8-27B |
+| BalkanBench | 102/156 | Target: 120+/156 |
+| DETAIL | 2/12 (regression) | Recover + Target: 8+/12 |
+| LOGIC/LOGIC2 | 3/12, 7/12 | Target: 8+/12 |
+| HALLU | 10/12 | Target: 12/12 |
+| Reasoning | Improved analysis | Full chain-of-thought training |
+
+## 11 · Zahvala
+
+Zora exists because of open source. We give our formal, heartfelt thanks:
+
+- **Above all, to the Qwen team at Alibaba** — for developing and open-sourcing **Qwen3** (Apache-2.0),
+  the foundation model Zora is built upon. Without their generosity, Zora would not exist.
+- To the **platforms and structures** that made this possible — **Kaggle, Modal,
+  HuggingFace, Ollama, Unsloth** — for the compute, the tools, and the open infrastructure.
+- To the **open-source community**, for the models, code, and knowledge freely shared with everyone.
+- To the **people of the Balkans** — whose languages, voices, stories and perspectives are Zora's very heart.
+- To **rag.ai.in.rs** for the RAG infrastructure and 87,284 chunks of Balkan knowledge.
+- And to **all that is.**
+
+*зора = "zora". Jedna da ih sve ujedini.* — **Sovasoft** ([ai.in.rs](https://ai.in.rs))
+
+## 12 · Citiranje
+
+```bibtex
+@software{zora_v113,
+  author       = {Vignjevic, Oliver},
+  title        = {Zora v1.13: An Open, Honest LLM for the Balkans \& Southeast Europe},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  url          = {https://huggingface.co/sovasoft/zora-v1.13},
+  license      = {Apache-2.0},
+  base_model   = {Qwen/Qwen3-8B},
+  languages    = {sr, hr, bs, mk, sl, sq, cnr, bg, el, tr, ro, hu}
+}
+```
+
+---
+
+*Sovasoft · ai.in.rs · one to unite them all*