slm-rd release

README.md

@@ -17,13 +17,113 @@ model-index:
 
 # tr-risk-sql-gemma-e4b
 
-**Türkçe risk sorularını SQL'e çeviren küçük model · A small model that turns Turkish risk questions into SQL**
+**A small model that turns Turkish risk questions into SQL · Türkçe risk sorularını SQL'e çeviren küçük model**
 
 Can Demir · [eqlx.ai](https://eqlx.ai) · 2026-11
 
-> **16 dolarlık bir Ar-Ge çalışması:** tek GPU'ya sığan 8 milyar parametreli açık bir model, bankacılık risk yönetimi gibi
-> uzmanlık isteyen bir işte büyük modellere yaklaştırıldı. Hiç görmediği veri ambarlarında doğruluk **%56'dan %87'ye**
-> çıktı.
+> **A 16-dollar R&D study:** an open 8-billion-parameter model that fits on a single GPU was brought close to large
+> models on an expert task, bank risk management. On data marts it had never seen, accuracy rose **from 56% to 87%**.
+>
+> **16 dolarlık bir Ar-Ge çalışması:** tek GPU'ya sığan 8 milyar parametreli açık bir model, bankacılık risk yönetimi
+> gibi uzmanlık isteyen bir işte büyük modellere yaklaştırıldı. Hiç görmediği veri ambarlarında doğruluk **%56'dan
+> %87'ye** çıktı.
+
+---
+
+## English
+
+### Why this work
+
+In sectors that cannot send data outside the organisation (banking, insurance, healthcare), the common assumption is that
+expert tasks need the largest models, and that running them in-house needs a large GPU investment. This work shows the
+opposite: **on a narrow, well-defined expert task, a small model can be brought close to large ones with the right
+engineering.** The large model is needed once, as a teacher; the model that runs every day can be small.
+
+### What it does
+
+A risk analyst asks a question in Turkish about the bank's risk data mart. The model reads the schema, writes a short plan
+and one `SELECT` portable to Oracle and PostgreSQL; a second call summarises the result in Turkish and proposes a chart.
+Domains covered include IFRS 9 expected credit loss, liquidity (LCR, HQLA), collateral, market risk (VaR, backtesting),
+operational risk, credit portfolio, non-performing loans and counterparty risk.
+
+### Results
+
+A query counts as correct only when it is **executed on three databases** and returns exactly the reference result.
+
+| Model (same 160 questions, 6 risk data marts) | Parameters | Accuracy |
+|---|---|---|
+| Frontier-class large model (teacher) | – | 99.4% |
+| Gemma 4 31B (larger sibling) | 31B | 93.1% |
+| **This model + verify-and-vote loop** | **8B** | **88.8%** |
+| This model, single answer | 8B | 79.7% |
+| Gemma 4 E4B, untuned (start) | 8B | 57.3% |
+
+The small model closed **88%** of the gap to its 4x larger sibling and **75%** of the gap to the frontier model.
+
+**Unseen data marts** (sealed test set: 125 questions, five risk domains absent from training), second-generation
+model (v2) with the loop against the untuned model:
+
+| | Untuned | Trained + loop |
+|---|---|---|
+| Overall accuracy | 56.0% | **86.8%** |
+| SQL, plain questions | 53.0% | 91.0% |
+| SQL, banking-jargon questions | 44.5% | 81.0% |
+| Hard questions | 35.6% | 74.5% |
+| Turkish summary | 71.2% | 80.8% |
+
+On a sealed test set of 125 questions over five risk domains absent from training, the second-generation model of this
+pipeline (v2) with the loop went from **56.0% to 86.8%** against the untuned model: +30.8 points, 95% confidence interval
++21.6 to +37.2 (bootstrap over schema families). SQL accuracy rose from 53% to 91% on plain questions and from 44.5% to
+81% on banking-jargon questions; queries that ran but returned a wrong number fell from 62 cases to 25. The model in this
+repository (v3) adds one round of self-distillation on top of v2: single-answer accuracy rose from 77.7% to 79.7% (a
+confirmed gain) and general ability stayed at the untuned model's level (85.0% vs 86.7%).
+
+**Total cost: 15.72 USD** (one rented GPU plus teacher API calls), about 14 hours.
+
+### How it was built
+
+1. **Measurement first.** A frozen execution verifier: every query runs on three seeded databases, results are compared
+   cell by cell regardless of row and column order, and each query is transpiled to Oracle and PostgreSQL. Test sets are
+   split by schema family; the noise floor (±3 points) and paired confidence intervals decide every comparison.
+2. **Synthetic risk worlds.** 53 training and 15 test data marts with Turkish column comments, coded fields and the traps
+   of real data. No real bank data at any stage.
+3. **Cross-verified teacher data.** One large model writes the question and SQL, a second independent model answers from
+   the schema alone; a pair is kept only when both return the same result on all three databases. Four question styles,
+   seven query patterns, 2,236 verified pairs, 2,935 training rows.
+4. **LoRA fine-tuning** of the text decoder only (rank 64, 139.5M trainable parameters). → 72.2%
+5. **Logit distillation** from Gemma 4 31B: the larger sibling's token distributions as a second target (0.5 CE + 0.5 KL,
+   top-64 plus remainder). → 75.2%
+6. **Self-distillation (RFT):** eight sampled answers per training question, executed, the verified ones trained on. → 78.6%
+7. **Plan before SQL:** a 3-7 line plan in Turkish precedes every query.
+8. **Structural diversity:** hand-designed schema shapes (hierarchies, restructuring history, syndication shares,
+   multi-currency positions) under the same blind verification.
+9. **Verify-and-vote loop at inference:** five candidates, executed, failures repaired from the error message, the result
+   most candidates agree on wins. The largest single gain, with no training. → 86.6%, 88.1% with the second generation
+10. **Second self-distillation (this release):** the loop's consistency moved into the weights; a single greedy answer with
+    repair reaches 86.3% in 1-3 calls. → **88.8%** with the loop
+11. **Reinforcement learning study:** GRPO with execution reward, training and rollouts sharing one GPU. Finding: at this
+    scale, self-distillation on verified answers delivers the same idea at a sixth of the cost with a confirmed gain.
+
+### Usage
+
+```python
+from peft import PeftModel
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+base = AutoModelForCausalLM.from_pretrained("google/gemma-4-E4B-it", torch_dtype="auto", device_map="auto")
+model = PeftModel.from_pretrained(base, "eqlx-ai/tr-risk-sql-gemma-e4b", subfolder="adapter")
+tok = AutoTokenizer.from_pretrained("google/gemma-4-E4B-it")
+```
+
+Prompt format: the system message is `code/system_sql.txt`; the user message carries the table definitions (with
+column comments) under `### Şema` and the Turkish question under `### Soru`. The verify-and-vote loop is in
+`code/infer_loop.py` and works with any OpenAI-compatible server (for example vLLM).
+
+### Scope and licence
+
+A research release, developed and measured on synthetic data marts, designed as an analyst's assistant: the analyst sees
+and approves the query and its result. Base model `google/gemma-4-E4B-it` (Apache 2.0). This adapter: **CC BY-NC 4.0**;
+for commercial use contact [eqlx.ai](https://eqlx.ai).
 
 ---
 
@@ -146,74 +246,3 @@ uyumlu herhangi bir sunucuyla (ör. vLLM) çalışır.
 **Kapsam.** Bu sürüm bir araştırma çalışmasıdır ve sentetik veri ambarlarında geliştirilip ölçülmüştür. Analistin
 yardımcısı olarak tasarlanmıştır: sorguyu ve sonucu analist görür ve onaylar. Lisans CC BY-NC 4.0'dır; ticari kullanım
 için [eqlx.ai](https://eqlx.ai) ile iletişime geçin.
-
----
-
-## English
-
-### Why this work
-
-In sectors that cannot send data outside the organisation (banking, insurance, healthcare), the common assumption is that
-expert tasks need the largest models, and that running them in-house needs a large GPU investment. This work shows the
-opposite: **on a narrow, well-defined expert task, a small model can be brought close to large ones with the right
-engineering.** The large model is needed once, as a teacher; the model that runs every day can be small.
-
-### What it does
-
-A risk analyst asks a question in Turkish about the bank's risk data mart. The model reads the schema, writes a short plan
-and one `SELECT` portable to Oracle and PostgreSQL; a second call summarises the result in Turkish and proposes a chart.
-Domains covered include IFRS 9 expected credit loss, liquidity (LCR, HQLA), collateral, market risk (VaR, backtesting),
-operational risk, credit portfolio, non-performing loans and counterparty risk.
-
-### Results
-
-A query counts as correct only when it is **executed on three databases** and returns exactly the reference result.
-
-| Model (same 160 questions, 6 risk data marts) | Parameters | Accuracy |
-|---|---|---|
-| Frontier-class large model (teacher) | – | 99.4% |
-| Gemma 4 31B (larger sibling) | 31B | 93.1% |
-| **This model + verify-and-vote loop** | **8B** | **88.8%** |
-| This model, single answer | 8B | 79.7% |
-| Gemma 4 E4B, untuned (start) | 8B | 57.3% |
-
-The small model closed **88%** of the gap to its 4x larger sibling and **75%** of the gap to the frontier model.
-
-On a sealed test set of 125 questions over five risk domains absent from training, the second-generation model of this
-pipeline (v2) with the loop went from **56.0% to 86.8%** against the untuned model: +30.8 points, 95% confidence interval
-+21.6 to +37.2 (bootstrap over schema families). SQL accuracy rose from 53% to 91% on plain questions and from 44.5% to
-81% on banking-jargon questions; queries that ran but returned a wrong number fell from 62 cases to 25. The model in this
-repository (v3) adds one round of self-distillation on top of v2: single-answer accuracy rose from 77.7% to 79.7% (a
-confirmed gain) and general ability stayed at the untuned model's level (85.0% vs 86.7%).
-
-**Total cost: 15.72 USD** (one rented GPU plus teacher API calls), about 14 hours.
-
-### How it was built
-
-1. **Measurement first.** A frozen execution verifier: every query runs on three seeded databases, results are compared
-   cell by cell regardless of row and column order, and each query is transpiled to Oracle and PostgreSQL. Test sets are
-   split by schema family; the noise floor (±3 points) and paired confidence intervals decide every comparison.
-2. **Synthetic risk worlds.** 53 training and 15 test data marts with Turkish column comments, coded fields and the traps
-   of real data. No real bank data at any stage.
-3. **Cross-verified teacher data.** One large model writes the question and SQL, a second independent model answers from
-   the schema alone; a pair is kept only when both return the same result on all three databases. Four question styles,
-   seven query patterns, 2,236 verified pairs, 2,935 training rows.
-4. **LoRA fine-tuning** of the text decoder only (rank 64, 139.5M trainable parameters). → 72.2%
-5. **Logit distillation** from Gemma 4 31B: the larger sibling's token distributions as a second target (0.5 CE + 0.5 KL,
-   top-64 plus remainder). → 75.2%
-6. **Self-distillation (RFT):** eight sampled answers per training question, executed, the verified ones trained on. → 78.6%
-7. **Plan before SQL:** a 3-7 line plan in Turkish precedes every query.
-8. **Structural diversity:** hand-designed schema shapes (hierarchies, restructuring history, syndication shares,
-   multi-currency positions) under the same blind verification.
-9. **Verify-and-vote loop at inference:** five candidates, executed, failures repaired from the error message, the result
-   most candidates agree on wins. The largest single gain, with no training. → 86.6%, 88.1% with the second generation
-10. **Second self-distillation (this release):** the loop's consistency moved into the weights; a single greedy answer with
-    repair reaches 86.3% in 1-3 calls. → **88.8%** with the loop
-11. **Reinforcement learning study:** GRPO with execution reward, training and rollouts sharing one GPU. Finding: at this
-    scale, self-distillation on verified answers delivers the same idea at a sixth of the cost with a confirmed gain.
-
-### Scope and licence
-
-A research release, developed and measured on synthetic data marts, designed as an analyst's assistant: the analyst sees
-and approves the query and its result. Base model `google/gemma-4-E4B-it` (Apache 2.0). This adapter: **CC BY-NC 4.0**;
-for commercial use contact [eqlx.ai](https://eqlx.ai).