Hugging Face's logo

eqlx-ai
/
tr-risk-sql-gemma-e4b

Text Generation
PEFT
Safetensors
Turkish
English
text-to-sql
turkish
banking
risk-management
lora
distillation
small-language-model
slm-rd
Eval Results (legacy)
Model card Files
xet
 Community
candemirx commited on
Commit
80f8ea8
·
verified ·
1 Parent(s): d92cf3a

Remove raw run table; the card carries the results

Browse files
Files changed (2) hide show
  1. eval/SCOREBOARD.md +0 -90
  2. eval/scoreboard.json +0 -1325
eval/SCOREBOARD.md DELETED
@@ -1,90 +0,0 @@
1
- # Scoreboard
2
-
3
- Updated: 2026-11-11T17:27:40Z · Goal: A risk-management analyst at a Turkish bank asks a question in Turkish about a given risk data-mart schema; one small model (Gemma 4 E4B) writes a correct SQL query (portable to Oracle/PostgreSQL), a 1-3 sentence Turkish summary of the result with faithful numbers, and a short chart spec (type, x, y, title) a renderer draws. R&D goal: CPT + SFT with off-policy logit distillation + RL (RFT, short GRPO) on one student, each stage measured, within 12 active hours and a 20 USD envelope.
4
- Target: "sql_execution clearly above the untuned student (CI excludes 0) on the sealed holdout; turkish_understanding, turkish_summary and chart_spec do not drop; forgetting within noise"
5
- **Best:** loop / eval 0.894 [0.819, 0.954] · vs untuned student +0.320 [0.244, 0.387]
6
- Total spend: 15.72 USD
7
-
8
- Intervals come from a 2000-fold family bootstrap; a difference whose interval excludes zero is real.
9
-
10
- ## runs/bakeoff/cases.jsonl (50 cases)
11
-
12
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | chart_spec | sql_execution | turkish_summary | turkish_understanding | Critical failures | Latency p50 (s) | Cost (USD) | Note |
13
- |---|---|---|---|---|---|---|---|---|---|---|---|
14
- | bakeoff-openrouter-deepseek_deepseek-v4.1-flash / openrouter-deepseek_deepseek-v4.1-flash (frontier) | deepseek/deepseek-v4.1-flash | 0.900 [0.879, 0.947] | – | 1.000 | 0.923 | 0.692 | 1.000 | 0.040 | 4.332 | – | older verifier |
15
- | bakeoff-openrouter-z-ai_glm-5.3-flash / openrouter-z-ai_glm-5.3-flash (frontier) | z-ai/glm-5.3-flash | 0.960 [0.909, 1.000] | – | 1.000 | 1.000 | 0.846 | 1.000 | 0.020 | 8.409 | – | older verifier |
16
- | bakeoff-openrouter-xiaomi_mimo-v2.6-pro / openrouter-xiaomi_mimo-v2.6-pro (frontier) | xiaomi/mimo-v2.6-pro | 0.880 [0.842, 0.923] | – | 1.000 | 1.000 | 0.538 | 1.000 | 0.060 | 5.671 | – | older verifier |
17
- | bakeoff / openrouter-z-ai_glm-5.3-flash-rescore (frontier) | z-ai/glm-5.3-flash | 1.000 [1.000, 1.000] | – | 1.000 | 1.000 | 1.000 | 1.000 | 0.000 | 8.409 | – | rescore |
18
- | bakeoff / openrouter-deepseek_deepseek-v4.1-flash-rescore (frontier) | deepseek/deepseek-v4.1-flash | 0.980 [0.966, 1.000] | – | 1.000 | 0.923 | 1.000 | 1.000 | 0.000 | 4.332 | – | rescore |
19
- | bakeoff / openrouter-xiaomi_mimo-v2.6-pro-rescore (frontier) | xiaomi/mimo-v2.6-pro | 0.980 [0.930, 1.000] | – | 1.000 | 1.000 | 0.923 | 1.000 | 0.020 | 5.671 | – | rescore |
20
-
21
- ## eval/cases/dev.jsonl (24 cases)
22
-
23
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | sql_execution | turkish_understanding | Critical failures | Latency p50 (s) | Cost (USD) | Note |
24
- |---|---|---|---|---|---|---|---|---|---|
25
- | base / eval-probe (experiment) | google/gemma-4-E4B-it | 0.542 [0.500, 0.750] | – | 0.583 | 0.500 | 0.292 | 26.848 | – | |
26
-
27
- ## eval/cases/dev.jsonl (160 cases)
28
-
29
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | chart_spec | sql_execution | turkish_summary | turkish_understanding | Critical failures | Latency p50 (s) | Cost (USD) | Note |
30
- |---|---|---|---|---|---|---|---|---|---|---|---|
31
- | base / eval-dev (student) | google/gemma-4-E4B-it | 0.573 [0.502, 0.641] | – | 0.912 | 0.523 | 0.812 | 0.431 | 0.494 | 5.206 | – | baseline |
32
- | teacher / eval-dev (frontier) | openrouter:z-ai/glm-5.3-flash | 0.000 [0.000, 0.000] | -0.573 [-0.641, -0.501] | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | – | – | invalid |
33
- | run1 / eval-dev (experiment) | run1 | 0.000 [0.000, 0.000] | -0.573 [-0.641, -0.501] | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | – | – | invalid; ⚠ regression: chart_spec, sql_execution, turkish_summary, turkish_understanding |
34
- | run1 / dev (experiment) | run1 | 0.722 [0.650, 0.777] | +0.148 [0.081, 0.204] | 0.988 | 0.725 | 0.800 | 0.604 | 0.375 | 6.706 | – | |
35
- | run2 / dev (experiment) | run2 | 0.752 [0.662, 0.821] | +0.178 [0.077, 0.250] | 1.000 | 0.738 | 0.738 | 0.688 | 0.319 | 5.870 | – | |
36
- | rft1 / dev (experiment) | rft1 | 0.786 [0.691, 0.863] | +0.212 [0.110, 0.294] | 1.000 | 0.750 | 0.887 | 0.717 | 0.244 | 5.978 | – | |
37
- | rft1 / export-dev (experiment) | rft1 | 0.797 [0.700, 0.872] | +0.223 [0.124, 0.302] | 1.000 | 0.762 | 0.900 | 0.729 | 0.237 | 6.032 | – | |
38
- | loop / eval (experiment) | rft1-loop-t0 | 0.819 [0.734, 0.883] | +0.245 [0.173, 0.302] | 1.000 | 0.750 | 1.000 | 0.767 | 0.131 | 3.700 | – | |
39
- | loop / eval (experiment) | rft1-loop-t0r2 | 0.838 [0.735, 0.906] | +0.264 [0.202, 0.321] | 1.000 | 0.783 | 0.950 | 0.800 | 0.163 | 3.282 | – | |
40
- | loop / eval (experiment) | rft1-loop-t1r2 | 0.838 [0.759, 0.889] | +0.264 [0.206, 0.313] | 1.000 | 0.838 | 0.863 | 0.775 | 0.281 | 3.555 | – | |
41
- | loop / eval (experiment) | rft1-loop-k5r2 | 0.866 [0.793, 0.926] | +0.292 [0.216, 0.359] | 1.000 | 0.850 | 0.925 | 0.817 | 0.175 | 17.995 | – | |
42
- | loop / eval (experiment) | rft1-loop-k5r2t07 | 0.850 [0.747, 0.921] | +0.277 [0.216, 0.338] | 1.000 | 0.833 | 0.825 | 0.825 | 0.181 | 16.406 | – | |
43
- | run4 / dev (experiment) | run4 | 0.777 [0.687, 0.852] | +0.203 [0.123, 0.280] | 1.000 | 0.750 | 0.850 | 0.704 | 0.375 | 8.329 | – | |
44
- | loop / eval (experiment) | run4-loop-k5r2 | 0.881 [0.819, 0.935] | +0.308 [0.241, 0.366] | 1.000 | 0.883 | 0.850 | 0.850 | 0.156 | 27.918 | – | |
45
- | grpo2 / dev (experiment) | grpo2 | 0.781 [0.695, 0.852] | +0.208 [0.135, 0.280] | 1.000 | 0.750 | 0.825 | 0.725 | 0.325 | 9.489 | – | |
46
- | loop / eval (experiment) | grpo2-loop-k5r2 | 0.894 [0.819, 0.954] | +0.320 [0.244, 0.387] | 1.000 | 0.892 | 0.925 | 0.850 | 0.169 | 28.351 | – | |
47
- | rft2 / dev (experiment) | rft2 | 0.797 [0.702, 0.875] | +0.223 [0.146, 0.300] | 1.000 | 0.775 | 0.963 | 0.696 | 0.331 | 11.448 | – | |
48
- | loop / eval (experiment) | rft2-loop-rft2-k5r2 | 0.887 [0.787, 0.956] | +0.314 [0.248, 0.376] | 1.000 | 0.858 | 0.950 | 0.858 | 0.144 | 35.439 | – | |
49
- | loop / eval (experiment) | rft2-loop-rft2-k1r2-t0 | 0.863 [0.784, 0.931] | +0.289 [0.215, 0.361] | 1.000 | 0.867 | 1.000 | 0.767 | 0.138 | 6.566 | – | |
50
-
51
- ## knowledge/forgetting/general_tr_en.jsonl (240 cases)
52
-
53
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | forgetting | Critical failures | Latency p50 (s) | Cost (USD) | Note |
54
- |---|---|---|---|---|---|---|---|---|
55
- | base / eval-forgetting (student) | google/gemma-4-E4B-it | 0.867 [0.821, 0.908] | – | 0.867 | 0.000 | 0.595 | – | baseline |
56
- | run1 / eval-forgetting (experiment) | run1 | 0.000 [0.000, 0.000] | -0.867 [-0.908, -0.821] | 0.000 | 0.000 | – | – | invalid; ⚠ regression: forgetting |
57
- | run1 / forgetting (experiment) | run1 | 0.787 [0.729, 0.838] | -0.079 [-0.125, -0.037] | 0.787 | 0.000 | 0.599 | – | ⚠ regression: forgetting |
58
- | run2 / forgetting (experiment) | run2 | 0.787 [0.733, 0.838] | -0.079 [-0.121, -0.042] | 0.787 | 0.000 | 0.611 | – | ⚠ regression: forgetting |
59
- | rft1 / forgetting (experiment) | rft1 | 0.804 [0.750, 0.854] | -0.062 [-0.113, -0.017] | 0.804 | 0.000 | 0.631 | – | ⚠ regression: forgetting |
60
- | rft2 / forgetting (experiment) | rft2 | 0.850 [0.800, 0.896] | -0.017 [-0.067, 0.029] | 0.850 | 0.000 | 0.665 | – | |
61
-
62
- ## runs/teacher-full/cases.jsonl (160 cases)
63
-
64
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | chart_spec | sql_execution | turkish_summary | turkish_understanding | Critical failures | Latency p50 (s) | Cost (USD) | Note |
65
- |---|---|---|---|---|---|---|---|---|---|---|---|
66
- | bakeoff-openrouter-z-ai_glm-5.3-flash / openrouter-z-ai_glm-5.3-flash (frontier) | z-ai/glm-5.3-flash | 0.994 [0.981, 1.000] | – | 1.000 | 1.000 | 1.000 | 0.983 | 0.000 | 10.628 | – | |
67
- | bakeoff-openrouter-google_gemma-4-31b-it / openrouter-google_gemma-4-31b-it (frontier) | google/gemma-4-31b-it | 0.931 [0.894, 0.969] | – | 1.000 | 0.933 | 0.800 | 0.950 | 0.069 | 5.132 | – | |
68
-
69
- ## data/prompts/rft_cases.jsonl (1567 cases)
70
-
71
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | report | sql_execution | Critical failures | Latency p50 (s) | Cost (USD) | Note |
72
- |---|---|---|---|---|---|---|---|---|---|
73
- | run2 / train-samples (experiment) | run2 | 0.770 [0.752, 0.789] | – | 0.875 | 0.741 | 0.344 | 3.367 | – | |
74
-
75
- ## eval/cases/holdout.jsonl (100 cases)
76
-
77
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | chart_spec | sql_execution | turkish_summary | turkish_understanding | Critical failures | Latency p50 (s) | Cost (USD) | Note |
78
- |---|---|---|---|---|---|---|---|---|---|---|---|
79
- | base / holdout (student) | google/gemma-4-E4B-it | 0.532 [0.410, 0.655] | – | 0.800 | 0.531 | 0.900 | 0.375 | 0.500 | 6.296 | – | baseline |
80
- | rft1 / holdout (experiment) | rft1 | 0.675 [0.615, 0.743] | +0.142 [0.080, 0.205] | 1.000 | 0.650 | 0.900 | 0.562 | 0.340 | 6.162 | – | |
81
-
82
- ## eval/cases/holdout2.jsonl (125 cases)
83
-
84
- | Run | Model | Score avg@N [95% interval] | Δ vs untuned student | chart_spec | sql_execution | turkish_summary | turkish_understanding | Critical failures | Latency p50 (s) | Cost (USD) | Note |
85
- |---|---|---|---|---|---|---|---|---|---|---|---|
86
- | rft1 / holdout2 (experiment) | rft1 | 0.750 [0.672, 0.824] | +0.190 [0.054, 0.310] | 1.000 | 0.710 | 0.942 | 0.680 | 0.336 | 6.553 | – | |
87
- | loop / eval (experiment) | rft1-loop-k5r2 | 0.796 [0.732, 0.856] | +0.236 [0.114, 0.344] | 1.000 | 0.800 | 0.846 | 0.730 | 0.248 | 13.693 | – | |
88
- | run4 / holdout2 (experiment) | run4 | 0.744 [0.680, 0.818] | +0.184 [0.086, 0.294] | 1.000 | 0.715 | 0.865 | 0.680 | 0.376 | 9.291 | – | |
89
- | loop / eval (experiment) | run4-loop-k5r2 | 0.868 [0.824, 0.920] | +0.308 [0.216, 0.372] | 1.000 | 0.910 | 0.808 | 0.810 | 0.200 | 31.140 | – | |
90
- | base / holdout2 (student) | google/gemma-4-E4B-it | 0.560 [0.514, 0.618] | – | 1.000 | 0.530 | 0.712 | 0.445 | 0.496 | 6.226 | – | baseline |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eval/scoreboard.json DELETED
@@ -1,1325 +0,0 @@
1
- {
2
- "updated": "2026-11-11T17:27:40Z",
3
- "objective": "A risk-management analyst at a Turkish bank asks a question in Turkish about a given risk data-mart schema; one small model (Gemma 4 E4B) writes a correct SQL query (portable to Oracle/PostgreSQL), a 1-3 sentence Turkish summary of the result with faithful numbers, and a short chart spec (type, x, y, title) a renderer draws. R&D goal: CPT + SFT with off-policy logit distillation + RL (RFT, short GRPO) on one student, each stage measured, within 12 active hours and a 20 USD envelope.",
4
- "success": "sql_execution clearly above the untuned student (CI excludes 0) on the sealed holdout; turkish_understanding, turkish_summary and chart_spec do not drop; forgetting within noise",
5
- "axes_targets": [
6
- {
7
- "axis": "sql_execution",
8
- "target": null
9
- },
10
- {
11
- "axis": "turkish_understanding",
12
- "target": null
13
- },
14
- {
15
- "axis": "turkish_summary",
16
- "target": null
17
- },
18
- {
19
- "axis": "chart_spec",
20
- "target": null
21
- },
22
- {
23
- "axis": "forgetting",
24
- "target": null
25
- }
26
- ],
27
- "total_usd": 15.7159,
28
- "sets": [
29
- {
30
- "cases": "runs/bakeoff/cases.jsonl",
31
- "n": 50,
32
- "cases_hash": "sha256:d6e6e9f283bb06dc003de8f14184a305aac838d281aa23c6a1f855ac5c51321d",
33
- "axes": [
34
- "chart_spec",
35
- "sql_execution",
36
- "turkish_summary",
37
- "turkish_understanding"
38
- ],
39
- "baseline": null,
40
- "rows": [
41
- {
42
- "run": "bakeoff-openrouter-deepseek_deepseek-v4.1-flash / openrouter-deepseek_deepseek-v4.1-flash",
43
- "model": "deepseek/deepseek-v4.1-flash",
44
- "role": "frontier",
45
- "avg": 0.9,
46
- "ci": [
47
- 0.8788,
48
- 0.9474
49
- ],
50
- "base": false,
51
- "usd": null,
52
- "stale_verifier": true,
53
- "critical_rate": 0.04,
54
- "latency_p50": 4.332,
55
- "simulated": false,
56
- "invalid": false,
57
- "rescore": false,
58
- "by_axis": {
59
- "sql_execution": 0.9231,
60
- "turkish_summary": 0.6923,
61
- "turkish_understanding": 1.0,
62
- "chart_spec": 1.0
63
- }
64
- },
65
- {
66
- "run": "bakeoff-openrouter-z-ai_glm-5.3-flash / openrouter-z-ai_glm-5.3-flash",
67
- "model": "z-ai/glm-5.3-flash",
68
- "role": "frontier",
69
- "avg": 0.96,
70
- "ci": [
71
- 0.9091,
72
- 1.0
73
- ],
74
- "base": false,
75
- "usd": null,
76
- "stale_verifier": true,
77
- "critical_rate": 0.02,
78
- "latency_p50": 8.409,
79
- "simulated": false,
80
- "invalid": false,
81
- "rescore": false,
82
- "by_axis": {
83
- "sql_execution": 1.0,
84
- "turkish_summary": 0.8462,
85
- "turkish_understanding": 1.0,
86
- "chart_spec": 1.0
87
- }
88
- },
89
- {
90
- "run": "bakeoff-openrouter-xiaomi_mimo-v2.6-pro / openrouter-xiaomi_mimo-v2.6-pro",
91
- "model": "xiaomi/mimo-v2.6-pro",
92
- "role": "frontier",
93
- "avg": 0.88,
94
- "ci": [
95
- 0.8421,
96
- 0.9231
97
- ],
98
- "base": false,
99
- "usd": null,
100
- "stale_verifier": true,
101
- "critical_rate": 0.06,
102
- "latency_p50": 5.671,
103
- "simulated": false,
104
- "invalid": false,
105
- "rescore": false,
106
- "by_axis": {
107
- "sql_execution": 1.0,
108
- "turkish_summary": 0.5385,
109
- "turkish_understanding": 1.0,
110
- "chart_spec": 1.0
111
- }
112
- },
113
- {
114
- "run": "bakeoff / openrouter-z-ai_glm-5.3-flash-rescore",
115
- "model": "z-ai/glm-5.3-flash",
116
- "role": "frontier",
117
- "avg": 1.0,
118
- "ci": [
119
- 1.0,
120
- 1.0
121
- ],
122
- "base": false,
123
- "usd": null,
124
- "stale_verifier": false,
125
- "critical_rate": 0.0,
126
- "latency_p50": 8.409,
127
- "simulated": false,
128
- "invalid": false,
129
- "rescore": true,
130
- "by_axis": {
131
- "sql_execution": 1.0,
132
- "turkish_summary": 1.0,
133
- "turkish_understanding": 1.0,
134
- "chart_spec": 1.0
135
- }
136
- },
137
- {
138
- "run": "bakeoff / openrouter-deepseek_deepseek-v4.1-flash-rescore",
139
- "model": "deepseek/deepseek-v4.1-flash",
140
- "role": "frontier",
141
- "avg": 0.98,
142
- "ci": [
143
- 0.9659,
144
- 1.0
145
- ],
146
- "base": false,
147
- "usd": null,
148
- "stale_verifier": false,
149
- "critical_rate": 0.0,
150
- "latency_p50": 4.332,
151
- "simulated": false,
152
- "invalid": false,
153
- "rescore": true,
154
- "by_axis": {
155
- "sql_execution": 0.9231,
156
- "turkish_summary": 1.0,
157
- "turkish_understanding": 1.0,
158
- "chart_spec": 1.0
159
- }
160
- },
161
- {
162
- "run": "bakeoff / openrouter-xiaomi_mimo-v2.6-pro-rescore",
163
- "model": "xiaomi/mimo-v2.6-pro",
164
- "role": "frontier",
165
- "avg": 0.98,
166
- "ci": [
167
- 0.9302,
168
- 1.0
169
- ],
170
- "base": false,
171
- "usd": null,
172
- "stale_verifier": false,
173
- "critical_rate": 0.02,
174
- "latency_p50": 5.671,
175
- "simulated": false,
176
- "invalid": false,
177
- "rescore": true,
178
- "by_axis": {
179
- "sql_execution": 1.0,
180
- "turkish_summary": 0.9231,
181
- "turkish_understanding": 1.0,
182
- "chart_spec": 1.0
183
- }
184
- }
185
- ],
186
- "holdout": false
187
- },
188
- {
189
- "cases": "eval/cases/dev.jsonl",
190
- "n": 24,
191
- "cases_hash": "sha256:6c9654ef96f61edb1f0ed0bb4ed00e4a43283c62edfeb7b6753999a0b358d20d",
192
- "axes": [
193
- "sql_execution",
194
- "turkish_understanding"
195
- ],
196
- "baseline": null,
197
- "rows": [
198
- {
199
- "run": "base / eval-probe",
200
- "model": "google/gemma-4-E4B-it",
201
- "role": "experiment",
202
- "avg": 0.5417,
203
- "ci": [
204
- 0.5,
205
- 0.75
206
- ],
207
- "base": false,
208
- "usd": null,
209
- "stale_verifier": false,
210
- "critical_rate": 0.2917,
211
- "latency_p50": 26.848,
212
- "simulated": false,
213
- "invalid": false,
214
- "rescore": false,
215
- "by_axis": {
216
- "sql_execution": 0.5833,
217
- "turkish_understanding": 0.5
218
- }
219
- }
220
- ],
221
- "holdout": false
222
- },
223
- {
224
- "cases": "eval/cases/dev.jsonl",
225
- "n": 160,
226
- "cases_hash": "sha256:35e152e3be377a60b1b9c1d67bc785596ef793fc5fdf19fd8c224cb6c9535171",
227
- "axes": [
228
- "chart_spec",
229
- "sql_execution",
230
- "turkish_summary",
231
- "turkish_understanding"
232
- ],
233
- "baseline": "base / eval-dev",
234
- "rows": [
235
- {
236
- "run": "base / eval-dev",
237
- "model": "google/gemma-4-E4B-it",
238
- "role": "student",
239
- "avg": 0.5734,
240
- "ci": [
241
- 0.5023,
242
- 0.641
243
- ],
244
- "base": true,
245
- "usd": null,
246
- "stale_verifier": false,
247
- "critical_rate": 0.4938,
248
- "latency_p50": 5.206,
249
- "simulated": false,
250
- "invalid": false,
251
- "rescore": false,
252
- "by_axis": {
253
- "sql_execution": 0.5229,
254
- "turkish_understanding": 0.4313,
255
- "turkish_summary": 0.8125,
256
- "chart_spec": 0.9125
257
- }
258
- },
259
- {
260
- "run": "teacher / eval-dev",
261
- "model": "openrouter:z-ai/glm-5.3-flash",
262
- "role": "frontier",
263
- "avg": 0.0,
264
- "ci": [
265
- 0.0,
266
- 0.0
267
- ],
268
- "base": false,
269
- "usd": null,
270
- "stale_verifier": false,
271
- "critical_rate": 0.0,
272
- "latency_p50": null,
273
- "simulated": false,
274
- "invalid": true,
275
- "rescore": false,
276
- "by_axis": {
277
- "sql_execution": 0.0,
278
- "turkish_understanding": 0.0,
279
- "turkish_summary": 0.0,
280
- "chart_spec": 0.0
281
- },
282
- "delta": -0.5734,
283
- "delta_ci": [
284
- -0.641,
285
- -0.5008
286
- ],
287
- "regressed_axes": []
288
- },
289
- {
290
- "run": "run1 / eval-dev",
291
- "model": "run1",
292
- "role": "experiment",
293
- "avg": 0.0,
294
- "ci": [
295
- 0.0,
296
- 0.0
297
- ],
298
- "base": false,
299
- "usd": null,
300
- "stale_verifier": false,
301
- "critical_rate": 0.0,
302
- "latency_p50": null,
303
- "simulated": false,
304
- "invalid": true,
305
- "rescore": false,
306
- "by_axis": {
307
- "sql_execution": 0.0,
308
- "turkish_understanding": 0.0,
309
- "turkish_summary": 0.0,
310
- "chart_spec": 0.0
311
- },
312
- "delta": -0.5734,
313
- "delta_ci": [
314
- -0.641,
315
- -0.5008
316
- ],
317
- "regressed_axes": [
318
- "chart_spec",
319
- "sql_execution",
320
- "turkish_summary",
321
- "turkish_understanding"
322
- ]
323
- },
324
- {
325
- "run": "run1 / dev",
326
- "model": "run1",
327
- "role": "experiment",
328
- "avg": 0.7219,
329
- "ci": [
330
- 0.6503,
331
- 0.7766
332
- ],
333
- "base": false,
334
- "usd": null,
335
- "stale_verifier": false,
336
- "critical_rate": 0.375,
337
- "latency_p50": 6.706,
338
- "simulated": false,
339
- "invalid": false,
340
- "rescore": false,
341
- "by_axis": {
342
- "sql_execution": 0.725,
343
- "turkish_understanding": 0.6042,
344
- "turkish_summary": 0.8,
345
- "chart_spec": 0.9875
346
- },
347
- "delta": 0.1484,
348
- "delta_ci": [
349
- 0.0807,
350
- 0.2045
351
- ],
352
- "regressed_axes": []
353
- },
354
- {
355
- "run": "run2 / dev",
356
- "model": "run2",
357
- "role": "experiment",
358
- "avg": 0.7516,
359
- "ci": [
360
- 0.6625,
361
- 0.821
362
- ],
363
- "base": false,
364
- "usd": null,
365
- "stale_verifier": false,
366
- "critical_rate": 0.3187,
367
- "latency_p50": 5.87,
368
- "simulated": false,
369
- "invalid": false,
370
- "rescore": false,
371
- "by_axis": {
372
- "sql_execution": 0.7375,
373
- "turkish_understanding": 0.6875,
374
- "turkish_summary": 0.7375,
375
- "chart_spec": 1.0
376
- },
377
- "delta": 0.1781,
378
- "delta_ci": [
379
- 0.0769,
380
- 0.25
381
- ],
382
- "regressed_axes": []
383
- },
384
- {
385
- "run": "rft1 / dev",
386
- "model": "rft1",
387
- "role": "experiment",
388
- "avg": 0.7859,
389
- "ci": [
390
- 0.6907,
391
- 0.8627
392
- ],
393
- "base": false,
394
- "usd": null,
395
- "stale_verifier": false,
396
- "critical_rate": 0.2437,
397
- "latency_p50": 5.978,
398
- "simulated": false,
399
- "invalid": false,
400
- "rescore": false,
401
- "by_axis": {
402
- "sql_execution": 0.75,
403
- "turkish_understanding": 0.7167,
404
- "turkish_summary": 0.8875,
405
- "chart_spec": 1.0
406
- },
407
- "delta": 0.2125,
408
- "delta_ci": [
409
- 0.1104,
410
- 0.294
411
- ],
412
- "regressed_axes": []
413
- },
414
- {
415
- "run": "rft1 / export-dev",
416
- "model": "rft1",
417
- "role": "experiment",
418
- "avg": 0.7969,
419
- "ci": [
420
- 0.7003,
421
- 0.8719
422
- ],
423
- "base": false,
424
- "usd": null,
425
- "stale_verifier": false,
426
- "critical_rate": 0.2375,
427
- "latency_p50": 6.032,
428
- "simulated": false,
429
- "invalid": false,
430
- "rescore": false,
431
- "by_axis": {
432
- "sql_execution": 0.7625,
433
- "turkish_understanding": 0.7292,
434
- "turkish_summary": 0.9,
435
- "chart_spec": 1.0
436
- },
437
- "delta": 0.2234,
438
- "delta_ci": [
439
- 0.1242,
440
- 0.3025
441
- ],
442
- "regressed_axes": []
443
- },
444
- {
445
- "run": "loop / eval",
446
- "model": "rft1-loop-t0",
447
- "role": "experiment",
448
- "avg": 0.8187,
449
- "ci": [
450
- 0.7342,
451
- 0.8827
452
- ],
453
- "base": false,
454
- "usd": null,
455
- "stale_verifier": false,
456
- "critical_rate": 0.1313,
457
- "latency_p50": 3.7,
458
- "simulated": false,
459
- "invalid": false,
460
- "rescore": false,
461
- "by_axis": {
462
- "sql_execution": 0.75,
463
- "turkish_understanding": 0.7667,
464
- "turkish_summary": 1.0,
465
- "chart_spec": 1.0
466
- },
467
- "delta": 0.2453,
468
- "delta_ci": [
469
- 0.1733,
470
- 0.3025
471
- ],
472
- "regressed_axes": []
473
- },
474
- {
475
- "run": "loop / eval",
476
- "model": "rft1-loop-t0r2",
477
- "role": "experiment",
478
- "avg": 0.8375,
479
- "ci": [
480
- 0.7346,
481
- 0.9062
482
- ],
483
- "base": false,
484
- "usd": null,
485
- "stale_verifier": false,
486
- "critical_rate": 0.1625,
487
- "latency_p50": 3.282,
488
- "simulated": false,
489
- "invalid": false,
490
- "rescore": false,
491
- "by_axis": {
492
- "sql_execution": 0.7833,
493
- "turkish_understanding": 0.8,
494
- "turkish_summary": 0.95,
495
- "chart_spec": 1.0
496
- },
497
- "delta": 0.2641,
498
- "delta_ci": [
499
- 0.2021,
500
- 0.321
501
- ],
502
- "regressed_axes": []
503
- },
504
- {
505
- "run": "loop / eval",
506
- "model": "rft1-loop-t1r2",
507
- "role": "experiment",
508
- "avg": 0.8375,
509
- "ci": [
510
- 0.7594,
511
- 0.8889
512
- ],
513
- "base": false,
514
- "usd": null,
515
- "stale_verifier": false,
516
- "critical_rate": 0.2812,
517
- "latency_p50": 3.555,
518
- "simulated": false,
519
- "invalid": false,
520
- "rescore": false,
521
- "by_axis": {
522
- "sql_execution": 0.8375,
523
- "turkish_understanding": 0.775,
524
- "turkish_summary": 0.8625,
525
- "chart_spec": 1.0
526
- },
527
- "delta": 0.2641,
528
- "delta_ci": [
529
- 0.2059,
530
- 0.3133
531
- ],
532
- "regressed_axes": []
533
- },
534
- {
535
- "run": "loop / eval",
536
- "model": "rft1-loop-k5r2",
537
- "role": "experiment",
538
- "avg": 0.8656,
539
- "ci": [
540
- 0.7932,
541
- 0.9259
542
- ],
543
- "base": false,
544
- "usd": null,
545
- "stale_verifier": false,
546
- "critical_rate": 0.175,
547
- "latency_p50": 17.995,
548
- "simulated": false,
549
- "invalid": false,
550
- "rescore": false,
551
- "by_axis": {
552
- "sql_execution": 0.85,
553
- "turkish_understanding": 0.8167,
554
- "turkish_summary": 0.925,
555
- "chart_spec": 1.0
556
- },
557
- "delta": 0.2922,
558
- "delta_ci": [
559
- 0.2163,
560
- 0.3594
561
- ],
562
- "regressed_axes": []
563
- },
564
- {
565
- "run": "loop / eval",
566
- "model": "rft1-loop-k5r2t07",
567
- "role": "experiment",
568
- "avg": 0.85,
569
- "ci": [
570
- 0.7469,
571
- 0.9209
572
- ],
573
- "base": false,
574
- "usd": null,
575
- "stale_verifier": false,
576
- "critical_rate": 0.1812,
577
- "latency_p50": 16.406,
578
- "simulated": false,
579
- "invalid": false,
580
- "rescore": false,
581
- "by_axis": {
582
- "sql_execution": 0.8333,
583
- "turkish_understanding": 0.825,
584
- "turkish_summary": 0.825,
585
- "chart_spec": 1.0
586
- },
587
- "delta": 0.2766,
588
- "delta_ci": [
589
- 0.2163,
590
- 0.338
591
- ],
592
- "regressed_axes": []
593
- },
594
- {
595
- "run": "run4 / dev",
596
- "model": "run4",
597
- "role": "experiment",
598
- "avg": 0.7766,
599
- "ci": [
600
- 0.6867,
601
- 0.8519
602
- ],
603
- "base": false,
604
- "usd": null,
605
- "stale_verifier": false,
606
- "critical_rate": 0.375,
607
- "latency_p50": 8.329,
608
- "simulated": false,
609
- "invalid": false,
610
- "rescore": false,
611
- "by_axis": {
612
- "sql_execution": 0.75,
613
- "turkish_understanding": 0.7042,
614
- "turkish_summary": 0.85,
615
- "chart_spec": 1.0
616
- },
617
- "delta": 0.2031,
618
- "delta_ci": [
619
- 0.1234,
620
- 0.2801
621
- ],
622
- "regressed_axes": []
623
- },
624
- {
625
- "run": "loop / eval",
626
- "model": "run4-loop-k5r2",
627
- "role": "experiment",
628
- "avg": 0.8812,
629
- "ci": [
630
- 0.8187,
631
- 0.9352
632
- ],
633
- "base": false,
634
- "usd": null,
635
- "stale_verifier": false,
636
- "critical_rate": 0.1562,
637
- "latency_p50": 27.918,
638
- "simulated": false,
639
- "invalid": false,
640
- "rescore": false,
641
- "by_axis": {
642
- "sql_execution": 0.8833,
643
- "turkish_understanding": 0.85,
644
- "turkish_summary": 0.85,
645
- "chart_spec": 1.0
646
- },
647
- "delta": 0.3078,
648
- "delta_ci": [
649
- 0.2412,
650
- 0.3657
651
- ],
652
- "regressed_axes": []
653
- },
654
- {
655
- "run": "grpo2 / dev",
656
- "model": "grpo2",
657
- "role": "experiment",
658
- "avg": 0.7812,
659
- "ci": [
660
- 0.6946,
661
- 0.8516
662
- ],
663
- "base": false,
664
- "usd": null,
665
- "stale_verifier": false,
666
- "critical_rate": 0.325,
667
- "latency_p50": 9.489,
668
- "simulated": false,
669
- "invalid": false,
670
- "rescore": false,
671
- "by_axis": {
672
- "sql_execution": 0.75,
673
- "turkish_understanding": 0.725,
674
- "turkish_summary": 0.825,
675
- "chart_spec": 1.0
676
- },
677
- "delta": 0.2078,
678
- "delta_ci": [
679
- 0.1346,
680
- 0.2801
681
- ],
682
- "regressed_axes": []
683
- },
684
- {
685
- "run": "loop / eval",
686
- "model": "grpo2-loop-k5r2",
687
- "role": "experiment",
688
- "avg": 0.8938,
689
- "ci": [
690
- 0.8187,
691
- 0.9537
692
- ],
693
- "base": false,
694
- "usd": null,
695
- "stale_verifier": false,
696
- "critical_rate": 0.1688,
697
- "latency_p50": 28.351,
698
- "simulated": false,
699
- "invalid": false,
700
- "rescore": false,
701
- "by_axis": {
702
- "sql_execution": 0.8917,
703
- "turkish_understanding": 0.85,
704
- "turkish_summary": 0.925,
705
- "chart_spec": 1.0
706
- },
707
- "delta": 0.3203,
708
- "delta_ci": [
709
- 0.2444,
710
- 0.3866
711
- ],
712
- "regressed_axes": []
713
- },
714
- {
715
- "run": "rft2 / dev",
716
- "model": "rft2",
717
- "role": "experiment",
718
- "avg": 0.7969,
719
- "ci": [
720
- 0.7016,
721
- 0.875
722
- ],
723
- "base": false,
724
- "usd": null,
725
- "stale_verifier": false,
726
- "critical_rate": 0.3312,
727
- "latency_p50": 11.448,
728
- "simulated": false,
729
- "invalid": false,
730
- "rescore": false,
731
- "by_axis": {
732
- "sql_execution": 0.775,
733
- "turkish_understanding": 0.6958,
734
- "turkish_summary": 0.9625,
735
- "chart_spec": 1.0
736
- },
737
- "delta": 0.2234,
738
- "delta_ci": [
739
- 0.1456,
740
- 0.3002
741
- ],
742
- "regressed_axes": []
743
- },
744
- {
745
- "run": "loop / eval",
746
- "model": "rft2-loop-rft2-k5r2",
747
- "role": "experiment",
748
- "avg": 0.8875,
749
- "ci": [
750
- 0.787,
751
- 0.9563
752
- ],
753
- "base": false,
754
- "usd": null,
755
- "stale_verifier": false,
756
- "critical_rate": 0.1437,
757
- "latency_p50": 35.439,
758
- "simulated": false,
759
- "invalid": false,
760
- "rescore": false,
761
- "by_axis": {
762
- "sql_execution": 0.8583,
763
- "turkish_understanding": 0.8583,
764
- "turkish_summary": 0.95,
765
- "chart_spec": 1.0
766
- },
767
- "delta": 0.3141,
768
- "delta_ci": [
769
- 0.2484,
770
- 0.3758
771
- ],
772
- "regressed_axes": []
773
- },
774
- {
775
- "run": "loop / eval",
776
- "model": "rft2-loop-rft2-k1r2-t0",
777
- "role": "experiment",
778
- "avg": 0.8625,
779
- "ci": [
780
- 0.784,
781
- 0.9313
782
- ],
783
- "base": false,
784
- "usd": null,
785
- "stale_verifier": false,
786
- "critical_rate": 0.1375,
787
- "latency_p50": 6.566,
788
- "simulated": false,
789
- "invalid": false,
790
- "rescore": false,
791
- "by_axis": {
792
- "sql_execution": 0.8667,
793
- "turkish_understanding": 0.7667,
794
- "turkish_summary": 1.0,
795
- "chart_spec": 1.0
796
- },
797
- "delta": 0.2891,
798
- "delta_ci": [
799
- 0.2152,
800
- 0.3611
801
- ],
802
- "regressed_axes": []
803
- }
804
- ],
805
- "holdout": false
806
- },
807
- {
808
- "cases": "knowledge/forgetting/general_tr_en.jsonl",
809
- "n": 240,
810
- "cases_hash": "sha256:6ae5e67a1b455c4d159ab43502b62c0c55959aae39342715e9be42680fe1068a",
811
- "axes": [
812
- "forgetting"
813
- ],
814
- "baseline": "base / eval-forgetting",
815
- "rows": [
816
- {
817
- "run": "base / eval-forgetting",
818
- "model": "google/gemma-4-E4B-it",
819
- "role": "student",
820
- "avg": 0.8667,
821
- "ci": [
822
- 0.8208,
823
- 0.9083
824
- ],
825
- "base": true,
826
- "usd": null,
827
- "stale_verifier": false,
828
- "critical_rate": 0.0,
829
- "latency_p50": 0.595,
830
- "simulated": false,
831
- "invalid": false,
832
- "rescore": false,
833
- "by_axis": {
834
- "forgetting": 0.8667
835
- }
836
- },
837
- {
838
- "run": "run1 / eval-forgetting",
839
- "model": "run1",
840
- "role": "experiment",
841
- "avg": 0.0,
842
- "ci": [
843
- 0.0,
844
- 0.0
845
- ],
846
- "base": false,
847
- "usd": null,
848
- "stale_verifier": false,
849
- "critical_rate": 0.0,
850
- "latency_p50": null,
851
- "simulated": false,
852
- "invalid": true,
853
- "rescore": false,
854
- "by_axis": {
855
- "forgetting": 0.0
856
- },
857
- "delta": -0.8667,
858
- "delta_ci": [
859
- -0.9083,
860
- -0.8208
861
- ],
862
- "regressed_axes": [
863
- "forgetting"
864
- ]
865
- },
866
- {
867
- "run": "run1 / forgetting",
868
- "model": "run1",
869
- "role": "experiment",
870
- "avg": 0.7875,
871
- "ci": [
872
- 0.7292,
873
- 0.8375
874
- ],
875
- "base": false,
876
- "usd": null,
877
- "stale_verifier": false,
878
- "critical_rate": 0.0,
879
- "latency_p50": 0.599,
880
- "simulated": false,
881
- "invalid": false,
882
- "rescore": false,
883
- "by_axis": {
884
- "forgetting": 0.7875
885
- },
886
- "delta": -0.0792,
887
- "delta_ci": [
888
- -0.125,
889
- -0.0375
890
- ],
891
- "regressed_axes": [
892
- "forgetting"
893
- ]
894
- },
895
- {
896
- "run": "run2 / forgetting",
897
- "model": "run2",
898
- "role": "experiment",
899
- "avg": 0.7875,
900
- "ci": [
901
- 0.7333,
902
- 0.8375
903
- ],
904
- "base": false,
905
- "usd": null,
906
- "stale_verifier": false,
907
- "critical_rate": 0.0,
908
- "latency_p50": 0.611,
909
- "simulated": false,
910
- "invalid": false,
911
- "rescore": false,
912
- "by_axis": {
913
- "forgetting": 0.7875
914
- },
915
- "delta": -0.0792,
916
- "delta_ci": [
917
- -0.1208,
918
- -0.0417
919
- ],
920
- "regressed_axes": [
921
- "forgetting"
922
- ]
923
- },
924
- {
925
- "run": "rft1 / forgetting",
926
- "model": "rft1",
927
- "role": "experiment",
928
- "avg": 0.8042,
929
- "ci": [
930
- 0.75,
931
- 0.8542
932
- ],
933
- "base": false,
934
- "usd": null,
935
- "stale_verifier": false,
936
- "critical_rate": 0.0,
937
- "latency_p50": 0.631,
938
- "simulated": false,
939
- "invalid": false,
940
- "rescore": false,
941
- "by_axis": {
942
- "forgetting": 0.8042
943
- },
944
- "delta": -0.0625,
945
- "delta_ci": [
946
- -0.1125,
947
- -0.0167
948
- ],
949
- "regressed_axes": [
950
- "forgetting"
951
- ]
952
- },
953
- {
954
- "run": "rft2 / forgetting",
955
- "model": "rft2",
956
- "role": "experiment",
957
- "avg": 0.85,
958
- "ci": [
959
- 0.8,
960
- 0.8958
961
- ],
962
- "base": false,
963
- "usd": null,
964
- "stale_verifier": false,
965
- "critical_rate": 0.0,
966
- "latency_p50": 0.665,
967
- "simulated": false,
968
- "invalid": false,
969
- "rescore": false,
970
- "by_axis": {
971
- "forgetting": 0.85
972
- },
973
- "delta": -0.0167,
974
- "delta_ci": [
975
- -0.0667,
976
- 0.0292
977
- ],
978
- "regressed_axes": []
979
- }
980
- ],
981
- "holdout": false
982
- },
983
- {
984
- "cases": "runs/teacher-full/cases.jsonl",
985
- "n": 160,
986
- "cases_hash": "sha256:5c7987db982feda07b5a1262fedb91f4453dcfea7f46f8e8605701d2d279b5ff",
987
- "axes": [
988
- "chart_spec",
989
- "sql_execution",
990
- "turkish_summary",
991
- "turkish_understanding"
992
- ],
993
- "baseline": null,
994
- "rows": [
995
- {
996
- "run": "bakeoff-openrouter-z-ai_glm-5.3-flash / openrouter-z-ai_glm-5.3-flash",
997
- "model": "z-ai/glm-5.3-flash",
998
- "role": "frontier",
999
- "avg": 0.9938,
1000
- "ci": [
1001
- 0.9812,
1002
- 1.0
1003
- ],
1004
- "base": false,
1005
- "usd": null,
1006
- "stale_verifier": false,
1007
- "critical_rate": 0.0,
1008
- "latency_p50": 10.628,
1009
- "simulated": false,
1010
- "invalid": false,
1011
- "rescore": false,
1012
- "by_axis": {
1013
- "sql_execution": 1.0,
1014
- "turkish_summary": 1.0,
1015
- "turkish_understanding": 0.9833,
1016
- "chart_spec": 1.0
1017
- }
1018
- },
1019
- {
1020
- "run": "bakeoff-openrouter-google_gemma-4-31b-it / openrouter-google_gemma-4-31b-it",
1021
- "model": "google/gemma-4-31b-it",
1022
- "role": "frontier",
1023
- "avg": 0.9313,
1024
- "ci": [
1025
- 0.8938,
1026
- 0.9688
1027
- ],
1028
- "base": false,
1029
- "usd": null,
1030
- "stale_verifier": false,
1031
- "critical_rate": 0.0688,
1032
- "latency_p50": 5.132,
1033
- "simulated": false,
1034
- "invalid": false,
1035
- "rescore": false,
1036
- "by_axis": {
1037
- "sql_execution": 0.9333,
1038
- "turkish_summary": 0.8,
1039
- "turkish_understanding": 0.95,
1040
- "chart_spec": 1.0
1041
- }
1042
- }
1043
- ],
1044
- "holdout": false
1045
- },
1046
- {
1047
- "cases": "data/prompts/rft_cases.jsonl",
1048
- "n": 1567,
1049
- "cases_hash": "sha256:9c395458b332021d9783066dd87b137c8e431d15e2a6477af8dd7c108e742689",
1050
- "axes": [
1051
- "report",
1052
- "sql_execution"
1053
- ],
1054
- "baseline": null,
1055
- "rows": [
1056
- {
1057
- "run": "run2 / train-samples",
1058
- "model": "run2",
1059
- "role": "experiment",
1060
- "avg": 0.7705,
1061
- "ci": [
1062
- 0.752,
1063
- 0.7892
1064
- ],
1065
- "base": false,
1066
- "usd": null,
1067
- "stale_verifier": false,
1068
- "critical_rate": 0.344,
1069
- "latency_p50": 3.367,
1070
- "simulated": false,
1071
- "invalid": false,
1072
- "rescore": false,
1073
- "by_axis": {
1074
- "sql_execution": 0.7406,
1075
- "report": 0.8746
1076
- }
1077
- }
1078
- ],
1079
- "holdout": false
1080
- },
1081
- {
1082
- "cases": "eval/cases/holdout.jsonl",
1083
- "n": 100,
1084
- "cases_hash": "sha256:2daa51c6b49b153cea03d9fdf22ed1c6c1fe5211b278c5bf95022b06e7895e00",
1085
- "axes": [
1086
- "chart_spec",
1087
- "sql_execution",
1088
- "turkish_summary",
1089
- "turkish_understanding"
1090
- ],
1091
- "baseline": "base / holdout",
1092
- "rows": [
1093
- {
1094
- "run": "base / holdout",
1095
- "model": "google/gemma-4-E4B-it",
1096
- "role": "student",
1097
- "avg": 0.5325,
1098
- "ci": [
1099
- 0.41,
1100
- 0.655
1101
- ],
1102
- "base": true,
1103
- "usd": null,
1104
- "stale_verifier": false,
1105
- "critical_rate": 0.5,
1106
- "latency_p50": 6.296,
1107
- "simulated": false,
1108
- "invalid": false,
1109
- "rescore": false,
1110
- "by_axis": {
1111
- "sql_execution": 0.5312,
1112
- "turkish_understanding": 0.375,
1113
- "turkish_summary": 0.9,
1114
- "chart_spec": 0.8
1115
- }
1116
- },
1117
- {
1118
- "run": "rft1 / holdout",
1119
- "model": "rft1",
1120
- "role": "experiment",
1121
- "avg": 0.675,
1122
- "ci": [
1123
- 0.615,
1124
- 0.7425
1125
- ],
1126
- "base": false,
1127
- "usd": null,
1128
- "stale_verifier": false,
1129
- "critical_rate": 0.34,
1130
- "latency_p50": 6.162,
1131
- "simulated": false,
1132
- "invalid": false,
1133
- "rescore": false,
1134
- "by_axis": {
1135
- "sql_execution": 0.65,
1136
- "turkish_understanding": 0.5625,
1137
- "turkish_summary": 0.9,
1138
- "chart_spec": 1.0
1139
- },
1140
- "delta": 0.1425,
1141
- "delta_ci": [
1142
- 0.08,
1143
- 0.205
1144
- ],
1145
- "regressed_axes": []
1146
- }
1147
- ],
1148
- "holdout": true
1149
- },
1150
- {
1151
- "cases": "eval/cases/holdout2.jsonl",
1152
- "n": 125,
1153
- "cases_hash": "sha256:bc7165e82acdef0f3470936c85b785c9cf0b4f404aff2f5225def6a82d8625b5",
1154
- "axes": [
1155
- "chart_spec",
1156
- "sql_execution",
1157
- "turkish_summary",
1158
- "turkish_understanding"
1159
- ],
1160
- "baseline": "base / holdout2",
1161
- "rows": [
1162
- {
1163
- "run": "rft1 / holdout2",
1164
- "model": "rft1",
1165
- "role": "experiment",
1166
- "avg": 0.75,
1167
- "ci": [
1168
- 0.672,
1169
- 0.824
1170
- ],
1171
- "base": false,
1172
- "usd": null,
1173
- "stale_verifier": false,
1174
- "critical_rate": 0.336,
1175
- "latency_p50": 6.553,
1176
- "simulated": false,
1177
- "invalid": false,
1178
- "rescore": false,
1179
- "by_axis": {
1180
- "sql_execution": 0.71,
1181
- "turkish_understanding": 0.68,
1182
- "turkish_summary": 0.9423,
1183
- "chart_spec": 1.0
1184
- },
1185
- "delta": 0.19,
1186
- "delta_ci": [
1187
- 0.054,
1188
- 0.31
1189
- ],
1190
- "regressed_axes": []
1191
- },
1192
- {
1193
- "run": "loop / eval",
1194
- "model": "rft1-loop-k5r2",
1195
- "role": "experiment",
1196
- "avg": 0.796,
1197
- "ci": [
1198
- 0.732,
1199
- 0.856
1200
- ],
1201
- "base": false,
1202
- "usd": null,
1203
- "stale_verifier": false,
1204
- "critical_rate": 0.248,
1205
- "latency_p50": 13.693,
1206
- "simulated": false,
1207
- "invalid": false,
1208
- "rescore": false,
1209
- "by_axis": {
1210
- "sql_execution": 0.8,
1211
- "turkish_understanding": 0.73,
1212
- "turkish_summary": 0.8462,
1213
- "chart_spec": 1.0
1214
- },
1215
- "delta": 0.236,
1216
- "delta_ci": [
1217
- 0.114,
1218
- 0.344
1219
- ],
1220
- "regressed_axes": []
1221
- },
1222
- {
1223
- "run": "run4 / holdout2",
1224
- "model": "run4",
1225
- "role": "experiment",
1226
- "avg": 0.744,
1227
- "ci": [
1228
- 0.68,
1229
- 0.818
1230
- ],
1231
- "base": false,
1232
- "usd": null,
1233
- "stale_verifier": false,
1234
- "critical_rate": 0.376,
1235
- "latency_p50": 9.291,
1236
- "simulated": false,
1237
- "invalid": false,
1238
- "rescore": false,
1239
- "by_axis": {
1240
- "sql_execution": 0.715,
1241
- "turkish_understanding": 0.68,
1242
- "turkish_summary": 0.8654,
1243
- "chart_spec": 1.0
1244
- },
1245
- "delta": 0.184,
1246
- "delta_ci": [
1247
- 0.086,
1248
- 0.294
1249
- ],
1250
- "regressed_axes": []
1251
- },
1252
- {
1253
- "run": "loop / eval",
1254
- "model": "run4-loop-k5r2",
1255
- "role": "experiment",
1256
- "avg": 0.868,
1257
- "ci": [
1258
- 0.824,
1259
- 0.92
1260
- ],
1261
- "base": false,
1262
- "usd": null,
1263
- "stale_verifier": false,
1264
- "critical_rate": 0.2,
1265
- "latency_p50": 31.14,
1266
- "simulated": false,
1267
- "invalid": false,
1268
- "rescore": false,
1269
- "by_axis": {
1270
- "sql_execution": 0.91,
1271
- "turkish_understanding": 0.81,
1272
- "turkish_summary": 0.8077,
1273
- "chart_spec": 1.0
1274
- },
1275
- "delta": 0.308,
1276
- "delta_ci": [
1277
- 0.216,
1278
- 0.372
1279
- ],
1280
- "regressed_axes": []
1281
- },
1282
- {
1283
- "run": "base / holdout2",
1284
- "model": "google/gemma-4-E4B-it",
1285
- "role": "student",
1286
- "avg": 0.56,
1287
- "ci": [
1288
- 0.514,
1289
- 0.618
1290
- ],
1291
- "base": true,
1292
- "usd": null,
1293
- "stale_verifier": false,
1294
- "critical_rate": 0.496,
1295
- "latency_p50": 6.226,
1296
- "simulated": false,
1297
- "invalid": false,
1298
- "rescore": false,
1299
- "by_axis": {
1300
- "sql_execution": 0.53,
1301
- "turkish_understanding": 0.445,
1302
- "turkish_summary": 0.7115,
1303
- "chart_spec": 1.0
1304
- }
1305
- }
1306
- ],
1307
- "holdout": true
1308
- }
1309
- ],
1310
- "best": {
1311
- "run": "loop / eval",
1312
- "simulated": false,
1313
- "avg": 0.8938,
1314
- "ci": [
1315
- 0.8187,
1316
- 0.9537
1317
- ],
1318
- "delta": 0.3203,
1319
- "delta_ci": [
1320
- 0.2444,
1321
- 0.3866
1322
- ],
1323
- "left_to_target": null
1324
- }
1325
- }