Interchained commited on
Commit
4df1e20
·
verified ·
1 Parent(s): b741596

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +24 -18
README.md CHANGED
@@ -142,24 +142,30 @@ A previously reported v12 result of 40/40 on a specific held-out evaluation is *
142
  - The precise upstream licensing terms and redistribution permissions should be verified before publication. Replace the provisional `license: other` metadata with the correct Hugging Face license identifier or a linked license notice once confirmed.
143
 
144
  ## ================ FINAL EVAL ================
145
- imagine-v12 light tooluse 25/55 (45.5%)
146
- imagine-v12 light temporal 0/15 (0.0%)
147
- imagine-v12 light discrimination 0/120 (0.0%)
148
- imagine-v12 light ALL 25/190 (13.2%)
149
- imagine-v12 bare tooluse 25/55 (45.5%)
150
- imagine-v12 bare temporal 0/15 (0.0%)
151
- imagine-v12 bare discrimination 3/120 (2.5%)
152
- imagine-v12 bare ALL 28/190 (14.7%)
153
- imagine-v12 both ALL 53/380 (13.9%)
154
- imagine-v13 light tooluse 26/55 (47.3%)
155
- imagine-v13 light temporal 6/15 (40.0%)
156
- imagine-v13 light discrimination 53/120 (44.2%)
157
- imagine-v13 light ALL 85/190 (44.7%)
158
- imagine-v13 bare tooluse 25/55 (45.5%)
159
- imagine-v13 bare temporal 8/15 (53.3%)
160
- imagine-v13 bare discrimination 48/120 (40.0%)
161
- imagine-v13 bare ALL 81/190 (42.6%)
162
- imagine-v13 both ALL 166/380 (43.7%)
 
 
 
 
 
 
163
 
164
 
165
  ## Provenance
 
142
  - The precise upstream licensing terms and redistribution permissions should be verified before publication. Replace the provisional `license: other` metadata with the correct Hugging Face license identifier or a linked license notice once confirmed.
143
 
144
  ## ================ FINAL EVAL ================
145
+
146
+ Measured 2026-11-03, after training, on the held-out eval split of the v13 corpus: **310 fixtures** (tool use 55, temporal grounding 15, discrimination 240 = 120 minimal pairs × 2 variants) whose schemas and tables appear **nowhere in training**. Both models were evaluated on identical fixtures, identical decoding (temperature 0, Q4_K_M GGUF, served side by side through the Hearth gateway), under two system-prompt conditions:
147
+
148
+ - **light** — one system line: "You are Imagine, a text-to-SQL assistant built by Interchained."
149
+ - **bare** — no system prompt at all.
150
+
151
+ | Family | v12 light | v12 bare | v13 light | v13 bare |
152
+ |---|---|---|---|---|
153
+ | Tool use (55) | 25/55 (45.5%) | 25/55 (45.5%) | 26/55 (47.3%) | 25/55 (45.5%) |
154
+ | Temporal (15) | 0/15 (0.0%) | 0/15 (0.0%) | 5/15 (33.3%) | 7/15 (46.7%) |
155
+ | Discrimination (240) | 0/240 (0.0%) | 3/240 (1.2%) | 116/240 (48.3%) | 103/240 (42.9%) |
156
+ | **All (310 per condition)** | **25/310 (8.1%)** | **28/310 (9.0%)** | **147/310 (47.4%)** | **135/310 (43.5%)** |
157
+
158
+ **Combined: v12 53/620 (8.5%) — v13 282/620 (45.5%).**
159
+
160
+ Scoring is deterministic and per family: discrimination by response mode (SQL vs single tool call vs tool chain, plus the expected tool name where one applies), tool use by behavior (call the right tool, abstain, clarify, or declare unanswerable), temporal by normalized SQL match against the reference.
161
+
162
+ How to read this, plainly:
163
+
164
+ - The gains are exactly where v13 aimed. Discrimination went from effectively zero to roughly half; temporal grounding went from zero to a third (light) and nearly half (bare). v13's new skills are in the weights, not the prompt: on temporal and discrimination it scores as well or better with **no system prompt** as with one.
165
+ - Tool use is a tie. These fixtures were built for the corpus's native tool-protocol system prompt, which this evaluation deliberately withholds, so both models are understated equally. The comparison is fair; the absolute level is pessimistic.
166
+ - Discrimination at ~45% also means the mode call is still wrong more often than right. v13 is a large, measured improvement over v12 on the families it was built for. It is not a finished model, and this card will not pretend otherwise.
167
+
168
+ Not yet measured for v13: the Tier-0 held-out telemetry suite (v12's 40/40 was on v12's own fixtures and is not a v13 number) and multi-turn behavior. They will be added here when they exist, the same way these were.
169
 
170
 
171
  ## Provenance