Vveerrgg commited on
Commit
8f78b29
·
verified ·
1 Parent(s): 8c89ef0

card: recipe-only numbers throughout

Browse files
Files changed (1) hide show
  1. README.md +6 -5
README.md CHANGED
@@ -72,8 +72,9 @@ only, every training record traceable to its source, and the checks published al
72
 
73
  ## Results
74
 
75
- All cooking scores are on a fixed held-out set of 682 questions (300 with the recipe shown, 300 that need a recipe
76
- the model is not shown, 82 general questions), in 26 languages, never used in training. Answers from base Qwen3-8B
 
77
  (thinking off) and this model are compared by an open-weights judge from a different model family
78
  (DeepSeek-V4-Flash, MIT licence), blind, in both orders. 95% confidence intervals in brackets.
79
 
@@ -81,7 +82,7 @@ the model is not shown, 82 general questions), in 26 languages, never used in tr
81
  |---|---|---|
82
  | **Invents recipe details when no recipe is shown** | 30.0% [25.1, 35.4] | **9.0%** [6.3, 12.8] |
83
  | same, after a rule check that removes judge false alarms | 27.7% | **4.0%** [2.3, 6.9] |
84
- | **Correct when the recipe is shown** | 84.7% | **93.0%** |
85
  | Preferred by the judge, all 682 questions | — | **0.610** [0.581, 0.639] (0.50 = tie) |
86
  | Preferred by the judge, general questions only | — | 0.460 [0.369, 0.549] (level with base) |
87
 
@@ -118,7 +119,7 @@ itself, or a common default ("a pinch of salt") offered with a hedge.
118
  | | Q4_K_M | Q6_K | Q8_0 |
119
  |---|---|---|---|
120
  | Judge preference vs full precision | 0.492 [0.475, 0.509] | 0.501 [0.485, 0.517] | 0.481 [0.466, 0.494] |
121
- | Correct when the recipe is shown | 94.0% | 93.3% | 93.7% |
122
  | Invents with no recipe (full precision on that run) | 9.3% (7.3%) | 6.3% (6.3%) | 6.7% (6.7%) |
123
  | Invents with no recipe, 3 of 4 judges (full precision) | 4.7% (3.3%) | 3.0% (3.3%) | 3.3% (2.7%) |
124
  | Correct when the recipe is shown, 3 of 4 judges (full precision) | 95.5% (94.1%) | 95.5% (94.1%) | 95.2% (94.8%) |
@@ -127,7 +128,7 @@ With all four judges, no quantised file differs significantly from full precisio
127
  Q4_K_M p = 0.34, Q6_K p = 1.0, Q8_0 p = 0.69 for inventing; p ≥ 0.22 for correctness with the recipe shown). On Q4_K_M two of the four judges on their own
128
  (GLM-4.7-Flash, gpt-oss-120b) do see slightly more invented details (p = 0.02 and 0.04); pick Q6_K or Q8_0 if that matters.
129
 
130
- No quantisation changes the invents rate significantly (McNemar p ≥ 0.29). Q8_0's small dip comes from the 82
131
  general questions (0.418) and is most likely judge noise on a small set; Q4_K_M and Q6_K, which remove more bits,
132
  sit at parity.
133
 
 
72
 
73
  ## Results
74
 
75
+ All cooking scores are on a fixed held-out set of 682 questions in 26 languages, never used in training: 290 with the
76
+ recipe shown, 10 recall questions with no recipe ("how do you make X?"), 300 that need a recipe the model is not shown,
77
+ and 82 general questions. Answers from base Qwen3-8B
78
  (thinking off) and this model are compared by an open-weights judge from a different model family
79
  (DeepSeek-V4-Flash, MIT licence), blind, in both orders. 95% confidence intervals in brackets.
80
 
 
82
  |---|---|---|
83
  | **Invents recipe details when no recipe is shown** | 30.0% [25.1, 35.4] | **9.0%** [6.3, 12.8] |
84
  | same, after a rule check that removes judge false alarms | 27.7% | **4.0%** [2.3, 6.9] |
85
+ | **Correct when the recipe is shown** (290 questions) | 87.6% | **96.2%** |
86
  | Preferred by the judge, all 682 questions | — | **0.610** [0.581, 0.639] (0.50 = tie) |
87
  | Preferred by the judge, general questions only | — | 0.460 [0.369, 0.549] (level with base) |
88
 
 
119
  | | Q4_K_M | Q6_K | Q8_0 |
120
  |---|---|---|---|
121
  | Judge preference vs full precision | 0.492 [0.475, 0.509] | 0.501 [0.485, 0.517] | 0.481 [0.466, 0.494] |
122
+ | Correct when the recipe is shown, 290 questions (full precision on that run) | 97.2% (96.6%) | 96.6% (96.2%) | 96.9% (96.2%) |
123
  | Invents with no recipe (full precision on that run) | 9.3% (7.3%) | 6.3% (6.3%) | 6.7% (6.7%) |
124
  | Invents with no recipe, 3 of 4 judges (full precision) | 4.7% (3.3%) | 3.0% (3.3%) | 3.3% (2.7%) |
125
  | Correct when the recipe is shown, 3 of 4 judges (full precision) | 95.5% (94.1%) | 95.5% (94.1%) | 95.2% (94.8%) |
 
128
  Q4_K_M p = 0.34, Q6_K p = 1.0, Q8_0 p = 0.69 for inventing; p ≥ 0.22 for correctness with the recipe shown). On Q4_K_M two of the four judges on their own
129
  (GLM-4.7-Flash, gpt-oss-120b) do see slightly more invented details (p = 0.02 and 0.04); pick Q6_K or Q8_0 if that matters.
130
 
131
+ With the DeepSeek judge alone, no quantisation changes the invents rate significantly (McNemar p ≥ 0.29). Q8_0's small dip comes from the 82
132
  general questions (0.418) and is most likely judge noise on a small set; Q4_K_M and Q6_K, which remove more bits,
133
  sit at parity.
134