Vveerrgg commited on
Commit
8c89ef0
·
verified ·
1 Parent(s): 0cb6244

card: recipe-shown correctness reported apart (94.8% vs 83.1%), quant rows corrected

Browse files
Files changed (1) hide show
  1. README.md +9 -2
README.md CHANGED
@@ -98,6 +98,13 @@ applied to all four, the share of no-recipe answers that invent recipe details:
98
  | all four say so | 19.1% | 1.0% |
99
  | any one says so | 49.2% | 18.3% |
100
 
 
 
 
 
 
 
 
101
  Every judge, and every way of combining them, shows a significant drop (3 of 4: 72 questions where only the base
102
  invents, 3 where only this model does, McNemar p < 1e-17). The judges agree moderately on base answers (Fleiss kappa
103
  0.63). GLM-4.7-Flash flags more of this model's answers than the others; reading them, most repeat or restate a line
@@ -114,10 +121,10 @@ itself, or a common default ("a pinch of salt") offered with a hedge.
114
  | Correct when the recipe is shown | 94.0% | 93.3% | 93.7% |
115
  | Invents with no recipe (full precision on that run) | 9.3% (7.3%) | 6.3% (6.3%) | 6.7% (6.7%) |
116
  | Invents with no recipe, 3 of 4 judges (full precision) | 4.7% (3.3%) | 3.0% (3.3%) | 3.3% (2.7%) |
117
- | Correct when the recipe is shown, 3 of 4 judges (full precision) | 81.4% (80.4%) | 82.7% (80.6%) | 80.4% (80.6%) |
118
 
119
  With all four judges, no quantised file differs significantly from full precision (McNemar on the 3-of-4 verdict:
120
- Q4_K_M p = 0.34, Q6_K p = 1.0, Q8_0 p = 0.69 for inventing). On Q4_K_M two of the four judges on their own
121
  (GLM-4.7-Flash, gpt-oss-120b) do see slightly more invented details (p = 0.02 and 0.04); pick Q6_K or Q8_0 if that matters.
122
 
123
  No quantisation changes the invents rate significantly (McNemar p ≥ 0.29). Q8_0's small dip comes from the 82
 
98
  | all four say so | 19.1% | 1.0% |
99
  | any one says so | 49.2% | 18.3% |
100
 
101
+ The same four judges on the questions where the recipe **is** shown and the model has to read it (290 questions; 3 of
102
+ 4 must agree): Qwen3-8B correct 83.1% [78.4, 87.0], this model **94.8%** [91.6, 96.8] (40 questions only this model
103
+ gets right, 6 only the base does, McNemar p < 1e-6). Two other kinds of question in the set are reported apart:
104
+ 82 general questions graded against a reference answer (40.2% for both models; the judges disagree widely there) and
105
+ 10 "how do you make X?" questions with no recipe whose reference is the original 1800s text (0% for both: the model
106
+ is built not to recall historical recipes from memory).
107
+
108
  Every judge, and every way of combining them, shows a significant drop (3 of 4: 72 questions where only the base
109
  invents, 3 where only this model does, McNemar p < 1e-17). The judges agree moderately on base answers (Fleiss kappa
110
  0.63). GLM-4.7-Flash flags more of this model's answers than the others; reading them, most repeat or restate a line
 
121
  | Correct when the recipe is shown | 94.0% | 93.3% | 93.7% |
122
  | Invents with no recipe (full precision on that run) | 9.3% (7.3%) | 6.3% (6.3%) | 6.7% (6.7%) |
123
  | Invents with no recipe, 3 of 4 judges (full precision) | 4.7% (3.3%) | 3.0% (3.3%) | 3.3% (2.7%) |
124
+ | Correct when the recipe is shown, 3 of 4 judges (full precision) | 95.5% (94.1%) | 95.5% (94.1%) | 95.2% (94.8%) |
125
 
126
  With all four judges, no quantised file differs significantly from full precision (McNemar on the 3-of-4 verdict:
127
+ Q4_K_M p = 0.34, Q6_K p = 1.0, Q8_0 p = 0.69 for inventing; p ≥ 0.22 for correctness with the recipe shown). On Q4_K_M two of the four judges on their own
128
  (GLM-4.7-Flash, gpt-oss-120b) do see slightly more invented details (p = 0.02 and 0.04); pick Q6_K or Q8_0 if that matters.
129
 
130
  No quantisation changes the invents rate significantly (McNemar p ≥ 0.29). Q8_0's small dip comes from the 82