zeechimp commited on
Commit
62b33c7
·
verified ·
1 Parent(s): 6ddc33e

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +410 -0
README.md ADDED
@@ -0,0 +1,410 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ library_name: learning-report
4
+ tags:
5
+ - evaluation
6
+ - calibration
7
+ - out-of-distribution
8
+ - model-diagnostics
9
+ - training-data-volume
10
+ - reproducibility
11
+ - numpy
12
+ - cpu
13
+ pipeline_tag: other
14
+ language:
15
+ - en
16
+ ---
17
+
18
+ # learning-report
19
+
20
+ **Two tools for making an honest claim about a model.**
21
+
22
+ A model card says `"our model achieves 97% accuracy."` That number can
23
+ be true and still hide four separate failures: no held-out evaluation,
24
+ no OOD split, a temperature that hit the edge of the search grid, and
25
+ an output distribution that has collapsed to one class. None of these
26
+ show up in the single accuracy number.
27
+
28
+ `learning-report` provides the missing pieces:
29
+
30
+ - **A diagnostic checklist** that produces a report you cannot
31
+ summarise in one number: four splits, mean confidence and ECE on
32
+ each, temperature-boundary detection, output-uniformity canary,
33
+ and calibration thresholds. The verdict counts the issues.
34
+ - **A training-data-volume calculator** that answers "how many
35
+ examples do I need?" before you run the experiment. Two
36
+ estimates: a formula and an empirical ladder with two curve
37
+ families (exponential saturation and Hill cooperativity).
38
+
39
+ Both are pure numpy, train in seconds, and ship as code — there are
40
+ no weights to download.
41
+
42
+ ## The claim in one sentence
43
+
44
+ A model claim that can be summarised in a single accuracy number is
45
+ an incomplete claim; a complete claim names the splits, reports
46
+ calibration on each, and states how many examples the target
47
+ required in the first place.
48
+
49
+ ## What the diagnostic reports
50
+
51
+ Every `LearningReport` contains:
52
+
53
+ - `parameters` — model size
54
+ - `training examples` — dataset size
55
+ - `training accuracy` — fit, not evidence of learning
56
+ - `temperature` — fitted value, with a `[BOUNDARY]` flag if it landed
57
+ on the edge of the search grid
58
+ - `output uniformity` — normalized entropy of the model's prediction
59
+ histogram; low values mean the model has collapsed to one class
60
+ - `mean max-prob` — average top-class probability; near-uniform
61
+ values are not calibration
62
+ - `verdict` — `[ok]`, `[?]`, `[??]`, or `[???]` by issue count
63
+ - `issues` — plain-English list of every problem detected
64
+ - `splits` — four rows, each with `n`, `accuracy`, `mean_confidence`,
65
+ `ece`, and an OOD marker
66
+
67
+ The four splits are:
68
+
69
+ | split | operands | operation | what it tests |
70
+ |---|---|---|---|
71
+ | interpolation | in training range | same | fit + generalization on distribution |
72
+ | near-OOD | one axis extended | same | partial transfer |
73
+ | far-OOD | all outside training range | same | generalization to new inputs |
74
+ | structural-OOD | in training range | different | transfer to a new operation |
75
+
76
+ The last split is the load-bearing one and it is almost never
77
+ reported. It distinguishes "learned the sum rule" from "learned the
78
+ input-output table."
79
+
80
+ ## What the volume calculator reports
81
+
82
+ Two independent estimates of training-set size:
83
+
84
+ **Formula.** `N ≈ k · P / log2(C) · 1 / (1 - target_acc)` with
85
+ `k = 0.05` calibrated on the demo task. Fast, no training, useful for
86
+ the first pass.
87
+
88
+ **Empirical ladder.** Train the model at a ladder of dataset sizes,
89
+ fit a saturating curve to the resulting accuracy, and invert to
90
+ recommend an N. Two curve families:
91
+
92
+ | family | form | when it fits |
93
+ |---|---|---|
94
+ | exponential | `acc(N) = acc_max · (1 - exp(-N / N_half))` | smooth saturation |
95
+ | Hill | `acc(N) = acc_max · N^h / (N_half^h + N^h)` | step-shaped ladder |
96
+
97
+ The Hill exponent `h` is a diagnostic: `h > 1` means the transition
98
+ is cooperative, and the exponential fit is systematically
99
+ overestimating the required N.
100
+
101
+ ## Install
102
+
103
+ ```bash
104
+ pip install numpy
105
+ python learning_report.py
106
+ ```
107
+
108
+ No downloads. No tokenizer files. No model hub. Runs in under a
109
+ minute on CPU.
110
+
111
+ ## Usage
112
+
113
+ ### Diagnostic checklist
114
+
115
+ ```python
116
+ from learning_report import (
117
+ MLP, train, fit_temperature, diagnose, print_report,
118
+ sample_pairs, encode_pairs, labels_add, labels_mul,
119
+ )
120
+
121
+ # Train a model (or bring your own that exposes .predict_proba).
122
+ model = MLP(in_dim=40, hidden=32, out_dim=40, seed=0)
123
+
124
+ # Build four splits as lists of raw (a, b) tuples.
125
+ interp = sample_pairs(0, 9, 0, 9, 80, seed=200)
126
+ near_ood = sample_pairs(0, 9, 10, 14, 60, seed=201)
127
+ far_ood = sample_pairs(10, 19, 10, 19, 60, seed=202)
128
+ struct = sample_pairs(0, 5, 0, 5, 36, seed=203)
129
+
130
+ # diagnose() encodes internally; pass raw pairs, not arrays.
131
+ report = diagnose(
132
+ model,
133
+ train_pairs, y_train,
134
+ interp, labels_add(interp),
135
+ near_ood, labels_add(near_ood),
136
+ far_ood, labels_add(far_ood),
137
+ struct, labels_mul(struct),
138
+ )
139
+ print_report(report)
140
+ ```
141
+
142
+ ### Volume calculator
143
+
144
+ ```python
145
+ from learning_report import (
146
+ estimate_volume_formula,
147
+ estimate_volume_empirical,
148
+ make_add_data,
149
+ make_add_model,
150
+ )
151
+
152
+ # Formula only.
153
+ n = estimate_volume_formula(n_params=2632, n_classes=40,
154
+ target_acc=0.80) # → 124
155
+
156
+ # Empirical ladder with exponential fit.
157
+ recommended_exp, diag_exp = estimate_volume_empirical(
158
+ make_data=make_add_data,
159
+ model_factory=make_add_model,
160
+ N_grid=(16, 32, 64, 128, 256, 512, 1024),
161
+ epochs=500,
162
+ seeds=2,
163
+ target_acc=0.80,
164
+ form="exp",
165
+ )
166
+
167
+ # Empirical ladder with Hill fit.
168
+ recommended_hill, diag_hill = estimate_volume_empirical(
169
+ make_data=make_add_model and make_add_data,
170
+ model_factory=make_add_model,
171
+ target_acc=0.80,
172
+ form="hill",
173
+ )
174
+ ```
175
+
176
+ ## Results from the demo
177
+
178
+ The demo trains a 2,632-parameter MLP on `a + b` for `a, b ∈ [0, 9]`
179
+ with 60 examples.
180
+
181
+ ### Self-test
182
+
183
+ **21/21 checks pass.** The load-bearing ones:
184
+
185
+ | check | guards against |
186
+ |---|---|
187
+ | `threshold: ece fires when ece>0.20` | high ECE going unreported |
188
+ | `threshold: underconf fires when conf<0.40 and acc>0.5` | temperature over-correction going unreported |
189
+ | `curve fit hill beats exp on step data` | the fit family being chosen for the wrong shape |
190
+ | `boundary: T=20 flagged` | temperature hitting the grid edge |
191
+
192
+ ### Diagnostic report
193
+
194
+ ```
195
+ parameters : 2632
196
+ training examples : 60
197
+ training accuracy : 100.0%
198
+ temperature : 4.45
199
+ output uniformity : 0.658
200
+ mean max-prob : 0.244
201
+ verdict : [???]
202
+
203
+ issues:
204
+ - cannot generalise to OOD inputs: far-OOD accuracy 0.00
205
+ - cannot transfer to a different operation: structural-OOD accuracy 0.11
206
+ - poor calibration on interpolation: ECE=0.33 (threshold 0.20)
207
+ - underconfident on interpolation (acc=0.60, conf=0.38); temperature may be over-corrected
208
+
209
+ split n acc mean_conf ECE
210
+ --------------------------------------------------------
211
+ interpolation 80 0.60 0.38 0.33
212
+ near-OOD 50 0.00 0.17 0.17 (OOD)
213
+ far-OOD 60 0.00 0.07 0.07 (OOD)
214
+ structural-OOD 36 0.11 0.39 0.28 (OOD)
215
+ ```
216
+
217
+ A model that fits its training data perfectly (`100.0%` training
218
+ accuracy) and generalizes to `60%` on-distribution, `0%` far-OOD,
219
+ and `11%` structural-OOD. The report names four issues. A model card
220
+ that reported only the training accuracy would be technically true
221
+ and completely misleading.
222
+
223
+ ### Volume calculator
224
+
225
+ **Formula:**
226
+
227
+ | target acc | estimated N |
228
+ |---|---|
229
+ | 0.30 | 36 |
230
+ | 0.50 | 50 |
231
+ | 0.70 | 83 |
232
+ | 0.80 | 124 |
233
+ | 0.90 | 248 |
234
+
235
+ **Empirical ladder:**
236
+
237
+ | N | val_acc |
238
+ |---|---|
239
+ | 16 | 0.165 |
240
+ | 32 | 0.320 |
241
+ | 64 | 0.640 |
242
+ | 128 | 1.000 |
243
+ | 256 | 1.000 |
244
+ | 512 | 1.000 |
245
+ | 1024 | 1.000 |
246
+
247
+ **Two fits:**
248
+
249
+ | family | ceiling | N_half | h | MSE | recommended N for 0.80 |
250
+ |---|---|---|---|---|---|
251
+ | exp | 1.000 | 67.7 | — | 0.0042 | 110 |
252
+ | **hill** | **1.000** | **40.8** | **2.04** | **0.0027** | **81** |
253
+
254
+ The Hill fit has lower MSE (0.0027 vs 0.0042), and the fitted `h = 2.04`
255
+ is the signature of a step, not a smooth curve. The ladder jumps from
256
+ 0.640 at N=64 to 1.000 at N=128; the exponential fit smooths over
257
+ that and recommends 110, while the Hill fit recommends 81.
258
+
259
+ **Verification:**
260
+
261
+ | N | held-out accuracy | target met? |
262
+ |---|---|---|
263
+ | 81 (hill) | 0.810 | yes, exactly |
264
+ | 110 (exp) | 1.000 | yes, overshot |
265
+ | 124 (formula) | 1.000 | yes, overshot |
266
+
267
+ All three recommendations land above the target. The Hill
268
+ recommendation is the most economical — it hits the target without
269
+ overshooting — because the fit correctly identified the ladder as a
270
+ step rather than a smooth curve.
271
+
272
+ ## The bug class
273
+
274
+ `"Our model achieves X% accuracy."`
275
+
276
+ The single number hides:
277
+
278
+ - which split the X% was measured on
279
+ - whether any OOD split was tested
280
+ - whether confidence is calibrated
281
+ - whether the temperature fit degenerated
282
+ - whether the output distribution collapsed
283
+ - how many examples the target required in the first place
284
+
285
+ Every one of these can be wrong while X% is true.
286
+
287
+ ## The six defenses
288
+
289
+ | level | mechanism | action |
290
+ |---|---|---|
291
+ | 1. split | four evaluation axes | OOD is measured, not assumed |
292
+ | 2. calibrate | mean confidence + ECE on each split | confidence claims are checked |
293
+ | 3. canary | output uniformity | degenerate models are flagged |
294
+ | 4. bound | temperature boundary check | degenerate calibration is flagged |
295
+ | 5. threshold | ECE and underconfidence thresholds | calibration issues become issues |
296
+ | 6. plan | volume calculator (formula + empirical) | the target N is known before training |
297
+
298
+ Any evaluation can choose its level. The choice is explicit in the
299
+ report.
300
+
301
+ ## When to use it
302
+
303
+ - **Before publishing any accuracy number.** Run the diagnostic and
304
+ quote the full report, not the top line.
305
+ - **Before running an expensive training job.** The volume
306
+ calculator tells you how many examples the target requires. If
307
+ you do not have them, you have a data problem, not a model
308
+ problem.
309
+ - **When comparing two models.** Same splits, same calibration
310
+ metric, same OOD axes. A model that wins on interpolation and
311
+ loses on structural-OOD is a different model.
312
+ - **When a model's accuracy looks too good.** The diagnostic
313
+ includes a formula check: if `N * log2(C) / P` is under 0.05, the
314
+ model is over-parameterized for the data.
315
+
316
+ ## When *not* to use it
317
+
318
+ - **When the task has no meaningful OOD axis.** Some tasks are
319
+ bounded by construction (a fixed vocabulary, a closed label set).
320
+ The structural-OOD split requires two different operations on the
321
+ same inputs; if your task has only one, skip that split.
322
+ - **When the model is already validated against a public benchmark.**
323
+ Public benchmarks handle splitting. Use the diagnostic for the
324
+ cases the benchmark does not cover.
325
+ - **As a substitute for domain expertise.** The report flags
326
+ problems; it does not tell you which ones matter for your
327
+ application.
328
+
329
+ ## Honest limitations
330
+
331
+ - **The formula constant `k = 0.05` is calibrated on one demo.**
332
+ It is a rule of thumb, not a law. Re-calibrate on your own task
333
+ before trusting the number.
334
+ - **The empirical ladder is expensive.** Seven training runs at
335
+ two seeds each. If training is expensive, use the formula and
336
+ validate with a single held-out training run.
337
+ - **The Hill fit is chosen by MSE, not by hypothesis.** A Hill fit
338
+ beating an exp fit is evidence that the ladder has a step, not
339
+ proof. The curve families are approximations.
340
+ - **The four splits do not cover every OOD axis.** Adversarial
341
+ inputs, distribution shift in the feature distribution, label
342
+ noise, and length-based generalization are separate axes not
343
+ measured here.
344
+ - **The diagnostic does not know about your loss function.** ECE is
345
+ meaningful for classification. For regression, use a proper
346
+ scoring rule and report residuals.
347
+ - **`temperature_hit_boundary` is a heuristic.** A temperature of
348
+ 19.9 on a grid with upper bound 20.0 is technically inside the
349
+ grid but suspiciously close. The check uses `>= upper - 1e-6`,
350
+ which catches only exact boundary hits.
351
+ - **The volume calculator assumes the curve shape is fixed.** Real
352
+ learning curves can have plateaus, jumps, and non-monotonic
353
+ regions. The two fitted families cover two shapes; other shapes
354
+ are silently mis-modelled.
355
+
356
+ ## What this tool demonstrates
357
+
358
+ Four findings from the demo, each visible in the report:
359
+
360
+ **1. Training accuracy is not evidence.** The model reaches 100%
361
+ on training data and 0% on far-OOD.
362
+
363
+ **2. Calibration is not accuracy.** Mean confidence is 0.38 on
364
+ interpolation where accuracy is 0.60 — the model is
365
+ underconfident, not overconfident, and the temperature-scaled ECE
366
+ is 0.33.
367
+
368
+ **3. A single number is not a report.** Four issues, all invisible
369
+ in the training accuracy figure.
370
+
371
+ **4. The fit family matters.** Exponential and Hill fits produce
372
+ recommendations of 110 and 81 examples for the same target. The
373
+ Hill fit is more economical because it correctly identifies the
374
+ ladder as a step.
375
+
376
+ ## Version history
377
+
378
+ | version | change |
379
+ |---|---|
380
+ | 0.1.0 | `LearningReport`, `diagnose()`, `print_report()`, four splits |
381
+ | 0.2.0 | `estimate_volume_formula()` |
382
+ | 0.3.0 | `estimate_volume_empirical()`, exponential saturation fit |
383
+ | 0.4.0 | temperature-boundary check, output-uniformity canary |
384
+ | 0.5.0 | ECE and underconfidence thresholds in `diagnose()` |
385
+ | 0.6.0 | Hill cooperativity fit; both curve families supported |
386
+
387
+ ## Reference
388
+
389
+ Part of a series of small tools built in one session:
390
+
391
+ | tool | reads | answers |
392
+ |---|---|---|
393
+ | `hv-manifold` | a corpus | the geometry of style space |
394
+ | `hv-reader` | one text | how it reads |
395
+ | `anomaly-or-bug` | a number and a matrix | is this a bug or a discovery? |
396
+ | `frontier-check` | one claim | where does it sit relative to the frontier? |
397
+ | `numerical-provenance` | a numeric pipeline | can I trust this number? |
398
+ | `token-provenance` | an LLM pipeline | can I trust this token count? |
399
+ | `provenance-reconstruct` | a bare number | what produced this, and how sure are you? |
400
+ | `math-ood` | a small model | did it learn a rule or memorize a table? |
401
+ | **`learning-report`** | **a trained model** | **can I make an honest claim about what it learned?** |
402
+
403
+ The design principle — that a model claim should be as structured
404
+ as the provenance of a single number — came out of a session in
405
+ which the same diagnostic (four splits, calibration, OOD) was
406
+ applied by hand to five different models, and then written down.
407
+
408
+ ## License
409
+
410
+ Apache-2.0