Update logbook: Repro - Instance-Level Costs for Nuanced Classifier Evaluation

logbook.json

@@ -11,7 +11,7 @@
     "icml2026-repro",
     "paper-qMI1xD8O3x"
   ],
-  "updated_at": "2026-07-18T16:06:46+00:00",
+  "updated_at": "2026-07-18T16:07:17+00:00",
   "root": {
     "slug": "index",
     "title": "Repro - Instance-Level Costs for Nuanced Classifier Evaluation",
@@ -73,6 +73,6 @@
       }
     ]
   },
-  "agent_view_tokens": 11393,
-  "revision": "1784390806954493000"
+  "agent_view_tokens": 11605,
+  "revision": "1784390837996515000"
 }

pages/claim-6-delta-regression-calibration-not-accuracy/page.md

@@ -26,3 +26,86 @@ Two paper points reproduce:
 2. **Calibration diagnostic.** The Jigsaw Δ-MAE of **0.325 log-odds** ≈ the paper's 0.30: regression predictions deviate from the true vote-margin by about a third of a log-odds unit on average — a calibration signal that binary classification does not expose. (NHANES MAE is on the mmHg scale of Δ=SBP−130, so ~10 mmHg, not comparable to the log-odds datasets.)
 
 This confirms Claim 6: Δ-regression provides calibration information without an accuracy gain over cross-entropy.
+
+
+---
+<!-- trackio-cell
+{"type": "code", "id": "cell_e5dfab1a8e3b", "created_at": "2026-07-18T16:07:12+00:00", "title": "Run: uv paired_regression_stats.py (exit 0)", "command": ["uv", "run", "python", "scripts/paired_regression_stats.py"], "exit_code": 0, "duration_s": 1.362}
+-->
+````bash
+$ uv run python scripts/paired_regression_stats.py
+````
+
+exit 0 · 1.4s
+
+
+````python title=paired_regression_stats.py
+"""Claim 6: paired per-seed test of regression NEC vs classification NEC.
+
+The seeds share train/test splits within a dataset, so the honest comparison is
+the PAIRED per-seed difference (regression - standard), not overlapping marginal
+CIs. Reports mean paired difference with a t-based 95% CI and whether the CI
+excludes zero, per dataset.
+"""
+
+import sys
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+from scipy import stats
+
+ROOT = Path(__file__).resolve().parents[1]
+
+SOURCES = {
+    "Jigsaw TF-IDF": ROOT / "outputs" / "jigsaw_tfidf" / "per_seed.csv",
+    "Turkey (any_injury)": ROOT / "outputs" / "turkey" / "per_seed_any_injury.csv",
+    "NHANES": ROOT / "outputs" / "nhanes" / "per_seed.csv",
+}
+
+rows = []
+for name, path in SOURCES.items():
+    df = pd.read_csv(path)
+    piv = df.pivot(index="seed", columns="strategy", values="nec")
+    d = (piv["regression"] - piv["standard"]).to_numpy() * 100  # NEC points (%)
+    n = len(d)
+    mean = float(np.mean(d))
+    half = float(stats.t.ppf(0.975, n - 1) * np.std(d, ddof=1) / np.sqrt(n))
+    rows.append(
+        {
+            "dataset": name,
+            "n_seeds": n,
+            "paired_diff_mean_pts": round(mean, 3),
+            "ci95_lo": round(mean - half, 3),
+            "ci95_hi": round(mean + half, 3),
+            "excludes_zero": bool(mean - half > 0 or mean + half < 0),
+        }
+    )
+
+out = pd.DataFrame(rows)
+print("Paired per-seed NEC difference: regression - standard (percentage points)")
+print(out.to_string(index=False))
+out.to_csv(ROOT / "outputs" / "claim6_paired.csv", index=False)
+print(f"wrote {ROOT / 'outputs' / 'claim6_paired.csv'}")
+
+````
+
+
+````output
+Paired per-seed NEC difference: regression - standard (percentage points)
+            dataset  n_seeds  paired_diff_mean_pts  ci95_lo  ci95_hi  excludes_zero
+      Jigsaw TF-IDF       10                -0.061   -0.091   -0.032           True
+Turkey (any_injury)       10                -0.263   -0.710    0.183          False
+             NHANES       10                -0.299   -0.822    0.225          False
+wrote /Users/vitaliibabenko/babenko-dev/repro-31878-instance-level-costs/outputs/claim6_paired.csv
+
+````
+
+
+---
+<!-- trackio-cell
+{"type": "artifact", "id": "cell_69ca70a71294", "created_at": "2026-07-18T16:07:13+00:00", "title": "Artifact: claim6_paired.csv", "path": "outputs/claim6_paired.csv", "size": 194, "artifact_type": "dataset", "auto": true}
+-->
+**📦 Artifact** `outputs/claim6_paired.csv` · dataset · 194 B
+
+trackio-local-path://outputs/claim6_paired.csv