Update logbook: Repro - Instance-Level Costs for Nuanced Classifier Evaluation

logbook.json

@@ -11,7 +11,7 @@
     "icml2026-repro",
     "paper-qMI1xD8O3x"
   ],
-  "updated_at": "2026-07-18T16:42:49+00:00",
+  "updated_at": "2026-07-18T16:49:47+00:00",
   "root": {
     "slug": "index",
     "title": "Repro - Instance-Level Costs for Nuanced Classifier Evaluation",
@@ -70,9 +70,15 @@
         "title": "Conclusion",
         "file": "pages/conclusion/page.md",
         "children": []
+      },
+      {
+        "slug": "repro-31878-bundle",
+        "title": "repro-31878-bundle",
+        "file": "pages/repro-31878-bundle/page.md",
+        "children": []
       }
     ]
   },
-  "agent_view_tokens": 12916,
-  "revision": "1784392969962222000"
+  "agent_view_tokens": 13307,
+  "revision": "1784393387850790000"
 }

pages/claim-3-cost-derivation-sources/page.md

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:9cd351eb814c387be680afbc9c19ec413eda466860747d7b8b54c91de072657a
-size 23426748
+oid sha256:4a04836fc8cd91b561dc702ab9388bd48f2ab1852df314691b88b7b64ce46a08
+size 23426914

pages/claim-5-cost-weighted-training-is-inconsistent/page.md

@@ -125,7 +125,7 @@ https://huggingface.co/buckets/vbabenko97/repro-instance-level-costs-artifacts#l
 
 Agreements: Turkey (−11.6% vs −14.5%) and NHANES (−1.9% vs ~−2.5%) match the paper's direction and rough size. Divergences: on Jigsaw the paper reports weighting as neutral (NEC 1.8→1.8) while our weighted NEC rises by +11.4% with error up 5.91→6.42 — statistically large against the ±0.04 seed CIs, i.e. weighting *hurts* here; and the synthetic positive control failed to show the paper's benefit (next paragraph). Both divergences point the same way as the paper's bottom line — don't rely on cost-weighted training — but they are divergences, not confirmations.
 
-**Documented discrepancy — the synthetic positive control.** The paper reports weighting helps in the idealized synthetic case (−18.6%); this reproduction finds it neutral (+0.2%). Root cause, verified analytically and by a σ/d/capacity sweep: when Δ is a *symmetric* linear-Gaussian function of features, the cost-weighted-optimal and error-optimal decision boundaries **coincide** (both at Δ=0), so weighting cannot help a well-specified model at scale. A finite-sample variant (N=2,000) does show a small weighting benefit (ΔNEC ≈ −0.06 pp), the regime where the paper's effect likely arises. The large error/NEC gap in the synthetic (3.8x) reproduces regardless.
+**Documented discrepancy — the synthetic positive control.** The paper reports weighting helps in the idealized synthetic case (−18.6%); this reproduction finds it neutral (+0.2%). The root cause is analytic: when Δ is a *symmetric* linear-Gaussian function of features, the cost-weighted-optimal and error-optimal decision boundaries **coincide** (both at Δ=0), so weighting cannot help a well-specified model. A logged sweep (run below, `run_synthetic_sweep.py`) tests whether this neutrality is configuration-specific by varying one factor at a time around the base config — N (2,000/20,000), σ (0.2/0.7), d (10/50), and regularization C (0.01/100): the weighted-minus-standard NEC difference is statistically indistinguishable from zero in **every** configuration (all 95% CIs include 0; the N=2,000 point estimate is −0.06 pts but its CI is [−0.71, +0.59]). The paper's −18.6% benefit could not be recovered under any tested variant of this construction, which suggests the paper's synthetic differs in some unstated way (e.g. asymmetry or model misspecification). The large error/NEC gap in the synthetic (3.8x) reproduces regardless.
 
 **Mechanism status.** The paper's §5.3 mechanism — weighting helps only when costs are predictable from features — is directionally consistent with the real-data ranking (Turkey/NHANES, whose features carry injury/BP-adjacent signal, see gains; Jigsaw, whose annotator idiosyncrasy is orthogonal to TF-IDF, does not). But the one experiment designed to isolate that mechanism (the synthetic control) did not reproduce the paper's effect, so the mechanism should be treated as not confirmed by this reproduction.
 
@@ -289,3 +289,12 @@ wrote outputs/synthetic/sweep.csv
 **📦 Artifact** `outputs/synthetic/sweep.csv` · dataset · 311 B
 
 trackio-local-path://outputs/synthetic/sweep.csv
+
+
+---
+<!-- trackio-cell
+{"type": "dashboard", "id": "cell_951462ca1f8e", "created_at": "2026-07-18T16:49:43+00:00", "title": "Dashboard: repro-31878-bundle", "dashboard_project": "repro-31878-bundle"}
+-->
+**🎯 Trackio dashboard** `repro-31878-bundle`
+
+trackio-local-dashboard://repro-31878-bundle

pages/claim-6-delta-regression-calibration-not-accuracy/page.md

@@ -12,7 +12,7 @@ Claim 6 (§4.2/§5.5): Δ-regression (Eq. 7 — predict the signed margin with s
 <!-- trackio-cell
 {"type": "markdown", "id": "cell_8c9930e1a7bc", "created_at": "2026-07-18T14:18:35+00:00", "title": "Result: MAE matches (0.325 vs 0.30); accuracy differences small, one significant"}
 -->
-**Reproduced with a caveat.** The Jigsaw MAE lands on the paper's value, and Δ-regression gives no meaningful accuracy loss or gain in practical terms — but a paired per-seed test (run below) shows the differences are not statistically zero everywhere, so we do not claim strict equivalence:
+**Partially reproduced.** The Jigsaw MAE lands on the paper's value, and Δ-regression gives no meaningful accuracy loss or gain in practical terms — but a paired per-seed test (run below) shows the differences are not statistically zero everywhere, so we do not claim strict equivalence, and the paper's "inconsistent in direction" description does not match our runs:
 
 | Dataset | Classification NEC | Regression NEC | Paired diff (reg − std, 95% CI) | Regression MAE (Δ) |
 |---|---|---|---|---|

pages/index.md

@@ -13,3 +13,4 @@
 | [Claim 5: Cost-weighted training is inconsistent](#/claim-5-cost-weighted-training-is-inconsistent) | Partial (inconsistency holds; Jigsaw + synthetic diverge) |
 | [Claim 6: Delta-regression = calibration, not accuracy](#/claim-6-delta-regression-calibration-not-accuracy) | MAE matches (0.325 vs 0.30); equivalence caveat |
 | [Conclusion](#/conclusion) | Executive summary + poster |
+| [repro-31878-bundle](#/repro-31878-bundle) |  |

pages/repro-31878-bundle/page.md

@@ -3,8 +3,8 @@
 
 ---
 <!-- trackio-cell
-{"type": "artifact", "id": "cell_5c2bed35f9b9", "created_at": "2026-07-18T16:26:49+00:00", "title": "Artifact: repro-31878-bundle/repro-bundle:v1", "artifact": "repro-31878-bundle/repro-bundle:v1", "artifact_type": "dataset"}
+{"type": "artifact", "id": "cell_a3885b937258", "created_at": "2026-07-18T16:49:45+00:00", "title": "Artifact: repro-31878-bundle/repro-bundle:v2", "artifact": "repro-31878-bundle/repro-bundle:v2", "artifact_type": "dataset"}
 -->
-**📦 Artifact** `repro-31878-bundle/repro-bundle:v1` · dataset · 0.02 GB
+**📦 Artifact** `repro-31878-bundle/repro-bundle:v2` · dataset · 0.02 GB
 
-trackio-artifact://repro-31878-bundle/repro-bundle:v1
+trackio-artifact://repro-31878-bundle/repro-bundle:v2