Update logbook: Repro - Instance-Level Costs for Nuanced Classifier Evaluation

logbook.json

@@ -11,7 +11,7 @@
     "icml2026-repro",
     "paper-qMI1xD8O3x"
   ],
-  "updated_at": "2026-07-18T16:07:17+00:00",
+  "updated_at": "2026-07-18T16:26:52+00:00",
   "root": {
     "slug": "index",
     "title": "Repro - Instance-Level Costs for Nuanced Classifier Evaluation",
@@ -70,9 +70,15 @@
         "title": "Conclusion",
         "file": "pages/conclusion/page.md",
         "children": []
+      },
+      {
+        "slug": "repro-31878-bundle",
+        "title": "repro-31878-bundle",
+        "file": "pages/repro-31878-bundle/page.md",
+        "children": []
       }
     ]
   },
-  "agent_view_tokens": 11605,
-  "revision": "1784390837996515000"
+  "agent_view_tokens": 12791,
+  "revision": "1784392012668433000"
 }

pages/claim-1-jigsaw-nec-vs-error-3x-gap/page.md

@@ -192,17 +192,9 @@ https://huggingface.co/buckets/vbabenko97/repro-instance-level-costs-artifacts#l
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_aeb8af2c6d1a", "created_at": "2026-07-18T14:15:53+00:00", "title": "Result: 2.92x error/NEC gap (paper ~3x) — reproduced"}
+{"type": "markdown", "id": "cell_aeb8af2c6d1a", "created_at": "2026-07-18T14:15:53+00:00", "title": "First-run result (superseded — see leak-free rerun below)"}
 -->
-**Reproduced — the ~3x gap holds.** On the real Jigsaw comments with recovered vote margins (300k stratified subsample, 10 seeds), standard-CE TF-IDF logistic regression gives:
-
-| Metric | This repro (N=300k) | Paper (N=1.8M) |
-|---|---|---|
-| Error rate | 5.92 ± 0.03 % | 5.34 % |
-| NEC | 2.03 ± 0.04 % | 1.76 % |
-| Error / NEC ratio | **2.92x** | **~3.0x** |
-
-The headline finding reproduces: on Jigsaw, error rate is ~3x higher than NEC, so the model's mistakes concentrate on ambiguous, low-cost comments where annotators disagreed, and it handles high-consensus comments well. Absolute NEC/error are slightly higher than the paper (2.03 vs 1.76; 5.92 vs 5.34), consistent with the smaller sample (300k vs 1.8M) and minor TF-IDF hyperparameter differences; the ratio — the paper's actual claim — matches within 3%. Δ recovered exactly from `toxicity`×`toxicity_annotator_count` (Eq. 3); no proxy.
+**Superseded.** The run above fit the TF-IDF vectorizer on the full 300k subsample *before* splitting, so vocabulary/IDF statistics included test documents — evaluation leakage (not label leakage), flagged in an external audit. The experiment was rerun with TF-IDF fit on the train split only (run below); numbers moved ≤0.01–0.02 points and the ratio is unchanged. This cell is kept as the record of the first run: standard CE 2.03 ± 0.04 NEC / 5.92 ± 0.03 error, ratio 2.92x.
 
 
 ---
@@ -214,7 +206,7 @@ The headline finding reproduces: on Jigsaw, error rate is ~3x higher than NEC, s
 <head><meta charset="utf-8" /></head>
 <body>
     <div style="height:100%; width:100%;">                        <script>window.PlotlyConfig = {MathJaxConfig: 'local'};</script>
-        <script charset="utf-8" src="https://cdn.plot.ly/plotly-3.7.0.min.js" integrity="sha256-jvTGqxNp8AGWEcvNLVuKr+8j5dGe9Yw51LQkmDH+IYA=" crossorigin="anonymous"></script>                <div id="e5548d53-2cb8-4f6b-8743-98173c1cb495" class="plotly-graph-div" style="height:100%; width:100%;"></div>            <script>                window.PLOTLYENV=window.PLOTLYENV || {};                                if (document.getElementById("e5548d53-2cb8-4f6b-8743-98173c1cb495")) {                    Plotly.newPlot(                        "e5548d53-2cb8-4f6b-8743-98173c1cb495",                        [{"marker":{"color":"#2563eb"},"name":"NEC (cost-weighted)","text":["2.0"],"textposition":"outside","x":["Jigsaw TF-IDF (repro)"],"y":[2.03],"type":"bar"},{"marker":{"color":"#f97316"},"name":"Error rate","text":["5.9"],"textposition":"outside","x":["Jigsaw TF-IDF (repro)"],"y":[5.92],"type":"bar"}],                        {"template":{"data":{"barpolar":[{"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"type":"carpet"}],"choropleth":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"choropleth"}],"contourcarpet":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"contourcarpet"}],"contour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"contour"}],"heatmap":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmap"}],"histogram2dcontour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2dcontour"}],"histogram2d":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2d"}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"mesh3d":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"mesh3d"}],"parcoords":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"parcoords"}],"pie":[{"automargin":true,"type":"pie"}],"scatter3d":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter3d"}],"scattercarpet":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattercarpet"}],"scattergeo":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergeo"}],"scattergl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergl"}],"scattermapbox":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermapbox"}],"scattermap":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermap"}],"scatterpolargl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolargl"}],"scatterpolar":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolar"}],"scatter":[{"fillpattern":{"fillmode":"overlay","size":10,"solidity":0.2},"type":"scatter"}],"scatterternary":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterternary"}],"surface":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"surface"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}]},"layout":{"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"autotypenumbers":"strict","coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]],"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]},"colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"geo":{"bgcolor":"white","lakecolor":"white","landcolor":"white","showlakes":true,"showland":true,"subunitcolor":"#C8D4E3"},"hoverlabel":{"align":"left"},"hovermode":"closest","mapbox":{"style":"light"},"paper_bgcolor":"white","plot_bgcolor":"white","polar":{"angularaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""},"bgcolor":"white","radialaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""}},"scene":{"xaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"yaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"zaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"ternary":{"aaxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"baxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"bgcolor":"white","caxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""}},"title":{"x":0.05},"xaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2},"yaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2}}},"legend":{"orientation":"h","y":1.08},"margin":{"t":70,"r":20,"l":50,"b":40},"title":{"text":"Claim 1 \u2014 Jigsaw TF-IDF: NEC vs Error rate (10 seeds, N=300k)"},"barmode":"group","yaxis":{"title":{"text":"%"}}},                        {"responsive": true}                    )                };            </script>        </div>
+        <script charset="utf-8" src="https://cdn.plot.ly/plotly-3.7.0.min.js" integrity="sha256-jvTGqxNp8AGWEcvNLVuKr+8j5dGe9Yw51LQkmDH+IYA=" crossorigin="anonymous"></script>                <div id="e5548d53-2cb8-4f6b-8743-98173c1cb495" class="plotly-graph-div" style="height:100%; width:100%;"></div>            <script>                window.PLOTLYENV=window.PLOTLYENV || {};                                if (document.getElementById("e5548d53-2cb8-4f6b-8743-98173c1cb495")) {                    Plotly.newPlot(                        "e5548d53-2cb8-4f6b-8743-98173c1cb495",                        [{"marker":{"color":"#2563eb"},"name":"NEC (cost-weighted)","text":["2.0"],"textposition":"outside","x":["Jigsaw TF-IDF (repro)"],"y":[2.02],"type":"bar"},{"marker":{"color":"#f97316"},"name":"Error rate","text":["5.9"],"textposition":"outside","x":["Jigsaw TF-IDF (repro)"],"y":[5.91],"type":"bar"}],                        {"template":{"data":{"barpolar":[{"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"type":"carpet"}],"choropleth":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"choropleth"}],"contourcarpet":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"contourcarpet"}],"contour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"contour"}],"heatmap":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmap"}],"histogram2dcontour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2dcontour"}],"histogram2d":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2d"}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"mesh3d":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"mesh3d"}],"parcoords":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"parcoords"}],"pie":[{"automargin":true,"type":"pie"}],"scatter3d":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter3d"}],"scattercarpet":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattercarpet"}],"scattergeo":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergeo"}],"scattergl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergl"}],"scattermapbox":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermapbox"}],"scattermap":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermap"}],"scatterpolargl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolargl"}],"scatterpolar":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolar"}],"scatter":[{"fillpattern":{"fillmode":"overlay","size":10,"solidity":0.2},"type":"scatter"}],"scatterternary":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterternary"}],"surface":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"surface"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}]},"layout":{"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"autotypenumbers":"strict","coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]],"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]},"colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"geo":{"bgcolor":"white","lakecolor":"white","landcolor":"white","showlakes":true,"showland":true,"subunitcolor":"#C8D4E3"},"hoverlabel":{"align":"left"},"hovermode":"closest","mapbox":{"style":"light"},"paper_bgcolor":"white","plot_bgcolor":"white","polar":{"angularaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""},"bgcolor":"white","radialaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""}},"scene":{"xaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"yaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"zaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"ternary":{"aaxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"baxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"bgcolor":"white","caxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""}},"title":{"x":0.05},"xaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2},"yaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2}}},"legend":{"orientation":"h","y":1.08},"margin":{"t":70,"r":20,"l":50,"b":40},"title":{"text":"Claim 1 \u2014 Jigsaw TF-IDF: NEC vs Error rate (10 seeds, N=300k)"},"barmode":"group","yaxis":{"title":{"text":"%"}}},                        {"responsive": true}                    )                };            </script>        </div>
 </body>
 </html>
 ````
@@ -223,8 +215,8 @@ The headline finding reproduces: on Jigsaw, error rate is ~3x higher than NEC, s
 [
   {
     "label":"Jigsaw TF-IDF (repro)",
-    "nec":2.03,
-    "error":5.92,
+    "nec":2.02,
+    "error":5.91,
     "ratio_err_over_nec":2.92
   }
 ]
@@ -386,7 +378,7 @@ Jigsaw TF-IDF, N=299,999, 10 seeds
    tdown50       10 2.19 ± 0.04  2.189664 6.25 ± 0.03    6.246333   NaN                2.85
    tdown70       10 2.08 ± 0.05  2.075193 5.91 ± 0.04    5.907667   NaN                2.85
 regression       10 1.96 ± 0.04  1.962724 5.93 ± 0.04    5.925333 0.325                3.02
-wrote /Users/vitaliibabenko/babenko-dev/repro-31878-instance-level-costs/outputs/jigsaw_tfidf/per_seed.csv and summary.csv
+wrote outputs/jigsaw_tfidf/per_seed.csv and summary.csv
 
 ````
 
@@ -407,3 +399,18 @@ trackio-local-path://outputs/jigsaw_tfidf/per_seed.csv
 **📦 Artifact** `outputs/jigsaw_tfidf/summary.csv` · dataset · 622 B
 
 trackio-local-path://outputs/jigsaw_tfidf/summary.csv
+
+
+---
+<!-- trackio-cell
+{"type": "markdown", "id": "cell_f1e2d3c4b5a6", "created_at": "2026-07-18T16:20:00+00:00", "title": "Result (leak-free): 2.92x error/NEC gap (paper ~3x) — reproduced"}
+-->
+**Reproduced — the ~3x gap holds under the leak-free protocol.** TF-IDF is now fit on each seed's train split only (test documents never touch vocabulary/IDF statistics). On the real Jigsaw comments with recovered vote margins (300k stratified subsample of the 1.8M-comment dataset, 10 seeds), standard-CE TF-IDF logistic regression gives:
+
+| Metric | This repro (300k subsample, leak-free) | Paper (N=1.8M) |
+|---|---|---|
+| Error rate | 5.91 ± 0.03 % | 5.34 % |
+| NEC | 2.02 ± 0.04 % | 1.76 % |
+| Error / NEC ratio | **2.92x** | **~3.0x** |
+
+The headline finding reproduces: error rate is ~3x higher than NEC, so the model's mistakes concentrate on ambiguous, low-cost comments where annotators disagreed, and it handles high-consensus comments well. The preprocessing-leakage fix moved results by ≤0.02 points (2.03→2.02 NEC, 5.92→5.91 error) — the leak was immaterial here, but the reported numbers now come from the clean protocol. Absolute NEC/error remain slightly above the paper (2.02 vs 1.76; 5.91 vs 5.34), consistent with the smaller sample (300k vs 1.8M) and unspecified TF-IDF hyperparameters; the ratio — the paper's actual claim — matches within 3%. Δ recovered exactly from `toxicity`×`toxicity_annotator_count` (Eq. 3); no proxy.

pages/claim-2-divergence-across-modalities/page.md

@@ -398,12 +398,12 @@ https://huggingface.co/buckets/vbabenko97/repro-instance-level-costs-artifacts#l
 
 | Modality (model) | Error (repro) | NEC (repro) | Ratio (repro) | Paper ratio | Paper NEC/Err |
 |---|---|---|---|---|---|
-| Jigsaw text (TF-IDF) | 5.92 % | 2.03 % | 2.92x | ~3.0x | 1.8 / 5.3 |
+| Jigsaw text (TF-IDF, leak-free rerun — see Claim 1) | 5.91 % | 2.02 % | 2.92x | ~3.0x | 1.8 / 5.3 |
 | Turkey image (ResNet-50 frozen) | 6.70 % | 4.06 % | **1.65x** | 1.6x | 4.2 / 6.7 |
 | NHANES tabular (HistGBM) | 21.57 % | 15.70 % | **1.37x** | 1.5x | 14.8 / 21.7 |
 | iNaturalist image | — blocked — | — | — | 1.1x | 9.3 / 11.1 |
 
-The divergence holds across every modality tested: NEC is materially below error rate in text, image, and tabular data, and the ranking of gaps matches the paper (text largest, tabular smaller). Turkey lands almost exactly on the paper (4.06/6.70 vs 4.16/6.68, 1.65x vs 1.6x) using the any-injury binarization — head+plumage vs not-injured; head-only gives 4.38/6.63, 1.51x (both reported on the run above). NHANES reproduces at 1.37x vs the paper's 1.5x (N=7,457 vs 7,455; standard 15.70/21.57 vs 14.8/21.7) — the modest shortfall is expected from unspecified feature encoding and the HistGBM defaults.
+The divergence holds across every modality tested: NEC is materially below error rate in text, image, and tabular data, and the ranking of gaps matches the paper (text largest, tabular smaller). Turkey lands almost exactly on the paper (4.06/6.70 vs 4.16/6.68, 1.65x vs 1.6x) using the any-injury binarization — head+plumage vs not-injured; the paper's "injured is the minority class" wording instead implies head-only, which gives 4.38/6.63, 1.51x (both binarizations reported on the run above; the divergence claim holds under either). NHANES reproduces at 1.37x vs the paper's 1.5x (N=7,457 vs 7,455; standard 15.70/21.57 vs 14.8/21.7) — the modest shortfall is expected from unspecified feature encoding and the HistGBM defaults.
 
 **iNaturalist blocked (documented, not fabricated).** The paper's 9,956-image sample and its Gemini 2.5-pro 7-point ratings — which define Δ for that dataset — were never released, and no author code repo exists (verified: GitHub/Zenodo/HF/PapersWithCode all empty). The iNaturalist API's native `captive_cultivated` flag is a different, binary signal and cannot reconstruct the graded Δ, so the 1.1x iNaturalist point is reported as blocked rather than proxied. See the Feasibility gates page for the download-verified availability matrix.
 

pages/claim-5-cost-weighted-training-is-inconsistent/page.md

@@ -112,22 +112,22 @@ https://huggingface.co/buckets/vbabenko97/repro-instance-level-costs-artifacts#l
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_40ebd493540e", "created_at": "2026-07-18T14:17:59+00:00", "title": "Result: inconsistent benefit reproduced (real data); synthetic discrepancy explained"}
+{"type": "markdown", "id": "cell_40ebd493540e", "created_at": "2026-07-18T14:17:59+00:00", "title": "Result: partial — inconsistency holds; Jigsaw and synthetic diverge from paper"}
 -->
-**Reproduced — the core "inconsistent benefit" claim holds; one honest discrepancy on the synthetic control.** Relative change in NEC from standard to |Δ|-weighted CE (frozen features, 10 seeds):
+**Partially reproduced.** The paper's broad claim — cost-weighted training gives inconsistent, dataset-dependent benefits — holds in our runs, and holds *more strongly* than in the paper: where the paper found weighting neutral on Jigsaw, we find it actively harmful. But two of four per-dataset results diverge materially from the paper's numbers, so the per-dataset pattern is only partially reproduced. Relative change in NEC from standard to |Δ|-weighted CE (frozen features, 10 seeds; Jigsaw from the leak-free rerun):
 
 | Dataset | Standard NEC | Weighted NEC | Rel. change (repro) | Paper |
 |---|---|---|---|---|
 | Turkey | 4.06 % | 3.59 % | **−11.6 %** (helps) | −14.5 % (helps) |
 | NHANES | 15.70 % | 15.40 % | **−1.9 %** (modest) | 14.83→14.46 ≈ −2.5 % |
-| Jigsaw | 2.03 % | 2.26 % | +11.4 % (no benefit) | negligible (no benefit) |
-| Synthetic | 2.81 % | 2.81 % | +0.2 % (neutral) | −18.6 % (helps) |
+| Jigsaw | 2.02 % | 2.26 % | **+11.4 % (harms)** — diverges | ≈0 % (1.8→1.8, neutral) |
+| Synthetic | 2.81 % | 2.81 % | +0.2 % (neutral) — diverges | −18.6 % (helps) |
 
-On the **real** datasets the reproduction matches the paper's message closely: cost-weighting helps on Turkey (−11.6% vs −14.5%), helps modestly on NHANES (−1.9% vs ~−2.5%), and gives no benefit on Jigsaw. This directly reproduces Claim 5's core point that cost-weighted training gives inconsistent, dataset-dependent benefits.
+Agreements: Turkey (−11.6% vs −14.5%) and NHANES (−1.9% vs ~−2.5%) match the paper's direction and rough size. Divergences: on Jigsaw the paper reports weighting as neutral (NEC 1.8→1.8) while our weighted NEC rises by +11.4% with error up 5.91→6.42 — statistically large against the ±0.04 seed CIs, i.e. weighting *hurts* here; and the synthetic positive control failed to show the paper's benefit (next paragraph). Both divergences point the same way as the paper's bottom line — don't rely on cost-weighted training — but they are divergences, not confirmations.
 
-**Documented discrepancy — the synthetic positive control.** The paper reports weighting helps in the idealized synthetic case (−18.6%); this reproduction finds it neutral (+0.2%). Root cause, verified analytically and by a σ/d/capacity sweep: when Δ is a *symmetric* linear-Gaussian function of features, the cost-weighted-optimal and error-optimal decision boundaries **coincide** (both at Δ=0), so weighting cannot help a well-specified model at scale. A finite-sample variant (N=2,000) does show a small weighting benefit (ΔNEC ≈ −0.06 pp), the regime where the paper's effect likely arises. The large error/NEC gap in the synthetic (3.8x) reproduces regardless. This discrepancy strengthens, rather than undercuts, the paper's overall conclusion that cost-weighting benefits are marginal and only appear under specific conditions.
+**Documented discrepancy — the synthetic positive control.** The paper reports weighting helps in the idealized synthetic case (−18.6%); this reproduction finds it neutral (+0.2%). Root cause, verified analytically and by a σ/d/capacity sweep: when Δ is a *symmetric* linear-Gaussian function of features, the cost-weighted-optimal and error-optimal decision boundaries **coincide** (both at Δ=0), so weighting cannot help a well-specified model at scale. A finite-sample variant (N=2,000) does show a small weighting benefit (ΔNEC ≈ −0.06 pp), the regime where the paper's effect likely arises. The large error/NEC gap in the synthetic (3.8x) reproduces regardless.
 
-The paper's §5.3 mechanism — weighting helps only when costs are predictable from features — is consistent with these results: Turkey/NHANES (features carry injury/BP-adjacent signal) see gains, Jigsaw (annotator idiosyncrasy orthogonal to TF-IDF) does not.
+**Mechanism status.** The paper's §5.3 mechanism — weighting helps only when costs are predictable from features — is directionally consistent with the real-data ranking (Turkey/NHANES, whose features carry injury/BP-adjacent signal, see gains; Jigsaw, whose annotator idiosyncrasy is orthogonal to TF-IDF, does not). But the one experiment designed to isolate that mechanism (the synthetic control) did not reproduce the paper's effect, so the mechanism should be treated as not confirmed by this reproduction.
 
 
 ---
@@ -139,7 +139,7 @@ The paper's §5.3 mechanism — weighting helps only when costs are predictable
 <head><meta charset="utf-8" /></head>
 <body>
     <div style="height:100%; width:100%;">                        <script>window.PlotlyConfig = {MathJaxConfig: 'local'};</script>
-        <script charset="utf-8" src="https://cdn.plot.ly/plotly-3.7.0.min.js" integrity="sha256-jvTGqxNp8AGWEcvNLVuKr+8j5dGe9Yw51LQkmDH+IYA=" crossorigin="anonymous"></script>                <div id="0d8e9fc0-a9b8-4bca-8bec-40b411d56b93" class="plotly-graph-div" style="height:100%; width:100%;"></div>            <script>                window.PLOTLYENV=window.PLOTLYENV || {};                                if (document.getElementById("0d8e9fc0-a9b8-4bca-8bec-40b411d56b93")) {                    Plotly.newPlot(                        "0d8e9fc0-a9b8-4bca-8bec-40b411d56b93",                        [{"marker":{"color":"#94a3b8"},"name":"Standard CE","text":{"dtype":"f8","bdata":"PQrXo3A9AEA9CtejcD0QQGZmZmZmZi9AexSuR+F6BkA="},"textposition":"outside","x":["Jigsaw","Turkey","NHANES","Synthetic"],"y":{"dtype":"f8","bdata":"PQrXo3A9AEA9CtejcD0QQGZmZmZmZi9AexSuR+F6BkA="},"type":"bar"},{"marker":{"color":"#2563eb"},"name":"|\u0394|-weighted CE","text":{"dtype":"f8","bdata":"FK5H4XoUAkC4HoXrUbgMQM3MzMzMzC5AexSuR+F6BkA="},"textposition":"outside","x":["Jigsaw","Turkey","NHANES","Synthetic"],"y":{"dtype":"f8","bdata":"FK5H4XoUAkC4HoXrUbgMQM3MzMzMzC5AexSuR+F6BkA="},"type":"bar"}],                        {"template":{"data":{"barpolar":[{"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"type":"carpet"}],"choropleth":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"choropleth"}],"contourcarpet":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"contourcarpet"}],"contour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"contour"}],"heatmap":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmap"}],"histogram2dcontour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2dcontour"}],"histogram2d":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2d"}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"mesh3d":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"mesh3d"}],"parcoords":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"parcoords"}],"pie":[{"automargin":true,"type":"pie"}],"scatter3d":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter3d"}],"scattercarpet":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattercarpet"}],"scattergeo":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergeo"}],"scattergl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergl"}],"scattermapbox":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermapbox"}],"scattermap":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermap"}],"scatterpolargl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolargl"}],"scatterpolar":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolar"}],"scatter":[{"fillpattern":{"fillmode":"overlay","size":10,"solidity":0.2},"type":"scatter"}],"scatterternary":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterternary"}],"surface":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"surface"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}]},"layout":{"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"autotypenumbers":"strict","coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]],"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]},"colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"geo":{"bgcolor":"white","lakecolor":"white","landcolor":"white","showlakes":true,"showland":true,"subunitcolor":"#C8D4E3"},"hoverlabel":{"align":"left"},"hovermode":"closest","mapbox":{"style":"light"},"paper_bgcolor":"white","plot_bgcolor":"white","polar":{"angularaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""},"bgcolor":"white","radialaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""}},"scene":{"xaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"yaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"zaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"ternary":{"aaxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"baxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"bgcolor":"white","caxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""}},"title":{"x":0.05},"xaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2},"yaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2}}},"legend":{"orientation":"h","y":1.1},"margin":{"t":80},"title":{"text":"Claim 5 \u2014 Cost-weighted vs standard training: NEC (frozen features, 10 seeds)"},"barmode":"group","yaxis":{"title":{"text":"NEC %"}}},                        {"responsive": true}                    )                };            </script>        </div>
+        <script charset="utf-8" src="https://cdn.plot.ly/plotly-3.7.0.min.js" integrity="sha256-jvTGqxNp8AGWEcvNLVuKr+8j5dGe9Yw51LQkmDH+IYA=" crossorigin="anonymous"></script>                <div id="0d8e9fc0-a9b8-4bca-8bec-40b411d56b93" class="plotly-graph-div" style="height:100%; width:100%;"></div>            <script>                window.PLOTLYENV=window.PLOTLYENV || {};                                if (document.getElementById("0d8e9fc0-a9b8-4bca-8bec-40b411d56b93")) {                    Plotly.newPlot(                        "0d8e9fc0-a9b8-4bca-8bec-40b411d56b93",                        [{"marker":{"color":"#94a3b8"},"name":"Standard CE","text":{"dtype":"f8","bdata":"KVyPwvUoAEA9CtejcD0QQGZmZmZmZi9AexSuR+F6BkA="},"textposition":"outside","x":["Jigsaw","Turkey","NHANES","Synthetic"],"y":{"dtype":"f8","bdata":"KVyPwvUoAEA9CtejcD0QQGZmZmZmZi9AexSuR+F6BkA="},"type":"bar"},{"marker":{"color":"#2563eb"},"name":"|\u0394|-weighted CE","text":{"dtype":"f8","bdata":"FK5H4XoUAkC4HoXrUbgMQM3MzMzMzC5AexSuR+F6BkA="},"textposition":"outside","x":["Jigsaw","Turkey","NHANES","Synthetic"],"y":{"dtype":"f8","bdata":"FK5H4XoUAkC4HoXrUbgMQM3MzMzMzC5AexSuR+F6BkA="},"type":"bar"}],                        {"template":{"data":{"barpolar":[{"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"barpolar"}],"bar":[{"error_x":{"color":"#2a3f5f"},"error_y":{"color":"#2a3f5f"},"marker":{"line":{"color":"white","width":0.5},"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"bar"}],"carpet":[{"aaxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"baxis":{"endlinecolor":"#2a3f5f","gridcolor":"#C8D4E3","linecolor":"#C8D4E3","minorgridcolor":"#C8D4E3","startlinecolor":"#2a3f5f"},"type":"carpet"}],"choropleth":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"choropleth"}],"contourcarpet":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"contourcarpet"}],"contour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"contour"}],"heatmap":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"heatmap"}],"histogram2dcontour":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2dcontour"}],"histogram2d":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"histogram2d"}],"histogram":[{"marker":{"pattern":{"fillmode":"overlay","size":10,"solidity":0.2}},"type":"histogram"}],"mesh3d":[{"colorbar":{"outlinewidth":0,"ticks":""},"type":"mesh3d"}],"parcoords":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"parcoords"}],"pie":[{"automargin":true,"type":"pie"}],"scatter3d":[{"line":{"colorbar":{"outlinewidth":0,"ticks":""}},"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatter3d"}],"scattercarpet":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattercarpet"}],"scattergeo":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergeo"}],"scattergl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattergl"}],"scattermapbox":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermapbox"}],"scattermap":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scattermap"}],"scatterpolargl":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolargl"}],"scatterpolar":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterpolar"}],"scatter":[{"fillpattern":{"fillmode":"overlay","size":10,"solidity":0.2},"type":"scatter"}],"scatterternary":[{"marker":{"colorbar":{"outlinewidth":0,"ticks":""}},"type":"scatterternary"}],"surface":[{"colorbar":{"outlinewidth":0,"ticks":""},"colorscale":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"type":"surface"}],"table":[{"cells":{"fill":{"color":"#EBF0F8"},"line":{"color":"white"}},"header":{"fill":{"color":"#C8D4E3"},"line":{"color":"white"}},"type":"table"}]},"layout":{"annotationdefaults":{"arrowcolor":"#2a3f5f","arrowhead":0,"arrowwidth":1},"autotypenumbers":"strict","coloraxis":{"colorbar":{"outlinewidth":0,"ticks":""}},"colorscale":{"diverging":[[0,"#8e0152"],[0.1,"#c51b7d"],[0.2,"#de77ae"],[0.3,"#f1b6da"],[0.4,"#fde0ef"],[0.5,"#f7f7f7"],[0.6,"#e6f5d0"],[0.7,"#b8e186"],[0.8,"#7fbc41"],[0.9,"#4d9221"],[1,"#276419"]],"sequential":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]],"sequentialminus":[[0.0,"#0d0887"],[0.1111111111111111,"#46039f"],[0.2222222222222222,"#7201a8"],[0.3333333333333333,"#9c179e"],[0.4444444444444444,"#bd3786"],[0.5555555555555556,"#d8576b"],[0.6666666666666666,"#ed7953"],[0.7777777777777778,"#fb9f3a"],[0.8888888888888888,"#fdca26"],[1.0,"#f0f921"]]},"colorway":["#636efa","#EF553B","#00cc96","#ab63fa","#FFA15A","#19d3f3","#FF6692","#B6E880","#FF97FF","#FECB52"],"font":{"color":"#2a3f5f"},"geo":{"bgcolor":"white","lakecolor":"white","landcolor":"white","showlakes":true,"showland":true,"subunitcolor":"#C8D4E3"},"hoverlabel":{"align":"left"},"hovermode":"closest","mapbox":{"style":"light"},"paper_bgcolor":"white","plot_bgcolor":"white","polar":{"angularaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""},"bgcolor":"white","radialaxis":{"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":""}},"scene":{"xaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"yaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"},"zaxis":{"backgroundcolor":"white","gridcolor":"#DFE8F3","gridwidth":2,"linecolor":"#EBF0F8","showbackground":true,"ticks":"","zerolinecolor":"#EBF0F8"}},"shapedefaults":{"line":{"color":"#2a3f5f"}},"ternary":{"aaxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"baxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""},"bgcolor":"white","caxis":{"gridcolor":"#DFE8F3","linecolor":"#A2B1C6","ticks":""}},"title":{"x":0.05},"xaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2},"yaxis":{"automargin":true,"gridcolor":"#EBF0F8","linecolor":"#EBF0F8","ticks":"","title":{"standoff":15},"zerolinecolor":"#EBF0F8","zerolinewidth":2}}},"legend":{"orientation":"h","y":1.1},"margin":{"t":80},"title":{"text":"Claim 5 \u2014 Cost-weighted vs standard training: NEC (frozen features, 10 seeds)"},"barmode":"group","yaxis":{"title":{"text":"NEC %"}}},                        {"responsive": true}                    )                };            </script>        </div>
 </body>
 </html>
 ````
@@ -148,7 +148,7 @@ The paper's §5.3 mechanism — weighting helps only when costs are predictable
 [
   {
     "dataset":"Jigsaw",
-    "std_nec":2.03,
+    "std_nec":2.02,
     "wt_nec":2.26,
     "rel_pct":11.4
   },

pages/claim-6-delta-regression-calibration-not-accuracy/page.md

@@ -10,22 +10,21 @@ Claim 6 (§4.2/§5.5): Δ-regression (Eq. 7 — predict the signed margin with s
 
 ---
 <!-- trackio-cell
-{"type": "markdown", "id": "cell_8c9930e1a7bc", "created_at": "2026-07-18T14:18:35+00:00", "title": "Result: regression ≈ classification NEC; Jigsaw MAE 0.325 (paper 0.30)"}
+{"type": "markdown", "id": "cell_8c9930e1a7bc", "created_at": "2026-07-18T14:18:35+00:00", "title": "Result: MAE matches (0.325 vs 0.30); accuracy differences small, one significant"}
 -->
-**Reproduced.** Δ-regression matches classification NEC across every dataset (differences small and inconsistent in direction), and the Jigsaw MAE lands on the paper's value:
+**Reproduced with a caveat.** The Jigsaw MAE lands on the paper's value, and Δ-regression gives no meaningful accuracy loss or gain in practical terms — but a paired per-seed test (run below) shows the differences are not statistically zero everywhere, so we do not claim strict equivalence:
 
-| Dataset | Classification NEC | Regression NEC | Regression MAE (Δ) |
-|---|---|---|---|
-| Jigsaw TF-IDF | 2.03 % | 1.96 % | **0.325** (paper 0.30) |
-| Turkey ResNet | 4.06 % | 3.80 % | 0.795 |
-| NHANES HistGBM | 15.70 % | 15.40 % | 10.47 (mmHg-scale) |
-| Synthetic | 2.81 % | 2.77 % | 0.281 |
+| Dataset | Classification NEC | Regression NEC | Paired diff (reg − std, 95% CI) | Regression MAE (Δ) |
+|---|---|---|---|---|
+| Jigsaw TF-IDF | 2.02 % | 1.96 % | **−0.06 pts [−0.09, −0.03]** (significant) | **0.325** (paper 0.30) |
+| Turkey ResNet | 4.06 % | 3.80 % | −0.26 pts [−0.71, +0.18] (n.s.) | 0.795 |
+| NHANES HistGBM | 15.70 % | 15.40 % | −0.30 pts [−0.82, +0.23] (n.s.) | 10.47 (mmHg-scale) |
+| Synthetic | 2.81 % | 2.77 % | — | 0.281 |
 
-Two paper points reproduce:
-1. **No accuracy advantage.** Regression NEC is within noise of classification NEC everywhere (Jigsaw 1.96 vs 2.03, Turkey 3.80 vs 4.06, NHANES 15.40 vs 15.70) — the paper's "differences are small and inconsistent in direction."
-2. **Calibration diagnostic.** The Jigsaw Δ-MAE of **0.325 log-odds** ≈ the paper's 0.30: regression predictions deviate from the true vote-margin by about a third of a log-odds unit on average — a calibration signal that binary classification does not expose. (NHANES MAE is on the mmHg scale of Δ=SBP−130, so ~10 mmHg, not comparable to the log-odds datasets.)
+1. **Accuracy comparison (caveat vs paper).** The paper describes the regression-vs-classification differences as "small and inconsistent in direction". Ours are small but *consistent* in direction — regression NEC is slightly lower on all three real datasets, and on Jigsaw the paired difference excludes zero (−0.06 NEC points, ~3% relative). So the practical takeaway holds (switching to Δ-regression costs no accuracy), but the paper's "inconsistent in direction" description and statistical equivalence are not confirmed by our runs.
+2. **Δ-prediction fidelity.** The Jigsaw Δ-MAE of **0.325 log-odds** ≈ the paper's 0.30: regression predictions deviate from the true vote-margin by about a third of a log-odds unit on average. This is the paper's proposed diagnostic — a prediction-error measure of the margin estimate that binary classification does not expose. (A full calibration analysis — predicted vs observed Δ — was not run; NHANES MAE is on the mmHg scale of Δ=SBP−130, so ~10 mmHg, not comparable to the log-odds datasets.)
 
-This confirms Claim 6: Δ-regression provides calibration information without an accuracy gain over cross-entropy.
+Claim 6 outcome: MAE matches the paper; regression trades no accuracy for margin information. Statistical equivalence of NEC was tested (paired, below) and does not strictly hold on Jigsaw.
 
 
 ---
@@ -97,7 +96,7 @@ Paired per-seed NEC difference: regression - standard (percentage points)
       Jigsaw TF-IDF       10                -0.061   -0.091   -0.032           True
 Turkey (any_injury)       10                -0.263   -0.710    0.183          False
              NHANES       10                -0.299   -0.822    0.225          False
-wrote /Users/vitaliibabenko/babenko-dev/repro-31878-instance-level-costs/outputs/claim6_paired.csv
+wrote outputs/claim6_paired.csv
 
 ````
 
@@ -109,3 +108,12 @@ wrote /Users/vitaliibabenko/babenko-dev/repro-31878-instance-level-costs/outputs
 **📦 Artifact** `outputs/claim6_paired.csv` · dataset · 194 B
 
 trackio-local-path://outputs/claim6_paired.csv
+
+
+---
+<!-- trackio-cell
+{"type": "dashboard", "id": "cell_382f9da4312f", "created_at": "2026-07-18T16:26:48+00:00", "title": "Dashboard: repro-31878-bundle", "dashboard_project": "repro-31878-bundle"}
+-->
+**🎯 Trackio dashboard** `repro-31878-bundle`
+
+trackio-local-dashboard://repro-31878-bundle

pages/index.md

@@ -6,10 +6,11 @@
 | --- | --- |
 | [Paper & claims](#/paper-claims) | 6 claims verified vs arXiv text |
 | [Feasibility gates](#/feasibility-gates) | 3/4 datasets public; GPU 402-blocked |
-| [Claim 1: Jigsaw NEC vs error (3x gap)](#/claim-1-jigsaw-nec-vs-error-3x-gap) | Reproduced (2.92x vs ~3x) |
+| [Claim 1: Jigsaw NEC vs error (3x gap)](#/claim-1-jigsaw-nec-vs-error-3x-gap) | Reproduced (2.92x vs ~3x; leak-free rerun) |
 | [Claim 2: Divergence across modalities](#/claim-2-divergence-across-modalities) | Reproduced (Turkey/NHANES); iNat blocked |
-| [Claim 3: Cost derivation sources](#/claim-3-cost-derivation-sources) | Reproduced (Eq. 3/4/rating + Fig. 2) |
-| [Claim 4: Fine-tuned results (toy)](#/claim-4-fine-tuned-results-toy) | Direction only; magnitudes GPU-blocked |
-| [Claim 5: Cost-weighted training is inconsistent](#/claim-5-cost-weighted-training-is-inconsistent) | Reproduced (real data); synthetic explained |
-| [Claim 6: Delta-regression = calibration, not accuracy](#/claim-6-delta-regression-calibration-not-accuracy) | Reproduced (MAE 0.325 vs 0.30) |
+| [Claim 3: Cost derivation sources](#/claim-3-cost-derivation-sources) | Partial (Eq. 3/4 on real data; rating synthetic) |
+| [Claim 4: Fine-tuned results (toy)](#/claim-4-fine-tuned-results-toy) | Not reproduced; toy direction only (no GPU credits) |
+| [Claim 5: Cost-weighted training is inconsistent](#/claim-5-cost-weighted-training-is-inconsistent) | Partial (inconsistency holds; Jigsaw + synthetic diverge) |
+| [Claim 6: Delta-regression = calibration, not accuracy](#/claim-6-delta-regression-calibration-not-accuracy) | MAE matches (0.325 vs 0.30); equivalence caveat |
 | [Conclusion](#/conclusion) | Executive summary + poster |
+| [repro-31878-bundle](#/repro-31878-bundle) |  |

pages/repro-31878-bundle/page.md

@@ -0,0 +1,10 @@
+# repro-31878-bundle
+
+
+---
+<!-- trackio-cell
+{"type": "artifact", "id": "cell_5c2bed35f9b9", "created_at": "2026-07-18T16:26:49+00:00", "title": "Artifact: repro-31878-bundle/repro-bundle:v1", "artifact": "repro-31878-bundle/repro-bundle:v1", "artifact_type": "dataset"}
+-->
+**📦 Artifact** `repro-31878-bundle/repro-bundle:v1` · dataset · 0.02 GB
+
+trackio-artifact://repro-31878-bundle/repro-bundle:v1