Closed at this problem framing: converged results beside pre-convergence ones, empty-result and competitiveness notice, viewer with unseen conditions

.gitattributes

@@ -35,3 +35,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 figures/fig1_recall_by_condition.dark.png filter=lfs diff=lfs merge=lfs -text
 figures/fig1_recall_by_condition.light.png filter=lfs diff=lfs merge=lfs -text
+figures/pre-convergence/fig1_recall_by_condition.dark.png filter=lfs diff=lfs merge=lfs -text
+figures/pre-convergence/fig1_recall_by_condition.light.png filter=lfs diff=lfs merge=lfs -text
+figures/val_curve.dark.png filter=lfs diff=lfs merge=lfs -text
+figures/val_curve.light.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -18,17 +18,30 @@ tags:
   - redaction
   - speech
   - negative-results
+  - closed
 ---
 
 # Bleep
 
 **Find *where* sensitive speech is, without transcribing *what* was said.**
 
-A results dashboard for a published **negative result**: at the ASR baseline's
-own measured false-alarm rate, Bleep reaches 0.16 recall against the baseline's
-0.83. The page sets out the three separable reasons the benchmark was built to
-distinguish, and includes a timeline viewer over real test utterances showing
-ground truth against what each detector actually found.
+> **An empty result means nothing was flagged. It does not mean nothing is there.**
+>
+> This detector is not competitive with an ASR-based pipeline outside the
+> conditions it was trained on. See the
+> [model card](https://huggingface.co/NagaYu/bleep-0.09b).
+
+**Status: closed at this problem framing.** A perfect transcript fed to the
+same text rules scores 0.986, so the room above the ASR baseline (0.826) is
+bounded by the ASR's word errors. Retrained to convergence on 2026-10-18,
+Bleep reaches 0.903 at the baseline's false-alarm rate on the degradation types
+it was trained on (0.156 before convergence); on unseen ones it raises 3.3–3.5×
+the baseline's false alarms.
+
+A results dashboard with a timeline viewer over real test utterances — from
+conditions the model was trained on and from conditions it never saw — showing
+ground truth against what each detector actually found, with the
+pre-convergence model drawn as its own lane.
 
 No audio is shipped — the corpus was rendered with macOS system voices, which
 are licensed for use and not for republication. The viewer draws a downsampled

index.html

@@ -46,6 +46,10 @@
     border:1px solid var(--rule);border-radius:9px;text-decoration:none;
     color:var(--ink);background:var(--panel);font-size:.92rem}
   .links a:hover{border-color:var(--a)}
+  .banner{border:2px solid var(--warn-br);background:var(--warn-bg);border-radius:12px;
+    padding:14px 18px;margin:18px 0 6px;font-size:1rem}
+  .banner p{margin:0 0 4px}
+  .banner p:last-child{margin:0}
   .callout{background:var(--warn-bg);border-left:3px solid var(--warn-br);
     border-radius:0 10px 10px 0;padding:18px 22px;margin:26px 0}
   .callout h2{margin-top:0}
@@ -92,42 +96,63 @@
     <a href="https://huggingface.co/datasets/NagaYu/bleep-spans">📊 Dataset</a>
   </div>
 
+  <div class="banner" role="note">
+    <p><strong>An empty result means nothing was flagged. It does not mean nothing is there.</strong></p>
+    <p>This detector is not competitive with an ASR-based pipeline outside the
+    conditions it was trained on. See the
+    <a href="https://huggingface.co/NagaYu/bleep-0.09b">model card</a>.</p>
+  </div>
+
   <div class="callout">
-    <h2>This is a published negative result</h2>
-    <p>At the ASR baseline's own measured false-alarm rate, on a speaker-disjoint
-    test split, <strong>Bleep loses badly</strong>. That is the finding, and it
-    belongs at the top rather than in a footnote.</p>
+    <h2>STATUS: closed at this problem framing</h2>
+    <p><strong>The oracle arm — a perfect transcript fed to the same text rules —
+    scores <span id="oracleRecall">…</span>. <span id="missSplit">…</span> At this
+    problem framing the room above the baseline is bounded by that gap —
+    <span id="headroom">…</span> recall on the test split — and this is the
+    measurement that establishes it.</strong></p>
+    <p>Retrained to convergence on 2026-10-18. Pre-convergence numbers are kept
+    beside the new ones. These are final; there will not be a third training
+    run.</p>
     <div class="tablewrap"><table id="armTable"></table></div>
     <p class="note" id="armNote"></p>
   </div>
 
-  <h2>Why, specifically</h2>
-  <p>The benchmark was built to separate the reasons, not just report the
-  number.</p>
+  <h2>Where it does not hold</h2>
+  <p id="heldoutText">…</p>
+  <div class="tablewrap"><table id="heldTable"></table></div>
 
-  <h3>1 · The failure mode is real, but the corpus put it in the wrong place</h3>
-  <p id="whyCond">…</p>
+  <h2>Why it was negative before convergence</h2>
 
-  <h3>2 · The text rules were never the bottleneck</h3>
-  <p>An <em>oracle</em> arm — a perfect transcript fed to the same text-PII
-  rules — scores 0.96–1.00 in every condition. So every baseline miss is an ASR
-  <em>word</em> error rather than a detector gap. That one arm turns “the
-  baseline failed” into “the baseline failed here, for this reason”.</p>
+  <h3>The training conditions were too mild</h3>
+  <p id="whyCond">…</p>
 
-  <h3>3 · The model is badly undertrained</h3>
+  <h3>The model was badly undertrained</h3>
   <p>1500 steps, batch 4, ~1.8 epochs over 9.15 hours — 70 minutes on one
-  consumer GPU. Validation loss was still falling monotonically at the final
-  step (0.218 → 0.183): a stopped run, not a converged one.</p>
+  consumer GPU. The validation loss recorded during that run (0.218, 0.200,
+  0.191, 0.203, 0.192, 0.183) was measured on freshly re-sampled crops, rose
+  once, and ended under a learning rate already annealed to 2% of its peak: a
+  stopped run, not a converged one. (This page previously said the loss was
+  "still falling monotonically"; that was not accurate.) The retrain ran to a
+  stopping rule fixed and hashed before it started, and stopped by that rule.</p>
+  <figure>
+    <img src="figures/val_curve.light.png" alt="Dev loss against training step for the retrain"
+         id="figValLight">
+    <figcaption>The retrain's dev loss on a fixed set of crops. The dashed line
+    is the pre-convergence checkpoint on the same crops.</figcaption>
+  </figure>
 
   <h3>One directional hint</h3>
   <p id="whyCat">…</p>
 
-  <h2>What did hold up</h2>
+  <h2>What is complete and load-bearing</h2>
   <p id="heldUp">…</p>
 
   <h2>Timeline viewer</h2>
-  <p>Real utterances from the test split. Ground truth against what each
-  detector actually found, at the operating points the benchmark reports.</p>
+  <p>Real utterances: the first ones from <code>test</code> (conditions the
+  model was trained on) and from <code>test_heldout_v2</code> (conditions it
+  never saw). Ground truth against what each detector actually found, at the
+  operating points the benchmark reports. The pre-convergence model has its own
+  lane.</p>
   <p class="note"><strong>No audio is shipped.</strong> These recordings were
   rendered with the macOS system voices, which are licensed for use and not for
   republication as a corpus. What you see is a downsampled amplitude envelope —
@@ -140,7 +165,8 @@
     <div class="legend">
       <span><i class="sw" style="background:var(--neutral)"></i>ground truth</span>
       <span><i class="sw" style="background:var(--a)"></i>ASR + text PII</span>
-      <span><i class="sw" style="background:var(--c)"></i>Bleep</span>
+      <span><i class="sw" style="background:var(--c)"></i>Bleep (converged)</span>
+      <span><i class="sw" style="background:var(--b)"></i>Bleep (before convergence)</span>
       <span><i class="sw" style="background:var(--d)"></i>union</span>
     </div>
   </div>
@@ -150,15 +176,16 @@
     <img src="figures/fig1_recall_by_condition.light.png"
          alt="Recall by degradation condition for each arm, training conditions"
          id="figCondLight">
-    <figcaption>Training conditions. The subtitle is generated from the data, so
-    the chart cannot assert a conclusion the numbers do not support — it prints
-    the opposite of the project's hypothesis.</figcaption>
+    <figcaption>Training conditions, converged model. The subtitle is generated
+    from the data, so the chart cannot assert a conclusion the numbers do not
+    support.</figcaption>
   </figure>
   <figure>
-    <img src="figures/fig1_recall_by_condition.test_heldout.light.png"
-         alt="Recall by held-out degradation condition" id="figHeldLight">
-    <figcaption>Held-out conditions — the more informative set. The ASR arm's
-    collapse under heavy overlap appears only here.</figcaption>
+    <img src="figures/fig4_recall_vs_false_alarms.test_heldout_v2.light.png"
+         alt="Recall against false-alarm rate on unseen degradations" id="figFaLight">
+    <figcaption>Unseen conditions: the converged model's whole recall/false-alarm
+    curve sits to the right of the ASR baseline — no threshold reaches the
+    baseline's false-alarm rate.</figcaption>
   </figure>
 
   <h2>Recall by category</h2>
@@ -166,24 +193,32 @@
     <img src="figures/fig3_recall_by_category.light.png"
          alt="Recall by category of sensitive speech" id="figCatLight">
     <figcaption>★ marks the failure modes reported for transcription-first
-    pipelines.</figcaption>
+    pipelines. Converged model; the pre-convergence figures are in
+    <code>figures/pre-convergence/</code>.</figcaption>
   </figure>
 
   <h2>What this is not</h2>
-  <p><strong>Not a redaction guarantee.</strong> It misses most things. An empty
-  result means “nothing was flagged”, never “nothing is there”.</p>
-  <p><strong>Not a replacement for your ASR-based detector.</strong> It is
-  designed for union mode, and the union is proven never to drop a region the
-  existing detector found — zero violations across both splits.</p>
+  <p><strong>Not a redaction guarantee.</strong> Even on the conditions it was
+  trained on it misses about one region in ten; on conditions it was not, roughly
+  one in five to one in four, while raising several times the baseline's false
+  alarms. An empty result means “nothing was flagged”, never “nothing is
+  there”.</p>
+  <p><strong>Not a replacement for your ASR-based detector.</strong> It beats the
+  baseline only on the degradation types it was trained on. It is designed for
+  union mode, and the union is verified never to drop a region the existing
+  detector found — zero violations on every utterance of all three evaluation
+  splits.</p>
   <p><strong>Not trained on real audio.</strong> Every recording is synthetic.
   No real call, no clinical recording, nothing derived from either, at any
   stage. That is also a limitation: TTS speech is far easier to transcribe than
   conversation, which is part of why the baseline looks so strong here.</p>
+  <p><strong>English only.</strong> Japanese generators exist in the code but are
+  not benchmarked.</p>
 
   <div class="foot">
     <p id="foot">…</p>
     <p>Apache-2.0. Negative results are cheap to hide and expensive to
-    rediscover.</p>
+    rediscover; closed projects are worth closing in writing.</p>
   </div>
 </div>
 
@@ -194,74 +229,116 @@ const el = (t, a = {}) => { const n = document.createElementNS(NS, t);
 const f3 = v => (v === null || v === undefined || Number.isNaN(v)) ? "—" : v.toFixed(3);
 
 fetch("data.json").then(r => r.json()).then(D => {
-  const S = D.summary.test, m = S.matched_fa, arms = S.arms, comp = S.complementarity;
-
-  // ---- arm table -------------------------------------------------------
+  const S = D.summary.test, B = D.summary_before.test;
+  const m = S.matched_fa, arms = S.arms, comp = S.complementarity;
+  const f1 = v => (v === null || v === undefined) ? "—" : v.toFixed(1);
+
+  // ---- status numbers (from the data, not typed) ------------------------
+  const oracle = arms.oracle.recall, baseA = arms.A.recall;
+  document.getElementById("oracleRecall").textContent = f3(oracle);
+  document.getElementById("headroom").textContent = f3(oracle - baseA);
+  const C = D.ceiling;
+  document.getElementById("missSplit").textContent =
+    `${C.baseline_misses - C.also_missed_by_oracle} of the baseline's `
+    + `${C.baseline_misses} misses on the test split are ASR word errors, not `
+    + `detector gaps (the oracle misses the other ${C.also_missed_by_oracle} too).`;
+
+  // ---- arm table: before and after convergence ---------------------------
   const rows = [
     ["A — ASR (whisper-tiny.en) + text PII", arms.A, false],
     ["B — ASR (whisper-small.en) + text PII", arms.B, false],
-    ["C — Bleep alone", arms.C, true],
-    ["D — A ∪ Bleep", arms.D, false],
+    ["C — Bleep alone, converged", arms.C, true],
+    ["C — Bleep alone, before convergence", B.arms.C, false],
+    ["D — A ∪ Bleep, converged", arms.D, false],
+    ["D — A ∪ Bleep, before convergence", B.arms.D, false],
     ["oracle — perfect transcript + same rules", arms.oracle, false],
   ].filter(r => r[1]);
   const t = document.getElementById("armTable");
-  t.innerHTML = "<tr><th>arm</th><th style='text-align:right'>recall</th>"
+  t.innerHTML = "<tr><th>arm (test split)</th><th style='text-align:right'>recall</th>"
     + "<th style='text-align:right'>false alarms/hour</th></tr>"
     + rows.map(([name, a, hero]) =>
         `<tr class="${hero ? "hero" : ""}"><td>${name}</td>`
         + `<td class="num">${f3(a.recall)}</td>`
-        + `<td class="num">${a.fa_per_hour.toFixed(1)}</td></tr>`).join("");
+        + `<td class="num">${f1(a.fa_per_hour)}</td></tr>`).join("");
   document.getElementById("armNote").textContent =
-    `${S.n_utterances} utterances (${S.audio_hours.toFixed(2)} h). Bleep's `
-    + `threshold was calibrated on a disjoint split to match arm A's measured `
-    + `false-alarm rate of ${m.arm_a_fa_per_hour_on_test.toFixed(1)}/hour, so the `
-    + `comparison is like-for-like. Of the regions arm A missed, Bleep found `
-    + `${(comp.asr_misses_rescued_by_bleep * 100).toFixed(0)}%.`;
+    `${S.n_utterances} utterances (${S.audio_hours.toFixed(2)} h) from the eight `
+    + `degradation conditions the model was trained on, speakers it never saw. `
+    + `Bleep's threshold was calibrated on a disjoint split to arm A's measured `
+    + `false-alarm rate of ${f1(m.arm_a_fa_per_hour_on_test)}/hour. Of the regions `
+    + `arm A missed, the converged model found `
+    + `${(comp.asr_misses_rescued_by_bleep * 100).toFixed(0)}% (before convergence: `
+    + `${(B.complementarity.asr_misses_rescued_by_bleep * 100).toFixed(0)}%).`;
+
+  // ---- held-out: the false-alarm failure ---------------------------------
+  const H = D.summary.test_heldout, H2 = D.summary.test_heldout_v2;
+  const ratio = h => h.arms.C.fa_per_hour / h.matched_fa.arm_a_fa_per_hour_on_test;
+  document.getElementById("heldoutText").textContent =
+    `On degradation conditions it was not trained on, the threshold carried over `
+    + `from the calibration split does not carry its false-alarm rate: the `
+    + `converged model raises ${ratio(H).toFixed(1)}–${ratio(H2).toFixed(1)}× the `
+    + `baseline's false alarms, and no threshold brings it down to the baseline's `
+    + `rate. None of these recall comparisons is at a matched false-alarm rate.`;
+  const ht = document.getElementById("heldTable");
+  const hrow = (name, h) => `<tr><td>${name}</td>`
+    + `<td class="num">${f3(h.arms.A.recall)} · ${f1(h.arms.A.fa_per_hour)}</td>`
+    + `<td class="num">${f3(h.arms.C.recall)} · ${f1(h.arms.C.fa_per_hour)}</td></tr>`;
+  ht.innerHTML = "<tr><th>held-out split</th><th style='text-align:right'>ASR baseline: recall · FA/h</th>"
+    + "<th style='text-align:right'>Bleep, converged: recall · FA/h</th></tr>"
+    + hrow("test_heldout (2 of 5 conditions later trained)", H)
+    + hrow("test_heldout_v2 (all 5 conditions unseen)", H2);
 
   // ---- data-derived prose ---------------------------------------------
-  const pc = S.per_condition, ph = D.summary.test_heldout.per_condition;
+  const pc = S.per_condition, ph = H.per_condition, ph2 = H2.per_condition;
   document.getElementById("whyCond").textContent =
     `On the training-condition test split the ASR baseline is essentially flat: `
     + `${f3(pc.clean.A)} on clean audio and ${f3(pc.telephony_8k.A)} at 8 kHz — `
     + `i.e. better than clean. A band-limit plus G.711 is not a hard condition `
-    + `for a modern ASR on hyper-articulated TTS speech. The collapse appears `
+    + `for a modern ASR on hyper-articulated TTS speech. The collapse appeared `
     + `only in the harsher held-out conditions: `
     + `${f3(ph.overlap_0db_three_way.A)} at 0 dB three-way overlap and `
-    + `${f3(ph.telephony_overlap_3db.A)} at narrowband-plus-overlap. The premise `
-    + `survives; the training conditions were too mild to exercise it.`;
+    + `${f3(ph.telephony_overlap_3db.A)} at narrowband-plus-overlap. For the `
+    + `retrain those two were moved into training; of five new unseen conditions `
+    + `the baseline collapses only under overlap again (`
+    + `${f3(ph2.overlap_minus3db.A)} at −3 dB).`;
 
-  const cat = S.per_category;
-  const best = Object.entries(cat).filter(([, v]) => v.C !== null)
+  const catB = B.per_category, cat = S.per_category;
+  const best = Object.entries(catB).filter(([, v]) => v.C !== null)
     .sort((a, b) => b[1].C - a[1].C).slice(0, 2);
   document.getElementById("whyCat").textContent =
-    `Bleep's per-category recall has the shape the acoustic argument predicts. `
-    + `Its best categories are ${best.map(([k, v]) =>
+    `Before convergence, Bleep's per-category recall had the shape the acoustic `
+    + `argument predicts. Its best categories were ${best.map(([k, v]) =>
         k.replace(/_/g, " ").toLowerCase() + " (" + f3(v.C) + ")").join(" and ")}`
     + ` — the two with the most distinctive rhythm — against ~0.00 for run-on `
-    + `digits, money and identifiers. The cue family it managed to learn is the `
-    + `predicted one. There is just far too little of it to be useful.`;
+    + `digits, money and identifiers. After convergence those two reach `
+    + `${best.map(([k]) => f3(cat[k].C)).join(" and ")}.`;
 
   const sp = S.speed.bleep, asr = S.speed.asr_arms_rtf;
   document.getElementById("heldUp").textContent =
     `Speed: ${sp.rtf_median} real-time factor on CPU with the front end `
     + `included — ${sp.xrealtime_best}× real time, `
     + `${(asr.A / sp.rtf_median).toFixed(1)}× faster than the cheapest ASR arm `
-    + `and ${(asr.B / sp.rtf_median).toFixed(0)}× faster than the better one. `
-    + `${(sp.param_count / 1e6).toFixed(1)} M parameters, inside the committed `
-    + `0.05–0.15 B band. The union containment guarantee held with zero `
-    + `violations across both splits, and the no-content invariant is enforced `
-    + `structurally by tests rather than asserted.`;
+    + `and ${(asr.B / sp.rtf_median).toFixed(0)}× faster than the better one; `
+    + `on the Apple Neural Engine via Core ML, about 0.002 real-time factor. `
+    + `${(sp.param_count / 1e6).toFixed(1)} M parameters. The no-content invariant `
+    + `is enforced structurally by tests (an AST walk over the module graph); a `
+    + `span carries only start, end, label, score and source. The union never `
+    + `drops a region the existing detector found: zero violations on every `
+    + `utterance of all three evaluation splits. Chunked inference is bit-exact `
+    + `against whole-file inference, the receptive field is capped at 8.98 s on `
+    + `purpose, and redaction writes a review plan rather than touching audio.`;
 
   document.getElementById("foot").textContent =
-    `Benchmark generated ${D.generated_utc}. Every figure on this page is `
-    + `produced by scripts in the repository from the benchmark's own JSON.`;
+    `Converged benchmark generated ${D.generated_utc}; pre-convergence benchmark `
+    + `${D.generated_utc_before}. Every number on this page is read from the `
+    + `benchmarks' own JSON by the page itself.`;
 
   // ---- timeline viewer -------------------------------------------------
   const pick = document.getElementById("pick");
   D.examples.forEach((e, i) => {
     const o = document.createElement("option");
     o.value = i;
-    o.textContent = `${e.condition.replace(/_/g, " ")} — ${e.duration.toFixed(1)}s `
+    o.textContent = `${e.split === "test" ? "trained" : "UNSEEN"} · `
+      + `${e.condition.replace(/_/g, " ")} — ${e.duration.toFixed(1)}s `
       + `(${e.truth.length} sensitive region${e.truth.length === 1 ? "" : "s"})`;
     pick.appendChild(o);
   });
@@ -270,6 +347,7 @@ fetch("data.json").then(r => r.json()).then(D => {
     ["ground truth", "truth", "var(--neutral)"],
     ["ASR + text PII", "asr", "var(--a)"],
     ["Bleep", "bleep", "var(--c)"],
+    ["Bleep (before)", "bleep_before", "var(--b)"],
     ["union", "union", "var(--d)"],
   ];
 
@@ -345,7 +423,8 @@ fetch("data.json").then(r => r.json()).then(D => {
   // dark-mode figure swap
   const swap = dark => {
     [["figCondLight", "fig1_recall_by_condition"],
-     ["figHeldLight", "fig1_recall_by_condition.test_heldout"],
+     ["figFaLight", "fig4_recall_vs_false_alarms.test_heldout_v2"],
+     ["figValLight", "val_curve"],
      ["figCatLight", "fig3_recall_by_category"]].forEach(([id, base]) => {
        const n = document.getElementById(id);
        if (n) n.src = `figures/${base}.${dark ? "dark" : "light"}.png`;