v23: the ORGANIZATION-boundary build — every gate line passes on both seeds

README.md

@@ -29,73 +29,97 @@ classifier only produces a second, weaker opinion that then competes with the
 layer that was right. This model is built to leave them alone — and measurably
 does (see the touch rate below).
 
-## Verified results
-
-Build **v15**. Gate run 2026-09-03, scored through the production Rust inference
-engine, not a Python decode of the same ONNX. Corpus 147,753 training rows
-(128,584 generated + 19,169 augmented) and 5,300 held-out rows, 2 epochs. Every
-figure below was measured, none estimated; the gate was re-run on the stored
-results before this upload and every check passed.
-
-| | **v15 (this build)** | previous build on this repo (v13) |
-|---|---|---|
-| **regex-owned identifiers wrongly touched** (hivetrace ru) | **0.0%** (0/770) | 0.0% (0/770) |
-| **known-defect suite** | **21/21** (int8 and fp32) | 21/21 |
-| hivetrace ru PII (synthetic, expert-annotated) — F1 / P / R | **0.8656 / 0.8000 / 0.9429** | 0.8036 / 0.6888 / 0.9643 |
-| Wojood ar NER — F1 / P / R | **0.5870 / 0.6537 / 0.5327** | 0.4945 / 0.6143 / 0.4137 |
-| IAHLT ar — F1 | **0.5444** | 0.4697 |
-| TAB en incl. quasi-identifiers — F1 | 0.7009 | 0.7428 |
-| production-engine filtered F1, frozen eval file `cf21c01b1759` (n = 3,300) | **0.9348** | 0.8563 |
-| … PERSON only / production-masked (PERSON + ADDRESS) | **0.9772 / 0.9864** | 0.8094 / — |
-| production-engine filtered F1, register eval `c72af982d47c` (n = 3,300) | **0.8786** | 0.7884 |
-| … LOCATION / ADDRESS | **0.927 / 0.942** | 0.568 / 0.793 |
-| int8 vs fp32 F1 cost (95% CI) | +0.0086 [+0.0058, +0.0112] | +0.0020 |
-| latency p95 @ 512 tok (CPU) | 191.7 ms (measured under load avg 10–14; an upper bound) | 190.8 ms |
-| name-origin bias delta | **0.0000** — every origin scores 1.000 | 0.0469 |
-
-The two "frozen eval" rows are the only cross-build deltas in this table that
-compare like with like: the same 3,300 rows, the same scorer, both builds. The
-register file stresses LOCATION and ADDRESS in syntactic roles the original file
-under-represents; v15 lifted LOCATION there from 0.568 to 0.927.
-
-**What changed in v15.** Name-origin bias closed to zero (v14's frame work had
-left it at 0.0469) by drawing names from the Mimesis pools for every origin
-(anglo, arabic, slavic, south_asian, east_asian, african, hispanic). Address
-abbreviations (Apt/Ste/Whg, а/я, ص.ب, postcodes, PO boxes) and LOCATION in
-syntactic roles are now covered. TAB en moved slightly against this build
-(0.7428 → 0.7009); it is reported, not gated.
-
-Head-to-head vs `onnx-community/multilang-pii-ner-ONNX` (PERSON only, WikiANN
-test[:500]): es +0.65 · it +0.62 · fr +0.63 · en +0.61 · ar +0.59 · de +0.56 ·
-ru +0.12, with the 95% CI lower bound above the −0.02 floor in every language.
-Disclosure: in that comparison ours is scored as fp32 torch and the incumbent as
-int8 ONNX; the +0.0086 quantization cost above does not change the conclusion.
-
-Nothing from this repo has shipped to devices yet: the repo is private and the
-agent's model catalog does not point at it. "Previous build" means the previous
-commit here, not a fleet.
+## This build: v23
+
+Gate run 2026-09-09, scored through the production Rust inference engine, not a
+Python decode of the same ONNX. **Every one of the eight gate lines passed, on
+both training seeds** — the first build since v15 to do so.
+
+| gate line | v23 seed 1234 | v23 seed 4321 | bar |
+|---|---|---|---|
+| observed defects | 21/21 | 21/21 | all must pass |
+| regex-owned identifiers wrongly touched | **0.1 %** | **0.0 %** | ≤ 15 % |
+| recall on real Russian PII (hivetrace) | 0.9429 | 0.9357 | ≥ 0.90 |
+| int8 vs fp32 F1 cost | −0.0003 | +0.0043 | ≤ 0.02 |
+| latency p95 @ 512 tok, CPU | 192.6 ms | 188.9 ms | ≤ 200 |
+| name-origin bias delta | **0.0000** | **0.0000** | ≤ 0.05 |
+| vs incumbent, worst language (ru) | +0.1538 [+0.1012, +0.2082] | +0.1308 [+0.0809, +0.1870] | CI lower ≥ −0.02 |
+| held-out register frames, engine-filtered | **0.8870** | 0.8860 | ≥ 0.8786 |
+
+Production-engine numbers on the frozen file (`n = 3,300`): filtered F1 **0.9353**
+(P 0.9899 / R 0.8864), production-masked (PERSON + ADDRESS) **0.9795**. Per
+entity: ADDRESS 1.000 · ORGANIZATION 0.9988 · LOCATION 0.9984 · PERSON 0.9657.
+On the held-out register frames, raw F1 0.9438 and filtered 0.8870.
+
+Real-text benchmarks, unchanged in method from previous builds and **reported,
+not gated**: hivetrace ru 0.8713 · TAB en incl. quasi-identifiers 0.7111 ·
+IAHLT ar 0.5490 · Wojood ar 0.5384.
+
+**What changed from v15.** One lever: `ORG_ARTICLE_FRAMES` — training frames
+whose definite article or head noun sits immediately before the organisation
+slot and **outside** the span. It targets the largest single ORGANIZATION error
+class this project has measured: on two unsteered real-text sets the previous
+build lost 9–10 % of GermEval's gold organisations and 16 % of TAB's to a span
+that started one word early.
+
+## Where it stands against other PII models
+
+Scored on [piibench](https://huggingface.co/datasets/ScienceSoft/piibench), our
+own open benchmark — real text carried in the wire shapes outbound traffic
+actually uses (JSON bodies, XML, SQL, CSV, key=value, logs, OCR noise), scored
+character-level and label-agnostically.
+
+**This artifact scores 0.772 against the strongest independent entrant's 0.753**
+(NVIDIA's GLiNER PII), a paired difference of **+0.0192, 95 % CI
+[+0.0100, +0.0269]** — separable.
+
+Two things belong in the same breath as that number.
+
+**The claim is about this artifact, not the recipe.** The same corpus trained on
+a second seed scores 0.759, and *that* difference from GLiNER is +0.0057
+[−0.0031, +0.0143] — **not** separable. A shipped model is one specific set of
+weights, so the board position is real; but do not read it as "this recipe beats
+GLiNER every time", because measured across two seeds it does not.
+
+**We win on recall and lose on precision.** 0.908 against their 0.785 on recall;
+0.685 against their 0.743 on precision. For a data-loss-prevention product that
+is the right side of the trade — a miss is personal data leaving the machine,
+a false positive is a word masked that needn't have been — but it is a trade,
+and the aggregate hides it.
+
+**Latency, measured the same way for every entrant**: 20.3 ms median per request
+against GLiNER's 140.6 ms, batch size 1, p95 33.5 ms against 199.7 ms. Both on
+the same shared workstation, ours under a *higher* load average (20.7 vs 11.1),
+so that gap is if anything understated. Absolute milliseconds from a machine
+under load are pessimistic; the ordering is the informative part.
+
+**Touch rate 0.020 against GLiNER's 0.304.** This model fires on the identifiers
+a regex already owns roughly fifteen times less often. That is the property the
+label set was designed for, and it is not on the ranking axis.
 
 ## Read this before using it
 
-**Arabic is the weakest language, and the gap is recall.** Arabic F1 sits at
-0.54–0.59 on the external benchmarks against 0.87 for Russian and 0.70 for
-English, concentrated in LOCATION (Wojood LOCATION recall 0.334). Do not read
-the aggregate numbers as uniform accuracy across languages.
+**Arabic is the weakest language, and the gap is recall.** 0.54–0.55 on the
+Arabic benchmarks against 0.87 for Russian. Do not read the aggregate as uniform
+accuracy across languages.
 
 **Trained only on synthetic data.** Entity pools are finite, so scores on
-generated text overstate real-world ability. Judge it on the external
-benchmarks above (Wojood, IAHLT, TAB), not the synthetic figures — and note that
-hivetrace is itself a synthetic benchmark, expert-annotated but not real text.
+generated text overstate real-world ability. Judge it on the external benchmarks
+and on piibench, not on the synthetic figures — and note that hivetrace is
+itself synthetic, expert-annotated but not real text.
 
 **Arabizi is a reconstruction.** No corpus contains it; its conventions
 (3=ain, 7=haa, 2=hamza) are our model of how people type, not observed data.
 
-**DATE_TIME scores 0.000 in the production-filtered number by design, not by
-defect.** The production filter keeps a date span only next to an explicit
-birth-context cue (`born on`, `DOB:`, …); this eval corpus carries none, so
-every date the model correctly finds is filtered back out before it reaches
-the number above. It is not gated and should not be read as a date-detection
-failure — see `precision.rs`'s date-context rule in the agent's source.
+**DATE_TIME scores 0.000 on the frozen file by design, not by defect.** The
+production filter keeps a date span only next to an explicit birth-context cue
+(`born on`, `DOB:`, …), and that corpus carries none, so every date the model
+correctly finds is filtered back out before it reaches the number. On the
+held-out register frames, where such cues do occur, filtered DATE_TIME is 0.4933
+at precision 0.9418.
+
+**Nothing from this repo has shipped to devices.** The repo is private and the
+agent's model catalog does not point at it.
 
 ## Use
 
@@ -121,22 +145,15 @@ logits = sess.run(None, {i.name: enc[i.name] for i in sess.get_inputs()})[0]
 **Include `<s>`/`</s>` in the tokenizer call for full-fidelity results** — this
 is what closed most of the gap between the Python-decode number and the
 production-engine number above. `AutoTokenizer.__call__` does this by default;
-only a hand-rolled encode path that skips special tokens needs to add them back.
-
-Pair it with a regex/checksum layer for the structured identifiers it
-deliberately ignores.
-
-## Training data
-
-Fully synthetic, no real person's data. A meaningful share of rows contain no
-entity at all; those entity-free negatives are what keeps the touch rate at
-zero. The published corpus `ScienceSoft/scnsoft-pii-synthetic-corpus` is the
-v13 build's corpus; v15 trained on a later generator (address shapes, LOCATION
-roles, Arabic address vocabulary, Mimesis name pools) whose export has not been
-published yet.
+a manual `encode(..., add_special_tokens=False)` does not.
 
-Environment: transformers 4.57.1, tokenizers 0.22.1, torch 2.9.0a0, faker 37.12.0.
+## Provenance
 
-## Licence
+Corpus generated by this project's own generator; no production traffic and no
+scraped text is ever used for training. The generator switch set for this build,
+its seeds and the library versions are recorded with the run, and the
+organisation-name pool (`REAL_ORGS`) is **off** — it was measured in an earlier
+arm and cost more than it bought.
 
-Apache-2.0.
+Training stack: `transformers==4.57.1`, `tokenizers==0.22.1`, `torch==2.9.0a0`,
+`faker==37.12.0`. Two epochs.

onnx/model_quantized.onnx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:af1250c3c3dd69151528d28efeb1222088af4ad8b5dd3513cc4a2451b350d99c
-size 278234412
+oid sha256:c996fc5ad1118e0571263ee637f541b3d0981bec648e57211ad4676a3dc1ab47
+size 278234411