Updated README.md to have inference latency evals
Browse files
README.md
CHANGED
|
@@ -129,6 +129,36 @@ documents; it misses them in casual first-person writing.
|
|
| 129 |
phi-redact v7 is essentially tied with Desert Ant's English composite at half
|
| 130 |
the parameters and half the on-device footprint.
|
| 131 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 132 |
---
|
| 133 |
|
| 134 |
## Training data
|
|
|
|
| 129 |
phi-redact v7 is essentially tied with Desert Ant's English composite at half
|
| 130 |
the parameters and half the on-device footprint.
|
| 131 |
|
| 132 |
+
### Inference latency
|
| 133 |
+
|
| 134 |
+
Measured 2026-10-03 against the live demo (corytrimm.com/phi-redact): the
|
| 135 |
+
same 15-template typing-length corpus as `evals/latency-bench/bench.py`,
|
| 136 |
+
15 templates × 10 cycles = 150 redactions, timed from the page's own
|
| 137 |
+
per-redaction readout — TypeScript rules scan + ONNX detect + merge +
|
| 138 |
+
placeholders, end to end, onnxruntime-web WASM (Chromium on Linux).
|
| 139 |
+
|
| 140 |
+
| Input | p50 | p95 | p99 |
|
| 141 |
+
|---|---|---|---|
|
| 142 |
+
| Chat-length (150 samples) | **6.0 ms** | 8.0 ms | 9.0 ms |
|
| 143 |
+
| Long text (3,959 chars, 139 entities) | 572 ms | — | — |
|
| 144 |
+
|
| 145 |
+
Model load: 0.5 s cached, 4.4 s cold (11.3 MB int8 ONNX + runtime download).
|
| 146 |
+
|
| 147 |
+
Caveats: the model was cached for this run; browser version and hardware
|
| 148 |
+
details were not obtainable from the measurement setup (Linux VM, not a
|
| 149 |
+
phone); readout resolution is whole milliseconds. Not a head-to-head with
|
| 150 |
+
any vendor's published number — different hardware, corpus, and setup.
|
| 151 |
+
Methodology travels with the number; see `evals/latency-bench/`.
|
| 152 |
+
|
| 153 |
+
### Synthetic redaction eval (164 cases)
|
| 154 |
+
|
| 155 |
+
`evals/synthetic-164/` carries a 164-case synthetic redaction test set, the
|
| 156 |
+
scoring harness, and v7's results: **0.812 full-value recall** (term recall
|
| 157 |
+
0.615, precision 0.698, clean-case FP rate 0.120). Perfect 1.000 on
|
| 158 |
+
BANK_ACCOUNT, CREDIT_CARD, DRIVERS_LICENSE, EMAIL, PHONE, ROUTING_NUMBER;
|
| 159 |
+
weak on NAME (0.347) and ADDRESS (0.333). See the directory README for
|
| 160 |
+
methodology and the taxonomy note.
|
| 161 |
+
|
| 162 |
---
|
| 163 |
|
| 164 |
## Training data
|