| > Current development result: [Gate03e taxonomy and review readiness](anonymization-gate03e.md). |
| > Three returned religion-status canaries improve; one organization partial mask |
| > and five safe abstentions remain. No new independent holdout; model qualification, |
| > private deployment, sensitive processing and automatic release remain NO-GO. |
|
|
| > Historical first independent result: [Gate03d religious terminology qualification](anonymization-gate03d.md) |
| > preserves the failed Gate03c result, compares limited fixes, and records a FAILED |
| > new independent 160-case first holdout. Model recognition, real sensitive processing, |
| > private deployment and automated release remain NO-GO. Earlier sections are history. |
|
|
| # Anonymization Qualification Report |
|
|
| Current staging/sensitive-processing decision: **NO-GO**. The fresh 120-case |
| [Gate 03c holdout](anonymization-gate03c.md) has two unmasked religious-reference |
| canaries; real identity integration remains BLOCKED. Earlier findings below are |
| preserved as historical evidence. |
|
|
| ## Mission Objective |
| Investigate and test the existing anonymization implementation before integrating it into the OCR workflow. |
|
|
| ## 1. Implementation Audit |
| - **Recognizers Available:** Presidio (Analyzer/Anonymizer) is supported but optional. The code defaults to `nltk` (using `word_tokenize`, `pos_tag`, `ne_chunk`). |
| - **Entity Categories Supported:** Only `PERSON` and `LOCATION` are supported explicitly by both NLTK and Presidio configurations in `anonymize_text`. Organizations, Emails, Phone Numbers, and Religious Affiliations are **not** supported. |
| - **Languages:** English (`en`) is hardcoded in Presidio. German support is virtually non-existent, leading to substantial classification errors (e.g., mislabeling German words as English entities or completely missing them). |
| - **Span Replacement:** When it does detect an entity, the span replacement works reasonably well, but the NLTK logic sometimes joins punctuation incorrectly or misses multi-word entities. |
| - **Fail-Open Security Flaw (Baseline Defect):** If an exception occurs in `anonymize_text_nltk` (e.g., during tokenization or chunking), it catches the exception and returns the *original, unredacted text*. This means a recognition failure leaks the original sensitive content. |
| - **Integration with OCR:** The `ocr/handoff.py` script provides a secure bridge (`anonymization_handoff`) that properly returns the result as `release_review_required=True` and `anonymous=False`. |
|
|
| ## 2. Evaluation Metrics (Document-Level Exact-Match Scoring) |
| A synthetic dataset of 40 text cases was tested. Because the current `anonymize_text` implementation does not yield character offsets (spans), entity-level Precision, Recall, and F1 metrics are explicitly marked as **UNAVAILABLE**. |
|
|
| *Note: The following metrics are published benchmark observations derived from local test runs using the baseline NLTK implementation. They are historical/unverified in standard CI, as the real model evaluation is marked as an opt-in test (`RUN_OPTIONAL_OCR_INTEGRATION=1`) to avoid arbitrary heavy ML dependencies in the default GitHub Actions run.* |
|
|
| ```text |
| --- Benchmark Observations --- |
| PERSON: {'precision': 'UNAVAILABLE', 'recall': 'UNAVAILABLE', 'f1': 'UNAVAILABLE', 'unredacted_canaries': 5} |
| CITY: {'precision': 'UNAVAILABLE', 'recall': 'UNAVAILABLE', 'f1': 'UNAVAILABLE', 'unredacted_canaries': 3} |
| RELIGION: {'precision': 'UNAVAILABLE', 'recall': 'UNAVAILABLE', 'f1': 'UNAVAILABLE', 'unredacted_canaries': 5} |
| EMAIL: {'precision': 'UNAVAILABLE', 'recall': 'UNAVAILABLE', 'f1': 'UNAVAILABLE', 'unredacted_canaries': 2} |
| PHONE: {'precision': 'UNAVAILABLE', 'recall': 'UNAVAILABLE', 'f1': 'UNAVAILABLE', 'unredacted_canaries': 1} |
| ORGANIZATION: {'precision': 'UNAVAILABLE', 'recall': 'UNAVAILABLE', 'f1': 'UNAVAILABLE', 'unredacted_canaries': 2} |
| document_exact_match_rate: 0.175 |
| avg_runtime_ms: ~840.0 |
| ``` |
|
|
| - **Document Exact Match Rate**: Only 17.5% of the test documents were redacted exactly as specified in the gold standard. |
| - **Canary Leakage**: Every required entity category saw significant canary leakage. |
| - **Unsupported Categories**: `RELIGION`, `EMAIL`, `PHONE`, and `ORGANIZATION` are unhandled by the NLTK/Presidio implementations. This leads to large numbers of unredacted canary leaks. |
|
|
| ## 3. Testing Context and Integration |
| The independent verification tests executing globally in CI cover: |
| - Validation of all 40 synthetic gold-standard entity spans. |
| - Acceptance tests proving `anonymization_handoff` handles malformed outputs securely (`SafeError`) and correctly bounds metadata (`release_review_required=True`, `anonymous=False`). |
| - Baseline demonstration of the NLTK fail-open security flaw. |
|
|
| An opt-in integration test (`test_ocr_to_anonymization_integration`) simulates the OCR-to-anonymizer integration boundary. It demonstrates that the legacy baseline **fails to fully redact** required canaries like "Lina", "Wolkenstein", "Berlin", and "evangelisch". |
|
|
| ## 4. Final Decision |
| **Decision:** `NO-GO` |
|
|
| **Reasons:** |
| 1. **Fail-Open Behavior:** The existing NLTK baseline implementation returns the original text on failure. This is a critical security vulnerability. |
| 2. **Missing Entity Support:** Essential categories (`[RELIGION]`, `[EMAIL]`, `[PHONE]`, `[ORGANIZATION]`) are completely unhandled, immediately leading to compliance failures (canary leakage). |
| 3. **Missing Multilingual Support:** The hardcoded English Presidio configuration and NLTK fallback severely underperform on German text. |
|
|
| **Recommendation:** |
| The next implementation gate needs to: |
| - Introduce a multilingual GLiNER-based recognizer (or similar robust multilingual model) to properly handle German and custom entity tags like `RELIGION` without downloading arbitrary corpora. |
| - The recognizer must return explicit spans to calculate accurate P/R/F1. |
| - Implement explicit catch blocks inside the tagger wrappers that fail closed (throw `Exception`) ensuring `ocr/handoff.py` correctly converts it to a `SafeError`. |
|
|
|
|
| --- |
|
|
| ## New span engine qualification — 2026-10-18 |
|
|
| This section qualifies the new explicit implementation. All preceding legacy |
| findings and characterization tests are preserved unchanged; their NO-GO decision |
| still applies to the legacy engine. A test demonstrating its fail-open behavior |
| is not a passing acceptance test for security. |
|
|
| **Decision: PARTIAL for reviewed private integration; NO-GO for automated release.** |
| The boundary/replacement implementation passes its offline acceptance tests. The |
| actual pinned recognizer still leaves one required synthetic CITY canary unmasked. |
| The opt-in canary acceptance test fails deliberately without xfail or altered gold |
| labels. Every candidate remains ineligible for automatic release. |
|
|
| ### Synchronization evidence |
|
|
| - Original checkout: `work`, `9fc91e6f92b4ae9cc902359fd56ef0011a10f9f1`. |
| - Prior review worktree: `review/pr-1-fixes`, `a8f5d7b22c6c5787621973aa5207311d2cc75dca`; uncommitted corrections preserved untouched. |
| - No pre-existing local anonymization implementation branch or commits existed. |
| - Fetched/verified integration branch: `jules-agent-0-deployment-9610133099829518547`, `281c84172281a2294227f15287a4aded8a7ea948`. GitHub confirms PR #2 merged at that SHA. |
| - Feature branch: `feat/ocr-anonymization-failclosed`, initial local HEAD `281c84172281a2294227f15287a4aded8a7ea948`. No rebase/conflicts were necessary; shared integration branch was not changed. |
| - Worktree: `/workspace/augmentoolkit-anonymization`. Final delivery SHA/PR/CI are in the delivery report; `git merge-base --is-ancestor origin/jules-agent-0-deployment-9610133099829518547 HEAD` verifies ancestry. |
|
|
| ### Actual-model evaluation |
|
|
| Selected `VAGOsolutions/SauerkrautLM-GLiNER` revision |
| `ba967ee030b7e679dd5422cb16fcd1b379da38e1`, native local CPU runtime, threshold |
| 0.5, two inference threads. Linux container CPU quota: four CPUs; memory ceiling |
| 32 GiB. No GPU, external model endpoint, public worker or real private documents |
| were used. The actual recognition tests start from synthetic mocked OCRResult |
| objects; this task did not rerun image/PDF OCR benchmarks. |
|
|
| Raw reproducible aggregate evidence: [cpu-pinned.json](anonymization-evaluation/cpu-pinned.json). |
| Installation/runtime pins and exact commands: [integration](anonymization-integration.md). |
|
|
| The 40-case corpus is a faithful copy of merged qualification cases, with language |
| and noise annotations added. A separate 10-case validation set was written before |
| model evaluation and is not read by the runtime. No test entity names are hardcoded |
| in recognizers/detectors. Generic line-wrap handling was developed after the initial |
| evaluation; this is development feedback, not independent external validation. |
| Further held-out real-world evaluation is required to assess overfitting/domain shift. |
|
|
| Metrics below match exact **(start, end, label)** tuples. Wrong boundaries or labels |
| count as both a false positive and a false negative. NER-only metrics and combined |
| NER + deterministic detector metrics are separately recorded in JSON; their final |
| counts match on these small corpora. A canary fails if any original alphanumeric |
| character in its gold span remains uncovered, even if partial masking removed the |
| full literal string. Literal-string checks are also recorded separately. |
|
|
| #### Merged qualification corpus (40 cases) |
|
|
| | Entity | Precision | Recall | F1 | FP | FN | Unredacted canaries | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| | PERSON | 0.818 | 0.900 | 0.857 | 4 | 2 | 0 | |
| | CITY | 1.000 | 0.824 | 0.903 | 0 | 3 | 1 | |
| | RELIGION | 1.000 | 1.000 | 1.000 | 0 | 0 | 0 | |
| | EMAIL | 1.000 | 1.000 | 1.000 | 0 | 0 | 0 | |
| | PHONE | 1.000 | 1.000 | 1.000 | 0 | 0 | 0 | |
| | ORGANIZATION | 0.250 | 0.500 | 0.333 | 3 | 1 | 0 | |
|
|
| Document exact-match rate: **82.5%**; blocked documents: 0. |
|
|
| | Group | Documents | Exact matches | Entity FN | |
| | --- | ---: | ---: | ---: | |
| | de | 10 | 8 | 1 | |
| | en | 29 | 24 | 5 | |
| | mixed | 1 | 1 | 0 | |
| | ocr_noise | 4 | 2 | 2 | |
| |
| Warm latency median 89.4 ms, maximum 335.5 ms. |
| |
| #### Separate validation corpus (10 cases) |
| |
| | Entity | Precision | Recall | F1 | FP | FN | Unredacted canaries | |
| | --- | ---: | ---: | ---: | ---: | ---: | ---: | |
| | PERSON | 1.000 | 1.000 | 1.000 | 0 | 0 | 0 | |
| | CITY | 1.000 | 0.800 | 0.889 | 0 | 1 | 0 | |
| | RELIGION | 0.667 | 1.000 | 0.800 | 1 | 0 | 0 | |
| | EMAIL | 1.000 | 1.000 | 1.000 | 0 | 0 | 0 | |
| | PHONE | 1.000 | 1.000 | 1.000 | 0 | 0 | 0 | |
| | ORGANIZATION | 0.500 | 1.000 | 0.667 | 2 | 0 | 0 | |
| |
| Document exact-match rate: **70.0%**; blocked documents: 0. |
| |
| | Group | Documents | Exact matches | Entity FN | |
| | --- | ---: | ---: | ---: | |
| | de | 5 | 3 | 1 | |
| | en | 5 | 4 | 0 | |
| | ocr_noise | 2 | 2 | 0 | |
|
|
| Warm latency median 96.3 ms, maximum 121.7 ms. |
|
|
| Model loading (including artifact integrity verification and runtime import): |
| **12.42 seconds**. Process peak RSS: **3061.2 MiB**. |
| These are local observations, not a throughput SLA or HF CPU Basic sizing claim. |
|
|
| Gold limitations are retained rather than silently repaired: the historical |
| organization fixture includes its terminal period inside a gold span but expects |
| the period after replacement. Replacement correctly follows source spans; the |
| replacement unit test documents that inconsistency separately. Some historical |
| CITY annotations describe states/rivers, which differs from the prompt's strict |
| city meaning. Aggregate strict scores are still reported against the unchanged |
| gold labels. Remaining category/boundary errors and one canary preclude acceptance |
| of autonomous release. No recognition threshold or name dictionary was tuned to |
| make the suite green. |
|
|
| ### Tests and security qualification matrix |
|
|
| Baseline before implementation: **64 passed, 6 skipped**. Final default suite: |
| **114 passed, 8 skipped**, one existing Starlette TestClient deprecation warning. |
| Ruff check and format check pass. Two skips are explicit real-model tests, three |
| are legacy NLTK/optional characterization cases (NLTK not installed), and three |
| are existing opt-in OCR/backend tests. The historical tests were not edited. |
|
|
| Separate actual-model pytest run: **1 passed, 1 failed**. Real local recognition |
| and two-page OCR handoff passed; the zero-canary release qualification failed. |
| The aggregate evaluator completed successfully and records |
| `recognition_qualification_passed=false`. Its successful execution does not |
| change that failed gate. Default CI intentionally does not download a NER model. |
|
|
| | Boundary | Evidence | Result | |
| | --- | --- | --- | |
| | Missing/corrupt/pin-mismatched model artifacts | Offline cache and manifest tests | PASS: constant error, no candidate | |
| | Model initialization/inference exception | Injected failures, no raw exception/document logging | PASS | |
| | Malformed spans, scores, labels, extra fields, forged copies | Offline strict-schema tests | PASS | |
| | Untrusted tagger text/metadata | New path accepts validated model spans; historical dictionaries cannot enter release | PASS | |
| | Overlap/repeat/umlaut/line wrap/spacing | Gold replacement tests and source-offset mapping tests | PASS for replacement; recognition remains imperfect | |
| | Incomplete/missing/duplicate pages or classification mismatch | Multi-page failures and second-page inference exception | PASS: all-or-nothing | |
| | Raw Markdown/blocks/alternatives/engine/IDs | Canary-rich mocked OCR evidence and immutable snapshot test | PASS: omitted from candidate/export | |
| | D1/D2 local handling | Classification-retention and remote-recognizer rejection tests; no remote adapter | PASS within trusted application boundary | |
| | Empty predictions | Review reason, approval false, automated eligibility false | PASS | |
| | Unauthorized/unreviewed/expired/forged/mismatched approval | Identity callback, digest/HMAC, expiry tests | PASS | |
| | Unsupported exports and D1/D2 bare text export | Export policy tests | PASS: blocked | |
| | Legacy pipeline implicit anonymization | Real chunking entry function rejects anonymize=True | PASS: migration required | |
| | Model detects every required canary | Actual pinned CPU model, 40 + 10 synthetic cases | **FAIL: one CITY canary** | |
| | Original OCR runtime/public worker packaging | Existing OCR tests; no worker/API/Docker/runtime dependency edits | PASS regression; no production deployment performed | |
|
|
| Review authorization is a host application integration contract, not a mocked |
| claim of deployed SSO/ACL enforcement. Tests use synthetic actors/signing keys. |
| Production approval storage, identity integration, CUDA, live Debian-Devil execution |
| and a reviewed destination for D1/D2 export remain operator integration work. No |
| public Debian port was opened, HF deployment changed, production service restarted, |
| or PR automatically merged. |
|
|
|
|
| --- |
|
|
| ## PR #3 follow-up — recognition and private release hardening |
|
|
| The [follow-up report](anonymization-pr3-hardening.md) supersedes the initial new-engine |
| results above while retaining them as historical evidence. The actual final model |
| tests now both pass; zero canaries remain in 40 qualification + 10 validation cases |
| after the general CITY `location` prompt fix. Strict per-category errors are still |
| reported; no gold corpus changes were made. Approval exports now require verified |
| review evidence, destination binding and durable single-use replay protection. |
| Automated release and deployment remain **NO-GO**. See the linked report for all |
| comparison metrics, security regressions and four separate readiness decisions. |
|
|
