Spaces:
Sleeping
Sleeping
SCRUB-WP implement pure Zorgfilter recognizers
Browse filesImplement sixteen context-bound care entities, pass all frozen contracts and corpus expectations, preserve clinical content and keep app registration gated.
- CARE_RECOGNIZER_IMPLEMENTATION_V1.md +76 -0
- CHANGELOG.md +39 -0
- RISK_REGISTER.md +2 -0
- ROADMAP.md +2 -0
- WORKPACKAGES.md +44 -0
- care_recognizer_validation.py +139 -0
- dutch_care_recognizers.py +411 -0
- handover/workpackages/20260803_1652_care_profile_recognizer_implementation.md +61 -0
- output/validation/care_recognizer_implementation_validation.json +23 -0
- scripts/generate_care_recognizer_validation.py +32 -0
- tests/test_care_recognizer_validation.py +47 -0
- tests/test_dutch_care_recognizers.py +168 -0
- workpackage_claims/scrub_wp_care_profile_recognizer_implementation.md +22 -0
CARE_RECOGNIZER_IMPLEMENTATION_V1.md
ADDED
|
@@ -0,0 +1,76 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Zorgfilter v1 — pure recognizer implementation
|
| 2 |
+
|
| 3 |
+
Status: implemented and contract-validated; app registration remains gated
|
| 4 |
+
Workpackage: `SCRUB-WP_CARE_PROFILE_RECOGNIZER_IMPLEMENTATION`
|
| 5 |
+
|
| 6 |
+
## Result
|
| 7 |
+
|
| 8 |
+
The pure module `dutch_care_recognizers.py` implements the frozen Zorgfilter v1 contract without changing the current application.
|
| 9 |
+
|
| 10 |
+
Public API:
|
| 11 |
+
|
| 12 |
+
```python
|
| 13 |
+
get_dutch_care_entity_names()
|
| 14 |
+
get_dutch_care_recognizers(supported_language="en")
|
| 15 |
+
```
|
| 16 |
+
|
| 17 |
+
Implemented entity families:
|
| 18 |
+
|
| 19 |
+
- patient, client, medical-record and EPD/ECD identifiers;
|
| 20 |
+
- health-insurance, referral, treatment and laboratory references;
|
| 21 |
+
- care incident and care indication references;
|
| 22 |
+
- strong-context AGB codes;
|
| 23 |
+
- care-provider names after bounded professional roles;
|
| 24 |
+
- labeled care organizations and bounded care locations;
|
| 25 |
+
- room, bed and apartment references;
|
| 26 |
+
- labeled care-event dates distinct from date of birth.
|
| 27 |
+
|
| 28 |
+
## Validation result
|
| 29 |
+
|
| 30 |
+
```text
|
| 31 |
+
Dedicated entities: 16
|
| 32 |
+
Frozen positive contracts: 37/37 passed
|
| 33 |
+
Forbidden positive collisions: 0
|
| 34 |
+
Frozen negative/collision contracts: 16/16 passed
|
| 35 |
+
Dedicated expectations in eight-document corpus: 54/54 passed
|
| 36 |
+
Protected clinical phrase overlaps: 0
|
| 37 |
+
Full repository regression run #1854: 953 tests passed
|
| 38 |
+
App registration: false
|
| 39 |
+
Production readiness: false
|
| 40 |
+
```
|
| 41 |
+
|
| 42 |
+
Evidence:
|
| 43 |
+
|
| 44 |
+
- `output/validation/care_recognizer_implementation_validation.json`
|
| 45 |
+
- `tests/test_dutch_care_recognizers.py`
|
| 46 |
+
- `tests/test_care_recognizer_validation.py`
|
| 47 |
+
|
| 48 |
+
## Design choices
|
| 49 |
+
|
| 50 |
+
### Strong care labels for references
|
| 51 |
+
|
| 52 |
+
Patient-specific codes are recognized only after explicit labels such as patient number, client number, EPD/ECD number, referral number, sample number or incident number. The recognizer returns only the value span and preserves the label.
|
| 53 |
+
|
| 54 |
+
### Provider roles remain readable
|
| 55 |
+
|
| 56 |
+
A provider name is recognized only after bounded roles such as nurse, GP, treating specialist, personal carer, prescriber or incident reporter. The role and titles such as `dr.` remain readable.
|
| 57 |
+
|
| 58 |
+
### Conservative organization and location handling
|
| 59 |
+
|
| 60 |
+
Organization detection is restricted to labeled fields. Location detection is restricted to bounded phrases such as `location`, `ward`, `team` or `residential location`. The module does not infer every narrative organization as a care organization.
|
| 61 |
+
|
| 62 |
+
### AGB collision protection
|
| 63 |
+
|
| 64 |
+
AGB requires explicit AGB context and exactly eight digits. The dedicated module does not infer AGB from arbitrary numbers and passes the frozen BSN/BIG negative cases. Cross-recognizer precedence with the existing BSN recognizer remains an explicit profile-integration responsibility.
|
| 65 |
+
|
| 66 |
+
### Clinical meaning is preserved
|
| 67 |
+
|
| 68 |
+
The module does not match medication, dosage, administration times, vital signs, laboratory values, pain scores, DBC/ICD codes or role words without a person name. The eight-document corpus produced zero overlaps with protected clinical passages.
|
| 69 |
+
|
| 70 |
+
## Integration boundary
|
| 71 |
+
|
| 72 |
+
The recognizers are intentionally not imported or registered by `presidio_helpers.py` or `presidio_streamlit.py` in this package. The next workpackage centralizes profile composition so Zorg can be added without further scattering profile-specific `if` statements across the UI.
|
| 73 |
+
|
| 74 |
+
No export, Scrub Key, reinsert, runtime, dependency or cloud-processing behavior changed.
|
| 75 |
+
|
| 76 |
+
Human review remains mandatory. Passing the synthetic contract does not establish production readiness.
|
CHANGELOG.md
CHANGED
|
@@ -1,3 +1,42 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
## 2026-08-03 16:34 Europe/Amsterdam — SCRUB-WP_CARE_PROFILE_RECOGNIZER_CONTRACT_TESTS
|
| 2 |
|
| 3 |
Status: completed; test/specification contract frozen.
|
|
|
|
| 1 |
+
## 2026-08-03 16:52 Europe/Amsterdam — SCRUB-WP_CARE_PROFILE_RECOGNIZER_IMPLEMENTATION
|
| 2 |
+
|
| 3 |
+
Status: completed; pure recognizer implementation validated.
|
| 4 |
+
|
| 5 |
+
Purpose:
|
| 6 |
+
- Implement the frozen Zorgfilter recognizer contract without registering it in the current app.
|
| 7 |
+
|
| 8 |
+
Files added:
|
| 9 |
+
- `dutch_care_recognizers.py`
|
| 10 |
+
- `care_recognizer_validation.py`
|
| 11 |
+
- `CARE_RECOGNIZER_IMPLEMENTATION_V1.md`
|
| 12 |
+
- `scripts/generate_care_recognizer_validation.py`
|
| 13 |
+
- `tests/test_dutch_care_recognizers.py`
|
| 14 |
+
- `tests/test_care_recognizer_validation.py`
|
| 15 |
+
- `output/validation/care_recognizer_implementation_validation.json`
|
| 16 |
+
- `workpackage_claims/scrub_wp_care_profile_recognizer_implementation.md`
|
| 17 |
+
- `handover/workpackages/20260803_1652_care_profile_recognizer_implementation.md`
|
| 18 |
+
|
| 19 |
+
Implementation result:
|
| 20 |
+
- implemented sixteen dedicated, context-bound care entities;
|
| 21 |
+
- returned exact value-only Presidio spans with explanations and metadata;
|
| 22 |
+
- corrected bounded variants for Dutch client labels, prescription numbers, generic incident labels and lowercase residential-care organization labels;
|
| 23 |
+
- passed all frozen positive and negative contracts;
|
| 24 |
+
- covered all 54 dedicated corpus expectations with zero protected-clinical overlaps;
|
| 25 |
+
- kept app registration explicitly false.
|
| 26 |
+
|
| 27 |
+
Validation:
|
| 28 |
+
- initial run #1850 exposed five bounded missing context variants and no clinical-overmasking failure;
|
| 29 |
+
- corrected GitHub Actions run #1854 passed: 953 tests;
|
| 30 |
+
- final clean validation pending after governance finalization;
|
| 31 |
+
- Hugging Face sync not functionally relevant;
|
| 32 |
+
- app verification not applicable.
|
| 33 |
+
|
| 34 |
+
Intentionally not changed:
|
| 35 |
+
- `presidio_helpers.py`, `presidio_streamlit.py` or current profile behavior;
|
| 36 |
+
- thresholds, entity defaults or generic NER;
|
| 37 |
+
- review, export, Scrub Key or reinsert semantics;
|
| 38 |
+
- runtime, dependencies or cloud processing.
|
| 39 |
+
|
| 40 |
## 2026-08-03 16:34 Europe/Amsterdam — SCRUB-WP_CARE_PROFILE_RECOGNIZER_CONTRACT_TESTS
|
| 41 |
|
| 42 |
Status: completed; test/specification contract frozen.
|
RISK_REGISTER.md
CHANGED
|
@@ -229,6 +229,8 @@ Gap triage classified all 81 expectations. The largest unresolved family is cont
|
|
| 229 |
|
| 230 |
The recognizer contract is now frozen with sixteen dedicated entities, 37 positive exact-span cases and 16 negative/collision/preservation cases. Implementation must pass these fixtures before any app registration or UI promotion.
|
| 231 |
|
|
|
|
|
|
|
| 232 |
---
|
| 233 |
|
| 234 |
## Product-claim boundary
|
|
|
|
| 229 |
|
| 230 |
The recognizer contract is now frozen with sixteen dedicated entities, 37 positive exact-span cases and 16 negative/collision/preservation cases. Implementation must pass these fixtures before any app registration or UI promotion.
|
| 231 |
|
| 232 |
+
The pure recognizer implementation now passes all frozen fixtures and all 54 dedicated corpus expectations with zero protected-clinical overlaps. Risk R10 remains open because generic NER composition, AGB/BSN cross-recognizer precedence, visible profile policy, cross-profile regression and live app verification are not yet complete.
|
| 233 |
+
|
| 234 |
---
|
| 235 |
|
| 236 |
## Product-claim boundary
|
ROADMAP.md
CHANGED
|
@@ -467,6 +467,8 @@ The existing broad `NL_HEALTHCARE_REFERENCE` category must be assessed and split
|
|
| 467 |
|
| 468 |
Current-engine baseline evidence is now recorded: 25 of 81 expected replace/review values were found as exact normalized spans, only 14 under the intended entity type, 11 were misclassified and 56 were missed. The bounded custom-rule baseline produced zero overlaps with the designated clinical preserve passages. Generic NER was excluded, so PERSON and e-mail results are not full-app measurements.
|
| 469 |
|
|
|
|
|
|
|
| 470 |
### Current and final interface direction
|
| 471 |
|
| 472 |
Current prototype after test-gated integration:
|
|
|
|
| 467 |
|
| 468 |
Current-engine baseline evidence is now recorded: 25 of 81 expected replace/review values were found as exact normalized spans, only 14 under the intended entity type, 11 were misclassified and 56 were missed. The bounded custom-rule baseline produced zero overlaps with the designated clinical preserve passages. Generic NER was excluded, so PERSON and e-mail results are not full-app measurements.
|
| 469 |
|
| 470 |
+
The dedicated pure recognizer implementation is now green but remains unregistered: sixteen care entities pass 37/37 positive contracts, 16/16 negative/collision contracts and all 54 dedicated expectations in the eight-document corpus, with zero protected-clinical overlaps. The next gate is central profile composition and AGB/BSN precedence before any visible Zorg profile is added.
|
| 471 |
+
|
| 472 |
### Current and final interface direction
|
| 473 |
|
| 474 |
Current prototype after test-gated integration:
|
WORKPACKAGES.md
CHANGED
|
@@ -1,3 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
## 2026-08-03 16:34 Europe/Amsterdam — SCRUB-WP_CARE_PROFILE_RECOGNIZER_CONTRACT_TESTS
|
| 2 |
|
| 3 |
Status: completed; recognizer contract frozen before implementation.
|
|
|
|
| 1 |
+
## 2026-08-03 16:52 Europe/Amsterdam — SCRUB-WP_CARE_PROFILE_RECOGNIZER_IMPLEMENTATION
|
| 2 |
+
|
| 3 |
+
Status: completed; pure recognizers implemented and contract-validated, app registration still closed.
|
| 4 |
+
|
| 5 |
+
Goal:
|
| 6 |
+
- Implement the frozen sixteen-entity Zorgfilter recognizer module without changing current app behavior.
|
| 7 |
+
|
| 8 |
+
Result:
|
| 9 |
+
|
| 10 |
+
```text
|
| 11 |
+
Dedicated entities: 16
|
| 12 |
+
Positive contracts: 37/37 passed
|
| 13 |
+
Forbidden positive collisions: 0
|
| 14 |
+
Negative/collision contracts: 16/16 passed
|
| 15 |
+
Dedicated corpus expectations: 54/54 passed
|
| 16 |
+
Protected clinical phrase overlaps: 0
|
| 17 |
+
Full regression run #1854: 953 tests passed
|
| 18 |
+
App registration: false
|
| 19 |
+
```
|
| 20 |
+
|
| 21 |
+
Implementation:
|
| 22 |
+
- `dutch_care_recognizers.py` with value-only Presidio capture results;
|
| 23 |
+
- strong-context administrative references and AGB;
|
| 24 |
+
- provider-name recognition preserving professional roles;
|
| 25 |
+
- labeled organizations and bounded locations;
|
| 26 |
+
- room/bed/apartment references;
|
| 27 |
+
- care-event dates separated from date of birth;
|
| 28 |
+
- no Streamlit, network, AI, cloud or file-write behavior.
|
| 29 |
+
|
| 30 |
+
Evidence:
|
| 31 |
+
- `CARE_RECOGNIZER_IMPLEMENTATION_V1.md`
|
| 32 |
+
- `output/validation/care_recognizer_implementation_validation.json`
|
| 33 |
+
|
| 34 |
+
Active next package:
|
| 35 |
+
- `SCRUB-WP_RECOGNITION_PROFILE_CONFIGURATION_REFACTOR`
|
| 36 |
+
|
| 37 |
+
Boundaries:
|
| 38 |
+
- recognizers are not registered in `presidio_helpers.py` or the UI;
|
| 39 |
+
- generic PERSON/e-mail remain generic-profile dependencies;
|
| 40 |
+
- AGB/BSN profile-level precedence remains to be validated;
|
| 41 |
+
- no export, Scrub Key or reinsert change;
|
| 42 |
+
- human review remains required;
|
| 43 |
+
- production readiness remains false.
|
| 44 |
+
|
| 45 |
## 2026-08-03 16:34 Europe/Amsterdam — SCRUB-WP_CARE_PROFILE_RECOGNIZER_CONTRACT_TESTS
|
| 46 |
|
| 47 |
Status: completed; recognizer contract frozen before implementation.
|
care_recognizer_validation.py
ADDED
|
@@ -0,0 +1,139 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pure validation report for the Zorgfilter v1 recognizer implementation."""
|
| 2 |
+
|
| 3 |
+
from __future__ import annotations
|
| 4 |
+
|
| 5 |
+
from typing import Any, Dict
|
| 6 |
+
|
| 7 |
+
from care_recognizer_contracts import ALL_POSITIVE_CASES, NEGATIVE_CASES
|
| 8 |
+
from care_test_examples import TEST_CASES
|
| 9 |
+
from dutch_care_recognizers import (
|
| 10 |
+
get_dutch_care_entity_names,
|
| 11 |
+
get_dutch_care_recognizers,
|
| 12 |
+
)
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
def _detect(text: str) -> list[dict[str, Any]]:
|
| 16 |
+
entities = get_dutch_care_entity_names()
|
| 17 |
+
rows: list[dict[str, Any]] = []
|
| 18 |
+
for recognizer in get_dutch_care_recognizers(supported_language="en"):
|
| 19 |
+
for result in recognizer.analyze(text, entities=entities, nlp_artifacts=None):
|
| 20 |
+
rows.append(
|
| 21 |
+
{
|
| 22 |
+
"entity_type": result.entity_type,
|
| 23 |
+
"start": int(result.start),
|
| 24 |
+
"end": int(result.end),
|
| 25 |
+
"text": text[result.start:result.end],
|
| 26 |
+
"score": float(result.score),
|
| 27 |
+
"recognizer": recognizer.name,
|
| 28 |
+
}
|
| 29 |
+
)
|
| 30 |
+
return sorted(rows, key=lambda row: (row["start"], row["end"], row["entity_type"]))
|
| 31 |
+
|
| 32 |
+
|
| 33 |
+
def _has_exact(rows: list[dict[str, Any]], entity_type: str, value: str) -> bool:
|
| 34 |
+
return any(
|
| 35 |
+
row["entity_type"] == entity_type and row["text"] == value
|
| 36 |
+
for row in rows
|
| 37 |
+
)
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def build_care_recognizer_validation() -> Dict[str, Any]:
|
| 41 |
+
"""Return deterministic contract and corpus implementation evidence."""
|
| 42 |
+
|
| 43 |
+
positive_failures: list[dict[str, Any]] = []
|
| 44 |
+
forbidden_positive_failures: list[dict[str, Any]] = []
|
| 45 |
+
for case in ALL_POSITIVE_CASES:
|
| 46 |
+
rows = _detect(case["text"])
|
| 47 |
+
if not _has_exact(rows, case["entity_type"], case["expected_value"]):
|
| 48 |
+
positive_failures.append(
|
| 49 |
+
{
|
| 50 |
+
"id": case["id"],
|
| 51 |
+
"entity_type": case["entity_type"],
|
| 52 |
+
"expected_value": case["expected_value"],
|
| 53 |
+
"detected": rows,
|
| 54 |
+
}
|
| 55 |
+
)
|
| 56 |
+
for forbidden_entity in case["forbidden_entities"]:
|
| 57 |
+
if _has_exact(rows, forbidden_entity, case["expected_value"]):
|
| 58 |
+
forbidden_positive_failures.append(
|
| 59 |
+
{
|
| 60 |
+
"id": case["id"],
|
| 61 |
+
"forbidden_entity": forbidden_entity,
|
| 62 |
+
"value": case["expected_value"],
|
| 63 |
+
}
|
| 64 |
+
)
|
| 65 |
+
|
| 66 |
+
negative_failures: list[dict[str, Any]] = []
|
| 67 |
+
for case in NEGATIVE_CASES:
|
| 68 |
+
rows = _detect(case["text"])
|
| 69 |
+
violations = [
|
| 70 |
+
row for row in rows
|
| 71 |
+
if row["entity_type"] in set(case["forbidden_entities"])
|
| 72 |
+
]
|
| 73 |
+
if violations:
|
| 74 |
+
negative_failures.append(
|
| 75 |
+
{
|
| 76 |
+
"id": case["id"],
|
| 77 |
+
"violations": violations,
|
| 78 |
+
}
|
| 79 |
+
)
|
| 80 |
+
|
| 81 |
+
dedicated_entities = set(get_dutch_care_entity_names())
|
| 82 |
+
corpus_expected = 0
|
| 83 |
+
corpus_found = 0
|
| 84 |
+
corpus_failures: list[dict[str, Any]] = []
|
| 85 |
+
preserve_overlaps: list[dict[str, Any]] = []
|
| 86 |
+
for case in TEST_CASES:
|
| 87 |
+
rows = _detect(case["text"])
|
| 88 |
+
for bucket in ("replace", "review_selected"):
|
| 89 |
+
for expectation in case[bucket]:
|
| 90 |
+
if expectation["entity_type"] not in dedicated_entities:
|
| 91 |
+
continue
|
| 92 |
+
corpus_expected += 1
|
| 93 |
+
if _has_exact(rows, expectation["entity_type"], expectation["value"]):
|
| 94 |
+
corpus_found += 1
|
| 95 |
+
else:
|
| 96 |
+
corpus_failures.append(
|
| 97 |
+
{
|
| 98 |
+
"case_id": case["id"],
|
| 99 |
+
"entity_type": expectation["entity_type"],
|
| 100 |
+
"value": expectation["value"],
|
| 101 |
+
}
|
| 102 |
+
)
|
| 103 |
+
for phrase in case["preserve"]:
|
| 104 |
+
phrase_start = case["text"].index(phrase)
|
| 105 |
+
phrase_end = phrase_start + len(phrase)
|
| 106 |
+
for row in rows:
|
| 107 |
+
if phrase_start < row["end"] and row["start"] < phrase_end:
|
| 108 |
+
preserve_overlaps.append(
|
| 109 |
+
{
|
| 110 |
+
"case_id": case["id"],
|
| 111 |
+
"phrase": phrase,
|
| 112 |
+
"entity_type": row["entity_type"],
|
| 113 |
+
"detected_text": row["text"],
|
| 114 |
+
}
|
| 115 |
+
)
|
| 116 |
+
|
| 117 |
+
return {
|
| 118 |
+
"schema_version": "1.0",
|
| 119 |
+
"module": "dutch_care_recognizers",
|
| 120 |
+
"synthetic_data_only": True,
|
| 121 |
+
"production_ready": False,
|
| 122 |
+
"human_review_required": True,
|
| 123 |
+
"entity_count": len(dedicated_entities),
|
| 124 |
+
"positive_contract_count": len(ALL_POSITIVE_CASES),
|
| 125 |
+
"positive_contract_passed": len(ALL_POSITIVE_CASES) - len(positive_failures),
|
| 126 |
+
"positive_contract_failures": positive_failures,
|
| 127 |
+
"forbidden_positive_failure_count": len(forbidden_positive_failures),
|
| 128 |
+
"forbidden_positive_failures": forbidden_positive_failures,
|
| 129 |
+
"negative_contract_count": len(NEGATIVE_CASES),
|
| 130 |
+
"negative_contract_passed": len(NEGATIVE_CASES) - len(negative_failures),
|
| 131 |
+
"negative_contract_failures": negative_failures,
|
| 132 |
+
"corpus_dedicated_expectation_count": corpus_expected,
|
| 133 |
+
"corpus_dedicated_found_count": corpus_found,
|
| 134 |
+
"corpus_dedicated_failures": corpus_failures,
|
| 135 |
+
"clinical_preserve_overlap_count": len(preserve_overlaps),
|
| 136 |
+
"clinical_preserve_overlaps": preserve_overlaps,
|
| 137 |
+
"app_registered": False,
|
| 138 |
+
"next_workpackage": "SCRUB-WP_RECOGNITION_PROFILE_CONFIGURATION_REFACTOR",
|
| 139 |
+
}
|
dutch_care_recognizers.py
ADDED
|
@@ -0,0 +1,411 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Pure Dutch care recognizers for SolidPrivacy Scrub Zorgfilter v1.
|
| 2 |
+
|
| 3 |
+
The module implements the frozen care-recognizer contract only. It is not
|
| 4 |
+
registered in the current app yet and contains no Streamlit, network, AI, cloud
|
| 5 |
+
or file-system behavior.
|
| 6 |
+
|
| 7 |
+
The current application analyses with ``language="en"`` while adding Dutch
|
| 8 |
+
custom recognizers. Therefore these recognizers default to
|
| 9 |
+
``supported_language="en"``.
|
| 10 |
+
"""
|
| 11 |
+
|
| 12 |
+
from __future__ import annotations
|
| 13 |
+
|
| 14 |
+
import re
|
| 15 |
+
from typing import Iterable, List, Sequence, Tuple
|
| 16 |
+
|
| 17 |
+
from presidio_analyzer import (
|
| 18 |
+
AnalysisExplanation,
|
| 19 |
+
EntityRecognizer,
|
| 20 |
+
RecognizerResult,
|
| 21 |
+
)
|
| 22 |
+
|
| 23 |
+
|
| 24 |
+
DUTCH_CARE_ENTITY_NAMES: tuple[str, ...] = (
|
| 25 |
+
"NL_PATIENT_NUMBER",
|
| 26 |
+
"NL_CARE_CLIENT_NUMBER",
|
| 27 |
+
"NL_MEDICAL_RECORD_NUMBER",
|
| 28 |
+
"NL_EPD_ECD_NUMBER",
|
| 29 |
+
"NL_HEALTH_INSURANCE_NUMBER",
|
| 30 |
+
"NL_REFERRAL_NUMBER",
|
| 31 |
+
"NL_TREATMENT_REFERENCE",
|
| 32 |
+
"NL_LAB_SAMPLE_NUMBER",
|
| 33 |
+
"NL_CARE_INCIDENT_NUMBER",
|
| 34 |
+
"NL_CARE_INDICATION_REFERENCE",
|
| 35 |
+
"NL_AGB_CODE",
|
| 36 |
+
"NL_CARE_PROVIDER_NAME",
|
| 37 |
+
"NL_CARE_ORGANIZATION",
|
| 38 |
+
"NL_CARE_LOCATION_REFERENCE",
|
| 39 |
+
"NL_ROOM_OR_BED_REFERENCE",
|
| 40 |
+
"NL_CARE_EVENT_DATE",
|
| 41 |
+
)
|
| 42 |
+
|
| 43 |
+
# Keep name recognition strict and label-bound. Generic PERSON recognition stays
|
| 44 |
+
# in the generic local NER/profile layer.
|
| 45 |
+
UPPER = r"(?-i:[A-ZÀ-ÖØ-Þ])"
|
| 46 |
+
NAME_TOKEN = rf"{UPPER}[A-Za-zÀ-ÖØ-öø-ÿ'’\-]*"
|
| 47 |
+
NAME_PARTICLE = r"(?:van|de|der|den|ten|ter|el|al|la|du|op|aan|bin|ibn)"
|
| 48 |
+
NAME_GAP = r"[ \t]+"
|
| 49 |
+
NAME_VALUE = (
|
| 50 |
+
rf"{NAME_TOKEN}"
|
| 51 |
+
rf"(?:(?:{NAME_GAP}{NAME_PARTICLE})?{NAME_GAP}{NAME_TOKEN}){{0,4}}"
|
| 52 |
+
)
|
| 53 |
+
|
| 54 |
+
DATE_VALUE = r"\d{1,2}[-/.]\d{1,2}[-/.]\d{2,4}|\d{4}[-/.]\d{1,2}[-/.]\d{1,2}"
|
| 55 |
+
# Strong labels make numeric patient identifiers safe to support. Unlabelled
|
| 56 |
+
# numeric strings are deliberately outside this module.
|
| 57 |
+
REFERENCE_VALUE = r"(?:[A-Z0-9]+(?:[-_/][A-Z0-9]+){1,8}|\d{5,16})"
|
| 58 |
+
|
| 59 |
+
|
| 60 |
+
class CareRegexCaptureRecognizer(EntityRecognizer):
|
| 61 |
+
"""Return only a named ``value`` capture from bounded care-context regexes."""
|
| 62 |
+
|
| 63 |
+
def __init__(
|
| 64 |
+
self,
|
| 65 |
+
entity: str,
|
| 66 |
+
patterns: Sequence[Tuple[str, str]],
|
| 67 |
+
score: float,
|
| 68 |
+
context: Iterable[str] | None = None,
|
| 69 |
+
supported_language: str = "en",
|
| 70 |
+
) -> None:
|
| 71 |
+
super().__init__(
|
| 72 |
+
supported_entities=[entity],
|
| 73 |
+
supported_language=supported_language,
|
| 74 |
+
name=f"{entity}_care_capture_recognizer",
|
| 75 |
+
)
|
| 76 |
+
self.entity = entity
|
| 77 |
+
self.patterns = tuple(
|
| 78 |
+
(
|
| 79 |
+
name,
|
| 80 |
+
re.compile(pattern, flags=re.IGNORECASE | re.MULTILINE),
|
| 81 |
+
)
|
| 82 |
+
for name, pattern in patterns
|
| 83 |
+
)
|
| 84 |
+
self.score = float(score)
|
| 85 |
+
self.context = tuple(context or ())
|
| 86 |
+
|
| 87 |
+
def load(self) -> None:
|
| 88 |
+
return None
|
| 89 |
+
|
| 90 |
+
def analyze(
|
| 91 |
+
self,
|
| 92 |
+
text: str,
|
| 93 |
+
entities: List[str],
|
| 94 |
+
nlp_artifacts=None,
|
| 95 |
+
) -> List[RecognizerResult]:
|
| 96 |
+
if entities and self.entity not in entities:
|
| 97 |
+
return []
|
| 98 |
+
|
| 99 |
+
source_text = text or ""
|
| 100 |
+
results: List[RecognizerResult] = []
|
| 101 |
+
seen: set[tuple[int, int]] = set()
|
| 102 |
+
|
| 103 |
+
for pattern_name, pattern in self.patterns:
|
| 104 |
+
for match in pattern.finditer(source_text):
|
| 105 |
+
if "value" not in match.groupdict() or match.group("value") is None:
|
| 106 |
+
continue
|
| 107 |
+
start, end = match.span("value")
|
| 108 |
+
|
| 109 |
+
while start < end and source_text[start].isspace():
|
| 110 |
+
start += 1
|
| 111 |
+
while end > start and source_text[end - 1] in " \t\r\n.,;:":
|
| 112 |
+
end -= 1
|
| 113 |
+
if start >= end or (start, end) in seen:
|
| 114 |
+
continue
|
| 115 |
+
seen.add((start, end))
|
| 116 |
+
|
| 117 |
+
explanation = AnalysisExplanation(
|
| 118 |
+
recognizer=self.name,
|
| 119 |
+
original_score=self.score,
|
| 120 |
+
pattern_name=pattern_name,
|
| 121 |
+
pattern=pattern.pattern,
|
| 122 |
+
textual_explanation=(
|
| 123 |
+
f"Detected by `{self.name}` using bounded care-context "
|
| 124 |
+
f"pattern `{pattern_name}`. Only the named value span is "
|
| 125 |
+
"returned so the care label, professional role and clinical "
|
| 126 |
+
"meaning remain readable."
|
| 127 |
+
),
|
| 128 |
+
)
|
| 129 |
+
results.append(
|
| 130 |
+
RecognizerResult(
|
| 131 |
+
entity_type=self.entity,
|
| 132 |
+
start=start,
|
| 133 |
+
end=end,
|
| 134 |
+
score=self.score,
|
| 135 |
+
analysis_explanation=explanation,
|
| 136 |
+
recognition_metadata={
|
| 137 |
+
RecognizerResult.RECOGNIZER_NAME_KEY: self.name,
|
| 138 |
+
RecognizerResult.RECOGNIZER_IDENTIFIER_KEY: getattr(
|
| 139 |
+
self, "id", self.name
|
| 140 |
+
),
|
| 141 |
+
},
|
| 142 |
+
)
|
| 143 |
+
)
|
| 144 |
+
|
| 145 |
+
return results
|
| 146 |
+
|
| 147 |
+
|
| 148 |
+
def _reference_recognizer(
|
| 149 |
+
entity: str,
|
| 150 |
+
labels_regex: str,
|
| 151 |
+
score: float,
|
| 152 |
+
supported_language: str,
|
| 153 |
+
) -> CareRegexCaptureRecognizer:
|
| 154 |
+
return CareRegexCaptureRecognizer(
|
| 155 |
+
entity=entity,
|
| 156 |
+
patterns=[
|
| 157 |
+
(
|
| 158 |
+
f"{entity.lower()}_labelled_value",
|
| 159 |
+
rf"(?<!\w)(?:{labels_regex})(?!\w)\s*(?:is|:|#|=|-)?\s*"
|
| 160 |
+
rf"(?P<value>{REFERENCE_VALUE})(?![A-Z0-9_/-])",
|
| 161 |
+
)
|
| 162 |
+
],
|
| 163 |
+
score=score,
|
| 164 |
+
context=re.split(r"\|", labels_regex),
|
| 165 |
+
supported_language=supported_language,
|
| 166 |
+
)
|
| 167 |
+
|
| 168 |
+
|
| 169 |
+
def _provider_recognizer(supported_language: str) -> CareRegexCaptureRecognizer:
|
| 170 |
+
title = r"(?:(?:dr|dokter)\.?\s+)?"
|
| 171 |
+
patterns = [
|
| 172 |
+
(
|
| 173 |
+
"reporter_with_professional_role",
|
| 174 |
+
rf"\brapporteur\s*:\s*(?:verpleegkundige|verzorgende(?:\s+IG)?|"
|
| 175 |
+
rf"persoonlijk\s+begeleider|zorgco[oö]rdinator)\s+{title}"
|
| 176 |
+
rf"(?P<value>{NAME_VALUE})\b",
|
| 177 |
+
),
|
| 178 |
+
(
|
| 179 |
+
"reporting_or_transfer_role",
|
| 180 |
+
rf"\b(?:overdragend\s+verpleegkundige|persoonlijk\s+begeleider|"
|
| 181 |
+
rf"behandelend\s+(?:internist|arts|specialist)|hoofdbehandelaar|"
|
| 182 |
+
rf"regiebehandelaar|huisarts|betrokken\s+arts|aanvrager)\s*:\s*"
|
| 183 |
+
rf"{title}(?P<value>{NAME_VALUE})\b",
|
| 184 |
+
),
|
| 185 |
+
(
|
| 186 |
+
"prescriber_with_optional_role",
|
| 187 |
+
rf"\bvoorschrijver\s*:\s*(?:(?:huisarts|arts|specialist|apotheker)\s+)?"
|
| 188 |
+
rf"{title}(?P<value>{NAME_VALUE})\b",
|
| 189 |
+
),
|
| 190 |
+
(
|
| 191 |
+
"incident_reporter_with_role",
|
| 192 |
+
rf"\bmelder\s*:\s*(?:verzorgende(?:\s+IG)?|verpleegkundige|arts|"
|
| 193 |
+
rf"persoonlijk\s+begeleider)\s+{title}(?P<value>{NAME_VALUE})\b",
|
| 194 |
+
),
|
| 195 |
+
]
|
| 196 |
+
return CareRegexCaptureRecognizer(
|
| 197 |
+
entity="NL_CARE_PROVIDER_NAME",
|
| 198 |
+
patterns=patterns,
|
| 199 |
+
score=0.84,
|
| 200 |
+
context=(
|
| 201 |
+
"rapporteur",
|
| 202 |
+
"verpleegkundige",
|
| 203 |
+
"huisarts",
|
| 204 |
+
"behandelend arts",
|
| 205 |
+
"persoonlijk begeleider",
|
| 206 |
+
"voorschrijver",
|
| 207 |
+
"melder",
|
| 208 |
+
),
|
| 209 |
+
supported_language=supported_language,
|
| 210 |
+
)
|
| 211 |
+
|
| 212 |
+
|
| 213 |
+
def _organization_recognizer(supported_language: str) -> CareRegexCaptureRecognizer:
|
| 214 |
+
# Bounded to labelled fields and stopped before sublocation punctuation.
|
| 215 |
+
organization_value = r"(?-i:[A-ZÀ-ÖØ-Þ])[^,\.\r\n]{1,79}?"
|
| 216 |
+
patterns = [
|
| 217 |
+
(
|
| 218 |
+
"labelled_care_organization",
|
| 219 |
+
rf"\b(?:zorgorganisatie|ziekenhuis|apotheek|laboratorium|"
|
| 220 |
+
rf"verzendende\s+organisatie|ontvangende\s+organisatie)\s*:\s*"
|
| 221 |
+
rf"(?P<value>{organization_value})(?=\s*(?:,|\.|\r?$))",
|
| 222 |
+
),
|
| 223 |
+
(
|
| 224 |
+
"location_line_primary_organization",
|
| 225 |
+
rf"\blocatie\s*:\s*(?P<value>{organization_value})"
|
| 226 |
+
rf"(?=\s*,\s*(?:woonlocatie|locatie|afdeling|team|kamer|bed|appartement)\b)",
|
| 227 |
+
),
|
| 228 |
+
(
|
| 229 |
+
"location_line_lowercase_care_organization",
|
| 230 |
+
r"\blocatie\s*:\s*(?P<value>(?:woonzorgcentrum|zorgcentrum|"
|
| 231 |
+
r"verpleeghuis|ziekenhuis|zorggroep)\s+"
|
| 232 |
+
r"(?-i:[A-ZÀ-ÖØ-Þ])[^,\.\r\n]{1,60}?)"
|
| 233 |
+
r"(?=\s*,\s*(?:woonlocatie|locatie|afdeling|team|kamer|bed|appartement)\b)",
|
| 234 |
+
),
|
| 235 |
+
(
|
| 236 |
+
"labelled_practice_phrase",
|
| 237 |
+
rf"(?P<value>praktijk\s+(?-i:[A-ZÀ-ÖØ-Þ])[^,\.\r\n]{{1,60}}?)"
|
| 238 |
+
rf"(?=\s*(?:,|\.|\r?$))",
|
| 239 |
+
),
|
| 240 |
+
]
|
| 241 |
+
return CareRegexCaptureRecognizer(
|
| 242 |
+
entity="NL_CARE_ORGANIZATION",
|
| 243 |
+
patterns=patterns,
|
| 244 |
+
score=0.74,
|
| 245 |
+
context=(
|
| 246 |
+
"zorgorganisatie",
|
| 247 |
+
"ziekenhuis",
|
| 248 |
+
"apotheek",
|
| 249 |
+
"laboratorium",
|
| 250 |
+
"praktijk",
|
| 251 |
+
"verzendende organisatie",
|
| 252 |
+
"ontvangende organisatie",
|
| 253 |
+
),
|
| 254 |
+
supported_language=supported_language,
|
| 255 |
+
)
|
| 256 |
+
|
| 257 |
+
|
| 258 |
+
def _location_recognizer(supported_language: str) -> CareRegexCaptureRecognizer:
|
| 259 |
+
patterns = [
|
| 260 |
+
(
|
| 261 |
+
"bounded_named_care_location",
|
| 262 |
+
r"(?P<value>\b(?:woonlocatie|locatie|afdeling|team)\s+"
|
| 263 |
+
r"(?-i:[A-ZÀ-ÖØ-Þ])[A-Za-zÀ-ÖØ-öø-ÿ0-9'’\-]*"
|
| 264 |
+
r"(?:[ \t]+[A-Za-zÀ-ÖØ-öø-ÿ0-9'’\-]+){0,4})"
|
| 265 |
+
r"(?=\s*(?:,|\.|\r?$))",
|
| 266 |
+
)
|
| 267 |
+
]
|
| 268 |
+
return CareRegexCaptureRecognizer(
|
| 269 |
+
entity="NL_CARE_LOCATION_REFERENCE",
|
| 270 |
+
patterns=patterns,
|
| 271 |
+
score=0.72,
|
| 272 |
+
context=("woonlocatie", "locatie", "afdeling", "team"),
|
| 273 |
+
supported_language=supported_language,
|
| 274 |
+
)
|
| 275 |
+
|
| 276 |
+
|
| 277 |
+
def _room_bed_recognizer(supported_language: str) -> CareRegexCaptureRecognizer:
|
| 278 |
+
return CareRegexCaptureRecognizer(
|
| 279 |
+
entity="NL_ROOM_OR_BED_REFERENCE",
|
| 280 |
+
patterns=[
|
| 281 |
+
(
|
| 282 |
+
"room_bed_or_apartment_code",
|
| 283 |
+
r"(?P<value>\b(?:kamer|bed|appartement)\s+"
|
| 284 |
+
r"(?:[A-Z](?:-[A-Z0-9]+)?|\d{1,4}[A-Z]?|[A-Z]\d{1,4}|"
|
| 285 |
+
r"[A-Z0-9]+-[A-Z0-9]+))\b",
|
| 286 |
+
)
|
| 287 |
+
],
|
| 288 |
+
score=0.72,
|
| 289 |
+
context=("kamer", "bed", "appartement"),
|
| 290 |
+
supported_language=supported_language,
|
| 291 |
+
)
|
| 292 |
+
|
| 293 |
+
|
| 294 |
+
def _event_date_recognizer(supported_language: str) -> CareRegexCaptureRecognizer:
|
| 295 |
+
labels = (
|
| 296 |
+
r"opnamedatum|ontslagdatum|rapportagedatum|evaluatiedatum|incidentdatum|"
|
| 297 |
+
r"afnamedatum|afspraakdatum|behandeldatum|onderzoeksdatum"
|
| 298 |
+
)
|
| 299 |
+
return CareRegexCaptureRecognizer(
|
| 300 |
+
entity="NL_CARE_EVENT_DATE",
|
| 301 |
+
patterns=[
|
| 302 |
+
(
|
| 303 |
+
"labelled_care_event_date",
|
| 304 |
+
rf"\b(?:{labels})\s*(?:is|:)?\s*(?P<value>{DATE_VALUE})\b",
|
| 305 |
+
)
|
| 306 |
+
],
|
| 307 |
+
score=0.82,
|
| 308 |
+
context=tuple(labels.split("|")),
|
| 309 |
+
supported_language=supported_language,
|
| 310 |
+
)
|
| 311 |
+
|
| 312 |
+
|
| 313 |
+
def get_dutch_care_entity_names() -> List[str]:
|
| 314 |
+
"""Return the frozen dedicated Zorgfilter v1 entity names."""
|
| 315 |
+
|
| 316 |
+
return list(DUTCH_CARE_ENTITY_NAMES)
|
| 317 |
+
|
| 318 |
+
|
| 319 |
+
def get_dutch_care_recognizers(
|
| 320 |
+
supported_language: str = "en",
|
| 321 |
+
) -> List[EntityRecognizer]:
|
| 322 |
+
"""Return pure, context-bound Zorgfilter v1 recognizers.
|
| 323 |
+
|
| 324 |
+
The returned recognizers are not automatically registered in the current
|
| 325 |
+
application. Profile composition is handled in a later workpackage.
|
| 326 |
+
"""
|
| 327 |
+
|
| 328 |
+
recognizers: List[EntityRecognizer] = [
|
| 329 |
+
_reference_recognizer(
|
| 330 |
+
"NL_PATIENT_NUMBER",
|
| 331 |
+
r"pati[eë]ntnummer|patientnummer|pati[eë]ntnr\.?|patientnr\. ?",
|
| 332 |
+
0.91,
|
| 333 |
+
supported_language,
|
| 334 |
+
),
|
| 335 |
+
_reference_recognizer(
|
| 336 |
+
"NL_CARE_CLIENT_NUMBER",
|
| 337 |
+
r"cli[eë]ntnummer|clientnummer|cli[eë]ntnr\.?|clientnr\.?",
|
| 338 |
+
0.90,
|
| 339 |
+
supported_language,
|
| 340 |
+
),
|
| 341 |
+
_reference_recognizer(
|
| 342 |
+
"NL_MEDICAL_RECORD_NUMBER",
|
| 343 |
+
r"medisch\s+dossiernummer|zorgdossiernummer|medisch\s+dossiernr\. ?",
|
| 344 |
+
0.90,
|
| 345 |
+
supported_language,
|
| 346 |
+
),
|
| 347 |
+
_reference_recognizer(
|
| 348 |
+
"NL_EPD_ECD_NUMBER",
|
| 349 |
+
r"EPD[-\s]?nummer|ECD[-\s]?nummer|EPD\s*nr\.?|ECD\s*nr\. ?",
|
| 350 |
+
0.92,
|
| 351 |
+
supported_language,
|
| 352 |
+
),
|
| 353 |
+
_reference_recognizer(
|
| 354 |
+
"NL_HEALTH_INSURANCE_NUMBER",
|
| 355 |
+
r"verzekerdennummer|zorgverzekeringsnummer|verzekerdennr\. ?",
|
| 356 |
+
0.90,
|
| 357 |
+
supported_language,
|
| 358 |
+
),
|
| 359 |
+
_reference_recognizer(
|
| 360 |
+
"NL_REFERRAL_NUMBER",
|
| 361 |
+
r"verwijsnummer|verwijzingsnummer|verwijsnr\. ?",
|
| 362 |
+
0.90,
|
| 363 |
+
supported_language,
|
| 364 |
+
),
|
| 365 |
+
_reference_recognizer(
|
| 366 |
+
"NL_TREATMENT_REFERENCE",
|
| 367 |
+
r"behandelnummer|zorgtrajectnummer|receptnummer|behandelnr\.?|trajectnummer",
|
| 368 |
+
0.88,
|
| 369 |
+
supported_language,
|
| 370 |
+
),
|
| 371 |
+
_reference_recognizer(
|
| 372 |
+
"NL_LAB_SAMPLE_NUMBER",
|
| 373 |
+
r"monsternummer|sample[-\s]?ID|labnummer|laboratoriumnummer|"
|
| 374 |
+
r"accessienummer|onderzoeks[-\s]?ID",
|
| 375 |
+
0.90,
|
| 376 |
+
supported_language,
|
| 377 |
+
),
|
| 378 |
+
_reference_recognizer(
|
| 379 |
+
"NL_CARE_INCIDENT_NUMBER",
|
| 380 |
+
r"MIC[-\s]?nummer|MIM[-\s]?nummer|VIM[-\s]?nummer|"
|
| 381 |
+
r"zorgincidentnummer|incidentnummer|medicatiefoutnummer",
|
| 382 |
+
0.91,
|
| 383 |
+
supported_language,
|
| 384 |
+
),
|
| 385 |
+
_reference_recognizer(
|
| 386 |
+
"NL_CARE_INDICATION_REFERENCE",
|
| 387 |
+
r"CIZ[-\s]?nummer|Wlz[-\s]?indicatie|Wmo[-\s]?beschikking|"
|
| 388 |
+
r"zorgtoewijzing|PGB[-\s]?nummer|indicatiebesluit",
|
| 389 |
+
0.90,
|
| 390 |
+
supported_language,
|
| 391 |
+
),
|
| 392 |
+
CareRegexCaptureRecognizer(
|
| 393 |
+
entity="NL_AGB_CODE",
|
| 394 |
+
patterns=[
|
| 395 |
+
(
|
| 396 |
+
"strong_agb_context_eight_digits",
|
| 397 |
+
r"\b(?:AGB[-\s]?(?:code|nummer|nr\.?)|AGB)\s*(?:is|:|#|=|-)?\s*"
|
| 398 |
+
r"(?P<value>\d{8})\b",
|
| 399 |
+
)
|
| 400 |
+
],
|
| 401 |
+
score=0.94,
|
| 402 |
+
context=("AGB-code", "AGB-nummer", "AGB"),
|
| 403 |
+
supported_language=supported_language,
|
| 404 |
+
),
|
| 405 |
+
_provider_recognizer(supported_language),
|
| 406 |
+
_organization_recognizer(supported_language),
|
| 407 |
+
_location_recognizer(supported_language),
|
| 408 |
+
_room_bed_recognizer(supported_language),
|
| 409 |
+
_event_date_recognizer(supported_language),
|
| 410 |
+
]
|
| 411 |
+
return recognizers
|
handover/workpackages/20260803_1652_care_profile_recognizer_implementation.md
ADDED
|
@@ -0,0 +1,61 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Handover — SCRUB-WP_CARE_PROFILE_RECOGNIZER_IMPLEMENTATION
|
| 2 |
+
|
| 3 |
+
Repository: solidprivacy-nl/scrub
|
| 4 |
+
Workpackage title: Zorgfilter v1 recognizer implementation
|
| 5 |
+
Status: completed; final clean PR validation initiated
|
| 6 |
+
|
| 7 |
+
## Summary
|
| 8 |
+
|
| 9 |
+
Implemented the pure `dutch_care_recognizers.py` module against the frozen contract. Sixteen dedicated care entities are recognized through bounded labels and professional context. The module is not registered in the app yet. All 37 positive contracts, 16 negative/collision contracts and 54 dedicated expectations in the eight-document synthetic corpus pass, with zero protected clinical phrase overlaps.
|
| 10 |
+
|
| 11 |
+
## Files added
|
| 12 |
+
|
| 13 |
+
- `dutch_care_recognizers.py`
|
| 14 |
+
- `care_recognizer_validation.py`
|
| 15 |
+
- `CARE_RECOGNIZER_IMPLEMENTATION_V1.md`
|
| 16 |
+
- `scripts/generate_care_recognizer_validation.py`
|
| 17 |
+
- `tests/test_dutch_care_recognizers.py`
|
| 18 |
+
- `tests/test_care_recognizer_validation.py`
|
| 19 |
+
- `output/validation/care_recognizer_implementation_validation.json`
|
| 20 |
+
- `workpackage_claims/scrub_wp_care_profile_recognizer_implementation.md`
|
| 21 |
+
- `handover/workpackages/20260803_1652_care_profile_recognizer_implementation.md`
|
| 22 |
+
|
| 23 |
+
## Files changed
|
| 24 |
+
|
| 25 |
+
- `ROADMAP.md`
|
| 26 |
+
- `WORKPACKAGES.md`
|
| 27 |
+
- `CHANGELOG.md`
|
| 28 |
+
- `RISK_REGISTER.md`
|
| 29 |
+
|
| 30 |
+
## Tests
|
| 31 |
+
|
| 32 |
+
- Frozen public API and sixteen-entity scope.
|
| 33 |
+
- All 37 positive exact-span contracts.
|
| 34 |
+
- All 16 negative, collision and clinical-preservation contracts.
|
| 35 |
+
- Exact Presidio result spans, explanations and metadata.
|
| 36 |
+
- Module-purity import guard.
|
| 37 |
+
- All 54 dedicated care expectations in the eight-document synthetic corpus.
|
| 38 |
+
- Zero overlap with protected clinical passages.
|
| 39 |
+
- Reproducibility of the committed validation artifact.
|
| 40 |
+
|
| 41 |
+
## Validation
|
| 42 |
+
|
| 43 |
+
- Initial run #1850 identified five bounded missing context variants; no clinical-overmasking failure.
|
| 44 |
+
- Corrected variants: Dutch `cliëntnummer`, `receptnummer`, generic `incidentnummer`, and lowercase `woonzorgcentrum` after a labeled location field.
|
| 45 |
+
- Corrected GitHub Actions run #1854: 953 tests passed.
|
| 46 |
+
- All temporary operator and workflow files are removed.
|
| 47 |
+
- Final clean GitHub Actions: initiated on this handover commit.
|
| 48 |
+
- Hugging Face sync: not functionally relevant; no current runtime/UI registration changed.
|
| 49 |
+
- App verification: not applicable.
|
| 50 |
+
|
| 51 |
+
## Remaining risks
|
| 52 |
+
|
| 53 |
+
- The recognizers are not yet composed into a Zorg profile or registered in the app.
|
| 54 |
+
- Generic PERSON and e-mail remain generic-profile dependencies.
|
| 55 |
+
- Existing BSN and new AGB overlap requires deterministic profile-level precedence validation.
|
| 56 |
+
- Synthetic contract success does not establish production precision or recall.
|
| 57 |
+
- Human review remains mandatory and production readiness remains false.
|
| 58 |
+
|
| 59 |
+
## Next recommended step
|
| 60 |
+
|
| 61 |
+
Start `SCRUB-WP_RECOGNITION_PROFILE_CONFIGURATION_REFACTOR` to centralize the four profile definitions and collision/policy composition before changing the visible Streamlit selector.
|
output/validation/care_recognizer_implementation_validation.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": "1.0",
|
| 3 |
+
"module": "dutch_care_recognizers",
|
| 4 |
+
"synthetic_data_only": true,
|
| 5 |
+
"production_ready": false,
|
| 6 |
+
"human_review_required": true,
|
| 7 |
+
"entity_count": 16,
|
| 8 |
+
"positive_contract_count": 37,
|
| 9 |
+
"positive_contract_passed": 37,
|
| 10 |
+
"positive_contract_failures": [],
|
| 11 |
+
"forbidden_positive_failure_count": 0,
|
| 12 |
+
"forbidden_positive_failures": [],
|
| 13 |
+
"negative_contract_count": 16,
|
| 14 |
+
"negative_contract_passed": 16,
|
| 15 |
+
"negative_contract_failures": [],
|
| 16 |
+
"corpus_dedicated_expectation_count": 54,
|
| 17 |
+
"corpus_dedicated_found_count": 54,
|
| 18 |
+
"corpus_dedicated_failures": [],
|
| 19 |
+
"clinical_preserve_overlap_count": 0,
|
| 20 |
+
"clinical_preserve_overlaps": [],
|
| 21 |
+
"app_registered": false,
|
| 22 |
+
"next_workpackage": "SCRUB-WP_RECOGNITION_PROFILE_CONFIGURATION_REFACTOR"
|
| 23 |
+
}
|
scripts/generate_care_recognizer_validation.py
ADDED
|
@@ -0,0 +1,32 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Generate the machine-readable Zorgfilter recognizer validation report."""
|
| 3 |
+
|
| 4 |
+
from __future__ import annotations
|
| 5 |
+
|
| 6 |
+
import argparse
|
| 7 |
+
import json
|
| 8 |
+
from pathlib import Path
|
| 9 |
+
|
| 10 |
+
from care_recognizer_validation import build_care_recognizer_validation
|
| 11 |
+
|
| 12 |
+
|
| 13 |
+
def main() -> int:
|
| 14 |
+
parser = argparse.ArgumentParser()
|
| 15 |
+
parser.add_argument(
|
| 16 |
+
"--output",
|
| 17 |
+
default="output/validation/care_recognizer_implementation_validation.json",
|
| 18 |
+
)
|
| 19 |
+
args = parser.parse_args()
|
| 20 |
+
|
| 21 |
+
output_path = Path(args.output)
|
| 22 |
+
output_path.parent.mkdir(parents=True, exist_ok=True)
|
| 23 |
+
output_path.write_text(
|
| 24 |
+
json.dumps(build_care_recognizer_validation(), ensure_ascii=False, indent=2) + "\n",
|
| 25 |
+
encoding="utf-8",
|
| 26 |
+
)
|
| 27 |
+
print(output_path)
|
| 28 |
+
return 0
|
| 29 |
+
|
| 30 |
+
|
| 31 |
+
if __name__ == "__main__":
|
| 32 |
+
raise SystemExit(main())
|
tests/test_care_recognizer_validation.py
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import json
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
|
| 4 |
+
from care_recognizer_validation import build_care_recognizer_validation
|
| 5 |
+
|
| 6 |
+
|
| 7 |
+
VALIDATION_PATH = Path("output/validation/care_recognizer_implementation_validation.json")
|
| 8 |
+
|
| 9 |
+
|
| 10 |
+
def test_care_recognizer_implementation_passes_frozen_contracts():
|
| 11 |
+
report = build_care_recognizer_validation()
|
| 12 |
+
|
| 13 |
+
assert report["schema_version"] == "1.0"
|
| 14 |
+
assert report["entity_count"] == 16
|
| 15 |
+
assert report["positive_contract_count"] == 37
|
| 16 |
+
assert report["positive_contract_passed"] == 37
|
| 17 |
+
assert report["positive_contract_failures"] == []
|
| 18 |
+
assert report["forbidden_positive_failure_count"] == 0
|
| 19 |
+
assert report["forbidden_positive_failures"] == []
|
| 20 |
+
assert report["negative_contract_count"] == 16
|
| 21 |
+
assert report["negative_contract_passed"] == 16
|
| 22 |
+
assert report["negative_contract_failures"] == []
|
| 23 |
+
|
| 24 |
+
|
| 25 |
+
def test_care_recognizers_cover_all_dedicated_corpus_expectations():
|
| 26 |
+
report = build_care_recognizer_validation()
|
| 27 |
+
|
| 28 |
+
assert report["corpus_dedicated_expectation_count"] == 54
|
| 29 |
+
assert report["corpus_dedicated_found_count"] == 54
|
| 30 |
+
assert report["corpus_dedicated_failures"] == []
|
| 31 |
+
assert report["clinical_preserve_overlap_count"] == 0
|
| 32 |
+
assert report["clinical_preserve_overlaps"] == []
|
| 33 |
+
|
| 34 |
+
|
| 35 |
+
def test_care_recognizer_validation_keeps_integration_gates_closed():
|
| 36 |
+
report = build_care_recognizer_validation()
|
| 37 |
+
|
| 38 |
+
assert report["synthetic_data_only"] is True
|
| 39 |
+
assert report["production_ready"] is False
|
| 40 |
+
assert report["human_review_required"] is True
|
| 41 |
+
assert report["app_registered"] is False
|
| 42 |
+
assert report["next_workpackage"] == "SCRUB-WP_RECOGNITION_PROFILE_CONFIGURATION_REFACTOR"
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def test_committed_care_recognizer_validation_is_reproducible():
|
| 46 |
+
committed = json.loads(VALIDATION_PATH.read_text(encoding="utf-8"))
|
| 47 |
+
assert committed == build_care_recognizer_validation()
|
tests/test_dutch_care_recognizers.py
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
import ast
|
| 2 |
+
import inspect
|
| 3 |
+
from pathlib import Path
|
| 4 |
+
|
| 5 |
+
from care_recognizer_contracts import (
|
| 6 |
+
ALL_POSITIVE_CASES,
|
| 7 |
+
CARE_RECOGNIZER_ENTITY_NAMES,
|
| 8 |
+
NEGATIVE_CASES,
|
| 9 |
+
)
|
| 10 |
+
from care_test_examples import TEST_CASES
|
| 11 |
+
from dutch_care_recognizers import (
|
| 12 |
+
CareRegexCaptureRecognizer,
|
| 13 |
+
get_dutch_care_entity_names,
|
| 14 |
+
get_dutch_care_recognizers,
|
| 15 |
+
)
|
| 16 |
+
|
| 17 |
+
|
| 18 |
+
def _detect(text):
|
| 19 |
+
entities = get_dutch_care_entity_names()
|
| 20 |
+
rows = []
|
| 21 |
+
for recognizer in get_dutch_care_recognizers(supported_language="en"):
|
| 22 |
+
for result in recognizer.analyze(text, entities=entities, nlp_artifacts=None):
|
| 23 |
+
rows.append(
|
| 24 |
+
{
|
| 25 |
+
"entity_type": result.entity_type,
|
| 26 |
+
"start": result.start,
|
| 27 |
+
"end": result.end,
|
| 28 |
+
"text": text[result.start:result.end],
|
| 29 |
+
"score": result.score,
|
| 30 |
+
"recognizer": recognizer.name,
|
| 31 |
+
}
|
| 32 |
+
)
|
| 33 |
+
return sorted(rows, key=lambda row: (row["start"], row["end"], row["entity_type"]))
|
| 34 |
+
|
| 35 |
+
|
| 36 |
+
def _exact(rows, entity_type, value):
|
| 37 |
+
return [
|
| 38 |
+
row for row in rows
|
| 39 |
+
if row["entity_type"] == entity_type and row["text"] == value
|
| 40 |
+
]
|
| 41 |
+
|
| 42 |
+
|
| 43 |
+
def test_public_api_matches_frozen_contract():
|
| 44 |
+
assert get_dutch_care_entity_names() == list(CARE_RECOGNIZER_ENTITY_NAMES)
|
| 45 |
+
recognizers = get_dutch_care_recognizers()
|
| 46 |
+
assert len(recognizers) == len(CARE_RECOGNIZER_ENTITY_NAMES)
|
| 47 |
+
assert {entity for recognizer in recognizers for entity in recognizer.supported_entities} == set(
|
| 48 |
+
CARE_RECOGNIZER_ENTITY_NAMES
|
| 49 |
+
)
|
| 50 |
+
assert all(recognizer.supported_language == "en" for recognizer in recognizers)
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def test_all_positive_contracts_return_the_exact_value_span():
|
| 54 |
+
for case in ALL_POSITIVE_CASES:
|
| 55 |
+
rows = _detect(case["text"])
|
| 56 |
+
matches = _exact(rows, case["entity_type"], case["expected_value"])
|
| 57 |
+
assert len(matches) == 1, (case["id"], rows)
|
| 58 |
+
|
| 59 |
+
expected_start = case["text"].index(case["expected_value"])
|
| 60 |
+
assert matches[0]["start"] == expected_start
|
| 61 |
+
assert matches[0]["end"] == expected_start + len(case["expected_value"])
|
| 62 |
+
|
| 63 |
+
for forbidden_entity in case["forbidden_entities"]:
|
| 64 |
+
assert not _exact(rows, forbidden_entity, case["expected_value"]), (
|
| 65 |
+
case["id"],
|
| 66 |
+
forbidden_entity,
|
| 67 |
+
rows,
|
| 68 |
+
)
|
| 69 |
+
|
| 70 |
+
|
| 71 |
+
def test_all_negative_contracts_avoid_forbidden_care_entities():
|
| 72 |
+
for case in NEGATIVE_CASES:
|
| 73 |
+
rows = _detect(case["text"])
|
| 74 |
+
forbidden = set(case["forbidden_entities"])
|
| 75 |
+
violations = [row for row in rows if row["entity_type"] in forbidden]
|
| 76 |
+
assert violations == [], (case["id"], violations)
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def test_capture_results_have_explanations_and_stable_metadata():
|
| 80 |
+
text = "Patiëntnummer: PAT-2026-1148."
|
| 81 |
+
recognizer = next(
|
| 82 |
+
recognizer for recognizer in get_dutch_care_recognizers()
|
| 83 |
+
if recognizer.supported_entities == ["NL_PATIENT_NUMBER"]
|
| 84 |
+
)
|
| 85 |
+
results = recognizer.analyze(
|
| 86 |
+
text,
|
| 87 |
+
entities=get_dutch_care_entity_names(),
|
| 88 |
+
nlp_artifacts=None,
|
| 89 |
+
)
|
| 90 |
+
|
| 91 |
+
assert len(results) == 1
|
| 92 |
+
result = results[0]
|
| 93 |
+
assert result.entity_type == "NL_PATIENT_NUMBER"
|
| 94 |
+
assert text[result.start:result.end] == "PAT-2026-1148"
|
| 95 |
+
assert result.analysis_explanation is not None
|
| 96 |
+
assert result.recognition_metadata
|
| 97 |
+
|
| 98 |
+
|
| 99 |
+
def test_care_module_has_no_streamlit_network_cloud_or_file_write_imports():
|
| 100 |
+
source_path = Path(inspect.getsourcefile(CareRegexCaptureRecognizer))
|
| 101 |
+
tree = ast.parse(source_path.read_text(encoding="utf-8"))
|
| 102 |
+
imported_roots = set()
|
| 103 |
+
for node in ast.walk(tree):
|
| 104 |
+
if isinstance(node, ast.Import):
|
| 105 |
+
imported_roots.update(alias.name.split(".")[0] for alias in node.names)
|
| 106 |
+
elif isinstance(node, ast.ImportFrom) and node.module:
|
| 107 |
+
imported_roots.add(node.module.split(".")[0])
|
| 108 |
+
|
| 109 |
+
assert imported_roots.isdisjoint(
|
| 110 |
+
{
|
| 111 |
+
"streamlit",
|
| 112 |
+
"requests",
|
| 113 |
+
"httpx",
|
| 114 |
+
"urllib",
|
| 115 |
+
"openai",
|
| 116 |
+
"azure",
|
| 117 |
+
"boto3",
|
| 118 |
+
"pathlib",
|
| 119 |
+
"os",
|
| 120 |
+
"subprocess",
|
| 121 |
+
}
|
| 122 |
+
)
|
| 123 |
+
|
| 124 |
+
|
| 125 |
+
def test_dedicated_care_expectations_in_full_corpus_are_detected_exactly():
|
| 126 |
+
dedicated_entities = set(get_dutch_care_entity_names())
|
| 127 |
+
checked = 0
|
| 128 |
+
missing = []
|
| 129 |
+
|
| 130 |
+
for case in TEST_CASES:
|
| 131 |
+
rows = _detect(case["text"])
|
| 132 |
+
for bucket in ("replace", "review_selected"):
|
| 133 |
+
for expectation in case[bucket]:
|
| 134 |
+
if expectation["entity_type"] not in dedicated_entities:
|
| 135 |
+
continue
|
| 136 |
+
checked += 1
|
| 137 |
+
if not _exact(rows, expectation["entity_type"], expectation["value"]):
|
| 138 |
+
missing.append(
|
| 139 |
+
{
|
| 140 |
+
"case_id": case["id"],
|
| 141 |
+
"entity_type": expectation["entity_type"],
|
| 142 |
+
"value": expectation["value"],
|
| 143 |
+
"detected": rows,
|
| 144 |
+
}
|
| 145 |
+
)
|
| 146 |
+
|
| 147 |
+
assert checked > 0
|
| 148 |
+
assert missing == []
|
| 149 |
+
|
| 150 |
+
|
| 151 |
+
def test_full_corpus_clinical_preserve_phrases_do_not_overlap_care_results():
|
| 152 |
+
overlaps = []
|
| 153 |
+
for case in TEST_CASES:
|
| 154 |
+
rows = _detect(case["text"])
|
| 155 |
+
for phrase in case["preserve"]:
|
| 156 |
+
phrase_start = case["text"].index(phrase)
|
| 157 |
+
phrase_end = phrase_start + len(phrase)
|
| 158 |
+
for row in rows:
|
| 159 |
+
if phrase_start < row["end"] and row["start"] < phrase_end:
|
| 160 |
+
overlaps.append(
|
| 161 |
+
{
|
| 162 |
+
"case_id": case["id"],
|
| 163 |
+
"phrase": phrase,
|
| 164 |
+
"row": row,
|
| 165 |
+
}
|
| 166 |
+
)
|
| 167 |
+
|
| 168 |
+
assert overlaps == []
|
workpackage_claims/scrub_wp_care_profile_recognizer_implementation.md
ADDED
|
@@ -0,0 +1,22 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Workpackage claim — SCRUB-WP_CARE_PROFILE_RECOGNIZER_IMPLEMENTATION
|
| 2 |
+
|
| 3 |
+
Repository: `solidprivacy-nl/scrub`
|
| 4 |
+
Claimed at: 2026-08-03 16:42 Europe/Amsterdam
|
| 5 |
+
Status: completed
|
| 6 |
+
|
| 7 |
+
## Scope
|
| 8 |
+
|
| 9 |
+
- Implement the frozen pure `dutch_care_recognizers.py` module.
|
| 10 |
+
- Implement exactly the sixteen dedicated care entities and two public helper functions.
|
| 11 |
+
- Pass the 37 positive exact-span and 16 negative/collision/clinical-preservation contracts.
|
| 12 |
+
- Add implementation validation and a machine-readable result artifact.
|
| 13 |
+
- Do not register the recognizers in the current app yet.
|
| 14 |
+
|
| 15 |
+
## Boundaries
|
| 16 |
+
|
| 17 |
+
- Synthetic data only.
|
| 18 |
+
- No Streamlit, network, AI, cloud or file-write behavior in the recognizer module.
|
| 19 |
+
- No current profile selector, threshold or entity-default change.
|
| 20 |
+
- No export, Scrub Key or reinsert semantic change.
|
| 21 |
+
- Generic PERSON and e-mail remain generic-profile responsibilities.
|
| 22 |
+
- Human review remains required and production readiness remains false.
|