|
Download experiment-methods.md from EasyOctopus/german-support-redactor: direct link, hf CLI and curl.
- Browser
- Download file 4.54 kB
-
https://huggingface.co/spaces/EasyOctopus/german-support-redactor/resolve/main/experiment-methods.md
- Command line
-
hf download hf://spaces/EasyOctopus/german-support-redactor/experiment-methods.md
-
curl -L -o experiment-methods.md https://huggingface.co/spaces/EasyOctopus/german-support-redactor/resolve/main/experiment-methods.md
4.54 kB
| # Training a German redaction policy | |
| ## What the experiment changes | |
| The released German DistilBERT model proposes private spans. A small second model decides whether each proposal should stay masked in its surrounding German sentence. The policy sees context outside the candidate and its predicted type. The browser still applies the user’s confidence cutoff. It cannot discover an identifier the first model missed. That limitation makes full-span recall and previously covered individual spans important release checks. | |
| The supervised and reinforcement learning policies receive the same examples and start with supervised training. The supervised comparator continues to learn from annotated actions. The RL branch samples keep/mask decisions and updates the policy using the measured reward of each outcome relative to the other sampled outcomes. A contextual bandit fits this one-step decision. A long-running agent harness would change the task. | |
| ## Methods and sources | |
| | Source | What we used | | |
| |---|---| | |
| | [Ahmadian et al., ACL 2024](https://aclanthology.org/2024.acl-long.662/) and [Hugging Face TRL implementation](https://github.com/huggingface/trl/blob/main/trl/trainer/rloo_trainer.py) | Sample several decisions for a document. Estimate a policy gradient with a leave-one-out reward baseline. Our binary span policy has its own implementation. Results from preference training of language models do not establish gains on redaction. | | |
| | [Minimum Risk Training, Shen et al.](https://aclanthology.org/P16-1159/) | An output-level task measure can differ from token accuracy. Count whether a whole sensitive span remains protected and charge for masking ordinary text. | | |
| | [Text Anonymization Benchmark paper](https://aclanthology.org/2022.cl-4.19/) and [authors' evaluator](https://github.com/NorskRegnesentral/text-anonymisation-benchmark) | Score full coverage of a sensitive mention. Track loss of information from unnecessary masking separately. Our German task uses a narrower definition than full document anonymisation. | | |
| | [RedactionBench](https://arxiv.org/abs/2606.18782) | Hold the name or reference constant and change its role. A product title and a customer's name can have the same spelling. | | |
| | [CheckList](https://aclanthology.org/2020.acl-main.442/) and [companion code](https://github.com/marcotcr/checklist) | Evaluate clean controls, paired directional changes and mixed cases. Include ordinary German, line breaks, emoji and combining characters. | | |
| | [Counterfactually augmented data](https://github.com/acmi-lab/counterfactually-augmented-data) | Use closely related examples with different intended decisions. Reserve unseen scenario families and phrases for evaluation. | | |
| | [Presidio recognizer architecture](https://microsoft.github.io/presidio/analyzer/) and [source repository](https://github.com/data-privacy-stack/presidio) | Evaluate a separate decoder intervention that completes the boundaries of an already detected, syntactically valid identifier. Only a matching detector proposal can trigger it. | | |
| | [OpenAI Privacy Filter model card](https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf), [reference implementation](https://github.com/openai/privacy-filter) and [Digitflow German fine-tune](https://huggingface.co/digitflow/privacy-filter-de-ft) | Reference alternatives for bidirectional classification and multilingual redaction. They use different workloads and weights. This experiment measures our changes against the shipping browser detector and a matched supervised control. | | |
| | [Gretel synthetic financial documents](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual) | Use the entire German test split as an external transfer check. Count annotated names, addresses, linked customer IDs, contact details, birth dates, bank identifiers and IP addresses. The dataset can omit other private text. An unmatched prediction alone is insufficient evidence of a false alarm. | | |
| ## Evidence boundaries | |
| Contabo owns training. The Mac holds the sealed 120-message support set and the separately authored German financial test split. The blind support set includes 36 pairs, mixed messages, ordinary negatives, structured identifiers and Unicode edge cases. The frozen release rule checks complete mentions and false alarms on the same paired predictions. In the external check we use all 453 documents and count only annotations in the mapped categories. Neither synthetic test establishes customer performance. | |