Expand model card: usage, evaluation, training data and procedure, limitations

README.md

@@ -1,12 +1,318 @@
 ---
-language: uk
+language:
+- uk
+license: mit
 library_name: transformers
 pipeline_tag: token-classification
-license: mit
+base_model: benjamin/roberta-base-wechsel-ukrainian
+datasets:
+- lang-uk/court-decisions-uk
+metrics:
+- f1
+- precision
+- recall
+tags:
+- token-classification
+- ner
+- named-entity-recognition
+- pii
+- pii-detection
+- personal-data
+- anonymization
+- pseudonymization
+- de-identification
+- data-privacy
+- privacy
+- redaction
+- legal
+- legal-nlp
+- court-decisions
+- ukrainian
+- ukraine
+- synthetic-data
+- roberta
+new_version: davidmelash/ukrainian-pii-ner-roberta-base
+widget:
+- text: >-
+    Позивач Мельничук Андрій Васильович, 12.03.1985 року народження, зареєстрований за адресою: м. Житомир, вул. Лесі Українки, буд. 14, кв. 7, РНОКПП 3124509876, проходив службу у військовій частині А4123.
+- text: >-
+    Представник позивача - адвокат Кравчук Олена Миколаївна, електронна пошта: kravchuk.law@ukr.net. Транспортний засіб Toyota Camry, номерний знак АА1234ВС, VIN JTNB11HK103456789, належить відповідачу Шевчуку Т. П.
+model-index:
+- name: wechsel_base
+  results:
+  - task:
+      type: token-classification
+      name: Personal data detection (NER)
+    dataset:
+      type: ukrainian-court-decisions-pseudonymization-test
+      name: Ukrainian court decisions pseudonymization test set (240 decisions)
+      split: test
+    metrics:
+    - type: f1
+      value: 0.8699
+      name: Macro F1 (relaxed, IoU >= 0.5)
+    - type: precision
+      value: 0.9511
+      name: Macro precision (relaxed)
+    - type: recall
+      value: 0.8084
+      name: Macro recall (relaxed)
+    - type: f1
+      value: 0.8281
+      name: Macro F1 (strict, exact boundaries)
+    - type: f1
+      value: 0.9961
+      name: F1 ОСОБА (relaxed)
+    - type: f1
+      value: 0.7907
+      name: F1 АДРЕСА (relaxed)
+    - type: f1
+      value: 0.9269
+      name: F1 НОМЕР (relaxed)
+    - type: f1
+      value: 0.7660
+      name: F1 ІНФОРМАЦІЯ (relaxed)
 ---
 
-# wechsel_base
+# Ukrainian PII NER for court decisions — RoBERTa-base (WECHSEL), data v2, seed 42
+
+**wechsel_base** is a Ukrainian **named-entity recognition (NER)** model that finds **personal data (PII)** in court decisions
+and other legal texts, so they can be **anonymized or pseudonymized** before publication. It tags people's names,
+addresses, personal ID numbers and other identifying details (dates of birth, e-mails, military units) with the four
+categories that Ukrainian courts use when they de-identify decisions. It is a 124M-parameter
+RoBERTa-base fine-tuned from [benjamin/roberta-base-wechsel-ukrainian](https://huggingface.co/benjamin/roberta-base-wechsel-ukrainian) on synthetic court decisions, and it runs
+offline on a CPU.
+
+> **Legacy release.** This is the first model of the family, trained on an early version of the data. The v7 models score about 0.09 higher macro F1, mostly on addresses, numbers and ІНФОРМАЦІЯ: use [ukrainian-pii-ner-roberta-base](https://huggingface.co/davidmelash/ukrainian-pii-ner-roberta-base) instead (same size and speed).
+
+- **Macro F1 0.870** on a 240-document court-decision test set (precision 0.951, recall
+  0.808): ОСОБА 0.996, АДРЕСА 0.791, НОМЕР 0.927, ІНФОРМАЦІЯ 0.766.
+- Only **15 of 864 people** in the test decisions have any mention of their name left unmasked.
+- Trained without any real personal data: 24,097 published decisions whose placeholders were refilled with synthetic values.
+- CPU-friendly, no GPU needed: the 240 test decisions (1.4M characters) take about 100 s with PyTorch on a laptop CPU.
+- MIT licence; code, data generator and evaluation on [GitHub](https://github.com/EnderLancer/anonimimizer).
+
+## Entity types
+
+| Label | Meaning | Examples |
+|---|---|---|
+| `ОСОБА` (person) | Name of a natural person in any form and grammatical case | «Мельничук Андрій Васильович», «Мельничука А. В.», «А. Мельничук» |
+| `АДРЕСА` (address) | Address of residence, stay or property | «м. Житомир, вул. Лесі Українки, буд. 14, кв. 7» |
+| `НОМЕР` (number) | Identifier of a particular person: taxpayer number (РНОКПП / ІПН), passport or ID card, phone, IBAN, bank card, licence plate, VIN, military unit number, enforcement-proceeding number | «3124509876», «АА1234ВС», «А4123» |
+| `ІНФОРМАЦІЯ` (information) | Other identifying information: date of birth or death, e-mail, URL, social-media nickname, name of a military unit or military body that is a party to the case | «12.03.1985 року народження», «kravchuk.law@ukr.net», «Житомирський обласний ТЦК та СП» as the defendant |
+
+Labels use the BIO scheme (`B-ОСОБА`, `I-ОСОБА`, …, `O`). Following the conventions of the court register, case numbers,
+cadastral numbers, contract numbers, accounts of state institutions and names of courts are *not* personal data, while
+every natural person — including judges, lawyers and court secretaries — is labelled `ОСОБА`.
+
+## How to use
+
+```python
+from transformers import pipeline
+
+ner = pipeline("token-classification", model="davidmelash/wechsel_base", aggregation_strategy="simple")
+
+text = (
+    "Позивач Мельничук Андрій Васильович, 12.03.1985 року народження, зареєстрований за адресою: "
+    "м. Житомир, вул. Лесі Українки, буд. 14, кв. 7, РНОКПП 3124509876, проходив службу у військовій частині А4123."
+)
+for e in ner(text):
+    print(e["entity_group"], text[e["start"]:e["end"]])
+# ОСОБА Мельничук Андрій Васильович
+# ІНФОРМАЦІЯ 12.03.1985 року народження
+# АДРЕСА м. Житомир, вул. Лесі Українки, буд. 14, кв. 7
+# НОМЕР 3124509876
+# НОМЕР А4123
+```
+
+### Long documents
+
+The model reads 512 tokens (about 1,500–2,000 characters of Ukrainian) at a time, and a court decision is usually
+longer. Split the text into overlapping windows and merge the results (the pipeline's own `stride` option did not
+cover long texts in transformers 5.19):
+
+```python
+def find_entities(text: str, window: int = 500, stride: int = 100) -> list[dict]:
+    """Run the pipeline on overlapping windows of the text and merge the results."""
+    offsets = ner.tokenizer(text, add_special_tokens=False, return_offsets_mapping=True)["offset_mapping"]
+    found, start = [], 0
+    while start < len(offsets):
+        end = min(start + window, len(offsets))
+        a, b = offsets[start][0], offsets[end - 1][1]
+        found += [dict(e, start=e["start"] + a, end=e["end"] + a) for e in ner(text[a:b])]
+        if end == len(offsets):
+            break
+        start = end - stride
+    entities = []
+    for e in sorted(found, key=lambda e: (e["start"], e["start"] - e["end"])):
+        if not entities or e["start"] >= entities[-1]["end"]:  # an entity seen by two windows is kept once
+            entities.append(e)
+    return entities
+```
+
+### Pseudonymize a document
+
+Deterministic pseudonymization replaces each entity with a code and gives the same code to the same value, as Ukrainian
+courts do:
+
+```python
+def pseudonymize(text: str) -> str:
+    codes, counts, out, last = {}, {}, [], 0
+    for e in find_entities(text):
+        label, value = e["entity_group"], text[e["start"]:e["end"]]
+        if (label, value) not in codes:
+            counts[label] = counts.get(label, 0) + 1
+            codes[(label, value)] = f"{label}_{counts[label]}"
+        out += [text[last:e["start"]], codes[(label, value)]]
+        last = e["end"]
+    return "".join(out) + text[last:]
+
+print(pseudonymize(text))
+# Позивач ОСОБА_1, ІНФОРМАЦІЯ_1, зареєстрований за адресою: АДРЕСА_1, РНОКПП НОМЕР_1, проходив службу у військовій частині НОМЕР_2.
+```
+
+The [GitHub repository](https://github.com/EnderLancer/anonimimizer) adds what a production pipeline needs on top of the model: linking of different
+forms of one name («Мельничук Андрій Васильович», «Мельничук А. В.», «позивача Мельничука») to one pseudonym, optional
+regex rules, a Gradio demo and an offline Docker image with ONNX Runtime.
+
+## Model family
+
+All models were trained with the same code and evaluated on the same test set with the same code (model only). Two
+seeds of one recipe differ by up to 0.010 macro F1, so compare recipes by their seed average. *Names leaked*
+counts the people (of 864) with at least one mention of their name left unmasked.
+
+| Model | Architecture | Training data | Seed | Macro F1 | Strict F1 | ОСОБА | АДРЕСА | НОМЕР | ІНФОРМАЦІЯ | Names leaked |
+|---|---|---|---|---|---|---|---|---|---|---|
+| [ukrainian-pii-ner-roberta-base](https://huggingface.co/davidmelash/ukrainian-pii-ner-roberta-base) | RoBERTa-base | data v7-large | 42 | 0.958 | 0.911 | 0.996 | 0.972 | 0.980 | 0.885 | 15 |
+| [wechsel_base_v7large_2026-10-09_r4_s43](https://huggingface.co/davidmelash/wechsel_base_v7large_2026-10-09_r4_s43) | RoBERTa-base | data v7-large | 43 | 0.957 | 0.905 | 0.996 | 0.972 | 0.984 | 0.878 | 14 |
+| [ukrainian-pii-ner-modernbert-large](https://huggingface.co/davidmelash/ukrainian-pii-ner-modernbert-large) | ModernBERT-large | data v7 | 43 | 0.957 | 0.921 | 0.996 | 0.967 | 0.982 | 0.882 | 15 |
+| [wechsel_base_v7_2026-10-09_r5_s43](https://huggingface.co/davidmelash/wechsel_base_v7_2026-10-09_r5_s43) | RoBERTa-base | data v7 | 43 | 0.956 | 0.902 | 0.998 | 0.968 | 0.980 | 0.879 | 11 |
+| [wechsel_base_v7_2026-10-09_r5](https://huggingface.co/davidmelash/wechsel_base_v7_2026-10-09_r5) | RoBERTa-base | data v7 | 42 | 0.955 | 0.904 | 0.997 | 0.968 | 0.980 | 0.873 | 12 |
+| [modern_liberta_large_w2048_v7large_2026-10-09_r4_s43](https://huggingface.co/davidmelash/modern_liberta_large_w2048_v7large_2026-10-09_r4_s43) | ModernBERT-large | data v7-large | 43 | 0.953 | 0.925 | 0.997 | 0.972 | 0.963 | 0.881 | 19 |
+| [modern_liberta_large_w2048_v7large_2026-10-09_r4](https://huggingface.co/davidmelash/modern_liberta_large_w2048_v7large_2026-10-09_r4) | ModernBERT-large | data v7-large | 42 | 0.952 | 0.920 | 0.996 | 0.974 | 0.956 | 0.883 | 17 |
+| [modern_liberta_large_w2048_v7_2026-10-09_r5](https://huggingface.co/davidmelash/modern_liberta_large_w2048_v7_2026-10-09_r5) | ModernBERT-large | data v7 | 42 | 0.947 | 0.914 | 0.997 | 0.954 | 0.967 | 0.870 | 15 |
+| **wechsel_base** (this model) | RoBERTa-base | data v2 | 42 | **0.870** | 0.828 | 0.996 | 0.791 | 0.927 | 0.766 | 15 |
+
+## Evaluation
+
+The model was scored on the test set of a Ukrainian court-decision pseudonymization challenge: **240 real court decisions**
+from the Unified State Register of Court Decisions of Ukraine (ЄДРСР, 1.4M characters), whose anonymized fragments the
+organisers replaced with realistic synthetic values and annotated at the character level (3,685 entity mentions; 864 people). The test set was never used for training or checkpoint selection.
+
+**Metric:** F1 per entity type, a predicted span counts as correct when it has the right type and an intersection over
+union of at least 0.5 with a gold span (*relaxed*); *strict* requires exact boundaries. The headline number is the
+unweighted (macro) average over the four types, so the rare types weigh as much as persons. The scores are for the
+model alone — no rules, no post-processing — computed with exactly the code of [How to use](#how-to-use).
+
+| Entity type | Precision | Recall | F1 (relaxed) | F1 (strict) | Gold mentions |
+|---|---|---|---|---|---|
+| ОСОБА | 0.997 | 0.995 | **0.996** | 0.914 | 2,837 |
+| АДРЕСА | 0.863 | 0.730 | **0.791** | 0.753 | 233 |
+| НОМЕР | 0.997 | 0.866 | **0.927** | 0.901 | 447 |
+| ІНФОРМАЦІЯ | 0.947 | 0.643 | **0.766** | 0.745 | 168 |
+| **Macro average** | 0.951 | 0.808 | **0.870** | 0.828 | |
+
+| Other views of the same predictions | Value |
+|---|---|
+| Macro F1 on the corrected gold (18 mis-annotated test documents fixed) | 0.862 |
+| Persons whose name leaks at least once (deanonymization risk by name, lower is better) | 0.017 (15 of 864) |
+
+A 95 % bootstrap interval over documents is about ±0.03 for the macro F1, so models closer than about 0.01 are
+statistically indistinguishable.
+
+## Training data
+
+The model was trained on an early version (v2) of the synthetic data pipeline: **24,097 court decisions** from [lang-uk/court-decisions-uk](https://huggingface.co/datasets/lang-uk/court-decisions-uk) with **403,029 labelled entity spans** (ОСОБА 78 %, ІНФОРМАЦІЯ 8 %, НОМЕР 8 %, АДРЕСА 6 %) and 3,558 masked spans. Addresses come from the Ukrposhta directory only. This version predates the third corpus (2024 Supreme Court decisions from the court register), the generator of military-body names and the masking of ambiguous values used by the v7 models.
+
+There is no public, labelled corpus of Ukrainian personal data, so the training set is built by *distant supervision*
+from published court decisions (code: [`datagen/`](https://github.com/EnderLancer/anonimimizer/tree/main/datagen)). The v2 data of this model was built with an early version of these steps; the list describes the current (v7) pipeline.
+
+1. **Placeholders.** Court clerks publish decisions with personal data replaced by placeholders such as `ОСОБА_1`,
+   `АДРЕСА_1`, `НОМЕР_1`, `ІНФОРМАЦІЯ_1`. Each placeholder tells where an entity was and of which type.
+2. **Synthetic values.** Every placeholder is filled with a realistic value in the grammatical case its context needs:
+   names from [lang-uk/name_freq_dict_uk](https://github.com/lang-uk/name_freq_dict_uk); addresses from the Ukrposhta
+   directory in several writing styles; taxpayer numbers (РНОКПП), passports, ID cards, phones,
+   IBANs, bank cards, licence plates in Cyrillic and Latin letters, VINs, IMEIs, military unit numbers and
+   enforcement-proceeding numbers; dates of birth and death in five formats; e-mails, URLs and names of military bodies.
+   The same placeholder gets the same value throughout a decision.
+3. **Labels.** The filled values become character-level labels. People whose names the clerk left open (judges,
+   lawyers, secretaries) are labelled ОСОБА as well, by role, initials, heading and signature rules checked against a
+   general Ukrainian NER model, because in the target data *every* natural person is pseudonymized.
+4. **Masks.** Values whose type the context does not settle — for example a civil body or company in an anonymized
+   slot, an open phone number with no person nearby, or an enforcement-proceeding number with no natural person
+   involved — are excluded from the loss (label `-100`) instead of being taught as either entity or non-entity.
+5. **Hygiene.** The 250 decisions the challenge test set was cut from, other decisions of the same court cases and
+   near-duplicates (word 5-gram containment ≥ 0.5) are removed; the train / dev split is grouped by case number.
+   An independent audit of the v2 data, done after this model was trained, found no test document in it, but 3 training decisions belong to the same court cases as test documents: v2 predates the exclusion step.
+
+## Training procedure
+
+Fine-tuned from [benjamin/roberta-base-wechsel-ukrainian](https://huggingface.co/benjamin/roberta-base-wechsel-ukrainian), a Ukrainian RoBERTa-base obtained from English RoBERTa with WECHSEL embedding transfer (MIT), with a token-classification head over 9 BIO labels (`O`, `B-`/`I-` × 4 types).
+
+| Setting | Value |
+|---|---|
+| Epochs | 3 (the last checkpoint is published) |
+| Learning rate | 4e-05, linear decay, warm-up 6 % of the steps |
+| Optimizer | AdamW, weight decay 0.01 |
+| Batch size | 64 windows |
+| Window | 512 tokens, stride 128 (long decisions are split into overlapping windows) |
+| Oversampling | decisions with a НОМЕР entity are seen twice per epoch (31,978 training documents, 97,592 windows) |
+| Precision | bf16 mixed precision |
+| Seed | 42 |
+| Hardware | 1 × A100-SXM4-40GB, 22 min |
+| Final training loss | 0.0195 |
+| Dev F1 per epoch | 0.9869, 0.9893, 0.9906 (500 synthetic Supreme Court decisions; the dev set saturates and does not rank models like the test set) |
+
+Training code: [`train/`](https://github.com/EnderLancer/anonimimizer/tree/main/train) (`python scripts/train.py --model benjamin/roberta-base-wechsel-ukrainian --epochs 3 --lr 4e-05 --batch 64 --seed 42`).
+
+## Intended use
+
+- Pseudonymization of Ukrainian court decisions before publication in an open register, in the style of the Unified
+  State Register of Court Decisions (ЄДРСР).
+- Finding personal data in other Ukrainian legal and administrative documents: claims, rulings, orders, letters.
+- A PII detector inside a larger de-identification pipeline, together with rules and human review.
+- Research on Ukrainian legal NLP, PII detection and synthetic training data.
+
+## Limitations
+
+- **Domain.** The model learnt from court decisions, where names follow procedural roles («позивач», «відповідач»). On
+  news, encyclopaedia and web text an earlier model of this family found fewer than one name in five, so do not use it
+  as a general-purpose Ukrainian NER model.
+- **Label conventions.** It follows the de-identification practice of Ukrainian courts, not a generic PII taxonomy:
+  civil bodies and companies are not labelled, a date is labelled only when it is a date of birth or death, and a
+  military body is labelled only as a party to the case. Clerks apply these rules inconsistently, which caps the
+  ІНФОРМАЦІЯ score.
+- **Casing and noise.** Training text is normally cased: names written entirely in capitals (for example in signature
+  blocks, «Ольга КАШПУР») and words with Latin look-alike letters are recognised less reliably.
+- **Phones and rare identifiers.** Open phone numbers were mostly masked in training, so phone recall is lower than for
+  other numbers.
+- **Synthetic data.** Values were generated, so some inflected forms are imperfect, and the test set itself is
+  synthetic; real decisions may contain patterns that neither covers. A few data-generation choices (date formats,
+  excerpt length) follow published statistics of the test set.
+- **Not a guarantee.** Automatic pseudonymization reduces but does not remove the risk of re-identification. Review
+  the output before publishing personal data.
+
+## Deployment
+
+The model needs no GPU: with PyTorch the 240 test decisions (1.4M characters) take about 100 s on a laptop CPU (Apple M1 Pro); exported to ONNX with int8 quantization, the same test runs in a Docker container in about 290 s on 4 CPU cores with 0.5 GB of RAM (p50 1.2 s per decision), at a cost of about 0.001 F1. This fits the challenge limits of 8 GB RAM and 2 hours for the whole test set by
+a wide margin. See [`docker/`](https://github.com/EnderLancer/anonimimizer/tree/main/docker) for the ONNX export and the offline image.
+
+## Licence and attribution
+
+The weights are released under the **MIT licence**. Training texts come from
+[lang-uk/court-decisions-uk](https://huggingface.co/datasets/lang-uk/court-decisions-uk) (MIT).
+Synthetic names come from [lang-uk/name_freq_dict_uk](https://github.com/lang-uk/name_freq_dict_uk) and addresses from
+the Ukrposhta directory.
 
-Token classification model that finds personal data in Ukrainian court decisions for pseudonymization.
+## Citation
 
-Entity types: `ОСОБА`, `АДРЕСА`, `НОМЕР`, `ІНФОРМАЦІЯ`.
+```bibtex
+@misc{wechsel_base,
+  title        = {Ukrainian PII NER for court decisions — RoBERTa-base (WECHSEL), data v2, seed 42},
+  author       = {Melashych, David},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/davidmelash/wechsel_base}},
+  note         = {Code: https://github.com/EnderLancer/anonimimizer}
+}
+```