Update README.md
Browse files
README.md
CHANGED
|
@@ -31,12 +31,11 @@ metrics:
|
|
| 31 |
---
|
| 32 |
|
| 33 |
# 1. Overview
|
| 34 |
-
This repository provides the INT8 weight-only ONNX Runtime serving artifact for a Korean/English PII detection model for the finance domain. The model was built by full fine-tuning [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter)
|
| 35 |
-
BIOES token classes at token level.
|
| 36 |
|
| 37 |
-
The model is designed as the NER layer of a multi-layer PII-masking gateway in front of LLM services
|
| 38 |
|
| 39 |
-
This repository does not contain PyTorch weights. `AutoModelForTokenClassification.from_pretrained()`
|
| 40 |
and the standard Transformers token-classification pipeline therefore do not load this artifact. Use ONNX Runtime as shown in [Usage](#13-usage).
|
| 41 |
|
| 42 |
The serving chain is:
|
|
@@ -55,14 +54,14 @@ text
|
|
| 55 |
* **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
|
| 56 |
* **Domain / Language**: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
|
| 57 |
* **Task**: Token classification (BIOES) → character-offset PII spans → masking
|
| 58 |
-
* **Labels (18)**: `PERSON, RRN, FRN, CARD_NUMBER, ACCOUNT_NUMBER, SECRET, USER_ID, EMAIL, PHONE, PASSPORT, DRIVER_LICENSE, GENERIC_ID, ADDRESS, ZIPCODE, DATE, CARD_EXPIRY, CVC, IPIN`
|
| 59 |
* **Method**: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
|
| 60 |
* **Decoding**: **constrained BIOES Viterbi** (not per-token argmax) + whitespace span refinement — the bundled `viterbi_calibration.json` exposes precision↔recall operating-point biases without retraining
|
| 61 |
* **Format**: INT8 weight-only ONNX (`MatMulNBits`, `QMoE`, `GatherBlockQuantized`) with FP32 activations and logits; graph + external tensor data + tokenizer + label taxonomy + Viterbi calibration sidecar
|
| 62 |
* **Sequence length**: trained on sequences ≤768 tokens — chunk longer inputs
|
| 63 |
-
* **Intended use**
|
| 64 |
-
|
| 65 |
-
|
| 66 |
|
| 67 |
## 1.2. Label Taxonomy (N=18)
|
| 68 |
The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (`GIVENNAME`/`SURNAME` → `PERSON`, `CITY`/`STREET`/`BUILDINGNUM` → `ADDRESS`) and adding Korea-specific classes absent upstream (`RRN`, `FRN`, `IPIN`, `CARD_EXPIRY`, `CVC`, `SECRET`). `data source` records the row-source buckets in which each label occurs: `ko` means `openpii-1.5m-ko`, `en` means `openpii-1.5m-en`, and `domain` means locally synthesized rows.
|
|
@@ -71,7 +70,7 @@ The 18 labels re-map the upstream ai4privacy source labels to the granularity a
|
|
| 71 |
|-------|-------------|-------------|
|
| 72 |
| `PERSON` | full name (surname + given, single span) | ko, en, domain |
|
| 73 |
| `RRN` | resident registration number (Korea) | ko, domain |
|
| 74 |
-
| `FRN` | foreign registration number | domain |
|
| 75 |
| `CARD_NUMBER` | credit/debit card PAN | ko, en, domain |
|
| 76 |
| `ACCOUNT_NUMBER` | bank account number | ko, domain |
|
| 77 |
| `SECRET` | auth secret (password / API key / token) | ko, domain |
|
|
@@ -80,15 +79,15 @@ The 18 labels re-map the upstream ai4privacy source labels to the granularity a
|
|
| 80 |
| `PHONE` | phone number (mobile / landline) | ko, en, domain |
|
| 81 |
| `PASSPORT` | passport number | ko, en, domain |
|
| 82 |
| `DRIVER_LICENSE` | driver's license number | ko, en, domain |
|
| 83 |
-
| `GENERIC_ID` | generic identifier
|
| 84 |
| `ADDRESS` | address (city / street / building, single span) | ko, en, domain |
|
| 85 |
| `ZIPCODE` | postal code | ko, en, domain |
|
| 86 |
| `DATE` | date / time | ko, en, domain |
|
| 87 |
| `CARD_EXPIRY` | card expiry date | domain |
|
| 88 |
| `CVC` | card verification code | domain |
|
| 89 |
-
| `IPIN` | I-PIN number | domain |
|
| 90 |
|
| 91 |
-
Each entity type has `B-`, `I-`, `E-` and `S-` boundary classes, plus the background class `O`.
|
| 92 |
|
| 93 |
## 1.3. Usage
|
| 94 |
Install the CPU runtime and client dependencies:
|
|
@@ -170,18 +169,16 @@ The offsets use Python's half-open character interval `[start, end)`. Masking is
|
|
| 170 |
|
| 171 |
For batches, enable right padding and pass only `input_ids` and `attention_mask` to the graph. `offset_mapping` stays outside ONNX and is used only to map decoded token tags back to the original text. Inputs are INT64 and the graph always returns FP32 logits.
|
| 172 |
|
| 173 |
-
|
| 174 |
## 1.4. Training Data
|
| 175 |
| Dataset | Role | Size |
|
| 176 |
|---------|------|------|
|
| 177 |
| (Public) [BCCard/pii-masking-openpii-finance](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance) (v2) | Training / validation | ~58.5k train rows · ~14.5k validation rows |
|
| 178 |
-
| (Private) BCCard/pii-masking-openpii-finance-test (v2) | Golden Set (release
|
| 179 |
|
| 180 |
-
* Sources: curated Korean subset of `ai4privacy/pii-masking-openpii-1.5m` (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + **~30% English replay** (forgetting guard)
|
| 181 |
* Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
|
| 182 |
* All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)
|
| 183 |
|
| 184 |
-
|
| 185 |
## 1.5. Training Procedure
|
| 186 |
| Item | Value |
|
| 187 |
|------|-------|
|
|
@@ -217,14 +214,14 @@ The INT8 artifact was evaluated on all 14,543 v2 validation rows with the same t
|
|
| 217 |
| ko macro F1 | 0.9571 |
|
| 218 |
| en strict micro F1 | 0.9689 |
|
| 219 |
| en macro F1 | 0.9641 |
|
| 220 |
-
| masking coverage
|
| 221 |
-
| ko masking coverage
|
| 222 |
-
| en masking coverage
|
| 223 |
|
| 224 |
-
|
| 225 |
|
| 226 |
## 2.2. INT8 Parity Against FP32 ONNX
|
| 227 |
-
| Metric | FP32 ONNX | INT8 ONNX |
|
| 228 |
|---|---:|---:|---:|
|
| 229 |
| strict micro F1 | 0.959801 | 0.959886 | +0.000085 |
|
| 230 |
| macro F1 | 0.960264 | 0.960662 | +0.000398 |
|
|
@@ -272,7 +269,7 @@ INT8 predictions reached strict micro F1 0.998568 against FP32 predictions and m
|
|
| 272 |
* Source data attribution - `ai4privacy/pii-masking-openpii-1.5m` under CC-BY-4.0
|
| 273 |
|
| 274 |
## 4.3. Viterbi Reference Implementation
|
| 275 |
-
These functions apply the BIOES transition constraints and calibration biases used for validation, then recover character-offset spans with whitespace boundary refinement.
|
| 276 |
|
| 277 |
<details>
|
| 278 |
<summary>Show the constrained Viterbi and span-decoding reference code</summary>
|
|
|
|
| 31 |
---
|
| 32 |
|
| 33 |
# 1. Overview
|
| 34 |
+
This repository provides the **INT8 weight-only ONNX Runtime serving artifact** for a Korean/English **PII detection model for the finance domain**. The model was built by full fine-tuning [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) (1.4B MoE, 50M active) on synthetic finance-domain PII data. It tags **18 PII entity types** (73 BIOES classes) at token level.
|
|
|
|
| 35 |
|
| 36 |
+
The model is designed as the **NER layer of a multi-layer PII-masking gateway** in front of LLM services and as a PII detection component for offline privacy review and audit workflows. High-sensitivity deployments require in-domain evaluation and ongoing monitoring.
|
| 37 |
|
| 38 |
+
This repository does **not contain PyTorch weights**. `AutoModelForTokenClassification.from_pretrained()`
|
| 39 |
and the standard Transformers token-classification pipeline therefore do not load this artifact. Use ONNX Runtime as shown in [Usage](#13-usage).
|
| 40 |
|
| 41 |
The serving chain is:
|
|
|
|
| 54 |
* **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
|
| 55 |
* **Domain / Language**: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
|
| 56 |
* **Task**: Token classification (BIOES) → character-offset PII spans → masking
|
| 57 |
+
* **Labels (18)**: `PERSON`, `RRN`, `FRN`, `CARD_NUMBER`, `ACCOUNT_NUMBER`, `SECRET`, `USER_ID`, `EMAIL`, `PHONE`, `PASSPORT`, `DRIVER_LICENSE`, `GENERIC_ID`, `ADDRESS`, `ZIPCODE`, `DATE`, `CARD_EXPIRY`, `CVC`, `IPIN`
|
| 58 |
* **Method**: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
|
| 59 |
* **Decoding**: **constrained BIOES Viterbi** (not per-token argmax) + whitespace span refinement — the bundled `viterbi_calibration.json` exposes precision↔recall operating-point biases without retraining
|
| 60 |
* **Format**: INT8 weight-only ONNX (`MatMulNBits`, `QMoE`, `GatherBlockQuantized`) with FP32 activations and logits; graph + external tensor data + tokenizer + label taxonomy + Viterbi calibration sidecar
|
| 61 |
* **Sequence length**: trained on sequences ≤768 tokens — chunk longer inputs
|
| 62 |
+
* **Intended use**:
|
| 63 |
+
1. In-house PII masking gateway (detect → mask before text reaches an LLM)
|
| 64 |
+
2. Offline privacy review and audit support (PII discovery in stored text, logs and documents)
|
| 65 |
|
| 66 |
## 1.2. Label Taxonomy (N=18)
|
| 67 |
The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (`GIVENNAME`/`SURNAME` → `PERSON`, `CITY`/`STREET`/`BUILDINGNUM` → `ADDRESS`) and adding Korea-specific classes absent upstream (`RRN`, `FRN`, `IPIN`, `CARD_EXPIRY`, `CVC`, `SECRET`). `data source` records the row-source buckets in which each label occurs: `ko` means `openpii-1.5m-ko`, `en` means `openpii-1.5m-en`, and `domain` means locally synthesized rows.
|
|
|
|
| 70 |
|-------|-------------|-------------|
|
| 71 |
| `PERSON` | full name (surname + given, single span) | ko, en, domain |
|
| 72 |
| `RRN` | resident registration number (Korea) | ko, domain |
|
| 73 |
+
| `FRN` | foreign registration number (Korea) | domain |
|
| 74 |
| `CARD_NUMBER` | credit/debit card PAN | ko, en, domain |
|
| 75 |
| `ACCOUNT_NUMBER` | bank account number | ko, domain |
|
| 76 |
| `SECRET` | auth secret (password / API key / token) | ko, domain |
|
|
|
|
| 79 |
| `PHONE` | phone number (mobile / landline) | ko, en, domain |
|
| 80 |
| `PASSPORT` | passport number | ko, en, domain |
|
| 81 |
| `DRIVER_LICENSE` | driver's license number | ko, en, domain |
|
| 82 |
+
| `GENERIC_ID` | generic identifier without a more specific taxonomy class | ko, en, domain |
|
| 83 |
| `ADDRESS` | address (city / street / building, single span) | ko, en, domain |
|
| 84 |
| `ZIPCODE` | postal code | ko, en, domain |
|
| 85 |
| `DATE` | date / time | ko, en, domain |
|
| 86 |
| `CARD_EXPIRY` | card expiry date | domain |
|
| 87 |
| `CVC` | card verification code | domain |
|
| 88 |
+
| `IPIN` | I-PIN number (Korea only) | domain |
|
| 89 |
|
| 90 |
+
Each entity type has `B-`, `I-`, `E-` and `S-` boundary classes, plus the background class `O`.
|
| 91 |
|
| 92 |
## 1.3. Usage
|
| 93 |
Install the CPU runtime and client dependencies:
|
|
|
|
| 169 |
|
| 170 |
For batches, enable right padding and pass only `input_ids` and `attention_mask` to the graph. `offset_mapping` stays outside ONNX and is used only to map decoded token tags back to the original text. Inputs are INT64 and the graph always returns FP32 logits.
|
| 171 |
|
|
|
|
| 172 |
## 1.4. Training Data
|
| 173 |
| Dataset | Role | Size |
|
| 174 |
|---------|------|------|
|
| 175 |
| (Public) [BCCard/pii-masking-openpii-finance](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance) (v2) | Training / validation | ~58.5k train rows · ~14.5k validation rows |
|
| 176 |
+
| (Private) BCCard/pii-masking-openpii-finance-test (v2) | Golden Set (release evaluation; not used for training/tuning) | 2,000 rows (ko 1,460 / en 540) |
|
| 177 |
|
| 178 |
+
* Sources: curated Korean subset of `ai4privacy/pii-masking-openpii-1.5m` (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + **~30% English replay** (catastrophic forgetting guard)
|
| 179 |
* Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
|
| 180 |
* All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)
|
| 181 |
|
|
|
|
| 182 |
## 1.5. Training Procedure
|
| 183 |
| Item | Value |
|
| 184 |
|------|-------|
|
|
|
|
| 214 |
| ko macro F1 | 0.9571 |
|
| 215 |
| en strict micro F1 | 0.9689 |
|
| 216 |
| en macro F1 | 0.9641 |
|
| 217 |
+
| **masking coverage** | **0.9980** |
|
| 218 |
+
| **ko masking coverage** | **0.9987** |
|
| 219 |
+
| **en masking coverage** | **0.9967** |
|
| 220 |
|
| 221 |
+
**Observed masking coverage is 99.8% overall and at least 99.6% in both the Korean and English validation slices.** This leakage-oriented metric is reported as a diagnostic rather than a release gate.
|
| 222 |
|
| 223 |
## 2.2. INT8 Parity Against FP32 ONNX
|
| 224 |
+
| Metric | FP32 ONNX | INT8 ONNX | Δ |
|
| 225 |
|---|---:|---:|---:|
|
| 226 |
| strict micro F1 | 0.959801 | 0.959886 | +0.000085 |
|
| 227 |
| macro F1 | 0.960264 | 0.960662 | +0.000398 |
|
|
|
|
| 269 |
* Source data attribution - `ai4privacy/pii-masking-openpii-1.5m` under CC-BY-4.0
|
| 270 |
|
| 271 |
## 4.3. Viterbi Reference Implementation
|
| 272 |
+
These functions apply the BIOES transition constraints and calibration biases used for validation, then recover character-offset spans with whitespace boundary refinement. `config.json` defines the class-ID order for logits, while `viterbi_calibration.json` supplies the additive transition biases used by the decoder.
|
| 273 |
|
| 274 |
<details>
|
| 275 |
<summary>Show the constrained Viterbi and span-decoding reference code</summary>
|