becleverksh commited on
Commit
63f53a1
·
verified ·
1 Parent(s): ad336d5

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +19 -22
README.md CHANGED
@@ -31,12 +31,11 @@ metrics:
31
  ---
32
 
33
  # 1. Overview
34
- This repository provides the INT8 weight-only ONNX Runtime serving artifact for a Korean/English PII detection model for the finance domain. The model was built by full fine-tuning [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) - a 1.4B-parameter MoE with 50M active parameters - on synthetic finance-domain PII data. It tags 18 PII entity types using 73
35
- BIOES token classes at token level.
36
 
37
- The model is designed as the NER layer of a multi-layer PII-masking gateway in front of LLM services. It should be deployed with a deterministic regex backstop for fully structured identifiers and must not be treated as a standalone anonymization, compliance or privacy guarantee. High-sensitivity deployments require in-domain evaluation and ongoing monitoring.
38
 
39
- This repository does not contain PyTorch weights. `AutoModelForTokenClassification.from_pretrained()`
40
  and the standard Transformers token-classification pipeline therefore do not load this artifact. Use ONNX Runtime as shown in [Usage](#13-usage).
41
 
42
  The serving chain is:
@@ -55,14 +54,14 @@ text
55
  * **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
56
  * **Domain / Language**: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
57
  * **Task**: Token classification (BIOES) → character-offset PII spans → masking
58
- * **Labels (18)**: `PERSON, RRN, FRN, CARD_NUMBER, ACCOUNT_NUMBER, SECRET, USER_ID, EMAIL, PHONE, PASSPORT, DRIVER_LICENSE, GENERIC_ID, ADDRESS, ZIPCODE, DATE, CARD_EXPIRY, CVC, IPIN`
59
  * **Method**: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
60
  * **Decoding**: **constrained BIOES Viterbi** (not per-token argmax) + whitespace span refinement — the bundled `viterbi_calibration.json` exposes precision↔recall operating-point biases without retraining
61
  * **Format**: INT8 weight-only ONNX (`MatMulNBits`, `QMoE`, `GatherBlockQuantized`) with FP32 activations and logits; graph + external tensor data + tokenizer + label taxonomy + Viterbi calibration sidecar
62
  * **Sequence length**: trained on sequences ≤768 tokens — chunk longer inputs
63
- * **Intended use**
64
- - In-house **PII masking gateway** (detect → mask before text reaches an LLM)
65
- - Korean-centric finance text with mixed English (IDs, e-mails, card numbers)
66
 
67
  ## 1.2. Label Taxonomy (N=18)
68
  The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (`GIVENNAME`/`SURNAME` → `PERSON`, `CITY`/`STREET`/`BUILDINGNUM` → `ADDRESS`) and adding Korea-specific classes absent upstream (`RRN`, `FRN`, `IPIN`, `CARD_EXPIRY`, `CVC`, `SECRET`). `data source` records the row-source buckets in which each label occurs: `ko` means `openpii-1.5m-ko`, `en` means `openpii-1.5m-en`, and `domain` means locally synthesized rows.
@@ -71,7 +70,7 @@ The 18 labels re-map the upstream ai4privacy source labels to the granularity a
71
  |-------|-------------|-------------|
72
  | `PERSON` | full name (surname + given, single span) | ko, en, domain |
73
  | `RRN` | resident registration number (Korea) | ko, domain |
74
- | `FRN` | foreign registration number | domain |
75
  | `CARD_NUMBER` | credit/debit card PAN | ko, en, domain |
76
  | `ACCOUNT_NUMBER` | bank account number | ko, domain |
77
  | `SECRET` | auth secret (password / API key / token) | ko, domain |
@@ -80,15 +79,15 @@ The 18 labels re-map the upstream ai4privacy source labels to the granularity a
80
  | `PHONE` | phone number (mobile / landline) | ko, en, domain |
81
  | `PASSPORT` | passport number | ko, en, domain |
82
  | `DRIVER_LICENSE` | driver's license number | ko, en, domain |
83
- | `GENERIC_ID` | generic identifier (no KO counterpart) | ko, en, domain |
84
  | `ADDRESS` | address (city / street / building, single span) | ko, en, domain |
85
  | `ZIPCODE` | postal code | ko, en, domain |
86
  | `DATE` | date / time | ko, en, domain |
87
  | `CARD_EXPIRY` | card expiry date | domain |
88
  | `CVC` | card verification code | domain |
89
- | `IPIN` | I-PIN number | domain |
90
 
91
- Each entity type has `B-`, `I-`, `E-` and `S-` boundary classes, plus the background class `O`. This yields 73 output classes. The bundled `label-taxonomy.yaml` and `config.json` must remain in the same label order.
92
 
93
  ## 1.3. Usage
94
  Install the CPU runtime and client dependencies:
@@ -170,18 +169,16 @@ The offsets use Python's half-open character interval `[start, end)`. Masking is
170
 
171
  For batches, enable right padding and pass only `input_ids` and `attention_mask` to the graph. `offset_mapping` stays outside ONNX and is used only to map decoded token tags back to the original text. Inputs are INT64 and the graph always returns FP32 logits.
172
 
173
-
174
  ## 1.4. Training Data
175
  | Dataset | Role | Size |
176
  |---------|------|------|
177
  | (Public) [BCCard/pii-masking-openpii-finance](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance) (v2) | Training / validation | ~58.5k train rows · ~14.5k validation rows |
178
- | (Private) BCCard/pii-masking-openpii-finance-test (v2) | Golden Set (release gate; never used for training/tuning) | 2,000 rows (ko 1,460 / en 540) |
179
 
180
- * Sources: curated Korean subset of `ai4privacy/pii-masking-openpii-1.5m` (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + **~30% English replay** (forgetting guard)
181
  * Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
182
  * All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)
183
 
184
-
185
  ## 1.5. Training Procedure
186
  | Item | Value |
187
  |------|-------|
@@ -217,14 +214,14 @@ The INT8 artifact was evaluated on all 14,543 v2 validation rows with the same t
217
  | ko macro F1 | 0.9571 |
218
  | en strict micro F1 | 0.9689 |
219
  | en macro F1 | 0.9641 |
220
- | masking coverage - diagnostic | 0.9980 |
221
- | ko masking coverage - diagnostic | 0.9987 |
222
- | en masking coverage - diagnostic | 0.9967 |
223
 
224
- Masking coverage is the proportion of gold PII characters covered by the union of predicted spans, regardless of predicted label. It is a diagnostic value and was not used as the INT8 pass or fail gate.
225
 
226
  ## 2.2. INT8 Parity Against FP32 ONNX
227
- | Metric | FP32 ONNX | INT8 ONNX | Delta |
228
  |---|---:|---:|---:|
229
  | strict micro F1 | 0.959801 | 0.959886 | +0.000085 |
230
  | macro F1 | 0.960264 | 0.960662 | +0.000398 |
@@ -272,7 +269,7 @@ INT8 predictions reached strict micro F1 0.998568 against FP32 predictions and m
272
  * Source data attribution - `ai4privacy/pii-masking-openpii-1.5m` under CC-BY-4.0
273
 
274
  ## 4.3. Viterbi Reference Implementation
275
- These functions apply the BIOES transition constraints and calibration biases used for validation, then recover character-offset spans with whitespace boundary refinement. Keep `config.json`, `label-taxonomy.yaml` and `viterbi_calibration.json` in their bundled label order and configuration.
276
 
277
  <details>
278
  <summary>Show the constrained Viterbi and span-decoding reference code</summary>
 
31
  ---
32
 
33
  # 1. Overview
34
+ This repository provides the **INT8 weight-only ONNX Runtime serving artifact** for a Korean/English **PII detection model for the finance domain**. The model was built by full fine-tuning [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) (1.4B MoE, 50M active) on synthetic finance-domain PII data. It tags **18 PII entity types** (73 BIOES classes) at token level.
 
35
 
36
+ The model is designed as the **NER layer of a multi-layer PII-masking gateway** in front of LLM services and as a PII detection component for offline privacy review and audit workflows. High-sensitivity deployments require in-domain evaluation and ongoing monitoring.
37
 
38
+ This repository does **not contain PyTorch weights**. `AutoModelForTokenClassification.from_pretrained()`
39
  and the standard Transformers token-classification pipeline therefore do not load this artifact. Use ONNX Runtime as shown in [Usage](#13-usage).
40
 
41
  The serving chain is:
 
54
  * **Base model**: [`openai/privacy-filter`](https://huggingface.co/openai/privacy-filter) — 1.4B-parameter MoE (128 experts, 50M active), 8 layers, hidden 640, bidirectional banded attention (±128), o200k tokenizer
55
  * **Domain / Language**: Finance (BC Card — cards, accounts, national IDs, customer service text) / Korean + English
56
  * **Task**: Token classification (BIOES) → character-offset PII spans → masking
57
+ * **Labels (18)**: `PERSON`, `RRN`, `FRN`, `CARD_NUMBER`, `ACCOUNT_NUMBER`, `SECRET`, `USER_ID`, `EMAIL`, `PHONE`, `PASSPORT`, `DRIVER_LICENSE`, `GENERIC_ID`, `ADDRESS`, `ZIPCODE`, `DATE`, `CARD_EXPIRY`, `CVC`, `IPIN`
58
  * **Method**: Full fine-tuning (all parameters incl. experts & router) with a re-initialized 73-class head (rows copied from the base head by taxonomy mapping)
59
  * **Decoding**: **constrained BIOES Viterbi** (not per-token argmax) + whitespace span refinement — the bundled `viterbi_calibration.json` exposes precision↔recall operating-point biases without retraining
60
  * **Format**: INT8 weight-only ONNX (`MatMulNBits`, `QMoE`, `GatherBlockQuantized`) with FP32 activations and logits; graph + external tensor data + tokenizer + label taxonomy + Viterbi calibration sidecar
61
  * **Sequence length**: trained on sequences ≤768 tokens — chunk longer inputs
62
+ * **Intended use**:
63
+ 1. In-house PII masking gateway (detect → mask before text reaches an LLM)
64
+ 2. Offline privacy review and audit support (PII discovery in stored text, logs and documents)
65
 
66
  ## 1.2. Label Taxonomy (N=18)
67
  The 18 labels re-map the upstream ai4privacy source labels to the granularity a Korean financial masking policy needs - merging fragments into single spans (`GIVENNAME`/`SURNAME` → `PERSON`, `CITY`/`STREET`/`BUILDINGNUM` → `ADDRESS`) and adding Korea-specific classes absent upstream (`RRN`, `FRN`, `IPIN`, `CARD_EXPIRY`, `CVC`, `SECRET`). `data source` records the row-source buckets in which each label occurs: `ko` means `openpii-1.5m-ko`, `en` means `openpii-1.5m-en`, and `domain` means locally synthesized rows.
 
70
  |-------|-------------|-------------|
71
  | `PERSON` | full name (surname + given, single span) | ko, en, domain |
72
  | `RRN` | resident registration number (Korea) | ko, domain |
73
+ | `FRN` | foreign registration number (Korea) | domain |
74
  | `CARD_NUMBER` | credit/debit card PAN | ko, en, domain |
75
  | `ACCOUNT_NUMBER` | bank account number | ko, domain |
76
  | `SECRET` | auth secret (password / API key / token) | ko, domain |
 
79
  | `PHONE` | phone number (mobile / landline) | ko, en, domain |
80
  | `PASSPORT` | passport number | ko, en, domain |
81
  | `DRIVER_LICENSE` | driver's license number | ko, en, domain |
82
+ | `GENERIC_ID` | generic identifier without a more specific taxonomy class | ko, en, domain |
83
  | `ADDRESS` | address (city / street / building, single span) | ko, en, domain |
84
  | `ZIPCODE` | postal code | ko, en, domain |
85
  | `DATE` | date / time | ko, en, domain |
86
  | `CARD_EXPIRY` | card expiry date | domain |
87
  | `CVC` | card verification code | domain |
88
+ | `IPIN` | I-PIN number (Korea only) | domain |
89
 
90
+ Each entity type has `B-`, `I-`, `E-` and `S-` boundary classes, plus the background class `O`.
91
 
92
  ## 1.3. Usage
93
  Install the CPU runtime and client dependencies:
 
169
 
170
  For batches, enable right padding and pass only `input_ids` and `attention_mask` to the graph. `offset_mapping` stays outside ONNX and is used only to map decoded token tags back to the original text. Inputs are INT64 and the graph always returns FP32 logits.
171
 
 
172
  ## 1.4. Training Data
173
  | Dataset | Role | Size |
174
  |---------|------|------|
175
  | (Public) [BCCard/pii-masking-openpii-finance](https://huggingface.co/datasets/BCCard/pii-masking-openpii-finance) (v2) | Training / validation | ~58.5k train rows · ~14.5k validation rows |
176
+ | (Private) BCCard/pii-masking-openpii-finance-test (v2) | Golden Set (release evaluation; not used for training/tuning) | 2,000 rows (ko 1,460 / en 540) |
177
 
178
+ * Sources: curated Korean subset of `ai4privacy/pii-masking-openpii-1.5m` (label taxonomy remapped, name spans merged & naturalized) + finance-domain synthetic templates + **~30% English replay** (catastrophic forgetting guard)
179
  * Hard-example design baked into v2: surface-similar non-PII decoys (FP suppression), label-confusion pairs in one sentence (RRN↔FRN, DRIVER_LICENSE↔GENERIC_ID), weak-context true PII (FN suppression), long-span address boundary variants
180
  * All values are synthetic; validity-pattern collisions with real identifiers are removed at generation time (e.g. card numbers are forced to fail Luhn)
181
 
 
182
  ## 1.5. Training Procedure
183
  | Item | Value |
184
  |------|-------|
 
214
  | ko macro F1 | 0.9571 |
215
  | en strict micro F1 | 0.9689 |
216
  | en macro F1 | 0.9641 |
217
+ | **masking coverage** | **0.9980** |
218
+ | **ko masking coverage** | **0.9987** |
219
+ | **en masking coverage** | **0.9967** |
220
 
221
+ **Observed masking coverage is 99.8% overall and at least 99.6% in both the Korean and English validation slices.** This leakage-oriented metric is reported as a diagnostic rather than a release gate.
222
 
223
  ## 2.2. INT8 Parity Against FP32 ONNX
224
+ | Metric | FP32 ONNX | INT8 ONNX | Δ |
225
  |---|---:|---:|---:|
226
  | strict micro F1 | 0.959801 | 0.959886 | +0.000085 |
227
  | macro F1 | 0.960264 | 0.960662 | +0.000398 |
 
269
  * Source data attribution - `ai4privacy/pii-masking-openpii-1.5m` under CC-BY-4.0
270
 
271
  ## 4.3. Viterbi Reference Implementation
272
+ These functions apply the BIOES transition constraints and calibration biases used for validation, then recover character-offset spans with whitespace boundary refinement. `config.json` defines the class-ID order for logits, while `viterbi_calibration.json` supplies the additive transition biases used by the decoder.
273
 
274
  <details>
275
  <summary>Show the constrained Viterbi and span-decoding reference code</summary>