Instructions to use atonlee/koelectra-ko-pii-ner with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use atonlee/koelectra-ko-pii-ner with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="atonlee/koelectra-ko-pii-ner")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("atonlee/koelectra-ko-pii-ner") model = AutoModelForTokenClassification.from_pretrained("atonlee/koelectra-ko-pii-ner", device_map="auto") - Notebooks
- Google Colab
- Kaggle
koelectra-ko-pii-ner Β· Korean PII Detection Model
A 14M-parameter NER model that detects personal data in Korean text and returns its position and type.
Used for masking personal data, or for identifying it before text is sent to an external LLM or to the cloud.
| tier | criterion | recommended action | main labels |
|---|---|---|---|
tier1 |
identifies a person on its own | do not send off-device | resident registration number Β· alien registration number Β· passport Β· driver's licence Β· card number Β· account number Β· password |
tier2 |
identifies a person in combination with other data | mask, then send | name Β· phone Β· email Β· address Β· date of birth Β· user id Β· employee id Β· vehicle plate Β· IP |
none |
entity that is not treated as personal data | send as-is | date Β· time Β· quantity Β· place Β· organisation |
none does not mean the sentence contains no personal data. It means an entity that
is detected but not treated as personal data, such as PLACE or ORGANIZATION.
Usage
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
MODEL_ID = "atonlee/koelectra-ko-pii-ner"
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
model = AutoModelForTokenClassification.from_pretrained(MODEL_ID).eval()
text = "κΉλ―Όμ€ μ¨νν
010-1234-5678λ‘ μ°λ½ν΄μ€"
inputs = tokenizer(
text,
return_offsets_mapping=True,
return_tensors="pt",
truncation=True,
max_length=512,
)
offsets = inputs.pop("offset_mapping")[0]
with torch.no_grad():
logits = model(**inputs).logits
predictions = logits.argmax(dim=-1)[0]
for pred, (start, end) in zip(predictions, offsets):
if start == end:
continue
label = model.config.id2label[pred.item()]
if label != "O":
print(label, repr(text[start:end]), f"[{start}:{end}]")
Output:
B-NAME 'κΉλ―Ό' [0:2]
I-NAME 'μ€' [2:3]
B-PHONE '010' [8:11]
I-PHONE '-' [11:12]
I-PHONE '123' [12:15]
I-PHONE '4' [15:16]
I-PHONE '-' [16:17]
I-PHONE '56' [17:19]
I-PHONE '78' [19:21]
The model predicts personal data using the BIO scheme.
B-β start of a regionI-β continuation of the same regionOβ token that is not a target
Tokens are subword units, so consecutive BIO tags have to be joined to recover the actual region.
κΉλ―Όμ€ NAME
010-1234-5678 PHONE
900101-1234567 RRN
5310-99**-****-1122 CARD_NUMBER
μμΈμ κ°λ¨κ΅¬ ν
ν€λλ‘ 152 3μΈ΅ ADDRESS
One joined region is one entity. κΉλ―Όμ€ = NAME (0, 3) is one. A sentence can
contain several, and the evaluation below counts entities, not sentences.
Labels
29 labels.
tier1 β do not send off-device
Identifies a person on its own.
| label | description | example |
|---|---|---|
RRN |
resident registration number | 900101-1234567 |
ALIEN_ID |
alien registration number | 920414-6613077 |
PASSPORT |
passport number | W858W0028 |
DRIVER_LICENSE |
driver's licence number | 19-61-620553-00 |
IPIN |
i-PIN Β· connecting information | 7275209485124 |
CARD_NUMBER |
card number | 4594-1722-2866-5259 |
CARD_EXPIRY |
card expiry date | 10/2028 |
CVC |
card security code | 488 |
ACCOUNT_NUMBER |
bank account number | 719-938-027247 |
SECRET |
password Β· credential | A2UfgtKg!W1x |
tier2 β mask, then send
Identifies a person in combination with other data.
| label | description | example |
|---|---|---|
NAME |
person's name | κΉλ―Όμ€ |
PHONE |
phone number | 010-6517-8506 |
EMAIL |
email address | jkim899@example.com |
ADDRESS |
address with a building or unit number | μΈμ²κ΄μμ μ°μꡬ μ‘λλ 6 |
ZIPCODE |
postal code | 55857 |
BIRTHDATE |
date of birth | 2001λ
5μ 25μΌ |
USER_ID |
account id | quiet_owl22 |
GENERIC_ID |
assigned number β employee id, membership id | MEM-865502 |
VEHICLE_NUMBER |
vehicle plate | 89λΌ 8008 |
IP_ADDRESS |
IP address | 221.139.235.53 |
URL |
link containing a personal identifier | https://example.co.kr/account/9uemxnc5l |
DEPARTMENT |
department | μΈμ¬ν |
POSITION |
job title | νμ₯ |
ATTRIBUTE |
personal attribute β blood type, religion, major | Aν Β· μ²μ£Όκ΅ |
none β send as-is
Detected, but not treated as personal data.
| label | description | example |
|---|---|---|
PLACE |
place name without a building number | κ°λ¦μ Β· λ©κ°λ°μ€ κ°λ¨ |
ORGANIZATION |
institution Β· company Β· group | νκ²½λΆ Β· NAVER |
DATE |
date | 1939-10-14 |
TIME |
time | μ€μ |
QUANTITY |
quantity | 40% |
ADDRESS and PLACE are split by form. With a building or unit number it is
ADDRESS (tier2); without one it is PLACE (none).
ONNX
ONNX models are included.
| file | size |
|---|---|
model.onnx |
56.6 MB |
model_fp16.onnx |
28.5 MB |
model_int8.onnx |
14.7 MB |
model_uint8.onnx |
14.7 MB |
model_quantized.onnx |
14.7 MB |
model_q4f16.onnx |
15.0 MB |
model_quantized.onnx is the uint8 model under the name Transformers.js looks for.
Training
| Base model | monologg/koelectra-small-v3-discriminator |
| Parameters | 14.1M |
| Tokenizer | Korean WordPiece |
| Task | Token Classification |
| Tagging | BIO |
| Max input | 512 tokens |
Training used public datasets and hand-written PII examples.
| source | licence | contribution |
|---|---|---|
BCCard/pii-masking-openpii-finance |
CC BY 4.0 | financial and administrative prose, structured identifiers |
townboy/korean-pii-dataset |
CC BY 4.0 | Korean names, affiliations, personal-data expressions |
The following were written for this model and added to the training data.
- masked identifiers
- partially revealed identifiers
- ordinary sentences containing no personal data
- strings shaped like identifiers but not personal data
Neither the source datasets nor the merged training corpus is redistributed here.
Evaluation
5,171 test sentences carry ground-truth annotations β 17,537 entities in total, personal data included. Predictions were compared against those annotations.
Three criteria, differing in how much has to match.
| criterion | counts as correct when | needed for |
|---|---|---|
| position | it overlaps the ground truth | deciding whether personal data is present |
| type | it overlaps and the label matches | tier routing |
| boundary | start and end character positions match too | masking |
| criterion | Precision | Recall | F1 |
|---|---|---|---|
| position | 93.8% | 98.4% | 96.1% |
| type | 92.0% | 96.5% | 94.2% |
| boundary | 86.1% | 90.3% | 88.1% |
By tier, on the type criterion.
| tier | ground truth | Precision | Recall | F1 |
|---|---|---|---|---|
tier1 |
1,979 | 84.8% | 94.3% | 89.3% |
tier2 |
8,975 | 94.9% | 98.5% | 96.7% |
none |
6,583 | 90.2% | 94.3% | 92.2% |
Licence
Model weights: Apache-2.0
Inherited from the base model. Each external dataset used in training keeps its own licence.
BCCard/pii-masking-openpii-financeβ CC BY 4.0townboy/korean-pii-datasetβ CC BY 4.0
This repository does not relicense or redistribute the source datasets.
- Downloads last month
- 2,072
Model tree for atonlee/koelectra-ko-pii-ner
Base model
monologg/koelectra-small-v3-discriminator