Instructions to use mulkzz/kestrel-edu-ner-22l with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mulkzz/kestrel-edu-ner-22l with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("token-classification", model="mulkzz/kestrel-edu-ner-22l")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForTokenClassification tokenizer = AutoTokenizer.from_pretrained("mulkzz/kestrel-edu-ner-22l") model = AutoModelForTokenClassification.from_pretrained("mulkzz/kestrel-edu-ner-22l", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Korean | English
1. ๋ชจ๋ธ ์์ฝ (v1.2.1, ๋ผ๋ฒจ ํ๋กํ์ผ edu-v1+fmt ยท 29ํด๋์ค BIO 22์ธต ๊ต์ฌ ๋ชจ๋ธ)
mulkzz/kestrel-edu-ner-22l(v1.2.1, ๋ผ๋ฒจ ํ๋กํ์ผ edu-v1+fmt)๋ ๊ฒฝ๋ ์๋น ํ์ ๋ชจ๋ธ(mulkzz/kestrel-edu-ner-6l)์ ๋ค์ค ์ ํธ ์ง์ ์ฆ๋ฅ(Stage 2)๋ฅผ ๊ฐ๋
ํ๊ธฐ ์ํด ํ์ต๋ 22์ธต ์์ ๊ต์ฌ(Teacher) ํ ํฐ ๋ถ๋ฅ ๋ชจ๋ธ(149.34M ํ๋ผ๋ฏธํฐ / ๋น์๋ฒ ๋ฉ 110.94M)์
๋๋ค. skt/A.X-Encoder-base(9708f9c, Apache-2.0) ์์ 4๋ ํต์ฌ ๋น์ ํ ๊ฐ์ฒด(PERSON, SCHOOL, ORG, ADDRESS) + 10์ข
์ ํ ๋ณด์กฐ ๊ฐ์ฒด(PHONE, EMAIL, RRN, PASSPORT, DRIVER_LICENSE, ACCOUNT, CARD, BRN, URL, CODE) ์ด 29ํด๋์ค BIO ํค๋๋ฅผ ์น๊ณ , ์กฐ์ฌ ๋ถ๋ฆฌ ์ ๋ ฌ(particle_aware)๊ณผ ๋ํ์ด์์ด(amb ๋ amb_neg) + ๋๋ช
์ด์ธ(D1=B ๊ต๊ณผ ๊ณต์ธ O ๋ ์ค์ ํ์ PERSON) ๋์ผ ๋ฏธ๋๋ฐฐ์น ๋์กฐ ์ํ๋ฌ๋ก ๋ฏธ์ธ์กฐ์ ํ์ต๋๋ค.
- ๋ธ๋์น ยท ๋ฆฌ๋น์ ๋ฐ ์๋น ์๋ด:
- ๊ฒฝ๋ 6์ธต ์๋น ํ์ ๋ชจ๋ธ (์ด์ ๊ถ์ฅ): mulkzz/kestrel-edu-ner-6l (
69.62 MBINT8 ONNX, CPU2.17ร๊ฐ์,dorms-pii-ner-bench v1.1post-v1.1์ ์ค๋ฅ91.6% [589/643]/ ๋๋ฝ8.4%). mainยทv1.2.1ยทv1.2: 29ํด๋์คedu-v1+fmt๊ต์ฌ ๊ฐ์ค์น(sha256: 9998e608...) ๋ฐv1.2.1์ต์ ๋ฒค์น๋งํฌ ๋ํ๋ฅผ ์ ๊ณตํฉ๋๋ค. ์ด๊ธฐ 9ํด๋์ค(edu-v1) ๋จ๋ฐ ๊ธฐ์ค์ ์revision="v1.0"(133302653e95c30d0e8ea0729f0e6f9649d4a0eb)์ ์๊ตฌ ๋ณด์กด๋ฉ๋๋ค.- ์ค์๊ฐ CPU ์๋น ์ฐธ๊ณ : 22์ธต ๊ต์ฌ ๋ชจ๋ธ์ ECI
C-4(4 vCPU) ๊ธฐ์ค20,000์ร๋์ 4์์ฒญํฌ์ ์ง์ฐ์ด~54.5s(40s์ํ ์ด๊ณผ)์ด๋ฏ๋ก ์ค์๊ฐ ๊ฒ์ดํธ์จ์ดยท์จ๋๋ฐ์ด์ค ๋ธ๋ฆฌ์ง์๋ 6์ธต ์ฆ๋ฅ ํ์ ๋ชจ๋ธ์ ์ฌ์ฉํ๋ฉฐ, ๋ณธ ์ ์ฅ์๋ ์ง์ ์ฆ๋ฅ ์ฌํยท์คํ๋ผ์ธ ๊ณ ์ ๋ฐ ๊ธฐ์ค์ ยทํ์ ์ฐ๊ตฌ์ฉ์ผ๋ก ๊ณต๊ฐํฉ๋๋ค.
- ๊ฒฝ๋ 6์ธต ์๋น ํ์ ๋ชจ๋ธ (์ด์ ๊ถ์ฅ): mulkzz/kestrel-edu-ner-6l (
- ํ์ต ๋ฐ์ดํฐ ์์น: ์ค์ ํ์ ์๋ฃ
0๊ฑด, ์ธ๋ถ ์์ฉ LLM ์์ฑ ํ์ต ๋ฐ์ดํฐ0๊ฑด(28,000๊ฑด ๊ฒฐ์ ๋ก ์ ํฉ์ฑ ๊ต์ก ๋ฌธ์).
6์ธต ์ฆ๋ฅ ํ์์ post-v1.1 ๋ฒค์น๋งํฌ ๋ฐ ๊ต์ฌ-ํ์ ์ฉ๋ ๋น๊ต
Fig. 1 | 22์ธต ๊ต์ฌ(
kestrel-edu-ner-22l) โ 6์ธต ํ์(kestrel-edu-ner-6l) ์์ถ ์ ์ง์ฐยท๋ฉ๋ชจ๋ฆฌ ์ ๊ฐ ๋ฐv1.2.14์ค ๊ฒ์ดํธ ์์ฝ. ๋น์๋ฒ ๋ฉ ์ฐ์ฐ ํ๋ผ๋ฏธํฐ๋ฅผ110.94M โ 30.70M(-72.3%)์ผ๋ก ์ค์ฌ Local Bridge 20k์ ์ง์ฐ์6.49s โ 1.82s, PII Worker20kร4์ง์ฐ์17.03s โ 4.40s๋ก ๋จ์ถํ๋ฉด์๋ ๋ฏธ๋ ธ์ถ ๋จ๋ฐitest_v1_2(3,000๊ฑด) ๋งคํฌ๋ก ํต์ฌ ์ฌํ์จ97.84% โ 97.44%(์ ํ 1,817 span100.0% โ 100.0%)๋ฅผ ์ ์งํจ. 6์ธต ํ์์dorms-pii-ner-bench v1.1(post-v1.1, ๋๋ฝ ๋ฌธํญ17.0% โ 8.4%, $p < 0.0001$) ์์ธ ์ฑ์ ์ mulkzz/kestrel-edu-ner-6l ์นด๋ ๋ฐ./figures/ner_bench_v1_1_6l.svg์ฐธ์กฐ.
Fig. 2 | Stage 1 22์ธต ๊ต์ฌ ๋ฏธ์ธ์กฐ์ ์๋ ด ๋์ญํ ๋ฐ ๋ถํฌ ๋ด(
dev_v1_2/ ๋จ๋ฐitest_v1_23,000๊ฑด) ์ฑ์ .28,000๊ฑด ํฉ์ฑ ๊ต์ก ๋ฌธ์์์ ์ฐจ๋ฑ ํ์ต๋ฅ (backbone 3e-5,classifier 1e-4, ์ฝ์ฌ์ธ ์ค์ผ์ค, ์คํจ ๋ฐฐ์น32,bf16)๋ก ํ์ตํ์ฌ1.5์ํญ(step=1,783,7.44Mํ ํฐ)์์ ์ต๊ณ ๊ฒฐํฉ ์ ์(selection_score=1.7843, ์ ํ ์ฌํ์จ100.0%, ๊ณต์ธ ์คํ2.31%)๋ฅผ ๋ฌ์ฑํจ.
2. Stage 1 ๊ต์ฌ ํ์ต ์ค์ ๋ฐ ์ํญ๋ณ ์๋ ด ์ถ์ด
- ๋ฐ์ดํฐ์
: ๊ต์ฌ ์
๋ฌด 12์ฅ๋ฅด ๋๊ฒฐ ํ ์ํ ๊ธฐ๋ฐ ๊ฒฐ์ ๋ก ์ ํฉ์ฑ ๋ฐ์ดํฐ(
train 28,000/dev_v1_2 3,000/ ๋จ๋ฐitest_v1_2 3,000). ์ค์ ํ์ ๋ฐ์ดํฐ0๊ฑด, ์ธ๋ถ ์์ฉ LLM ์์ฑ๋ฌธ0๊ฑด. - ์ต์ ํ:
AdamW(backbone_lr=3e-5,classifier_lr=1e-4,weight_decay=0.01,warmup_ratio=0.06,max_grad_norm=1.0,micro_batch=16 ร grad_accum=2), ์ฌ๋กฏ ๋์ ์นํ ์ฆ๊ฐ15%, ๋ค์ค ์ค์ผ์ผ ์ฐฝ(W-40030% ํผํฉ).
ํ๊ฐ ์์ (epoch) |
๊ธ๋ก๋ฒ ์คํ | ๋์ ์ ํจ ํ ํฐ | dev_loss |
selection_score |
macro_core_recall |
formal_recall |
homonym_student |
public_person_fp |
๋น๊ณ |
|---|---|---|---|---|---|---|---|---|---|
| 0.5 | 593 | 2.55M | 0.00408 | 1.7837 | 98.58% | 100.00% | 97.57% | 0.33% | ์ด๊ธฐ ์๋ ด |
| 1.0 | 1,187 | 4.95M | 0.00446 | 1.7700 | 98.55% | 100.00% | 93.69% | 0.33% | no_imp=1 |
| 1.5 | 1,783 | 7.44M | 0.00620 | 1.7843 | 98.60% | 100.00% | 97.09% | 2.31% | ์ต์ข
์ฑํ (*BEST*) |
| 2.0 | 2,379 | 9.91M | 0.00496 | 1.7584 | 98.62% | 100.00% | 96.60% | 8.91% | no_imp=1 |
| 2.5 | 2,976 | 12.40M | 0.00861 | 1.7708 | 98.64% | 100.00% | 96.12% | 8.25% | no_imp=2 |
| 3.0 | 3,573 | 14.88M | 0.01840 | 1.7785 | 98.66% | 100.00% | 98.06% | 9.24% | ์กฐ๊ธฐ ์ข
๋ฃ (patience=3) |
๋ถํฌ ๋ด dev_v1_2 (3,000๊ฑด) ๋ฐ ๋ฏธ๋
ธ์ถ ๋จ๋ฐ itest_v1_2 (3,000๊ฑด) ๋น๊ต (W-400, pipeline ๋ทฐ)
| ๋ถํ | ๋ชจ๋ธ | macro_core_recall |
overall_core_recall |
formal_recall |
PERSON |
SCHOOL |
ORG |
ADDRESS |
public_person_fp (NER) |
|---|---|---|---|---|---|---|---|---|---|
dev_v1_2 (3,000) |
kestrel-edu-ner-22l (๊ต์ฌ) |
98.71% | 98.32% | 100.00% | 98.24% | 100.00% | 96.60% | 100.00% | 2.31% |
dev_v1_2 (3,000) |
kestrel-edu-ner-6l (๋ณด์ ํ์) |
98.45% | 97.84% | 100.00% | 97.69% | 99.60% | 96.60% | 99.90% | 5.28% |
itest_v1_2 (3,000) |
kestrel-edu-ner-22l (๊ต์ฌ) |
97.84% | 96.79% | 100.00% | 96.60% | 100.00% | 94.90% | 99.85% | 0.00% |
itest_v1_2 (3,000) |
kestrel-edu-ner-6l (๋ณด์ ํ์) |
97.44% | 95.77% | 100.00% | 95.44% | 99.56% | 94.75% | 100.00% | 3.74% |
3. ํธ์ถ ์์ (transformers) ๋ฐ ๊ณ ์ง
import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer
REPO_ID = "mulkzz/kestrel-edu-ner-22l"
REV = "v1.2.1" # ๋๋ ๊ธฐ๋ณธ ๋ธ๋์น "main"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision=REV)
model = AutoModelForTokenClassification.from_pretrained(REPO_ID, revision=REV).eval()
text = "๋ณ๋น๊ณ ๋ฐ ๋ฏธ๋ ํ์๊ณผ ์ด๋กฑ์ด๊ฐ ํด์ค๋ฆ์๋ด์ผํฐ(๋ฌผ์๊ฐ๋ก 58, 1์ธต)์ ๋ฐฉ๋ฌธํ๋ค."
inputs = tokenizer(text, return_offsets_mapping=True, return_tensors="pt")
offsets = inputs.pop("offset_mapping")[0].tolist()
with torch.no_grad():
pred = model(**inputs).logits[0].argmax(dim=-1).tolist()
for pid, (s, e) in zip(pred, offsets):
if e > s and pid != 0:
print(model.config.id2label[pid], text[s:e], (s, e))
- ์ฃผ์ (
pad_token_id = 49999): ์๋ ๋ฐฐ์น ํจ๋ฉ ์ ๋ฐ๋์49999๋ฅผ ์ฌ์ฉํ์ญ์์ค(0์[CLS]). - ํ๊ณยท์ธ๋ถ ๋ฐ์ดํฐ ๊ท์ยท๋ผ์ด์ ์ค: ๋ณธ ๋ชจ๋ธ์ ๊ฐ๋ช
ํ ๋ณด์กฐ ์ฐ๊ตฌยท์ฆ๋ฅ์ฉ ์์ ๋ชจ๋ธ์ด๋ฉฐ ๋ชจ๋ ๊ฐ์ธ์ ๋ณด์ ์์ ํ ํ์ง๋ ๋ฒ์ ๋ฉด์ฑ
์ ๋ณด์ฆํ์ง ์์ต๋๋ค. ์ธ๋ถ ํ๊ฐ ์ฝํผ์ค(
townboy/korean-pii-datasetCC BY 4.0,KDPIIZenodo10968609CC BY 4.0, ์ฝ๊ธฐ ์ ์ฉ ์ง๊ณ๋ง ์ฌ์ฉ) ์์ธ ๊ฒฐ๊ณผ๋ mulkzz/kestrel-edu-ner-6l ๋ชจ๋ธ ์นด๋๋ฅผ ์ฐธ์กฐํ์ญ์์ค. ๋ผ์ด์ ์ค๋ Apache-2.0 (NOTICE)์ ๋๋ค.
- Downloads last month
- 64
Model tree for mulkzz/kestrel-edu-ner-22l
Base model
skt/A.X-Encoder-base