kestrel-edu-ner-6l v1.2.1 โ€” ํ•œ๊ตญ์–ด ๊ต์œก ๋„๋ฉ”์ธ ๊ฒฝ๋Ÿ‰ PII ๊ฐœ์ฒด๋ช… ์ธ์‹ ๋ชจ๋ธ (edu-v1+fmt)

Korean | English

1. ๋ชจ๋ธ ์š”์•ฝ (v1.2.1, ๋ผ๋ฒจ ํ”„๋กœํŒŒ์ผ edu-v1+fmt ยท 29ํด๋ž˜์Šค BIO)

mulkzz/kestrel-edu-ner-6l(v1.2.1, ๋ผ๋ฒจ ํ”„๋กœํŒŒ์ผ edu-v1+fmt)๋Š” ๊ต์‚ฌ๊ฐ€ ์™ธ๋ถ€ LLM(OpenRouter ๋“ฑ)์— ํ•™์ƒ ๊ธฐ๋กยท์ƒ๋‹ด ์ผ์ง€ยท์ƒํ™œ๊ธฐ๋ก๋ถ€ ์ดˆ์•ˆยทํ•™๋ถ€๋ชจ ์•ˆ๋‚ด๋ฌธ์„ ๋ณด๋‚ด๊ธฐ ์ „์— ๊ฐœ์ธ์‹๋ณ„์ •๋ณด(PII)๋ฅผ ํƒ์ง€ยท๊ฐ€๋ช…ํ™”ํ•˜๊ธฐ ์œ„ํ•ด ์„ค๊ณ„๋œ ํ•œ๊ตญ์–ด ๊ต์œก ๋„๋ฉ”์ธ ์ „์šฉ 6์ธต ๊ฒฝ๋Ÿ‰ ํ† ํฐ ๋ถ„๋ฅ˜(NER) ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค. skt/A.X-Encoder-base(9708f9c, Apache-2.0)๋ฅผ 22์ธต ๊ต์‚ฌ ๋ชจ๋ธ(mulkzz/kestrel-edu-ner-22l@20133bbd)๋กœ ๋ฏธ์„ธ์กฐ์ •ํ•œ ๋’ค, ์ „์—ญยท๊ตญ์†Œ ์–ดํ…์…˜ ๋ฐ RoPE ์ฃผํŒŒ์ˆ˜๊ฐ€ 1:1๋กœ ๋Œ€์‘ํ•˜๋Š” 6๊ฐœ ์ธต([0, 4, 9, 13, 18, 20], global_attn_every_n_layers=2)์œผ๋กœ ๋‹ค์ค‘ ์‹ ํ˜ธ ์ง€์‹ ์ฆ๋ฅ˜(Logit KL + ์€๋‹‰์ธต ์ฝ”์‚ฌ์ธ ์ •๋ ฌ + ํ•˜๋“œ CE)ํ•˜๊ณ  ๋ฌด์ฝ”๋“œ ์˜จ๋„ยท๋งˆ์ง„ ๋ณด์ •($T = 1.428$, $\delta_O = +0.225$) ๋ฐ ์ฑ„๋„๋ณ„ ๋™์  INT8 ์–‘์žํ™”(onnx/model_int8.onnx, 69.62 MB)๋ฅผ ์ ์šฉํ–ˆ์Šต๋‹ˆ๋‹ค.

๋ณธ v1.2.1 ๋ฆด๋ฆฌ์Šค๋Š” v1.2์˜ 29ํด๋ž˜์Šค ์‹ ๊ฒฝ๋ง ๊ฐ€์ค‘์น˜(sha256: b8010203..., ๋ฌด์žฌํ•™์Šต 100% ๋™์ผ) ์œ„์— ๋Ÿฐํƒ€์ž„ ํ•œ๊ตญ์–ด ์กฐ์‚ฌยท์–ด๊ฐ„ ํ›„์ฒ˜๋ฆฌ ๊ฒฐํ•จ 3๊ฑด(1์Œ์ ˆ ํ•œ๊ธ€ยทํ•œ์ž ์ธ๋ช… ๋ณด์กด, ๊ณต๋ฐฑ ์„ฑ๋ช… ์–ด๊ฐ„ ํ•˜ํ•œ, ๋ฐ›์นจ ์Œ์šด ์กฐ๊ฑด๋ถ€ ์กฐ์‚ฌ ์ ˆ๋‹จ)์„ ์ˆ˜์ •ํ•˜์—ฌ dorms-pii-ner-bench v1.1 ํ•ต์‹ฌ ์ •๋‹ต ๋ˆ„๋ฝ ๋ฌธํ•ญ ๋น„์œจ์„ 17.0% โ†’ 8.4% (-8.6%p, ์–‘์ธก McNemar $p < 0.0001$)๋กœ ์ ˆ๋ฐ˜ ์ดํ•˜ ๊ฐ์ถ•ํ•œ ์ตœ์‹  ๋ฆด๋ฆฌ์Šค์ž…๋‹ˆ๋‹ค.

  • ๋ธŒ๋žœ์น˜ ยท ๋ฆฌ๋น„์ „ ๋ฐ ๋ฐฐํฌ ํ˜„ํ™ฉ ์•ˆ๋‚ด:
    • main ยท v1.2.1 ยท v1.2 (ํ˜„์žฌ ๊ธฐ๋ณธ๊ฐ’): 29ํด๋ž˜์Šค edu-v1+fmt ๊ฐ€์ค‘์น˜ ๋ฐ v1.2.1 ํ‰๊ฐ€ยทํ›„์ฒ˜๋ฆฌ ๊ทœ๊ฒฉ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. DoRms Alias ๋กœ์ปฌ ๋ธŒ๋ฆฌ์ง€(alias-bridge) ์˜จ๋””๋ฐ”์ด์Šค ๊ธฐ๋ณธ ๋ชจ๋ธ ๋ฐ ์„œ๋ฒ„ pii-worker ๋น„๋™๊ธฐ ๋ฌด๋…ธ์ถœ Shadow ๊ฒ€์ฆ ๋ชจ๋ธ๋กœ ์ฑ„ํƒ๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.
    • v1.0 ๋ธŒ๋žœ์น˜ ยท ํƒœ๊ทธ (์˜๊ตฌ ๋™๊ฒฐ ๊ธฐ์ค€์„ ): ์ตœ์ดˆ ๋‹จ๋ฐœ(Single-Shot) 9ํด๋ž˜์Šค(edu-v1) ๊ธฐ์ค€์„  ๊ฐ€์ค‘์น˜์™€ ๊ธฐ๋ก์€ revision="v1.0"(b61b3e8231491d16dca16566ef03e34ed57fcda0)์— ๋ฌด๋ณ€๊ฒฝ ์ƒํƒœ๋กœ ์˜๊ตฌ ๋ณด์กด๋ฉ๋‹ˆ๋‹ค.
  • ์ƒ์œ„ 22์ธต ๊ต์‚ฌ ๋ชจ๋ธ: mulkzz/kestrel-edu-ner-22l (20133bbde74f46bee9d51dcbb937cd58683a9200)
  • ๊ธฐ๋ฐ˜ ์ธ์ฝ”๋”: skt/A.X-Encoder-base (9708f9c404ace91efd25c06fac2d73413616f4ef, pad_token_id=49999, Apache-2.0)
  • ํฌํ•จ ๊ฐ€์ค‘์น˜ (v1.2์™€ ๋น„ํŠธ ๋‹จ์œ„ ๋™์ผ ํ•ด์‹œ):
    • model.safetensors: ์˜จ๋„ยท๋งˆ์ง„ ๋ณด์ • ํก์ˆ˜ ์™„๋ฃŒ(T = 1.428, O ๋งˆ์ง„ +0.225) 6์ธต FP32 ๊ฐ€์ค‘์น˜ (276.42 MB, ์ด ํŒŒ๋ผ๋ฏธํ„ฐ 69.10M / ๋น„์ž„๋ฒ ๋”ฉ ํ–‰๋ ฌ ์—ฐ์‚ฐ ํŒŒ๋ผ๋ฏธํ„ฐ 30.70M)
    • onnx/model.onnx: ONNX Opset 17 FP32 ๊ธฐ์ค€ ๊ทธ๋ž˜ํ”„ (276.60 MB, PyTorch ๋Œ€๋น„ max |ฮ”logit| = 7.49e-5, argmax ์ผ์น˜ 100.00%)
    • onnx/model_int8.onnx: MatMul + Gemm + Gather ์ฑ„๋„๋ณ„ ๋™์  INT8 ๋ฐฐํฌ ๊ทธ๋ž˜ํ”„ (69.62 MB, 29ํด๋ž˜์Šค classifier ์„ ํ˜•์ธต FP32 ๋ณด์กด)
  • ํ•™์Šต ๋ฐ์ดํ„ฐ ์›์น™: ์‹ค์ œ ํ•™์ƒ ์ž๋ฃŒ 0๊ฑด, ์™ธ๋ถ€ ์ƒ์šฉ LLM(OpenAIยทGemini/Gemma ๋“ฑ) ์ƒ์„ฑ๋ฌธ 0๊ฑด. ๊ฒฐ์ •๋ก ์  ๊ทœ์น™ยท์Šฌ๋กฏ ๋ฌธ๋ฒ•๊ณผ ๊ต์‚ฌ ์—…๋ฌด 12์žฅ๋ฅด ๋™๊ฒฐ ํ‹€ ์€ํ–‰(๋ฒค์น˜๋งˆํฌ์™€ ํด๋Ÿฌ์Šคํ„ฐยท8-gram ์™„์ „ ๋ถ„๋ฆฌ)์—์„œ ํ•ฉ์„ฑํ•œ ๋ผ๋ฒจ 28,000๊ฑด + ๋ฌด๋ผ๋ฒจ 100,000๊ฑด์œผ๋กœ NVIDIA L40S์—์„œ ํ•™์Šตยท์ฆ๋ฅ˜ํ–ˆ์Šต๋‹ˆ๋‹ค(9,397์Šคํ…, 76.12M ์œ ํšจ ํ† ํฐ).

๋…๋ฆฝ ๋ฒค์น˜๋งˆํฌ dorms-pii-ner-bench v1.1 ์‚ฌํ›„ ๋ณด์™„ ์ธก์ • ์š”์•ฝ (post-v1.1, n = 848๋ฌธํ•ญ ยท 2,009 span)

์ธก์ • ์„ฑ๊ฒฉ ๊ณ ์ง€: dorms-pii-ner-bench v1.1์€ v1.0 ๋‹จ๋ฐœ ์ธก์ • ํ›„ ์‚ฌํ›„ ์›์ธ ๊ฐ์‚ฌ๋ฅผ ๋ฐ˜์˜ํ•œ v1.2 ์žฌํ•™์Šต ๋ฐ v1.2.1 ํ•œ๊ตญ์–ด ์กฐ์‚ฌ ํ›„์ฒ˜๋ฆฌ ๊ฒฐํ•จ ์ˆ˜์ •์„ ๊ฑฐ์ณ ์žฌ์ธก์ •ํ–ˆ์œผ๋ฏ€๋กœ ๋ชจ๋“  v1.1 ์ˆ˜์น˜์— post-v1.1(์‚ฌํ›„ ๋ณด์™„ ์ธก์ •) ํ‘œ๊ธฐ๋ฅผ ๋ณ‘๊ธฐํ•ฉ๋‹ˆ๋‹ค. ์—ด๋žŒ ์ด๋ ฅ์ด ์ „ํ˜€ ์—†๋Š” ๋ฏธ๋…ธ์ถœ ํ•ฉ์„ฑ ๋‹จ๋ฐœ ํ‰๊ฐ€๋Š” ๋ณ„๋„ ๋™๊ฒฐ๋œ itest_v1_2(3,000๋ฌธํ•ญ, ํ•˜๋‹จ 6์ ˆ)๋กœ ๊ต์ฐจ ๊ฒ€์ฆํ–ˆ์Šต๋‹ˆ๋‹ค.

Fig. 1 | post-v1.1 evaluation of kestrel-edu-ner-6l v1.2.1 vs koredact-int8

Fig. 1 | dorms-pii-ner-bench v1.1 (post-v1.1, 848๋ฌธํ•ญ ยท ํ•ต์‹ฌ ์ •๋‹ต 1,693๊ฐœ ยท ํŒ๋‹จ ์œ ๋ณด 316๊ฐœ)์—์„œ kestrel-edu-ner-6l v1.2.1 (onnx/model_int8.onnx, 29 BIO)๊ณผ ํ˜„ํ–‰ ๊ธฐ์ค€ ๋ชจ๋ธ koredact-int8 (infobank-corp/koredact-bert-base-onnx-int8@a2e7696, 12์ธต BERT INT8) ๋น„๊ต. a, ๊ทœ์น™ ๋‹จ๋…(NER disabled), ๊ทœ์น™+koredact-int8, ๊ทœ์น™+kestrel-6l v1.2.1์˜ ๋ฌธํ•ญ ์™„์ „ ์ ์ค‘๋ฅ (1 โˆ’ ๋ˆ„๋ฝ ๋ฌธํ•ญ ๋น„์œจ) ๋ฐ 4๋Œ€ ํ•ต์‹ฌ ์œ ํ˜• ์žฌํ˜„์œจ(Wilson 95% CI). ํ•ต์‹ฌ ์ •๋‹ต์ด ํ•˜๋‚˜๋ผ๋„ ๋ˆ„๋ฝ๋œ ๋ฌธํ•ญ ๋น„์œจ์ด 17.0% (109/643)(๋™์ผ v1.2.1 ๋Ÿฐํƒ€์ž„ ๊ธฐ์ค€ 16.8% 108/643)์—์„œ 8.4% (54/643)(์ ์ค‘๋ฅ  91.6%)๋กœ ์ ˆ๋ฐ˜ ์ดํ•˜ ๊ฐ์ถ•๋จ(๋ถˆ์ผ์น˜ 27 ๋Œ€ 82, ๋ฌธํ•ญ ๋‹จ์œ„ ์ •ํ™• ์–‘์ธก McNemar $p < 0.0001$). b, ๋ฌธํ•ญ ๊ตฐ์ง‘ ๋ถ€ํŠธ์ŠคํŠธ๋žฉ(2,000ํšŒ, Miller 2024 arXiv:2411.00640) ์ •๋‹ต ์žฌํ˜„์œจ ์Œ๋Œ€ ์ฐจ์ด: ์ „์ฒด ํ•ต์‹ฌ ์ •๋‹ต +4.8%p [+3.1, +6.6]($p < 0.0001$; NER ๋‹จ๋… +8.9%p [+6.7, +11.1]), ORG +63.7%p [+53.1, +74.0]($p < 0.0001$), SCHOOL +11.8%p [+5.2, +19.8]($p = 0.0020$; NER ๋‹จ๋… 19.4% โ†’ 96.8%), PERSON +0.5%p [-1.0, +2.1]($p = 0.8939$, 96.1% โ†’ 96.6%), ADDRESS -1.8%p [-5.8, +0.0]($p = 1.0000$, ๋„๋กœ๋ช… 96.0%). c, doc URL_ID(100%), ๊ฒฐํ•ฉ ํ˜ธ์นญ D5(100%), ๋กœ๋งˆ์ž ์ธ๋ช…(100%), 1์Œ์ ˆ ์™ธ์ž ํ•œ๊ธ€(98.4%), ๋„๋กœ๋ช… ์ฃผ์†Œ(96.0%), ๋„์–ด ์“ด ์„ฑ๋ช…(95.0%), ์• ์นญ(75.4% โ†’ 90.2%), PERSON ๋”ฑ ๋งž์Œ ๊ฒฝ๊ณ„(87.7% โ†’ 92.0%) ๋“ฑ ์ „ ์„ธ๋ถ€ ์Šฌ๋ผ์ด์Šค ์šฐ์„ธ ํ™•์ธ. d, ์ฐจ๋‹จ ๋ชจ๋“œ ํ‰๊ฐ€์…‹(holdout_w12/w13/w13b/w13c, teacher_eval)์˜ ๋ฌด๋ช… ์š”์ฒญ ์˜ค์ฐจ๋‹จ์œจ ์ผ์ œ ๊ฐ์†Œ ๋ฐ post-v1.1 ๋ฌดํ•ด ๋ฌธํ•ญ ์˜คํƒ๋ฅ  14.8% (18/122)(koredact 16.4% ํ•˜ํšŒ) ๋‹ฌ์„ฑ.

Fig. 2 | v1.2.1 quantization parity, latency/memory capacity, and 4-Gate scorecard

Fig. 2 | v1.2.1 ์–‘์žํ™” ๋™๋“ฑ์„ฑ(FP32 ๋Œ€ INT8), CPU ์ถ”๋ก  ์ง€์—ฐยท๋ฉ”๋ชจ๋ฆฌ ์‹ค์ธก, 4์ค‘ ์ฑ„ํƒ ๊ฒŒ์ดํŠธ ์ด๊ด„ํ‘œ. a, onnx/model.onnx(FP32, 276.60 MB) ๋Œ€ onnx/model_int8.onnx(int8_pc_gather, 69.62 MB) ๋น„๊ต: ๋ฌธํ•ญ ์ ์ค‘๋ฅ  91.1% โ†’ 91.6%($p = 0.2500$), ๊ณ ์œ  ๊ฐ’ ํ•ต์‹ฌ ์žฌํ˜„์œจ ์ฐจ์ด +0.1%p [-0.1, +0.4](SCHOOL/ORG/ADDRESS $p = 1.0000$, PERSON $p = 0.5000$)๋กœ 29ํด๋ž˜์Šค ํ™•์žฅ ํ›„์—๋„ ์–‘์žํ™” ์†์‹ค์ด 0์ž„์„ ์ž…์ฆํ•จ. b, CPU ์ถ”๋ก  ์ง€์—ฐ(post-v1.1 848๋ฌธํ•ญ 55.3s โ†’ 25.5s 2.17ร— ๊ฐ€์†; w13c_eval 8,484๊ฑด 156s โ†’ 72s 2.17ร— ๊ฐ€์†; PII Worker 20kร—4 p50 10.02s โ†’ 4.40s; Local Bridge 20k p50 3.84s โ†’ 1.82s; ECI C-4 ํˆฌ์˜ 32.0s โ†’ 14.1s๋กœ 40s ์ƒํ•œ ํ†ต๊ณผ). c, ๋””์Šคํฌ ํฌ๊ธฐ(111.2 โ†’ 69.6 MB, -37.4%) ๋ฐ ์„œ๋น™ ๋ฉ”๋ชจ๋ฆฌ ์˜ˆ์‚ฐ(Local Bridge 436.0 MiB, PII Worker 715.8 MiB). d, ์‚ฌ์ „ ๋“ฑ๋ก 4์ค‘ ๊ฒŒ์ดํŠธ ์„ธ๋ถ€ ์„ฑ์  ๋ฐ ๋ฐฐํฌ ํ˜„ํ™ฉ(alias-bridge ๊ธฐ๋ณธ ๋ชจ๋ธ + ์„œ๋ฒ„ ๋น„๋™๊ธฐ Shadow ๋ชจ๋ธ).

๊ตฌ๋ถ„ ํ‰๊ฐ€ ์ง€ํ‘œ (dorms-pii-ner-bench v1.1 post-v1.1, pipeline ๊ธฐ์ค€) ํ‘œ๋ณธ ์ˆ˜ ๊ทœ์น™+koredact-int8 ๊ทœ์น™+kestrel-6l v1.2.1 ์Œ๋Œ€ ์ฐจ์ด ฮ” (kestrel โˆ’ koredact) ๋ฐ ํŒ์ •
์ฃผ ์ง€ํ‘œ ํ•ต์‹ฌ ์ •๋‹ต ๋ˆ„๋ฝ ๋ฌธํ•ญ ๋น„์œจ (๋‚ฎ์„์ˆ˜๋ก ์šฐ์ˆ˜) 643 ๋ฌธํ•ญ 17.0% (109/643) [14โ€“20] 8.4% (54/643) [6โ€“11] -8.6%p (๋ถˆ์ผ์น˜ 27 ๋Œ€ 82, McNemar $p < 0.0001$, ๋Œ€ํญ ๊ฐœ์„ )
๊ณ ์œ  ๊ฐ’ ํ•ต์‹ฌ ์žฌํ˜„์œจ / ์ „์ฒด span ์Œ๋Œ€ ์ฐจ์ด 1,495 / 1,693 90.9% (1359/1495) 96.3% (1439/1495) span +4.8%p [+3.1, +6.6] ($p < 0.0001$; NER ๋‹จ๋… +8.9%p)
์œ ํ˜•๋ณ„ ์žฌํ˜„์œจ PERSON ํ•ต์‹ฌ ์žฌํ˜„์œจ (NER ๋‹จ๋…) 1,442 span 96.1% (94.7%) 96.6% (95.9%) +0.5%p [-1.0, +2.1], $p = 0.8939$ (๋‹จ๋… +1.2%p, ๋น„์—ด๋“ฑยท์ƒํšŒ)
SCHOOL ํ•ต์‹ฌ ์žฌํ˜„์œจ (NER ๋‹จ๋…) 93 span 86.0% (19.4%) 97.8% (96.8%) +11.8%p [+5.2, +19.8], $p = 0.0020$ (NER ๋‹จ๋… +77.4%p)
ORG ์ง€์—ญ ๊ธฐ๊ด€ยทํ•™์›ยท๋ณ‘์› ์žฌํ˜„์œจ (NER ๋‹จ๋…) 102 span 30.4% (30.4%) 94.1% (93.1%) +63.7%p [+53.1, +74.0], $p < 0.0001$ (3.1๋ฐฐ ๋„์•ฝ)
ADDRESS ํ•ต์‹ฌ ์žฌํ˜„์œจ (apt/lot/road) 56 span 85.7% (100/100/100%) 83.9% (100/100/96.0%) -1.8%p [-5.8, +0.0], $p = 1.0000$ (๋™๋“ฑ์„ฑ ํšŒ๋ณต, v1.0 -12.5%p ํ•ด์†Œ)
์„ธ๋ถ€ ์ผ๋ฐ˜ํ™” ํ’€ ๋ฐ– ํ˜•ํƒœ / 1์Œ์ ˆ ์™ธ์ž(single) / ํ•œ์ž(hanja) 133 / 63 / 10 80.5% / 93.7% / 100.0% 91.7% / 98.4% / 100.0% ํ’€ ๋ฐ– +11.2%p, ์™ธ์ž ํ•œ๊ธ€ 62/63 ํฌ์ฐฉ
์• ์นญ(nickname) / ๋„์–ด ์“ด ์„ฑ๋ช…(spaced) / ๋กœ๋งˆ์ž(roman) 61 / 20 / 22 75.4% / 90.0% / 63.6% 90.2% / 95.0% / 100.0% ์• ์นญ +14.8%p, ๋„์–ด ์“ด ์„ฑ๋ช… 19/20, ๋กœ๋งˆ์žยท๊ฒฐํ•ฉํ˜ธ์นญ(D5) ๋งŒ์ 
๊ฒฝ๊ณ„ ๋ฐ ์˜คํƒ PERSON ๋”ฑ ๋งž์Œ ๋น„์œจ / ์šฐ์ธก ๋„˜์นจ ๋น„์œจ (NER ๋‹จ๋…) 1,442 span 87.7% / 8.8% (7.7%) 92.0% / 4.8% (3.6%) ์ •๋ฐ€ ๊ฒฝ๊ณ„ +4.3%p ์ƒ์Šน, ์šฐ์ธก ์กฐ์‚ฌ ๋„˜์นจ ์ ˆ๋ฐ˜ ์ดํ•˜ ๊ฐ์ถ•
ADDRESS ๋”ฑ ๋งž์Œ / ORG ๋”ฑ ๋งž์Œ / SCHOOL ๋”ฑ ๋งž์Œ 56 / 102 / 93 57.1% / 27.5% / 84.9% 76.8% / 93.1% / 91.4% 4๋Œ€ ํ•ต์‹ฌ ์œ ํ˜• ์ „ ๊ตฌ๊ฐ„์—์„œ ๊ฒฝ๊ณ„ ์ •๋ฐ€๋„ ๋Œ€ํญ ํ–ฅ์ƒ
๋ฌดํ•ด ๋ฌธํ•ญ(negยทamb_neg) ์˜คํƒ๋ฅ  / X.echo / doc URL_ID 122 / 6 / 9 16.4% (20/122) / 3 / 100% 14.8% (18/122) / 1 / 100% ๋ฌดํ•ด ๋ฌธํ•ญ ์˜คํƒ๋ฅ ์ด koredact ์•„๋ž˜๋กœ ํ•˜๋ฝ, URL_ID 9/9 ๋งŒ์  ๋ณต๊ตฌ

v1.0 โ†’ v1.2 โ†’ v1.2.1 ๋‹จ๊ณ„๋ณ„ ์›์ธ ํ•ด๊ฒฐ ๋ฐ ๋ฌด์žฌํ•™์Šต ํ›„์ฒ˜๋ฆฌ ๊ฐœ์„  ์š”์•ฝ

v1.0(9ํด๋ž˜์Šค edu-v1) ๋‹จ๋ฐœ ์ธก์ • ์งํ›„ ๊ทœ๋ช…๋œ 5๋Œ€ ๋ฐ์ดํ„ฐยท์Šคํ‚ค๋งˆ ๊ณผ์ œ๋ฅผ v1.2(29ํด๋ž˜์Šค edu-v1+fmt) ์žฌํ•™์Šต์œผ๋กœ ํ•ด๊ฒฐํ•œ ๋’ค, v1.2 ํ‰๊ฐ€ ๊ณผ์ •์—์„œ ๋™์ผ ๊ฐ€์ค‘์น˜ koredact์กฐ์ฐจ ํ•จ๊ป˜ ํ•˜๋ฝ์‹œ์ผฐ๋˜ ๋Ÿฐํƒ€์ž„ ํ•œ๊ตญ์–ด ์กฐ์‚ฌยท์–ด๊ฐ„ ํ›„์ฒ˜๋ฆฌ ๊ฒฐํ•จ 3๊ฑด์„ v1.2.1์—์„œ ๊ฐ€์ค‘์น˜ ์žฌํ•™์Šต ์—†์ด ๋ฐ”๋กœ์žก์•˜์Šต๋‹ˆ๋‹ค:

๋‹จ๊ณ„ ํ•ด๊ฒฐ ๋Œ€์ƒ ๊ณผ์ œ koredact-int8 kestrel v1.0 kestrel v1.2 kestrel v1.2.1 (ํ˜„์žฌ) ํ•ต์‹ฌ ํ•ด๊ฒฐ ๋ฉ”์ปค๋‹ˆ์ฆ˜
v1.2 ์Šคํ‚ค๋งˆ ์ •ํ˜• ๋ณด์กฐ ๋ผ๋ฒจ ๋ณต์› (doc URL_ID ยท teacher_eval Soft ๋ˆ„๋ฝ) 9/9 ยท 0/172 0/9 ยท 12/172 9/9 ยท 0/172 9/9 (100%) ยท 0/172 10์ข… ์ •ํ˜• ๋ณด์กฐ BIO(9..28) ๋ณต์› (edu-v1+fmt, 29ํด๋ž˜์Šค)
v1.2 ํ•ฉ์„ฑ ๋„๋กœ๋ช… ์ฃผ์†Œ ", N์ธต" ์‰ผํ‘œ ๋ถ„์ ˆ (ADDRESS / road) 85.7% / 100% 73.2% / 72.0% 83.9% / 96.0% 83.9% / 96.0% ($p=1.00$) ์‰ผํ‘œ ์ธต์ˆ˜ยท๋†์–ด์ดŒ ๋„๋กœ๋ช… ํ•ฉ์„ฑ ๋ณด๊ฐ• (tight 57.1% โ†’ 76.8%)
v1.2 ํ•ฉ์„ฑ ๋ฌดํ•ด ๋ฌธํ•ญ(negยทamb_neg) ๊ต๊ณผ ์ผ๋ฐ˜๋ช…์‚ฌ ์˜คํƒ 16.4% (20/122) 27.0% (33/122) 14.8% (18/122) 14.8% (18/122) ๊ต๊ณผ ๊ฐœ๋…์–ดยท๊ต์‹ค ์‚ฌ๋ฌผ ์Œ์„ฑ ๋Œ€์กฐ ๊ฐ•ํ™” (X.echo 3/6 โ†’ 1/6)
v1.2.1 ํ›„์ฒ˜๋ฆฌ 1 1์Œ์ ˆ ์™ธ์ž ํ•œ๊ธ€ยทํ•œ์ž ์ธ๋ช… ์ ˆ์‚ญ (PERSON single / hanja) 93.7% / 100% 96.8% / 100% 74.6% / 90.0% 98.4% (62/63) / 100% ๋ถˆ์šฉ ๋Œ€๋ช…์‚ฌ(์ด/๊ทธ/์ €/์š”) ์™ธ 1๊ธ€์ž ํ•œ๊ธ€ยทCJK ํ•œ์ž PERSON ๋ณด์กด
v1.2.1 ํ›„์ฒ˜๋ฆฌ 2 ๋„์–ด ์“ด ์„ฑ๋ช… ์–ด๊ฐ„ ์ ˆ๋‹จ ("๋ฐ• ๋ฏธ๋ž‘", PERSON spaced) 90.0% (18/20) 20.0% (4/20) 90.0% (18/20) 95.0% (19/20) ๊ณต๋ฐฑ ์ œ์™ธ ์–ด๊ฐ„ ๊ธธ์ด ํ•˜ํ•œ ๊ณ„์‚ฐ ๋ฐ ๊ณต๋ฐฑ ๋’ค ํ˜ธ์นญ ์ ‘๋ฏธ(์”จ/๋‹˜/๊ตฐ/์–‘)๋งŒ ๋ถ„๋ฆฌ
v1.2.1 ํ›„์ฒ˜๋ฆฌ 3 ๋ฐ›์นจ ๋ฌด๊ด€ "-์ด" ์ ˆ๋‹จ ์ถฉ๋Œ (nickname / PERSON tight) 75.4% / 87.7% 88.5% / 84.9% 49.2% / 88.8% 90.2% / 92.0% ์„ ํ–‰ ์Œ์ ˆ ๋ฐ›์นจ(์ข…์„ฑ) ์กด์žฌ ์‹œ์—๋งŒ ์ด/์€/์„/๊ณผ/์œผ๋กœ ๊ณ„์—ด ์กฐ์‚ฌ ๋ถ„๋ฆฌ
์ข…ํ•ฉ ๊ฒฐ๊ณผ ํ•ต์‹ฌ ์ •๋‹ต ๋ˆ„๋ฝ ๋ฌธํ•ญ ๋น„์œจ / PERSON ์ „์ฒด ์žฌํ˜„์œจ 17.0% / 96.1% 12.3% / 95.7% 13.5% / 93.4% 8.4% (54/643) / 96.6% ๋ˆ„๋ฝ ๋ฌธํ•ญ ์ ˆ๋ฐ˜ ์ดํ•˜ ๊ฐ์ถ• (-8.6%p, ์–‘์ธก McNemar $p < 0.0001$)

2. ์•„ํ‚คํ…์ฒ˜ ยท ๋‹ค์ค‘ ์‹ ํ˜ธ ์ง€์‹ ์ฆ๋ฅ˜ ยท ๋ฌด์ฝ”๋“œ ๋ณด์ • ์„ค๊ณ„

  • ๊ต์œก ๋„๋ฉ”์ธ ์ •์ฑ… (D1=B ยท D2 ยท D5):
    • ๊ณต์ธยท์ž‘ํ’ˆ ์ธ๋ฌผ ๋น„๊ฐœ์ฒด ์œ ์ง€ (D1=B): ์ˆ˜์—… ์ž๋ฃŒ ์† ์—ญ์‚ฌ ์ธ๋ฌผยท์ž‘๊ฐ€ยท๋ฌธํ•™ ๋“ฑ์žฅ์ธ๋ฌผ("์œค๋™์ฃผ ์‹œ์ธ์˜ ์„œ์‹œ", "ํ™๊ธธ๋™์ „")์€ O๋กœ ์œ ์ง€ํ•˜๋˜, ๋™๋ช…์ด์ธ ์‹ค์ œ ํ•™์ƒ("2ํ•™๋…„ 3๋ฐ˜ ์œค๋™์ฃผ ํ•™์ƒ ์ƒ๋‹ด")์€ PERSON์œผ๋กœ ์žก๋„๋ก ๋™์ผ ์ธ๋ช… ์ˆ˜์—… ๋ฌธ๋งฅ(O) ๋Œ€ ํ•™์  ๋ฌธ๋งฅ(PERSON) ๋Œ€์กฐ ์Œ์„ ๊ฐ™์€ ๋ฏธ๋‹ˆ๋ฐฐ์น˜์— ๊ตฌ์„ฑํ–ˆ์Šต๋‹ˆ๋‹ค(teacher_eval ๊ณต์ธ ์˜ค์ฐจ๋‹จ 85.6% โ†’ 14.4%, holdout_w13c ๊ณต์ธ ์ˆ˜์—… ๋Œ€ํ™” ์˜ค์ฐจ๋‹จ 56.5% โ†’ 34.8%, $p = 0.0020$).
    • ์ „๊ตญ ๊ณต์  ๊ธฐ๊ด€ ๋ฐ ์„ฑ+์งํ•จ ๋น„๊ฐœ์ฒด (D2): ๊ต์œก๋ถ€ยท๋‚˜์ด์Šค(NEIS)ยทEBS ๋“ฑ ์ „๊ตญ ๋‹จ์œ„ ๊ณต์  ์ฒด๊ณ„์™€ "๊น€ ์„ ์ƒ๋‹˜"๋ฅ˜ 1๊ธ€์ž ์„ฑ+ํ˜ธ์นญ์€ O๋กœ ๋‘์–ด ๊ต์‚ฌ ํ”„๋กฌํ”„ํŠธ ์†์ƒ์„ ๋ง‰์Šต๋‹ˆ๋‹ค.
    • ๊ฒฐํ•ฉ ํ˜ธ์นญ ๊ฒฝ๊ณ„ ์ •๋ ฌ (D5): "์„œ์•„๋ง˜", "๋ฏผ์ค€ํ•™์ƒ"์—์„œ ์ด๋ฆ„ ๋ณธ์ฒด("์„œ์•„", "๋ฏผ์ค€")๋งŒ PERSON์œผ๋กœ ํ•™์Šตํ•˜๊ณ  ์กฐ์‚ฌ ๋ถ„๋ฆฌ BIO ์ •๋ ฌ(particle_aware)์„ ์ ์šฉํ•ด ๊ฒฐํ•ฉ ํ˜ธ์นญ ์žฌํ˜„์œจ 100.0% (22/22) ๋ฅผ ๋‹ฌ์„ฑํ–ˆ์Šต๋‹ˆ๋‹ค.
  • ์ „์—ญยท๊ตญ์†Œ ์–ดํ…์…˜ ๋ฐ RoPE ์ฃผํŒŒ์ˆ˜ ์ผ์น˜ 6์ธต ๋ณต์ œ (backbone.py):
    • transformers์˜ ModernBertAttention์€ layer_id % global_attn_every_n_layers == 0 ์ˆ˜์‹์œผ๋กœ ์ „์—ญ ์–ดํ…์…˜(rope_theta=160,000)๊ณผ ๊ตญ์†Œ ์Šฌ๋ผ์ด๋”ฉ ์–ดํ…์…˜(local_attention=128, rope_theta=10,000)์„ ๊ฒฐ์ •ํ•ฉ๋‹ˆ๋‹ค.
    • 6์ธต ํ•™์ƒ ๋ชจ๋ธ์€ global_attn_every_n_layers=2๋กœ ์„ค์ •ํ•˜์—ฌ [์ „์—ญ, ๊ตญ์†Œ, ์ „์—ญ, ๊ตญ์†Œ, ์ „์—ญ, ๊ตญ์†Œ] ๊ตฌ์กฐ๋ฅผ ๋งŒ๋“ค๊ณ , 22์ธต ๊ต์‚ฌ์˜ (0, 4, 9, 13, 18, 20)๋ฒˆ ์ธต์„ 1:1 ๋ณต์ œํ•ด trust_remote_code=False ํ‘œ์ค€ ๋กœ๋”์—์„œ๋„ RoPE inv_freq์™€ ์–ดํ…์…˜ ์ฐฝ์ด ์ •ํ™•ํžˆ ์ผ์น˜ํ•ฉ๋‹ˆ๋‹ค(๋น„์ž„๋ฒ ๋”ฉ ์—ฐ์‚ฐ ํŒŒ๋ผ๋ฏธํ„ฐ 110.94M โ†’ 30.70M, -72.3%).
  • ๋‹ค์ค‘ ์‹ ํ˜ธ ์ง€์‹ ์ฆ๋ฅ˜ (Stage 2):
    • ๋ผ๋ฒจ ํ•ฉ์„ฑ ๋ฌธ์„œ 28,000๊ฑด($\alpha=0.3$)๊ณผ ๋ฌด๋ผ๋ฒจ ํ•ฉ์„ฑ ๊ต์œก ๋ฌธ์„œ 100,000๊ฑด($\alpha=0.0$)์— ๋Œ€ํ•ด ์˜จ๋„ $\tau = 3.0$ ๋กœ์ง“ KL ๋ฐœ์‚ฐ๊ณผ 6๊ฐœ ๋Œ€์‘ ์ธต ์€๋‹‰ ํ‘œํ˜„ ์ฝ”์‚ฌ์ธ ์†์‹ค($\beta = 0.2$)์„ ๊ฒฐํ•ฉํ•ด NVIDIA L40S์—์„œ ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค(9,397์Šคํ…, ๋ˆ„์  ์œ ํšจ ํ† ํฐ 76.12M, 1.5์—ํญ step=4,025 ์ตœ์  ์ฑ„ํƒ).
  • ๋ฌด์ฝ”๋“œ ์‚ฌํ›„ ๋ณด์ • ๋ฐ ์„ ํƒ์  INT8 ์–‘์žํ™”:
    • dev_v1_2(3,000๊ฑด, ์œ ํšจ ํ† ํฐ 392,794๊ฐœ)์—์„œ ํ† ํฐ NLL์„ ์ตœ์†Œํ™”ํ•˜๋Š” ์˜จ๋„ $T = 1.428$(NLL 0.01555 โ†’ 0.01262, ECE 0.00229 โ†’ 0.00099, -56.6%)๊ณผ O ๋กœ์ง“ ๋งˆ์ง„ $\delta_O = +0.225$๋ฅผ ์„ ํ˜• ๋ถ„๋ฅ˜๊ธฐ ๊ฐ€์ค‘์น˜(W /= T, b /= T, b[0] += delta_O)์— ์ง์ ‘ ํก์ˆ˜ํ•ด ์ถ”๋ก  ์‹œ argmax๋งŒ์œผ๋กœ ๋ณด์ •์ด ์ ์šฉ๋ฉ๋‹ˆ๋‹ค(max |ฮ”logit| = 2.86e-6, argmax ์ผ์น˜ 100.0%).
    • onnxruntime ๋™์  ์–‘์žํ™” ์‹œ MatMul + Gemm + Gather(5๋งŒ ์–ดํœ˜ ์ž„๋ฒ ๋”ฉ ํฌํ•จ)๋งŒ ์ฑ„๋„๋ณ„ QInt8๋กœ ์–‘์žํ™”ํ•˜๊ณ  ๋งˆ์ง€๋ง‰ 29ํด๋ž˜์Šค classifier ๋…ธ๋“œ๋Š” FP32๋กœ ๋ณด์กดํ•ด 69.62 MB ์šฉ๋Ÿ‰๊ณผ FP32 ๋™๋“ฑ ์ •ํ™•๋„๋ฅผ ๋‹ฌ์„ฑํ–ˆ์Šต๋‹ˆ๋‹ค.

3. ๋ผ๋ฒจ ์Šคํ‚ค๋งˆ (edu-v1+fmt, 29ํด๋ž˜์Šค BIO) ๋ฐ ์ž…์ถœ๋ ฅ ๊ทœ๊ฒฉ

ID config.json BIO ๋ผ๋ฒจ ์„ค๋ช… ๋Ÿฐํƒ€์ž„ ์ •์ฑ… ๋งคํ•‘ ๋ฐ ๋น„๊ณ 
0 O ๋น„๊ฐœ์ฒดยท์ผ๋ฐ˜ ์–ดํœ˜ยท์กฐ์‚ฌ ๊ต๊ณผ ๊ณต์ธ/๋ฌธํ•™ ์ธ๋ฌผ(D1=B), ์ „๊ตญ ๊ณต์  ๊ธฐ๊ด€ยท๊น€ ์„ ์ƒ๋‹˜(D2)
1, 2 B-PERSON, I-PERSON ํ•™์ƒยทํ•™๋ถ€๋ชจยท๊ต์ง์› ์‹ค๋ช…, ์™ธ์žยทํ•œ์žยท์• ์นญยท๋กœ๋งˆ์žยท์™ธ๋ž˜ ์ธ๋ช… PERSON (ํ•ต์‹ฌ ๋น„์ •ํ˜•, ์กฐ์‚ฌยทํ˜ธ์นญ ์ œ์™ธ ๋ณธ์ฒด ์ •๋ ฌ)
3, 4 B-SCHOOL, I-SCHOOL ์œ ์น˜์›ยท์ดˆยท์ค‘ยท๊ณ ยทํŠน์ˆ˜ํ•™๊ตยท๋Œ€ํ•™๊ต ์ •์‹ ๊ต๋ช… ๋ฐ ์•ฝ์นญ(ํ•œ๋น›์ดˆ) SCHOOL (ํ•ต์‹ฌ ๋น„์ •ํ˜•, ์šฐ๋ฆฌ ํ•™๊ต ๋“ฑ ๋ณดํ†ต๋ช…์‚ฌ ์ œ์™ธ)
5, 6 B-ORG, I-ORG ์ง€์—ญ ํ•™์›ยท๊ต์Šต์†Œยท์ƒ๋‹ด์„ผํ„ฐยท๋ณ‘์˜์›ยท์ง€์—ญ ๋ณต์ง€๊ด€ยท๋ณดํ˜ธ์ž ์ง์žฅ๋ช… ORG (ํ•ต์‹ฌ ๋น„์ •ํ˜•, ์ „๊ตญ ๊ณต์  ๊ธฐ๊ด€ ์ œ์™ธ)
7, 8 B-ADDRESS, I-ADDRESS ๋„๋กœ๋ช…(์‰ผํ‘œ ์ธต์ˆ˜ ํฌํ•จ)ยท์ง€๋ฒˆยท์•„ํŒŒํŠธ ๋™/ํ˜ธ์ˆ˜ยท์ƒ์„ธ ๊ฑฐ์ฃผ์ง€ ์ฃผ์†Œ ADDRESS (ํ•ต์‹ฌ ๋น„์ •ํ˜•, ์‹œ/๊ตฌ ๋‹จ๋… ํ‘œ๊ธฐ ์ œ์™ธ)
9, 10 B-PHONE, I-PHONE ์œ ๋ฌด์„  ์ „ํ™”๋ฒˆํ˜ธยท๋ณ€์น™ ๊ตฌ๋ถ„์ž ๋ฒˆํ˜ธ PHONE (์ •ํ˜• 2์ฐจ ๋ฐฉ์–ด์„ )
11, 12 B-EMAIL, I-EMAIL ์ด๋ฉ”์ผ ์ฃผ์†Œ EMAIL (์ •ํ˜• 2์ฐจ ๋ฐฉ์–ด์„ )
13, 14 B-RRN, I-RRN ์ฃผ๋ฏผ๋“ฑ๋ก๋ฒˆํ˜ธยท์™ธ๊ตญ์ธ๋“ฑ๋ก๋ฒˆํ˜ธ RRN (์ •ํ˜• 2์ฐจ ๋ฐฉ์–ด์„ )
15, 16 B-PASSPORT, I-PASSPORT ์—ฌ๊ถŒ ๋ฒˆํ˜ธ DoRms Alias ๋Ÿฐํƒ€์ž„์—์„œ RRN ๊ณ ์œ„ํ—˜ ์‹ ๋ถ„์ฆ ๊ตฐ์œผ๋กœ ํ†ตํ•ฉ
17, 18 B-DRIVER_LICENSE, I-DRIVER_LICENSE ์šด์ „๋ฉดํ—ˆ๋ฒˆํ˜ธ DoRms Alias ๋Ÿฐํƒ€์ž„์—์„œ RRN ๊ณ ์œ„ํ—˜ ์‹ ๋ถ„์ฆ ๊ตฐ์œผ๋กœ ํ†ตํ•ฉ
19, 20 B-ACCOUNT, I-ACCOUNT ์€ํ–‰ ๊ณ„์ขŒ๋ฒˆํ˜ธ ACCOUNT (์ •ํ˜• 2์ฐจ ๋ฐฉ์–ด์„ )
21, 22 B-CARD, I-CARD ์นด๋“œ๋ฒˆํ˜ธ DoRms Alias ๋Ÿฐํƒ€์ž„์—์„œ ACCOUNT ๊ธˆ์œต ์‹๋ณ„์ž ๊ตฐ์œผ๋กœ ํ†ตํ•ฉ
23, 24 B-BRN, I-BRN ์‚ฌ์—…์ž๋“ฑ๋ก๋ฒˆํ˜ธ DoRms Alias ๋Ÿฐํƒ€์ž„์—์„œ ACCOUNT ๊ธˆ์œต ์‹๋ณ„์ž ๊ตฐ์œผ๋กœ ํ†ตํ•ฉ
25, 26 B-URL, I-URL ํ•™๊ต ์•Œ๋ฆผ์žฅยทํ•™๊ธ‰ ๋ฌธ์„œยท๊ฐœ์ธ ์‹๋ณ„ URL DoRms Alias ๋Ÿฐํƒ€์ž„์—์„œ URL_ID๋กœ ๋งคํ•‘ (doc 9/9 ํฌ์ฐฉ)
27, 28 B-CODE, I-CODE ์ž„์‹œ ๋น„๋ฐ€๋ฒˆํ˜ธยท์ดˆ๋Œ€์ฝ”๋“œยท์ธ์ฆ ์‹œํฌ๋ฆฟ DoRms Alias ๋Ÿฐํƒ€์ž„์—์„œ SECRET์œผ๋กœ ๋งคํ•‘
  • ํ•„์ˆ˜ ์ถ”๋ก  ๋ฐ ํ›„์ฒ˜๋ฆฌ ๊ทœ๊ฒฉ (v1.2.1):
    1. ํŒจ๋”ฉ ํ† ํฐ ID (pad_token_id = 49999): skt/A.X-Encoder-base ํ† ํฌ๋‚˜์ด์ €์˜ [PAD] ID๋Š” 0('[CLS]')์ด ์•„๋‹Œ 49999 ์ž…๋‹ˆ๋‹ค. ์ˆ˜๋™ ๋ฐฐ์น˜ ๊ตฌ์„ฑ ์‹œ ๋ฐ˜๋“œ์‹œ 49999๋กœ ํŒจ๋”ฉํ•˜์‹ญ์‹œ์˜ค.
    2. ์Šฌ๋ผ์ด๋”ฉ ์ฐฝ (W-400) ๋ฐ O -> I-* ๋ณต์›: ๊ธด ๋ฌธ์„œ๋Š” ์ž๋ฅด์ง€ ๋ง๊ณ  400์ž ์ฐฝ / 100์ž ๊ฒน์นจ(max_tokens=512, batch_size=8)์œผ๋กœ ์ถ”๋ก ํ•œ ๋’ค, ๋‹จ๋… I-TYPE ์‹œ์ž‘ ํ† ํฐ์„ B-TYPE์œผ๋กœ ์Šน๊ฒฉํ•ด ๋ณ‘ํ•ฉํ•ฉ๋‹ˆ๋‹ค.
    3. ํ•œ๊ตญ์–ด ์Œ์šด ์กฐ๊ฑด๋ถ€ ์กฐ์‚ฌ ๋ถ„๋ฆฌ (v1.2.1 ๊ถŒ์žฅ): PERSON ์ŠคํŒฌ ์šฐ์ธก์— ๊ฒฐํ•ฉ๋œ ์กฐ์‚ฌ๋ฅผ ์ •๋ฆฌํ•  ๋•Œ ์€/์„/๊ณผ/์œผ๋กœ/์ด/์ด๊ฐ€/์ด๋Š”/์ด๋ฅผ/์ด์™€/์ด์˜/์ด๋ž‘ ๋“ฑ ๋ฐ›์นจ ์˜์กด ์กฐ์‚ฌ๋Š” ์ง์ „ ํ•œ๊ธ€ ์Œ์ ˆ์— ๋ฐ›์นจ(์ข…์„ฑ)์ด ์žˆ์„ ๋•Œ๋งŒ ์ œ๊ฑฐํ•˜๊ณ , ๊ณต๋ฐฑ ํฌํ•จ ์„ฑ๋ช…์€ ๊ณต๋ฐฑ์„ ์ œ์™ธํ•œ ์Œ์ ˆ ์ˆ˜ ๊ธฐ์ค€์œผ๋กœ ์ตœ์†Œ ์–ด๊ฐ„ ๊ธธ์ด๋ฅผ ํŒ์ •ํ•˜์‹ญ์‹œ์˜ค.

4. ์„œ๋น™ ๋ฐ ํ˜ธ์ถœ ์˜ˆ์ œ

1. ONNX Runtime CPU ์ถ”๋ก  (๊ถŒ์žฅ, onnx/model_int8.onnx)

import json
import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from tokenizers import Tokenizer

REPO_ID = "mulkzz/kestrel-edu-ner-6l"
REV = "v1.2.1"  # ๋˜๋Š” ๊ธฐ๋ณธ ๋ธŒ๋žœ์น˜ "main"
cfg = json.loads(open(hf_hub_download(REPO_ID, "config.json", revision=REV), encoding="utf-8").read())
tok = Tokenizer.from_file(hf_hub_download(REPO_ID, "tokenizer.json", revision=REV))
tok.no_truncation()
tok.no_padding()

so = ort.SessionOptions()
so.intra_op_num_threads = 2
so.inter_op_num_threads = 1
session = ort.InferenceSession(
    hf_hub_download(REPO_ID, "onnx/model_int8.onnx", revision=REV),
    so,
    providers=["CPUExecutionProvider"],
)

text = "๋ณ„๋น›๊ณ  ๋ฐ• ๋ฏธ๋ž‘ ํ•™์ƒ๊ณผ ์ดˆ๋กฑ์ด๊ฐ€ ํ•ด์˜ค๋ฆ„์ƒ๋‹ด์„ผํ„ฐ(๋ฌผ์•ˆ๊ฐœ๋กœ 58, 1์ธต)์— ๋ฐฉ๋ฌธํ–ˆ๋‹ค."
enc = tok.encode(text)
pad_id = int(cfg.get("pad_token_id", 49999))
ids = np.full((1, len(enc.ids)), pad_id, dtype=np.int64)
mask = np.ones((1, len(enc.ids)), dtype=np.int64)
ids[0, :] = enc.ids

logits = session.run(None, {"input_ids": ids, "attention_mask": mask})[0][0]
pred_ids = logits.argmax(axis=-1).tolist()
id2label = {int(k): v for k, v in cfg["id2label"].items()}

spans = []
cur_type, cur_s, cur_e = None, 0, 0
for tid, (s, e), sp in zip(pred_ids, enc.offsets, enc.special_tokens_mask):
    tag = "O" if sp or e <= s else id2label[tid]
    if tag == "O":
        if cur_type:
            spans.append((cur_type, cur_s, cur_e, text[cur_s:cur_e]))
            cur_type = None
        continue
    prefix, etype = tag.split("-", 1)
    if prefix == "B" or cur_type != etype:
        if cur_type:
            spans.append((cur_type, cur_s, cur_e, text[cur_s:cur_e]))
        cur_type, cur_s, cur_e = etype, s, e
    else:
        cur_e = e
if cur_type:
    spans.append((cur_type, cur_s, cur_e, text[cur_s:cur_e]))

print(spans)

2. Transformers PyTorch ์ถ”๋ก  (model.safetensors)

import torch
from transformers import AutoModelForTokenClassification, AutoTokenizer

REPO_ID = "mulkzz/kestrel-edu-ner-6l"
REV = "v1.2.1"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision=REV)
model = AutoModelForTokenClassification.from_pretrained(REPO_ID, revision=REV).eval()

text = "์œค๋™์ฃผ ์‹œ์ธ์˜ ์„œ์‹œ๋ฅผ ๋ฐฐ์šด ๋’ค ๋™๋ช…์ด์ธ ์œค๋™์ฃผ ํ•™์ƒ์˜ ๋…์„œ ๊ธฐ๋ก์„ ์ž‘์„ฑํ–ˆ๋‹ค."
inputs = tokenizer(text, return_offsets_mapping=True, return_tensors="pt")
offsets = inputs.pop("offset_mapping")[0].tolist()

with torch.no_grad():
    pred = model(**inputs).logits[0].argmax(dim=-1).tolist()

for pid, (s, e) in zip(pred, offsets):
    if e > s and pid != 0:
        print(model.config.id2label[pid], text[s:e], (s, e))

5. ์‚ฌ์šฉ ๋ฒ”์œ„ ยท ์ž”์—ฌ ํ•œ๊ณ„ ยท ์™ธ๋ถ€ ํ‰๊ฐ€ ๋ฐ์ดํ„ฐ ๊ณ ์ง€ ยท ๋ผ์ด์„ ์Šค

  • ๋ณด์กฐ ๋„๊ตฌ ํ•œ๊ณ„ ๊ณ ์ง€: ์ด ๋ชจ๋ธ์€ ๊ต์‚ฌ์˜ ๊ฐ€๋ช…ํ™” ์ž‘์—…์„ ๋•๋Š” ๊ธฐ์ˆ ์  ๋ณด์กฐ ๊ณ„์ธต์ด๋ฉฐ, ๋ชจ๋“  ๊ฐœ์ธ์ •๋ณด์˜ ์™„์ „ํ•œ ํƒ์ง€๋‚˜ ๋ฒ•์  ๋ฉด์ฑ…์„ ๋ณด์ฆํ•˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค. ๊ฒฐ์ •๋ก ์  ์ •๊ทœ์‹ยท์ฒดํฌ์„ฌ ๊ทœ์น™ ์—”์ง„ ๋ฐ ์‚ฌ๋žŒ์˜ ์ตœ์ข… ๊ฒ€ํ† ์™€ ํ•จ๊ป˜ ์‚ฌ์šฉํ•˜์‹ญ์‹œ์˜ค.
  • v1.2.1์˜ ์•Œ๋ ค์ง„ ์ž”์—ฌ ํ•œ๊ณ„ ๋ฐ ๋ฐฐํฌ ์›์น™ (ํˆฌ๋ช… ๊ณต๊ฐœ):
    1. dorms-pii-ner-bench v1.1์—์„œ ๋ฌดํ•ด ๋ฌธํ•ญ(negยทamb_neg) ์˜คํƒ๋ฅ ์€ 14.8%๋กœ koredact(16.4%)๋ณด๋‹ค ๋‚ฎ์œผ๋‚˜, ์–‘์„ฑ ๋ฌธํ•ญ ๋‚ด ๋ถ€์ˆ˜ ์˜คํƒ์„ ํฌํ•จํ•œ ์ „์ฒด 848๋ฌธํ•ญ 1๋งŒ ์ž๋‹น ์˜คํƒ(20.3 ๋Œ€ 13.4, ๋‹จ๋… 12.7 ๋Œ€ 5.8), ๋ฐ˜๋ก€ word ์ ์ค‘ 4/89 (4.5%)(์ด์Šฌ, ๋‹จ๋น„), ๋ณดํ†ต๋ช…์‚ฌ ๋™ํ˜•์ด์˜ ์ธ๋ช…(amb ๊ฐˆ๋ž˜ 83.3% ๋Œ€ 94.7%)์—์„œ ์„ธ๋ถ€ ๊ฒฉ์ฐจ๊ฐ€ ๋‚จ์•„ ์žˆ์Šต๋‹ˆ๋‹ค.
    2. ์™ธ๋ถ€ ์ผ๋ฐ˜ ๋„๋ฉ”์ธ ์ฝ”ํผ์Šค Hard ๋‹จ๊ณ„ ๋ˆ„๋ฝ๋ฅ ์€ 3๊ฐœ ์…‹ ๋ชจ๋‘ koredact๋ณด๋‹ค ๋‚ฎ์œผ๋‚˜(townboy_eval 25.6% โ†’ 24.3%, townboy_dev 8.0% โ†’ 6.7%, kdpii_test 43.8% โ†’ 40.8%), ๋น„๊ต์œก ์ผ๋ฐ˜ ๊ธˆ์œตยท์—ฌ๊ถŒ ๋ฒˆํ˜ธ Soft ๋ˆ„๋ฝ(townboy_eval 0.8% 5/605, kdpii_test 34.8%)๊ณผ ๋ฌธ๋งฅ ์—†์ด ์—ฌ๋Ÿฌ ํ„ด์— ์ชผ๊ฐ  ์ˆซ์ž(w13c split)๊ฐ€ ๋‚จ์•„ ์žˆ์Šต๋‹ˆ๋‹ค. ์ด์— ๋”ฐ๋ผ ์‚ฌ์ „ ๋“ฑ๋ก 4์ค‘ ๊ฒŒ์ดํŠธ ์›์น™์„ ์—„๊ฒฉํžˆ ์ ์šฉํ•ด ์˜จ๋””๋ฐ”์ด์Šค alias-bridge ๊ธฐ๋ณธ ๋ชจ๋ธ ๋ฐ ์„œ๋ฒ„ ๋น„๋™๊ธฐ Shadow ๋ชจ๋ธ๋กœ ์šฐ์„  ์šด์˜ํ•˜๋ฉฐ ์„œ๋ฒ„ ์ฃผ ๊ธฐ๋ณธ๊ฐ’ ๊ต์ฒด ์—ฌ๋ถ€๋Š” ํ˜„์žฅ Shadow ์ง‘๊ณ„ ํ›„ ๊ฒฐ์ •ํ•ฉ๋‹ˆ๋‹ค.
    3. ๋™์ผ ๋ฌธ์„œ ๋‚ด ๋™๋ช…์ด์ธ ๋ฒˆํ˜ธ ๋ถ„๋ฆฌ ๋ฐ ์•”์‹œ์  ์ •ํ™ฉ์„ ํ†ตํ•œ ์‹ ์› ํŠน์ •์€ ํ† ํฐ ๋ถ„๋ฅ˜ NER์˜ ๋ฒ”์œ„ ๋ฐ–์ž…๋‹ˆ๋‹ค.
  • ์™ธ๋ถ€ ํ‰๊ฐ€ ๋ฐ์ดํ„ฐ ์ถœ์ฒ˜ ๋ฐ ๋ผ์ด์„ ์Šค:
    • townboy/korean-pii-dataset (076419f07f7073b6b28cdfdb6fbd8b75e1278053, CC BY 4.0, ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ์…‹)
    • KDPII ์‹œํ—˜ ๋ถ„ํ•  (Zenodo 10968609, CC BY 4.0, ์—ฐ์„ธ๋Œ€ํ•™๊ต ์—ฐ๊ตฌ์ง„, IEEE Access, vol. 12, 2024, doi:10.1109/ACCESS.2024.3461804) โ€” ์˜ค์ง ์ฝ๊ธฐ ์ „์šฉ ์ง‘๊ณ„ ํ‰๊ฐ€์—๋งŒ ์‚ฌ์šฉํ–ˆ์œผ๋ฉฐ, ์›๋ฌธ ํ–‰์ด๋‚˜ ๋ถ€๋ถ„ ๋ฌธ์ž์—ด์€ ๊ฐ€์ค‘์น˜ยท์ฝ”๋“œยทfigures/data/ ์–ด๋””์—๋„ ํฌํ•จ๋˜์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
  • ๋ผ์ด์„ ์Šค: ๊ธฐ๋ฐ˜ ๋ชจ๋ธ(skt/A.X-Encoder-base)๊ณผ ๋™์ผํ•˜๊ฒŒ Apache-2.0 ์„ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค(๊ท€์† ๋ฐ ๋ณ€๊ฒฝ ๊ณ ์ง€๋Š” NOTICE ์ฐธ์กฐ).

6. ํ•™์Šตยท์ฆ๋ฅ˜ ๋™์—ญํ•™ ๋ฐ ๋ถ„ํฌ ๋‚ด(dev_v1_2 ยท ๋‹จ๋ฐœ itest_v1_2) ์ƒ์„ธ ์ง€ํ‘œ

Fig. 3 | v1.2 Stage 1 teacher fine-tuning, Stage 2 6L distillation dynamics, and itest_v1_2

Fig. 3 | Stage 1 22์ธต ๊ต์‚ฌ ๋ฏธ์„ธ์กฐ์ • ยท Stage 2 6์ธต ํ•™์ƒ ๋‹ค์ค‘ ์‹ ํ˜ธ ์ฆ๋ฅ˜ ๋™์—ญํ•™ ๋ฐ ๋ฏธ๋…ธ์ถœ ๋‹จ๋ฐœ itest_v1_2 ๊ฒ€์ฆ. a, 22์ธต ๊ต์‚ฌ(3,573์Šคํ…, 14.88M ํ† ํฐ, 29 BIO)์˜ ์Šคํ…๋ณ„ ํ•™์Šต ์†์‹ค๊ณผ 0.5์—ํญ ์ฃผ๊ธฐ dev_v1_2 ์†์‹ค(1.5์—ํญ step=1,783 ์ตœ์  ์ฑ„ํƒ, selection_score=1.7843). b, 6์ธต ํ•™์ƒ(9,397์Šคํ…, 76.12M ํ† ํฐ, ๋ผ๋ฒจ 28k + ๋ฌด๋ผ๋ฒจ 100k)์˜ ์ด ์ฆ๋ฅ˜ ์†์‹ค, $\tau^2\mathrm{KL}$ ๋ฐœ์‚ฐ, ์€๋‹‰ ์ฝ”์‚ฌ์ธ ์†์‹ค($1-\cos$), dev_v1_2 ์†์‹ค(1.5์—ํญ step=4,025 ์ตœ์  ์ฑ„ํƒ, selection_score=1.7481). c, ์ •๊ทœํ™” ์ง„ํ–‰๋ฅ  ๋Œ€๋น„ ๊ฒฝ์‚ฌ $L_2$ ๋…ธ๋ฆ„($|g|_2$) ์ˆ˜๋ ด ์ถ”์ด. d, ๋‚ด๋ถ€ dev_v1_2(3,000๊ฑด) ๋ฐ ๋ฏธ๋…ธ์ถœ ํ‹€ยท์—”ํ„ฐํ‹ฐ ํ’€๋กœ ๊ตฌ์„ฑ๋œ ๋‹จ๋ฐœ itest_v1_2(3,000๊ฑด)์˜ 22์ธต ๊ต์‚ฌ ๋Œ€ 6์ธต ๋ณด์ • ํ•™์ƒ ๋น„๊ต(ํ•ฉ์„ฑ ๋ถ„ํฌ ๋‚ด ์ง€ํ‘œ์ด๋ฏ€๋กœ ์‹ค์ œ ์ผ๋ฐ˜ํ™” ์„ฑ๋Šฅ์€ ์œ„ 1์ ˆ์˜ dorms-pii-ner-bench v1.1์„ ๊ธฐ์ค€์œผ๋กœ ํ•ด์„ํ•ด์•ผ ํ•จ).

1. Stage 2 ์ฆ๋ฅ˜ ์—ํญ๋ณ„ ํ‰๊ฐ€ ์ถ”์ด (dev_v1_2, n = 3,000)

ํ‰๊ฐ€ ์‹œ์  (epoch) ๊ธ€๋กœ๋ฒŒ ์Šคํ… ๋ˆ„์  ์œ ํšจ ํ† ํฐ dev_loss selection_score macro_core_recall formal_recall public_person_fp ๋น„๊ณ 
0.5 1,343 10.92M 0.01486 1.7376 98.21% 100.00% 27.39% ์ดˆ๊ธฐ ์ˆ˜๋ ด
1.0 2,682 21.72M 0.02027 1.7070 98.02% 100.00% 27.39% no_imp=1
1.5 4,025 32.53M 0.01548 1.7481 98.27% 100.00% 5.61% ์ตœ์ข… ์ฑ„ํƒ (*BEST*)
2.0 5,366 43.45M 0.01644 1.7363 98.31% 100.00% 12.87% no_imp=1
2.5 6,706 54.24M 0.01465 1.7358 98.36% 100.00% 10.23% no_imp=2
3.0 8,052 65.19M 0.01648 1.7216 98.46% 100.00% 16.50% no_imp=3
3.5 9,397 76.12M 0.01571 1.7227 98.41% 100.00% 15.51% ์กฐ๊ธฐ ์ข…๋ฃŒ (patience=4)

2. ๋ฏธ๋…ธ์ถœ ๋‹จ๋ฐœ itest_v1_2 (n = 3,000) ๊ต์‚ฌ ๋Œ€ ํ•™์ƒ ์„ฑ์  (W-400, pipeline ๋ทฐ)

๋ชจ๋ธ (itest_v1_2, ๋‹จ 1ํšŒ ์ธก์ •) macro_core_recall overall_core_recall formal_recall (1,817 span) PERSON SCHOOL ORG ADDRESS overall_tight innocent_fp_item (NER)
kestrel-edu-ner-22l (22์ธต ๊ต์‚ฌ) 97.84% 96.79% 100.00% 96.60% 100.00% 94.90% 99.85% 97.74% 4.79% (amb_neg 1.28%)
kestrel-edu-ner-6l (6์ธต ๋ณด์ • ํ•™์ƒ) 97.44% 95.77% 100.00% 95.44% 99.56% 94.75% 100.00% 97.37% 1.14%

3. PyTorch FP32 ๋Œ€ ONNX ๋ณ€ํ˜•๋ณ„ ์ˆ˜์น˜ ๋™๋“ฑ์„ฑ (dev_v1_2 ์‹ค์ œ ์ฐฝ 200๊ฑด / 11,713 ์œ ํšจ ํ† ํฐ)

| ๊ทธ๋ž˜ํ”„ ๋ณ€ํ˜• | ํŒŒ์ผ ๊ฒฝ๋กœ | ํฌ๊ธฐ | max |ฮ”logit| | p99 |ฮ”logit| | argmax ์ผ์น˜์œจ | dev_v1_2 3,000๊ฑด pipeline Macro Recall | |---|---|---:|---:|---:|---:|---:| | PyTorch FP32 (safetensors) | model.safetensors | 276.42 MB | ๊ธฐ์ค€ (0) | ๊ธฐ์ค€ (0) | 100% | 98.44% | | ONNX FP32 | onnx/model.onnx | 276.60 MB | 7.49 ร— 10โปโต | 7.27 ร— 10โปโถ | 100.0000% | 98.44% | | ONNX INT8-PC-Gather (์ฃผ๋ ฅ) | onnx/model_int8.onnx | 69.62 MB | 0.7832 | 0.0660 | 99.9573% | 98.45% (McNemar $p = 1.0000$) |

Downloads last month
69
Safetensors
Model size
69.1M params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for mulkzz/kestrel-edu-ner-6l

Quantized
(4)
this model

Paper for mulkzz/kestrel-edu-ner-6l