KJH-Judge-4B-ko / README.md
coderecode95's picture
Update README.md
01c9adf verified
|
Raw History Blame Contribute Delete
30.1 kB
---
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3-4B/blob/main/LICENSE
base_model: Qwen/Qwen3-4B
base_model_relation: finetune
language:
- ko
library_name: transformers
tags:
- korean
- classification
- natural-language-inference
- function-calling
- tool-use
- calibrated
- multiple-choice
model-index:
- name: KJH-Judge-4B-ko
results:
- task:
type: text-classification
name: Sentiment classification
dataset:
name: NSMC
type: e9t/nsmc
split: test
metrics:
- type: accuracy
value: 85.7
name: accuracy (0-shot
n=1000): null
- task:
type: text-classification
name: Natural language inference
dataset:
name: KorNLI (XNLI)
type: kakaobrain/kor_nli
config: xnli
split: test
metrics:
- type: accuracy
value: 74.5
name: accuracy (0-shot
n=1000): null
- task:
type: text-classification
name: Paraphrase detection (binary STS)
dataset:
name: KLUE-STS
type: klue/klue
config: sts
split: validation
metrics:
- type: accuracy
value: 85.9
name: accuracy (0-shot
n=519): null
- task:
type: text-classification
name: Hate speech detection
dataset:
name: Korean UnSmile
type: smilegate-ai/kor_unsmile
split: valid
metrics:
- type: accuracy
value: 82.1
name: accuracy (0-shot
n=1000): null
- task:
type: multiple-choice
name: Knowledge QA
dataset:
name: KMMLU
type: HAERAE-HUB/KMMLU
split: test
metrics:
- type: accuracy
value: 42.6
name: accuracy (0-shot
n=990): null
- task:
type: multiple-choice
name: Korean knowledge QA
dataset:
name: HAE-RAE Bench 1.0
type: HAERAE-HUB/HAE_RAE_BENCH_1.0
split: test
metrics:
- type: accuracy
value: 56.1
name: accuracy (0-shot
n=983): null
---
# KJH-Judge-4B-ko
> [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)๋ฅผ ๋ณด๊ณ  ํŒ์ • ๋ชจ๋ธ์„ ์ง์ ‘ ๋งŒ๋“ค์–ด ๋ณธ **๊ฐœ์ธ ํ•™์Šต ํ”„๋กœ์ ํŠธ**์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์ˆ˜์น˜๋Š” ๋ชจ๋‘ ์ง์ ‘ ์žฐ ๊ฒฐ๊ณผ์ด๋ฉฐ, ์‹ค์„œ๋น„์Šค์—์„œ ๊ฒ€์ฆํ•œ ๋ชจ๋ธ์€ ์•„๋‹™๋‹ˆ๋‹ค.
**์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณจ๋ผ, ๋ณด์ •๋œ ํ™•๋ฅ ๊ณผ ํ•จ๊ป˜ ๋Œ๋ ค์ฃผ๋Š” ํ•œ๊ตญ์–ด ํŒ์ • ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.** ๊ฐ์ •ยท์ฃผ์ œ ๋ถ„๋ฅ˜, ๋ฌธ์žฅ ๊ด€๊ณ„(ํ•จ์˜ยท๋ชจ์ˆœ), ํ˜์˜ค ํ‘œํ˜„ ๊ฒ€์ˆ˜, ์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ ์„ ํƒ, ์—์ด์ „ํŠธ์˜ ๋„๊ตฌ ํ˜ธ์ถœ ํŒ๋‹จ(๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€) ๊ฐ™์€ ์ผ์— ์”๋‹ˆ๋‹ค. ๋‹ค๋ฅธ ๋ชจ๋ธ์˜ ๋‹ต๋ณ€์„ ์ฑ„์ ํ•˜๋Š” LLM-as-a-judge ๋ชจ๋ธ์ด ์•„๋‹™๋‹ˆ๋‹ค.
> **English summary.** A personal learning project: a Korean decision model fine-tuned from Qwen3-4B. Given a task description, an item, and 2โ€“26 options, it returns the chosen option with **calibrated probabilities** from a single forward pass (no text generation, ~tens of ms per decision on a GPU). Good for classification, NLI, moderation, policy actions, and tool-call routing. It does not generate text, extract spans, or do numeric reasoning.
![ํ•™์Šตํ•˜์ง€ ์•Š์€ ํ•œ๊ตญ์–ด ๋ฒค์น˜๋งˆํฌ](assets/benchmark.png)
## ์ด ๋ชจ๋ธ์˜ ํŠน์ง•
- **๋ณด๊ธฐ ๊ณ ๋ฅด๊ธฐ๋งŒ ํ•ฉ๋‹ˆ๋‹ค.** ๊ธ€์„ ์ƒ์„ฑํ•˜์ง€ ์•Š๊ณ  ํ”„๋กฌํ”„ํŠธ๋ฅผ ํ•œ ๋ฒˆ ๊ณ„์‚ฐํ•œ ๋’ค, ๋ณด๊ธฐ ๊ธฐํ˜ธ(A, B, โ€ฆ) ํ† ํฐ์˜ ์ ์ˆ˜๋งŒ ์ฝ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ๋น ๋ฅด๊ณ (GPU ์„œ๋ฒ„ ์•ˆ ๊ณ„์‚ฐ ๊ธฐ์ค€ ํŒ์ • ํ•œ ๋ฒˆ ์•ฝ 90ms), ๋‹ต์ด ํ•ญ์ƒ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค.
- **ํ™•๋ฅ ์ด ๋ณด์ •๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.** ๊ฒ€์ฆ์…‹์œผ๋กœ ๋งž์ถ˜ ์˜จ๋„(1.72)๋กœ ๋ณด์ •ํ•ด, Korean JEV Benchmark์—์„œ ๊ธฐ๋Œ€ ๋ณด์ • ์˜ค์ฐจ(ECE)๊ฐ€ 0.027์ž…๋‹ˆ๋‹ค. โ€œํ™•์‹  70% ๋ฏธ๋งŒ์ด๋ฉด ์‚ฌ๋žŒ์—๊ฒŒ ๋„˜๊ธฐ๊ธฐโ€ ๊ฐ™์€ ๊ธฐ์ค€์„ ๊ฑธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ ํ•™์Šตํ•˜์ง€ ์•Š์€ ์ถœ์ฒ˜์˜ ๋ฐ์ดํ„ฐ์—์„œ๋Š” ํ‰๊ท  ํ™•์‹ ์ด ์ •ํ™•๋„๋ณด๋‹ค ์ตœ๋Œ€ 10%p ์ •๋„ ๋†’๊ฒŒ ๋‚˜์™”์œผ๋‹ˆ, ๊ธฐ์ค€๊ฐ’์€ ์“ฐ๋ ค๋Š” ๋ฐ์ดํ„ฐ์—์„œ ๋‹ค์‹œ ๋งž์ถ”๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
- **๋ณด๊ธฐ ์ˆœ์„œ์— ํ”๋“ค๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.** ๋ณด๊ธฐ๋ฅผ ์ •ํ™•ํžˆ ๊ฑฐ๊พธ๋กœ ๋†“๊ณ  ๋‹ค์‹œ ๋ฌผ์–ด๋„ 97.9%๊ฐ€ ๊ฐ™์€ ๋‹ต์ž…๋‹ˆ๋‹ค.
## ๋น ๋ฅด๊ฒŒ ์‹œ์ž‘ํ•˜๊ธฐ
```bash
pip install torch "transformers>=5" accelerate
```
```python
from transformers import pipeline
judge = pipeline(model="coderecode95/KJH-Judge-4B-ko", trust_remote_code=True,
device=0, dtype="bfloat16") # GPU๊ฐ€ ์—†์œผ๋ฉด device, dtype ๋ฅผ ๋นผ์„ธ์š” (๋งค์šฐ ๋А๋ฆผ)
```
์“ฐ๋Š” ๋ฐฉ๋ฒ•์€ ์„ธ ๊ฐ€์ง€์ด๊ณ , **๊ฒฐ๊ณผ๋Š” ๋ชจ๋‘ ๊ฐ™์€ ๋ชจ์–‘**์ž…๋‹ˆ๋‹ค.
```python
{'choice': '๊ธ์ •', 'confidence': 0.97, 'probabilities': {'๊ธ์ •': 0.97, '๋ถ€์ •': 0.03}}
# ๊ณ ๋ฅธ ๋ณด๊ธฐ ๊ทธ ๋ณด๊ธฐ์˜ ํ™•๋ฅ (๋ณด์ •๋จ) ๋ณด๊ธฐ๋ณ„ ํ™•๋ฅ 
```
#### 1. ์ž์ฃผ ์“ฐ๋Š” ํŒ์ • โ€” ๋ฉ”์„œ๋“œ ํ•˜๋‚˜๋กœ
ํ•™์Šต ๋•Œ์™€ ๊ฐ™์€ ์ง€์นจยท๋ณด๊ธฐ ๋ฌธ๊ตฌ๊ฐ€ ์ž๋™์œผ๋กœ ๋“ค์–ด๊ฐ€์„œ, ๋”ฐ๋กœ ํ”„๋กฌํ”„ํŠธ๋ฅผ ์“ธ ํ•„์š”๊ฐ€ ์—†๊ณ  ์ •ํ™•๋„๋„ ๊ฐ€์žฅ ์ž˜ ๋‚˜์˜ต๋‹ˆ๋‹ค.
```python
judge.sentiment("๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”") # โ†’ ๊ธ์ •
judge.nli("๋ณด์ฆ๊ธˆ์€ ํ‡ด๊ฑฐ ํ›„ 30์ผ ์ด๋‚ด ๋ฐ˜ํ™˜ํ•œ๋‹ค.", "๋ณด์ฆ๊ธˆ์€ ํ‡ด๊ฑฐ 40์ผ ๋’ค์— ๋ฐ›๋Š”๋‹ค.") # โ†’ contradiction
judge.next_action("๊ฐ•๋‚จ์—ญ์—์„œ ์„œ์šธ์—ญ๊นŒ์ง€ ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜", tools) # โ†’ CALL:call_taxi
```
| ๋ฉ”์„œ๋“œ | ํŒ์ • | ๊ณ ๋ฅด๋Š” ๊ฐ’ |
|---|---|---|
| `sentiment(๋ฌธ์žฅ)` | ๊ฐ์ • | ๊ธ์ • / ๋ถ€์ • |
| `topic(๋‰ด์Šค ์ œ๋ชฉ)` | ๋‰ด์Šค ์ฃผ์ œ | ์‚ฌํšŒ / IT๊ณผํ•™ / ์„ธ๊ณ„ / ์ƒํ™œ๋ฌธํ™” / ์ •์น˜ / ์Šคํฌ์ธ  / ๊ฒฝ์ œ |
| `hate(๋ฌธ์žฅ)` | ํ˜์˜คยท๋น„ํ•˜ ํ‘œํ˜„ | ๋น„ํ˜์˜ค / ํ˜์˜ค |
| `offensive(๋Œ“๊ธ€, news_title=None)` | ๋Œ“๊ธ€ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„ | ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค |
| `nli(์ „์ œ, ๊ฐ€์„ค)` | ๋ฌธ์žฅ ๊ด€๊ณ„ | entailment / contradiction / neutral |
| `similar(๋ฌธ์žฅ1, ๋ฌธ์žฅ2)` | ๊ฐ™์€ ๋œป์ธ์ง€ | ๊ฐ™์Œ / ๋‹ค๋ฆ„ |
| `yes_no(์งˆ๋ฌธ, ์ง€๋ฌธ)` | ์ง€๋ฌธ ๊ทผ๊ฑฐ ์˜ˆ/์•„๋‹ˆ์˜ค | ์˜ˆ / ์•„๋‹ˆ์˜ค |
| `same_meaning(๋‹จ์–ด, ๋ฌธ๋งฅ1, ๋ฌธ๋งฅ2)` | ๋‹จ์–ด ๋œป์ด ๊ฐ™์€์ง€ | ๊ฐ™์Œ / ๋‹ค๋ฆ„ |
| `reading(์ง€๋ฌธ, ์งˆ๋ฌธ, ๋ณด๊ธฐ)` | ๋…ํ•ด ๊ฐ๊ด€์‹ | ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜ |
| `next_action(๋Œ€ํ™”, ๋„๊ตฌ ๋ชฉ๋ก)` | ์—์ด์ „ํŠธ ๋‹ค์Œ ํ–‰๋™ | CALL:<๋„๊ตฌ> / ASK / RESPOND |
| `policy_action(์ •์ฑ…, ๊ณ ๊ฐ ์š”์ฒญ)` | ์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ | ANSWER / ASK_CLARIFICATION / ASK_CONFIRMATION / CALL_TOOL / CALL_MULTIPLE_TOOLS / ESCALATE / REFUSE |
๋ฌธ์žฅ ํ•˜๋‚˜๋งŒ ๋ฐ›๋Š” ๋ฉ”์„œ๋“œ(`sentiment`, `topic`, `hate`, `nli` ๋“ฑ)๋Š” **๋ฆฌ์ŠคํŠธ๋ฅผ ๋„ฃ์œผ๋ฉด ์—ฌ๋Ÿฌ ๊ฐœ๋ฅผ ํ•œ ๋ฒˆ์—** ํŒ์ •ํ•ฉ๋‹ˆ๋‹ค.
#### 2. ๋‚ด ํŒ์ • ๋งŒ๋“ค๊ธฐ โ€” JEV ํ˜•์‹
[Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)ยทJEV API์™€ ๊ฐ™์€ ๋ชจ์–‘์ž…๋‹ˆ๋‹ค. **์ƒํ™ฉ(`state`) ํ•˜๋‚˜์— ์ด๋ฆ„ ๋ถ™์ธ ์งˆ๋ฌธ ์—ฌ๋Ÿฌ ๊ฐœ**๋ฅผ ํ•œ ๋ฒˆ์— ๋ฌป๊ณ , ๋ณด๊ธฐ์—๋Š” **๋‚ด๊ฐ€ ์“ธ ID**๋ฅผ ๋ถ™์ž…๋‹ˆ๋‹ค. ๊ฒฐ๊ณผ๋Š” ๊ทธ ID๋กœ ๋Œ์•„์˜ต๋‹ˆ๋‹ค.
```python
judge.decide(
state={"๊ณ ๊ฐ ๋ฌธ์˜": "์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ์ฒ˜๋ฆฌํ•ด ์ฃผ์„ธ์š”."},
questions={
"๋ถ€์„œ": {
"instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.",
"criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ"},
},
"๊ธด๊ธ‰": {
"instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๋Š”์ง€ ํŒ๋‹จํ•œ๋‹ค.",
"criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ์š”๊ตฌ", "normal": "์ผ๋ฐ˜ ๋ฌธ์˜"},
},
},
)
# {'๋ถ€์„œ': {'choice': 'billing', 'confidence': 0.9โ€ฆ, 'probabilities': {...}},
# '๊ธด๊ธ‰': {'choice': 'urgent', 'confidence': 0.9โ€ฆ, 'probabilities': {...}}}
```
- **`state`**: ํŒ๋‹จํ•  ๋‚ด์šฉ. `{"์ด๋ฆ„": ๊ฐ’}`์ด ํ”„๋กฌํ”„ํŠธ์— ํ•œ ์ค„์”ฉ ๋“ค์–ด๊ฐ‘๋‹ˆ๋‹ค. ๋ฌธ์žฅ ํ•˜๋‚˜๋ฉด ๊ทธ๋ƒฅ ๋ฌธ์ž์—ด๋กœ ๋„ฃ์–ด๋„ ๋ฉ๋‹ˆ๋‹ค.
- **`instructions`**: ๋ฌด์—‡์„ ์–ด๋–ค ๊ธฐ์ค€์œผ๋กœ ๊ณ ๋ฅผ์ง€. ํŒ๋‹จ ๊ธฐ์ค€์„ ํ•œ๋‘ ๋ฌธ์žฅ ๋ง๋ถ™์ด๋ฉด(์˜ˆ: โ€œ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ํฌ๊ฒŒ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
- **`criteria`**: `{ID: ์„ค๋ช…}`, 2~26๊ฐœ. ID๋Š” ๊ฒฐ๊ณผ๋กœ ๋Œ๋ ค๋ฐ›๋Š” ๊ฐ’์ด๊ณ , ๋ชจ๋ธ์€ ์„ค๋ช…์„ ๋ณด๊ณ  ๊ณ ๋ฆ…๋‹ˆ๋‹ค. ID๊ฐ€ ํ•„์š” ์—†์œผ๋ฉด `["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก"]`์ฒ˜๋Ÿผ ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ์–ด๋„ ๋ฉ๋‹ˆ๋‹ค.
#### 3. ํ•œ ์ค„๋กœ
```python
judge("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”", choices=["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก", "๊ธฐํƒ€"],
instruction="๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.")
```
`judge.decide`์—์„œ ์งˆ๋ฌธ์ด ํ•˜๋‚˜์ผ ๋•Œ์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค. ๋ฆฌ์ŠคํŠธ๋ฅผ ๋„ฃ๊ณ  `batch_size=8`์ฒ˜๋Ÿผ ์ฃผ๋ฉด ๋ฌถ์–ด์„œ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค.
---
`trust_remote_code=True`๋Š” ์ด ์ €์žฅ์†Œ์˜ [`judge_pipeline.py`](judge_pipeline.py)์™€ [`judge.py`](judge.py)(ํ”„๋กฌํ”„ํŠธ ๋งŒ๋“ค๊ธฐ์™€ ๋ณด๊ธฐ ์ ์ˆ˜ ๊ณ„์‚ฐ)๋ฅผ ์‹คํ–‰ํ•œ๋‹ค๋Š” ๋œป์ž…๋‹ˆ๋‹ค. ์›๊ฒฉ ์ฝ”๋“œ๋ฅผ ์“ฐ๊ณ  ์‹ถ์ง€ ์•Š์œผ๋ฉด `judge.py`๋ฅผ ๋‚ด๋ ค๋ฐ›์•„ `from judge import Judge; judge = Judge()`๋กœ ๋˜‘๊ฐ™์ด ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. [`quickstart.ipynb`](quickstart.ipynb)์— ์˜ˆ์‹œ๋ฅผ ๋ชจ์•„ ๋‘์—ˆ์Šต๋‹ˆ๋‹ค.
## ์‚ฌ์šฉ ์˜ˆ์‹œ
์ด ๋ชจ๋ธ์€ **โ€œ๋ฌด์—‡์„ ํ• ์ง€โ€๋ฅผ ๋น ๋ฅด๊ฒŒ ์ •ํ•˜๋Š” ์•ž๋‹จ**์— ๋‘๋Š” ๊ฒƒ์ด ๊ฐ€์žฅ ์ž˜ ๋งž์Šต๋‹ˆ๋‹ค. ์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณ ๋ฅด๋Š” ํŒ๋‹จ์€ ์ด ๋ชจ๋ธ์ด ํ•˜๊ณ , ๊ธ€ ์“ฐ๊ธฐ๋‚˜ ๊ฐ’ ์ฑ„์šฐ๊ธฐ์ฒ˜๋Ÿผ ์ƒ์„ฑ์ด ํ•„์š”ํ•œ ์ผ๋งŒ ํฐ ๋ชจ๋ธ์ด๋‚˜ ์‚ฌ๋žŒ์—๊ฒŒ ๋„˜๊ธฐ๋Š” ๊ตฌ์„ฑ์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์˜ˆ์‹œ๋Š” ๋ชจ๋‘ ์œ„์—์„œ ๋งŒ๋“  `judge`๋กœ ๋ฐ”๋กœ ๋Œ์•„๊ฐ‘๋‹ˆ๋‹ค. ์ฃผ์„์— ์ ์€ ๊ฒฐ๊ณผ๋Š” ์˜๋„ํ•œ ์ •๋‹ต์ด๊ณ , ์‹ค์ œ ํ™•๋ฅ ์€ ์ง์ ‘ ์‹คํ–‰ํ•ด ํ™•์ธํ•˜์„ธ์š”.
### 1. ๊ณ ๊ฐ ๋ฌธ์˜ ๋ถ„๋ฅ˜ โ€” ๋ถ€์„œ์™€ ๊ธด๊ธ‰๋„๋ฅผ ํ•œ ๋ฒˆ์—
๋ฌธ์˜ ํ•˜๋‚˜๋ฅผ ์—ฌ๋Ÿฌ ๊ด€์ ์œผ๋กœ ๋™์‹œ์— ๋ถ„๋ฅ˜ํ•ฉ๋‹ˆ๋‹ค. ์งˆ๋ฌธ๋งˆ๋‹ค ๋ณด๊ธฐ ID๋ฅผ ์ •ํ•ด ๋‘๋ฉด ๊ฒฐ๊ณผ๋ฅผ ๋ฐ”๋กœ ์—…๋ฌด ์ฝ”๋“œ์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
```python
def route_ticket(text):
r = judge.decide(
state={"๊ณ ๊ฐ ๋ฌธ์˜": text},
questions={
"๋ถ€์„œ": {
"instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค. ์—ฌ๋Ÿฌ ๋‚ด์šฉ์ด ์„ž์—ฌ ์žˆ์œผ๋ฉด ๊ณ ๊ฐ์ด ํ•ด๊ฒฐ์„ ์›ํ•˜๋Š” ๋ฌธ์ œ๋ฅผ ์šฐ์„ ํ•œ๋‹ค.",
"criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ",
"product": "์ œํ’ˆ ์‚ฌ์šฉยท๊ณ ์žฅ", "other": "๊ธฐํƒ€"},
},
"๊ธด๊ธ‰": {
"instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๊ฑฐ๋‚˜ ๊ธˆ์ „ ํ”ผํ•ด๊ฐ€ ์ง„ํ–‰ ์ค‘์ธ์ง€ ํŒ๋‹จํ•œ๋‹ค.",
"criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ํ•„์š”", "normal": "์ผ๋ฐ˜ ์ฒ˜๋ฆฌ"},
},
},
)
return r["๋ถ€์„œ"]["choice"], r["๊ธด๊ธ‰"]["choice"]
route_ticket("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ํ™˜๋ถˆํ•ด ์ฃผ์„ธ์š”.") # ('billing', 'urgent')
```
- ์ง€์นจ์— **ํ—ท๊ฐˆ๋ฆฌ๋Š” ๊ฒฝ์šฐ์˜ ๊ธฐ์ค€**์„ ํ•œ ๋ฌธ์žฅ ๋„ฃ์œผ๋ฉด(โ€œ์—ฌ๋Ÿฌ ๋‚ด์šฉ์ด ์„ž์—ฌ ์žˆ์œผ๋ฉด โ€ฆ์„ ์šฐ์„ ํ•œ๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
- ๋ณด๊ธฐ์— `๊ธฐํƒ€`๋ฅผ ๋‘๋ฉด ์–ด๋””์—๋„ ์•ˆ ๋งž๋Š” ๋ฌธ์˜๋ฅผ ์–ต์ง€๋กœ ๋ผ์›Œ ๋„ฃ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
### 2. ๋ฆฌ๋ทฐ ๊ฐ์ • ๋ถ„์„ โ€” ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ์™€ ์‚ฌ๋žŒ ๊ฒ€ํ†  ๊ธฐ์ค€
๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ์œผ๋ฉด ๋ฌถ์–ด์„œ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ๋”ฐ๋กœ ๋ชจ์•„ ์‚ฌ๋žŒ์ด ๋ด…๋‹ˆ๋‹ค.
```python
reviews = ["๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”", "๊ฐ€๊ฒฉ ๋Œ€๋น„ ๊ดœ์ฐฎ์•„์š”", "๋‹ค์‹œ๋Š” ์•ˆ ์‚ฝ๋‹ˆ๋‹ค", "๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”"]
results = judge.sentiment(reviews)
auto = [(t, r["choice"]) for t, r in zip(reviews, results) if r["confidence"] >= 0.7]
review = [t for t, r in zip(reviews, results) if r["confidence"] < 0.7] # ์‚ฌ๋žŒ์ด ํ™•์ธ
```
- **๋ฐ˜์–ดยท๋น„๊ผผ**(โ€œ๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”โ€)์€ ๋†’์€ ํ™•์‹ ์œผ๋กœ ํ‹€๋ฆฌ๊ธฐ๋„ ํ•ฉ๋‹ˆ๋‹ค. ์ค‘์š”ํ•œ ๊ณณ์—๋Š” ํ‘œ๋ณธ์„ ์‚ฌ๋žŒ์ด ๋‹ค์‹œ ๋ณด์„ธ์š”.
- ๊ธฐ์ค€๊ฐ’ 0.7์€ ์˜ˆ์‹œ์ž…๋‹ˆ๋‹ค. ์‹ค์ œ ๋ฐ์ดํ„ฐ ์ˆ˜๋ฐฑ ๊ฑด์— ์ •๋‹ต์„ ๋‹ฌ์•„ ๋ณด๊ณ  โ€œ์ด ๊ธฐ์ค€ ์ด์ƒ์ด๋ฉด ๋ช‡ %๊ฐ€ ๋งž๋Š”์ง€โ€๋ฅผ ํ™•์ธํ•ด ์ •ํ•˜์„ธ์š”.
### 3. ๋ฆฌ๋ทฐ๋ฅผ ์—ฌ๋Ÿฌ ์ธก๋ฉด์œผ๋กœ โ€” ๋ฐฐ์†กยทํ’ˆ์งˆยท๊ฐ€๊ฒฉ ๊ฐ๊ฐ
ํ•œ ๋ฆฌ๋ทฐ์—์„œ ์ธก๋ฉด๋ณ„๋กœ ๋”ฐ๋กœ ํŒ๋‹จํ•ฉ๋‹ˆ๋‹ค. ๊ฐ™์€ `state`์— ์งˆ๋ฌธ๋งŒ ๋Š˜๋ฆฌ๋ฉด ๋ฉ๋‹ˆ๋‹ค.
```python
ASPECT = {"pos": "๊ธ์ •์ ์œผ๋กœ ์–ธ๊ธ‰", "neg": "๋ถ€์ •์ ์œผ๋กœ ์–ธ๊ธ‰", "none": "์–ธ๊ธ‰ํ•˜์ง€ ์•Š์Œ"}
judge.decide(
state={"๋ฆฌ๋ทฐ": "๋ฐฐ์†ก์€ ํ•˜๋ฃจ ๋งŒ์— ์™”๋Š”๋ฐ ๋งˆ๊ฐ์ด ์—‰๋ง์ด๋„ค์š”. ์ด ๊ฐ€๊ฒฉ์ด๋ฉด ๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”."},
questions={
"๋ฐฐ์†ก": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ๋ฐฐ์†ก(์†๋„ยทํฌ์žฅ)์„ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค.", "criteria": ASPECT},
"ํ’ˆ์งˆ": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ์ œํ’ˆ ํ’ˆ์งˆ(๋งˆ๊ฐยท๋‚ด๊ตฌ์„ฑยท์„ฑ๋Šฅ)์„ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค.", "criteria": ASPECT},
"๊ฐ€๊ฒฉ": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ๊ฐ€๊ฒฉยท๊ฐ€์„ฑ๋น„๋ฅผ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค. ์• ๋งคํ•œ ํ‰๊ฐ€๋Š” ์–ธ๊ธ‰ํ•œ ๊ฒƒ์œผ๋กœ ๋ณด๋˜ ๊ธ์ •ยท๋ถ€์ •์„ ๋ฌธ๋งฅ์œผ๋กœ ํŒ๋‹จํ•œ๋‹ค.", "criteria": ASPECT},
},
)
# {'๋ฐฐ์†ก': {'choice': 'pos', โ€ฆ}, 'ํ’ˆ์งˆ': {'choice': 'neg', โ€ฆ}, '๊ฐ€๊ฒฉ': {โ€ฆ}}
```
### 4. ๋Œ“๊ธ€ ๊ฒ€์ˆ˜ โ€” ํ˜์˜ค ์—ฌ๋ถ€์™€ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€
`hate`๋กœ ์ฐจ๋ณ„ยท๋น„ํ•˜ ์—ฌ๋ถ€๋ฅผ ๊ฑฐ๋ฅด๊ณ , `offensive`๋กœ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€์„ ๋‚˜๋ˆ•๋‹ˆ๋‹ค. ๊ธฐ์‚ฌ ๋Œ“๊ธ€์ด๋ฉด ์ œ๋ชฉ์„ ๋ฌธ๋งฅ์œผ๋กœ ํ•จ๊ป˜ ๋„ฃ์Šต๋‹ˆ๋‹ค.
```python
comment = "์ด๋Ÿฐ ๊ธฐ์‚ฌ ์“ฐ๋Š” ๊ธฐ์ž๋Š” ๋จธ๋ฆฌ๊ฐ€ ์žˆ๋Š” ๊ฑด์ง€"
judge.hate(comment) # ๋น„ํ˜์˜ค / ํ˜์˜ค
judge.offensive(comment, news_title="์ •๋ถ€, ๋‚ด๋…„ ์˜ˆ์‚ฐ์•ˆ ๋ฐœํ‘œ") # ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค
```
- ๋‘ ๋ฉ”์„œ๋“œ๋Š” ๊ธฐ์ค€์ด ๋‹ค๋ฆ…๋‹ˆ๋‹ค. `hate`๋Š” ์„ฑ๋ณ„ยท๋‚˜์ดยท์ถœ์‹  ๊ฐ™์€ ์ง‘๋‹จ์— ๋Œ€ํ•œ ์ฐจ๋ณ„ยท๋น„ํ•˜(APEACH ๊ธฐ์ค€), `offensive`๋Š” ๊ฐœ์ธ์„ ํ–ฅํ•œ ๋ชจ์š•๊นŒ์ง€ ํฌํ•จํ•œ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„(BEEP ๊ธฐ์ค€)์ž…๋‹ˆ๋‹ค.
- ์‚ฌ๋žŒ์—๊ฒŒ ๋ถˆ์ด์ต์„ ์ฃผ๋Š” ๊ฒฐ์ •(์‚ญ์ œยท์ œ์žฌ)์— ์“ธ ๋•Œ๋Š” ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ๋ฐ˜๋“œ์‹œ ์‚ฌ๋žŒ์ด ๊ฒ€ํ† ํ•˜๊ฒŒ ํ•˜์„ธ์š”.
### 5. ์—์ด์ „ํŠธ ๋ผ์šฐํ„ฐ โ€” ๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€
์—์ด์ „ํŠธ๊ฐ€ ๋งค ํ„ด ํฐ ๋ชจ๋ธ์— โ€œ๋‹ค์Œ์— ๋ญ˜ ํ• ์ง€โ€๋ฅผ ๋ฌป๋Š” ๋Œ€์‹ , ์ด ๋ชจ๋ธ์ด ๋จผ์ € ์ •ํ•ฉ๋‹ˆ๋‹ค. ๋„๊ตฌ ์ธ์ž ์ฑ„์šฐ๊ธฐ์™€ ๋‹ต๋ณ€ ์ž‘์„ฑ๋งŒ ํฐ ๋ชจ๋ธ์— ๋งก๊น๋‹ˆ๋‹ค.
```python
tools = [
{"type": "function", "function": {"name": "call_taxi", "description": "ํƒ์‹œ๋ฅผ ํ˜ธ์ถœํ•œ๋‹ค",
"parameters": {"type": "object", "properties": {"pickup": {"type": "string"}, "destination": {"type": "string"}},
"required": ["pickup", "destination"]}}},
{"type": "function", "function": {"name": "get_weather", "description": "์ง€์—ญ์˜ ํ˜„์žฌ ๋‚ ์”จ๋ฅผ ์•Œ๋ ค์ค€๋‹ค",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}},
]
messages = [{"role": "user", "content": "๊ฐ•๋‚จ์—ญ์—์„œ ์„œ์šธ์—ญ๊นŒ์ง€ ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜"}]
r = judge.next_action(messages, tools)
if r["confidence"] < 0.7:
action = "ํฐ ๋ชจ๋ธ์—๊ฒŒ ๋‹ค์Œ ํ–‰๋™์„ ๋งก๊น€"
elif r["choice"].startswith("CALL:"):
action = f"ํฐ ๋ชจ๋ธ๋กœ {r['choice'][5:]} ์˜ ์ธ์ž๋ฅผ ์ฑ„์›Œ ํ˜ธ์ถœ" # ์ด ๋ชจ๋ธ์€ ์ธ์ž๋ฅผ ์ฑ„์šฐ์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค
elif r["choice"] == "ASK":
action = "๋น ์ง„ ํ•„์ˆ˜ ์ •๋ณด๋ฅผ ์‚ฌ์šฉ์ž์—๊ฒŒ ๋˜๋ฌป๊ธฐ"
else: # RESPOND
action = "๋„๊ตฌ ์—†์ด ๋ฐ”๋กœ ๋‹ต๋ณ€"
```
- ๋„๊ตฌ๋Š” **OpenAI ํ˜•์‹ ๊ทธ๋Œ€๋กœ**, ํŠนํžˆ `required`๋ฅผ ๊ผญ ๋„ฃ์œผ์„ธ์š”. ์ด๋ฆ„๋งŒ ์ฃผ๋ฉด ํ•„์ˆ˜ ์ธ์ž๋ฅผ ๋ณผ ์ˆ˜ ์—†์–ด ASK์™€ CALL์„ ๊ตฌ๋ณ„ํ•˜์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค.
- ๋Œ€ํ™” ๊ธฐ๋ก(์•ž ํ„ด, ๋„๊ตฌ ๊ฒฐ๊ณผ `{"role": "tool", โ€ฆ}`)์„ ํ•จ๊ป˜ ๋„ฃ์œผ๋ฉด โ€œ์ด๋ฏธ ์ฒ˜๋ฆฌ๋์œผ๋‹ˆ ๊ฒฐ๊ณผ ์•ˆ๋‚ด(RESPOND)โ€ ๊ฐ™์€ ํŒ๋‹จ๋„ ํ•ฉ๋‹ˆ๋‹ค.
- ๋„๊ตฌ๋Š” ์ตœ๋Œ€ 24๊ฐœ๊นŒ์ง€ ๋„ฃ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค (๋ณด๊ธฐ 26๊ฐœ ์ค‘ 2๊ฐœ๊ฐ€ ASKยทRESPOND).
### 6. ์ •์ฑ…์— ๋”ฐ๋ฅธ ๊ณ ๊ฐ์„ผํ„ฐ ํ–‰๋™ ๊ฒฐ์ •
์ •์ฑ… ๋ฌธ์„œ์™€ ๊ณ ๊ฐ ์š”์ฒญ์„ ์ฃผ๋ฉด ์‹คํ–‰ / ํ™•์ธ / ๋˜๋ฌป๊ธฐ / ์ด๊ด€ / ๊ฑฐ์ ˆ ์ค‘ ๋ฌด์—‡์„ ํ• ์ง€ ๊ณ ๋ฆ…๋‹ˆ๋‹ค. ์˜ˆ์™ธ ์กฐํ•ญ์ด ์ ์šฉ๋˜๋Š”์ง€๊นŒ์ง€ ๋”ฐ์ง‘๋‹ˆ๋‹ค.
```python
policy = ("ํ•ด์™ธ ์นด๋“œ ์Šน์ธ ๊ธˆ์•ก์ด 100,000์› ์ด์ƒ์ด๋ฉด ๊ณ ๊ฐ์˜ ํ™•์ธ์„ ๋ฐ›์€ ํ›„์—๋งŒ ์‹คํ–‰ํ•œ๋‹ค. "
"๋‹ค๋งŒ VIP ๋“ฑ๊ธ‰ ๊ณ ๊ฐ์€ ํ™•์ธ ์—†์ด ๋ฐ”๋กœ ์‹คํ–‰ํ•œ๋‹ค.")
judge.policy_action(policy, "ํ™์ฝฉ์—์„œ ์นด๋“œ๋กœ 149,000์› ๊ฒฐ์ œํ•˜๋ ค๋Š”๋ฐ ์Šน์ธํ•ด์ค˜. ๋‚ด ๋“ฑ๊ธ‰์€ ์ผ๋ฐ˜์ด์•ผ.")
# โ†’ ASK_CONFIRMATION (๊ธˆ์•ก ๊ธฐ์ค€์„ ๋„˜์—ˆ๊ณ  VIP ์˜ˆ์™ธ๊ฐ€ ์•„๋‹˜)
```
- ์ •์ฑ…์— **์ ํ˜€ ์žˆ๋Š” ๊ฒƒ๋งŒ** ๊ทผ๊ฑฐ๋กœ ํŒ๋‹จํ•˜๋„๋ก ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค. ์‹ค์ œ ์—…๋ฌด ๊ทœ์ •์„ ๋ชจ๋ธ์ด ์•Œ์•„์„œ ๋ณด์ถฉํ•˜์ง€ ์•Š์œผ๋‹ˆ, ํ•„์š”ํ•œ ์กฐ๊ฑด์€ ์ •์ฑ… ๋ฌธ์žฅ์— ๋‹ค ์“ฐ์„ธ์š”.
- ๋ณด๊ธฐ: `ANSWER`, `ASK_CLARIFICATION`, `ASK_CONFIRMATION`, `CALL_TOOL`, `CALL_MULTIPLE_TOOLS`, `ESCALATE`, `REFUSE`.
### 7. ๋‹ต๋ณ€์ด ๊ทผ๊ฑฐ ๋ฌธ์„œ์™€ ๋งž๋Š”์ง€ ํ™•์ธ (RAG ๊ฒ€์ฆ)
๊ฒ€์ƒ‰ํ•œ ๋ฌธ์„œ๋ฅผ ์ „์ œ๋กœ, ์ƒ์„ฑ๋œ ๋‹ต๋ณ€์˜ ๋ฌธ์žฅ์„ ๊ฐ€์„ค๋กœ ๋„ฃ์Šต๋‹ˆ๋‹ค. `contradiction`์ด๋ฉด ๋ฌธ์„œ์™€ ์–ด๊ธ‹๋‚œ ๋‹ต์ด๊ณ , `neutral`์ด๋ฉด ๋ฌธ์„œ์— ๊ทผ๊ฑฐ๊ฐ€ ์—†๋Š” ๋‚ด์šฉ์ž…๋‹ˆ๋‹ค.
```python
doc = "์ž„๋Œ€์ธ์€ ๊ณ„์•ฝ์ด ๋๋‚˜๊ณ  ์ž„์ฐจ์ธ์ด ์ง‘์„ ๋น„์šด ๋‚ ๋ถ€ํ„ฐ 30์ผ ์ด๋‚ด์— ๋ณด์ฆ๊ธˆ์„ ๋Œ๋ ค์ค€๋‹ค."
answer_sentences = ["๋ณด์ฆ๊ธˆ์€ ์ง‘์„ ๋น„์šด ๋’ค 30์ผ ์•ˆ์— ๋Œ๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.",
"๋ณด์ฆ๊ธˆ์€ ์ง‘์„ ๋น„์šด ๋’ค 40์ผ ๋’ค์— ๋Œ๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.",
"๋ณด์ฆ๊ธˆ์—๋Š” ์—ฐ 3% ์ด์ž๊ฐ€ ๋ถ™์Šต๋‹ˆ๋‹ค."]
for s, r in zip(answer_sentences, judge.nli([doc] * len(answer_sentences), answer_sentences)):
print(r["choice"], s)
# entailment โ€ฆ 30์ผ ์•ˆ์— โ€ฆ โ†’ ๊ทผ๊ฑฐ ์žˆ์Œ
# contradiction โ€ฆ 40์ผ ๋’ค์— โ€ฆ โ†’ ๋ฌธ์„œ์™€ ์–ด๊ธ‹๋‚จ
# neutral โ€ฆ ์—ฐ 3% ์ด์ž โ€ฆ โ†’ ๋ฌธ์„œ์— ๊ทผ๊ฑฐ ์—†์Œ
```
- ๋‹ต๋ณ€์€ **๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋‚˜๋ˆ ** ๋„ฃ์œผ์„ธ์š”. ์—ฌ๋Ÿฌ ์ฃผ์žฅ์ด ์„ž์ธ ๊ธด ๋‹ต๋ณ€์€ ํŒ๋‹จ์ด ํ๋ ค์ง‘๋‹ˆ๋‹ค.
- ํ•™์Šต ์ตœ๋Œ€ ๊ธธ์ด๊ฐ€ 2,048ํ† ํฐ์ด๋ผ ๊ธด ๋ฌธ์„œ๋Š” ๊ด€๋ จ ๋‹จ๋ฝ๋งŒ ์ž˜๋ผ ๋„ฃ์œผ์„ธ์š”.
### 8. ์•ฝ๊ด€ยท๋ฌธ์„œ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค
๋ฌธ์„œ๋ฅผ ์ฃผ๊ณ  ์งˆ๋ฌธ์— ์˜ˆ/์•„๋‹ˆ์˜ค๋กœ ๋‹ตํ•˜๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ๋ฌธ์„œ์— ์—†๋Š” ์ƒ์‹์œผ๋กœ ๋‹ตํ•˜์ง€ ์•Š๋„๋ก ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
```python
terms = "๊ตฌ๋งค ํ›„ 7์ผ ์ด๋‚ด์—๋Š” ๋‹จ์ˆœ ๋ณ€์‹ฌ์œผ๋กœ๋„ ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‹ค. ๋‹จ, ํฌ์žฅ์„ ๋œฏ์€ ์†Œํ”„ํŠธ์›จ์–ด๋Š” ํ™˜๋ถˆ ๋Œ€์ƒ์—์„œ ์ œ์™ธํ•œ๋‹ค."
judge.yes_no("ํฌ์žฅ์„ ๋œฏ์€ ๊ฒŒ์ž„ CD๋ฅผ 3์ผ ๋งŒ์— ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‚˜์š”?", terms) # โ†’ ์•„๋‹ˆ์˜ค
judge.yes_no("๋œฏ์ง€ ์•Š์€ ํ‚ค๋ณด๋“œ๋ฅผ 5์ผ ๋งŒ์— ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‚˜์š”?", terms) # โ†’ ์˜ˆ
```
### 9. ๋น„์Šทํ•œ ์งˆ๋ฌธ ์ฐพ๊ธฐ โ€” FAQ ์ค‘๋ณต ํ™•์ธ
์ƒˆ๋กœ ๋“ค์–ด์˜จ ์งˆ๋ฌธ์ด ๊ธฐ์กด FAQ์™€ ๊ฐ™์€ ๋œป์ธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ํ›„๋ณด๋Š” ๊ฒ€์ƒ‰(์ž„๋ฒ ๋”ฉ ๋“ฑ)์œผ๋กœ ๋จผ์ € ์ขํžˆ๊ณ , ์ตœ์ข… ํ™•์ธ์— ์ด ๋ชจ๋ธ์„ ์“ฐ๋Š” ๊ตฌ์„ฑ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
```python
faq = ["๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”", "๋ฐฐ์†ก์ง€๋Š” ์–ด๋–ป๊ฒŒ ๋ฐ”๊พธ๋‚˜์š”", "ํšŒ์› ํƒˆํ‡ด๋Š” ์–ด๋””์„œ ํ•˜๋‚˜์š”"]
q = "๋กœ๊ทธ์ธ ๋น„๋ฒˆ์ด ๊ธฐ์–ต์ด ์•ˆ ๋‚˜์š”"
hits = judge.similar([q] * len(faq), faq)
best = max(range(len(faq)), key=lambda i: hits[i]["probabilities"]["๊ฐ™์Œ"])
faq[best], hits[best]["choice"] # ('๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”', '๊ฐ™์Œ')
```
- `similar`๋Š” ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณผ์ œ์ž…๋‹ˆ๋‹ค (ํ•™์Šตํ•˜์ง€ ์•Š์€ KLUE-STS์—์„œ 85.9%).
### 10. ๋‚ด ๋ถ„๋ฅ˜ ์ฒด๊ณ„ ๊ทธ๋Œ€๋กœ ์“ฐ๊ธฐ
์–ด๋–ค ๋ถ„๋ฅ˜๋“  ์ง€์นจ๊ณผ ๋ณด๊ธฐ๋งŒ ์“ฐ๋ฉด ๋ฉ๋‹ˆ๋‹ค. ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณผ์ œ๋ผ๋„ **ํŒ๋‹จ ๊ธฐ์ค€์„ ์ง€์นจ์— ์ ์–ด ์ฃผ๋Š” ๊ฒƒ**์ด ์ •ํ™•๋„๋ฅผ ๊ฐ€์žฅ ํฌ๊ฒŒ ์ขŒ์šฐํ•ฉ๋‹ˆ๋‹ค.
```python
judge.decide(
state={"๋ณด๊ณ ์„œ ๋ฌธ์žฅ": "3๋ถ„๊ธฐ ๋งค์ถœ์€ ์ „๋…„ ๋Œ€๋น„ 12% ์ฆ๊ฐ€ํ–ˆ์œผ๋‚˜ ์˜์—…์ด์ต๋ฅ ์€ ์†Œํญ ํ•˜๋ฝํ–ˆ๋‹ค."},
questions={
"๋…ผ์กฐ": {
"instructions": "๋ฌธ์žฅ์ด ํšŒ์‚ฌ ์‹ค์ ์„ ์–ด๋–ค ๋…ผ์กฐ๋กœ ์ „๋‹ฌํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค. ์ข‹์€ ์ ๊ณผ ๋‚˜์œ ์ ์ด ํ•จ๊ป˜ ์žˆ์œผ๋ฉด ํ˜ผํ•ฉ์œผ๋กœ ๋ณธ๋‹ค.",
"criteria": {"positive": "๊ธ์ •์ ", "negative": "๋ถ€์ •์ ", "mixed": "ํ˜ผํ•ฉ", "neutral": "์‚ฌ์‹ค ์ „๋‹ฌ๋งŒ"},
},
},
)
```
**์ž˜ ์•ˆ ๋˜๋Š” ์“ฐ์ž„์ƒˆ**: ์ˆซ์ž ๋น„๊ตยท๊ณ„์‚ฐ์ด ํ•ต์‹ฌ์ธ ํŒ๋‹จ(โ€œ์–ด๋А ์ชฝ์ด ๋” ์‹ผ๊ฐ€โ€), ๋ฌธ์žฅ์—์„œ ๊ฐ’ ๋ฝ‘๊ธฐ, ๋‹ต๋ณ€ ์ž‘์„ฑ, ๋‹ค๋ฅธ ๋ชจ๋ธ ๋‹ต๋ณ€์˜ ํ’ˆ์งˆ ์ฑ„์ (LLM-as-a-judge). ์ด๋Ÿฐ ์ผ์€ ์ƒ์„ฑ ๋ชจ๋ธ์— ๋งก๊ธฐ์„ธ์š”.
### ์›๋ฆฌ: transformers๋งŒ์œผ๋กœ ์ง์ ‘ ๊ณ„์‚ฐํ•˜๊ธฐ
```python
import torch, json
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import hf_hub_download
name = "coderecode95/KJH-Judge-4B-ko"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16, device_map="auto").eval()
T = json.load(open(hf_hub_download(name, "calibration.json")))["temperature"] # 1.7179
choices = ["๊ธ์ •์  ๊ฐ์ •", "๋ถ€์ •์  ๊ฐ์ •"]
prompt = ("๋ฌธ์žฅ์˜ ์ „๋ฐ˜์ ์ธ ๊ฐ์ • ๊ทน์„ฑ์„ ํŒ๋‹จํ•œ๋‹ค.\n\n"
"text: ๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”\n\n"
"๋ณด๊ธฐ:\n" + "\n".join(f"{chr(65+i)}. {c}" for i, c in enumerate(choices)) +
"\n\n์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.")
ids = tok(f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n", return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
with torch.no_grad():
logits = model(ids).logits[0, -1]
letters = [tok.encode(chr(65 + i), add_special_tokens=False)[0] for i in range(len(choices))]
probs = torch.softmax(logits[letters].float() / T, -1)
print({c: round(p, 4) for c, p in zip(choices, probs.tolist())})
```
### ํ”„๋กฌํ”„ํŠธ ํ˜•์‹
ํ•™์Šต๊ณผ ํ‰๊ฐ€์— ์“ด ํ˜•์‹์ž…๋‹ˆ๋‹ค. ์ด ํ˜•์‹์„ ์ง€์ผœ์•ผ ์œ„ ์„ฑ๋Šฅ์ด ๋‚˜์˜ต๋‹ˆ๋‹ค.
```
<|im_start|>user
{๊ณผ์ œ ์„ค๋ช…} โ† instructions
{ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’} โ† state ์˜ ํ•ญ๋ชฉ๋งˆ๋‹ค ํ•œ ์ค„. ๊ฐ’์ด ๋ฌธ์ž์—ด์ด ์•„๋‹ˆ๋ฉด JSON
{ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’}
๋ณด๊ธฐ:
A. {๋ณด๊ธฐ 1} โ† criteria ์˜ ์„ค๋ช…
B. {๋ณด๊ธฐ 2}
โ€ฆ
์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.<|im_end|>
<|im_start|>assistant
```
- `judge.decide()`์™€ ๋ฉ”์„œ๋“œ๋“ค์€ ์ด ํ˜•์‹์„ ์ž๋™์œผ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์ง์ ‘ ๋งŒ๋“ค ๋•Œ๋งŒ ์‹ ๊ฒฝ ์“ฐ๋ฉด ๋ฉ๋‹ˆ๋‹ค.
- ๋ณด๊ธฐ๋Š” 2~26๊ฐœ์ž…๋‹ˆ๋‹ค.
- ๊ณผ์ œ ์„ค๋ช…์€ ์งง๊ณ  ๊ตฌ์ฒด์ ์œผ๋กœ ์“ฐ์„ธ์š”. ํŒ๋‹จ ๊ธฐ์ค€์„ ํ•œ๋‘ ๋ฌธ์žฅ ๋ง๋ถ™์ด๋ฉด(์˜ˆ: โ€œ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹คโ€, โ€œ์ •๋ณด๊ฐ€ ์—†๋‹ค๋Š” ์ด์œ ๋งŒ์œผ๋กœ ๋ชจ์ˆœ์„ ๊ณ ๋ฅด์ง€ ์•Š๋Š”๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ํฌ๊ฒŒ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
- `generate()`๋กœ ์“ธ ๊ฒฝ์šฐ `max_new_tokens=1`๋กœ ๋ณด๊ธฐ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋ฐ›์œผ์„ธ์š” (์ด ์ €์žฅ์†Œ์˜ `generation_config.json`์€ `do_sample: false`์ž…๋‹ˆ๋‹ค). ํ™•๋ฅ ์ด ํ•„์š”ํ•˜๋ฉด ์œ„์ฒ˜๋Ÿผ logits๋ฅผ ์ฝ์œผ์„ธ์š”.
## ์„ฑ๋Šฅ
### 1. ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณต๊ฐœ ๋ฒค์น˜๋งˆํฌ
์ด ๋ชจ๋ธ์˜ ํ•™์Šต์— ์“ฐ์ง€ ์•Š์€ ์ถœ์ฒ˜์˜ ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค (KLUE-STS๋Š” ํ•™์Šต์— ์“ด KLUE NLIยทYNAT์™€ ๊ฐ™์€ KLUE ๋ฌถ์Œ์ด์ง€๋งŒ, STS ๊ณผ์ œ๋Š” ํ•™์Šตํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค). 0-shot, ์œ„ ํ”„๋กฌํ”„ํŠธ ํ˜•์‹, ๋ณด๊ธฐ ๊ธฐํ˜ธ ์ ์ˆ˜ ๋ฐฉ์‹์œผ๋กœ ์Ÿ€์Šต๋‹ˆ๋‹ค (ํ‘œ๋ณธ์€ ๊ณ ์ • ์‹œ๋“œ, ๊ด„ํ˜ธ๋Š” 95% ์‹ ๋ขฐ๊ตฌ๊ฐ„). ์žฌํ˜„ ์Šคํฌ๋ฆฝํŠธ: [`eval/bench_heldout.py`](eval/bench_heldout.py).
<!-- HELDOUT:start -->
| ๋ฒค์น˜๋งˆํฌ | ๋ฌด์—‡์„ ์žฌ๋‚˜ | ๋ฌธํ•ญ | ์ •ํ™•๋„ (95% ๊ตฌ๊ฐ„) | ๋ฌด์ž‘์œ„ | ์ถœ์ฒ˜ ยท ๋ผ์ด์„ ์Šค |
|---|---|---:|---:|---:|---|
| NSMC | ์˜ํ™” ๋ฆฌ๋ทฐ ๊ฐ์„ฑ (๊ธ์ •/๋ถ€์ •) | 1,000 | **85.7%** (83.4~87.7) | 50% | e9t/nsmc ยท ratings_test ยท CC0 1.0 |
| KorNLI (XNLI) | ๋ฌธ์žฅ ๊ด€๊ณ„ 3๋ถ„๋ฅ˜ (XNLI ํ•œ๊ตญ์–ด ๋ฒˆ์—ญ) | 1,000 | **74.5%** (71.7~77.1) | 33% | kakaobrain/kor_nli ยท xnli/test ยท CC BY-SA 4.0 |
| KLUE-STS | ๋‘ ๋ฌธ์žฅ์ด ๊ฐ™์€ ๋œป์ธ์ง€ (์˜ˆ/์•„๋‹ˆ์˜ค) | 519 | **85.9%** (82.7~88.7) | 50% | klue/klue ยท sts/validation (binary-label) ยท CC BY-SA 4.0 |
| Korean UnSmile | ์•…ํ”Œยทํ˜์˜ค ํ‘œํ˜„ ์—ฌ๋ถ€ | 1,000 | **82.1%** (79.6~84.4) | 50% | smilegate-ai/kor_unsmile ยท valid (clean ์—ฌ๋ถ€) ยท CC BY-NC-ND 4.0 |
| KMMLU | ํ•œ๊ตญ์–ด ์ „๋ฌธ ์ง€์‹ 4์ง€์„ ๋‹ค (45๊ณผ๋ชฉ) | 990 | **42.6%** (39.6~45.7) | 25% | HAERAE-HUB/KMMLU ยท test ยท CC BY-ND 4.0 |
| HAE-RAE Bench 1.0 | ํ•œ๊ตญ ๋ฌธํ™”ยท์–ดํœ˜ ์ง€์‹ 5์ง€์„ ๋‹ค | 983 | **56.1%** (53.0~59.2) | 21% | HAERAE-HUB/HAE_RAE_BENCH_1.0 ยท test ยท CC BY-NC-ND 4.0 |
<!-- HELDOUT:end -->
<!-- COMPARE:start -->
**๊ฐ™์€ ์กฐ๊ฑด ๋น„๊ต** ยท ๊ฐ™์€ ๋ฌธํ•ญ ยท ๊ฐ™์€ ํ”„๋กฌํ”„ํŠธ ยท 0-shot ยท ๋ณด๊ธฐ ๊ธฐํ˜ธ ์ ์ˆ˜. ๋‹ค๋ฅธ ๋ชจ๋ธ์€ ์ƒ๊ฐ(thinking)์„ ๋„๊ณ  ๊ฐ์ž์˜ ๋Œ€ํ™” ํ˜•์‹์œผ๋กœ ๋ฌผ์—ˆ์Šต๋‹ˆ๋‹ค.
| ๋ฒค์น˜๋งˆํฌ | **KJH-Judge-4B-ko (4B)** | Gemma 4 26B-A4B-it (0-shot) |
|---|---:|---:|
| NSMC | **85.7%** | 88.7% |
| KorNLI (XNLI) | **74.5%** | 77.1% |
| KLUE-STS | **85.9%** | 88.4% |
| Korean UnSmile | **82.1%** | 81.3% |
| KMMLU | **42.6%** | 64.0% |
| HAE-RAE Bench 1.0 | **56.1%** | 79.7% |
<!-- COMPARE:end -->
์›๋ณธ Qwen3-4B์˜ ์‚ฌ์ „ํ•™์Šต์— ์ด ๋ฐ์ดํ„ฐ๊ฐ€ ๋“ค์–ด๊ฐ”๋Š”์ง€๋Š” ์•Œ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. KMMLUยทHAE-RAE๋Š” ์ „๋ฌธ ์ง€์‹์„ ๋ฌป๋Š” ์‹œํ—˜์ด๋ผ, ํŒ์ • ๋Šฅ๋ ฅ๋ณด๋‹ค ์›๋ณธ ๋ชจ๋ธ์˜ ์ง€์‹์ด ์ ์ˆ˜๋ฅผ ์ขŒ์šฐํ•ฉ๋‹ˆ๋‹ค.
### 2. Korean JEV Benchmark (์ฐธ๊ณ : ๊ฐ™์€ ์ถœ์ฒ˜์˜ ์‹œํ—˜ ๋ฌธํ•ญ)
[DaehanKim/korean-jev-benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark) 25,418๋ฌธํ•ญ ์ „์ฒด์ž…๋‹ˆ๋‹ค. ์ด ์‹œํ—˜ ๋ฌธํ•ญ ์ž์ฒด๋กœ ํ•™์Šตํ•˜์ง€๋Š” ์•Š์•˜์ง€๋งŒ, **๊ฐ™์€ ์ถœ์ฒ˜ ๋ฐ์ดํ„ฐ(KLUE, KoBEST ๋“ฑ)์˜ ํ•™์Šต ๋ถ„ํ• ๋กœ ํ•™์Šตํ–ˆ๊ธฐ ๋•Œ๋ฌธ์—** ๋‹ค๋ฅธ ๋ชจ๋ธ์˜ 0-shot ์ ์ˆ˜์™€ ๊ฐ™์€ ์กฐ๊ฑด์˜ ๋น„๊ต๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค.
| ํ•ญ๋ชฉ | ๊ฐ’ |
|---|---|
| ์ „์ฒด ์ •ํ™•๋„ | **90.37%** (ํ•™์Šต ์ „ Qwen3-4B 29.56%, ๋ฌด์ž‘์œ„ 29.86%) |
| ํ•™์Šต ๋ฐ์ดํ„ฐ๊ฐ€ ์—†๋Š” ๊ณผ๋ชฉ (Belebele ํ•œ๊ตญ์–ด ๋…ํ•ด) | **86.87%** |
| ๋ณด๊ธฐ๋ฅผ ๊ฑฐ๊พธ๋กœ ๋†“์•„๋„ ๊ฐ™์€ ๋‹ต | 97.93% |
| ๊ธฐ๋Œ€ ๋ณด์ • ์˜ค์ฐจ (ECE) | 0.0265 |
| ๊ณผ๋ชฉ | ์ •ํ™•๋„ | ๊ณผ๋ชฉ | ์ •ํ™•๋„ |
|---|---:|---|---:|
| KoBEST BoolQ | 95.65% | KLUE YNAT (๋‰ด์Šค ์ฃผ์ œ) | 93.93% |
| KoBEST HellaSwag | 94.65% | KLUE NLI | 92.39% |
| KoBEST SentiNeg | 93.91% | APEACH (ํ˜์˜ค) | 88.64% |
| KoBEST COPA | 92.55% | KoBEST WiC | 84.96% |
| SIB-200 | 94.44% | Korean Hate Speech (3๋ถ„๋ฅ˜) | 77.87% |
| FunctionChat ๋„๊ตฌ ๊ณ ๋ฅด๊ธฐ | 94.68% | EnterpriseOps ์ •์ฑ… ํ–‰๋™ (7๋ถ„๋ฅ˜) | 83.43% |
| FunctionChat ๋Œ€ํ™” | 93.50% | ๋ฒ•๋ น ์งˆ๋ฌธ (v2ยทv3, 6๊ฐœ ๋ฌถ์Œ) | 90.8~99.2% |
| FunctionChat ํ˜ธ์ถœ ํŒ๋‹จ | 84.55% | ๋ฒ•๋ น ๊ธฐ๊ฐ„ ๊ณ„์‚ฐ | 69.10% |
### ์†๋„
๊ฐœ์ธ GPU ์„œ๋ฒ„์—์„œ ๊ฐ™์€ ๋ฌธํ•ญ 60๊ฐœ(NSMCยทKorNLI)๋กœ ์žฐ ์ฐธ๊ณ ๊ฐ’์ž…๋‹ˆ๋‹ค (๋„คํŠธ์›Œํฌ ์™•๋ณต ํฌํ•จ, ์ธก์ • ์Šคํฌ๋ฆฝํŠธ: [`eval/bench_speed.py`](eval/bench_speed.py)).
| ์ธก์ • | **KJH-Judge-4B-ko** | Gemma 4 26B-A4B-it |
|---|---:|---:|
| ํŒ์ • ํ•œ ๋ฒˆ (์ค‘์•™๊ฐ’) | **250ms** (์„œ๋ฒ„ ์•ˆ ๊ณ„์‚ฐ 87ms) | 621ms |
| ๋™์‹œ 4๊ฐœ์”ฉ ๋ณด๋ƒˆ์„ ๋•Œ ์ฒ˜๋ฆฌ๋Ÿ‰ | 3.9๊ฑด/์ดˆ | **9.9๊ฑด/์ดˆ** |
| ์ƒ๊ฐ์„ ์ผœ๊ณ  ๋‹ต์„ ์ƒ์„ฑํ•  ๋•Œ | | 6.3์ดˆ |
- ๋™์‹œ ์ฒ˜๋ฆฌ๋Ÿ‰์€ ๋น„๊ต ๋ชจ๋ธ์ด ์•ž์„ฐ์Šต๋‹ˆ๋‹ค. ๋น„๊ต ๋ชจ๋ธ์€ ์š”์ฒญ์„ ๋ฌถ์–ด ์ฒ˜๋ฆฌํ•˜๋Š” ์„œ๋น™ ์—”์ง„์—์„œ, ์ด ๋ชจ๋ธ์€ ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌํ•˜๋Š” ๋‹จ์ˆœํ•œ ์„œ๋ฒ„์—์„œ ๋Œ์•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ์—๋Š” ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ๊ณ  `batch_size`๋ฅผ ์ฃผ์–ด ๋ฌถ์–ด ๊ณ„์‚ฐํ•˜์„ธ์š”.
- CPU(๋…ธํŠธ๋ถ, fp32)์—์„œ๋Š” ํŒ์ • ํ•œ ๋ฒˆ์— 6์ดˆ์—์„œ ์ˆ˜ ๋ถ„์ด ๊ฑธ๋ ค ์‹ค์‚ฌ์šฉ์ด ์–ด๋ ต์Šต๋‹ˆ๋‹ค.
## ์ž˜ํ•˜๋Š” ๊ฒƒ๊ณผ ๋ชปํ•˜๋Š” ๊ฒƒ
**์ž˜ํ•˜๋Š” ๊ฒƒ**: ๋ณด๊ธฐ๊ฐ€ ์ •ํ•ด์ง„ ๋ถ„๋ฅ˜, ๋ฌธ์žฅ ๊ด€๊ณ„ ํŒ๋‹จ, ์ง€๋ฌธ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค, ์ •์ฑ…ยท๊ทœ์น™์— ๋”ฐ๋ฅธ ํ–‰๋™ ์„ ํƒ, ์—์ด์ „ํŠธ์˜ ๋‹ค์Œ ํ–‰๋™(๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€) ๊ฒฐ์ •. ์ฒ˜์Œ ๋ณด๋Š” ์ง€์‹œ๋ฌธ์ด๋ผ๋„ ์ด๋Ÿฐ ํ˜•์‹์ด๋ฉด ์ž˜ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
**๋ชปํ•˜๋Š” ๊ฒƒ**
- **๊ธ€ ์ƒ์„ฑ, ๋ฌธ์žฅ์—์„œ ๊ฐ’ ๋ฝ‘๊ธฐ**: ๋ณด๊ธฐ๋ฅผ ๊ณ ๋ฅด๋Š” ๊ฒƒ๋งŒ ํ•ฉ๋‹ˆ๋‹ค. ๋„๊ตฌ ์ธ์ž ๊ฐ’์„ ์ฑ„์šฐ๊ฑฐ๋‚˜ ๋‹ต๋ณ€์„ ์“ฐ๋Š” ์ผ์€ ์ƒ์„ฑ ๋ชจ๋ธ์— ๋งก๊ธฐ์„ธ์š”.
- **์ˆซ์ž ๋น„๊ตยท๊ณ„์‚ฐ์ด ํ•ต์‹ฌ์ธ ํŒ๋‹จ**: ๊ฑฐ๋ฆฌยท์†๋„ยท์ ์ˆ˜ ๊ฐ™์€ ์ˆ˜์น˜๋ฅผ ๋น„๊ตํ•ด ๊ณ ๋ฅด๋Š” ์ผ์€ ์•ฝํ•ฉ๋‹ˆ๋‹ค.
- **๋ฐ˜์–ดยท๋น„๊ผผ**: โ€œ๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”โ€๋ฅผ ๊ธ์ •์œผ๋กœ ๋†’์€ ํ™•์‹ ๊ณผ ํ•จ๊ป˜ ๊ณ ๋ฅด๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
- **์ง€์‹œ๋ฌธ ๋ฌธ๊ตฌ์— ๋ฏผ๊ฐ**: โ€œ์‹ค์ œ ์กด์žฌํ•˜๋Š” ์ธ๋ฌผ์ธ๊ฐ€์š”?โ€๋ฅผ โ€œ์‹ค์ œ ์กด์žฌํ•˜๋‚˜์š”?โ€์ฒ˜๋Ÿผ ์ฝ๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ์ค€์„ ๋ช…ํ™•ํžˆ ์“ฐ์„ธ์š”.
๊ถŒ์žฅ ๊ตฌ์„ฑ: ํฐ LLM ์•ž๋‹จ์—์„œ ์ด ๋ชจ๋ธ์ด โ€œ๋ฌด์—‡์„ ํ• ์ง€โ€๋ฅผ ๋น ๋ฅด๊ณ  ์‹ธ๊ฒŒ ์ •ํ•˜๊ณ , ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ์‚ฌ๋žŒ์ด๋‚˜ ํฐ ๋ชจ๋ธ์— ๋„˜๊ธฐ์„ธ์š”.
## ํ•™์Šต
> ํ•™์Šต ๋ฐ์ดํ„ฐ ๊ตฌ์„ฑ, ๋ณด๊ธฐ ์ˆœ์„œ ์—ญ์ˆœ ํ•™์Šต, ์˜ค์—ผ ์ œ๊ฑฐ, ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ ๋งŒ๋“œ๋Š” ๋ฒ•, ํ™•๋ฅ  ๋ณด์ •, ์ฒดํฌํฌ์ธํŠธ๋ณ„ ์ •ํ™•๋„๊นŒ์ง€ ์ž์„ธํ•œ ๊ณผ์ •์€ **[TRAINING.md](TRAINING.md)**์— ์ •๋ฆฌํ–ˆ์Šต๋‹ˆ๋‹ค.
| ํ•ญ๋ชฉ | ๊ฐ’ |
|---|---|
| ์›๋ณธ ๋ชจ๋ธ | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) |
| ๋ฐฉ๋ฒ• | SFT, LoRA (r 16, ฮฑ 32, dropout 0.05, qยทkยทvยทoยทgateยทupยทdown) โ†’ ์›๋ณธ์— ๋ณ‘ํ•ฉ |
| ํ•™์Šต๋Ÿ‰ | 69,144๋ฌธํ•ญ ร— ๋ณด๊ธฐ ์ˆœ์„œ 2๊ฐ€์ง€(์„ž์€ ์ˆœ์„œ + ์—ญ์ˆœ), ์ถœ์ฒ˜ ๋น„์œจ๋Œ€๋กœ ์„ž์–ด 4,430 step ยท ๋ฐฐ์น˜ 32, ์ตœ๋Œ€ ๊ธธ์ด 2,048 |
| ์ตœ์ ํ™” | ํ•™์Šต๋ฅ  2e-4, cosine, warmup 3% |
| ์žฅ๋น„ | GPU 1์žฅ, ์•ฝ 3.5์‹œ๊ฐ„ |
| ์ •๋‹ต ํ˜•์‹ | ๋ณด๊ธฐ ๊ธฐํ˜ธ ํ•œ ๊ธ€์ž + `<|im_end|>` |
| ๋ณด์ • | ๊ฒ€์ฆ์…‹ 3,089๋ฌธํ•ญ์œผ๋กœ ๋‹จ์ผ ์˜จ๋„(1.7179)๋ฅผ ๋งž์ถค. ECE 0.066 โ†’ 0.013 |
**ํ•™์Šต ๋ฐ์ดํ„ฐ** (๋ชจ๋‘ ์œ„ ํ”„๋กฌํ”„ํŠธ ํ˜•์‹์œผ๋กœ ๋ฐ”๊ฟ” ์‚ฌ์šฉ, ํ‰๊ฐ€ ๋ฌธํ•ญ๊ณผ ๊ฒน์น˜๋Š” ๋ฌธ์žฅ์€ ์ œ์™ธ)
| ๋ฐ์ดํ„ฐ | ๋ผ์ด์„ ์Šค |
|---|---|
| KLUE NLI ยท YNAT (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
| KoBEST BoolQ ยท COPA ยท HellaSwag ยท SentiNeg ยท WiC (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
| kocohub/korean-hate-speech (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
| APEACH (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
| SIB-200 (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
| ์ •์ฑ… ํ–‰๋™ ยท ๋ฒ•๋ น ์งˆ๋ฌธ ยท ๋„๊ตฌ ํ˜ธ์ถœ ํŒ๋‹จ | ์ž์ฒด ํ•ฉ์„ฑ (๊ทœ์น™์œผ๋กœ ์ƒ์„ฑ, ์ •๋‹ต์€ ์ฝ”๋“œ๋กœ ๊ณ„์‚ฐ. ์ •์ฑ…ยท๋ฒ•๋ นยท๋„๊ตฌ๋Š” ๋ชจ๋‘ ์ง€์–ด๋‚ธ ๊ฒƒ์ด๋ฉฐ LLM ์ƒ์„ฑ๋ฌผ์„ ์“ฐ์ง€ ์•Š์Œ) |
## ๋ผ์ด์„ ์Šค์™€ ์ฃผ์˜
- ๊ณต๊ฐœ ํ•™์Šต ๋ฐ์ดํ„ฐ๋Š” ๋ชจ๋‘ CC BY-SA 4.0์ž…๋‹ˆ๋‹ค.
- [`judge.py`](judge.py)์˜ ๊ณผ์ œ๋ณ„ ์ง€์นจยท๋ณด๊ธฐ ๋ฌธ๊ตฌ(`TASKS`, `CRITERIA`)์™€ ํ‰๊ฐ€ ์Šคํฌ๋ฆฝํŠธ์˜ ์ง€์นจ ์ผ๋ถ€๋Š” [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)(DaehanKim)์˜ ๊ณผ์ œ ๋ฌธ๊ตฌ๋ฅผ ๊ทธ๋Œ€๋กœ ์“ด ๊ฒƒ์œผ๋กœ, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)์„ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
- `space/` ํด๋”(์‡ผ๋ฆด ํŽ˜์ด์ง€)๋Š” KoBESTยทKLUE ๋ฌธ์žฅ์„ ํฌํ•จํ•˜๊ณ  ์žˆ์–ด CC BY-SA 4.0์ž…๋‹ˆ๋‹ค.
- ํ˜์˜ค ํ‘œํ˜„ ํŒ์ •, ์ •์ฑ… ํŒ๋‹จ ๋“ฑ ์‚ฌ๋žŒ์—๊ฒŒ ์˜ํ–ฅ์„ ์ฃผ๋Š” ๊ฒฐ์ •์— ์“ธ ๋•Œ๋Š” ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฒฐ๊ณผ๋ฅผ ์‚ฌ๋žŒ์ด ๊ฒ€ํ† ํ•˜๋„๋ก ํ•˜์„ธ์š”.
- ๋ฒ•๋ฅ ยท๊ธˆ์œต ํŒ๋‹จ์˜ ๊ทผ๊ฑฐ๋กœ ์“ฐ์ง€ ๋งˆ์„ธ์š”. ์ฃผ์–ด์ง„ ๊ธ€ ์•ˆ์—์„œ ๋‚ด์šฉ์„ ์ฐพ๊ณ  ๋Œ€์กฐํ•˜๋Š” ๋„๊ตฌ์ž…๋‹ˆ๋‹ค.