coderecode95 commited on
Commit
01c9adf
ยท
verified ยท
1 Parent(s): af72c54

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +449 -448
README.md CHANGED
@@ -92,451 +92,452 @@ model-index:
92
  name: accuracy (0-shot
93
  n=983): null
94
  ---
95
-
96
- # KJH-Judge-4B-ko
97
-
98
- > [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)๋ฅผ ๋ณด๊ณ  ํŒ์ • ๋ชจ๋ธ์„ ์ง์ ‘ ๋งŒ๋“ค์–ด ๋ณธ **๊ฐœ์ธ ํ•™์Šต ํ”„๋กœ์ ํŠธ**์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์ˆ˜์น˜๋Š” ๋ชจ๋‘ ์ง์ ‘ ์žฐ ๊ฒฐ๊ณผ์ด๋ฉฐ, ์‹ค์„œ๋น„์Šค์—์„œ ๊ฒ€์ฆํ•œ ๋ชจ๋ธ์€ ์•„๋‹™๋‹ˆ๋‹ค.
99
-
100
- **์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณจ๋ผ, ๋ณด์ •๋œ ํ™•๋ฅ ๊ณผ ํ•จ๊ป˜ ๋Œ๋ ค์ฃผ๋Š” ํ•œ๊ตญ์–ด ํŒ์ • ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.** ๊ฐ์ •ยท์ฃผ์ œ ๋ถ„๋ฅ˜, ๋ฌธ์žฅ ๊ด€๊ณ„(ํ•จ์˜ยท๋ชจ์ˆœ), ํ˜์˜ค ํ‘œํ˜„ ๊ฒ€์ˆ˜, ์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ ์„ ํƒ, ์—์ด์ „ํŠธ์˜ ๋„๊ตฌ ํ˜ธ์ถœ ํŒ๋‹จ(๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€) ๊ฐ™์€ ์ผ์— ์”๋‹ˆ๋‹ค. ๋‹ค๋ฅธ ๋ชจ๋ธ์˜ ๋‹ต๋ณ€์„ ์ฑ„์ ํ•˜๋Š” LLM-as-a-judge ๋ชจ๋ธ์ด ์•„๋‹™๋‹ˆ๋‹ค.
101
-
102
- > **English summary.** A personal learning project: a Korean decision model fine-tuned from Qwen3-4B. Given a task description, an item, and 2โ€“26 options, it returns the chosen option with **calibrated probabilities** from a single forward pass (no text generation, ~tens of ms per decision on a GPU). Good for classification, NLI, moderation, policy actions, and tool-call routing. It does not generate text, extract spans, or do numeric reasoning.
103
-
104
- ![ํ•™์Šตํ•˜์ง€ ์•Š์€ ํ•œ๊ตญ์–ด ๋ฒค์น˜๋งˆํฌ](assets/benchmark.png)
105
-
106
- ## ์ด ๋ชจ๋ธ์˜ ํŠน์ง•
107
-
108
- - **๋ณด๊ธฐ ๊ณ ๋ฅด๊ธฐ๋งŒ ํ•ฉ๋‹ˆ๋‹ค.** ๊ธ€์„ ์ƒ์„ฑํ•˜์ง€ ์•Š๊ณ  ํ”„๋กฌํ”„ํŠธ๋ฅผ ํ•œ ๋ฒˆ ๊ณ„์‚ฐํ•œ ๋’ค, ๋ณด๊ธฐ ๊ธฐํ˜ธ(A, B, โ€ฆ) ํ† ํฐ์˜ ์ ์ˆ˜๋งŒ ์ฝ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ๋น ๋ฅด๊ณ (GPU ์„œ๋ฒ„ ์•ˆ ๊ณ„์‚ฐ ๊ธฐ์ค€ ํŒ์ • ํ•œ ๋ฒˆ ์•ฝ 90ms), ๋‹ต์ด ํ•ญ์ƒ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค.
109
- - **ํ™•๋ฅ ์ด ๋ณด์ •๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.** ๊ฒ€์ฆ์…‹์œผ๋กœ ๋งž์ถ˜ ์˜จ๋„(1.72)๋กœ ๋ณด์ •ํ•ด, Korean JEV Benchmark์—์„œ ๊ธฐ๋Œ€ ๋ณด์ • ์˜ค์ฐจ(ECE)๊ฐ€ 0.027์ž…๋‹ˆ๋‹ค. โ€œํ™•์‹  70% ๋ฏธ๋งŒ์ด๋ฉด ์‚ฌ๋žŒ์—๊ฒŒ ๋„˜๊ธฐ๊ธฐโ€ ๊ฐ™์€ ๊ธฐ์ค€์„ ๊ฑธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ ํ•™์Šตํ•˜์ง€ ์•Š์€ ์ถœ์ฒ˜์˜ ๋ฐ์ดํ„ฐ์—์„œ๋Š” ํ‰๊ท  ํ™•์‹ ์ด ์ •ํ™•๋„๋ณด๋‹ค ์ตœ๋Œ€ 10%p ์ •๋„ ๋†’๊ฒŒ ๋‚˜์™”์œผ๋‹ˆ, ๊ธฐ์ค€๊ฐ’์€ ์“ฐ๋ ค๋Š” ๋ฐ์ดํ„ฐ์—์„œ ๋‹ค์‹œ ๋งž์ถ”๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
110
- - **๋ณด๊ธฐ ์ˆœ์„œ์— ํ”๋“ค๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.** ๋ณด๊ธฐ๋ฅผ ์ •ํ™•ํžˆ ๊ฑฐ๊พธ๋กœ ๋†“๊ณ  ๋‹ค์‹œ ๋ฌผ์–ด๋„ 97.9%๊ฐ€ ๊ฐ™์€ ๋‹ต์ž…๋‹ˆ๋‹ค.
111
-
112
- ## ๋น ๋ฅด๊ฒŒ ์‹œ์ž‘ํ•˜๊ธฐ
113
-
114
- ```bash
115
- pip install torch "transformers>=5" accelerate
116
- ```
117
-
118
- ```python
119
- from transformers import pipeline
120
-
121
- judge = pipeline(model="coderecode95/KJH-Judge-4B-ko", trust_remote_code=True,
122
- device=0, dtype="bfloat16") # GPU๊ฐ€ ์—†์œผ๋ฉด device, dtype ๋ฅผ ๋นผ์„ธ์š” (๋งค์šฐ ๋А๋ฆผ)
123
- ```
124
-
125
- ์“ฐ๋Š” ๋ฐฉ๋ฒ•์€ ์„ธ ๊ฐ€์ง€์ด๊ณ , **๊ฒฐ๊ณผ๋Š” ๋ชจ๋‘ ๊ฐ™์€ ๋ชจ์–‘**์ž…๋‹ˆ๋‹ค.
126
-
127
- ```python
128
- {'choice': '๊ธ์ •', 'confidence': 0.97, 'probabilities': {'๊ธ์ •': 0.97, '๋ถ€์ •': 0.03}}
129
- # ๊ณ ๋ฅธ ๋ณด๊ธฐ ๊ทธ ๋ณด๊ธฐ์˜ ํ™•๋ฅ (๋ณด์ •๋จ) ๋ณด๊ธฐ๋ณ„ ํ™•๋ฅ 
130
- ```
131
-
132
- #### 1. ์ž์ฃผ ์“ฐ๋Š” ํŒ์ • โ€” ๋ฉ”์„œ๋“œ ํ•˜๋‚˜๋กœ
133
-
134
- ํ•™์Šต ๋•Œ์™€ ๊ฐ™์€ ์ง€์นจยท๋ณด๊ธฐ ๋ฌธ๊ตฌ๊ฐ€ ์ž๋™์œผ๋กœ ๋“ค์–ด๊ฐ€์„œ, ๋”ฐ๋กœ ํ”„๋กฌํ”„ํŠธ๋ฅผ ์“ธ ํ•„์š”๊ฐ€ ์—†๊ณ  ์ •ํ™•๋„๋„ ๊ฐ€์žฅ ์ž˜ ๋‚˜์˜ต๋‹ˆ๋‹ค.
135
-
136
- ```python
137
- judge.sentiment("๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”") # โ†’ ๊ธ์ •
138
- judge.nli("๋ณด์ฆ๊ธˆ์€ ํ‡ด๊ฑฐ ํ›„ 30์ผ ์ด๋‚ด ๋ฐ˜ํ™˜ํ•œ๋‹ค.", "๋ณด์ฆ๊ธˆ์€ ํ‡ด๊ฑฐ 40์ผ ๋’ค์— ๋ฐ›๋Š”๋‹ค.") # โ†’ contradiction
139
- judge.next_action("๊ฐ•๋‚จ์—ญ์—์„œ ์„œ์šธ์—ญ๊นŒ์ง€ ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜", tools) # โ†’ CALL:call_taxi
140
- ```
141
-
142
- | ๋ฉ”์„œ๋“œ | ํŒ์ • | ๊ณ ๋ฅด๋Š” ๊ฐ’ |
143
- |---|---|---|
144
- | `sentiment(๋ฌธ์žฅ)` | ๊ฐ์ • | ๊ธ์ • / ๋ถ€์ • |
145
- | `topic(๋‰ด์Šค ์ œ๋ชฉ)` | ๋‰ด์Šค ์ฃผ์ œ | ์‚ฌํšŒ / IT๊ณผํ•™ / ์„ธ๊ณ„ / ์ƒํ™œ๋ฌธํ™” / ์ •์น˜ / ์Šคํฌ์ธ  / ๊ฒฝ์ œ |
146
- | `hate(๋ฌธ์žฅ)` | ํ˜์˜คยท๋น„ํ•˜ ํ‘œํ˜„ | ๋น„ํ˜์˜ค / ํ˜์˜ค |
147
- | `offensive(๋Œ“๊ธ€, news_title=None)` | ๋Œ“๊ธ€ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„ | ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค |
148
- | `nli(์ „์ œ, ๊ฐ€์„ค)` | ๋ฌธ์žฅ ๊ด€๊ณ„ | entailment / contradiction / neutral |
149
- | `similar(๋ฌธ์žฅ1, ๋ฌธ์žฅ2)` | ๊ฐ™์€ ๋œป์ธ์ง€ | ๊ฐ™์Œ / ๋‹ค๋ฆ„ |
150
- | `yes_no(์งˆ๋ฌธ, ์ง€๋ฌธ)` | ์ง€๋ฌธ ๊ทผ๊ฑฐ ์˜ˆ/์•„๋‹ˆ์˜ค | ์˜ˆ / ์•„๋‹ˆ์˜ค |
151
- | `same_meaning(๋‹จ์–ด, ๋ฌธ๋งฅ1, ๋ฌธ๋งฅ2)` | ๋‹จ์–ด ๋œป์ด ๊ฐ™์€์ง€ | ๊ฐ™์Œ / ๋‹ค๋ฆ„ |
152
- | `reading(์ง€๋ฌธ, ์งˆ๋ฌธ, ๋ณด๊ธฐ)` | ๋…ํ•ด ๊ฐ๊ด€์‹ | ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜ |
153
- | `next_action(๋Œ€ํ™”, ๋„๊ตฌ ๋ชฉ๋ก)` | ์—์ด์ „ํŠธ ๋‹ค์Œ ํ–‰๋™ | CALL:<๋„๊ตฌ> / ASK / RESPOND |
154
- | `policy_action(์ •์ฑ…, ๊ณ ๊ฐ ์š”์ฒญ)` | ์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ | ANSWER / ASK_CLARIFICATION / ASK_CONFIRMATION / CALL_TOOL / CALL_MULTIPLE_TOOLS / ESCALATE / REFUSE |
155
-
156
- ๋ฌธ์žฅ ํ•˜๋‚˜๋งŒ ๋ฐ›๋Š” ๋ฉ”์„œ๋“œ(`sentiment`, `topic`, `hate`, `nli` ๋“ฑ)๋Š” **๋ฆฌ์ŠคํŠธ๋ฅผ ๋„ฃ์œผ๋ฉด ์—ฌ๋Ÿฌ ๊ฐœ๋ฅผ ํ•œ ๋ฒˆ์—** ํŒ์ •ํ•ฉ๋‹ˆ๋‹ค.
157
-
158
- #### 2. ๋‚ด ํŒ์ • ๋งŒ๋“ค๊ธฐ โ€” JEV ํ˜•์‹
159
-
160
- [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)ยทJEV API์™€ ๊ฐ™์€ ๋ชจ์–‘์ž…๋‹ˆ๋‹ค. **์ƒํ™ฉ(`state`) ํ•˜๋‚˜์— ์ด๋ฆ„ ๋ถ™์ธ ์งˆ๋ฌธ ์—ฌ๋Ÿฌ ๊ฐœ**๋ฅผ ํ•œ ๋ฒˆ์— ๋ฌป๊ณ , ๋ณด๊ธฐ์—๋Š” **๋‚ด๊ฐ€ ์“ธ ID**๋ฅผ ๋ถ™์ž…๋‹ˆ๋‹ค. ๊ฒฐ๊ณผ๋Š” ๊ทธ ID๋กœ ๋Œ์•„์˜ต๋‹ˆ๋‹ค.
161
-
162
- ```python
163
- judge.decide(
164
- state={"๊ณ ๊ฐ ๋ฌธ์˜": "์ง€๋‚œ๋‹ฌ ๏ฟฝ๏ฟฝ๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ์ฒ˜๋ฆฌํ•ด ์ฃผ์„ธ์š”."},
165
- questions={
166
- "๋ถ€์„œ": {
167
- "instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.",
168
- "criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ"},
169
- },
170
- "๊ธด๊ธ‰": {
171
- "instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๋Š”์ง€ ํŒ๋‹จํ•œ๋‹ค.",
172
- "criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ์š”๊ตฌ", "normal": "์ผ๋ฐ˜ ๋ฌธ์˜"},
173
- },
174
- },
175
- )
176
- # {'๋ถ€์„œ': {'choice': 'billing', 'confidence': 0.9โ€ฆ, 'probabilities': {...}},
177
- # '๊ธด๊ธ‰': {'choice': 'urgent', 'confidence': 0.9โ€ฆ, 'probabilities': {...}}}
178
- ```
179
-
180
- - **`state`**: ํŒ๋‹จํ•  ๋‚ด์šฉ. `{"์ด๋ฆ„": ๊ฐ’}`์ด ํ”„๋กฌํ”„ํŠธ์— ํ•œ ์ค„์”ฉ ๋“ค์–ด๊ฐ‘๋‹ˆ๋‹ค. ๋ฌธ์žฅ ํ•˜๋‚˜๋ฉด ๊ทธ๋ƒฅ ๋ฌธ์ž์—ด๋กœ ๋„ฃ์–ด๋„ ๋ฉ๋‹ˆ๋‹ค.
181
- - **`instructions`**: ๋ฌด์—‡์„ ์–ด๋–ค ๊ธฐ์ค€์œผ๋กœ ๊ณ ๋ฅผ์ง€. ํŒ๋‹จ ๊ธฐ์ค€์„ ํ•œ๋‘ ๋ฌธ์žฅ ๋ง๋ถ™์ด๋ฉด(์˜ˆ: โ€œ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ํฌ๊ฒŒ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
182
- - **`criteria`**: `{ID: ์„ค๋ช…}`, 2~26๊ฐœ. ID๋Š” ๊ฒฐ๊ณผ๋กœ ๋Œ๋ ค๋ฐ›๋Š” ๊ฐ’์ด๊ณ , ๋ชจ๋ธ์€ ์„ค๋ช…์„ ๋ณด๊ณ  ๊ณ ๋ฆ…๋‹ˆ๋‹ค. ID๊ฐ€ ํ•„์š” ์—†์œผ๋ฉด `["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก"]`์ฒ˜๋Ÿผ ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ์–ด๋„ ๋ฉ๋‹ˆ๋‹ค.
183
-
184
- #### 3. ํ•œ ์ค„๋กœ
185
-
186
- ```python
187
- judge("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”", choices=["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก", "๊ธฐํƒ€"],
188
- instruction="๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.")
189
- ```
190
-
191
- `judge.decide`์—์„œ ์งˆ๋ฌธ์ด ํ•˜๋‚˜์ผ ๋•Œ์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค. ๋ฆฌ์ŠคํŠธ๋ฅผ ๋„ฃ๊ณ  `batch_size=8`์ฒ˜๋Ÿผ ์ฃผ๋ฉด ๋ฌถ์–ด์„œ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค.
192
-
193
- ---
194
-
195
- `trust_remote_code=True`๋Š” ์ด ์ €์žฅ์†Œ์˜ [`judge_pipeline.py`](judge_pipeline.py)์™€ [`judge.py`](judge.py)(ํ”„๋กฌํ”„ํŠธ ๋งŒ๋“ค๊ธฐ์™€ ๋ณด๊ธฐ ์ ์ˆ˜ ๊ณ„์‚ฐ)๋ฅผ ์‹คํ–‰ํ•œ๋‹ค๋Š” ๋œป์ž…๋‹ˆ๋‹ค. ์›๊ฒฉ ์ฝ”๋“œ๋ฅผ ์“ฐ๊ณ  ์‹ถ์ง€ ์•Š์œผ๋ฉด `judge.py`๋ฅผ ๋‚ด๋ ค๋ฐ›์•„ `from judge import Judge; judge = Judge()`๋กœ ๋˜‘๊ฐ™์ด ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. [`quickstart.ipynb`](quickstart.ipynb)์— ์˜ˆ์‹œ๋ฅผ ๋ชจ์•„ ๋‘์—ˆ์Šต๋‹ˆ๋‹ค.
196
-
197
- ## ์‚ฌ์šฉ ์˜ˆ์‹œ
198
-
199
- ์ด ๋ชจ๋ธ์€ **โ€œ๋ฌด์—‡์„ ํ• ์ง€โ€๋ฅผ ๋น ๋ฅด๊ฒŒ ์ •ํ•˜๋Š” ์•ž๋‹จ**์— ๋‘๋Š” ๊ฒƒ์ด ๊ฐ€์žฅ ์ž˜ ๋งž์Šต๋‹ˆ๋‹ค. ์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณ ๋ฅด๋Š” ํŒ๋‹จ์€ ์ด ๋ชจ๋ธ์ด ํ•˜๊ณ , ๊ธ€ ์“ฐ๊ธฐ๋‚˜ ๊ฐ’ ์ฑ„์šฐ๊ธฐ์ฒ˜๋Ÿผ ์ƒ์„ฑ์ด ํ•„์š”ํ•œ ์ผ๋งŒ ํฐ ๋ชจ๋ธ์ด๋‚˜ ์‚ฌ๋žŒ์—๊ฒŒ ๋„˜๊ธฐ๋Š” ๊ตฌ์„ฑ์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์˜ˆ์‹œ๋Š” ๋ชจ๋‘ ์œ„์—์„œ ๋งŒ๋“  `judge`๋กœ ๋ฐ”๋กœ ๋Œ์•„๊ฐ‘๋‹ˆ๋‹ค. ์ฃผ์„์— ์ ์€ ๊ฒฐ๊ณผ๋Š” ์˜๋„ํ•œ ์ •๋‹ต์ด๊ณ , ์‹ค์ œ ํ™•๋ฅ ์€ ์ง์ ‘ ์‹คํ–‰ํ•ด ํ™•์ธํ•˜์„ธ์š”.
200
-
201
- ### 1. ๊ณ ๊ฐ ๋ฌธ์˜ ๋ถ„๋ฅ˜ โ€” ๋ถ€์„œ์™€ ๊ธด๊ธ‰๋„๋ฅผ ํ•œ ๋ฒˆ์—
202
-
203
- ๋ฌธ์˜ ํ•˜๋‚˜๋ฅผ ์—ฌ๋Ÿฌ ๊ด€์ ์œผ๋กœ ๋™์‹œ์— ๋ถ„๋ฅ˜ํ•ฉ๋‹ˆ๋‹ค. ์งˆ๋ฌธ๋งˆ๋‹ค ๋ณด๊ธฐ ID๋ฅผ ์ •ํ•ด ๋‘๋ฉด ๊ฒฐ๊ณผ๋ฅผ ๋ฐ”๋กœ ์—…๋ฌด ์ฝ”๋“œ์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
204
-
205
- ```python
206
- def route_ticket(text):
207
- r = judge.decide(
208
- state={"๊ณ ๊ฐ ๋ฌธ์˜": text},
209
- questions={
210
- "๋ถ€์„œ": {
211
- "instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค. ์—ฌ๋Ÿฌ ๋‚ด์šฉ์ด ์„ž์—ฌ ์žˆ์œผ๋ฉด ๊ณ ๊ฐ์ด ํ•ด๊ฒฐ์„ ์›ํ•˜๋Š” ๋ฌธ์ œ๋ฅผ ์šฐ์„ ํ•œ๋‹ค.",
212
- "criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ",
213
- "product": "์ œํ’ˆ ์‚ฌ์šฉยท๊ณ ์žฅ", "other": "๊ธฐํƒ€"},
214
- },
215
- "๊ธด๊ธ‰": {
216
- "instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๊ฑฐ๋‚˜ ๊ธˆ์ „ ํ”ผํ•ด๊ฐ€ ์ง„ํ–‰ ์ค‘์ธ์ง€ ํŒ๋‹จํ•œ๋‹ค.",
217
- "criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ํ•„์š”", "normal": "์ผ๋ฐ˜ ์ฒ˜๋ฆฌ"},
218
- },
219
- },
220
- )
221
- return r["๋ถ€์„œ"]["choice"], r["๊ธด๊ธ‰"]["choice"]
222
-
223
- route_ticket("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ํ™˜๋ถˆํ•ด ์ฃผ์„ธ์š”.") # ('billing', 'urgent')
224
- ```
225
-
226
- - ์ง€์นจ์— **ํ—ท๊ฐˆ๋ฆฌ๋Š” ๊ฒฝ์šฐ์˜ ๊ธฐ์ค€**์„ ํ•œ ๋ฌธ์žฅ ๋„ฃ์œผ๋ฉด(โ€œ์—ฌ๋Ÿฌ ๋‚ด์šฉ์ด ์„ž์—ฌ ์žˆ์œผ๋ฉด โ€ฆ์„ ์šฐ์„ ํ•œ๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
227
- - ๋ณด๊ธฐ์— `๊ธฐํƒ€`๋ฅผ ๋‘๋ฉด ์–ด๋””์—๋„ ์•ˆ ๋งž๋Š” ๋ฌธ์˜๋ฅผ ์–ต์ง€๋กœ ๋ผ์›Œ ๋„ฃ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
228
-
229
- ### 2. ๋ฆฌ๋ทฐ ๊ฐ์ • ๋ถ„์„ โ€” ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ์™€ ์‚ฌ๋žŒ ๊ฒ€ํ†  ๊ธฐ์ค€
230
-
231
- ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ์œผ๋ฉด ๋ฌถ์–ด์„œ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ๋”ฐ๋กœ ๋ชจ์•„ ์‚ฌ๋žŒ์ด ๋ด…๋‹ˆ๋‹ค.
232
-
233
- ```python
234
- reviews = ["๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”", "๊ฐ€๊ฒฉ ๋Œ€๋น„ ๊ดœ์ฐฎ์•„์š”", "๋‹ค์‹œ๋Š” ์•ˆ ์‚ฝ๋‹ˆ๋‹ค", "๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”"]
235
- results = judge.sentiment(reviews)
236
-
237
- auto = [(t, r["choice"]) for t, r in zip(reviews, results) if r["confidence"] >= 0.7]
238
- review = [t for t, r in zip(reviews, results) if r["confidence"] < 0.7] # ์‚ฌ๋žŒ์ด ํ™•์ธ
239
- ```
240
-
241
- - **๋ฐ˜์–ดยท๋น„๊ผผ**(โ€œ๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”โ€)์€ ๋†’์€ ํ™•์‹ ์œผ๋กœ ํ‹€๋ฆฌ๊ธฐ๋„ ํ•ฉ๋‹ˆ๋‹ค. ์ค‘์š”ํ•œ ๊ณณ์—๋Š” ํ‘œ๋ณธ์„ ์‚ฌ๋žŒ์ด ๋‹ค์‹œ ๋ณด์„ธ์š”.
242
- - ๊ธฐ์ค€๊ฐ’ 0.7์€ ์˜ˆ์‹œ์ž…๋‹ˆ๋‹ค. ์‹ค์ œ ๋ฐ์ดํ„ฐ ์ˆ˜๋ฐฑ ๊ฑด์— ์ •๋‹ต์„ ๋‹ฌ์•„ ๋ณด๊ณ  โ€œ์ด ๊ธฐ์ค€ ์ด์ƒ์ด๋ฉด ๋ช‡ %๊ฐ€ ๋งž๋Š”์ง€โ€๋ฅผ ํ™•์ธํ•ด ์ •ํ•˜์„ธ์š”.
243
-
244
- ### 3. ๋ฆฌ๋ทฐ๋ฅผ ์—ฌ๋Ÿฌ ์ธก๋ฉด์œผ๋กœ โ€” ๋ฐฐ์†กยทํ’ˆ์งˆยท๊ฐ€๊ฒฉ ๊ฐ๊ฐ
245
-
246
- ํ•œ ๏ฟฝ๏ฟฝ๋ทฐ์—์„œ ์ธก๋ฉด๋ณ„๋กœ ๋”ฐ๋กœ ํŒ๋‹จํ•ฉ๋‹ˆ๋‹ค. ๊ฐ™์€ `state`์— ์งˆ๋ฌธ๋งŒ ๋Š˜๋ฆฌ๋ฉด ๋ฉ๋‹ˆ๋‹ค.
247
-
248
- ```python
249
- ASPECT = {"pos": "๊ธ์ •์ ์œผ๋กœ ์–ธ๊ธ‰", "neg": "๋ถ€์ •์ ์œผ๋กœ ์–ธ๊ธ‰", "none": "์–ธ๊ธ‰ํ•˜์ง€ ์•Š์Œ"}
250
-
251
- judge.decide(
252
- state={"๋ฆฌ๋ทฐ": "๋ฐฐ์†ก์€ ํ•˜๋ฃจ ๋งŒ์— ์™”๋Š”๋ฐ ๋งˆ๊ฐ์ด ์—‰๋ง์ด๋„ค์š”. ์ด ๊ฐ€๊ฒฉ์ด๋ฉด ๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”."},
253
- questions={
254
- "๋ฐฐ์†ก": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ๋ฐฐ์†ก(์†๋„ยทํฌ์žฅ)์„ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค.", "criteria": ASPECT},
255
- "ํ’ˆ์งˆ": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ์ œํ’ˆ ํ’ˆ์งˆ(๋งˆ๊ฐยท๋‚ด๊ตฌ์„ฑยท์„ฑ๋Šฅ)์„ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค.", "criteria": ASPECT},
256
- "๊ฐ€๊ฒฉ": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ๊ฐ€๊ฒฉยท๊ฐ€์„ฑ๋น„๋ฅผ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค. ์• ๋งคํ•œ ํ‰๊ฐ€๋Š” ์–ธ๊ธ‰ํ•œ ๊ฒƒ์œผ๋กœ ๋ณด๋˜ ๊ธ์ •ยท๋ถ€์ •์„ ๋ฌธ๋งฅ์œผ๋กœ ํŒ๋‹จํ•œ๋‹ค.", "criteria": ASPECT},
257
- },
258
- )
259
- # {'๋ฐฐ์†ก': {'choice': 'pos', โ€ฆ}, 'ํ’ˆ์งˆ': {'choice': 'neg', โ€ฆ}, '๊ฐ€๊ฒฉ': {โ€ฆ}}
260
- ```
261
-
262
- ### 4. ๋Œ“๊ธ€ ๊ฒ€์ˆ˜ โ€” ํ˜์˜ค ์—ฌ๋ถ€์™€ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€
263
-
264
- `hate`๋กœ ์ฐจ๋ณ„ยท๋น„ํ•˜ ์—ฌ๋ถ€๋ฅผ ๊ฑฐ๋ฅด๊ณ , `offensive`๋กœ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€์„ ๋‚˜๋ˆ•๋‹ˆ๋‹ค. ๊ธฐ์‚ฌ ๋Œ“๊ธ€์ด๋ฉด ์ œ๋ชฉ์„ ๋ฌธ๋งฅ์œผ๋กœ ํ•จ๊ป˜ ๋„ฃ์Šต๋‹ˆ๋‹ค.
265
-
266
- ```python
267
- comment = "์ด๋Ÿฐ ๊ธฐ์‚ฌ ์“ฐ๋Š” ๊ธฐ์ž๋Š” ๋จธ๋ฆฌ๊ฐ€ ์žˆ๋Š” ๊ฑด์ง€"
268
-
269
- judge.hate(comment) # ๋น„ํ˜์˜ค / ํ˜์˜ค
270
- judge.offensive(comment, news_title="์ •๋ถ€, ๋‚ด๋…„ ์˜ˆ์‚ฐ์•ˆ ๋ฐœํ‘œ") # ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค
271
- ```
272
-
273
- - ๋‘ ๋ฉ”์„œ๋“œ๋Š” ๊ธฐ์ค€์ด ๋‹ค๋ฆ…๋‹ˆ๋‹ค. `hate`๋Š” ์„ฑ๋ณ„ยท๋‚˜์ดยท์ถœ์‹  ๊ฐ™์€ ์ง‘๋‹จ์— ๋Œ€ํ•œ ์ฐจ๋ณ„ยท๋น„ํ•˜(APEACH ๊ธฐ์ค€), `offensive`๋Š” ๊ฐœ์ธ์„ ํ–ฅํ•œ ๋ชจ์š•๊นŒ์ง€ ํฌํ•จํ•œ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„(BEEP ๊ธฐ์ค€)์ž…๋‹ˆ๋‹ค.
274
- - ์‚ฌ๋žŒ์—๊ฒŒ ๋ถˆ์ด์ต์„ ์ฃผ๋Š” ๊ฒฐ์ •(์‚ญ์ œยท์ œ์žฌ)์— ์“ธ ๋•Œ๋Š” ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ๋ฐ˜๋“œ์‹œ ์‚ฌ๋žŒ์ด ๊ฒ€ํ† ํ•˜๊ฒŒ ํ•˜์„ธ์š”.
275
-
276
- ### 5. ์—์ด์ „ํŠธ ๋ผ์šฐํ„ฐ โ€” ๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€
277
-
278
- ์—์ด์ „ํŠธ๊ฐ€ ๋งค ํ„ด ํฐ ๋ชจ๋ธ์— โ€œ๋‹ค์Œ์— ๋ญ˜ ํ• ์ง€โ€๋ฅผ ๋ฌป๋Š” ๋Œ€์‹ , ์ด ๋ชจ๋ธ์ด ๋จผ์ € ์ •ํ•ฉ๋‹ˆ๋‹ค. ๋„๊ตฌ ์ธ์ž ์ฑ„์šฐ๊ธฐ์™€ ๋‹ต๋ณ€ ์ž‘์„ฑ๋งŒ ํฐ ๋ชจ๋ธ์— ๋งก๊น๋‹ˆ๋‹ค.
279
-
280
- ```python
281
- tools = [
282
- {"type": "function", "function": {"name": "call_taxi", "description": "ํƒ์‹œ๋ฅผ ํ˜ธ์ถœํ•œ๋‹ค",
283
- "parameters": {"type": "object", "properties": {"pickup": {"type": "string"}, "destination": {"type": "string"}},
284
- "required": ["pickup", "destination"]}}},
285
- {"type": "function", "function": {"name": "get_weather", "description": "์ง€์—ญ์˜ ํ˜„์žฌ ๋‚ ์”จ๋ฅผ ์•Œ๋ ค์ค€๋‹ค",
286
- "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}},
287
- ]
288
- messages = [{"role": "user", "content": "๊ฐ•๋‚จ์—ญ์—์„œ ์„œ์šธ์—ญ๊นŒ์ง€ ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜"}]
289
-
290
- r = judge.next_action(messages, tools)
291
- if r["confidence"] < 0.7:
292
- action = "ํฐ ๋ชจ๋ธ์—๊ฒŒ ๋‹ค์Œ ํ–‰๋™์„ ๋งก๊น€"
293
- elif r["choice"].startswith("CALL:"):
294
- action = f"ํฐ ๋ชจ๋ธ๋กœ {r['choice'][5:]} ์˜ ์ธ์ž๋ฅผ ์ฑ„์›Œ ํ˜ธ์ถœ" # ์ด ๋ชจ๋ธ์€ ์ธ์ž๋ฅผ ์ฑ„์šฐ์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค
295
- elif r["choice"] == "ASK":
296
- action = "๋น ์ง„ ํ•„์ˆ˜ ์ •๋ณด๋ฅผ ์‚ฌ์šฉ์ž์—๊ฒŒ ๋˜๋ฌป๊ธฐ"
297
- else: # RESPOND
298
- action = "๋„๊ตฌ ์—†์ด ๋ฐ”๋กœ ๋‹ต๋ณ€"
299
- ```
300
-
301
- - ๋„๊ตฌ๋Š” **OpenAI ํ˜•์‹ ๊ทธ๋Œ€๋กœ**, ํŠนํžˆ `required`๋ฅผ ๊ผญ ๋„ฃ์œผ์„ธ์š”. ์ด๋ฆ„๋งŒ ์ฃผ๋ฉด ํ•„์ˆ˜ ์ธ์ž๋ฅผ ๋ณผ ์ˆ˜ ์—†์–ด ASK์™€ CALL์„ ๊ตฌ๋ณ„ํ•˜์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค.
302
- - ๋Œ€ํ™” ๊ธฐ๋ก(์•ž ํ„ด, ๋„๊ตฌ ๊ฒฐ๊ณผ `{"role": "tool", โ€ฆ}`)์„ ํ•จ๊ป˜ ๋„ฃ์œผ๋ฉด โ€œ์ด๋ฏธ ์ฒ˜๋ฆฌ๋์œผ๋‹ˆ ๊ฒฐ๊ณผ ์•ˆ๋‚ด(RESPOND)โ€ ๊ฐ™์€ ํŒ๋‹จ๋„ ํ•ฉ๋‹ˆ๋‹ค.
303
- - ๋„๊ตฌ๋Š” ์ตœ๋Œ€ 24๊ฐœ๊นŒ์ง€ ๋„ฃ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค (๋ณด๊ธฐ 26๊ฐœ ์ค‘ 2๊ฐœ๊ฐ€ ASKยทRESPOND).
304
-
305
- ### 6. ์ •์ฑ…์— ๋”ฐ๋ฅธ ๊ณ ๊ฐ์„ผํ„ฐ ํ–‰๋™ ๊ฒฐ์ •
306
-
307
- ์ •์ฑ… ๋ฌธ์„œ์™€ ๊ณ ๊ฐ ์š”์ฒญ์„ ์ฃผ๋ฉด ์‹คํ–‰ / ํ™•์ธ / ๋˜๋ฌป๊ธฐ / ์ด๊ด€ / ๊ฑฐ์ ˆ ์ค‘ ๋ฌด์—‡์„ ํ• ์ง€ ๊ณ ๋ฆ…๋‹ˆ๋‹ค. ์˜ˆ์™ธ ์กฐํ•ญ์ด ์ ์šฉ๋˜๋Š”์ง€๊นŒ์ง€ ๋”ฐ์ง‘๋‹ˆ๋‹ค.
308
-
309
- ```python
310
- policy = ("ํ•ด์™ธ ์นด๋“œ ์Šน์ธ ๊ธˆ์•ก์ด 100,000์› ์ด์ƒ์ด๋ฉด ๊ณ ๊ฐ์˜ ํ™•์ธ์„ ๋ฐ›์€ ํ›„์—๋งŒ ์‹คํ–‰ํ•œ๋‹ค. "
311
- "๋‹ค๋งŒ VIP ๋“ฑ๊ธ‰ ๊ณ ๊ฐ์€ ํ™•์ธ ์—†์ด ๋ฐ”๋กœ ์‹คํ–‰ํ•œ๋‹ค.")
312
-
313
- judge.policy_action(policy, "ํ™์ฝฉ์—์„œ ์นด๋“œ๋กœ 149,000์› ๊ฒฐ์ œํ•˜๋ ค๋Š”๋ฐ ์Šน์ธํ•ด์ค˜. ๋‚ด ๋“ฑ๊ธ‰์€ ์ผ๋ฐ˜์ด์•ผ.")
314
- # โ†’ ASK_CONFIRMATION (๊ธˆ์•ก ๊ธฐ์ค€์„ ๋„˜์—ˆ๊ณ  VIP ์˜ˆ์™ธ๊ฐ€ ์•„๋‹˜)
315
- ```
316
-
317
- - ์ •์ฑ…์— **์ ํ˜€ ์žˆ๋Š” ๊ฒƒ๋งŒ** ๊ทผ๊ฑฐ๋กœ ํŒ๋‹จํ•˜๋„๋ก ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค. ์‹ค์ œ ์—…๋ฌด ๊ทœ์ •์„ ๋ชจ๋ธ์ด ์•Œ์•„์„œ ๋ณด์ถฉํ•˜์ง€ ์•Š์œผ๋‹ˆ, ํ•„์š”ํ•œ ์กฐ๊ฑด์€ ์ •์ฑ… ๋ฌธ์žฅ์— ๋‹ค ์“ฐ์„ธ์š”.
318
- - ๋ณด๊ธฐ: `ANSWER`, `ASK_CLARIFICATION`, `ASK_CONFIRMATION`, `CALL_TOOL`, `CALL_MULTIPLE_TOOLS`, `ESCALATE`, `REFUSE`.
319
-
320
- ### 7. ๋‹ต๋ณ€์ด ๊ทผ๊ฑฐ ๋ฌธ์„œ์™€ ๋งž๋Š”์ง€ ํ™•์ธ (RAG ๊ฒ€์ฆ)
321
-
322
- ๊ฒ€์ƒ‰ํ•œ ๋ฌธ์„œ๋ฅผ ์ „์ œ๋กœ, ์ƒ์„ฑ๋œ ๋‹ต๋ณ€์˜ ๋ฌธ์žฅ์„ ๊ฐ€์„ค๋กœ ๋„ฃ์Šต๋‹ˆ๋‹ค. `contradiction`์ด๋ฉด ๋ฌธ์„œ์™€ ์–ด๊ธ‹๋‚œ ๋‹ต์ด๊ณ , `neutral`์ด๋ฉด ๋ฌธ์„œ์— ๊ทผ๊ฑฐ๊ฐ€ ์—†๋Š” ๋‚ด์šฉ์ž…๋‹ˆ๋‹ค.
323
-
324
- ```python
325
- doc = "์ž„๋Œ€์ธ์€ ๊ณ„์•ฝ์ด ๋๋‚˜๊ณ  ์ž„์ฐจ์ธ์ด ์ง‘์„ ๋น„์šด ๋‚ ๋ถ€ํ„ฐ 30์ผ ์ด๋‚ด์— ๋ณด์ฆ๊ธˆ์„ ๋Œ๋ ค์ค€๋‹ค."
326
- answer_sentences = ["๋ณด์ฆ๊ธˆ์€ ์ง‘์„ ๋น„์šด ๋’ค 30์ผ ์•ˆ์— ๋Œ๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.",
327
- "๋ณด์ฆ๊ธˆ์€ ์ง‘์„ ๋น„์šด ๋’ค 40์ผ ๋’ค์— ๋Œ๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.",
328
- "๋ณด์ฆ๊ธˆ์—๋Š” ์—ฐ 3% ์ด์ž๊ฐ€ ๋ถ™์Šต๋‹ˆ๋‹ค."]
329
-
330
- for s, r in zip(answer_sentences, judge.nli([doc] * len(answer_sentences), answer_sentences)):
331
- print(r["choice"], s)
332
- # entailment โ€ฆ 30์ผ ์•ˆ์— โ€ฆ โ†’ ๊ทผ๊ฑฐ ์žˆ์Œ
333
- # contradiction โ€ฆ 40์ผ ๋’ค์— โ€ฆ โ†’ ๋ฌธ์„œ์™€ ์–ด๊ธ‹๋‚จ
334
- # neutral โ€ฆ ์—ฐ 3% ์ด์ž โ€ฆ โ†’ ๋ฌธ์„œ์— ๊ทผ๊ฑฐ ์—†์Œ
335
- ```
336
-
337
- - ๋‹ต๋ณ€์€ **๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋‚˜๋ˆ ** ๋„ฃ์œผ์„ธ์š”. ์—ฌ๋Ÿฌ ์ฃผ์žฅ์ด ์„ž์ธ ๊ธด ๋‹ต๋ณ€์€ ํŒ๋‹จ์ด ํ๋ ค์ง‘๋‹ˆ๋‹ค.
338
- - ํ•™์Šต ์ตœ๋Œ€ ๊ธธ์ด๊ฐ€ 2,048ํ† ํฐ์ด๋ผ ๊ธด ๋ฌธ์„œ๋Š” ๊ด€๋ จ ๋‹จ๋ฝ๋งŒ ์ž˜๋ผ ๋„ฃ์œผ์„ธ์š”.
339
-
340
- ### 8. ์•ฝ๊ด€ยท๋ฌธ์„œ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค
341
-
342
- ๋ฌธ์„œ๋ฅผ ์ฃผ๊ณ  ์งˆ๋ฌธ์— ์˜ˆ/์•„๋‹ˆ์˜ค๋กœ ๋‹ตํ•˜๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ๋ฌธ์„œ์— ์—†๋Š” ์ƒ์‹์œผ๋กœ ๋‹ตํ•˜์ง€ ์•Š๋„๋ก ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
343
-
344
- ```python
345
- terms = "๊ตฌ๋งค ํ›„ 7์ผ ์ด๋‚ด์—๋Š” ๋‹จ์ˆœ ๋ณ€์‹ฌ์œผ๋กœ๋„ ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‹ค. ๋‹จ, ํฌ์žฅ์„ ๋œฏ์€ ์†Œํ”„ํŠธ์›จ์–ด๋Š” ํ™˜๋ถˆ ๋Œ€์ƒ์—์„œ ์ œ์™ธํ•œ๋‹ค."
346
-
347
- judge.yes_no("ํฌ์žฅ์„ ๋œฏ์€ ๊ฒŒ์ž„ CD๋ฅผ 3์ผ ๋งŒ์— ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‚˜์š”?", terms) # โ†’ ์•„๋‹ˆ์˜ค
348
- judge.yes_no("๋œฏ์ง€ ์•Š์€ ํ‚ค๋ณด๋“œ๋ฅผ 5์ผ ๋งŒ์— ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‚˜์š”?", terms) # โ†’ ์˜ˆ
349
- ```
350
-
351
- ### 9. ๋น„์Šทํ•œ ์งˆ๋ฌธ ์ฐพ๊ธฐ โ€” FAQ ์ค‘๋ณต ํ™•์ธ
352
-
353
- ์ƒˆ๋กœ ๋“ค์–ด์˜จ ์งˆ๋ฌธ์ด ๊ธฐ์กด FAQ์™€ ๊ฐ™์€ ๋œป์ธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ํ›„๋ณด๋Š” ๊ฒ€์ƒ‰(์ž„๋ฒ ๋”ฉ ๋“ฑ)์œผ๋กœ ๋จผ์ € ์ขํžˆ๊ณ , ์ตœ์ข… ํ™•์ธ์— ์ด ๋ชจ๋ธ์„ ์“ฐ๋Š” ๊ตฌ์„ฑ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
354
-
355
- ```python
356
- faq = ["๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”", "๋ฐฐ์†ก์ง€๋Š” ์–ด๋–ป๊ฒŒ ๋ฐ”๊พธ๋‚˜์š”", "ํšŒ์› ํƒˆํ‡ด๋Š” ์–ด๋””์„œ ํ•˜๋‚˜์š”"]
357
- q = "๋กœ๊ทธ์ธ ๋น„๋ฒˆ์ด ๊ธฐ์–ต์ด ์•ˆ ๋‚˜์š”"
358
-
359
- hits = judge.similar([q] * len(faq), faq)
360
- best = max(range(len(faq)), key=lambda i: hits[i]["probabilities"]["๊ฐ™์Œ"])
361
- faq[best], hits[best]["choice"] # ('๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”', '๊ฐ™์Œ')
362
- ```
363
-
364
- - `similar`๋Š” ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณผ์ œ์ž…๋‹ˆ๋‹ค (ํ•™์Šตํ•˜์ง€ ์•Š์€ KLUE-STS์—์„œ 85.9%).
365
-
366
- ### 10. ๋‚ด ๋ถ„๋ฅ˜ ์ฒด๊ณ„ ๊ทธ๋Œ€๋กœ ์“ฐ๊ธฐ
367
-
368
- ์–ด๋–ค ๋ถ„๋ฅ˜๋“  ์ง€์นจ๊ณผ ๋ณด๊ธฐ๋งŒ ์“ฐ๋ฉด ๋ฉ๋‹ˆ๋‹ค. ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณผ์ œ๋ผ๋„ **ํŒ๋‹จ ๊ธฐ์ค€์„ ์ง€์นจ์— ์ ์–ด ์ฃผ๋Š” ๊ฒƒ**์ด ์ •ํ™•๋„๋ฅผ ๊ฐ€์žฅ ํฌ๊ฒŒ ์ขŒ์šฐํ•ฉ๋‹ˆ๋‹ค.
369
-
370
- ```python
371
- judge.decide(
372
- state={"๋ณด๊ณ ์„œ ๋ฌธ์žฅ": "3๋ถ„๊ธฐ ๋งค์ถœ์€ ์ „๋…„ ๋Œ€๋น„ 12% ์ฆ๊ฐ€ํ–ˆ์œผ๋‚˜ ์˜์—…์ด์ต๋ฅ ์€ ์†Œํญ ํ•˜๋ฝํ–ˆ๋‹ค."},
373
- questions={
374
- "๋…ผ์กฐ": {
375
- "instructions": "๋ฌธ์žฅ์ด ํšŒ์‚ฌ ์‹ค์ ์„ ์–ด๋–ค ๋…ผ์กฐ๋กœ ์ „๋‹ฌํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค. ์ข‹์€ ์ ๊ณผ ๋‚˜์œ ์ ์ด ํ•จ๊ป˜ ์žˆ์œผ๋ฉด ํ˜ผํ•ฉ์œผ๋กœ ๋ณธ๋‹ค.",
376
- "criteria": {"positive": "๊ธ์ •์ ", "negative": "๋ถ€์ •์ ", "mixed": "ํ˜ผํ•ฉ", "neutral": "์‚ฌ์‹ค ์ „๋‹ฌ๋งŒ"},
377
- },
378
- },
379
- )
380
- ```
381
-
382
- **์ž˜ ์•ˆ ๋˜๋Š” ์“ฐ์ž„์ƒˆ**: ์ˆซ์ž ๋น„๊ตยท๊ณ„์‚ฐ์ด ํ•ต์‹ฌ์ธ ํŒ๋‹จ(โ€œ์–ด๋А ์ชฝ์ด ๋” ์‹ผ๊ฐ€โ€), ๋ฌธ์žฅ์—์„œ ๊ฐ’ ๋ฝ‘๊ธฐ, ๋‹ต๋ณ€ ์ž‘์„ฑ, ๋‹ค๋ฅธ ๋ชจ๋ธ ๋‹ต๋ณ€์˜ ํ’ˆ์งˆ ์ฑ„์ (LLM-as-a-judge). ์ด๋Ÿฐ ์ผ์€ ์ƒ์„ฑ ๋ชจ๋ธ์— ๋งก๊ธฐ์„ธ์š”.
383
-
384
- ### ์›๋ฆฌ: transformers๋งŒ์œผ๋กœ ์ง์ ‘ ๊ณ„์‚ฐํ•˜๊ธฐ
385
-
386
- ```python
387
- import torch, json
388
- from transformers import AutoModelForCausalLM, AutoTokenizer
389
- from huggingface_hub import hf_hub_download
390
-
391
- name = "coderecode95/KJH-Judge-4B-ko"
392
- tok = AutoTokenizer.from_pretrained(name)
393
- model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16, device_map="auto").eval()
394
- T = json.load(open(hf_hub_download(name, "calibration.json")))["temperature"] # 1.7179
395
-
396
- choices = ["๊ธ์ •์  ๊ฐ์ •", "๋ถ€์ •์  ๊ฐ์ •"]
397
- prompt = ("๋ฌธ์žฅ์˜ ์ „๋ฐ˜์ ์ธ ๊ฐ์ • ๊ทน์„ฑ์„ ํŒ๋‹จํ•œ๋‹ค.\n\n"
398
- "text: ๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”\n\n"
399
- "๋ณด๊ธฐ:\n" + "\n".join(f"{chr(65+i)}. {c}" for i, c in enumerate(choices)) +
400
- "\n\n์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.")
401
- ids = tok(f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n", return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
402
- with torch.no_grad():
403
- logits = model(ids).logits[0, -1]
404
- letters = [tok.encode(chr(65 + i), add_special_tokens=False)[0] for i in range(len(choices))]
405
- probs = torch.softmax(logits[letters].float() / T, -1)
406
- print({c: round(p, 4) for c, p in zip(choices, probs.tolist())})
407
- ```
408
-
409
- ### ํ”„๋กฌํ”„ํŠธ ํ˜•์‹
410
-
411
- ํ•™์Šต๊ณผ ํ‰๊ฐ€์— ์“ด ํ˜•์‹์ž…๋‹ˆ๋‹ค. ์ด ํ˜•์‹์„ ์ง€์ผœ์•ผ ์œ„ ์„ฑ๋Šฅ์ด ๋‚˜์˜ต๋‹ˆ๋‹ค.
412
-
413
- ```
414
- <|im_start|>user
415
- {๊ณผ์ œ ์„ค๋ช…} โ† instructions
416
-
417
- {ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’} โ† state ์˜ ํ•ญ๋ชฉ๋งˆ๋‹ค ํ•œ ์ค„. ๊ฐ’์ด ๋ฌธ์ž์—ด์ด ์•„๋‹ˆ๋ฉด JSON
418
- {ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’}
419
-
420
- ๋ณด๊ธฐ:
421
- A. {๋ณด๊ธฐ 1} โ† criteria ์˜ ์„ค๋ช…
422
- B. {๋ณด๊ธฐ 2}
423
- โ€ฆ
424
-
425
- ์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.<|im_end|>
426
- <|im_start|>assistant
427
- ```
428
-
429
- - `judge.decide()`์™€ ๋ฉ”์„œ๋“œ๋“ค์€ ์ด ํ˜•์‹์„ ์ž๋™์œผ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์ง์ ‘ ๋งŒ๋“ค ๋•Œ๋งŒ ์‹ ๊ฒฝ ๏ฟฝ๏ฟฝ๋ฉด ๋ฉ๋‹ˆ๋‹ค.
430
- - ๋ณด๊ธฐ๋Š” 2~26๊ฐœ์ž…๋‹ˆ๋‹ค.
431
- - ๊ณผ์ œ ์„ค๋ช…์€ ์งง๊ณ  ๊ตฌ์ฒด์ ์œผ๋กœ ์“ฐ์„ธ์š”. ํŒ๋‹จ ๊ธฐ์ค€์„ ํ•œ๋‘ ๋ฌธ์žฅ ๋ง๋ถ™์ด๋ฉด(์˜ˆ: โ€œ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹คโ€, โ€œ์ •๋ณด๊ฐ€ ์—†๋‹ค๋Š” ์ด์œ ๋งŒ์œผ๋กœ ๋ชจ์ˆœ์„ ๊ณ ๋ฅด์ง€ ์•Š๋Š”๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ํฌ๊ฒŒ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
432
- - `generate()`๋กœ ์“ธ ๊ฒฝ์šฐ `max_new_tokens=1`๋กœ ๋ณด๊ธฐ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋ฐ›์œผ์„ธ์š” (์ด ์ €์žฅ์†Œ์˜ `generation_config.json`์€ `do_sample: false`์ž…๋‹ˆ๋‹ค). ํ™•๋ฅ ์ด ํ•„์š”ํ•˜๋ฉด ์œ„์ฒ˜๋Ÿผ logits๋ฅผ ์ฝ์œผ์„ธ์š”.
433
-
434
- ## ์„ฑ๋Šฅ
435
-
436
- ### 1. ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณต๊ฐœ ๋ฒค์น˜๋งˆํฌ
437
-
438
- ์ด ๋ชจ๋ธ์˜ ํ•™์Šต์— ์“ฐ์ง€ ์•Š์€ ์ถœ์ฒ˜์˜ ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค (KLUE-STS๋Š” ํ•™์Šต์— ์“ด KLUE NLIยทYNAT์™€ ๊ฐ™์€ KLUE ๋ฌถ์Œ์ด์ง€๋งŒ, STS ๊ณผ์ œ๋Š” ํ•™์Šตํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค). 0-shot, ์œ„ ํ”„๋กฌํ”„ํŠธ ํ˜•์‹, ๋ณด๊ธฐ ๊ธฐํ˜ธ ์ ์ˆ˜ ๋ฐฉ์‹์œผ๋กœ ์Ÿ€์Šต๋‹ˆ๋‹ค (ํ‘œ๋ณธ์€ ๊ณ ์ • ์‹œ๋“œ, ๊ด„ํ˜ธ๋Š” 95% ์‹ ๋ขฐ๊ตฌ๊ฐ„). ์žฌํ˜„ ์Šคํฌ๋ฆฝํŠธ: [`eval/bench_heldout.py`](eval/bench_heldout.py).
439
-
440
- <!-- HELDOUT:start -->
441
- | ๋ฒค์น˜๋งˆํฌ | ๋ฌด์—‡์„ ์žฌ๋‚˜ | ๋ฌธํ•ญ | ์ •ํ™•๋„ (95% ๊ตฌ๊ฐ„) | ๋ฌด์ž‘์œ„ | ์ถœ์ฒ˜ ยท ๋ผ์ด์„ ์Šค |
442
- |---|---|---:|---:|---:|---|
443
- | NSMC | ์˜ํ™” ๋ฆฌ๋ทฐ ๊ฐ์„ฑ (๊ธ์ •/๋ถ€์ •) | 1,000 | **85.7%** (83.4~87.7) | 50% | e9t/nsmc ยท ratings_test ยท CC0 1.0 |
444
- | KorNLI (XNLI) | ๋ฌธ์žฅ ๊ด€๊ณ„ 3๋ถ„๋ฅ˜ (XNLI ํ•œ๊ตญ์–ด ๋ฒˆ์—ญ) | 1,000 | **74.5%** (71.7~77.1) | 33% | kakaobrain/kor_nli ยท xnli/test ยท CC BY-SA 4.0 |
445
- | KLUE-STS | ๋‘ ๋ฌธ์žฅ์ด ๊ฐ™์€ ๋œป์ธ์ง€ (์˜ˆ/์•„๋‹ˆ์˜ค) | 519 | **85.9%** (82.7~88.7) | 50% | klue/klue ยท sts/validation (binary-label) ยท CC BY-SA 4.0 |
446
- | Korean UnSmile | ์•…ํ”Œยทํ˜์˜ค ํ‘œํ˜„ ์—ฌ๋ถ€ | 1,000 | **82.1%** (79.6~84.4) | 50% | smilegate-ai/kor_unsmile ยท valid (clean ์—ฌ๋ถ€) ยท CC BY-NC-ND 4.0 |
447
- | KMMLU | ํ•œ๊ตญ์–ด ์ „๋ฌธ ์ง€์‹ 4์ง€์„ ๋‹ค (45๊ณผ๋ชฉ) | 990 | **42.6%** (39.6~45.7) | 25% | HAERAE-HUB/KMMLU ยท test ยท CC BY-ND 4.0 |
448
- | HAE-RAE Bench 1.0 | ํ•œ๊ตญ ๋ฌธํ™”ยท์–ดํœ˜ ์ง€์‹ 5์ง€์„ ๋‹ค | 983 | **56.1%** (53.0~59.2) | 21% | HAERAE-HUB/HAE_RAE_BENCH_1.0 ยท test ยท CC BY-NC-ND 4.0 |
449
- <!-- HELDOUT:end -->
450
-
451
- <!-- COMPARE:start -->
452
- **๊ฐ™์€ ์กฐ๊ฑด ๋น„๊ต** ยท ๊ฐ™์€ ๋ฌธํ•ญ ยท ๊ฐ™์€ ํ”„๋กฌํ”„ํŠธ ยท 0-shot ยท ๋ณด๊ธฐ ๊ธฐํ˜ธ ์ ์ˆ˜. ๋‹ค๋ฅธ ๋ชจ๋ธ์€ ์ƒ๊ฐ(thinking)์„ ๋„๊ณ  ๊ฐ์ž์˜ ๋Œ€ํ™” ํ˜•์‹์œผ๋กœ ๋ฌผ์—ˆ์Šต๋‹ˆ๋‹ค.
453
-
454
- | ๋ฒค์น˜๋งˆํฌ | **KJH-Judge-4B-ko (4B)** | Gemma 4 26B-A4B-it (0-shot) |
455
- |---|---:|---:|
456
- | NSMC | **85.7%** | 88.7% |
457
- | KorNLI (XNLI) | **74.5%** | 77.1% |
458
- | KLUE-STS | **85.9%** | 88.4% |
459
- | Korean UnSmile | **82.1%** | 81.3% |
460
- | KMMLU | **42.6%** | 64.0% |
461
- | HAE-RAE Bench 1.0 | **56.1%** | 79.7% |
462
- <!-- COMPARE:end -->
463
-
464
- ์›๋ณธ Qwen3-4B์˜ ์‚ฌ์ „ํ•™์Šต์— ์ด ๋ฐ์ดํ„ฐ๊ฐ€ ๋“ค์–ด๊ฐ”๋Š”์ง€๋Š” ์•Œ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. KMMLUยทHAE-RAE๋Š” ์ „๋ฌธ ์ง€์‹์„ ๋ฌป๋Š” ์‹œํ—˜์ด๋ผ, ํŒ์ • ๋Šฅ๋ ฅ๋ณด๋‹ค ์›๋ณธ ๋ชจ๋ธ์˜ ์ง€์‹์ด ์ ์ˆ˜๋ฅผ ์ขŒ์šฐํ•ฉ๋‹ˆ๋‹ค.
465
-
466
- ### 2. Korean JEV Benchmark (์ฐธ๊ณ : ๊ฐ™์€ ์ถœ์ฒ˜์˜ ์‹œํ—˜ ๋ฌธํ•ญ)
467
-
468
- [DaehanKim/korean-jev-benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark) 25,418๋ฌธํ•ญ ์ „์ฒด์ž…๋‹ˆ๋‹ค. ์ด ์‹œํ—˜ ๋ฌธํ•ญ ์ž์ฒด๋กœ ํ•™์Šตํ•˜์ง€๋Š” ์•Š์•˜์ง€๋งŒ, **๊ฐ™์€ ์ถœ์ฒ˜ ๋ฐ์ดํ„ฐ(KLUE, KoBEST ๋“ฑ)์˜ ํ•™์Šต ๋ถ„ํ• ๋กœ ํ•™์Šตํ–ˆ๊ธฐ ๋•Œ๋ฌธ์—** ๋‹ค๋ฅธ ๋ชจ๋ธ์˜ 0-shot ์ ์ˆ˜์™€ ๊ฐ™์€ ์กฐ๊ฑด์˜ ๋น„๊ต๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค.
469
-
470
- | ํ•ญ๋ชฉ | ๊ฐ’ |
471
- |---|---|
472
- | ์ „์ฒด ์ •ํ™•๋„ | **90.37%** (ํ•™์Šต ์ „ Qwen3-4B 29.56%, ๋ฌด์ž‘์œ„ 29.86%) |
473
- | ํ•™์Šต ๋ฐ์ดํ„ฐ๊ฐ€ ์—†๋Š” ๊ณผ๋ชฉ (Belebele ํ•œ๊ตญ์–ด ๋…ํ•ด) | **86.87%** |
474
- | ๋ณด๊ธฐ๋ฅผ ๊ฑฐ๊พธ๋กœ ๋†“์•„๋„ ๊ฐ™์€ ๋‹ต | 97.93% |
475
- | ๊ธฐ๋Œ€ ๋ณด์ • ์˜ค์ฐจ (ECE) | 0.0265 |
476
-
477
- | ๊ณผ๋ชฉ | ์ •ํ™•๋„ | ๊ณผ๋ชฉ | ์ •ํ™•๋„ |
478
- |---|---:|---|---:|
479
- | KoBEST BoolQ | 95.65% | KLUE YNAT (๋‰ด์Šค ์ฃผ์ œ) | 93.93% |
480
- | KoBEST HellaSwag | 94.65% | KLUE NLI | 92.39% |
481
- | KoBEST SentiNeg | 93.91% | APEACH (ํ˜์˜ค) | 88.64% |
482
- | KoBEST COPA | 92.55% | KoBEST WiC | 84.96% |
483
- | SIB-200 | 94.44% | Korean Hate Speech (3๋ถ„๋ฅ˜) | 77.87% |
484
- | FunctionChat ๋„๊ตฌ ๊ณ ๋ฅด๊ธฐ | 94.68% | EnterpriseOps ์ •์ฑ… ํ–‰๋™ (7๋ถ„๋ฅ˜) | 83.43% |
485
- | FunctionChat ๋Œ€ํ™” | 93.50% | ๋ฒ•๋ น ์งˆ๋ฌธ (v2ยทv3, 6๊ฐœ ๋ฌถ์Œ) | 90.8~99.2% |
486
- | FunctionChat ํ˜ธ์ถœ ํŒ๋‹จ | 84.55% | ๋ฒ•๋ น ๊ธฐ๊ฐ„ ๊ณ„์‚ฐ | 69.10% |
487
-
488
- ### ์†๋„
489
-
490
- ๊ฐœ์ธ GPU ์„œ๋ฒ„์—์„œ ๊ฐ™์€ ๋ฌธํ•ญ 60๊ฐœ(NSMCยทKorNLI)๋กœ ์žฐ ์ฐธ๊ณ ๊ฐ’์ž…๋‹ˆ๋‹ค (๋„คํŠธ์›Œํฌ ์™•๋ณต ํฌํ•จ, ์ธก์ • ์Šคํฌ๋ฆฝํŠธ: [`eval/bench_speed.py`](eval/bench_speed.py)).
491
-
492
- | ์ธก์ • | **KJH-Judge-4B-ko** | Gemma 4 26B-A4B-it |
493
- |---|---:|---:|
494
- | ํŒ์ • ํ•œ ๋ฒˆ (์ค‘์•™๊ฐ’) | **250ms** (์„œ๋ฒ„ ์•ˆ ๊ณ„์‚ฐ 87ms) | 621ms |
495
- | ๋™์‹œ 4๊ฐœ์”ฉ ๋ณด๋ƒˆ์„ ๋•Œ ์ฒ˜๋ฆฌ๋Ÿ‰ | 3.9๊ฑด/์ดˆ | **9.9๊ฑด/์ดˆ** |
496
- | ์ƒ๊ฐ์„ ์ผœ๊ณ  ๋‹ต์„ ์ƒ์„ฑํ•  ๋•Œ | | 6.3์ดˆ |
497
-
498
- - ๋™์‹œ ์ฒ˜๋ฆฌ๋Ÿ‰์€ ๋น„๊ต ๋ชจ๋ธ์ด ์•ž์„ฐ์Šต๋‹ˆ๋‹ค. ๋น„๊ต ๋ชจ๋ธ์€ ์š”์ฒญ์„ ๋ฌถ์–ด ์ฒ˜๋ฆฌํ•˜๋Š” ์„œ๋น™ ์—”์ง„์—์„œ, ์ด ๋ชจ๋ธ์€ ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌํ•˜๋Š” ๋‹จ์ˆœํ•œ ์„œ๋ฒ„์—์„œ ๋Œ์•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ์—๋Š” ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ๊ณ  `batch_size`๋ฅผ ์ฃผ์–ด ๋ฌถ์–ด ๊ณ„์‚ฐํ•˜์„ธ์š”.
499
- - CPU(๋…ธํŠธ๋ถ, fp32)์—์„œ๋Š” ํŒ์ • ํ•œ ๋ฒˆ์— 6์ดˆ์—์„œ ์ˆ˜ ๋ถ„์ด ๊ฑธ๋ ค ์‹ค์‚ฌ์šฉ์ด ์–ด๋ ต์Šต๋‹ˆ๋‹ค.
500
-
501
- ## ์ž˜ํ•˜๏ฟฝ๏ฟฝ๏ฟฝ ๊ฒƒ๊ณผ ๋ชปํ•˜๋Š” ๊ฒƒ
502
-
503
- **์ž˜ํ•˜๋Š” ๊ฒƒ**: ๋ณด๊ธฐ๊ฐ€ ์ •ํ•ด์ง„ ๋ถ„๋ฅ˜, ๋ฌธ์žฅ ๊ด€๊ณ„ ํŒ๋‹จ, ์ง€๋ฌธ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค, ์ •์ฑ…ยท๊ทœ์น™์— ๋”ฐ๋ฅธ ํ–‰๋™ ์„ ํƒ, ์—์ด์ „ํŠธ์˜ ๋‹ค์Œ ํ–‰๋™(๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€) ๊ฒฐ์ •. ์ฒ˜์Œ ๋ณด๋Š” ์ง€์‹œ๋ฌธ์ด๋ผ๋„ ์ด๋Ÿฐ ํ˜•์‹์ด๋ฉด ์ž˜ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
504
-
505
- **๋ชปํ•˜๋Š” ๊ฒƒ**
506
- - **๊ธ€ ์ƒ์„ฑ, ๋ฌธ์žฅ์—์„œ ๊ฐ’ ๋ฝ‘๊ธฐ**: ๋ณด๊ธฐ๋ฅผ ๊ณ ๋ฅด๋Š” ๊ฒƒ๋งŒ ํ•ฉ๋‹ˆ๋‹ค. ๋„๊ตฌ ์ธ์ž ๊ฐ’์„ ์ฑ„์šฐ๊ฑฐ๋‚˜ ๋‹ต๋ณ€์„ ์“ฐ๋Š” ์ผ์€ ์ƒ์„ฑ ๋ชจ๋ธ์— ๋งก๊ธฐ์„ธ์š”.
507
- - **์ˆซ์ž ๋น„๊ตยท๊ณ„์‚ฐ์ด ํ•ต์‹ฌ์ธ ํŒ๋‹จ**: ๊ฑฐ๋ฆฌยท์†๋„ยท์ ์ˆ˜ ๊ฐ™์€ ์ˆ˜์น˜๋ฅผ ๋น„๊ตํ•ด ๊ณ ๋ฅด๋Š” ์ผ์€ ์•ฝํ•ฉ๋‹ˆ๋‹ค.
508
- - **๋ฐ˜์–ดยท๋น„๊ผผ**: โ€œ๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”โ€๋ฅผ ๊ธ์ •์œผ๋กœ ๋†’์€ ํ™•์‹ ๊ณผ ํ•จ๊ป˜ ๊ณ ๋ฅด๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
509
- - **์ง€์‹œ๋ฌธ ๋ฌธ๊ตฌ์— ๋ฏผ๊ฐ**: โ€œ์‹ค์ œ ์กด์žฌํ•˜๋Š” ์ธ๋ฌผ์ธ๊ฐ€์š”?โ€๋ฅผ โ€œ์‹ค์ œ ์กด์žฌํ•˜๋‚˜์š”?โ€์ฒ˜๋Ÿผ ์ฝ๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ์ค€์„ ๋ช…ํ™•ํžˆ ์“ฐ์„ธ์š”.
510
-
511
- ๊ถŒ์žฅ ๊ตฌ์„ฑ: ํฐ LLM ์•ž๋‹จ์—์„œ ์ด ๋ชจ๋ธ์ด โ€œ๋ฌด์—‡์„ ํ• ์ง€โ€๋ฅผ ๋น ๋ฅด๊ณ  ์‹ธ๊ฒŒ ์ •ํ•˜๊ณ , ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ์‚ฌ๋žŒ์ด๋‚˜ ํฐ ๋ชจ๋ธ์— ๋„˜๊ธฐ์„ธ์š”.
512
-
513
- ## ํ•™์Šต
514
-
515
- > ํ•™์Šต ๋ฐ์ดํ„ฐ ๊ตฌ์„ฑ, ๋ณด๊ธฐ ์ˆœ์„œ ์—ญ์ˆœ ํ•™์Šต, ์˜ค์—ผ ์ œ๊ฑฐ, ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ ๋งŒ๋“œ๋Š” ๋ฒ•, ํ™•๋ฅ  ๋ณด์ •, ์ฒดํฌํฌ์ธํŠธ๋ณ„ ์ •ํ™•๋„๊นŒ์ง€ ์ž์„ธํ•œ ๊ณผ์ •์€ **[TRAINING.md](TRAINING.md)**์— ์ •๋ฆฌํ–ˆ์Šต๋‹ˆ๋‹ค.
516
-
517
- | ํ•ญ๋ชฉ | ๊ฐ’ |
518
- |---|---|
519
- | ์›๋ณธ ๋ชจ๋ธ | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) |
520
- | ๋ฐฉ๋ฒ• | SFT, LoRA (r 16, ฮฑ 32, dropout 0.05, qยทkยทvยทoยทgateยทupยทdown) โ†’ ์›๋ณธ์— ๋ณ‘ํ•ฉ |
521
- | ํ•™์Šต๋Ÿ‰ | 69,144๋ฌธํ•ญ ร— ๋ณด๊ธฐ ์ˆœ์„œ 2๊ฐ€์ง€(์„ž์€ ์ˆœ์„œ + ์—ญ์ˆœ), ์ถœ์ฒ˜ ๋น„์œจ๋Œ€๋กœ ์„ž์–ด 4,430 step ยท ๋ฐฐ์น˜ 32, ์ตœ๋Œ€ ๊ธธ์ด 2,048 |
522
- | ์ตœ์ ํ™” | ํ•™์Šต๋ฅ  2e-4, cosine, warmup 3% |
523
- | ์žฅ๋น„ | GPU 1์žฅ, ์•ฝ 3.5์‹œ๊ฐ„ |
524
- | ์ •๋‹ต ํ˜•์‹ | ๋ณด๊ธฐ ๊ธฐํ˜ธ ํ•œ ๊ธ€์ž + `<|im_end|>` |
525
- | ๋ณด์ • | ๊ฒ€์ฆ์…‹ 3,089๋ฌธํ•ญ์œผ๋กœ ๋‹จ์ผ ์˜จ๋„(1.7179)๋ฅผ ๋งž์ถค. ECE 0.066 โ†’ 0.013 |
526
-
527
- **ํ•™์Šต ๋ฐ์ดํ„ฐ** (๋ชจ๋‘ ์œ„ ํ”„๋กฌํ”„ํŠธ ํ˜•์‹์œผ๋กœ ๋ฐ”๊ฟ” ์‚ฌ์šฉ, ํ‰๊ฐ€ ๋ฌธํ•ญ๊ณผ ๊ฒน์น˜๋Š” ๋ฌธ์žฅ์€ ์ œ์™ธ)
528
-
529
- | ๋ฐ์ดํ„ฐ | ๋ผ์ด์„ ์Šค |
530
- |---|---|
531
- | KLUE NLI ยท YNAT (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
532
- | KoBEST BoolQ ยท COPA ยท HellaSwag ยท SentiNeg ยท WiC (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
533
- | kocohub/korean-hate-speech (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
534
- | APEACH (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
535
- | SIB-200 (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
536
- | ์ •์ฑ… ํ–‰๋™ ยท ๋ฒ•๋ น ์งˆ๋ฌธ ยท ๋„๊ตฌ ํ˜ธ์ถœ ํŒ๋‹จ | ์ž์ฒด ํ•ฉ์„ฑ (๊ทœ์น™์œผ๋กœ ์ƒ์„ฑ, ์ •๋‹ต์€ ์ฝ”๋“œ๋กœ ๊ณ„์‚ฐ. ์ •์ฑ…ยท๋ฒ•๋ นยท๋„๊ตฌ๋Š” ๋ชจ๋‘ ์ง€์–ด๋‚ธ ๊ฒƒ์ด๋ฉฐ LLM ์ƒ์„ฑ๋ฌผ์„ ์“ฐ์ง€ ์•Š์Œ) |
537
-
538
- ## ๋ผ์ด์„ ์Šค์™€ ์ฃผ์˜
539
-
540
- - ๋ชจ๋ธ ๊ฐ€์ค‘์น˜๋Š” ์›๋ณธ Qwen3-4B์™€ ๊ฐ™์€ **Apache 2.0**์ž…๋‹ˆ๋‹ค ([LICENSE](LICENSE)). ๊ณต๊ฐœ ํ•™์Šต ๋ฐ์ดํ„ฐ๋Š” ๋ชจ๋‘ CC BY-SA 4.0์ž…๋‹ˆ๋‹ค.
541
- - ํ˜์˜ค ํ‘œํ˜„ ํŒ์ •, ์ •์ฑ… ํŒ๋‹จ ๋“ฑ ์‚ฌ๋žŒ์—๊ฒŒ ์˜ํ–ฅ์„ ์ฃผ๋Š” ๊ฒฐ์ •์— ์“ธ ๋•Œ๋Š” ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฒฐ๊ณผ๋ฅผ ์‚ฌ๋žŒ์ด ๊ฒ€ํ† ํ•˜๋„๋ก ํ•˜์„ธ์š”.
542
- - ๋ฒ•๋ฅ ยท๊ธˆ์œต ํŒ๋‹จ์˜ ๊ทผ๊ฑฐ๋กœ ์“ฐ์ง€ ๋งˆ์„ธ์š”. ์ฃผ์–ด์ง„ ๊ธ€ ์•ˆ์—์„œ ๋‚ด์šฉ์„ ์ฐพ๊ณ  ๋Œ€์กฐํ•˜๋Š” ๋„๊ตฌ์ž…๋‹ˆ๋‹ค.
 
 
92
  name: accuracy (0-shot
93
  n=983): null
94
  ---
95
+
96
+ # KJH-Judge-4B-ko
97
+
98
+ > [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)๋ฅผ ๋ณด๊ณ  ํŒ์ • ๋ชจ๋ธ์„ ์ง์ ‘ ๋งŒ๋“ค์–ด ๋ณธ **๊ฐœ์ธ ํ•™์Šต ํ”„๋กœ์ ํŠธ**์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์ˆ˜์น˜๋Š” ๋ชจ๋‘ ์ง์ ‘ ์žฐ ๊ฒฐ๊ณผ์ด๋ฉฐ, ์‹ค์„œ๋น„์Šค์—์„œ ๊ฒ€์ฆํ•œ ๋ชจ๋ธ์€ ์•„๋‹™๋‹ˆ๋‹ค.
99
+
100
+ **์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณจ๋ผ, ๋ณด์ •๋œ ํ™•๋ฅ ๊ณผ ํ•จ๊ป˜ ๋Œ๋ ค์ฃผ๋Š” ํ•œ๊ตญ์–ด ํŒ์ • ๋ชจ๋ธ์ž…๋‹ˆ๋‹ค.** ๊ฐ์ •ยท์ฃผ์ œ ๋ถ„๋ฅ˜, ๋ฌธ์žฅ ๊ด€๊ณ„(ํ•จ์˜ยท๋ชจ์ˆœ), ํ˜์˜ค ํ‘œํ˜„ ๊ฒ€์ˆ˜, ์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ ์„ ํƒ, ์—์ด์ „ํŠธ์˜ ๋„๊ตฌ ํ˜ธ์ถœ ํŒ๋‹จ(๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€) ๊ฐ™์€ ์ผ์— ์”๋‹ˆ๋‹ค. ๋‹ค๋ฅธ ๋ชจ๋ธ์˜ ๋‹ต๋ณ€์„ ์ฑ„์ ํ•˜๋Š” LLM-as-a-judge ๋ชจ๋ธ์ด ์•„๋‹™๋‹ˆ๋‹ค.
101
+
102
+ > **English summary.** A personal learning project: a Korean decision model fine-tuned from Qwen3-4B. Given a task description, an item, and 2โ€“26 options, it returns the chosen option with **calibrated probabilities** from a single forward pass (no text generation, ~tens of ms per decision on a GPU). Good for classification, NLI, moderation, policy actions, and tool-call routing. It does not generate text, extract spans, or do numeric reasoning.
103
+
104
+ ![ํ•™์Šตํ•˜์ง€ ์•Š์€ ํ•œ๊ตญ์–ด ๋ฒค์น˜๋งˆํฌ](assets/benchmark.png)
105
+
106
+ ## ์ด ๋ชจ๋ธ์˜ ํŠน์ง•
107
+
108
+ - **๋ณด๊ธฐ ๊ณ ๋ฅด๊ธฐ๋งŒ ํ•ฉ๋‹ˆ๋‹ค.** ๊ธ€์„ ์ƒ์„ฑํ•˜์ง€ ์•Š๊ณ  ํ”„๋กฌํ”„ํŠธ๋ฅผ ํ•œ ๋ฒˆ ๊ณ„์‚ฐํ•œ ๋’ค, ๋ณด๊ธฐ ๊ธฐํ˜ธ(A, B, โ€ฆ) ํ† ํฐ์˜ ์ ์ˆ˜๋งŒ ์ฝ์Šต๋‹ˆ๋‹ค. ๊ทธ๋ž˜์„œ ๋น ๋ฅด๊ณ (GPU ์„œ๋ฒ„ ์•ˆ ๊ณ„์‚ฐ ๊ธฐ์ค€ ํŒ์ • ํ•œ ๋ฒˆ ์•ฝ 90ms), ๋‹ต์ด ํ•ญ์ƒ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜์ž…๋‹ˆ๋‹ค.
109
+ - **ํ™•๋ฅ ์ด ๋ณด์ •๋˜์–ด ์žˆ์Šต๋‹ˆ๋‹ค.** ๊ฒ€์ฆ์…‹์œผ๋กœ ๋งž์ถ˜ ์˜จ๋„(1.72)๋กœ ๋ณด์ •ํ•ด, Korean JEV Benchmark์—์„œ ๊ธฐ๋Œ€ ๋ณด์ • ์˜ค์ฐจ(ECE)๊ฐ€ 0.027์ž…๋‹ˆ๋‹ค. โ€œํ™•์‹  70% ๋ฏธ๋งŒ์ด๋ฉด ์‚ฌ๋žŒ์—๊ฒŒ ๋„˜๊ธฐ๊ธฐโ€ ๊ฐ™์€ ๊ธฐ์ค€์„ ๊ฑธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. ๋‹ค๋งŒ ํ•™์Šตํ•˜์ง€ ์•Š์€ ์ถœ์ฒ˜์˜ ๋ฐ์ดํ„ฐ์—์„œ๋Š” ํ‰๊ท  ํ™•์‹ ์ด ์ •ํ™•๋„๋ณด๋‹ค ์ตœ๋Œ€ 10%p ์ •๋„ ๋†’๊ฒŒ ๋‚˜๏ฟฝ๏ฟฝ๏ฟฝ์œผ๋‹ˆ, ๊ธฐ์ค€๊ฐ’์€ ์“ฐ๋ ค๋Š” ๋ฐ์ดํ„ฐ์—์„œ ๋‹ค์‹œ ๋งž์ถ”๋Š” ๊ฒƒ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
110
+ - **๋ณด๊ธฐ ์ˆœ์„œ์— ํ”๋“ค๋ฆฌ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.** ๋ณด๊ธฐ๋ฅผ ์ •ํ™•ํžˆ ๊ฑฐ๊พธ๋กœ ๋†“๊ณ  ๋‹ค์‹œ ๋ฌผ์–ด๋„ 97.9%๊ฐ€ ๊ฐ™์€ ๋‹ต์ž…๋‹ˆ๋‹ค.
111
+
112
+ ## ๋น ๋ฅด๊ฒŒ ์‹œ์ž‘ํ•˜๊ธฐ
113
+
114
+ ```bash
115
+ pip install torch "transformers>=5" accelerate
116
+ ```
117
+
118
+ ```python
119
+ from transformers import pipeline
120
+
121
+ judge = pipeline(model="coderecode95/KJH-Judge-4B-ko", trust_remote_code=True,
122
+ device=0, dtype="bfloat16") # GPU๊ฐ€ ์—†์œผ๋ฉด device, dtype ๋ฅผ ๋นผ์„ธ์š” (๋งค์šฐ ๋А๋ฆผ)
123
+ ```
124
+
125
+ ์“ฐ๋Š” ๋ฐฉ๋ฒ•์€ ์„ธ ๊ฐ€์ง€์ด๊ณ , **๊ฒฐ๊ณผ๋Š” ๋ชจ๋‘ ๊ฐ™์€ ๋ชจ์–‘**์ž…๋‹ˆ๋‹ค.
126
+
127
+ ```python
128
+ {'choice': '๊ธ์ •', 'confidence': 0.97, 'probabilities': {'๊ธ์ •': 0.97, '๋ถ€์ •': 0.03}}
129
+ # ๊ณ ๋ฅธ ๋ณด๊ธฐ ๊ทธ ๋ณด๊ธฐ์˜ ํ™•๋ฅ (๋ณด์ •๋จ) ๋ณด๊ธฐ๋ณ„ ํ™•๋ฅ 
130
+ ```
131
+
132
+ #### 1. ์ž์ฃผ ์“ฐ๋Š” ํŒ์ • โ€” ๋ฉ”์„œ๋“œ ํ•˜๋‚˜๋กœ
133
+
134
+ ํ•™์Šต ๋•Œ์™€ ๊ฐ™์€ ์ง€์นจยท๋ณด๊ธฐ ๋ฌธ๊ตฌ๊ฐ€ ์ž๋™์œผ๋กœ ๋“ค์–ด๊ฐ€์„œ, ๋”ฐ๋กœ ํ”„๋กฌํ”„ํŠธ๋ฅผ ์“ธ ํ•„์š”๊ฐ€ ์—†๊ณ  ์ •ํ™•๋„๋„ ๊ฐ€์žฅ ์ž˜ ๋‚˜์˜ต๋‹ˆ๋‹ค.
135
+
136
+ ```python
137
+ judge.sentiment("๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”") # โ†’ ๊ธ์ •
138
+ judge.nli("๋ณด์ฆ๊ธˆ์€ ํ‡ด๊ฑฐ ํ›„ 30์ผ ์ด๋‚ด ๋ฐ˜ํ™˜ํ•œ๋‹ค.", "๋ณด์ฆ๊ธˆ์€ ํ‡ด๊ฑฐ 40์ผ ๋’ค์— ๋ฐ›๋Š”๋‹ค.") # โ†’ contradiction
139
+ judge.next_action("๊ฐ•๋‚จ์—ญ์—์„œ ์„œ์šธ์—ญ๊นŒ์ง€ ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜", tools) # โ†’ CALL:call_taxi
140
+ ```
141
+
142
+ | ๋ฉ”์„œ๋“œ | ํŒ์ • | ๊ณ ๋ฅด๋Š” ๊ฐ’ |
143
+ |---|---|---|
144
+ | `sentiment(๋ฌธ์žฅ)` | ๊ฐ์ • | ๊ธ์ • / ๋ถ€์ • |
145
+ | `topic(๋‰ด์Šค ์ œ๋ชฉ)` | ๋‰ด์Šค ์ฃผ์ œ | ์‚ฌํšŒ / IT๊ณผํ•™ / ์„ธ๊ณ„ / ์ƒํ™œ๋ฌธํ™” / ์ •์น˜ / ์Šคํฌ์ธ  / ๊ฒฝ์ œ |
146
+ | `hate(๋ฌธ์žฅ)` | ํ˜์˜คยท๋น„ํ•˜ ํ‘œํ˜„ | ๋น„ํ˜์˜ค / ํ˜์˜ค |
147
+ | `offensive(๋Œ“๊ธ€, news_title=None)` | ๋Œ“๊ธ€ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„ | ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค |
148
+ | `nli(์ „์ œ, ๊ฐ€์„ค)` | ๋ฌธ์žฅ ๊ด€๊ณ„ | entailment / contradiction / neutral |
149
+ | `similar(๋ฌธ์žฅ1, ๋ฌธ์žฅ2)` | ๊ฐ™์€ ๋œป์ธ์ง€ | ๊ฐ™์Œ / ๋‹ค๋ฆ„ |
150
+ | `yes_no(์งˆ๋ฌธ, ์ง€๋ฌธ)` | ์ง€๋ฌธ ๊ทผ๊ฑฐ ์˜ˆ/์•„๋‹ˆ์˜ค | ์˜ˆ / ์•„๋‹ˆ์˜ค |
151
+ | `same_meaning(๋‹จ์–ด, ๋ฌธ๋งฅ1, ๋ฌธ๋งฅ2)` | ๋‹จ์–ด ๋œป์ด ๊ฐ™์€์ง€ | ๊ฐ™์Œ / ๋‹ค๋ฆ„ |
152
+ | `reading(์ง€๋ฌธ, ์งˆ๋ฌธ, ๋ณด๊ธฐ)` | ๋…ํ•ด ๊ฐ๊ด€์‹ | ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜ |
153
+ | `next_action(๋Œ€ํ™”, ๋„๊ตฌ ๋ชฉ๋ก)` | ์—์ด์ „ํŠธ ๋‹ค์Œ ํ–‰๋™ | CALL:<๋„๊ตฌ> / ASK / RESPOND |
154
+ | `policy_action(์ •์ฑ…, ๊ณ ๊ฐ ์š”์ฒญ)` | ์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ | ANSWER / ASK_CLARIFICATION / ASK_CONFIRMATION / CALL_TOOL / CALL_MULTIPLE_TOOLS / ESCALATE / REFUSE |
155
+
156
+ ๋ฌธ์žฅ ํ•˜๋‚˜๋งŒ ๋ฐ›๋Š” ๋ฉ”์„œ๋“œ(`sentiment`, `topic`, `hate`, `nli` ๋“ฑ)๋Š” **๋ฆฌ์ŠคํŠธ๋ฅผ ๋„ฃ์œผ๋ฉด ์—ฌ๋Ÿฌ ๊ฐœ๋ฅผ ํ•œ ๋ฒˆ์—** ํŒ์ •ํ•ฉ๋‹ˆ๋‹ค.
157
+
158
+ #### 2. ๋‚ด ํŒ์ • ๋งŒ๋“ค๊ธฐ โ€” JEV ํ˜•์‹
159
+
160
+ [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)ยทJEV API์™€ ๊ฐ™์€ ๋ชจ์–‘์ž…๋‹ˆ๋‹ค. **์ƒํ™ฉ(`state`) ํ•˜๋‚˜์— ์ด๋ฆ„ ๋ถ™์ธ ์งˆ๋ฌธ ์—ฌ๋Ÿฌ ๊ฐœ**๋ฅผ ํ•œ ๋ฒˆ์— ๋ฌป๊ณ , ๋ณด๊ธฐ์—๋Š” **๋‚ด๊ฐ€ ์“ธ ID**๋ฅผ ๋ถ™์ž…๋‹ˆ๋‹ค. ๊ฒฐ๊ณผ๋Š” ๊ทธ ID๋กœ ๋Œ์•„์˜ต๋‹ˆ๋‹ค.
161
+
162
+ ```python
163
+ judge.decide(
164
+ state={"๊ณ ๊ฐ ๋ฌธ์˜": "์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ์ฒ˜๋ฆฌํ•ด ์ฃผ์„ธ์š”."},
165
+ questions={
166
+ "๋ถ€์„œ": {
167
+ "instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.",
168
+ "criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ"},
169
+ },
170
+ "๊ธด๊ธ‰": {
171
+ "instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๋Š”์ง€ ํŒ๋‹จํ•œ๋‹ค.",
172
+ "criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ์š”๊ตฌ", "normal": "์ผ๋ฐ˜ ๋ฌธ์˜"},
173
+ },
174
+ },
175
+ )
176
+ # {'๋ถ€์„œ': {'choice': 'billing', 'confidence': 0.9โ€ฆ, 'probabilities': {...}},
177
+ # '๊ธด๊ธ‰': {'choice': 'urgent', 'confidence': 0.9โ€ฆ, 'probabilities': {...}}}
178
+ ```
179
+
180
+ - **`state`**: ํŒ๋‹จํ•  ๋‚ด์šฉ. `{"์ด๋ฆ„": ๊ฐ’}`์ด ํ”„๋กฌํ”„ํŠธ์— ํ•œ ์ค„์”ฉ ๋“ค์–ด๊ฐ‘๋‹ˆ๋‹ค. ๋ฌธ์žฅ ํ•˜๋‚˜๋ฉด ๊ทธ๋ƒฅ ๋ฌธ์ž์—ด๋กœ ๋„ฃ์–ด๋„ ๋ฉ๋‹ˆ๋‹ค.
181
+ - **`instructions`**: ๋ฌด์—‡์„ ์–ด๋–ค ๊ธฐ์ค€์œผ๋กœ ๊ณ ๋ฅผ์ง€. ํŒ๋‹จ ๊ธฐ์ค€์„ ํ•œ๋‘ ๋ฌธ์žฅ ๋ง๋ถ™์ด๋ฉด(์˜ˆ: โ€œ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ํฌ๊ฒŒ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
182
+ - **`criteria`**: `{ID: ์„ค๋ช…}`, 2~26๊ฐœ. ID๋Š” ๊ฒฐ๊ณผ๋กœ ๋Œ๋ ค๋ฐ›๋Š” ๊ฐ’์ด๊ณ , ๋ชจ๋ธ์€ ์„ค๋ช…์„ ๋ณด๊ณ  ๊ณ ๋ฆ…๋‹ˆ๋‹ค. ID๊ฐ€ ํ•„์š” ์—†์œผ๋ฉด `["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก"]`์ฒ˜๋Ÿผ ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ์–ด๋„ ๋ฉ๋‹ˆ๋‹ค.
183
+
184
+ #### 3. ํ•œ ์ค„๋กœ
185
+
186
+ ```python
187
+ judge("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”", choices=["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก", "๊ธฐํƒ€"],
188
+ instruction="๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.")
189
+ ```
190
+
191
+ `judge.decide`์—์„œ ์งˆ๋ฌธ์ด ํ•˜๋‚˜์ผ ๋•Œ์™€ ๊ฐ™์Šต๋‹ˆ๋‹ค. ๋ฆฌ์ŠคํŠธ๋ฅผ ๋„ฃ๊ณ  `batch_size=8`์ฒ˜๋Ÿผ ์ฃผ๋ฉด ๋ฌถ์–ด์„œ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค.
192
+
193
+ ---
194
+
195
+ `trust_remote_code=True`๋Š” ์ด ์ €์žฅ์†Œ์˜ [`judge_pipeline.py`](judge_pipeline.py)์™€ [`judge.py`](judge.py)(ํ”„๋กฌํ”„ํŠธ ๋งŒ๋“ค๊ธฐ์™€ ๋ณด๊ธฐ ์ ์ˆ˜ ๊ณ„์‚ฐ)๋ฅผ ์‹คํ–‰ํ•œ๋‹ค๋Š” ๋œป์ž…๋‹ˆ๋‹ค. ์›๊ฒฉ ์ฝ”๋“œ๋ฅผ ์“ฐ๊ณ  ์‹ถ์ง€ ์•Š์œผ๋ฉด `judge.py`๋ฅผ ๋‚ด๋ ค๋ฐ›์•„ `from judge import Judge; judge = Judge()`๋กœ ๋˜‘๊ฐ™์ด ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค. [`quickstart.ipynb`](quickstart.ipynb)์— ์˜ˆ์‹œ๋ฅผ ๋ชจ์•„ ๋‘์—ˆ์Šต๋‹ˆ๋‹ค.
196
+
197
+ ## ์‚ฌ์šฉ ์˜ˆ์‹œ
198
+
199
+ ์ด ๋ชจ๋ธ์€ **โ€œ๋ฌด๏ฟฝ๏ฟฝ๏ฟฝ์„ ํ• ์ง€โ€๋ฅผ ๋น ๋ฅด๊ฒŒ ์ •ํ•˜๋Š” ์•ž๋‹จ**์— ๋‘๋Š” ๊ฒƒ์ด ๊ฐ€์žฅ ์ž˜ ๋งž์Šต๋‹ˆ๋‹ค. ์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณ ๋ฅด๋Š” ํŒ๋‹จ์€ ์ด ๋ชจ๋ธ์ด ํ•˜๊ณ , ๊ธ€ ์“ฐ๊ธฐ๋‚˜ ๊ฐ’ ์ฑ„์šฐ๊ธฐ์ฒ˜๋Ÿผ ์ƒ์„ฑ์ด ํ•„์š”ํ•œ ์ผ๋งŒ ํฐ ๋ชจ๋ธ์ด๋‚˜ ์‚ฌ๋žŒ์—๊ฒŒ ๋„˜๊ธฐ๋Š” ๊ตฌ์„ฑ์ž…๋‹ˆ๋‹ค. ์•„๋ž˜ ์˜ˆ์‹œ๋Š” ๋ชจ๋‘ ์œ„์—์„œ ๋งŒ๋“  `judge`๋กœ ๋ฐ”๋กœ ๋Œ์•„๊ฐ‘๋‹ˆ๋‹ค. ์ฃผ์„์— ์ ์€ ๊ฒฐ๊ณผ๋Š” ์˜๋„ํ•œ ์ •๋‹ต์ด๊ณ , ์‹ค์ œ ํ™•๋ฅ ์€ ์ง์ ‘ ์‹คํ–‰ํ•ด ํ™•์ธํ•˜์„ธ์š”.
200
+
201
+ ### 1. ๊ณ ๊ฐ ๋ฌธ์˜ ๋ถ„๋ฅ˜ โ€” ๋ถ€์„œ์™€ ๊ธด๊ธ‰๋„๋ฅผ ํ•œ ๋ฒˆ์—
202
+
203
+ ๋ฌธ์˜ ํ•˜๋‚˜๋ฅผ ์—ฌ๋Ÿฌ ๊ด€์ ์œผ๋กœ ๋™์‹œ์— ๋ถ„๋ฅ˜ํ•ฉ๋‹ˆ๋‹ค. ์งˆ๋ฌธ๋งˆ๋‹ค ๋ณด๊ธฐ ID๋ฅผ ์ •ํ•ด ๋‘๋ฉด ๊ฒฐ๊ณผ๋ฅผ ๋ฐ”๋กœ ์—…๋ฌด ์ฝ”๋“œ์— ์—ฐ๊ฒฐํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
204
+
205
+ ```python
206
+ def route_ticket(text):
207
+ r = judge.decide(
208
+ state={"๊ณ ๊ฐ ๋ฌธ์˜": text},
209
+ questions={
210
+ "๋ถ€์„œ": {
211
+ "instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค. ์—ฌ๋Ÿฌ ๋‚ด์šฉ์ด ์„ž์—ฌ ์žˆ์œผ๋ฉด ๊ณ ๊ฐ์ด ํ•ด๊ฒฐ์„ ์›ํ•˜๋Š” ๋ฌธ์ œ๋ฅผ ์šฐ์„ ํ•œ๋‹ค.",
212
+ "criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ",
213
+ "product": "์ œํ’ˆ ์‚ฌ์šฉยท๊ณ ์žฅ", "other": "๊ธฐํƒ€"},
214
+ },
215
+ "๊ธด๊ธ‰": {
216
+ "instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๊ฑฐ๋‚˜ ๊ธˆ์ „ ํ”ผํ•ด๊ฐ€ ์ง„ํ–‰ ์ค‘์ธ์ง€ ํŒ๋‹จํ•œ๋‹ค.",
217
+ "criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ํ•„์š”", "normal": "์ผ๋ฐ˜ ์ฒ˜๋ฆฌ"},
218
+ },
219
+ },
220
+ )
221
+ return r["๋ถ€์„œ"]["choice"], r["๊ธด๊ธ‰"]["choice"]
222
+
223
+ route_ticket("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ํ™˜๋ถˆํ•ด ์ฃผ์„ธ์š”.") # ('billing', 'urgent')
224
+ ```
225
+
226
+ - ์ง€์นจ์— **ํ—ท๊ฐˆ๋ฆฌ๋Š” ๊ฒฝ์šฐ์˜ ๊ธฐ์ค€**์„ ํ•œ ๋ฌธ์žฅ ๋„ฃ์œผ๋ฉด(โ€œ์—ฌ๋Ÿฌ ๋‚ด์šฉ์ด ์„ž์—ฌ ์žˆ์œผ๋ฉด โ€ฆ์„ ์šฐ์„ ํ•œ๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
227
+ - ๋ณด๊ธฐ์— `๊ธฐํƒ€`๋ฅผ ๋‘๋ฉด ์–ด๋””์—๋„ ์•ˆ ๋งž๋Š” ๋ฌธ์˜๋ฅผ ์–ต์ง€๋กœ ๋ผ์›Œ ๋„ฃ์ง€ ์•Š์Šต๋‹ˆ๋‹ค.
228
+
229
+ ### 2. ๋ฆฌ๋ทฐ ๊ฐ์ • ๋ถ„์„ โ€” ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ์™€ ์‚ฌ๋žŒ ๊ฒ€ํ†  ๊ธฐ์ค€
230
+
231
+ ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ์œผ๋ฉด ๋ฌถ์–ด์„œ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค. ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ๋”ฐ๋กœ ๋ชจ์•„ ์‚ฌ๋žŒ์ด ๋ด…๋‹ˆ๋‹ค.
232
+
233
+ ```python
234
+ reviews = ["๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”", "๊ฐ€๊ฒฉ ๋Œ€๋น„ ๊ดœ์ฐฎ์•„์š”", "๋‹ค์‹œ๋Š” ์•ˆ ์‚ฝ๋‹ˆ๋‹ค", "๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”"]
235
+ results = judge.sentiment(reviews)
236
+
237
+ auto = [(t, r["choice"]) for t, r in zip(reviews, results) if r["confidence"] >= 0.7]
238
+ review = [t for t, r in zip(reviews, results) if r["confidence"] < 0.7] # ์‚ฌ๋žŒ์ด ํ™•์ธ
239
+ ```
240
+
241
+ - **๋ฐ˜์–ดยท๋น„๊ผผ**(โ€œ๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”โ€)์€ ๋†’์€ ํ™•์‹ ์œผ๋กœ ํ‹€๋ฆฌ๊ธฐ๋„ ํ•ฉ๋‹ˆ๋‹ค. ์ค‘์š”ํ•œ ๊ณณ์—๋Š” ํ‘œ๋ณธ์„ ์‚ฌ๋žŒ์ด ๋‹ค์‹œ ๋ณด์„ธ์š”.
242
+ - ๊ธฐ์ค€๊ฐ’ 0.7์€ ์˜ˆ์‹œ์ž…๋‹ˆ๋‹ค. ์‹ค์ œ ๋ฐ์ดํ„ฐ ์ˆ˜๋ฐฑ ๊ฑด์— ์ •๋‹ต์„ ๋‹ฌ์•„ ๋ณด๊ณ  โ€œ์ด ๊ธฐ์ค€ ์ด์ƒ์ด๋ฉด ๋ช‡ %๊ฐ€ ๋งž๋Š”์ง€โ€๋ฅผ ํ™•์ธํ•ด ์ •ํ•˜์„ธ์š”.
243
+
244
+ ### 3. ๋ฆฌ๋ทฐ๋ฅผ ์—ฌ๋Ÿฌ ์ธก๋ฉด์œผ๋กœ โ€” ๋ฐฐ์†กยทํ’ˆ์งˆยท๊ฐ€๊ฒฉ ๊ฐ๊ฐ
245
+
246
+ ํ•œ ๋ฆฌ๋ทฐ์—์„œ ์ธก๋ฉด๋ณ„๋กœ ๋”ฐ๋กœ ํŒ๋‹จํ•ฉ๋‹ˆ๋‹ค. ๊ฐ™์€ `state`์— ์งˆ๋ฌธ๋งŒ ๋Š˜๋ฆฌ๋ฉด ๋ฉ๋‹ˆ๋‹ค.
247
+
248
+ ```python
249
+ ASPECT = {"pos": "๊ธ์ •์ ์œผ๋กœ ์–ธ๊ธ‰", "neg": "๋ถ€์ •์ ์œผ๋กœ ์–ธ๊ธ‰", "none": "์–ธ๊ธ‰ํ•˜์ง€ ์•Š์Œ"}
250
+
251
+ judge.decide(
252
+ state={"๋ฆฌ๋ทฐ": "๋ฐฐ์†ก์€ ํ•˜๋ฃจ ๋งŒ์— ์™”๋Š”๋ฐ ๋งˆ๊ฐ์ด ์—‰๋ง์ด๋„ค์š”. ์ด ๊ฐ€๊ฒฉ์ด๋ฉด ๊ทธ๋ƒฅ ๊ทธ๋ž˜์š”."},
253
+ questions={
254
+ "๋ฐฐ์†ก": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ๋ฐฐ์†ก(์†๋„ยทํฌ์žฅ)์„ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค.", "criteria": ASPECT},
255
+ "ํ’ˆ์งˆ": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ์ œํ’ˆ ํ’ˆ์งˆ(๋งˆ๊ฐยท๋‚ด๊ตฌ์„ฑยท์„ฑ๋Šฅ)์„ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค.", "criteria": ASPECT},
256
+ "๊ฐ€๊ฒฉ": {"instructions": "๋ฆฌ๋ทฐ๊ฐ€ ๊ฐ€๊ฒฉยท๊ฐ€์„ฑ๋น„๋ฅผ ์–ด๋–ป๊ฒŒ ์–ธ๊ธ‰ํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค. ์• ๋งคํ•œ ํ‰๊ฐ€๋Š” ์–ธ๊ธ‰ํ•œ ๊ฒƒ์œผ๋กœ ๋ณด๋˜ ๊ธ์ •ยท๋ถ€์ •์„ ๋ฌธ๋งฅ์œผ๋กœ ํŒ๋‹จํ•œ๋‹ค.", "criteria": ASPECT},
257
+ },
258
+ )
259
+ # {'๋ฐฐ์†ก': {'choice': 'pos', โ€ฆ}, 'ํ’ˆ์งˆ': {'choice': 'neg', โ€ฆ}, '๊ฐ€๊ฒฉ': {โ€ฆ}}
260
+ ```
261
+
262
+ ### 4. ๋Œ“๊ธ€ ๊ฒ€์ˆ˜ โ€” ํ˜์˜ค ์—ฌ๋ถ€์™€ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€
263
+
264
+ `hate`๋กœ ์ฐจ๋ณ„ยท๋น„ํ•˜ ์—ฌ๋ถ€๋ฅผ ๊ฑฐ๋ฅด๊ณ , `offensive`๋กœ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€์„ ๋‚˜๋ˆ•๋‹ˆ๋‹ค. ๊ธฐ์‚ฌ ๋Œ“๊ธ€์ด๋ฉด ์ œ๋ชฉ์„ ๋ฌธ๋งฅ์œผ๋กœ ํ•จ๊ป˜ ๋„ฃ์Šต๋‹ˆ๋‹ค.
265
+
266
+ ```python
267
+ comment = "์ด๋Ÿฐ ๊ธฐ์‚ฌ ์“ฐ๋Š” ๊ธฐ์ž๋Š” ๋จธ๋ฆฌ๊ฐ€ ์žˆ๋Š” ๊ฑด์ง€"
268
+
269
+ judge.hate(comment) # ๋น„ํ˜์˜ค / ํ˜์˜ค
270
+ judge.offensive(comment, news_title="์ •๋ถ€, ๋‚ด๋…„ ์˜ˆ์‚ฐ์•ˆ ๋ฐœํ‘œ") # ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค
271
+ ```
272
+
273
+ - ๋‘ ๋ฉ”์„œ๋“œ๋Š” ๊ธฐ์ค€์ด ๋‹ค๋ฆ…๋‹ˆ๋‹ค. `hate`๋Š” ์„ฑ๋ณ„ยท๋‚˜์ดยท์ถœ์‹  ๊ฐ™์€ ์ง‘๋‹จ์— ๋Œ€ํ•œ ์ฐจ๋ณ„ยท๋น„ํ•˜(APEACH ๊ธฐ์ค€), `offensive`๋Š” ๊ฐœ์ธ์„ ํ–ฅํ•œ ๋ชจ์š•๊นŒ์ง€ ํฌํ•จํ•œ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„(BEEP ๊ธฐ์ค€)์ž…๋‹ˆ๋‹ค.
274
+ - ์‚ฌ๋žŒ์—๊ฒŒ ๋ถˆ์ด์ต์„ ์ฃผ๋Š” ๊ฒฐ์ •(์‚ญ์ œยท์ œ์žฌ)์— ์“ธ ๋•Œ๋Š” ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ๋ฐ˜๋“œ์‹œ ์‚ฌ๋žŒ์ด ๊ฒ€ํ† ํ•˜๊ฒŒ ํ•˜์„ธ์š”.
275
+
276
+ ### 5. ์—์ด์ „ํŠธ ๋ผ์šฐํ„ฐ โ€” ๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€
277
+
278
+ ์—์ด์ „ํŠธ๊ฐ€ ๋งค ํ„ด ํฐ ๋ชจ๋ธ์— โ€œ๋‹ค์Œ์— ๋ญ˜ ํ• ์ง€โ€๋ฅผ ๋ฌป๋Š” ๋Œ€์‹ , ์ด ๋ชจ๋ธ์ด ๋จผ์ € ์ •ํ•ฉ๋‹ˆ๋‹ค. ๋„๊ตฌ ์ธ์ž ์ฑ„์šฐ๊ธฐ์™€ ๋‹ต๋ณ€ ์ž‘์„ฑ๋งŒ ํฐ ๋ชจ๋ธ์— ๋งก๊น๋‹ˆ๋‹ค.
279
+
280
+ ```python
281
+ tools = [
282
+ {"type": "function", "function": {"name": "call_taxi", "description": "ํƒ์‹œ๋ฅผ ํ˜ธ์ถœํ•œ๋‹ค",
283
+ "parameters": {"type": "object", "properties": {"pickup": {"type": "string"}, "destination": {"type": "string"}},
284
+ "required": ["pickup", "destination"]}}},
285
+ {"type": "function", "function": {"name": "get_weather", "description": "์ง€์—ญ์˜ ํ˜„์žฌ ๋‚ ์”จ๋ฅผ ์•Œ๋ ค์ค€๋‹ค",
286
+ "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}},
287
+ ]
288
+ messages = [{"role": "user", "content": "๊ฐ•๋‚จ์—ญ์—์„œ ์„œ์šธ์—ญ๊นŒ์ง€ ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜"}]
289
+
290
+ r = judge.next_action(messages, tools)
291
+ if r["confidence"] < 0.7:
292
+ action = "ํฐ ๋ชจ๋ธ์—๊ฒŒ ๋‹ค์Œ ํ–‰๋™์„ ๋งก๊น€"
293
+ elif r["choice"].startswith("CALL:"):
294
+ action = f"ํฐ ๋ชจ๋ธ๋กœ {r['choice'][5:]} ์˜ ์ธ์ž๋ฅผ ์ฑ„์›Œ ํ˜ธ์ถœ" # ์ด ๋ชจ๋ธ์€ ์ธ์ž๋ฅผ ์ฑ„์šฐ์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค
295
+ elif r["choice"] == "ASK":
296
+ action = "๋น ์ง„ ํ•„์ˆ˜ ์ •๋ณด๋ฅผ ์‚ฌ์šฉ์ž์—๊ฒŒ ๋˜๋ฌป๊ธฐ"
297
+ else: # RESPOND
298
+ action = "๋„๊ตฌ ์—†์ด ๋ฐ”๋กœ ๋‹ต๋ณ€"
299
+ ```
300
+
301
+ - ๋„๊ตฌ๋Š” **OpenAI ํ˜•์‹ ๊ทธ๋Œ€๋กœ**, ํŠนํžˆ `required`๋ฅผ ๊ผญ ๋„ฃ์œผ์„ธ์š”. ์ด๋ฆ„๋งŒ ์ฃผ๋ฉด ํ•„์ˆ˜ ์ธ์ž๋ฅผ ๋ณผ ์ˆ˜ ์—†์–ด ASK์™€ CALL์„ ๊ตฌ๋ณ„ํ•˜์ง€ ๋ชปํ•ฉ๋‹ˆ๋‹ค.
302
+ - ๋Œ€ํ™” ๊ธฐ๋ก(์•ž ํ„ด, ๋„๊ตฌ ๊ฒฐ๊ณผ `{"role": "tool", โ€ฆ}`)์„ ํ•จ๊ป˜ ๋„ฃ์œผ๋ฉด โ€œ์ด๋ฏธ ์ฒ˜๋ฆฌ๋์œผ๋‹ˆ ๊ฒฐ๊ณผ ์•ˆ๋‚ด(RESPOND)โ€ ๊ฐ™์€ ํŒ๋‹จ๋„ ํ•ฉ๋‹ˆ๋‹ค.
303
+ - ๋„๊ตฌ๋Š” ์ตœ๋Œ€ 24๊ฐœ๊นŒ์ง€ ๋„ฃ์„ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค (๋ณด๊ธฐ 26๊ฐœ ์ค‘ 2๊ฐœ๊ฐ€ ASKยทRESPOND).
304
+
305
+ ### 6. ์ •์ฑ…์— ๋”ฐ๋ฅธ ๊ณ ๊ฐ์„ผํ„ฐ ํ–‰๋™ ๊ฒฐ์ •
306
+
307
+ ์ •์ฑ… ๋ฌธ์„œ์™€ ๊ณ ๊ฐ ์š”์ฒญ์„ ์ฃผ๋ฉด ์‹คํ–‰ / ํ™•์ธ / ๋˜๋ฌป๊ธฐ / ์ด๊ด€ / ๊ฑฐ์ ˆ ์ค‘ ๋ฌด์—‡์„ ํ• ์ง€ ๊ณ ๋ฆ…๋‹ˆ๋‹ค. ์˜ˆ์™ธ ์กฐํ•ญ์ด ์ ์šฉ๋˜๋Š”์ง€๊นŒ์ง€ ๋”ฐ์ง‘๋‹ˆ๋‹ค.
308
+
309
+ ```python
310
+ policy = ("ํ•ด์™ธ ์นด๋“œ ์Šน์ธ ๊ธˆ์•ก์ด 100,000์› ์ด์ƒ์ด๋ฉด ๊ณ ๊ฐ์˜ ํ™•์ธ์„ ๋ฐ›์€ ํ›„์—๋งŒ ์‹คํ–‰ํ•œ๋‹ค. "
311
+ "๋‹ค๋งŒ VIP ๋“ฑ๊ธ‰ ๊ณ ๊ฐ์€ ํ™•์ธ ์—†์ด ๋ฐ”๋กœ ์‹คํ–‰ํ•œ๋‹ค.")
312
+
313
+ judge.policy_action(policy, "ํ™์ฝฉ์—์„œ ์นด๋“œ๋กœ 149,000์› ๊ฒฐ์ œํ•˜๋ ค๋Š”๋ฐ ์Šน์ธํ•ด์ค˜. ๋‚ด ๋“ฑ๊ธ‰์€ ์ผ๋ฐ˜์ด์•ผ.")
314
+ # โ†’ ASK_CONFIRMATION (๊ธˆ์•ก ๊ธฐ์ค€์„ ๋„˜์—ˆ๊ณ  VIP ์˜ˆ์™ธ๊ฐ€ ์•„๋‹˜)
315
+ ```
316
+
317
+ - ์ •์ฑ…์— **์ ํ˜€ ์žˆ๋Š” ๊ฒƒ๋งŒ** ๊ทผ๊ฑฐ๋กœ ํŒ๋‹จํ•˜๋„๋ก ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค. ์‹ค์ œ ์—…๋ฌด ๊ทœ์ •์„ ๋ชจ๋ธ์ด ์•Œ์•„์„œ ๋ณด์ถฉํ•˜์ง€ ์•Š์œผ๋‹ˆ, ํ•„์š”ํ•œ ์กฐ๊ฑด์€ ์ •์ฑ… ๋ฌธ์žฅ์— ๋‹ค ์“ฐ์„ธ์š”.
318
+ - ๋ณด๊ธฐ: `ANSWER`, `ASK_CLARIFICATION`, `ASK_CONFIRMATION`, `CALL_TOOL`, `CALL_MULTIPLE_TOOLS`, `ESCALATE`, `REFUSE`.
319
+
320
+ ### 7. ๋‹ต๋ณ€์ด ๊ทผ๊ฑฐ ๋ฌธ์„œ์™€ ๋งž๋Š”์ง€ ํ™•์ธ (RAG ๊ฒ€์ฆ)
321
+
322
+ ๊ฒ€์ƒ‰ํ•œ ๋ฌธ์„œ๋ฅผ ์ „์ œ๋กœ, ์ƒ์„ฑ๋œ ๋‹ต๋ณ€์˜ ๋ฌธ์žฅ์„ ๊ฐ€์„ค๋กœ ๋„ฃ์Šต๋‹ˆ๋‹ค. `contradiction`์ด๋ฉด ๋ฌธ์„œ์™€ ์–ด๊ธ‹๋‚œ ๋‹ต์ด๊ณ , `neutral`์ด๋ฉด ๋ฌธ์„œ์— ๊ทผ๊ฑฐ๊ฐ€ ์—†๋Š” ๋‚ด์šฉ์ž…๋‹ˆ๋‹ค.
323
+
324
+ ```python
325
+ doc = "์ž„๋Œ€์ธ์€ ๊ณ„์•ฝ์ด ๋๋‚˜๊ณ  ์ž„์ฐจ์ธ์ด ์ง‘์„ ๋น„์šด ๋‚ ๋ถ€ํ„ฐ 30์ผ ์ด๋‚ด์— ๋ณด์ฆ๊ธˆ์„ ๋Œ๋ ค์ค€๋‹ค."
326
+ answer_sentences = ["๋ณด์ฆ๊ธˆ์€ ์ง‘์„ ๋น„์šด ๋’ค 30์ผ ์•ˆ์— ๋Œ๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.",
327
+ "๋ณด์ฆ๊ธˆ์€ ์ง‘์„ ๋น„์šด ๋’ค 40์ผ ๋’ค์— ๋Œ๋ ค๋ฐ›์Šต๋‹ˆ๋‹ค.",
328
+ "๋ณด์ฆ๊ธˆ์—๋Š” ์—ฐ 3% ์ด์ž๊ฐ€ ๋ถ™์Šต๋‹ˆ๋‹ค."]
329
+
330
+ for s, r in zip(answer_sentences, judge.nli([doc] * len(answer_sentences), answer_sentences)):
331
+ print(r["choice"], s)
332
+ # entailment โ€ฆ 30์ผ ์•ˆ์— โ€ฆ โ†’ ๊ทผ๊ฑฐ ์žˆ์Œ
333
+ # contradiction โ€ฆ 40์ผ ๋’ค์— โ€ฆ โ†’ ๋ฌธ์„œ์™€ ์–ด๊ธ‹๋‚จ
334
+ # neutral โ€ฆ ์—ฐ 3% ์ด์ž โ€ฆ โ†’ ๋ฌธ์„œ์— ๊ทผ๊ฑฐ ์—†์Œ
335
+ ```
336
+
337
+ - ๋‹ต๋ณ€์€ **๋ฌธ์žฅ ๋‹จ์œ„๋กœ ๋‚˜๋ˆ ** ๋„ฃ์œผ์„ธ์š”. ์—ฌ๋Ÿฌ ์ฃผ์žฅ์ด ์„ž์ธ ๊ธด ๋‹ต๋ณ€์€ ํŒ๋‹จ์ด ํ๋ ค์ง‘๋‹ˆ๋‹ค.
338
+ - ํ•™์Šต ์ตœ๋Œ€ ๊ธธ์ด๊ฐ€ 2,048ํ† ํฐ์ด๋ผ ๊ธด ๋ฌธ์„œ๋Š” ๊ด€๋ จ ๋‹จ๋ฝ๋งŒ ์ž˜๋ผ ๋„ฃ์œผ์„ธ์š”.
339
+
340
+ ### 8. ์•ฝ๊ด€ยท๋ฌธ์„œ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค
341
+
342
+ ๋ฌธ์„œ๋ฅผ ์ฃผ๊ณ  ์งˆ๋ฌธ์— ์˜ˆ/์•„๋‹ˆ์˜ค๋กœ ๋‹ตํ•˜๊ฒŒ ํ•ฉ๋‹ˆ๋‹ค. ๋ฌธ์„œ์— ์—†๋Š” ์ƒ์‹์œผ๋กœ ๋‹ตํ•˜์ง€ ์•Š๋„๋ก ํ•™์Šตํ–ˆ์Šต๋‹ˆ๋‹ค.
343
+
344
+ ```python
345
+ terms = "๊ตฌ๋งค ํ›„ 7์ผ ์ด๋‚ด์—๋Š” ๋‹จ์ˆœ ๋ณ€์‹ฌ์œผ๋กœ๋„ ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‹ค. ๋‹จ, ํฌ์žฅ์„ ๋œฏ์€ ์†Œํ”„ํŠธ์›จ์–ด๋Š” ํ™˜๋ถˆ ๋Œ€์ƒ์—์„œ ์ œ์™ธํ•œ๋‹ค."
346
+
347
+ judge.yes_no("ํฌ์žฅ์„ ๋œฏ์€ ๊ฒŒ์ž„ CD๋ฅผ 3์ผ ๋งŒ์— ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‚˜์š”?", terms) # โ†’ ์•„๋‹ˆ์˜ค
348
+ judge.yes_no("๋œฏ์ง€ ์•Š์€ ํ‚ค๋ณด๋“œ๋ฅผ 5์ผ ๋งŒ์— ํ™˜๋ถˆํ•  ์ˆ˜ ์žˆ๋‚˜์š”?", terms) # โ†’ ์˜ˆ
349
+ ```
350
+
351
+ ### 9. ๋น„์Šทํ•œ ์งˆ๋ฌธ ์ฐพ๊ธฐ โ€” FAQ ์ค‘๋ณต ํ™•์ธ
352
+
353
+ ์ƒˆ๋กœ ๋“ค์–ด์˜จ ์งˆ๋ฌธ์ด ๊ธฐ์กด FAQ์™€ ๊ฐ™์€ ๋œป์ธ์ง€ ํ™•์ธํ•ฉ๋‹ˆ๋‹ค. ํ›„๋ณด๋Š” ๊ฒ€์ƒ‰(์ž„๋ฒ ๋”ฉ ๋“ฑ)์œผ๋กœ ๋จผ์ € ์ขํžˆ๊ณ , ์ตœ์ข… ํ™•์ธ์— ์ด ๋ชจ๋ธ์„ ์“ฐ๋Š” ๊ตฌ์„ฑ์ด ์ข‹์Šต๋‹ˆ๋‹ค.
354
+
355
+ ```python
356
+ faq = ["๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”", "๋ฐฐ์†ก์ง€๋Š” ์–ด๋–ป๊ฒŒ ๋ฐ”๊พธ๋‚˜์š”", "ํšŒ์› ํƒˆํ‡ด๋Š” ์–ด๋””์„œ ํ•˜๋‚˜์š”"]
357
+ q = "๋กœ๊ทธ์ธ ๋น„๋ฒˆ์ด ๊ธฐ์–ต์ด ์•ˆ ๋‚˜์š”"
358
+
359
+ hits = judge.similar([q] * len(faq), faq)
360
+ best = max(range(len(faq)), key=lambda i: hits[i]["probabilities"]["๊ฐ™์Œ"])
361
+ faq[best], hits[best]["choice"] # ('๋น„๋ฐ€๋ฒˆํ˜ธ๋ฅผ ์žŠ์–ด๋ฒ„๋ ธ์–ด์š”', '๊ฐ™์Œ')
362
+ ```
363
+
364
+ - `similar`๋Š” ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณผ์ œ์ž…๋‹ˆ๋‹ค (ํ•™์Šตํ•˜์ง€ ์•Š์€ KLUE-STS์—์„œ 85.9%).
365
+
366
+ ### 10. ๋‚ด ๋ถ„๋ฅ˜ ์ฒด๊ณ„ ๊ทธ๋Œ€๋กœ ์“ฐ๊ธฐ
367
+
368
+ ์–ด๋–ค ๋ถ„๋ฅ˜๋“  ์ง€์นจ๊ณผ ๋ณด๊ธฐ๋งŒ ์“ฐ๋ฉด ๋ฉ๋‹ˆ๋‹ค. ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณผ์ œ๋ผ๋„ **ํŒ๋‹จ ๊ธฐ์ค€์„ ์ง€์นจ์— ์ ์–ด ์ฃผ๋Š” ๊ฒƒ**์ด ์ •ํ™•๋„๋ฅผ ๊ฐ€์žฅ ํฌ๊ฒŒ ์ขŒ์šฐํ•ฉ๋‹ˆ๋‹ค.
369
+
370
+ ```python
371
+ judge.decide(
372
+ state={"๋ณด๊ณ ์„œ ๋ฌธ์žฅ": "3๋ถ„๊ธฐ ๋งค์ถœ์€ ์ „๋…„ ๋Œ€๋น„ 12% ์ฆ๊ฐ€ํ–ˆ์œผ๋‚˜ ์˜์—…์ด์ต๋ฅ ์€ ์†Œํญ ํ•˜๋ฝํ–ˆ๋‹ค."},
373
+ questions={
374
+ "๋…ผ์กฐ": {
375
+ "instructions": "๋ฌธ์žฅ์ด ํšŒ์‚ฌ ์‹ค์ ์„ ์–ด๋–ค ๋…ผ์กฐ๋กœ ์ „๋‹ฌํ•˜๋Š”์ง€ ๊ณ ๋ฅธ๋‹ค. ์ข‹์€ ์ ๊ณผ ๋‚˜์œ ์ ์ด ํ•จ๊ป˜ ์žˆ์œผ๋ฉด ํ˜ผํ•ฉ์œผ๋กœ ๋ณธ๋‹ค.",
376
+ "criteria": {"positive": "๊ธ์ •์ ", "negative": "๋ถ€์ •์ ", "mixed": "ํ˜ผํ•ฉ", "neutral": "์‚ฌ์‹ค ์ „๋‹ฌ๋งŒ"},
377
+ },
378
+ },
379
+ )
380
+ ```
381
+
382
+ **์ž˜ ์•ˆ ๋˜๋Š” ์“ฐ์ž„์ƒˆ**: ์ˆซ์ž ๋น„๊ตยท๊ณ„์‚ฐ์ด ํ•ต์‹ฌ์ธ ํŒ๋‹จ(โ€œ์–ด๋А ์ชฝ์ด ๋” ์‹ผ๊ฐ€โ€), ๋ฌธ์žฅ์—์„œ ๊ฐ’ ๋ฝ‘๊ธฐ, ๋‹ต๋ณ€ ์ž‘์„ฑ, ๋‹ค๋ฅธ ๋ชจ๋ธ ๋‹ต๋ณ€์˜ ํ’ˆ์งˆ ์ฑ„์ (LLM-as-a-judge). ์ด๋Ÿฐ ์ผ์€ ์ƒ์„ฑ ๋ชจ๋ธ์— ๋งก๊ธฐ์„ธ์š”.
383
+
384
+ ### ์›๋ฆฌ: transformers๋งŒ์œผ๋กœ ์ง์ ‘ ๊ณ„์‚ฐํ•˜๊ธฐ
385
+
386
+ ```python
387
+ import torch, json
388
+ from transformers import AutoModelForCausalLM, AutoTokenizer
389
+ from huggingface_hub import hf_hub_download
390
+
391
+ name = "coderecode95/KJH-Judge-4B-ko"
392
+ tok = AutoTokenizer.from_pretrained(name)
393
+ model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.bfloat16, device_map="auto").eval()
394
+ T = json.load(open(hf_hub_download(name, "calibration.json")))["temperature"] # 1.7179
395
+
396
+ choices = ["๊ธ์ •์  ๊ฐ์ •", "๋ถ€์ •์  ๊ฐ์ •"]
397
+ prompt = ("๋ฌธ์žฅ์˜ ์ „๋ฐ˜์ ์ธ ๊ฐ์ • ๊ทน์„ฑ์„ ํŒ๋‹จํ•œ๋‹ค.\n\n"
398
+ "text: ๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”\n\n"
399
+ "๋ณด๊ธฐ:\n" + "\n".join(f"{chr(65+i)}. {c}" for i, c in enumerate(choices)) +
400
+ "\n\n์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.")
401
+ ids = tok(f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n", return_tensors="pt", add_special_tokens=False).input_ids.to(model.device)
402
+ with torch.no_grad():
403
+ logits = model(ids).logits[0, -1]
404
+ letters = [tok.encode(chr(65 + i), add_special_tokens=False)[0] for i in range(len(choices))]
405
+ probs = torch.softmax(logits[letters].float() / T, -1)
406
+ print({c: round(p, 4) for c, p in zip(choices, probs.tolist())})
407
+ ```
408
+
409
+ ### ํ”„๋กฌํ”„ํŠธ ํ˜•์‹
410
+
411
+ ํ•™์Šต๊ณผ ํ‰๊ฐ€์— ์“ด ํ˜•์‹์ž…๋‹ˆ๋‹ค. ์ด ํ˜•์‹์„ ์ง€์ผœ์•ผ ์œ„ ์„ฑ๋Šฅ์ด ๋‚˜์˜ต๋‹ˆ๋‹ค.
412
+
413
+ ```
414
+ <|im_start|>user
415
+ {๊ณผ์ œ ์„ค๋ช…} โ† instructions
416
+
417
+ {ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’} โ† state ์˜ ํ•ญ๋ชฉ๋งˆ๋‹ค ํ•œ ์ค„. ๊ฐ’์ด ๋ฌธ์ž์—ด์ด ์•„๋‹ˆ๋ฉด JSON
418
+ {ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’}
419
+
420
+ ๋ณด๊ธฐ:
421
+ A. {๋ณด๊ธฐ 1} โ† criteria ์˜ ์„ค๋ช…
422
+ B. {๋ณด๊ธฐ 2}
423
+ โ€ฆ
424
+
425
+ ์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.<|im_end|>
426
+ <|im_start|>assistant
427
+ ```
428
+
429
+ - `judge.decide()`์™€ ๋ฉ”์„œ๋“œ๋“ค์€ ์ด ํ˜•์‹์„ ์ž๋™์œผ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค. ์ง์ ‘ ๋งŒ๋“ค ๋•Œ๋งŒ ์‹ ๊ฒฝ ์“ฐ๋ฉด ๋ฉ๋‹ˆ๋‹ค.
430
+ - ๋ณด๊ธฐ๋Š” 2~26๊ฐœ์ž…๋‹ˆ๋‹ค.
431
+ - ๊ณผ์ œ ์„ค๋ช…์€ ์งง๊ณ  ๊ตฌ์ฒด์ ์œผ๋กœ ์“ฐ์„ธ์š”. ํŒ๋‹จ ๊ธฐ์ค€์„ ํ•œ๋‘ ๋ฌธ์žฅ ๋ง๋ถ™์ด๋ฉด(์˜ˆ: โ€œ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹คโ€, โ€œ์ •๋ณด๊ฐ€ ์—†๋‹ค๋Š” ์ด์œ ๋งŒ์œผ๋กœ ๋ชจ์ˆœ์„ ๊ณ ๋ฅด์ง€ ์•Š๋Š”๋‹คโ€) ์ •ํ™•๋„๊ฐ€ ํฌ๊ฒŒ ์˜ค๋ฆ…๋‹ˆ๋‹ค.
432
+ - `generate()`๋กœ ์“ธ ๊ฒฝ์šฐ `max_new_tokens=1`๋กœ ๋ณด๊ธฐ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋ฐ›์œผ์„ธ์š” (์ด ์ €์žฅ์†Œ์˜ `generation_config.json`์€ `do_sample: false`์ž…๋‹ˆ๋‹ค). ํ™•๋ฅ ์ด ํ•„์š”ํ•˜๋ฉด ์œ„์ฒ˜๋Ÿผ logits๋ฅผ ์ฝ์œผ์„ธ์š”.
433
+
434
+ ## ์„ฑ๋Šฅ
435
+
436
+ ### 1. ํ•™์Šตํ•˜์ง€ ์•Š์€ ๊ณต๊ฐœ ๋ฒค์น˜๋งˆํฌ
437
+
438
+ ์ด ๋ชจ๋ธ์˜ ํ•™์Šต์— ์“ฐ์ง€ ์•Š์€ ์ถœ์ฒ˜์˜ ๊ณต๊ฐœ ๋ฐ์ดํ„ฐ์ž…๋‹ˆ๋‹ค (KLUE-STS๋Š” ํ•™์Šต์— ์“ด KLUE NLIยทYNAT์™€ ๊ฐ™์€ KLUE ๋ฌถ์Œ์ด์ง€๋งŒ, STS ๊ณผ์ œ๋Š” ํ•™์Šตํ•˜์ง€ ์•Š์•˜์Šต๋‹ˆ๋‹ค). 0-shot, ์œ„ ํ”„๋กฌํ”„ํŠธ ํ˜•์‹, ๋ณด๊ธฐ ๊ธฐํ˜ธ ์ ์ˆ˜ ๋ฐฉ์‹์œผ๋กœ ์Ÿ€์Šต๋‹ˆ๋‹ค (ํ‘œ๋ณธ์€ ๊ณ ์ • ์‹œ๋“œ, ๊ด„ํ˜ธ๋Š” 95% ์‹ ๋ขฐ๊ตฌ๊ฐ„). ์žฌํ˜„ ์Šคํฌ๋ฆฝํŠธ: [`eval/bench_heldout.py`](eval/bench_heldout.py).
439
+
440
+ <!-- HELDOUT:start -->
441
+ | ๋ฒค์น˜๋งˆํฌ | ๋ฌด์—‡์„ ์žฌ๋‚˜ | ๋ฌธํ•ญ | ์ •ํ™•๋„ (95% ๊ตฌ๊ฐ„) | ๋ฌด์ž‘์œ„ | ์ถœ์ฒ˜ ยท ๋ผ์ด์„ ์Šค |
442
+ |---|---|---:|---:|---:|---|
443
+ | NSMC | ์˜ํ™” ๋ฆฌ๋ทฐ ๊ฐ์„ฑ (๊ธ์ •/๋ถ€์ •) | 1,000 | **85.7%** (83.4~87.7) | 50% | e9t/nsmc ยท ratings_test ยท CC0 1.0 |
444
+ | KorNLI (XNLI) | ๋ฌธ์žฅ ๊ด€๊ณ„ 3๋ถ„๋ฅ˜ (XNLI ํ•œ๊ตญ์–ด ๋ฒˆ์—ญ) | 1,000 | **74.5%** (71.7~77.1) | 33% | kakaobrain/kor_nli ยท xnli/test ยท CC BY-SA 4.0 |
445
+ | KLUE-STS | ๋‘ ๋ฌธ์žฅ์ด ๊ฐ™์€ ๋œป์ธ์ง€ (์˜ˆ/์•„๋‹ˆ์˜ค) | 519 | **85.9%** (82.7~88.7) | 50% | klue/klue ยท sts/validation (binary-label) ยท CC BY-SA 4.0 |
446
+ | Korean UnSmile | ์•…ํ”Œยทํ˜์˜ค ํ‘œํ˜„ ์—ฌ๋ถ€ | 1,000 | **82.1%** (79.6~84.4) | 50% | smilegate-ai/kor_unsmile ยท valid (clean ์—ฌ๋ถ€) ยท CC BY-NC-ND 4.0 |
447
+ | KMMLU | ํ•œ๊ตญ์–ด ์ „๋ฌธ ์ง€์‹ 4์ง€์„ ๋‹ค (45๊ณผ๋ชฉ) | 990 | **42.6%** (39.6~45.7) | 25% | HAERAE-HUB/KMMLU ยท test ยท CC BY-ND 4.0 |
448
+ | HAE-RAE Bench 1.0 | ํ•œ๊ตญ ๋ฌธํ™”ยท์–ดํœ˜ ์ง€์‹ 5์ง€์„ ๋‹ค | 983 | **56.1%** (53.0~59.2) | 21% | HAERAE-HUB/HAE_RAE_BENCH_1.0 ยท test ยท CC BY-NC-ND 4.0 |
449
+ <!-- HELDOUT:end -->
450
+
451
+ <!-- COMPARE:start -->
452
+ **๊ฐ™์€ ์กฐ๊ฑด ๋น„๊ต** ยท ๊ฐ™์€ ๋ฌธํ•ญ ยท ๊ฐ™์€ ํ”„๋กฌํ”„ํŠธ ยท 0-shot ยท ๋ณด๊ธฐ ๊ธฐํ˜ธ ์ ์ˆ˜. ๋‹ค๋ฅธ ๋ชจ๋ธ์€ ์ƒ๊ฐ(thinking)์„ ๋„๊ณ  ๊ฐ์ž์˜ ๋Œ€ํ™” ํ˜•์‹์œผ๋กœ ๋ฌผ์—ˆ์Šต๋‹ˆ๋‹ค.
453
+
454
+ | ๋ฒค์น˜๋งˆํฌ | **KJH-Judge-4B-ko (4B)** | Gemma 4 26B-A4B-it (0-shot) |
455
+ |---|---:|---:|
456
+ | NSMC | **85.7%** | 88.7% |
457
+ | KorNLI (XNLI) | **74.5%** | 77.1% |
458
+ | KLUE-STS | **85.9%** | 88.4% |
459
+ | Korean UnSmile | **82.1%** | 81.3% |
460
+ | KMMLU | **42.6%** | 64.0% |
461
+ | HAE-RAE Bench 1.0 | **56.1%** | 79.7% |
462
+ <!-- COMPARE:end -->
463
+
464
+ ์›๋ณธ Qwen3-4B์˜ ์‚ฌ์ „ํ•™์Šต์— ์ด ๋ฐ์ดํ„ฐ๊ฐ€ ๋“ค์–ด๊ฐ”๋Š”์ง€๋Š” ์•Œ ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค. KMMLUยทHAE-RAE๋Š” ์ „๋ฌธ ์ง€์‹์„ ๋ฌป๋Š” ์‹œํ—˜์ด๋ผ, ํŒ์ • ๋Šฅ๋ ฅ๋ณด๋‹ค ์›๋ณธ ๋ชจ๋ธ์˜ ์ง€์‹์ด ์ ์ˆ˜๋ฅผ ์ขŒ์šฐํ•ฉ๋‹ˆ๋‹ค.
465
+
466
+ ### 2. Korean JEV Benchmark (์ฐธ๊ณ : ๊ฐ™์€ ์ถœ์ฒ˜์˜ ์‹œํ—˜ ๋ฌธํ•ญ)
467
+
468
+ [DaehanKim/korean-jev-benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark) 25,418๋ฌธํ•ญ ์ „์ฒด์ž…๋‹ˆ๋‹ค. ์ด ์‹œํ—˜ ๋ฌธํ•ญ ์ž์ฒด๋กœ ํ•™์Šตํ•˜์ง€๋Š” ์•Š์•˜์ง€๋งŒ, **๊ฐ™์€ ์ถœ์ฒ˜ ๋ฐ์ดํ„ฐ(KLUE, KoBEST ๋“ฑ)์˜ ํ•™์Šต ๋ถ„ํ• ๋กœ ํ•™์Šตํ–ˆ๊ธฐ ๋•Œ๋ฌธ์—** ๋‹ค๋ฅธ ๋ชจ๋ธ์˜ 0-shot ์ ์ˆ˜์™€ ๊ฐ™์€ ์กฐ๊ฑด์˜ ๋น„๊ต๊ฐ€ ์•„๋‹™๋‹ˆ๋‹ค.
469
+
470
+ | ํ•ญ๋ชฉ | ๊ฐ’ |
471
+ |---|---|
472
+ | ์ „์ฒด ์ •ํ™•๋„ | **90.37%** (ํ•™์Šต ์ „ Qwen3-4B 29.56%, ๋ฌด์ž‘์œ„ 29.86%) |
473
+ | ํ•™์Šต ๋ฐ์ดํ„ฐ๊ฐ€ ์—†๋Š” ๊ณผ๋ชฉ (Belebele ํ•œ๊ตญ์–ด ๋…ํ•ด) | **86.87%** |
474
+ | ๋ณด๊ธฐ๋ฅผ ๊ฑฐ๊พธ๋กœ ๋†“์•„๋„ ๊ฐ™์€ ๋‹ต | 97.93% |
475
+ | ๊ธฐ๋Œ€ ๋ณด์ • ์˜ค์ฐจ (ECE) | 0.0265 |
476
+
477
+ | ๊ณผ๋ชฉ | ์ •ํ™•๋„ | ๊ณผ๋ชฉ | ์ •ํ™•๋„ |
478
+ |---|---:|---|---:|
479
+ | KoBEST BoolQ | 95.65% | KLUE YNAT (๋‰ด์Šค ์ฃผ์ œ) | 93.93% |
480
+ | KoBEST HellaSwag | 94.65% | KLUE NLI | 92.39% |
481
+ | KoBEST SentiNeg | 93.91% | APEACH (ํ˜์˜ค) | 88.64% |
482
+ | KoBEST COPA | 92.55% | KoBEST WiC | 84.96% |
483
+ | SIB-200 | 94.44% | Korean Hate Speech (3๋ถ„๋ฅ˜) | 77.87% |
484
+ | FunctionChat ๋„๊ตฌ ๊ณ ๋ฅด๊ธฐ | 94.68% | EnterpriseOps ์ •์ฑ… ํ–‰๋™ (7๋ถ„๋ฅ˜) | 83.43% |
485
+ | FunctionChat ๋Œ€ํ™” | 93.50% | ๋ฒ•๋ น ์งˆ๋ฌธ (v2ยทv3, 6๊ฐœ ๋ฌถ์Œ) | 90.8~99.2% |
486
+ | FunctionChat ํ˜ธ์ถœ ํŒ๋‹จ | 84.55% | ๋ฒ•๋ น ๊ธฐ๊ฐ„ ๊ณ„์‚ฐ | 69.10% |
487
+
488
+ ### ์†๋„
489
+
490
+ ๊ฐœ์ธ GPU ์„œ๋ฒ„์—์„œ ๊ฐ™์€ ๋ฌธํ•ญ 60๊ฐœ(NSMCยทKorNLI)๋กœ ์žฐ ์ฐธ๊ณ ๊ฐ’์ž…๋‹ˆ๋‹ค (๋„คํŠธ์›Œํฌ ์™•๋ณต ํฌํ•จ, ์ธก์ • ์Šคํฌ๋ฆฝํŠธ: [`eval/bench_speed.py`](eval/bench_speed.py)).
491
+
492
+ | ์ธก์ • | **KJH-Judge-4B-ko** | Gemma 4 26B-A4B-it |
493
+ |---|---:|---:|
494
+ | ํŒ์ • ํ•œ ๋ฒˆ (์ค‘์•™๊ฐ’) | **250ms** (์„œ๋ฒ„ ์•ˆ ๊ณ„์‚ฐ 87ms) | 621ms |
495
+ | ๋™์‹œ 4๊ฐœ์”ฉ ๋ณด๋ƒˆ์„ ๋•Œ ์ฒ˜๋ฆฌ๋Ÿ‰ | 3.9๊ฑด/์ดˆ | **9.9๊ฑด/์ดˆ** |
496
+ | ์ƒ๊ฐ์„ ์ผœ๊ณ  ๋‹ต์„ ์ƒ์„ฑํ•  ๋•Œ | | 6.3์ดˆ |
497
+
498
+ - ๋™์‹œ ์ฒ˜๋ฆฌ๋Ÿ‰์€ ๋น„๊ต ๋ชจ๋ธ์ด ์•ž์„ฐ์Šต๋‹ˆ๋‹ค. ๋น„๊ต ๋ชจ๋ธ์€ ์š”์ฒญ์„ ๋ฌถ์–ด ์ฒ˜๋ฆฌํ•˜๋Š” ์„œ๋น™ ์—”์ง„์—์„œ, ์ด ๋ชจ๋ธ์€ ํ•˜๋‚˜์”ฉ ์ฒ˜๋ฆฌํ•˜๋Š” ๋‹จ์ˆœํ•œ ์„œ๋ฒ„์—์„œ ๋Œ์•˜๊ธฐ ๋•Œ๋ฌธ์ž…๋‹ˆ๋‹ค. ๋Œ€๋Ÿ‰ ์ฒ˜๋ฆฌ์—๋Š” ๋ฆฌ์ŠคํŠธ๋กœ ๋„ฃ๊ณ  `batch_size`๋ฅผ ์ฃผ์–ด ๋ฌถ์–ด ๊ณ„์‚ฐํ•˜์„ธ์š”.
499
+ - CPU(๋…ธํŠธ๋ถ, fp32)์—์„œ๋Š” ํŒ์ • ํ•œ ๋ฒˆ์— 6์ดˆ์—์„œ ์ˆ˜ ๋ถ„์ด ๊ฑธ๋ ค ์‹ค์‚ฌ์šฉ์ด ์–ด๋ ต์Šต๋‹ˆ๋‹ค.
500
+
501
+ ## ์ž˜ํ•˜๋Š” ๊ฒƒ๊ณผ ๋ชปํ•˜๋Š” ๊ฒƒ
502
+
503
+ **์ž˜ํ•˜๋Š” ๊ฒƒ**: ๋ณด๊ธฐ๊ฐ€ ์ •ํ•ด์ง„ ๋ถ„๋ฅ˜, ๋ฌธ์žฅ ๊ด€๊ณ„ ํŒ๋‹จ, ์ง€๋ฌธ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค, ์ •์ฑ…ยท๊ทœ์น™์— ๋”ฐ๋ฅธ ํ–‰๋™ ์„ ํƒ, ์—์ด์ „ํŠธ์˜ ๋‹ค์Œ ํ–‰๋™(๋„๊ตฌ ํ˜ธ์ถœ / ๋˜๋ฌป๊ธฐ / ์ง์ ‘ ๋‹ต๋ณ€) ๊ฒฐ์ •. ์ฒ˜์Œ ๋ณด๋Š” ์ง€์‹œ๋ฌธ์ด๋ผ๋„ ์ด๋Ÿฐ ํ˜•์‹์ด๋ฉด ์ž˜ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
504
+
505
+ **๋ชปํ•˜๋Š” ๊ฒƒ**
506
+ - **๊ธ€ ์ƒ์„ฑ, ๋ฌธ์žฅ์—์„œ ๊ฐ’ ๋ฝ‘๊ธฐ**: ๋ณด๊ธฐ๋ฅผ ๊ณ ๋ฅด๋Š” ๊ฒƒ๋งŒ ํ•ฉ๋‹ˆ๋‹ค. ๋„๊ตฌ ์ธ์ž ๊ฐ’์„ ์ฑ„์šฐ๊ฑฐ๋‚˜ ๋‹ต๋ณ€์„ ์“ฐ๋Š” ์ผ์€ ์ƒ์„ฑ ๋ชจ๋ธ์— ๋งก๊ธฐ์„ธ์š”.
507
+ - **์ˆซ์ž ๋น„๊ตยท๊ณ„์‚ฐ์ด ํ•ต์‹ฌ์ธ ํŒ๋‹จ**: ๊ฑฐ๋ฆฌยท์†๋„ยท์ ์ˆ˜ ๊ฐ™์€ ์ˆ˜์น˜๋ฅผ ๋น„๊ตํ•ด ๊ณ ๋ฅด๋Š” ์ผ์€ ์•ฝํ•ฉ๋‹ˆ๋‹ค.
508
+ - **๋ฐ˜์–ดยท๋น„๊ผผ**: โ€œ๋ฐฐ์†ก ์ฐธ ๋น ๋ฅด๋„ค์š”. 2์ฃผ ๊ฑธ๋ ธ์–ด์š”โ€๋ฅผ ๊ธ์ •์œผ๋กœ ๋†’์€ ํ™•์‹ ๊ณผ ํ•จ๊ป˜ ๊ณ ๋ฅด๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
509
+ - **์ง€์‹œ๋ฌธ ๋ฌธ๊ตฌ์— ๋ฏผ๊ฐ**: โ€œ์‹ค์ œ ์กด์žฌํ•˜๋Š” ์ธ๋ฌผ์ธ๊ฐ€์š”?โ€๋ฅผ โ€œ์‹ค์ œ ์กด์žฌํ•˜๋‚˜์š”?โ€์ฒ˜๋Ÿผ ์ฝ๋Š” ๊ฒฝ์šฐ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ๊ธฐ์ค€์„ ๋ช…ํ™•ํžˆ ์“ฐ์„ธ์š”.
510
+
511
+ ๊ถŒ์žฅ ๊ตฌ์„ฑ: ํฐ LLM ์•ž๋‹จ์—์„œ ์ด ๋ชจ๋ธ์ด โ€œ๋ฌด์—‡์„ ํ• ์ง€โ€๋ฅผ ๋น ๋ฅด๊ณ  ์‹ธ๊ฒŒ ์ •ํ•˜๊ณ , ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฑด ์‚ฌ๋žŒ์ด๋‚˜ ํฐ ๋ชจ๋ธ์— ๋„˜๊ธฐ์„ธ์š”.
512
+
513
+ ## ํ•™์Šต
514
+
515
+ > ํ•™์Šต ๋ฐ์ดํ„ฐ ๊ตฌ์„ฑ, ๋ณด๊ธฐ ์ˆœ์„œ ์—ญ์ˆœ ํ•™์Šต, ์˜ค์—ผ ์ œ๊ฑฐ, ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ ๋งŒ๋“œ๋Š” ๋ฒ•, ํ™•๋ฅ  ๋ณด์ •, ์ฒดํฌํฌ์ธํŠธ๋ณ„ ์ •ํ™•๋„๊นŒ์ง€ ์ž์„ธํ•œ ๊ณผ์ •์€ **[TRAINING.md](TRAINING.md)**์— ์ •๋ฆฌํ–ˆ์Šต๋‹ˆ๋‹ค.
516
+
517
+ | ํ•ญ๋ชฉ | ๊ฐ’ |
518
+ |---|---|
519
+ | ์›๋ณธ ๋ชจ๋ธ | [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) |
520
+ | ๋ฐฉ๋ฒ• | SFT, LoRA (r 16, ฮฑ 32, dropout 0.05, qยทkยทvยทoยทgateยทupยทdown) โ†’ ์›๋ณธ์— ๋ณ‘ํ•ฉ |
521
+ | ํ•™์Šต๋Ÿ‰ | 69,144๋ฌธํ•ญ ร— ๋ณด๊ธฐ ์ˆœ์„œ 2๊ฐ€์ง€(์„ž์€ ์ˆœ์„œ + ์—ญ์ˆœ), ์ถœ์ฒ˜ ๋น„์œจ๋Œ€๋กœ ์„ž์–ด 4,430 step ยท ๋ฐฐ์น˜ 32, ์ตœ๋Œ€ ๊ธธ์ด 2,048 |
522
+ | ์ตœ์ ํ™” | ํ•™์Šต๋ฅ  2e-4, cosine, warmup 3% |
523
+ | ์žฅ๋น„ | GPU 1์žฅ, ์•ฝ 3.5์‹œ๊ฐ„ |
524
+ | ์ •๋‹ต ํ˜•์‹ | ๋ณด๊ธฐ ๊ธฐํ˜ธ ํ•œ ๊ธ€์ž + `<|im_end|>` |
525
+ | ๋ณด์ • | ๊ฒ€์ฆ์…‹ 3,089๋ฌธํ•ญ์œผ๋กœ ๋‹จ์ผ ์˜จ๋„(1.7179)๋ฅผ ๋งž์ถค. ECE 0.066 โ†’ 0.013 |
526
+
527
+ **ํ•™์Šต ๋ฐ์ดํ„ฐ** (๋ชจ๋‘ ์œ„ ํ”„๋กฌํ”„ํŠธ ํ˜•์‹์œผ๋กœ ๋ฐ”๊ฟ” ์‚ฌ์šฉ, ํ‰๊ฐ€ ๋ฌธํ•ญ๊ณผ ๊ฒน์น˜๋Š” ๋ฌธ์žฅ์€ ์ œ์™ธ)
528
+
529
+ | ๋ฐ์ดํ„ฐ | ๋ผ์ด์„ ์Šค |
530
+ |---|---|
531
+ | KLUE NLI ยท YNAT (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
532
+ | KoBEST BoolQ ยท COPA ยท HellaSwag ยท SentiNeg ยท WiC (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
533
+ | kocohub/korean-hate-speech (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
534
+ | APEACH (ํ•™์Šต ๋ถ„ํ• ) | CC BY-SA 4.0 |
535
+ | SIB-200 (ํ•™์Šต ๋ถ„๏ฟฝ๏ฟฝ) | CC BY-SA 4.0 |
536
+ | ์ •์ฑ… ํ–‰๋™ ยท ๋ฒ•๋ น ์งˆ๋ฌธ ยท ๋„๊ตฌ ํ˜ธ์ถœ ํŒ๋‹จ | ์ž์ฒด ํ•ฉ์„ฑ (๊ทœ์น™์œผ๋กœ ์ƒ์„ฑ, ์ •๋‹ต์€ ์ฝ”๋“œ๋กœ ๊ณ„์‚ฐ. ์ •์ฑ…ยท๋ฒ•๋ นยท๋„๊ตฌ๋Š” ๋ชจ๋‘ ์ง€์–ด๋‚ธ ๊ฒƒ์ด๋ฉฐ LLM ์ƒ์„ฑ๋ฌผ์„ ์“ฐ์ง€ ์•Š์Œ) |
537
+
538
+ ## ๋ผ์ด์„ ์Šค์™€ ์ฃผ์˜
539
+ - ๊ณต๊ฐœ ํ•™์Šต ๋ฐ์ดํ„ฐ๋Š” ๋ชจ๋‘ CC BY-SA 4.0์ž…๋‹ˆ๋‹ค.
540
+ - [`judge.py`](judge.py)์˜ ๊ณผ์ œ๋ณ„ ์ง€์นจยท๋ณด๊ธฐ ๋ฌธ๊ตฌ(`TASKS`, `CRITERIA`)์™€ ํ‰๊ฐ€ ์Šคํฌ๋ฆฝํŠธ์˜ ์ง€์นจ ์ผ๋ถ€๋Š” [Korean JEV Benchmark](https://huggingface.co/datasets/DaehanKim/korean-jev-benchmark)(DaehanKim)์˜ ๊ณผ์ œ ๋ฌธ๊ตฌ๋ฅผ ๊ทธ๋Œ€๋กœ ์“ด ๊ฒƒ์œผ๋กœ, [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)์„ ๋”ฐ๋ฆ…๋‹ˆ๋‹ค.
541
+ - `space/` ํด๋”(์‡ผ๋ฆด ํŽ˜์ด์ง€)๋Š” KoBESTยทKLUE ๋ฌธ์žฅ์„ ํฌํ•จํ•˜๊ณ  ์žˆ์–ด CC BY-SA 4.0์ž…๋‹ˆ๋‹ค.
542
+ - ํ˜์˜ค ํ‘œํ˜„ ํŒ์ •, ์ •์ฑ… ํŒ๋‹จ ๋“ฑ ์‚ฌ๋žŒ์—๊ฒŒ ์˜ํ–ฅ์„ ์ฃผ๋Š” ๊ฒฐ์ •์— ์“ธ ๋•Œ๋Š” ํ™•์‹ ์ด ๋‚ฎ์€ ๊ฒฐ๊ณผ๋ฅผ ์‚ฌ๋žŒ์ด ๊ฒ€ํ† ํ•˜๋„๋ก ํ•˜์„ธ์š”.
543
+ - ๋ฒ•๋ฅ ยท๊ธˆ์œต ํŒ๋‹จ์˜ ๊ทผ๊ฑฐ๋กœ ์“ฐ์ง€ ๋งˆ์„ธ์š”. ์ฃผ์–ด์ง„ ๊ธ€ ์•ˆ์—์„œ ๋‚ด์šฉ์„ ์ฐพ๊ณ  ๋Œ€์กฐํ•˜๋Š” ๋„๊ตฌ์ž…๋‹ˆ๋‹ค.