KJH-Judge-4B-ko / judge.py
coderecode95's picture
Upload folder using huggingface_hub
cca00cc verified
Raw History Blame Contribute Delete
20 kB
"""KJH-Judge-4B-ko ํŒ์ • ๋„์šฐ๋ฏธ โ€” ์ •ํ•ด์ง„ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜๋ฅผ ๊ณจ๋ผ ๋ณด์ •๋œ ํ™•๋ฅ ๊ณผ ํ•จ๊ป˜ ๋Œ๋ ค์ค€๋‹ค.
from transformers import pipeline
judge = pipeline(model="coderecode95/KJH-Judge-4B-ko", trust_remote_code=True, device=0, dtype="bfloat16")
# ๋˜๋Š”: from judge import Judge; judge = Judge() (์ด ํŒŒ์ผ์„ ๋‚ด๋ ค๋ฐ›์•„ ์“ธ ๋•Œ)
1) JEV ํ˜•์‹ โ€” ์ƒํ™ฉ(state) ํ•˜๋‚˜์— ์ด๋ฆ„ ๋ถ™์ธ ์งˆ๋ฌธ ์—ฌ๋Ÿฌ ๊ฐœ. ๋ณด๊ธฐ ID๋Š” ๋งˆ์Œ๋Œ€๋กœ ์ •ํ•œ๋‹ค.
judge.decide(
state={"๊ณ ๊ฐ ๋ฌธ์˜": "์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”. ์˜ค๋Š˜ ์•ˆ์— ์ฒ˜๋ฆฌํ•ด ์ฃผ์„ธ์š”."},
questions={
"๋ถ€์„œ": {"instructions": "๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.",
"criteria": {"billing": "๊ฒฐ์ œยทํ™˜๋ถˆ", "delivery": "๋ฐฐ์†ก", "account": "๊ณ„์ •ยท๋กœ๊ทธ์ธ"}},
"๊ธด๊ธ‰": {"instructions": "๊ณ ๊ฐ์ด ์ฆ‰์‹œ ์ฒ˜๋ฆฌ๋ฅผ ์š”๊ตฌํ•˜๋Š”์ง€ ํŒ๋‹จํ•œ๋‹ค.",
"criteria": {"urgent": "์ฆ‰์‹œ ์ฒ˜๋ฆฌ ์š”๊ตฌ", "normal": "์ผ๋ฐ˜ ๋ฌธ์˜"}},
},
)
# โ†’ {'๋ถ€์„œ': {'choice': 'billing', 'confidence': 0.98, 'probabilities': {...}},
# '๊ธด๊ธ‰': {'choice': 'urgent', 'confidence': 0.91, 'probabilities': {...}}}
2) ์ž์ฃผ ์“ฐ๋Š” ํŒ์ • โ€” ํ•™์Šต ๋•Œ์™€ ๊ฐ™์€ ์ง€์นจยท๋ณด๊ธฐ๋ฅผ ์ž๋™์œผ๋กœ ๋„ฃ๋Š”๋‹ค.
judge.sentiment("๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”") # โ†’ choice: ๊ธ์ • / ๋ถ€์ •
judge.nli("๋ณด์ฆ๊ธˆ์€ 30์ผ ์ด๋‚ด ๋ฐ˜ํ™˜ํ•œ๋‹ค.", "๋ณด์ฆ๊ธˆ์€ 40์ผ ๋’ค์— ๋ฐ›๋Š”๋‹ค.") # โ†’ entailment / contradiction / neutral
judge.next_action("ํƒ์‹œ ๋ถˆ๋Ÿฌ์ค˜", tools) # โ†’ CALL:call_taxi / ASK / RESPOND
3) ํ•œ ์ค„๋กœ โ€” judge(์ƒํ™ฉ, ๋ณด๊ธฐ, instruction=โ€ฆ). ๊ฒฐ๊ณผ ํ˜•์‹์€ ๋ชจ๋‘ ๊ฐ™๋‹ค.
judge("์ง€๋‚œ๋‹ฌ ์š”๊ธˆ์ด ๋‘ ๋ฒˆ ๋น ์ ธ๋‚˜๊ฐ”์–ด์š”", ["๊ฒฐ์ œยทํ™˜๋ถˆ", "๋ฐฐ์†ก", "๊ธฐํƒ€"], instruction="๊ณ ๊ฐ ๋ฌธ์˜๋ฅผ ๋‹ด๋‹น ๋ถ€์„œ๋กœ ๋ถ„๋ฅ˜ํ•œ๋‹ค.")
๋™์ž‘: ์•„๋ž˜ ํ˜•์‹์˜ ํ”„๋กฌํ”„ํŠธ๋ฅผ ํ•œ ๋ฒˆ ๊ณ„์‚ฐํ•˜๊ณ , ๋‹ค์Œ ํ† ํฐ ์ž๋ฆฌ์—์„œ ๋ณด๊ธฐ ๊ธฐํ˜ธ(A, B, โ€ฆ) ํ† ํฐ์˜ ์ ์ˆ˜๋งŒ ๊บผ๋‚ด
calibration.json ์˜ ์˜จ๋„๋กœ ๋‚˜๋ˆ  softmax ํ•œ๋‹ค. ๊ธ€์„ ์ƒ์„ฑํ•˜์ง€ ์•Š์œผ๋ฏ€๋กœ ๋น ๋ฅด๋‹ค.
<|im_start|>user
{์ง€์นจ}
{ํ•ญ๋ชฉ ์ด๋ฆ„}: {๊ฐ’} โ† state ์˜ ํ‚ค๋งˆ๋‹ค ํ•œ ์ค„ (๊ฐ’์ด ๋ฌธ์ž์—ด์ด ์•„๋‹ˆ๋ฉด JSON). ๋ฌธ์ž์—ด๋งŒ ์ฃผ๋ฉด "text: {๋ฌธ์ž์—ด}"
๋ณด๊ธฐ:
A. {๋ณด๊ธฐ 1}
B. {๋ณด๊ธฐ 2}
์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”.<|im_end|>
<|im_start|>assistant
"""
import json
import warnings
from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
REPO = "coderecode95/KJH-Judge-4B-ko"
HINT = "์ •๋‹ต ๋ณด๊ธฐ์˜ ๊ธฐํ˜ธ ํ•˜๋‚˜๋งŒ ๋‹ตํ•˜์„ธ์š”."
DEFAULT_INSTRUCTION = "์ฃผ์–ด์ง„ ๋‚ด์šฉ์„ ์ฝ๊ณ  ๊ฐ€์žฅ ์•Œ๋งž์€ ๋ณด๊ธฐ๋ฅผ ๊ณ ๋ฅธ๋‹ค."
# ๊ณผ์ œ๋ณ„ ์ง€์นจ โ€” ํ•™์ŠตยทKorean JEV Benchmark ์™€ ๊ฐ™์€ ๋ฌธ๊ตฌ (๋ฐ”๊พธ๋ฉด ์ •ํ™•๋„๊ฐ€ ๋–จ์–ด์งˆ ์ˆ˜ ์žˆ๋‹ค)
TASKS = {
"sentiment": "๋ฌธ์žฅ์˜ ์ „๋ฐ˜์ ์ธ ๊ฐ์ • ๊ทน์„ฑ์„ ํŒ๋‹จํ•œ๋‹ค. ๋ถ€์ • ํ‘œํ˜„๊ณผ ๋ฐ˜์–ด๋ฅผ ๊ณ ๋ คํ•œ๋‹ค.",
"nli": ("์ „์ œ๋ฅผ ์ฐธ์ด๋ผ๊ณ  ํ•  ๋•Œ ๊ฐ€์„ค์„ ๋ถ„๋ฅ˜ํ•œ๋‹ค. entailment: ์ „์ œ๋กœ๋ถ€ํ„ฐ ๊ฐ€์„ค์ด ๋”ฐ๋ผ์˜จ๋‹ค. contradiction: ์ „์ œ์™€ ๊ฐ€์„ค์ด ์–‘๋ฆฝํ•  ์ˆ˜ ์—†๋‹ค. neutral: ๋‘˜ ๋‹ค ์•„๋‹ˆ๋‹ค. "
"์ •๋ณด๊ฐ€ ์—†๋‹ค๋Š” ์ด์œ ๋งŒ์œผ๋กœ contradiction์„ ๊ณ ๋ฅด์ง€ ์•Š๋Š”๋‹ค. neutral์€ ์ •์ƒ ์ •๋‹ต์ด๋ฉฐ insufficient_context๊ฐ€ ์•„๋‹ˆ๋‹ค. ํ†ต์ƒ์ ์ธ ๋ฌธ์žฅ ์˜๋ฏธ๋กœ ํŒ๋‹จํ•˜๋ฉฐ ํฌ๋ฐ•ํ•œ ๊ฐ€์ •์„ ๋งŒ๋“ค์–ด ๋ชจํ˜ธํ•˜๋‹ค๊ณ  ํ•˜์ง€ ์•Š๋Š”๋‹ค. "
"์ „์ œ์˜ ์—ด๊ฑฐ ๋Œ€์ƒ๊ณผ ๊ฐ€์„ค์˜ ๋ฒ”์œ„๋ฅผ ๋น ์ง์—†์ด ๋Œ€์กฐํ•œ๋‹ค. ๊ฐ€์„ค์˜ ๋งŒ/๋ฐ–์—/์˜ค์ง/๋ชจ๋‘ ๊ฐ™์€ ํ‘œํ˜„์ด ์ „์ œ์— ๋ช…์‹œ๋œ ๋‹ค๋ฅธ ๋Œ€์ƒ์ด๋‚˜ ์‚ฌ๋ก€๋ฅผ ๋ฐฐ์ œํ•˜๋Š”์ง€ ํ™•์ธํ•œ๋‹ค. "
"์ „์ œ์— ๋ฐฐํƒ€ ํ‘œํ˜„์ด ์—†๋‹ค๋Š” ์‚ฌ์‹ค๋งŒ์œผ๋กœ ์ค‘๋ฆฝ์ด๋ผ๊ณ  ํ•˜์ง€ ์•Š๋Š”๋‹ค. ํ†ต์ƒ์ ์ธ ๋ฌธ๋งฅ์ด ๋ช…ํ™•ํ•˜๋ฉด ๋‹ค๋ฅธ ์›์ธยท์ƒํ™ฉ์„ ์ž„์˜๋กœ ์ถ”๊ฐ€ํ•˜์ง€ ์•Š๋Š”๋‹ค."),
"topic": "ํ•œ๊ตญ์–ด ๋‰ด์Šค ์ œ๋ชฉ์˜ ์ค‘์‹ฌ ์ฃผ์ œ๋ฅผ ๊ณ ๋ฅธ๋‹ค. ์—ฌ๋Ÿฌ ๋ถ„์•ผ๊ฐ€ ์–ธ๊ธ‰๋˜์–ด๋„ ๊ธฐ์‚ฌ์˜ ์ฃผ๋œ ์‚ฌ๊ฑด์„ ์šฐ์„ ํ•œ๋‹ค.",
"hate": ("APEACH ์› ๋…ผ๋ฌธ์˜ ์ƒ์„ฑ ๊ธฐ์ค€์„ ๋”ฐ๋ฅธ๋‹ค. ์„ฑ๋ณ„ยท์„ฑ์  ์ง€ํ–ฅ/์ •์ฒด์„ฑยท๋‚˜์ด/์‹ ๋ถ„/๊ฒฝ๋ ฅยท๊ตญ์ /์ธ์ข…ยท์ถœ์‹ /๊ฑฐ์ฃผ์ง€ยท์™ธ๋ชจยท์งˆ๋ณ‘/์žฅ์• ยท์‹์Šต๊ด€ยทํ•™๋ ฅ/์ „๊ณต ๊ด€๋ จ ์ฐจ๋ณ„ยท๋น„ํ•˜ยท๊ณต๊ฒฉ์„ ํฌํ•จํ•œ๋‹ค. "
"๋ณดํ˜ธ์ง‘๋‹จ ์ •์ฒด์„ฑ์„ ๋ช…์‹œํ•ด์•ผ ํ•œ๋‹ค๋Š” ์ œํ•œ์€ ์—†๋‹ค. ๋ถˆํ•„์š”ํ•˜๊ฑฐ๋‚˜ ๊ณต๊ฒฉ์ ์ธ ์™ธ๋ชจ ํ‰๊ฐ€, ์‹์Šต๊ด€ ๊ฐ•์š”/ํ„ํ•˜๋„ ํ•ด๋‹นํ•œ๋‹ค. ๋น„๊ณต๊ฒฉ์ ์ธ ์ฃผ์ œ ์–ธ๊ธ‰์ด๋‚˜ ์ฐจ๋ณ„์— ๋Œ€ํ•œ ๋ฐ˜๋Œ€ ๋ฐœ์–ธ์€ ๋น„ํ˜์˜ค๋‹ค."),
"offensive": "BEEP ์› ์ง€์นจ์— ๋”ฐ๋ฅธ ๊ณต๊ฒฉ์„ฑ ์ˆ˜์ค€. hate ๋Œ€์ƒ์€ ๋ณดํ˜ธ ์ง‘๋‹จ์— ํ•œ์ •๋˜์ง€ ์•Š๊ณ  ๊ธฐ์‚ฌ ์† ์ธ๋ฌผยท์ž‘์„ฑ์ž ๋“ฑ ๊ฐœ์ธ๋„ ํฌํ•จํ•œ๋‹ค. ๋น„์†์–ด ์กด์žฌ๋งŒ์œผ๋กœ hate๊ฐ€ ๋˜์ง€๋Š” ์•Š๋Š”๋‹ค. ๋‰ด์Šค ์ œ๋ชฉ์„ ๋ฌธ๋งฅ์œผ๋กœ ์‚ฌ์šฉํ•œ๋‹ค.",
"yes_no": "๋ฌธ๋‹จ์— ๊ทผ๊ฑฐํ•˜์—ฌ ์งˆ๋ฌธ์— ์˜ˆ/์•„๋‹ˆ์˜ค๋กœ ๋‹ตํ•œ๋‹ค.",
"same_meaning": "๋‘ ๋ฌธ๋งฅ์—์„œ ํ‘œ์‹œ๋œ ๋‹จ์–ด์˜ ์˜๋ฏธ๊ฐ€ ๊ฐ™์€์ง€ ํŒ๋‹จํ•œ๋‹ค.",
"similar": "๋‘ ๋ฌธ์žฅ์˜ ์˜๋ฏธ๊ฐ€ ๊ฐ™์€์ง€ ํŒ๋‹จํ•œ๋‹ค. ํ‘œํ˜„์ด ๋‹ฌ๋ผ๋„ ์ „๋‹ฌํ•˜๋Š” ๋‚ด์šฉ์ด ๊ฐ™์œผ๋ฉด ๊ฐ™์€ ์˜๋ฏธ๋‹ค.",
"reading": "์ง€๋ฌธ์— ๊ทผ๊ฑฐํ•˜์—ฌ ์งˆ๋ฌธ์˜ ๋‹ต์„ ๊ณ ๋ฅธ๋‹ค. ๋ถ€์ • ์งˆ๋ฌธ์— ์ฃผ์˜ํ•œ๋‹ค.",
"next_action": ("๋‹ค์Œ ํ•œ ๋‹จ๊ณ„์˜ ํ–‰๋™์„ ์„ ํƒํ•œ๋‹ค. CALL:<name>: ์ œ๊ณต๋œ ๋„๊ตฌ๋กœ ์š”์ฒญ์„ ์ฒ˜๋ฆฌํ•  ์ˆ˜ ์žˆ๊ณ  ํ•„์š”ํ•œ ํ•„์ˆ˜ ์ธ์ž๊ฐ€ ๋Œ€ํ™”์— ๋ชจ๋‘ ์žˆ๋‹ค. "
"ASK: ๊ด€๋ จ ๋„๊ตฌ๋Š” ์žˆ์œผ๋‚˜ ํ•„์ˆ˜ ์ธ์ž๊ฐ€ ๋ถ€์กฑํ•ด ์งˆ๋ฌธ์ด ํ•„์š”ํ•˜๋‹ค. RESPOND: ๋„๊ตฌ๊ฐ€ ๋ถˆํ•„์š”ํ•œ ์„ค๋ช…/์™„๋ฃŒ ์•ˆ๋‚ด ๋˜๋Š” ์ œ๊ณต ๋„๊ตฌ๋กœ ์ง€์›ํ•˜์ง€ ์•Š๋Š” ์š”์ฒญ์— ๋Œ€ํ•œ ์•ˆ๋‚ด. "
"์‹ค์ œ ๋„๊ตฌ ์‹คํ–‰ยท์ธ์ž ์ž‘์„ฑ์€ ํ•˜์ง€ ์•Š๋Š”๋‹ค. ๊ณผ๊ฑฐ ๋„๊ตฌ ๊ฒฐ๊ณผ๋ฅผ ํ™œ์šฉํ•˜๊ณ  ํ˜„์žฌ ์ •๋‹ต์ด๋‚˜ ๋ฏธ๋ž˜ ํ„ด์„ ์ถ”์ธกํ•˜์ง€ ์•Š๋Š”๋‹ค. "
"๋„๊ตฌ schema๊ฐ€ ํ•„์ˆ˜ ๊ฐ’์˜ ๊ธฐ๋ณธ๊ฐ’ยท๋ฌธ๋งฅ ์ถ”๋ก ยท๋ช…์นญ ์ƒ์„ฑ์„ ๋ช…์‹œ์ ์œผ๋กœ ํ—ˆ์šฉํ•˜๋ฉด ๊ทธ ์ •์˜๋ฅผ ๋”ฐ๋ฅธ๋‹ค. ๊ทผ๊ฑฐ ์—†๋Š” ๊ฐ’์„ ๋งŒ๋“œ๋Š” ๊ฒƒ๊ณผ ๋„๊ตฌ ์ •์˜๊ฐ€ ํ—ˆ์šฉํ•˜๋Š” ๊ฐ’์„ ๋„์ถœํ•˜๋Š” ๊ฒƒ์„ ๊ตฌ๋ณ„ํ•œ๋‹ค. "
"ํ˜„์žฌ๊นŒ์ง€์˜ ๋„๊ตฌ ๊ฒฐ๊ณผ๋กœ ์š”์ฒญ์ด ํ•ด๊ฒฐ๋์œผ๋ฉด ๊ฒฐ๊ณผ ์•ˆ๋‚ด๊ฐ€ ๋‹ค์Œ ํ–‰๋™์ด๋ฉฐ ๊ฐ™์€ ๋„๊ตฌ๋ฅผ ๋ฐ˜๋ณต ํ˜ธ์ถœํ•˜์ง€ ์•Š๋Š”๋‹ค."),
"policy_action": ("๋ช…์‹œ๋œ ์ •์ฑ…๊ณผ ์˜ˆ์™ธ์˜ ์กฐ๊ฑด์— ๋”ฐ๋ผ ๋‹ค์Œ ํ–‰๋™์„ ๊ณ ๋ฅธ๋‹ค. ์˜ˆ์™ธ๋Š” ์“ฐ์ธ ์ ์šฉ ์กฐ๊ฑด์„ ๋งŒ์กฑํ•  ๋•Œ๋งŒ ๊ธฐ๋ณธ ๊ทœ์น™์„ ๋ฎ์–ด์“ด๋‹ค. ๋ช…์‹œ๋˜์ง€ ์•Š์€ ์‹ค์ œ ๊ธˆ์œต ์‹ค๋ฌด ์š”๊ฑด์„ ์ถ”๊ฐ€ํ•˜์ง€ ์•Š๋Š”๋‹ค. "
"ASK_CLARIFICATION๊ณผ ASK_CONFIRMATION์„ ๊ตฌ๋ณ„ํ•œ๋‹ค. ๋จผ์ € ๊ธฐ๋ณธ ๊ทœ์น™์˜ ์ ์šฉ ์กฐ๊ฑด๊ณผ ์˜ˆ์™ธ์˜ ์ ์šฉ ์กฐ๊ฑด์„ ๊ฐ๊ฐ ํ™•์ธํ•œ๋‹ค. "
"์กฐ๊ฑด์„ ๋งŒ์กฑํ•˜๋Š” ๋ช…์‹œ์  ํ–‰๋™ ์ง€์‹œ๋ฅผ ์šฐ์„ ํ•˜๋ฉฐ, ๊ณ ๊ฐ ๋ฌธ์žฅ์ด ์งˆ๋ฌธ ํ˜•ํƒœ๋ผ๋Š” ์ด์œ ๋กœ ์ด๋ฅผ ๋ฌด์‹œํ•˜์ง€ ์•Š๋Š”๋‹ค. ์กฐ๊ฑด์ด ๋งž์ง€ ์•Š๋Š” ์˜ˆ์™ธ๋Š” ์ ์šฉํ•˜์ง€ ์•Š๋Š”๋‹ค. "
"์ •์ฑ…์— ๋ณ„๋„ ํ–‰๋™ ์ง€์‹œ๊ฐ€ ์—†์œผ๋ฉด ์ฃผ์–ด์ง„ ํ–‰๋™ ์ •์˜์™€ ์‹ค์ œ ์š”์ฒญ์— ๋งž์ถฐ ํŒ๋‹จํ•œ๋‹ค. ์ •์ฑ…ยท์ผ๋ฐ˜ ์‚ฌ์‹ค ์„ค๋ช…๊ณผ ์™ธ๋ถ€ ๊ณ„์ขŒ ์ •๋ณด ์กฐํšŒ๋ฅผ ๊ตฌ๋ณ„ํ•œ๋‹ค. "
"๋ช…์‹œ๋œ ํ–‰๋™์„ ๊ณ ๋ฅด๋Š” ์ถ”์ƒ ๊ณผ์ œ์ด๋ฏ€๋กœ ์‹ค์ œ ๋„๊ตฌ ์Šคํ‚ค๋งˆ๋‚˜ ๊ณ„์ขŒ ์ธ์ฆ ์ ˆ์ฐจ๋ฅผ ์ถ”๊ฐ€๋กœ ์š”๊ตฌํ•˜์ง€ ์•Š๋Š”๋‹ค. ๋‹จ, ์›๋ณธ์— ์—†๋Š” ๊ธฐ๋ณธ ํ–‰๋™์ด๋‚˜ ์ •์ฑ…์„ ์ƒˆ๋กœ ๋งŒ๋“ค์ง€ ์•Š๋Š”๋‹ค."),
}
# ๊ณผ์ œ๋ณ„ ๋ณด๊ธฐ โ€” {๋Œ๋ ค์ค„ ID: ๋ชจ๋ธ์—๊ฒŒ ๋ณด์—ฌ ์ค„ ๋ณด๊ธฐ ๋ฌธ๊ตฌ}
CRITERIA = {
"sentiment": {"๊ธ์ •": "๊ธ์ •์  ๊ฐ์ •", "๋ถ€์ •": "๋ถ€์ •์  ๊ฐ์ •"},
"nli": {"entailment": "entailment", "contradiction": "contradiction", "neutral": "neutral"},
"topic": {k: k for k in ["์‚ฌํšŒ", "IT๊ณผํ•™", "์„ธ๊ณ„", "์ƒํ™œ๋ฌธํ™”", "์ •์น˜", "์Šคํฌ์ธ ", "๊ฒฝ์ œ"]},
"hate": {"๋น„ํ˜์˜ค": "๋น„ํ˜์˜ค: ๋น„ํ•˜ยท๊ณต๊ฒฉ์„ฑ์ด ์—†๋Š” ํ‘œํ˜„", "ํ˜์˜ค": "ํ˜์˜คยท๋น„ํ•˜ยท๊ณต๊ฒฉ์  ํ‘œํ˜„"},
"offensive": {"์—†์Œ": "ํ˜์˜คยท๋ชจ์š•ยท๋ถˆ์พŒ ํ‘œํ˜„ ์—†์Œ",
"๊ณต๊ฒฉ์ ": "๋ฌด๋ก€ยท๊ณต๊ฒฉยท๋น„๊ผผ ๋“ฑ ๋ถˆ์พŒํ•˜์ง€๋งŒ ๊ฐ•ํ•œ ํ˜์˜ค/๋ชจ์š• ์ˆ˜์ค€์€ ์•„๋‹˜",
"ํ˜์˜ค": "๊ฐœ์ธ/์ง‘๋‹จ์— ๋Œ€ํ•œ ๊ฐ•ํ•œ ํ˜์˜ค ๋˜๋Š” ์‚ฌํšŒ์  ์ง€์œ„๋ฅผ ์‹ฌํ•˜๊ฒŒ ํ›ผ์†ํ•˜๋Š” ๋ชจ์š•; ์„ฑํฌ๋กฑ ํฌํ•จ"},
"yes_no": {"์˜ˆ": "์˜ˆ", "์•„๋‹ˆ์˜ค": "์•„๋‹ˆ์˜ค"},
"same_meaning": {"๊ฐ™์Œ": "๊ฐ™์€ ์˜๋ฏธ", "๋‹ค๋ฆ„": "์„œ๋กœ ๋‹ค๋ฅธ ์˜๋ฏธ"},
"similar": {"๊ฐ™์Œ": "๊ฐ™์€ ์˜๋ฏธ", "๋‹ค๋ฆ„": "๋‹ค๋ฅธ ์˜๋ฏธ"},
"policy_action": {"ASK_CLARIFICATION": "ASK_CLARIFICATION: ๋ถ€์กฑํ•œ ์ •๋ณด๋ฅผ ๊ณ ๊ฐ์—๊ฒŒ ๋˜๋ฌป๋Š”๋‹ค",
"ASK_CONFIRMATION": "ASK_CONFIRMATION: ์‹คํ–‰ ์ „์— ๋ช…์‹œ์  ํ™•์ธยท๋™์˜๋ฅผ ๋ฐ›๋Š”๋‹ค",
"ANSWER": "ANSWER: ๋„๊ตฌ ํ˜ธ์ถœ์ด๋‚˜ ์‹คํ–‰ ์—†์ด ์ •์ฑ…ยท์‚ฌ์‹ค ์ •๋ณด๋กœ ๋‹ตํ•œ๋‹ค",
"CALL_TOOL": "CALL_TOOL: ๋„๊ตฌ ํ•˜๋‚˜๋กœ ์‹คํ–‰ํ•˜๊ฑฐ๋‚˜ ์กฐํšŒํ•œ๋‹ค",
"CALL_MULTIPLE_TOOLS": "CALL_MULTIPLE_TOOLS: ๋‘˜ ์ด์ƒ์˜ ๋„๊ตฌ๋ฅผ ์ฐจ๋ก€๋กœ ํ˜ธ์ถœํ•œ๋‹ค",
"ESCALATE": "ESCALATE: ์ž๋™ ์ฒ˜๋ฆฌ๋ฅผ ๋ฉˆ์ถ”๊ณ  ์‚ฌ๋žŒ ์ƒ๋‹ด์‚ฌ์—๊ฒŒ ๋„˜๊ธด๋‹ค",
"REFUSE": "REFUSE: ์š”์ฒญ์„ ์ดํ–‰ํ•˜์ง€ ์•Š๊ณ  ์‚ฌ์œ ๋ฅผ ์•ˆ๋‚ดํ•œ๋‹ค"},
}
ASK = "ASK: ๋ถ€์กฑํ•œ ํ•„์ˆ˜ ์ •๋ณด ์งˆ๋ฌธ"
RESPOND = "RESPOND: ์ง์ ‘ ์„ค๋ช…ยท์™„๋ฃŒยท๋ฏธ์ง€์› ์•ˆ๋‚ด"
def render_prompt(instruction, item, choices):
if isinstance(item, str):
item = {"text": item}
lines = [f"{k}: {v if isinstance(v, str) else json.dumps(v, ensure_ascii=False)}" for k, v in item.items()]
opts = "\n".join(f"{chr(65 + i)}. {c}" for i, c in enumerate(choices))
return "\n\n".join([(instruction or DEFAULT_INSTRUCTION).strip(), "\n".join(lines), "๋ณด๊ธฐ:\n" + opts, HINT])
def chat_prefix(prompt):
return f"<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n"
def check_choices(choices):
choices = [c["text"] if isinstance(c, dict) else str(c) for c in choices]
if not 2 <= len(choices) <= 26:
raise ValueError(f"๋ณด๊ธฐ๋Š” 2~26๊ฐœ์—ฌ์•ผ ํ•ฉ๋‹ˆ๋‹ค (์ง€๊ธˆ {len(choices)}๊ฐœ)")
if len(set(choices)) != len(choices):
raise ValueError("๋ณด๊ธฐ ๋ฌธ๊ตฌ๊ฐ€ ๊ฒน์นฉ๋‹ˆ๋‹ค. ๋ณด๊ธฐ๋งˆ๋‹ค ๋‹ค๋ฅธ ๋ฌธ๊ตฌ๋ฅผ ์จ ์ฃผ์„ธ์š”")
return choices
def load_temperature(model):
p = Path(model) / "calibration.json"
if not p.is_file():
try:
from huggingface_hub import hf_hub_download
p = Path(hf_hub_download(model, "calibration.json"))
except Exception:
warnings.warn(f"{model} ์—์„œ calibration.json ์„ ์ฐพ์ง€ ๋ชปํ•ด ๋ณด์ • ์—†์ด(์˜จ๋„ 1.0) ํ™•๋ฅ ์„ ๊ณ„์‚ฐํ•ฉ๋‹ˆ๋‹ค.")
return 1.0
return float(json.loads(p.read_text(encoding="utf-8"))["temperature"])
def letter_ids(tokenizer):
return [tokenizer.encode(chr(65 + i), add_special_tokens=False)[0] for i in range(26)]
def as_result(ids, probs):
"""๋ณด๊ธฐ ์ˆœ์„œ์˜ ํ™•๋ฅ  โ†’ {'choice', 'confidence', 'probabilities'}"""
b = max(range(len(probs)), key=probs.__getitem__)
return {"choice": ids[b], "confidence": round(probs[b], 4), "probabilities": {i: round(p, 4) for i, p in zip(ids, probs)}}
def _criteria(criteria):
"""{ID: ์„ค๋ช…} ๋˜๋Š” [์„ค๋ช…, โ€ฆ] โ†’ (ID ๋ชฉ๋ก, ์„ค๋ช… ๋ชฉ๋ก). ๋ฆฌ์ŠคํŠธ๋ฉด ์„ค๋ช…์ด ๊ณง ID."""
if isinstance(criteria, dict):
return [str(k) for k in criteria], check_choices(list(criteria.values()))
texts = check_choices(criteria)
return texts, texts
def _rows(**fields):
"""๊ฐ’์ด ๋ชจ๋‘ ๋ฌธ์ž์—ด์ด๋ฉด ํ•ญ๋ชฉ ํ•˜๋‚˜, ๋ชจ๋‘ ๊ฐ™์€ ๊ธธ์ด์˜ ๋ฆฌ์ŠคํŠธ๋ฉด ํ•ญ๋ชฉ ์—ฌ๋Ÿฌ ๊ฐœ."""
vals = list(fields.values())
if all(isinstance(v, list) for v in vals):
if len({len(v) for v in vals}) != 1:
raise ValueError("๋ฆฌ์ŠคํŠธ๋กœ ์—ฌ๋Ÿฌ ๊ฐœ๋ฅผ ๋„ฃ์„ ๋•Œ๋Š” ๊ธธ์ด๊ฐ€ ๊ฐ™์•„์•ผ ํ•ฉ๋‹ˆ๋‹ค")
return [dict(zip(fields, row)) for row in zip(*vals)], True
return dict(fields), False
def _tool_name(t):
return (t.get("function") or t)["name"] if isinstance(t, dict) else str(t)
class Tasks:
"""JEV ํ˜•์‹ decide() ์™€ ์ž์ฃผ ์“ฐ๋Š” ํŒ์ •. self(์ƒํ™ฉ, choices=โ€ฆ, instruction=โ€ฆ) ๋งŒ ์žˆ์œผ๋ฉด ๋œ๋‹ค (Judge ์™€ pipeline ์ด ํ•จ๊ป˜ ์“ด๋‹ค)."""
def _choose(self, state, criteria, instructions, many=False):
ids, texts = _criteria(criteria)
res = self(state, choices=texts, instruction=instructions)
def rename(r):
return as_result(ids, [r["probabilities"][t] for t in texts])
return [rename(r) for r in res] if many else rename(res)
def decide(self, state, questions):
"""JEV ํ˜•์‹. state: ์ƒํ™ฉ(dict ๋˜๋Š” ๋ฌธ์žฅ). questions: {์งˆ๋ฌธ ์ด๋ฆ„: {"instructions": ์ง€์นจ, "criteria": {ID: ์„ค๋ช…}}}.
โ†’ {์งˆ๋ฌธ ์ด๋ฆ„: {'choice': ID, 'confidence': ํ™•๋ฅ , 'probabilities': {ID: ํ™•๋ฅ }}}"""
out = {}
for name, q in questions.items():
if q.get("type", "choice") != "choice":
raise ValueError(f"'{name}': ์ง€์›ํ•˜๋Š” type ์€ 'choice' ๋ฟ์ž…๋‹ˆ๋‹ค")
out[name] = self._choose(state, q["criteria"], q.get("instructions"))
return out
def sentiment(self, text):
"""๊ฐ์ • โ†’ ๊ธ์ • / ๋ถ€์ •. ๋ฌธ์žฅ ํ•˜๋‚˜ ๋˜๋Š” ๋ฆฌ์ŠคํŠธ."""
return self._choose(text, CRITERIA["sentiment"], TASKS["sentiment"], isinstance(text, list))
def topic(self, headline):
"""๋‰ด์Šค ์ œ๋ชฉ ์ฃผ์ œ โ†’ ์‚ฌํšŒ / IT๊ณผํ•™ / ์„ธ๊ณ„ / ์ƒํ™œ๋ฌธํ™” / ์ •์น˜ / ์Šคํฌ์ธ  / ๊ฒฝ์ œ."""
rows, many = _rows(headline=headline)
return self._choose(rows, CRITERIA["topic"], TASKS["topic"], many)
def hate(self, text):
"""ํ˜์˜คยท๋น„ํ•˜ ํ‘œํ˜„ ์—ฌ๋ถ€ โ†’ ๋น„ํ˜์˜ค / ํ˜์˜ค."""
return self._choose(text, CRITERIA["hate"], TASKS["hate"], isinstance(text, list))
def offensive(self, comment, news_title=None):
"""๋Œ“๊ธ€ ๊ณต๊ฒฉ์„ฑ 3๋‹จ๊ณ„ โ†’ ์—†์Œ / ๊ณต๊ฒฉ์  / ํ˜์˜ค. ๋‰ด์Šค ์ œ๋ชฉ์ด ์žˆ์œผ๋ฉด ๋ฌธ๋งฅ์œผ๋กœ ์“ด๋‹ค."""
rows, many = _rows(news_title=news_title, comments=comment) if news_title is not None else _rows(comments=comment)
return self._choose(rows, CRITERIA["offensive"], TASKS["offensive"], many)
def nli(self, premise, hypothesis):
"""๋ฌธ์žฅ ๊ด€๊ณ„ โ†’ entailment / contradiction / neutral."""
rows, many = _rows(premise=premise, hypothesis=hypothesis)
return self._choose(rows, CRITERIA["nli"], TASKS["nli"], many)
def similar(self, sentence1, sentence2):
"""๋‘ ๋ฌธ์žฅ์ด ๊ฐ™์€ ๋œป์ธ์ง€ โ†’ ๊ฐ™์Œ / ๋‹ค๋ฆ„."""
rows, many = _rows(sentence1=sentence1, sentence2=sentence2)
return self._choose(rows, CRITERIA["similar"], TASKS["similar"], many)
def yes_no(self, question, passage):
"""์ง€๋ฌธ์— ๊ทผ๊ฑฐํ•œ ์˜ˆ/์•„๋‹ˆ์˜ค โ†’ ์˜ˆ / ์•„๋‹ˆ์˜ค."""
rows, many = _rows(paragraph=passage, question=question)
return self._choose(rows, CRITERIA["yes_no"], TASKS["yes_no"], many)
def same_meaning(self, word, context_1, context_2):
"""๋‘ ๋ฌธ๋งฅ์—์„œ ๋‹จ์–ด ๋œป์ด ๊ฐ™์€์ง€ โ†’ ๊ฐ™์Œ / ๋‹ค๋ฆ„. ๋ฌธ๋งฅ ์•ˆ์˜ ๋‹จ์–ด๋Š” [๋‹จ์–ด]๋กœ ํ‘œ์‹œํ•˜๋ฉด ์ข‹๋‹ค."""
rows, many = _rows(word=word, context_1=context_1, context_2=context_2)
return self._choose(rows, CRITERIA["same_meaning"], TASKS["same_meaning"], many)
def reading(self, passage, question, choices):
"""์ง€๋ฌธ ๋…ํ•ด ๊ฐ๊ด€์‹ โ†’ ๋ณด๊ธฐ ์ค‘ ํ•˜๋‚˜ (๋ฆฌ์ŠคํŠธ๋ฉด ๋ณด๊ธฐ ๋ฌธ๊ตฌ, dict ๋ฉด ID)."""
return self._choose({"passage": passage, "question": question}, choices, TASKS["reading"])
def next_action(self, messages, tools):
"""์—์ด์ „ํŠธ์˜ ๋‹ค์Œ ํ–‰๋™ โ†’ CALL:<๋„๊ตฌ ์ด๋ฆ„> / ASK (ํ•„์ˆ˜ ์ •๋ณด ๋ถ€์กฑ) / RESPOND (์ง์ ‘ ๋‹ต๋ณ€).
messages: ์‚ฌ์šฉ์ž ๋ฌธ์žฅ(str) ๋˜๋Š” [{"role", "content"}, โ€ฆ]. tools: OpenAI ํ˜•์‹ ๋„๊ตฌ ๋ชฉ๋ก ๋˜๋Š” ์ด๋ฆ„ ๋ชฉ๋ก."""
if isinstance(messages, str):
messages = [{"role": "user", "content": messages}]
criteria = {f"CALL:{_tool_name(t)}": f"CALL:{_tool_name(t)}" for t in tools} | {"ASK": ASK, "RESPOND": RESPOND}
return self._choose({"messages": messages, "tools": tools}, criteria, TASKS["next_action"])
def policy_action(self, policy, request):
"""์ •์ฑ…์— ๋”ฐ๋ฅธ ํ–‰๋™ โ†’ ASK_CLARIFICATION / ASK_CONFIRMATION / ANSWER / CALL_TOOL / CALL_MULTIPLE_TOOLS / ESCALATE / REFUSE."""
text = f"์ •์ฑ…: {policy}\n\n๊ณ ๊ฐ ์š”์ฒญ: \"{request}\"\n์œ„ ์ •์ฑ…์— ๋”ฐ๋ผ ์˜ฌ๋ฐ”๋ฅด๊ฒŒ ์ฒ˜๋ฆฌํ•˜๋ผ."
return self._choose({"policy_and_request": text}, CRITERIA["policy_action"], TASKS["policy_action"])
class Judge(Tasks):
def __init__(self, model=REPO, device=None, dtype=None):
device = device or ("cuda" if torch.cuda.is_available() else "cpu")
dtype = dtype or (torch.bfloat16 if device != "cpu" else torch.float32)
self.tok = AutoTokenizer.from_pretrained(model)
self.model = AutoModelForCausalLM.from_pretrained(model, dtype=dtype).to(device).eval()
self.device = device
self.temperature = load_temperature(model)
self.letter_ids = letter_ids(self.tok)
def __call__(self, inputs, choices, instruction=None):
"""inputs: ๋ฌธ์žฅ(str) ยท ์ƒํ™ฉ(dict) ยท ๊ทธ ๋ฆฌ์ŠคํŠธ. choices: [์„ค๋ช…, โ€ฆ] ๋˜๋Š” {ID: ์„ค๋ช…}. ๋ฆฌ์ŠคํŠธ๋ฅผ ์ฃผ๋ฉด ๊ฒฐ๊ณผ๋„ ๋ฆฌ์ŠคํŠธ."""
many = isinstance(inputs, list)
ids, texts = _criteria(choices)
out = self.decide_batch([{"instruction": instruction, "item": x, "choices": texts} for x in (inputs if many else [inputs])])
res = [as_result(ids, [r["probs"][chr(65 + i)] for i in range(len(texts))]) for r in out]
return res if many else res[0]
@torch.no_grad()
def decide_batch(self, requests, batch_size=16, max_tokens=8000):
"""์ €์ˆ˜์ค€ ๋ฌถ์Œ ํŒ์ • (ํ‰๊ฐ€ ์Šคํฌ๋ฆฝํŠธ์šฉ).
requests: [{"instruction": str, "item": dict|str, "choices": [str, โ€ฆ]}, โ€ฆ] โ†’ ๊ฐ™์€ ์ˆœ์„œ์˜ ๊ฒฐ๊ณผ ๋ชฉ๋ก.
๊ฒฐ๊ณผ: {'choice': 'A', 'index': 0, 'text': ๋ณด๊ธฐ, 'confidence': ํ™•๋ฅ , 'probs': {'A': โ€ฆ, 'B': โ€ฆ}}
๊ธธ์ด๊ฐ€ ๋น„์Šทํ•œ ๊ฒƒ๋ผ๋ฆฌ ๋ฌถ์–ด ๊ณ„์‚ฐํ•œ๋‹ค (์˜ค๋ฅธ์ชฝ์„ ์ฑ„์šฐ๊ณ  ๊ฐ ๋ฌธ์žฅ์˜ ๋งˆ์ง€๋ง‰ ์ž๋ฆฌ๋งŒ ์ฝ์œผ๋ฏ€๋กœ ๊ฒฐ๊ณผ๋Š” ํ•˜๋‚˜์”ฉ ๊ณ„์‚ฐํ•œ ๊ฒƒ๊ณผ ๊ฐ™๋‹ค)."""
seqs = []
for r in requests:
ch = check_choices(r["choices"])
ids = self.tok.encode(chat_prefix(render_prompt(r.get("instruction"), r["item"], ch)), add_special_tokens=False)
seqs.append((ids, ch))
order = sorted(range(len(seqs)), key=lambda i: len(seqs[i][0]))
out = [None] * len(seqs)
W = self.model.get_output_embeddings().weight
i = 0
while i < len(order):
grp = [order[i]]
while i + len(grp) < len(order) and len(grp) < batch_size and (len(grp) + 1) * len(seqs[order[i + len(grp)]][0]) <= max_tokens:
grp.append(order[i + len(grp)])
i += len(grp)
n = max(len(seqs[k][0]) for k in grp)
x = torch.zeros(len(grp), n, dtype=torch.long)
for r, k in enumerate(grp):
x[r, :len(seqs[k][0])] = torch.tensor(seqs[k][0])
h = self.model.model(input_ids=x.to(self.device)).last_hidden_state # ์ตœ์ข… ์ •๊ทœํ™”๊นŒ์ง€ ๋งˆ์นœ ์€๋‹‰ ์ƒํƒœ
for r, k in enumerate(grp):
ids, ch = seqs[k]
rows = torch.tensor(self.letter_ids[:len(ch)], device=self.device)
p = torch.softmax((h[r, len(ids) - 1].float() @ W[rows].float().T) / self.temperature, -1).tolist()
b = max(range(len(p)), key=p.__getitem__)
out[k] = {"choice": chr(65 + b), "index": b, "text": ch[b], "confidence": round(p[b], 4),
"probs": {chr(65 + j): round(v, 4) for j, v in enumerate(p)}}
return out
if __name__ == "__main__":
import sys
j = Judge(sys.argv[1] if len(sys.argv) > 1 else ".")
print(j.sentiment("๋ฐฐ์†ก์€ ๋А๋ ธ์ง€๋งŒ ๋ฌผ๊ฑด์€ ์ •๋ง ๋งˆ์Œ์— ๋“ค์–ด์š”"))