"""속도 비교: 같은 문항을 같은 PC에서 보내 걸린 시간을 잰다 (네트워크 왕복 포함). $env:OPENAI_API_KEY = "…" # 비교 모델 API 키 python eval/bench_speed.py --kjh http://서버:32500 --api http://서버:30102/v1 측정 1. 판정 1회 지연: 문항을 하나씩 보내 응답까지 걸린 시간 (KJH: /api/decide, 비교 모델: 생각 끄고 max_tokens=1) 2. 동시 처리량: 동시에 N개씩 보내 초당 판정 수 3. 비교 모델을 평소처럼 쓸 때: 생각을 켜고 답을 생성하게 했을 때 시간과 생성 토큰 수 (일부 문항) """ import argparse import concurrent.futures as cf import json import os import random import statistics import sys import time import urllib.request from pathlib import Path HERE = Path(__file__).resolve().parent sys.path.insert(0, str(HERE)) sys.path.insert(0, str(HERE.parent)) from bench_heldout import load # noqa: E402 from judge import render_prompt # noqa: E402 RETRIES = {"n": 0} def post(url, body, hdr, timeout=300): """연결이 잠깐 거부되면 다시 시도한다. 재시도에 걸린 시간은 그 요청 시간에 포함되므로 재시도 횟수를 함께 기록한다.""" req = urllib.request.Request(url, json.dumps(body, ensure_ascii=False).encode(), {"Content-Type": "application/json", **hdr}) for k in range(10): try: return json.loads(urllib.request.urlopen(req, timeout=timeout).read()) except urllib.error.HTTPError: raise except Exception: if k == 9: raise RETRIES["n"] += 1 time.sleep(1.5) def stat(ms): s = sorted(ms) return {"n": len(s), "median_ms": round(statistics.median(s)), "p90_ms": round(s[int(len(s) * 0.9) - 1]), "mean_ms": round(statistics.mean(s))} def main(): ap = argparse.ArgumentParser() ap.add_argument("--kjh", required=True, help="KJH 판정 서버 (studio/serve.py)") ap.add_argument("--api", required=True, help="비교 모델 OpenAI 호환 주소 (…/v1)") ap.add_argument("--api-model") ap.add_argument("--n", type=int, default=60) ap.add_argument("--conc", type=int, default=4) ap.add_argument("--think-n", type=int, default=10) ap.add_argument("--out", default=str(HERE / "speed_results.json")) a = ap.parse_args() kjh, api = a.kjh.rstrip("/"), a.api.rstrip("/") key = os.environ.get("OPENAI_API_KEY", "") hdr = {"Authorization": f"Bearer {key}"} if key else {} if not a.api_model: a.api_model = json.loads(urllib.request.urlopen(urllib.request.Request(api + "/models", headers=hdr), timeout=30).read())["data"][0]["id"] rng = random.Random("speed") data = load("nsmc", rng, a.n // 2) + load("kornli", rng, a.n - a.n // 2) reqs = [r for r, _ in data] prompts = [render_prompt(r["instruction"], r["item"], r["choices"]) for r in reqs] def kjh_one(i): t = time.perf_counter() r = post(kjh + "/api/decide", {"instruction": reqs[i]["instruction"], "item": reqs[i]["item"], "choices": [{"text": c} for c in reqs[i]["choices"]]}, {}) return (time.perf_counter() - t) * 1000, r.get("ms") def api_one(i, think=False): body = {"model": a.api_model, "messages": [{"role": "user", "content": prompts[i]}], "temperature": 0, "chat_template_kwargs": {"enable_thinking": think}} body.update({"max_tokens": 4096} if think else {"max_tokens": 1, "logprobs": True, "top_logprobs": 20}) t = time.perf_counter() r = post(api + "/chat/completions", body, hdr, timeout=600) return (time.perf_counter() - t) * 1000, (r.get("usage") or {}).get("completion_tokens") for f in (kjh_one, api_one): # 예열 f(0) out = {"made": time.strftime("%Y-%m-%d %H:%M"), "api_model": a.api_model, "n": len(reqs), "note": "같은 PC에서 측정, 네트워크 왕복 포함"} k = [kjh_one(i) for i in range(len(reqs))] out["kjh_single"] = {**stat([x[0] for x in k]), "server_compute_median_ms": round(statistics.median([x[1] for x in k if x[1] is not None]))} print("KJH 1회", out["kjh_single"], flush=True) b = [api_one(i)[0] for i in range(len(reqs))] out["api_single"] = stat(b) print("비교 1회", out["api_single"], flush=True) for name, fn in (("kjh_throughput", kjh_one), ("api_throughput", api_one)): t = time.perf_counter() with cf.ThreadPoolExecutor(a.conc) as ex: list(ex.map(fn, range(len(reqs)))) el = time.perf_counter() - t out[name] = {"concurrency": a.conc, "decisions": len(reqs), "seconds": round(el, 2), "per_sec": round(len(reqs) / el, 1)} print(name, out[name], flush=True) th = [api_one(i, True) for i in range(min(a.think_n, len(reqs)))] out["api_thinking"] = {**stat([x[0] for x in th]), "completion_tokens_median": statistics.median([x[1] or 0 for x in th])} print("비교 생각 켬", out["api_thinking"], flush=True) out["connection_retries"] = RETRIES["n"] Path(a.out).write_text(json.dumps(out, ensure_ascii=False, indent=1), encoding="utf-8") print("저장:", a.out) if __name__ == "__main__": main()