quiz-llm-server / README.md
dev2nd's picture
Set Docker SDK and app_port
1df14c8 verified
|
Raw History Blame Contribute Delete
1.43 kB
metadata
title: Quiz LLM Server
emoji: ๐Ÿค–
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
suggested_hardware: a10g-small
pinned: false
short_description: OpenAI-compatible vLLM server for quiz generation benchmark

Quiz LLM Server (vLLM, OpenAI-compatible)

ํ€ด์ฆˆ ์ƒ์„ฑ ์‹œ์Šคํ…œ ์„ฑ๋Šฅ ์‹œํ—˜์šฉ ๋กœ์ปฌ LLM ์„œ๋ฒ„. vLLM์˜ OpenAI ํ˜ธํ™˜ ์—”๋“œํฌ์ธํŠธ๋ฅผ ๊ทธ๋Œ€๋กœ ๋…ธ์ถœํ•จ.

Endpoints

  • GET /v1/models
  • POST /v1/chat/completions
  • POST /v1/completions

ํ™˜๊ฒฝ๋ณ€์ˆ˜ (Space โ†’ Settings โ†’ Variables and secrets)

์ด๋ฆ„ ์˜ˆ์‹œ ๊ฐ’ ๋น„๊ณ 
MODEL_ID Qwen/Qwen2.5-7B-Instruct HF ๋ชจ๋ธ ID
MAX_MODEL_LEN 8192 VRAM์— ๋งž์ถฐ ์กฐ์ •
GPU_MEM_UTIL 0.90 0.0 ~ 1.0
HF_TOKEN hf_xxx (Secret์œผ๋กœ ๋“ฑ๋ก) gated ๋ชจ๋ธ ์‚ฌ์šฉ ์‹œ ํ•„์ˆ˜

์—ฐ๊ฒฐ ์˜ˆ์‹œ

from openai import OpenAI

client = OpenAI(
    base_url="https://dev2nd-quiz-llm-server.hf.space/v1",
    api_key="hf_xxx",
)

resp = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "ํ€ด์ฆˆ๋ฅผ ๋งŒ๋“ค์–ด์ค˜"}],
)
print(resp.choices[0].message.content)