quiz-llm-server / README.md
dev2nd's picture
Set Docker SDK and app_port
1df14c8 verified
|
Raw History Blame Contribute Delete
1.43 kB
---
title: Quiz LLM Server
emoji: ๐Ÿค–
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
suggested_hardware: a10g-small
pinned: false
short_description: OpenAI-compatible vLLM server for quiz generation benchmark
---
# Quiz LLM Server (vLLM, OpenAI-compatible)
ํ€ด์ฆˆ ์ƒ์„ฑ ์‹œ์Šคํ…œ ์„ฑ๋Šฅ ์‹œํ—˜์šฉ ๋กœ์ปฌ LLM ์„œ๋ฒ„.
vLLM์˜ OpenAI ํ˜ธํ™˜ ์—”๋“œํฌ์ธํŠธ๋ฅผ ๊ทธ๋Œ€๋กœ ๋…ธ์ถœํ•จ.
## Endpoints
- `GET /v1/models`
- `POST /v1/chat/completions`
- `POST /v1/completions`
## ํ™˜๊ฒฝ๋ณ€์ˆ˜ (Space โ†’ Settings โ†’ Variables and secrets)
| ์ด๋ฆ„ | ์˜ˆ์‹œ ๊ฐ’ | ๋น„๊ณ  |
|------------------|----------------------------------|----------------------------|
| `MODEL_ID` | `Qwen/Qwen2.5-7B-Instruct` | HF ๋ชจ๋ธ ID |
| `MAX_MODEL_LEN` | `8192` | VRAM์— ๋งž์ถฐ ์กฐ์ • |
| `GPU_MEM_UTIL` | `0.90` | 0.0 ~ 1.0 |
| `HF_TOKEN` | `hf_xxx` (Secret์œผ๋กœ ๋“ฑ๋ก) | gated ๋ชจ๋ธ ์‚ฌ์šฉ ์‹œ ํ•„์ˆ˜ |
## ์—ฐ๊ฒฐ ์˜ˆ์‹œ
```python
from openai import OpenAI
client = OpenAI(
base_url="https://dev2nd-quiz-llm-server.hf.space/v1",
api_key="hf_xxx",
)
resp = client.chat.completions.create(
model="Qwen/Qwen2.5-7B-Instruct",
messages=[{"role": "user", "content": "ํ€ด์ฆˆ๋ฅผ ๋งŒ๋“ค์–ด์ค˜"}],
)
print(resp.choices[0].message.content)
```