mhd-radin commited on
Commit
cab2dad
·
verified ·
1 Parent(s): 02e5079

Upload 6 files

Browse files
Files changed (6) hide show
  1. .env.example +10 -0
  2. .gitignore +6 -0
  3. README.md +168 -0
  4. app.py +136 -0
  5. packages.txt +4 -0
  6. requirements.txt +3 -0
.env.example ADDED
@@ -0,0 +1,10 @@
 
 
 
 
 
 
 
 
 
 
 
1
+ # Required. Put the real value in Hugging Face Space Secrets, not in Git.
2
+ LLAMA_API_KEY=replace-with-a-long-random-secret
3
+
4
+ # Optional performance/model overrides:
5
+ # LLAMA_N_THREADS=2
6
+ # LLAMA_N_THREADS_BATCH=2
7
+ # LLAMA_N_CTX=2048
8
+ # LLAMA_N_BATCH=512
9
+ # LLAMA_N_UBATCH=256
10
+ # MODEL_ALIAS=qwen2.5-coder-0.5b-instruct
.gitignore ADDED
@@ -0,0 +1,6 @@
 
 
 
 
 
 
 
1
+ .env
2
+ .venv/
3
+ __pycache__/
4
+ *.py[cod]
5
+ .cache/
6
+ models/
README.md ADDED
@@ -0,0 +1,168 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ title: Qwen2.5 Coder 0.5B API
3
+ emoji: ⚡
4
+ colorFrom: blue
5
+ colorTo: purple
6
+ sdk: gradio
7
+ python_version: "3.11"
8
+ app_file: app.py
9
+ ---
10
+
11
+ # Qwen2.5-Coder-0.5B-Instruct REST API
12
+
13
+ A Hugging Face **Gradio SDK Space** that serves the official Qwen2.5-Coder-0.5B-Instruct `Q4_K_M` GGUF model through a llama.cpp-backed, OpenAI-compatible REST API.
14
+
15
+ - Model: `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF`
16
+ - Quantization: `Q4_K_M` (about 491 MB on the Hub)
17
+ - Runtime: `llama-cpp-python` (Python bindings to llama.cpp)
18
+ - Default hardware target: Hugging Face CPU Basic (2 vCPU / 16 GB RAM)
19
+ - Authentication: one required bearer API key from the Space Secret `LLAMA_API_KEY`
20
+ - Streaming: supported by `/v1/chat/completions`
21
+ - Docker: **not required**
22
+
23
+ ## Deploy to Hugging Face Spaces
24
+
25
+ 1. Create a new Space and select **Gradio** as the SDK.
26
+ 2. Upload or push all files from this project to the Space repository.
27
+ 3. Open **Settings → Variables and secrets**.
28
+ 4. Add a **Secret** named `LLAMA_API_KEY` and set a long, random value. Do not put the key in this repository or in a normal variable.
29
+ 5. Select **CPU Basic** hardware if it is not already selected, then let the Space build and start.
30
+ 6. Once it is running, the API base URL is:
31
+
32
+ `https://YOUR-SPACE-NAME.hf.space/v1`
33
+
34
+ The model file is downloaded automatically from the official Qwen repository on first startup. It is not included in this ZIP.
35
+
36
+ > **Hugging Face plan note (checked November 19, 2026):** Hugging Face documents CPU Basic as 2 vCPU / 16 GB RAM with no hourly compute charge, but creating a compute-backed Gradio or Docker Space on a personal account requires PRO. Free personal accounts can still run up to two Gradio Spaces on ZeroGPU; this project is a CPU-based llama.cpp service, not a ZeroGPU app. So the runtime hardware can be CPU Basic at no hourly cost, but a free personal account alone may not be able to create/run this kind of Space.
37
+
38
+ ## API authentication
39
+
40
+ Every `/v1` inference/API request must include the single Space Secret as a Bearer token:
41
+
42
+ ```http
43
+ Authorization: Bearer YOUR_LLAMA_API_KEY
44
+ ```
45
+
46
+ The key is checked by the llama-cpp-python API server. The public `/health` endpoint is intentionally unauthenticated so hosting health checks can reach it. Do not expose or share the key.
47
+
48
+ ## Endpoints
49
+
50
+ | Method | Endpoint | Authentication | Purpose |
51
+ |---|---|---|---|
52
+ | `GET` | `/health` | No | Health check |
53
+ | `GET` | `/v1/models` | Bearer key | List the served model |
54
+ | `POST` | `/v1/chat/completions` | Bearer key | Chat Completions; supports streaming |
55
+ | `POST` | `/v1/completions` | Bearer key | Text Completions |
56
+ | `GET` | `/docs` | No | OpenAPI documentation |
57
+ | `GET` | `/console` | No | Small deployment/API information page |
58
+
59
+ ## Quick test with curl
60
+
61
+ Replace the Space URL and key:
62
+
63
+ ```bash
64
+ curl https://YOUR-SPACE-NAME.hf.space/health
65
+ ```
66
+
67
+ ```bash
68
+ curl https://YOUR-SPACE-NAME.hf.space/v1/chat/completions \
69
+ -H "Authorization: Bearer YOUR_LLAMA_API_KEY" \
70
+ -H "Content-Type: application/json" \
71
+ -d '{
72
+ "model": "qwen2.5-coder-0.5b-instruct",
73
+ "messages": [
74
+ {"role": "system", "content": "You are a precise coding assistant."},
75
+ {"role": "user", "content": "Write a JavaScript function that checks whether a string is a palindrome."}
76
+ ],
77
+ "temperature": 0.2,
78
+ "max_tokens": 256,
79
+ "stream": false
80
+ }'
81
+ ```
82
+
83
+ Streaming response:
84
+
85
+ ```bash
86
+ curl -N https://YOUR-SPACE-NAME.hf.space/v1/chat/completions \
87
+ -H "Authorization: Bearer YOUR_LLAMA_API_KEY" \
88
+ -H "Content-Type: application/json" \
89
+ -d '{
90
+ "model": "qwen2.5-coder-0.5b-instruct",
91
+ "messages": [{"role": "user", "content": "Explain JavaScript promises in three bullet points."}],
92
+ "max_tokens": 256,
93
+ "stream": true
94
+ }'
95
+ ```
96
+
97
+ ## Use with the OpenAI SDK
98
+
99
+ ```bash
100
+ npm install openai
101
+ ```
102
+
103
+ ```js
104
+ import OpenAI from "openai";
105
+
106
+ const client = new OpenAI({
107
+ baseURL: "https://YOUR-SPACE-NAME.hf.space/v1",
108
+ apiKey: process.env.LLAMA_API_KEY,
109
+ });
110
+
111
+ const response = await client.chat.completions.create({
112
+ model: "qwen2.5-coder-0.5b-instruct",
113
+ messages: [{ role: "user", content: "Write a TypeScript debounce function." }],
114
+ max_tokens: 256,
115
+ });
116
+
117
+ console.log(response.choices[0].message.content);
118
+ ```
119
+
120
+ Set `LLAMA_API_KEY` in your local shell for the client example. The server-side key belongs only in Hugging Face Space Secrets.
121
+
122
+ ## Performance settings
123
+
124
+ This project is biased toward generation speed on a small CPU instance:
125
+
126
+ - Builds llama-cpp-python from source with `GGML_NATIVE`, OpenMP, and OpenBLAS enabled. `GGML_NATIVE` lets the compiler use the CPU instructions available during the build.
127
+ - Uses two inference threads by default to match CPU Basic's two vCPUs.
128
+ - Uses memory-mapped model loading, a 512-token logical batch, and a 256-token physical micro-batch.
129
+ - Starts with a 2,048-token context to reduce per-request memory and attention work. Increase it when larger prompts are more important than speed.
130
+ - Uses Q4_K_M rather than a larger quantization to keep the model compact.
131
+
132
+ These are performance-oriented defaults, not a guaranteed tokens-per-second figure. Actual throughput depends on the Space host CPU, prompt length, generated length, and concurrent traffic. CPU Basic is not a GPU accelerator. The service serializes access to its single model instance to avoid CPU oversubscription and model-state conflicts.
133
+
134
+ ### Optional Space variables
135
+
136
+ Only `LLAMA_API_KEY` is required. The following variables are optional:
137
+
138
+ | Variable | Default | Meaning |
139
+ |---|---:|---|
140
+ | `MODEL_REPO_ID` | `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF` | Hugging Face model repository |
141
+ | `MODEL_FILENAME` | `qwen2.5-coder-0.5b-instruct-q4_k_m.gguf` | Exact GGUF file to download |
142
+ | `MODEL_ALIAS` | `qwen2.5-coder-0.5b-instruct` | Model name accepted in API requests |
143
+ | `LLAMA_N_THREADS` | `2` | Generation CPU threads |
144
+ | `LLAMA_N_THREADS_BATCH` | `2` | Prompt-processing CPU threads |
145
+ | `LLAMA_N_CTX` | `2048` | Context window size |
146
+ | `LLAMA_N_BATCH` | `512` | Logical prompt batch size |
147
+ | `LLAMA_N_UBATCH` | `256` | Physical prompt micro-batch size |
148
+
149
+ Do not set `LLAMA_API_KEY` as a public variable. Store it under **Secrets**.
150
+
151
+ ## Local run
152
+
153
+ This project targets the Hugging Face Spaces Gradio environment. To run locally, install the Debian packages listed in `packages.txt`, install `requirements.txt`, set `LLAMA_API_KEY`, and run:
154
+
155
+ ```bash
156
+ python app.py
157
+ ```
158
+
159
+ The server listens on port `7860`.
160
+
161
+ ## Model and API references
162
+
163
+ - [Official Qwen GGUF repository](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF)
164
+ - [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
165
+ - [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)
166
+ - [Hugging Face Spaces overview](https://huggingface.co/docs/hub/spaces-overview)
167
+
168
+ The Qwen model repository is licensed under Apache-2.0. Review the model card and license before redistribution or commercial use.
app.py ADDED
@@ -0,0 +1,136 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ from __future__ import annotations
2
+
3
+ import logging
4
+ import os
5
+ from pathlib import Path
6
+
7
+ import gradio as gr
8
+ import uvicorn
9
+ from huggingface_hub import hf_hub_download
10
+ from llama_cpp.server.app import create_app
11
+ from llama_cpp.server.settings import Settings
12
+
13
+ logging.basicConfig(
14
+ level=os.getenv("LOG_LEVEL", "INFO").upper(),
15
+ format="%(asctime)s %(levelname)s %(name)s: %(message)s",
16
+ )
17
+ logger = logging.getLogger("qwen-space")
18
+
19
+ SPACE_PORT = int(os.getenv("PORT", "7860"))
20
+ MODEL_REPO_ID = os.getenv(
21
+ "MODEL_REPO_ID", "Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF"
22
+ )
23
+ MODEL_FILENAME = os.getenv(
24
+ "MODEL_FILENAME", "qwen2.5-coder-0.5b-instruct-q4_k_m.gguf"
25
+ )
26
+ MODEL_ALIAS = os.getenv("MODEL_ALIAS", "qwen2.5-coder-0.5b-instruct")
27
+
28
+
29
+ def _read_positive_int(name: str, default: int) -> int:
30
+ raw = os.getenv(name, str(default))
31
+ try:
32
+ value = int(raw)
33
+ except ValueError as exc:
34
+ raise RuntimeError(f"{name} must be an integer; received {raw!r}") from exc
35
+ if value < 1:
36
+ raise RuntimeError(f"{name} must be >= 1; received {value}")
37
+ return value
38
+
39
+
40
+ def download_model() -> str:
41
+ """Fetch exactly the selected GGUF file into the Hugging Face cache."""
42
+ logger.info("Downloading model %s / %s", MODEL_REPO_ID, MODEL_FILENAME)
43
+ model_path = hf_hub_download(
44
+ repo_id=MODEL_REPO_ID,
45
+ filename=MODEL_FILENAME,
46
+ cache_dir=os.getenv("HF_HUB_CACHE") or None,
47
+ token=os.getenv("HF_TOKEN") or None,
48
+ )
49
+ size_mb = Path(model_path).stat().st_size / (1024 * 1024)
50
+ logger.info("Model ready at %s (%.1f MiB)", model_path, size_mb)
51
+ return model_path
52
+
53
+
54
+ def build_api_app():
55
+ api_key = os.getenv("LLAMA_API_KEY", "").strip()
56
+ if not api_key:
57
+ raise RuntimeError(
58
+ "Missing required Space Secret LLAMA_API_KEY. Add it under "
59
+ "Settings > Variables and secrets > New secret."
60
+ )
61
+
62
+ model_path = download_model()
63
+ threads = _read_positive_int("LLAMA_N_THREADS", 2)
64
+ batch_threads = _read_positive_int("LLAMA_N_THREADS_BATCH", 2)
65
+
66
+ # llama-cpp-python is the llama.cpp C/C++ inference runtime with its
67
+ # OpenAI-compatible FastAPI server and built-in Bearer-key authentication.
68
+ settings = Settings(
69
+ model=model_path,
70
+ model_alias=MODEL_ALIAS,
71
+ host="0.0.0.0",
72
+ port=SPACE_PORT,
73
+ api_key=api_key,
74
+ chat_format="chatml",
75
+ n_ctx=_read_positive_int("LLAMA_N_CTX", 2048),
76
+ n_threads=threads,
77
+ n_threads_batch=batch_threads,
78
+ n_batch=_read_positive_int("LLAMA_N_BATCH", 512),
79
+ n_ubatch=_read_positive_int("LLAMA_N_UBATCH", 256),
80
+ n_gpu_layers=0,
81
+ use_mmap=True,
82
+ use_mlock=False,
83
+ flash_attn=False,
84
+ verbose=False,
85
+ )
86
+ api_app = create_app(settings=settings)
87
+
88
+ # A lightweight public health route for Space monitoring. Inference routes
89
+ # remain protected by the server's built-in Bearer-key dependency.
90
+ @api_app.get("/health", include_in_schema=False)
91
+ async def health():
92
+ return {"status": "ok", "model": MODEL_ALIAS}
93
+
94
+ return api_app
95
+
96
+
97
+ def build_console() -> gr.Blocks:
98
+ with gr.Blocks(title="Qwen2.5 Coder API", analytics_enabled=False) as demo:
99
+ gr.Markdown(
100
+ """
101
+ # ⚡ Qwen2.5-Coder-0.5B-Instruct API
102
+
103
+ The model is running through llama.cpp. This Space exposes an
104
+ OpenAI-compatible REST API; inference endpoints require your
105
+ configured Bearer API key.
106
+
107
+ **API base URL**
108
+
109
+ ```text
110
+ /v1
111
+ ```
112
+
113
+ **Main endpoints**
114
+
115
+ - `GET /health` — public service health check
116
+ - `GET /v1/models` — model list (Bearer key required)
117
+ - `POST /v1/chat/completions` — chat and streaming completions
118
+ - `POST /v1/completions` — text completions
119
+ - `GET /docs` — OpenAPI schema
120
+
121
+ Send `Authorization: Bearer YOUR_LLAMA_API_KEY` with `/v1` requests.
122
+ The API key is never displayed on this page.
123
+ """
124
+ )
125
+ return demo
126
+
127
+
128
+ def main() -> None:
129
+ api_app = build_api_app()
130
+ app = gr.mount_gradio_app(api_app, build_console(), path="/console")
131
+ logger.info("Starting API on 0.0.0.0:%s", SPACE_PORT)
132
+ uvicorn.run(app, host="0.0.0.0", port=SPACE_PORT, log_level="info")
133
+
134
+
135
+ if __name__ == "__main__":
136
+ main()
packages.txt ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ build-essential
2
+ cmake
3
+ ninja-build
4
+ libopenblas-dev
requirements.txt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ --no-binary=llama-cpp-python
2
+ llama-cpp-python[server]>=0.3.16,<0.4 --config-settings=cmake.args="-DGGML_NATIVE=ON;-DGGML_OPENMP=ON;-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"
3
+ huggingface_hub>=0.27.0