Spaces:
Running on Zero
Running on Zero
Upload 6 files
Browse files- .env.example +10 -0
- .gitignore +6 -0
- README.md +168 -0
- app.py +136 -0
- packages.txt +4 -0
- requirements.txt +3 -0
.env.example
ADDED
|
@@ -0,0 +1,10 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Required. Put the real value in Hugging Face Space Secrets, not in Git.
|
| 2 |
+
LLAMA_API_KEY=replace-with-a-long-random-secret
|
| 3 |
+
|
| 4 |
+
# Optional performance/model overrides:
|
| 5 |
+
# LLAMA_N_THREADS=2
|
| 6 |
+
# LLAMA_N_THREADS_BATCH=2
|
| 7 |
+
# LLAMA_N_CTX=2048
|
| 8 |
+
# LLAMA_N_BATCH=512
|
| 9 |
+
# LLAMA_N_UBATCH=256
|
| 10 |
+
# MODEL_ALIAS=qwen2.5-coder-0.5b-instruct
|
.gitignore
ADDED
|
@@ -0,0 +1,6 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.env
|
| 2 |
+
.venv/
|
| 3 |
+
__pycache__/
|
| 4 |
+
*.py[cod]
|
| 5 |
+
.cache/
|
| 6 |
+
models/
|
README.md
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
title: Qwen2.5 Coder 0.5B API
|
| 3 |
+
emoji: ⚡
|
| 4 |
+
colorFrom: blue
|
| 5 |
+
colorTo: purple
|
| 6 |
+
sdk: gradio
|
| 7 |
+
python_version: "3.11"
|
| 8 |
+
app_file: app.py
|
| 9 |
+
---
|
| 10 |
+
|
| 11 |
+
# Qwen2.5-Coder-0.5B-Instruct REST API
|
| 12 |
+
|
| 13 |
+
A Hugging Face **Gradio SDK Space** that serves the official Qwen2.5-Coder-0.5B-Instruct `Q4_K_M` GGUF model through a llama.cpp-backed, OpenAI-compatible REST API.
|
| 14 |
+
|
| 15 |
+
- Model: `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF`
|
| 16 |
+
- Quantization: `Q4_K_M` (about 491 MB on the Hub)
|
| 17 |
+
- Runtime: `llama-cpp-python` (Python bindings to llama.cpp)
|
| 18 |
+
- Default hardware target: Hugging Face CPU Basic (2 vCPU / 16 GB RAM)
|
| 19 |
+
- Authentication: one required bearer API key from the Space Secret `LLAMA_API_KEY`
|
| 20 |
+
- Streaming: supported by `/v1/chat/completions`
|
| 21 |
+
- Docker: **not required**
|
| 22 |
+
|
| 23 |
+
## Deploy to Hugging Face Spaces
|
| 24 |
+
|
| 25 |
+
1. Create a new Space and select **Gradio** as the SDK.
|
| 26 |
+
2. Upload or push all files from this project to the Space repository.
|
| 27 |
+
3. Open **Settings → Variables and secrets**.
|
| 28 |
+
4. Add a **Secret** named `LLAMA_API_KEY` and set a long, random value. Do not put the key in this repository or in a normal variable.
|
| 29 |
+
5. Select **CPU Basic** hardware if it is not already selected, then let the Space build and start.
|
| 30 |
+
6. Once it is running, the API base URL is:
|
| 31 |
+
|
| 32 |
+
`https://YOUR-SPACE-NAME.hf.space/v1`
|
| 33 |
+
|
| 34 |
+
The model file is downloaded automatically from the official Qwen repository on first startup. It is not included in this ZIP.
|
| 35 |
+
|
| 36 |
+
> **Hugging Face plan note (checked November 19, 2026):** Hugging Face documents CPU Basic as 2 vCPU / 16 GB RAM with no hourly compute charge, but creating a compute-backed Gradio or Docker Space on a personal account requires PRO. Free personal accounts can still run up to two Gradio Spaces on ZeroGPU; this project is a CPU-based llama.cpp service, not a ZeroGPU app. So the runtime hardware can be CPU Basic at no hourly cost, but a free personal account alone may not be able to create/run this kind of Space.
|
| 37 |
+
|
| 38 |
+
## API authentication
|
| 39 |
+
|
| 40 |
+
Every `/v1` inference/API request must include the single Space Secret as a Bearer token:
|
| 41 |
+
|
| 42 |
+
```http
|
| 43 |
+
Authorization: Bearer YOUR_LLAMA_API_KEY
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
The key is checked by the llama-cpp-python API server. The public `/health` endpoint is intentionally unauthenticated so hosting health checks can reach it. Do not expose or share the key.
|
| 47 |
+
|
| 48 |
+
## Endpoints
|
| 49 |
+
|
| 50 |
+
| Method | Endpoint | Authentication | Purpose |
|
| 51 |
+
|---|---|---|---|
|
| 52 |
+
| `GET` | `/health` | No | Health check |
|
| 53 |
+
| `GET` | `/v1/models` | Bearer key | List the served model |
|
| 54 |
+
| `POST` | `/v1/chat/completions` | Bearer key | Chat Completions; supports streaming |
|
| 55 |
+
| `POST` | `/v1/completions` | Bearer key | Text Completions |
|
| 56 |
+
| `GET` | `/docs` | No | OpenAPI documentation |
|
| 57 |
+
| `GET` | `/console` | No | Small deployment/API information page |
|
| 58 |
+
|
| 59 |
+
## Quick test with curl
|
| 60 |
+
|
| 61 |
+
Replace the Space URL and key:
|
| 62 |
+
|
| 63 |
+
```bash
|
| 64 |
+
curl https://YOUR-SPACE-NAME.hf.space/health
|
| 65 |
+
```
|
| 66 |
+
|
| 67 |
+
```bash
|
| 68 |
+
curl https://YOUR-SPACE-NAME.hf.space/v1/chat/completions \
|
| 69 |
+
-H "Authorization: Bearer YOUR_LLAMA_API_KEY" \
|
| 70 |
+
-H "Content-Type: application/json" \
|
| 71 |
+
-d '{
|
| 72 |
+
"model": "qwen2.5-coder-0.5b-instruct",
|
| 73 |
+
"messages": [
|
| 74 |
+
{"role": "system", "content": "You are a precise coding assistant."},
|
| 75 |
+
{"role": "user", "content": "Write a JavaScript function that checks whether a string is a palindrome."}
|
| 76 |
+
],
|
| 77 |
+
"temperature": 0.2,
|
| 78 |
+
"max_tokens": 256,
|
| 79 |
+
"stream": false
|
| 80 |
+
}'
|
| 81 |
+
```
|
| 82 |
+
|
| 83 |
+
Streaming response:
|
| 84 |
+
|
| 85 |
+
```bash
|
| 86 |
+
curl -N https://YOUR-SPACE-NAME.hf.space/v1/chat/completions \
|
| 87 |
+
-H "Authorization: Bearer YOUR_LLAMA_API_KEY" \
|
| 88 |
+
-H "Content-Type: application/json" \
|
| 89 |
+
-d '{
|
| 90 |
+
"model": "qwen2.5-coder-0.5b-instruct",
|
| 91 |
+
"messages": [{"role": "user", "content": "Explain JavaScript promises in three bullet points."}],
|
| 92 |
+
"max_tokens": 256,
|
| 93 |
+
"stream": true
|
| 94 |
+
}'
|
| 95 |
+
```
|
| 96 |
+
|
| 97 |
+
## Use with the OpenAI SDK
|
| 98 |
+
|
| 99 |
+
```bash
|
| 100 |
+
npm install openai
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
```js
|
| 104 |
+
import OpenAI from "openai";
|
| 105 |
+
|
| 106 |
+
const client = new OpenAI({
|
| 107 |
+
baseURL: "https://YOUR-SPACE-NAME.hf.space/v1",
|
| 108 |
+
apiKey: process.env.LLAMA_API_KEY,
|
| 109 |
+
});
|
| 110 |
+
|
| 111 |
+
const response = await client.chat.completions.create({
|
| 112 |
+
model: "qwen2.5-coder-0.5b-instruct",
|
| 113 |
+
messages: [{ role: "user", content: "Write a TypeScript debounce function." }],
|
| 114 |
+
max_tokens: 256,
|
| 115 |
+
});
|
| 116 |
+
|
| 117 |
+
console.log(response.choices[0].message.content);
|
| 118 |
+
```
|
| 119 |
+
|
| 120 |
+
Set `LLAMA_API_KEY` in your local shell for the client example. The server-side key belongs only in Hugging Face Space Secrets.
|
| 121 |
+
|
| 122 |
+
## Performance settings
|
| 123 |
+
|
| 124 |
+
This project is biased toward generation speed on a small CPU instance:
|
| 125 |
+
|
| 126 |
+
- Builds llama-cpp-python from source with `GGML_NATIVE`, OpenMP, and OpenBLAS enabled. `GGML_NATIVE` lets the compiler use the CPU instructions available during the build.
|
| 127 |
+
- Uses two inference threads by default to match CPU Basic's two vCPUs.
|
| 128 |
+
- Uses memory-mapped model loading, a 512-token logical batch, and a 256-token physical micro-batch.
|
| 129 |
+
- Starts with a 2,048-token context to reduce per-request memory and attention work. Increase it when larger prompts are more important than speed.
|
| 130 |
+
- Uses Q4_K_M rather than a larger quantization to keep the model compact.
|
| 131 |
+
|
| 132 |
+
These are performance-oriented defaults, not a guaranteed tokens-per-second figure. Actual throughput depends on the Space host CPU, prompt length, generated length, and concurrent traffic. CPU Basic is not a GPU accelerator. The service serializes access to its single model instance to avoid CPU oversubscription and model-state conflicts.
|
| 133 |
+
|
| 134 |
+
### Optional Space variables
|
| 135 |
+
|
| 136 |
+
Only `LLAMA_API_KEY` is required. The following variables are optional:
|
| 137 |
+
|
| 138 |
+
| Variable | Default | Meaning |
|
| 139 |
+
|---|---:|---|
|
| 140 |
+
| `MODEL_REPO_ID` | `Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF` | Hugging Face model repository |
|
| 141 |
+
| `MODEL_FILENAME` | `qwen2.5-coder-0.5b-instruct-q4_k_m.gguf` | Exact GGUF file to download |
|
| 142 |
+
| `MODEL_ALIAS` | `qwen2.5-coder-0.5b-instruct` | Model name accepted in API requests |
|
| 143 |
+
| `LLAMA_N_THREADS` | `2` | Generation CPU threads |
|
| 144 |
+
| `LLAMA_N_THREADS_BATCH` | `2` | Prompt-processing CPU threads |
|
| 145 |
+
| `LLAMA_N_CTX` | `2048` | Context window size |
|
| 146 |
+
| `LLAMA_N_BATCH` | `512` | Logical prompt batch size |
|
| 147 |
+
| `LLAMA_N_UBATCH` | `256` | Physical prompt micro-batch size |
|
| 148 |
+
|
| 149 |
+
Do not set `LLAMA_API_KEY` as a public variable. Store it under **Secrets**.
|
| 150 |
+
|
| 151 |
+
## Local run
|
| 152 |
+
|
| 153 |
+
This project targets the Hugging Face Spaces Gradio environment. To run locally, install the Debian packages listed in `packages.txt`, install `requirements.txt`, set `LLAMA_API_KEY`, and run:
|
| 154 |
+
|
| 155 |
+
```bash
|
| 156 |
+
python app.py
|
| 157 |
+
```
|
| 158 |
+
|
| 159 |
+
The server listens on port `7860`.
|
| 160 |
+
|
| 161 |
+
## Model and API references
|
| 162 |
+
|
| 163 |
+
- [Official Qwen GGUF repository](https://huggingface.co/Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF)
|
| 164 |
+
- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python)
|
| 165 |
+
- [llama.cpp server documentation](https://github.com/ggml-org/llama.cpp/tree/master/tools/server)
|
| 166 |
+
- [Hugging Face Spaces overview](https://huggingface.co/docs/hub/spaces-overview)
|
| 167 |
+
|
| 168 |
+
The Qwen model repository is licensed under Apache-2.0. Review the model card and license before redistribution or commercial use.
|
app.py
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
from __future__ import annotations
|
| 2 |
+
|
| 3 |
+
import logging
|
| 4 |
+
import os
|
| 5 |
+
from pathlib import Path
|
| 6 |
+
|
| 7 |
+
import gradio as gr
|
| 8 |
+
import uvicorn
|
| 9 |
+
from huggingface_hub import hf_hub_download
|
| 10 |
+
from llama_cpp.server.app import create_app
|
| 11 |
+
from llama_cpp.server.settings import Settings
|
| 12 |
+
|
| 13 |
+
logging.basicConfig(
|
| 14 |
+
level=os.getenv("LOG_LEVEL", "INFO").upper(),
|
| 15 |
+
format="%(asctime)s %(levelname)s %(name)s: %(message)s",
|
| 16 |
+
)
|
| 17 |
+
logger = logging.getLogger("qwen-space")
|
| 18 |
+
|
| 19 |
+
SPACE_PORT = int(os.getenv("PORT", "7860"))
|
| 20 |
+
MODEL_REPO_ID = os.getenv(
|
| 21 |
+
"MODEL_REPO_ID", "Qwen/Qwen2.5-Coder-0.5B-Instruct-GGUF"
|
| 22 |
+
)
|
| 23 |
+
MODEL_FILENAME = os.getenv(
|
| 24 |
+
"MODEL_FILENAME", "qwen2.5-coder-0.5b-instruct-q4_k_m.gguf"
|
| 25 |
+
)
|
| 26 |
+
MODEL_ALIAS = os.getenv("MODEL_ALIAS", "qwen2.5-coder-0.5b-instruct")
|
| 27 |
+
|
| 28 |
+
|
| 29 |
+
def _read_positive_int(name: str, default: int) -> int:
|
| 30 |
+
raw = os.getenv(name, str(default))
|
| 31 |
+
try:
|
| 32 |
+
value = int(raw)
|
| 33 |
+
except ValueError as exc:
|
| 34 |
+
raise RuntimeError(f"{name} must be an integer; received {raw!r}") from exc
|
| 35 |
+
if value < 1:
|
| 36 |
+
raise RuntimeError(f"{name} must be >= 1; received {value}")
|
| 37 |
+
return value
|
| 38 |
+
|
| 39 |
+
|
| 40 |
+
def download_model() -> str:
|
| 41 |
+
"""Fetch exactly the selected GGUF file into the Hugging Face cache."""
|
| 42 |
+
logger.info("Downloading model %s / %s", MODEL_REPO_ID, MODEL_FILENAME)
|
| 43 |
+
model_path = hf_hub_download(
|
| 44 |
+
repo_id=MODEL_REPO_ID,
|
| 45 |
+
filename=MODEL_FILENAME,
|
| 46 |
+
cache_dir=os.getenv("HF_HUB_CACHE") or None,
|
| 47 |
+
token=os.getenv("HF_TOKEN") or None,
|
| 48 |
+
)
|
| 49 |
+
size_mb = Path(model_path).stat().st_size / (1024 * 1024)
|
| 50 |
+
logger.info("Model ready at %s (%.1f MiB)", model_path, size_mb)
|
| 51 |
+
return model_path
|
| 52 |
+
|
| 53 |
+
|
| 54 |
+
def build_api_app():
|
| 55 |
+
api_key = os.getenv("LLAMA_API_KEY", "").strip()
|
| 56 |
+
if not api_key:
|
| 57 |
+
raise RuntimeError(
|
| 58 |
+
"Missing required Space Secret LLAMA_API_KEY. Add it under "
|
| 59 |
+
"Settings > Variables and secrets > New secret."
|
| 60 |
+
)
|
| 61 |
+
|
| 62 |
+
model_path = download_model()
|
| 63 |
+
threads = _read_positive_int("LLAMA_N_THREADS", 2)
|
| 64 |
+
batch_threads = _read_positive_int("LLAMA_N_THREADS_BATCH", 2)
|
| 65 |
+
|
| 66 |
+
# llama-cpp-python is the llama.cpp C/C++ inference runtime with its
|
| 67 |
+
# OpenAI-compatible FastAPI server and built-in Bearer-key authentication.
|
| 68 |
+
settings = Settings(
|
| 69 |
+
model=model_path,
|
| 70 |
+
model_alias=MODEL_ALIAS,
|
| 71 |
+
host="0.0.0.0",
|
| 72 |
+
port=SPACE_PORT,
|
| 73 |
+
api_key=api_key,
|
| 74 |
+
chat_format="chatml",
|
| 75 |
+
n_ctx=_read_positive_int("LLAMA_N_CTX", 2048),
|
| 76 |
+
n_threads=threads,
|
| 77 |
+
n_threads_batch=batch_threads,
|
| 78 |
+
n_batch=_read_positive_int("LLAMA_N_BATCH", 512),
|
| 79 |
+
n_ubatch=_read_positive_int("LLAMA_N_UBATCH", 256),
|
| 80 |
+
n_gpu_layers=0,
|
| 81 |
+
use_mmap=True,
|
| 82 |
+
use_mlock=False,
|
| 83 |
+
flash_attn=False,
|
| 84 |
+
verbose=False,
|
| 85 |
+
)
|
| 86 |
+
api_app = create_app(settings=settings)
|
| 87 |
+
|
| 88 |
+
# A lightweight public health route for Space monitoring. Inference routes
|
| 89 |
+
# remain protected by the server's built-in Bearer-key dependency.
|
| 90 |
+
@api_app.get("/health", include_in_schema=False)
|
| 91 |
+
async def health():
|
| 92 |
+
return {"status": "ok", "model": MODEL_ALIAS}
|
| 93 |
+
|
| 94 |
+
return api_app
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def build_console() -> gr.Blocks:
|
| 98 |
+
with gr.Blocks(title="Qwen2.5 Coder API", analytics_enabled=False) as demo:
|
| 99 |
+
gr.Markdown(
|
| 100 |
+
"""
|
| 101 |
+
# ⚡ Qwen2.5-Coder-0.5B-Instruct API
|
| 102 |
+
|
| 103 |
+
The model is running through llama.cpp. This Space exposes an
|
| 104 |
+
OpenAI-compatible REST API; inference endpoints require your
|
| 105 |
+
configured Bearer API key.
|
| 106 |
+
|
| 107 |
+
**API base URL**
|
| 108 |
+
|
| 109 |
+
```text
|
| 110 |
+
/v1
|
| 111 |
+
```
|
| 112 |
+
|
| 113 |
+
**Main endpoints**
|
| 114 |
+
|
| 115 |
+
- `GET /health` — public service health check
|
| 116 |
+
- `GET /v1/models` — model list (Bearer key required)
|
| 117 |
+
- `POST /v1/chat/completions` — chat and streaming completions
|
| 118 |
+
- `POST /v1/completions` — text completions
|
| 119 |
+
- `GET /docs` — OpenAPI schema
|
| 120 |
+
|
| 121 |
+
Send `Authorization: Bearer YOUR_LLAMA_API_KEY` with `/v1` requests.
|
| 122 |
+
The API key is never displayed on this page.
|
| 123 |
+
"""
|
| 124 |
+
)
|
| 125 |
+
return demo
|
| 126 |
+
|
| 127 |
+
|
| 128 |
+
def main() -> None:
|
| 129 |
+
api_app = build_api_app()
|
| 130 |
+
app = gr.mount_gradio_app(api_app, build_console(), path="/console")
|
| 131 |
+
logger.info("Starting API on 0.0.0.0:%s", SPACE_PORT)
|
| 132 |
+
uvicorn.run(app, host="0.0.0.0", port=SPACE_PORT, log_level="info")
|
| 133 |
+
|
| 134 |
+
|
| 135 |
+
if __name__ == "__main__":
|
| 136 |
+
main()
|
packages.txt
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
build-essential
|
| 2 |
+
cmake
|
| 3 |
+
ninja-build
|
| 4 |
+
libopenblas-dev
|
requirements.txt
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
--no-binary=llama-cpp-python
|
| 2 |
+
llama-cpp-python[server]>=0.3.16,<0.4 --config-settings=cmake.args="-DGGML_NATIVE=ON;-DGGML_OPENMP=ON;-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"
|
| 3 |
+
huggingface_hub>=0.27.0
|