Update app.py
Browse files
app.py
CHANGED
|
@@ -1,10 +1,3 @@
|
|
| 1 |
-
I'm **Claude** (Anthropic). My reliable knowledge cutoff is **end of January 2025** — I know things up to around then; anything later (new llama.cpp release tags, brand‑new model repos like a "MiniCPM5") I can't verify, so I've made those **env‑configurable + auto‑detected** in the code below instead of hard‑coding something that might 404.
|
| 2 |
-
|
| 3 |
-
---
|
| 4 |
-
|
| 5 |
-
# 1. `app.py`
|
| 6 |
-
|
| 7 |
-
```python
|
| 8 |
# app.py — GGUF chat Space (prebuilt llama.cpp server from GitHub, NO local compile)
|
| 9 |
import os
|
| 10 |
import io
|
|
@@ -207,73 +200,4 @@ with gr.Blocks(title="GGUF Chat (llama.cpp prebuilt)") as demo:
|
|
| 207 |
demo.queue(max_size=20).launch(server_name="0.0.0.0",
|
| 208 |
server_port=int(os.getenv("GRADIO_PORT", "7860")),
|
| 209 |
ssr_mode=False,
|
| 210 |
-
show_api=True)
|
| 211 |
-
```
|
| 212 |
-
|
| 213 |
-
---
|
| 214 |
-
|
| 215 |
-
# 2. `requirements.txt`
|
| 216 |
-
|
| 217 |
-
```
|
| 218 |
-
gradio==5.9.1
|
| 219 |
-
huggingface_hub>=0.26.0
|
| 220 |
-
requests>=2.32.0
|
| 221 |
-
```
|
| 222 |
-
|
| 223 |
-
That's it — **no `llama-cpp-python`, no compiler, no `spaces`**. The engine is a downloaded ELF binary.
|
| 224 |
-
|
| 225 |
-
> If you *insist* on the Python binding instead of the HTTP server, the only no-compile way is a prebuilt wheel index:
|
| 226 |
-
> ```
|
| 227 |
-
> --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
|
| 228 |
-
> llama-cpp-python
|
| 229 |
-
> ```
|
| 230 |
-
> (wheel availability depends on your Python version — the subprocess approach above has zero such risk).
|
| 231 |
-
|
| 232 |
-
---
|
| 233 |
-
|
| 234 |
-
# 3. Short doc — the important bits
|
| 235 |
-
|
| 236 |
-
### A. How the original Edge-TTS code "avoids GPU"
|
| 237 |
-
| Thing | What it actually does |
|
| 238 |
-
|---|---|
|
| 239 |
-
| `import spaces` + `@spaces.GPU(duration=1)` | On **ZeroGPU** hardware, HF statically scans the file for a `@spaces.GPU` decorator at startup. No decorator → the Space is rejected/ idles wrongly. The `dummy_gpu_trigger` is a **decoy** so the container boots. |
|
| 240 |
-
| `dummy_btn = gr.Button(visible=False)` | Keeps the decorated fn wired into the graph (so the scanner sees it used) while hiding it from users. |
|
| 241 |
-
| Real work (`edge_tts`) is pure network I/O | Microsoft's cloud does the synthesis → **0 GPU seconds consumed**, no CUDA, no torch. |
|
| 242 |
-
| `duration=1` | If ZeroGPU ever *does* allocate, it reserves 1 second — negligible quota. |
|
| 243 |
-
|
| 244 |
-
👉 **On a plain CPU Space you should delete `import spaces` and the dummy entirely** (that's what I did above). Keep the trick only if the Space hardware is literally ZeroGPU and you want it to boot while doing CPU/network work.
|
| 245 |
-
|
| 246 |
-
### B. How it becomes an external API
|
| 247 |
-
`api_name="speech"` / `api_name="chat"` publishes a named route. Everything else is Gradio's built-in HTTP layer.
|
| 248 |
-
|
| 249 |
-
**Python client**
|
| 250 |
-
```python
|
| 251 |
-
from gradio_client import Client
|
| 252 |
-
c = Client("USER/SPACE") # or "http://localhost:7860"
|
| 253 |
-
print(c.predict("Hello", [], "You are helpful.", 512, 0.7, 0.95, api_name="/chat"))
|
| 254 |
-
```
|
| 255 |
-
|
| 256 |
-
**Raw REST (Gradio 5 = 2-step)**
|
| 257 |
-
```bash
|
| 258 |
-
EV=$(curl -s -XPOST https://USER-SPACE.hf.space/gradio_api/call/chat \
|
| 259 |
-
-H 'Content-Type: application/json' \
|
| 260 |
-
-d '{"data":["Hello",[],"You are helpful.",512,0.7,0.95]}' | jq -r .event_id)
|
| 261 |
-
curl -N https://USER-SPACE.hf.space/gradio_api/call/chat/$EV # SSE stream
|
| 262 |
-
```
|
| 263 |
-
* Live schema: **`https://<space>.hf.space/?view=api`**
|
| 264 |
-
* Private Space → header `Authorization: Bearer hf_xxx`.
|
| 265 |
-
* `show_api=True` + `demo.queue()` are required for stable concurrent API use.
|
| 266 |
-
* Generator functions stream over the API too (SSE), same as in the UI.
|
| 267 |
-
|
| 268 |
-
### C. llama.cpp specifics in this app
|
| 269 |
-
* Binary: `ggml-org/llama.cpp` GitHub **Releases → `llama-<tag>-bin-ubuntu-x64.zip`** (CPU, glibc). Pin it: `LLAMA_TAG=b4667` — `latest` can break asset names.
|
| 270 |
-
* Bundled `libllama.so`/`libggml*.so` sit next to the binary → **`LD_LIBRARY_PATH` must include that dir** or you get "cannot open shared object file".
|
| 271 |
-
* `llama-server` speaks **OpenAI v1** (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`) + `/health`, `/props`. Chat template is read from the GGUF metadata — no manual prompt formatting.
|
| 272 |
-
* It stays on `127.0.0.1` — only Gradio is exposed. To expose it directly, mount it or proxy it (HF Spaces only routes **one** port, 7860).
|
| 273 |
-
|
| 274 |
-
### D. Resources / gotchas for a ~1.6 GB Q4_K_M
|
| 275 |
-
* RAM ≈ model (1.56 GB) + KV-cache. KV at 4096 ctx for a 2B ≈ 0.3–0.6 GB → fits free CPU Spaces (16 GB) easily; lower `CTX` if tight.
|
| 276 |
-
* Speed on free 2-vCPU: roughly **5–15 tok/s** for a 1–2B Q4. Set `THREADS` = physical cores; more threads than cores *hurts*.
|
| 277 |
-
* First boot downloads ~50 MB binary + model → use `WORK_DIR=/data` on persistent storage, else it re-downloads every cold start.
|
| 278 |
-
* `-ngl 0` is mandatory for the CPU build; `-ngl 99` only works if you swap `LLAMA_ASSET` to a CUDA asset **and** run on GPU hardware (then you also need the matching cudart zip).
|
| 279 |
-
* `MODEL_REPO="openbmb/MiniCPM5-2B-GGUF"` — I can't confirm that repo exists (post-cutoff for me). If it 404s the code auto-falls back; just set `MODEL_REPO`/`MODEL_FILE` to whatever you see on the Hub.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# app.py — GGUF chat Space (prebuilt llama.cpp server from GitHub, NO local compile)
|
| 2 |
import os
|
| 3 |
import io
|
|
|
|
| 200 |
demo.queue(max_size=20).launch(server_name="0.0.0.0",
|
| 201 |
server_port=int(os.getenv("GRADIO_PORT", "7860")),
|
| 202 |
ssr_mode=False,
|
| 203 |
+
show_api=True)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|