Unmid commited on
Commit
fede21e
·
verified ·
1 Parent(s): 6a4954d

Update app.py

Browse files
Files changed (1) hide show
  1. app.py +1 -77
app.py CHANGED
@@ -1,10 +1,3 @@
1
- I'm **Claude** (Anthropic). My reliable knowledge cutoff is **end of January 2025** — I know things up to around then; anything later (new llama.cpp release tags, brand‑new model repos like a "MiniCPM5") I can't verify, so I've made those **env‑configurable + auto‑detected** in the code below instead of hard‑coding something that might 404.
2
-
3
- ---
4
-
5
- # 1. `app.py`
6
-
7
- ```python
8
  # app.py — GGUF chat Space (prebuilt llama.cpp server from GitHub, NO local compile)
9
  import os
10
  import io
@@ -207,73 +200,4 @@ with gr.Blocks(title="GGUF Chat (llama.cpp prebuilt)") as demo:
207
  demo.queue(max_size=20).launch(server_name="0.0.0.0",
208
  server_port=int(os.getenv("GRADIO_PORT", "7860")),
209
  ssr_mode=False,
210
- show_api=True)
211
- ```
212
-
213
- ---
214
-
215
- # 2. `requirements.txt`
216
-
217
- ```
218
- gradio==5.9.1
219
- huggingface_hub>=0.26.0
220
- requests>=2.32.0
221
- ```
222
-
223
- That's it — **no `llama-cpp-python`, no compiler, no `spaces`**. The engine is a downloaded ELF binary.
224
-
225
- > If you *insist* on the Python binding instead of the HTTP server, the only no-compile way is a prebuilt wheel index:
226
- > ```
227
- > --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu
228
- > llama-cpp-python
229
- > ```
230
- > (wheel availability depends on your Python version — the subprocess approach above has zero such risk).
231
-
232
- ---
233
-
234
- # 3. Short doc — the important bits
235
-
236
- ### A. How the original Edge-TTS code "avoids GPU"
237
- | Thing | What it actually does |
238
- |---|---|
239
- | `import spaces` + `@spaces.GPU(duration=1)` | On **ZeroGPU** hardware, HF statically scans the file for a `@spaces.GPU` decorator at startup. No decorator → the Space is rejected/ idles wrongly. The `dummy_gpu_trigger` is a **decoy** so the container boots. |
240
- | `dummy_btn = gr.Button(visible=False)` | Keeps the decorated fn wired into the graph (so the scanner sees it used) while hiding it from users. |
241
- | Real work (`edge_tts`) is pure network I/O | Microsoft's cloud does the synthesis → **0 GPU seconds consumed**, no CUDA, no torch. |
242
- | `duration=1` | If ZeroGPU ever *does* allocate, it reserves 1 second — negligible quota. |
243
-
244
- 👉 **On a plain CPU Space you should delete `import spaces` and the dummy entirely** (that's what I did above). Keep the trick only if the Space hardware is literally ZeroGPU and you want it to boot while doing CPU/network work.
245
-
246
- ### B. How it becomes an external API
247
- `api_name="speech"` / `api_name="chat"` publishes a named route. Everything else is Gradio's built-in HTTP layer.
248
-
249
- **Python client**
250
- ```python
251
- from gradio_client import Client
252
- c = Client("USER/SPACE") # or "http://localhost:7860"
253
- print(c.predict("Hello", [], "You are helpful.", 512, 0.7, 0.95, api_name="/chat"))
254
- ```
255
-
256
- **Raw REST (Gradio 5 = 2-step)**
257
- ```bash
258
- EV=$(curl -s -XPOST https://USER-SPACE.hf.space/gradio_api/call/chat \
259
- -H 'Content-Type: application/json' \
260
- -d '{"data":["Hello",[],"You are helpful.",512,0.7,0.95]}' | jq -r .event_id)
261
- curl -N https://USER-SPACE.hf.space/gradio_api/call/chat/$EV # SSE stream
262
- ```
263
- * Live schema: **`https://<space>.hf.space/?view=api`**
264
- * Private Space → header `Authorization: Bearer hf_xxx`.
265
- * `show_api=True` + `demo.queue()` are required for stable concurrent API use.
266
- * Generator functions stream over the API too (SSE), same as in the UI.
267
-
268
- ### C. llama.cpp specifics in this app
269
- * Binary: `ggml-org/llama.cpp` GitHub **Releases → `llama-<tag>-bin-ubuntu-x64.zip`** (CPU, glibc). Pin it: `LLAMA_TAG=b4667` — `latest` can break asset names.
270
- * Bundled `libllama.so`/`libggml*.so` sit next to the binary → **`LD_LIBRARY_PATH` must include that dir** or you get "cannot open shared object file".
271
- * `llama-server` speaks **OpenAI v1** (`/v1/chat/completions`, `/v1/completions`, `/v1/embeddings`) + `/health`, `/props`. Chat template is read from the GGUF metadata — no manual prompt formatting.
272
- * It stays on `127.0.0.1` — only Gradio is exposed. To expose it directly, mount it or proxy it (HF Spaces only routes **one** port, 7860).
273
-
274
- ### D. Resources / gotchas for a ~1.6 GB Q4_K_M
275
- * RAM ≈ model (1.56 GB) + KV-cache. KV at 4096 ctx for a 2B ≈ 0.3–0.6 GB → fits free CPU Spaces (16 GB) easily; lower `CTX` if tight.
276
- * Speed on free 2-vCPU: roughly **5–15 tok/s** for a 1–2B Q4. Set `THREADS` = physical cores; more threads than cores *hurts*.
277
- * First boot downloads ~50 MB binary + model → use `WORK_DIR=/data` on persistent storage, else it re-downloads every cold start.
278
- * `-ngl 0` is mandatory for the CPU build; `-ngl 99` only works if you swap `LLAMA_ASSET` to a CUDA asset **and** run on GPU hardware (then you also need the matching cudart zip).
279
- * `MODEL_REPO="openbmb/MiniCPM5-2B-GGUF"` — I can't confirm that repo exists (post-cutoff for me). If it 404s the code auto-falls back; just set `MODEL_REPO`/`MODEL_FILE` to whatever you see on the Hub.
 
 
 
 
 
 
 
 
1
  # app.py — GGUF chat Space (prebuilt llama.cpp server from GitHub, NO local compile)
2
  import os
3
  import io
 
200
  demo.queue(max_size=20).launch(server_name="0.0.0.0",
201
  server_port=int(os.getenv("GRADIO_PORT", "7860")),
202
  ssr_mode=False,
203
+ show_api=True)