|
Download README.md from zrxarchit/inference: direct link, hf CLI and curl.
- Browser
- Download file 7.68 kB
-
https://huggingface.co/spaces/zrxarchit/inference/resolve/main/README.md
- Command line
-
hf download hf://spaces/zrxarchit/inference/README.md
-
curl -L -o README.md https://huggingface.co/spaces/zrxarchit/inference/resolve/main/README.md
7.68 kB
| title: Inference | |
| emoji: ⚡ | |
| colorFrom: gray | |
| colorTo: indigo | |
| sdk: docker | |
| pinned: false | |
| # CPU Inference Backend | |
| This Space is a backend-only OpenAI-compatible inference server built on `llama.cpp` and optimized for CPU-only Hugging Face Docker Spaces. | |
| It automatically downloads the target Hugging Face model from `MODEL_NAME`, stores it under `/data/models/model.gguf`, and starts `llama-server` with OpenAI-compatible endpoints. | |
| ## Environment variables | |
| Set these in the Space settings: | |
| - `MODEL_NAME`: Hugging Face repo id that contains one or more GGUF files | |
| - `MODEL_FILE`: optional exact GGUF filename inside `MODEL_NAME`; useful for testing faster/lower quality quants | |
| - `HF_TOKEN`: Hugging Face token used for the download | |
| - `API_PASSWORD`: bearer token required by the API | |
| Optional tuning variables: | |
| - `CTX_SIZE`: context size, default `4096` (reduce to 2048-4096 for small models to improve speed) | |
| - `THREADS`: CPU thread count, default auto-detect via cgroup CPU quota, then `nproc` | |
| - `OMP_PROC_BIND`, `OMP_PLACES`, `OMP_WAIT_POLICY`: OpenMP CPU placement controls, default `FALSE`, `cores`, and `PASSIVE` | |
| - `THREADS_BATCH`: batch thread count, default matches THREADS | |
| - `BATCH_SIZE`: prompt batch size, default 128 for balanced mode | |
| - `UBATCH_SIZE`: micro-batch size, default 128 for balanced mode | |
| - `CACHE_TYPE_K`: KV cache type for keys, default `f16` (use `q4_0` for memory savings) | |
| - `CACHE_TYPE_V`: KV cache type for values, default `f16` (use `q4_0` for memory savings) | |
| - `PERF_PROFILE`: `low_latency`, `balanced` (default), or `throughput` | |
| - `REASONING`: reasoning mode, accepts `True`, `False`, or `auto` and maps to llama.cpp `--reasoning` | |
| - `LANGSEARCH_API_KEY`: optional API key for LangSearch web search tool (free tier: 1 req/sec, 60/min, 1000/day) | |
| - `ENABLE_TOOLS`: set to `1` to enable llama.cpp shell tool support when `LANGSEARCH_API_KEY` is present, default `0` | |
| - `HTTP_THREADS`: HTTP server worker threads, default `1` for personal single-request inference | |
| - `LOG_VERBOSITY`: llama.cpp log verbosity, default `1` to reduce runtime logging overhead | |
| - `MMAP`: set to `1` to use memory-mapped model loading, default `0` | |
| - `FLASH_ATTN`: set to `0` to disable flash attention if a model hangs or fails during startup, default `1` | |
| - `NO_WARMUP`: set to `1` to skip model warmup and save startup time | |
| - `PORT`: listen port, default `7860` | |
| ## Endpoints | |
| The server exposes: | |
| - `/v1/chat/completions` | |
| - `/v1/completions` | |
| - `/v1/models` | |
| Streaming is supported by `llama-server`. Bearer token authentication uses `--api-key` directly. | |
| ## Build and runtime | |
| The Docker image uses a multi-stage build on Debian bookworm. The builder stage installs `git`, `build-essential`, `cmake`, and OpenBLAS, clones `llama.cpp`, and compiles `llama-server` with `-Ofast`, `-march=native`, `-flto`, OpenBLAS, and native CPU optimizations. The runtime stage keeps only Python, OpenBLAS/OpenMP runtime libraries, the compiled server, and the downloader. | |
| Model downloads and HF cache live on the `/data` bucket so restarts do not redownload the model. | |
| ## Performance Optimization | |
| For small 2B models on CPU-only inference (~2-4 t/s baseline), optimize these settings: | |
| **High Impact:** | |
| - The server forces `--parallel 1`, so only one request runs at a time and all CPU is focused on the active query | |
| - Runtime uses `--no-mmap`, `--flash-attn on`, `--threads-http 1`, and low log verbosity to favor single-request inference over serving overhead | |
| - OpenBLAS is enabled because benchmark results on the Space showed BLAS-off hurt prompt processing and did not recover generation speed | |
| - `BATCH_SIZE` & `UBATCH_SIZE`: Use **64-256** for faster prompt ingestion; tiny values like 8 cause many small prompt batches | |
| - `PERF_PROFILE`: Use **`balanced`** for normal personal use; `throughput` only changes prompt batch sizing now | |
| **Medium Impact:** | |
| - `NO_WARMUP=1`: Skip model warmup to reduce startup latency by ~0.2s | |
| - `CACHE_TYPE_K`, `CACHE_TYPE_V`: Try `q4_0` instead of `f16` to reduce KV cache bandwidth and memory use at longer contexts | |
| - `ENABLE_TOOLS`: Leave as `0` unless you need tool calling. Built-in shell tools add prompt/tool overhead and should not be exposed publicly. | |
| - `MODEL_FILE`: If the repo has multiple GGUFs, test a lighter quant such as `Q4_0`, `Q4_K_S`, or `Q3_K_M` for speed/quality tradeoffs. | |
| - Compatibility: if a GGUF hangs during startup, try `FLASH_ATTN=0`, `CACHE_TYPE_K=f16`, `CACHE_TYPE_V=f16`, and then `MMAP=1`. | |
| **Example for 2B model (CPU-only):** | |
| ``` | |
| CTX_SIZE=2048 | |
| BATCH_SIZE=128 | |
| UBATCH_SIZE=128 | |
| PERF_PROFILE=balanced | |
| REASONING=False | |
| ENABLE_TOOLS=0 | |
| CACHE_TYPE_K=q4_0 | |
| CACHE_TYPE_V=q4_0 | |
| ``` | |
| Expected: faster first-token latency from prompt ingestion. Generation speed is still CPU-bound on the free tier, so a 2B Q4 model will usually remain in the low single-digit tokens/sec range. | |
| ## Web Search Tool (Standalone) | |
| When `ENABLE_TOOLS=1` and `LANGSEARCH_API_KEY` is set, `start.sh` enables llama.cpp's shell tool so the model can launch the local LangSearch wrapper. | |
| llama.cpp does not expose arbitrary external HTTP APIs as first-class built-in tools, so LangSearch is still implemented as a local wrapper that runs through the shell tool. | |
| **Use cases:** | |
| - Call from your application layer to augment model responses | |
| - Pre-process queries before sending to model | |
| - Post-process to fetch real-time data for specific topics | |
| **API Usage (from application):** | |
| ```python | |
| import requests | |
| import json | |
| response = requests.post( | |
| "http://localhost:7860/v1/chat/completions", | |
| headers={"Authorization": "Bearer YOUR_API_KEY"}, | |
| json={ | |
| "model": "model", | |
| "messages": [{"role": "user", "content": "What's the latest news?"}], | |
| "temperature": 0.7 | |
| } | |
| ) | |
| latest_response = response.json()['choices'][0]['message']['content'] | |
| ``` | |
| Then call the search tool from your backend or from the shell tool: | |
| ```bash | |
| python3 /app/search_tool.py "latest AI news" 5 true noLimit | |
| ``` | |
| Combine results before returning to user. | |
| **Command-line (Docker container):** | |
| Examples: | |
| ```bash | |
| python3 /app/search_tool.py "latest AI news" 5 true noLimit | |
| python3 /app/search_tool.py "Python 3.13 release" 3 false oneWeek | |
| ``` | |
| Parameters: | |
| - `query`: search string (required) | |
| - `count`: max results 1-10 (default: 5) | |
| - `summary`: include summaries (default: true) | |
| - `freshness`: `oneDay`, `oneWeek`, `oneMonth`, `oneYear`, `noLimit` (default: `noLimit`) | |
| Output is JSON with search results, URLs, snippets, and optional summaries. | |
| ## curl examples | |
| ```bash | |
| curl http://localhost:7860/v1/models \ | |
| -H "Authorization: Bearer $API_PASSWORD" | |
| ``` | |
| ```bash | |
| curl http://localhost:7860/v1/chat/completions \ | |
| -H "Authorization: Bearer $API_PASSWORD" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "model": "model", | |
| "messages": [ | |
| {"role": "user", "content": "Write a one-sentence summary of llama.cpp."} | |
| ], | |
| "stream": false | |
| }' | |
| ``` | |
| ```bash | |
| curl http://localhost:7860/v1/completions \ | |
| -H "Authorization: Bearer $API_PASSWORD" \ | |
| -H "Content-Type: application/json" \ | |
| -d '{ | |
| "model": "model", | |
| "prompt": "Explain KV cache in one paragraph.", | |
| "stream": false | |
| }' | |
| ``` | |
| ## OpenAI SDK example | |
| ```python | |
| import os | |
| from openai import OpenAI | |
| client = OpenAI( | |
| base_url="http://localhost:7860/v1", | |
| api_key=os.environ["API_PASSWORD"], | |
| ) | |
| response = client.chat.completions.create( | |
| model="model", | |
| messages=[{"role": "user", "content": "Hello"}], | |
| ) | |
| print(response.choices[0].message.content) | |
| ``` | |
| ## Hugging Face Spaces notes | |
| This repository is ready for a Docker Space with no frontend. The only required changes at deployment time are the three environment variables above. | |