mjpsm's picture
Deploy MyVillage Activity Instruction API (llama-cpp-python)
b832e97 verified
|
Raw History Blame Contribute Delete
6.63 kB
---
title: MyVillage Activity Instruction API
emoji: 🧭
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
models:
- mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf
---
# MyVillage Activity Instruction API β€” llama-cpp-python
A FastAPI Docker Space that loads the **Q4_K_M GGUF** directly via `llama_cpp.Llama`. **There is no internal `llama-server` process.** Only Uvicorn serves port 7860.
- Merged model: `mjpsm/activity-instruction-model-500-qwen3.5-2b`
- Evaluated merged revision: `9ee80b07eb127e22c2008ea95fa75c43230b5c0e`
- Public GGUF repo: [`mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf`](https://huggingface.co/mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf) (no token needed)
- Default GGUF filename: `activity-instruction-model-500-qwen3.5-2b-Q4_K_M.gguf` (1.27 GB); `...-Q8_0.gguf` (2.01 GB) is also in the repo
- Inference: 4096 context, 192 tokens maximum, temperature 0, repetition penalty 1.0, seed 42, thinking disabled
## Run locally with Uvicorn
Requires Python 3.11+. On macOS you also need CMake and the Xcode command line tools (`brew install cmake`, `xcode-select --install`), because `llama-cpp-python` is compiled from source.
```bash
python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
# macOS (Apple Silicon): build llama-cpp-python from source with Metal. Takes several minutes.
CMAKE_ARGS="-DGGML_METAL=on" pip install --no-cache-dir --no-binary llama-cpp-python -r requirements-dev.txt
# Linux x86_64: use the prebuilt CPU wheel instead, as the Dockerfile does.
# pip install --prefer-binary --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu -r requirements-dev.txt
```
Start the API:
```bash
uvicorn main:app --host 127.0.0.1 --port 7860 --workers 1
```
The first start downloads the 1.27 GB Q4_K_M GGUF into the Hugging Face cache (`~/.cache/huggingface`); later starts load it from disk. The API is ready when `GET /health` returns 200. Then, in a second terminal:
```bash
source .venv/bin/activate
python test_api.py # hits /health, /model and /generate
# or open http://127.0.0.1:7860/docs
```
Keep `--workers 1`: each worker loads its own copy of the model. Verified on macOS arm64 with Python 3.11 and `llama-cpp-python` 0.3.36; `/generate` takes roughly 15–20 seconds for the example request with the default `N_THREADS=2`.
## Endpoints
- `GET /` β€” service metadata
- `GET /health` β€” 200 once the model is loaded, otherwise 503
- `GET /model` β€” source, GGUF and inference settings
- `POST /generate` β€” generate validated 1–5 step instructions
- `GET /docs` β€” interactive Swagger UI
- `POST /predict` β€” backwards-compatible alias
## Input
Send the **actual Progression output object** and **Composition `output` object**. Do not include `success_evidence`.
```json
{
"progression": {
"knowledge_state": "PARTIAL",
"demonstrated_state": ["Created the base animation", "Imported the finalized imagery"],
"next_gap": "Complete the remaining animation and composition work.",
"activity_plan": {
"action": "complete the remaining animation work",
"target": "the year-end review video",
"avoid_repeating": ["creating the base animation", "reimporting finalized imagery"]
}
},
"composition": {
"activity_title": "Complete the Animation",
"activity_description": "A continuation of the remaining animation and composition work so the year-end review is ready as a complete video."
}
}
```
## Output
Example shape (generated wording may vary):
```json
{"output":{"instructions":["Complete the remaining animation work for the year-end review video.","Finish the remaining composition work using the existing imagery."]}}
```
## Space settings
1. Create a Docker Space named `mjpsm/activity-instruction-api`.
2. Upload **all root-level files** from this package to the Space root. Do not upload the ZIP as the app.
3. The GGUF repository is public, so no `HF_TOKEN` is needed. If you set `API_KEY` as a secret, clients must supply `X-API-Key` for `/generate`.
4. The default `MODEL_FILENAME` (Q4_K_M) is present in the repo, so no variable is required. To use the larger quantization, set `MODEL_FILENAME=activity-instruction-model-500-qwen3.5-2b-Q8_0.gguf`.
5. Start the Space; the GGUF downloads and loads at startup. The Space is ready when `/health` returns 200.
6. Run `test_api.py` or try `POST /generate` in `/docs`.
Expected URL after Space creation: `https://mjpsm-activity-instruction-api.hf.space/docs`.
## Configuration
| Variable | Default | Description |
| --- | --- | --- |
| `HF_TOKEN` | unset | Not needed while the GGUF repository is public; only set it if the repo is made private again |
| `MODEL_REPO` | `mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf` | Model repository |
| `MODEL_FILENAME` | `activity-instruction-model-500-qwen3.5-2b-Q4_K_M.gguf` | Exact quantized filename |
| `MODEL_REVISION` | unset | Optional GGUF commit hash pin |
| `N_CTX` | `4096` | Context size |
| `N_THREADS` | `2` | CPU threads |
| `MAX_NEW_TOKENS` | `192` | Maximum output tokens |
| `REPEAT_PENALTY` | `1.0` | Repetition penalty |
| `SEED` | `42` | Deterministic seed |
| `QUEUE_TIMEOUT` | `90` | Wait for concurrent inference (single model lock) |
| `API_KEY` | unset | Optional inbound authentication; strongly recommended on public Spaces |
| `ALLOWED_ORIGINS` | `*` | Optional CORS restriction |
### Why the prompt differs from Composition
The Composition Model was evaluated with a plain `INPUT:` / `OUTPUT:` format. The Instruction Model was evaluated as a **system + user chat**, with `enable_thinking=False`. `llama-cpp-python`'s raw `create_completion` is used for simplicity, but its prompt contains the Qwen3.5 chat role markers and an **empty `<think></think>` prefix** to represent disabled thinking. The system prompt is copied exactly from the final evaluation. This requires live comparison against the saved GGUF regression outputs because it uses a different inference binding and prompt renderer than the original `llama-server` benchmark.
The API **rejects** malformed model JSON with HTTP 502; it never silently substitutes a generic fallback activity or forwards invalid output to the Activity Generation wrapper. Test important diagnostic examples before deploying beyond a reviewed beta.
## Local Docker
```bash
docker build -t myvillage-instruction .
docker run --rm -p 7860:7860 myvillage-instruction
```
If you add an `API_KEY`, do not commit it into source control. For Space deployment, use Settings β†’ Variables and secrets instead of placing secrets in code.