mjpsm's picture
Deploy Progression API (llama-cpp-python)
63d4e3a verified
|
Raw History Blame Contribute Delete
7.67 kB
---
title: MyVillage Activity Progression API
emoji: 🧭
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false
---
# MyVillage Activity Progression API
FastAPI inference service that runs the Progression GGUF model **in-process**
with `llama-cpp-python`.
```text
Uvicorn
↓
FastAPI
↓
llama-cpp-python
↓
Q8_0 GGUF
```
`llama-server` is **no longer used**. There is no subprocess, no internal
inference port, and no second server to start. Uvicorn is the only process.
## Model
| | |
|---|---|
| Hugging Face repo | `mjpsm/activity-progression-model-600-qwen3.5-0.8b-gguf` |
| GGUF file | `activity-progression-model-600-qwen3.5-0.8b-Q8_0.gguf` |
| Quantization | `Q8_0` |
| Runtime | `llama-cpp-python` |
On startup the API downloads the GGUF with `huggingface_hub.hf_hub_download()`
(cached after the first run) and loads it into memory once. Requests are
serialized through a single inference lock, and the blocking generation call
runs in a worker thread so the event loop stays responsive.
## Local installation (macOS / Linux)
```bash
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```
### Apple Silicon notes
`llama-cpp-python` compiles llama.cpp during `pip install`, so you need the
Xcode Command Line Tools (`xcode-select --install`). `cmake` is fetched
automatically by the build.
- Use a native **arm64** Python (`python3 -c "import platform; print(platform.machine())"`
should print `arm64`). An x86_64/Rosetta Python produces a much slower build.
- The default macOS build already includes Metal support. The API still runs
on **CPU by default** (`LLAMA_GPU_LAYERS=0`). To use Metal:
```bash
LLAMA_GPU_LAYERS=-1 python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
```
- If your install was built without Metal and you want it, rebuild:
```bash
CMAKE_ARGS="-DGGML_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python
```
## Hugging Face authentication
The model repo is currently public, so no authentication is needed. If it is
made private again, either log in once:
```bash
hf auth login
```
or export a token with read access to the repo:
```bash
export HF_TOKEN=hf_...
```
No token is stored in this repository. Do not commit `.env`.
## Run
```bash
python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
```
That is the only command. Check readiness:
```bash
curl -s http://localhost:8000/health | python3 -m json.tool
```
```json
{
"status": "ok",
"model_ready": true
}
```
If the model failed to download or load, the API still starts, logs the
error, and reports `"status": "degraded", "model_ready": false`;
`/v1/progression/generate` then returns `503`.
## Local curl test
```bash
curl -s -X POST "http://localhost:8000/v1/progression/generate" -H "Content-Type: application/json" -d '{
"version": "v1",
"knowledge_submission": "I created the video and showed the finished result first. It received 1,800 views and more engagement than my previous videos. I noticed viewers stayed longer when the final result appeared before the process.",
"wisdom": {
"content": "When I presented in class, showing the most interesting result before explaining the process got peoples attention.",
"chapterTitle": "Lead With the Result",
"BookTypeName": "Testimony",
"BookType": "Personal Record"
},
"village_goal": "Learn how to create engaging social media content",
"previous_activity_title": "Test a stronger opening hook"
}' | python3 -m json.tool
```
The same request is in `scripts/local_curl.sh`.
## Routes
- `GET /`
- `GET /health`
- `GET /models`
- `GET /docs`
- `POST /v1/progression/generate`
## Request schema
```json
{
"version": "v1",
"knowledge_submission": "string",
"wisdom": {
"content": "string",
"chapterTitle": "string",
"BookTypeName": "string",
"BookType": "string"
},
"village_goal": "string",
"previous_activity_title": "string"
}
```
- The `wisdom` field names and capitalization are exact.
- Unknown fields are rejected with `422`.
- An unknown `version` returns `404`.
### Generation parameters are internal
Generation settings are **not** part of the public API; a request containing
a `generation` field is rejected. They are fixed in `app/config.py`:
| Setting | Value |
|---|---|
| `max_new_tokens` | `450` |
| `temperature` | `0.0` |
| `top_p` | `1.0` |
| `repeat_penalty` | `1.08` |
| `seed` | `42` |
## Response schema
```json
{
"progression": {
"knowledge_state": "COMPLETED | PARTIAL | UNCLEAR | BLOCKED | ADVANCED",
"demonstrated_state": ["string"],
"next_gap": "string",
"activity_plan": {
"action": "string",
"target": "string",
"avoid_repeating": ["string"]
}
},
"usage": {
"input_tokens": 312,
"output_tokens": 147
}
}
```
`usage` is counted with the loaded model's own tokenizer:
- `input_tokens` — tokens in the final raw SFT prompt sent to the model
- `output_tokens` — tokens in the generated completion
Model metadata is not returned here; use `GET /models`.
If the model output contains no parseable JSON object, or the JSON does not
match the Progression schema, the API returns `502` with the raw output for
debugging. Malformed output is never silently repaired.
## Prompt format
The model was fine-tuned on a raw completion prompt, so the API uses raw
completion (never a chat template or chat-completion API).
`previous_activity_title` is converted to the structure used for SFT:
```json
{
"village_goal": "...",
"previous_activity": {
"title": "..."
},
"knowledge_submission": "...",
"wisdom": {
"content": "...",
"chapterTitle": "...",
"BookTypeName": "...",
"BookType": "..."
}
}
```
and serialized as compact JSON into:
```text
<SYSTEM_INSTRUCTION>
INPUT:
<compact JSON>
OUTPUT:
```
Do not change this format unless the model is retrained and re-evaluated
with the new one.
## Configuration
Optional environment variables (see `.env.example`):
| Variable | Default | Purpose |
|---|---|---|
| `HF_TOKEN` | unset | Read token, only needed if the model repo is private |
| `LLAMA_CONTEXT_SIZE` | `2048` | Context window |
| `LLAMA_THREADS` | `2` | CPU threads for generation |
| `LLAMA_GPU_LAYERS` | `0` | Layers to offload (`-1` = all) |
## Tests
```bash
pip install -r requirements-dev.txt
python -m pytest
```
The tests use a fake runtime and do not download or load the model.
## Docker / Hugging Face Space deployment
The image contains only Uvicorn + FastAPI + `llama-cpp-python` and listens
on port `7860`.
Local Docker run:
```bash
docker build -t progression-api .
docker run --rm -p 7860:7860 progression-api
```
### Space setup
1. Create a **Docker Space**.
2. Only if the model repo is private, add this secret under
**Space → Settings → Variables and secrets → New secret**:
```text
HF_TOKEN=<Hugging Face token with read access to the private model repo>
```
3. Push this repo to the Space:
```bash
git clone https://huggingface.co/spaces/mjpsm/YOUR-SPACE-NAME
cd YOUR-SPACE-NAME
# copy this repo's files in, then:
git add .
git commit -m "Deploy Progression API (llama-cpp-python)"
git push
```
Hugging Face builds the Docker image and exposes port `7860`.
```bash
curl -s "https://YOUR-SPACE-HOST/health" | python3 -m json.tool
```
## CPU behavior
One model instance is loaded and generation is serialized through one
inference lock, so concurrent requests queue rather than compete for
memory/compute on a small Space. Benchmark real Progression requests before
raising `LLAMA_THREADS`.