mjpsm's picture
Deploy Progression API (llama-cpp-python)
63d4e3a verified
|
Raw History Blame Contribute Delete
7.67 kB
metadata
title: MyVillage Activity Progression API
emoji: 🧭
colorFrom: indigo
colorTo: blue
sdk: docker
app_port: 7860
pinned: false

MyVillage Activity Progression API

FastAPI inference service that runs the Progression GGUF model in-process with llama-cpp-python.

Uvicorn
    ↓
FastAPI
    ↓
llama-cpp-python
    ↓
Q8_0 GGUF

llama-server is no longer used. There is no subprocess, no internal inference port, and no second server to start. Uvicorn is the only process.

Model

Hugging Face repo mjpsm/activity-progression-model-600-qwen3.5-0.8b-gguf
GGUF file activity-progression-model-600-qwen3.5-0.8b-Q8_0.gguf
Quantization Q8_0
Runtime llama-cpp-python

On startup the API downloads the GGUF with huggingface_hub.hf_hub_download() (cached after the first run) and loads it into memory once. Requests are serialized through a single inference lock, and the blocking generation call runs in a worker thread so the event loop stays responsive.

Local installation (macOS / Linux)

python3 -m venv .venv
source .venv/bin/activate

pip install -r requirements.txt

Apple Silicon notes

llama-cpp-python compiles llama.cpp during pip install, so you need the Xcode Command Line Tools (xcode-select --install). cmake is fetched automatically by the build.

  • Use a native arm64 Python (python3 -c "import platform; print(platform.machine())" should print arm64). An x86_64/Rosetta Python produces a much slower build.

  • The default macOS build already includes Metal support. The API still runs on CPU by default (LLAMA_GPU_LAYERS=0). To use Metal:

    LLAMA_GPU_LAYERS=-1 python -m uvicorn app.main:app --host 0.0.0.0 --port 8000
    
  • If your install was built without Metal and you want it, rebuild:

    CMAKE_ARGS="-DGGML_METAL=on" pip install --force-reinstall --no-cache-dir llama-cpp-python
    

Hugging Face authentication

The model repo is currently public, so no authentication is needed. If it is made private again, either log in once:

hf auth login

or export a token with read access to the repo:

export HF_TOKEN=hf_...

No token is stored in this repository. Do not commit .env.

Run

python -m uvicorn app.main:app --host 0.0.0.0 --port 8000

That is the only command. Check readiness:

curl -s http://localhost:8000/health | python3 -m json.tool
{
  "status": "ok",
  "model_ready": true
}

If the model failed to download or load, the API still starts, logs the error, and reports "status": "degraded", "model_ready": false; /v1/progression/generate then returns 503.

Local curl test

curl -s -X POST   "http://localhost:8000/v1/progression/generate"   -H "Content-Type: application/json"   -d '{
    "version": "v1",
    "knowledge_submission": "I created the video and showed the finished result first. It received 1,800 views and more engagement than my previous videos. I noticed viewers stayed longer when the final result appeared before the process.",
    "wisdom": {
      "content": "When I presented in class, showing the most interesting result before explaining the process got peoples attention.",
      "chapterTitle": "Lead With the Result",
      "BookTypeName": "Testimony",
      "BookType": "Personal Record"
    },
    "village_goal": "Learn how to create engaging social media content",
    "previous_activity_title": "Test a stronger opening hook"
  }' | python3 -m json.tool

The same request is in scripts/local_curl.sh.

Routes

  • GET /
  • GET /health
  • GET /models
  • GET /docs
  • POST /v1/progression/generate

Request schema

{
  "version": "v1",
  "knowledge_submission": "string",
  "wisdom": {
    "content": "string",
    "chapterTitle": "string",
    "BookTypeName": "string",
    "BookType": "string"
  },
  "village_goal": "string",
  "previous_activity_title": "string"
}
  • The wisdom field names and capitalization are exact.
  • Unknown fields are rejected with 422.
  • An unknown version returns 404.

Generation parameters are internal

Generation settings are not part of the public API; a request containing a generation field is rejected. They are fixed in app/config.py:

Setting Value
max_new_tokens 450
temperature 0.0
top_p 1.0
repeat_penalty 1.08
seed 42

Response schema

{
  "progression": {
    "knowledge_state": "COMPLETED | PARTIAL | UNCLEAR | BLOCKED | ADVANCED",
    "demonstrated_state": ["string"],
    "next_gap": "string",
    "activity_plan": {
      "action": "string",
      "target": "string",
      "avoid_repeating": ["string"]
    }
  },
  "usage": {
    "input_tokens": 312,
    "output_tokens": 147
  }
}

usage is counted with the loaded model's own tokenizer:

  • input_tokens — tokens in the final raw SFT prompt sent to the model
  • output_tokens — tokens in the generated completion

Model metadata is not returned here; use GET /models.

If the model output contains no parseable JSON object, or the JSON does not match the Progression schema, the API returns 502 with the raw output for debugging. Malformed output is never silently repaired.

Prompt format

The model was fine-tuned on a raw completion prompt, so the API uses raw completion (never a chat template or chat-completion API).

previous_activity_title is converted to the structure used for SFT:

{
  "village_goal": "...",
  "previous_activity": {
    "title": "..."
  },
  "knowledge_submission": "...",
  "wisdom": {
    "content": "...",
    "chapterTitle": "...",
    "BookTypeName": "...",
    "BookType": "..."
  }
}

and serialized as compact JSON into:

<SYSTEM_INSTRUCTION>

INPUT:
<compact JSON>

OUTPUT:

Do not change this format unless the model is retrained and re-evaluated with the new one.

Configuration

Optional environment variables (see .env.example):

Variable Default Purpose
HF_TOKEN unset Read token, only needed if the model repo is private
LLAMA_CONTEXT_SIZE 2048 Context window
LLAMA_THREADS 2 CPU threads for generation
LLAMA_GPU_LAYERS 0 Layers to offload (-1 = all)

Tests

pip install -r requirements-dev.txt
python -m pytest

The tests use a fake runtime and do not download or load the model.

Docker / Hugging Face Space deployment

The image contains only Uvicorn + FastAPI + llama-cpp-python and listens on port 7860.

Local Docker run:

docker build -t progression-api .
docker run --rm -p 7860:7860 progression-api

Space setup

  1. Create a Docker Space.

  2. Only if the model repo is private, add this secret under Space → Settings → Variables and secrets → New secret:

    HF_TOKEN=<Hugging Face token with read access to the private model repo>
    
  3. Push this repo to the Space:

    git clone https://huggingface.co/spaces/mjpsm/YOUR-SPACE-NAME
    cd YOUR-SPACE-NAME
    # copy this repo's files in, then:
    git add .
    git commit -m "Deploy Progression API (llama-cpp-python)"
    git push
    

Hugging Face builds the Docker image and exposes port 7860.

curl -s "https://YOUR-SPACE-HOST/health" | python3 -m json.tool

CPU behavior

One model instance is loaded and generation is serialized through one inference lock, so concurrent requests queue rather than compete for memory/compute on a small Space. Benchmark real Progression requests before raising LLAMA_THREADS.