mjpsm's picture
Deploy MyVillage Activity Instruction API (llama-cpp-python)
b832e97 verified
|
Raw History Blame Contribute Delete
6.63 kB
metadata
title: MyVillage Activity Instruction API
emoji: 🧭
colorFrom: blue
colorTo: indigo
sdk: docker
app_port: 7860
models:
  - mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf

MyVillage Activity Instruction API β€” llama-cpp-python

A FastAPI Docker Space that loads the Q4_K_M GGUF directly via llama_cpp.Llama. There is no internal llama-server process. Only Uvicorn serves port 7860.

  • Merged model: mjpsm/activity-instruction-model-500-qwen3.5-2b
  • Evaluated merged revision: 9ee80b07eb127e22c2008ea95fa75c43230b5c0e
  • Public GGUF repo: mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf (no token needed)
  • Default GGUF filename: activity-instruction-model-500-qwen3.5-2b-Q4_K_M.gguf (1.27 GB); ...-Q8_0.gguf (2.01 GB) is also in the repo
  • Inference: 4096 context, 192 tokens maximum, temperature 0, repetition penalty 1.0, seed 42, thinking disabled

Run locally with Uvicorn

Requires Python 3.11+. On macOS you also need CMake and the Xcode command line tools (brew install cmake, xcode-select --install), because llama-cpp-python is compiled from source.

python3.11 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip

# macOS (Apple Silicon): build llama-cpp-python from source with Metal. Takes several minutes.
CMAKE_ARGS="-DGGML_METAL=on" pip install --no-cache-dir --no-binary llama-cpp-python -r requirements-dev.txt

# Linux x86_64: use the prebuilt CPU wheel instead, as the Dockerfile does.
# pip install --prefer-binary --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu -r requirements-dev.txt

Start the API:

uvicorn main:app --host 127.0.0.1 --port 7860 --workers 1

The first start downloads the 1.27 GB Q4_K_M GGUF into the Hugging Face cache (~/.cache/huggingface); later starts load it from disk. The API is ready when GET /health returns 200. Then, in a second terminal:

source .venv/bin/activate
python test_api.py                      # hits /health, /model and /generate
# or open http://127.0.0.1:7860/docs

Keep --workers 1: each worker loads its own copy of the model. Verified on macOS arm64 with Python 3.11 and llama-cpp-python 0.3.36; /generate takes roughly 15–20 seconds for the example request with the default N_THREADS=2.

Endpoints

  • GET / β€” service metadata
  • GET /health β€” 200 once the model is loaded, otherwise 503
  • GET /model β€” source, GGUF and inference settings
  • POST /generate β€” generate validated 1–5 step instructions
  • GET /docs β€” interactive Swagger UI
  • POST /predict β€” backwards-compatible alias

Input

Send the actual Progression output object and Composition output object. Do not include success_evidence.

{
  "progression": {
    "knowledge_state": "PARTIAL",
    "demonstrated_state": ["Created the base animation", "Imported the finalized imagery"],
    "next_gap": "Complete the remaining animation and composition work.",
    "activity_plan": {
      "action": "complete the remaining animation work",
      "target": "the year-end review video",
      "avoid_repeating": ["creating the base animation", "reimporting finalized imagery"]
    }
  },
  "composition": {
    "activity_title": "Complete the Animation",
    "activity_description": "A continuation of the remaining animation and composition work so the year-end review is ready as a complete video."
  }
}

Output

Example shape (generated wording may vary):

{"output":{"instructions":["Complete the remaining animation work for the year-end review video.","Finish the remaining composition work using the existing imagery."]}}

Space settings

  1. Create a Docker Space named mjpsm/activity-instruction-api.
  2. Upload all root-level files from this package to the Space root. Do not upload the ZIP as the app.
  3. The GGUF repository is public, so no HF_TOKEN is needed. If you set API_KEY as a secret, clients must supply X-API-Key for /generate.
  4. The default MODEL_FILENAME (Q4_K_M) is present in the repo, so no variable is required. To use the larger quantization, set MODEL_FILENAME=activity-instruction-model-500-qwen3.5-2b-Q8_0.gguf.
  5. Start the Space; the GGUF downloads and loads at startup. The Space is ready when /health returns 200.
  6. Run test_api.py or try POST /generate in /docs.

Expected URL after Space creation: https://mjpsm-activity-instruction-api.hf.space/docs.

Configuration

Variable Default Description
HF_TOKEN unset Not needed while the GGUF repository is public; only set it if the repo is made private again
MODEL_REPO mjpsm/activity-instruction-model-500-qwen3.5-2b-gguf Model repository
MODEL_FILENAME activity-instruction-model-500-qwen3.5-2b-Q4_K_M.gguf Exact quantized filename
MODEL_REVISION unset Optional GGUF commit hash pin
N_CTX 4096 Context size
N_THREADS 2 CPU threads
MAX_NEW_TOKENS 192 Maximum output tokens
REPEAT_PENALTY 1.0 Repetition penalty
SEED 42 Deterministic seed
QUEUE_TIMEOUT 90 Wait for concurrent inference (single model lock)
API_KEY unset Optional inbound authentication; strongly recommended on public Spaces
ALLOWED_ORIGINS * Optional CORS restriction

Why the prompt differs from Composition

The Composition Model was evaluated with a plain INPUT: / OUTPUT: format. The Instruction Model was evaluated as a system + user chat, with enable_thinking=False. llama-cpp-python's raw create_completion is used for simplicity, but its prompt contains the Qwen3.5 chat role markers and an empty <think></think> prefix to represent disabled thinking. The system prompt is copied exactly from the final evaluation. This requires live comparison against the saved GGUF regression outputs because it uses a different inference binding and prompt renderer than the original llama-server benchmark.

The API rejects malformed model JSON with HTTP 502; it never silently substitutes a generic fallback activity or forwards invalid output to the Activity Generation wrapper. Test important diagnostic examples before deploying beyond a reviewed beta.

Local Docker

docker build -t myvillage-instruction .
docker run --rm -p 7860:7860 myvillage-instruction

If you add an API_KEY, do not commit it into source control. For Space deployment, use Settings β†’ Variables and secrets instead of placing secrets in code.