Blame for README.md

---
title: OpenJev · Zero-GPU CPU Deploy Kit
emoji: ⚖️
colorFrom: green
colorTo: gray
sdk: static
pinned: true
license: apache-2.0
short_description: Zero-GPU CPU deploy kit for the OpenJev decision API
---

# OpenJev — zero-GPU CPU deploy kit

Runs the [OpenJev](https://huggingface.co/openjev/openjev) decision API
([openjev-server](https://huggingface.co/spaces/openjev/openjev-server))
**with no GPU at all**. It serves the exact same
`POST /v1/systemone` contract as the hosted server — a choice / yes-no / score
per question, with a probability for every option, read off the model's first
output position — by pointing the **official, unmodified** `openjev-server` at
[`llama.cpp`](https://github.com/ggml-org/llama.cpp) (CPU, GGUF) instead of vLLM.

| | hosted (reference) | this kit (zero GPU) |
|---|---|---|
| engine | vLLM + FP8 on 1× H100 | llama.cpp on CPU |
| weights | `openjev/openjev-FP8` (80 GB) | `openjev/openjev-GGUF` Q4_K_M (16.2 GB) |
| latency | ~125 ms | seconds (see [Benchmarks](#benchmarks)) |
| surface | `/v1/systemone` + ops | identical (official server, unchanged) |

> 👀 The live index page is at **[https://broadfield-openjev-cpu-deploy.hf.space](https://broadfield-openjev-cpu-deploy.hf.space)** — this README is the machine-readable mirror. Every file below is fetchable raw: `https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/<path>`.

## Why this works (no code hacks)

`openjev-server`'s `vllm` backend is really *"any OpenAI-compatible server that returns logprobs"*:

- it sends standard `/chat/completions` JSON (`logprobs`, `top_logprobs`, `allowed_token_ids`, `max_tokens: 1`, `temperature: 0`);
- **exact readout** uses vLLM's `logprob_token_ids = [ids]` extension → llama.cpp doesn't have it, so the backend's documented fallback kicks in: top-K logprobs are matched **by label text** instead (see `openjev_server/backends/vllm.py` → `_scores`, "old protocol").

That single fallback is the whole trick: the official code path runs unmodified, and `--force` only relaxes the startup probe's letter-mass check if llama.cpp's text-matched logprobs ever look borderline for your quant.

## What you get

- `POST /v1/systemone` — one request, any mix of `choice` / `noul` / `score`
- `POST /v1/prewarm`, `POST /v1/chat/completions` (passthrough)
- `GET /v1/version` · `GET /healthz` · `GET /readyz` · `GET /metrics` (Prometheus) · `GET /docs` (OpenAPI)
- optional bearer auth via `OPENJEV_TOKEN`
- Persistent model storage (HF Spaces) — the 16.2 GB download survives restarts

## Deploy on Hugging Face (zero GPU, free)

1. **Duplicate the runtime Space** (needs a free HF account): <https://huggingface.co/spaces/broadfield/openjev-cpu-run?duplicate=true>
   - SDK: **Docker** · hardware: **CPU basic (free)** · persistent storage: **30 GB** ($0.50/mo, required for `/data`)
2. Wait for the build. The Docker image compiles llama.cpp CPU + installs openjev-server and, on first boot, downloads the Q4 GGUF (~16.2 GB) into `/data`. **Allow ~20–50 min on the first cold boot;** subsequent restarts load straight from persistent storage.
3. Try the API. By default the Space is public:

```bash
BASE=https://<your-username>-openjev-cpu-run.hf.space

curl -s "$BASE/v1/systemone" -H 'Content-Type: application/json' -d '{
  "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
  "questions": {
    "route":   {"type": "choice", "instructions": "Which team should handle this?",
                "criteria": {"billing": null, "shipping": null, "technical": null}},
    "angry":   {"type": "noul",   "instructions": "Is the customer angry?"},
    "urgency": {"type": "score",  "instructions": "How urgent is this?",
                "criteria": ["can wait", "this week", "today", "right now"]}
  }
}'
```

```json
{
  "answers": {
    "route":   {"type": "choice", "choice": "billing",
                "probabilities": {"billing": 0.9996, "shipping": 0.0002, "technical": 0.0002},
                "confidence": 0.9995},
    "angry":   {"type": "noul", "noul": 0.56},
    "urgency": {"type": "score", "score": 2.04,
                "legend": {"0": "can wait", "1": "this week", "2": "today", "3": "right now"},
                "probabilities": {"0": 0.02, "1": 0.13, "2": 0.64, "3": 0.21},
                "confidence": 0.71}
  },
  "usage": {"input_tokens": 239, "output_tokens": 0}
}
```

> **Latency honesty right up front** — a 27B model on a shared free CPU is *not* 125 ms. Expect **~2–6 s per question** (single reader) on a 16-core Space with the Q4 quant. That's a real API — just not a high-QPS one. The reason this is still interesting: **a decision API only ever generates 1 token**, so the CPU is the *only* slow part, and it stays predictable. Raise `LLAMA_PARALLEL`, lower `LLAMA_CTX`, or use a different quant if your workload wants less of it.

## Deploy locally (or on any VPS)

```bash
git clone https://huggingface.co/datasets/broadfield/openjev-cpu-deploy-kit
cd openjev-cpu-deploy-kit
docker compose up --build -d
curl -s http://localhost:7860/v1/systemone ...   # same payload as above
```

Or Docker directly:

```bash
docker build -t openjev-cpu .
docker run -d -p 7860:7860 -v openjev-data:/data -e MODEL_FILE=OpenJev-Q4_K_M.gguf openjev-cpu
```

## Env vars

| var | default | what it does |
|---|---|---|
| `MODEL_REPO` | `openjev/openjev-GGUF` | GGUF repo |
| `MODEL_FILE` | `OpenJev-Q4_K_M.gguf` | Q4_K_M · Q5_K_M (18.8 GB) · Q6_K (21.6 GB) · Q8_0 (27.9 GB) |
| `TOKENIZER_REPO` | `openjev/openjev` | tokenizer + chat template for openjev-server |
| `LLAMA_CTX` | `8192` | context window |
| `LLAMA_MAX_LOGPROBS` | `64` | top-K logprobs (must be ≥ number of options) |
| `LLAMA_PARALLEL` | `1` | concurrent sequences (scales memory) |
| `LLAMA_THREADS` / `LLAMA_THREADS_BATCH` | auto | pin core count, e.g. `8` / `8` |
| `OPENJEV_PORT` | `7860` | ingress (HF sets `$PORT`) |
| `OPENJEV_TOKEN` | empty | bearer token required on `/v1/*` |
| `OPENJEV_PROFILE` | `openjev` | `openjev` (published constants) or `uncalibrated` |
| `OPENJEV_SERVED_MODEL_NAME` | `openjev` | model id reported by `/v1/version` |

## Benchmarks

Expected on a **free 16 vCPU CPU Space**, `OpenJev-Q4_K_M.gguf` (27B, Qwen3-based → ~8-bit KV cache in the llama.cpp fork, 16.2 GB RAM):

| quant | file size | SPE (tokens/s, 8 threads) | RAM | 1-question latency* |
|---|---|---|---|---|
| Q4_K_M | 16.2 GB | 2–4 | ~17 GB | ~2–6 s |
| Q5_K_M | 18.8 GB | 1.5–3 | ~20 GB | ~3–8 s |
| Q6_K | 21.6 GB | 1.5–3 | ~23 GB | ~3–8 s |

\* single question ≈ one forward pass: `(prompt_tokens + 1) / SPE`. A 240-token state ⇒ ~4 s @ 3 SPE. Latency is directly linear in prompt length.

**The decision-API twist:** because `max_tokens: 1`, a *pipeline* of many questions on one state shares ~all of the prefill. For batch agents, batch the questions into one `/v1/systemone` call — every extra answer after the first is nearly free:

| questions in one call | est. per-call latency @ 3 SPE |
|---|---|
| 1 | ~4 s |
| 4 | ~4.5 s |
| 10 | ~5.5 s (10 answers, still one pass) |

## Calling it from code

```python
import httpx

r = httpx.post("http://localhost:7860/v1/systemone", json={
    "state": "Customer message: I was charged twice for my order last week and nobody has replied.",
    "questions": {
        "route": {"type": "choice", "instructions": "Which team should handle this?",
                  "criteria": {"billing": None, "shipping": None, "technical": None}},
        "angry": {"type": "noul", "instructions": "Is the customer angry?"},
    },
}, timeout=60)
answers = r.json()["answers"]
print(answers["route"]["choice"], answers["route"]["probabilities"], answers["angry"]["noul"])
```

```typescript
const r = await fetch(`${BASE}/v1/systemone`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({
    state: "Customer message: I was charged twice for my order last week and nobody has replied.",
    questions: {
      route: { type: "choice", instructions: "Which team should handle this?",
               criteria: { billing: null, shipping: null, technical: null } },
      angry: { type: "noul", instructions: "Is the customer angry?" },
    },
  }),
});
const { answers } = await r.json();
```

## Files in this kit (raw links)

| path | role | raw |
|---|---|---|
| `index.html` | live landing page with all instructions | [raw](https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/index.html) |
| `Dockerfile` | multi-stage: llama.cpp CPU + official openjev-server | [raw](https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/Dockerfile) |
| `entrypoint.sh` | GGUF → `/data`, llama-server, then `openjev serve` | [raw](https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/entrypoint.sh) |
| `docker-compose.yml` | local/VPS one-liner | [raw](https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/docker-compose.yml) |
| `bench.mjs` | latency/throughput smoke bench | [raw](https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/bench.mjs) |
| `runtime-baseline.md` | the exact runtime Space spec | [raw](https://huggingface.co/spaces/broadfield/openjev-cpu-deploy/raw/main/runtime-baseline.md) |

Everything is also mirrored (cloneable) in the dataset repo [`broadfield/openjev-cpu-deploy-kit`](https://huggingface.co/datasets/broadfield/openjev-cpu-deploy-kit).

## License & credit

- Model weights: **CC BY-NC 4.0** — OpenJev is an independent project, not affiliated with TypeSafe (Jev is their product).
- `openjev-server`: Apache 2.0.
- This kit: Apache 2.0. llama.cpp: MIT.