deployment/README.md

APUS TypeSafe SDK gateway with explicit vLLM low/high routing

This directory adds the official typesafe-sdk==0.7.0 request/response protocol to APUS Reflex. The local service is an APUS model, not the hosted Jev service. Its probabilities, calibration, quality, and billing do not inherit the hosted service's guarantees.

Two separately loaded vLLM engines serve the same trained checkpoint at different layer depths:

Effort	Engine	Execution
high (default)	Original merged 9B-5949	32 layers
low	Verified prefix export of that same checkpoint	16 layers, original final norm and LM head

This is explicit request routing between two engines. It is not automatic confidence-based early exit, shared weights between processes, hidden-state continuation, KV broadcast, or candidate-only LM-head execution. Both vLLM engines compute the full output head; the gateway extracts the requested candidate scores. A client-requested low never silently falls back to high.

1. Install and download the model

Use Linux, Python 3.12, CUDA-capable hardware, and vLLM 0.29.0. The 9B deployment work targets one RTX PRO 6000 Blackwell 96 GB; other hardware needs its own memory and numerical checks. Do not run the reference native runtime's Transformers 5.16.1 requirements in this vLLM environment.

python3.12 -m venv .venv
source .venv/bin/activate
pip install vllm==0.29.0 transformers==5.17.0 huggingface-hub==1.32.0 ninja==1.13.2
pip install fastapi==0.128.8 uvicorn==0.40.0 httpx==0.28.1 httpx2==2.13.0 typesafe-sdk==0.7.0

On Blackwell, configure the compatible CUDA 13 compiler in the same environment:

pip install nvidia-cuda-nvcc==13.4.92 nvidia-cuda-crt==13.4.92 nvidia-cuda-cccl==13.3.4.3.1
export CUDA_HOME="$(python -c 'import sysconfig; print(sysconfig.get_path("purelib") + "/nvidia/cu13")')"
export PATH="$CUDA_HOME/bin:$PATH"
"$CUDA_HOME/bin/nvcc" --version

The launch script disables the FlashInfer sampler to match the tested configuration. After downloading this directory, pip install -r requirements.txt installs the pinned deployment dependencies. requirements-test.txt supports the CPU gateway tests without installing a GPU runtime.

Resolve the repository revision once, then download by that immutable revision. The repository is a family bundle: use the model subdirectory, not its root.

export APUS_REVISION="$(python -c 'from huggingface_hub import HfApi; print(HfApi().model_info("apus-ailab/APUS-Reflex").sha)')"
hf download apus-ailab/APUS-Reflex --revision "$APUS_REVISION" \
  --include '9B-5949/*' 'deployment/*' --local-dir ./APUS-Reflex
cd ./APUS-Reflex/deployment
export APUS_FULL_MODEL="$(cd ../9B-5949 && pwd)"

If you already downloaded the deployment code separately, the included model-only helper performs the same immutable-revision resolution and prints the resulting model path:

python download_model.py --local-dir ../models --revision "$APUS_REVISION"
# With this layout, set APUS_FULL_MODEL to ../models/9B-5949.

Create a new separate directory for the trained 16-layer prefix. The exporter refuses an existing destination and verifies retained tensors; it never modifies the original weights.

export APUS_LOW_MODEL="$(dirname "$APUS_FULL_MODEL")/9B-5949-low16"
python export_low.py --source "$APUS_FULL_MODEL" --output "$APUS_LOW_MODEL"

Start the full engine in terminal A, using an otherwise available GPU:

export JEV_VLLM_PYTHON="$VIRTUAL_ENV/bin/python"
CUDA_VISIBLE_DEVICES=0 JEV_EFFORT=high \
  JEV_MODEL_PATH="$APUS_FULL_MODEL" JEV_ENGINE_PORT=30002 \
  JEV_GPU_MEMORY_UTILIZATION=0.70 JEV_MAX_NUM_SEQS=8 \
  bash run_backend.sh

In terminal B, activate the same environment, set the same CUDA_HOME and APUS_LOW_MODEL paths, and start the shallow engine:

export JEV_VLLM_PYTHON="$VIRTUAL_ENV/bin/python"
CUDA_VISIBLE_DEVICES=0 JEV_EFFORT=low \
  JEV_MODEL_PATH="$APUS_LOW_MODEL" JEV_ENGINE_PORT=30004 \
  JEV_GPU_MEMORY_UTILIZATION=0.22 JEV_MAX_NUM_SEQS=8 \
  bash run_backend.sh

These defaults allocate separate engine budgets of 70% and 22% of the same 96 GB GPU. They are not appropriate if another workload already occupies that GPU. Compilation can take several minutes. The launcher verifies metadata and weight hashes before starting; this adds sequential disk reads. In terminal C, check both engines:

curl -f http://127.0.0.1:30002/health
curl -f http://127.0.0.1:30004/health

Do not proceed until both are ready. Each foreground process can be stopped with Ctrl-C in its own terminal. Never reuse a port or GPU allocation belonging to another workload. The two engines must belong to the same source checkpoint; archive the export provenance and source revision together. The package uses max_num_seqs=8 per engine; historical high-only measurements with 32 sequences are a different serving configuration and must not be copied as this setup's result.

2. Start the SDK gateway

export JEV_MODEL_PATH="$APUS_FULL_MODEL"
export JEV_HIGH_MODEL_PATH="$APUS_FULL_MODEL"
export JEV_LOW_MODEL_PATH="$APUS_LOW_MODEL"
export JEV_HIGH_URL=http://127.0.0.1:30002
export JEV_LOW_URL=http://127.0.0.1:30004
export JEV_HIGH_MODEL=apus-openjev-9b-high32
export JEV_LOW_MODEL=apus-openjev-9b-low16
export JEV_MODEL_NAME=APUS-OpenJev-v1-9B
# Supply your own local gateway secret; this is NOT your hosted TypeSafe API key.
export JEV_API_KEY='replace-with-your-local-gateway-secret'
python gateway.py --host 127.0.0.1 --port 30013

Startup requires both model directories on the same filesystem as the gateway. It checks 32/16-layer configs, merged indexes, completed low-export provenance, source/export metadata hashes, and each /v1/models alias/root against its configured artifact directory. Mismatches fail startup. The tokenizer must come from the full model directory. Engine URLs and aliases must be distinct. Depth headers report the selected verified backend configuration, not per-request layer-hook telemetry; live numerical evaluation remains necessary.

The gateway binds to localhost by default. Configure an authenticated HTTPS proxy before exposing it publicly. Backend URLs are operator configuration, never accepted from incoming requests. Both backend connections and the SDK client can reuse HTTP connections.

3. Use the official SDK against APUS

Use the API root as base_url — do not append /v1. The SDK appends /v1/systemone and /v1/models itself.

import os
from typesafe_sdk import Choice, Noul, RetryPolicy, TypeSafeClient

with TypeSafeClient(
    base_url="http://127.0.0.1:30013",
    api_key=os.environ["JEV_API_KEY"],
    model="APUS-OpenJev-v1-9B",
    retry=RetryPolicy(max_retries=0),
) as client:
    print(client.models.list())
    result = client.system_one(
        state={
            "message": "My parcel arrived damaged yesterday. Can I get a refund?",
            "policy": "Damaged items reported within 30 days qualify for a refund.",
        },
        questions={
            "queue": Choice(
                instructions="Choose the appropriate customer-support queue.",
                criteria={"returns": "Returns and refunds", "billing": "Payments and invoices"},
            ),
            "eligible": Noul(instructions="Is the customer eligible under this refund policy?"),
        },
        extra_body={"effort": "low"},
    )
    print(result.answers["queue"].choice)
    print(result.answers["eligible"].noul)
    print(result.raw_http_response.headers["x-openjev-executed-layers"])

Use extra_body={"effort": "high"} for the full model, or omit it because high is the default. effort is an APUS extension, not a promise that the hosted TypeSafe API accepts the same field.

Run the complete English customer-support example (Choice + Noul + ordinal Score):

python sdk_example.py --effort high
python sdk_example.py --effort low

Async uses the same fields:

import asyncio
import os
from typesafe_sdk import AsyncTypeSafeClient, Noul, RetryPolicy

async def main():
    async with AsyncTypeSafeClient(
        base_url="http://127.0.0.1:30013",
        api_key=os.environ["JEV_API_KEY"],
        model="APUS-OpenJev-v1-9B",
        retry=RetryPolicy(max_retries=0),
    ) as client:
        result = await client.system_one(
            state="The customer received a damaged item yesterday.",
            questions={"damaged": Noul(instructions="Did the customer report a damaged item?")},
            extra_body={"effort": "high"},
        )
        print(result.answers["damaged"].noul)

asyncio.run(main())

4. Use the official hosted TypeSafe service separately

This snippet sends a real request to the hosted service and can incur charges. Keep its credentials separate from the local gateway secret. It deliberately does not send the APUS effort extension.

import os
from typesafe_sdk import Choice, RetryPolicy, TypeSafeClient

with TypeSafeClient(
    base_url="https://api.typesafe.ai",
    api_key=os.environ["TYPESAFE_API_KEY"],
    model="jev-latest",
    retry=RetryPolicy(max_retries=0),
) as client:
    result = client.system_one(
        state="The customer was charged twice for the same order.",
        questions={"queue": Choice(
            instructions="Choose the appropriate customer-support queue.",
            criteria={"billing": "Payments and invoices", "shipping": "Delivery and tracking"},
        )},
    )
    print(result.answers["queue"].choice)

Contract and limits

• Supported SDK 0.7.0 methods: synchronous/asynchronous system_one and models.list. This SDK version exposes no models.retrieve method.
• GET /v1/models lists the APUS model and jev-latest as an explicitly labeled local compatibility alias. Responses name the actual APUS model.
• POST /v1/systemone accepts state, questions, model, and the optional APUS effort extension only. Unknown fields, invalid effort, unsupported models, malformed or duplicate-key JSON fail explicitly.
• State: string / object / array. Instructions: string / object / array / null. Choice descriptions also allow null; null uses the candidate name. JSON is serialized deterministically; ordinary string descriptions remain unchanged.
• Choice: 2–16 candidates. Noul: yes/no; optional true/false descriptions affect the question's meaning. Canonical true/false descriptions retain the original binary prompt. Custom descriptions are included explicitly.
• Score: 1–10 ordered levels, selected by their descriptions; returns the expected zero-based index, probabilities, and the original legend. This differs from the trained binary score_level task and has not established ordinal-score quality or calibration. A single-level Score has only one possible result: score 0, probability 1, no GPU request, zero model tokens, and executed-layers header 0. Its confidence 1 denotes a degenerate distribution, not evidence that the rubric is objectively correct.
• At most 16 questions, 1 MiB request body, and 8,191 compiled prompt tokens per question (one output token reserved). Inputs are never silently truncated. Maximum 32 in-flight requests, with a 120-second request deadline.
• Each question is compiled and scored separately. Usage sums the processed prompt tokens and one scoring token per question; billing_units means local decision count, not official monetary billing.
• Confidence is normalized distribution concentration. Candidate probabilities are uncalibrated conditional softmax; neither quantity inherits official calibration semantics.
• Response shape uses model, answers, usage; Choice has choice/confidence/probabilities, Noul has noul, Score has score/confidence/probabilities/legend.
• Headers x-openjev-effort, x-openjev-executed-layers, and x-openjev-engine-model record the chosen route. x-typesafe-request-id is a local request identifier.

Verification status

CPU integration tests use the real official SDK 0.7.0, both sync and async, through an ASGI gateway with a deterministic mocked inference backend. They verify protocol, routing, payload validation, and numerical response assembly; they are not GPU quality or performance measurements.

pip install -r requirements-test.txt
python -m pytest -q tests/test_typesafe_gateway.py

Before reporting GPU performance, record immutable model/export identities, prove 16/32-layer engine admission, run the same frozen requests through both routes, retain raw responses and collection timestamps, and compare against a reference using the same compiled prompts. In particular, a historic score_level record mapped to TypeSafe Noul changes its prompt primitive; do not silently compare it as byte-identical input.

Run a real service smoke test and preserve its raw requests, responses, UTC timestamps, and per-call latency:

python smoke_sdk.py --base-url http://127.0.0.1:30013 --output ./sdk-smoke-new-run

This sends five HTTP calls: models, high multi-question, low multi-question, an expected 422 for invalid effort, and an async low request. The two multi-question requests include all three primitives. Use a new output directory for every run. No authentication headers are saved.

Deployment code retains the included Apache 2.0 license and source notices. Model weights retain their applicable upstream licenses.