Deployment validation — 2026-09-21
The official TypeSafe Python SDK 0.7.0 successfully called this gateway on a real GPU deployment. high reached a 32-layer vLLM engine; low reached a separately exported 16-layer engine. This is explicit depth selection, not automatic confidence routing.
Live SDK HTTP results
RTX PRO 6000 Blackwell 96 GB, vLLM 0.29.0, BF16, merged 9B-5949, one GPU with two resident engines. Local loopback HTTP, one persistent SDK client, concurrency 1, retries disabled. Each effort ran the same frozen80 panel three times. Collection: 2026-09-21 13:46:20–13:46:52 UTC. Timing covers SDK invocation through complete response validation; it is not streaming TTFT.
| SDK route | Successful | Accuracy per round | Mean | P50 | P95 | P99 |
|---|---|---|---|---|---|---|
| high / 32 layers | 240/240 | 67/80 in all three | 70.11 ms | 23.75 ms | 243.10 ms | 290.51 ms |
| low / 16 layers | 240/240 | 67/80 in all three | 49.17 ms | 22.01 ms | 154.61 ms | 177.58 ms |
Percentiles pool all 240 calls per route; they are not averages of per-round percentiles. Each run scored Browser 12/16, HelpSteer3 13/16, BoolQ 16/16, MNLI 12/16, and Score-derived Noul 14/16. There are only 80 distinct questions from 79 parent groups. These development questions have informed the project; neither significance nor general noninferiority follows from this result.
Mean latency fell 29.86% (about 1.43× speed) in this sequential local workload. Median improvement was much smaller. This does not establish a doubling of throughput or a public-network speedup. Serving budgets: high gpu_memory_utilization=0.70, max_num_seqs=32; low 0.22, max_num_seqs=8; both max length 8192, prefix cache disabled. The supplied launcher defaults to 8 sequences for each route; its concurrent throughput must be benchmarked separately. Resident GPU memory was approximately 90,580 MiB with both engines loaded.
The SDK smoke separately passed model listing, multi-question Choice/Noul/Score at both efforts, invalid-effort error handling, and an asynchronous low request: 5/5 checks. Full request/response bodies, UTC timestamps, route headers, and raw evaluation records were retained internally; no hosted API credentials are needed for this service.
Numerical verification and limits
The exporter retained 548 tensors, preserving their bytes, including the original embedding, final norm and LM head, while removing decoder layers 16–31. The source weights and metadata were checked against their release manifest. A native reference run used layer hooks to confirm that only layers 0–15 ran for all 80 questions.
A separate raw vLLM candidate-scoring comparison used the same original compiled prompts:
| Route | Correct | Same selected candidate as native reference | Maximum probability difference |
|---|---|---|---|
| high32 | 67/80 | 80/80 | 0.06422 |
| low16 | 67/80 | 78/80 | 0.08855 |
Low changed two Browser decisions: one incorrect answer became correct and one correct answer became incorrect. Equal aggregate accuracy does not establish identical decisions or numerical parity. Original probability thresholds must be recalibrated for the deployed runtime. The first raw low request took about 1,552 ms before warm execution; SDK measurements above followed raw verification and smoke requests.
The wire benchmark uses the same state, instructions, candidate descriptions and frozen IDs as the official API requests, with the APUS model and effort extension. Historic binary score_level records map to TypeSafe Noul, changing the internal prompt primitive; the raw-reference comparison and SDK-wire comparison therefore answer different questions. Ordinal Score protocol smoke is not a scored quality benchmark for ordinal rubrics.
Reproducibility
- Source HF family:
apus-ailab/APUS-Reflex, model subdirectory9B-5949/; inspected source repository revisionc8a31769918d34e4142b9338b231a7bef59de3f3. Serving-code publication does not change weights. - Frozen panel SHA256:
d6dd45a9c8cb9f5b50254ac5f8f550e0ab1341e993c8657eb89232f64686ebf6. - Protocol tests: 12 passing, real official SDK sync/async against a real ASGI gateway and mocked scoring engines.
- Export tests: 9 passing, standalone CPU tiny-Qwen prefix/readout equivalence, retained tensor identity, source modification and weight-tampering rejection.
- Live collector tests: 2 passing; independent offline verification checks all 480 saved requests, IDs, gold scoring, route headers and recorded timing.
- Serving bundle passes Ruff lint/format and shell syntax checks. A clean OS dependency installation was not tested; GPU tests used the existing verified Runpod environment.
The official hosted service was measured separately over a WAN: the earlier 240 pooled responses had P50/P95/P99 280.80/376.56/830.45 ms. A subsequent SDK single-connection 160-call test averaged 301.63 ms. Those include geography/network effects and must not be divided by these loopback numbers to claim a model speedup.
Compatibility boundaries
The gateway supports the SDK's synchronous/asynchronous system_one and models.list methods, typed responses, local bearer auth and documented errors. See README for candidate, context and request limits. This release does not assert unlimited compatibility with every request accepted by the hosted service, official calibration, automatic early exit, shared KV broadcast, or production Browser task completion.