docs/structured-output-postmortem.md

Structured output (response_format) postmortem

Symptom

POST /v1/chat/completions with response_format: {"type": "json_object"} or {"type": "json_schema", ...} returned:

HTTP 400
{"error":{"code":400,"message":"Failed to initialize samplers: std::exception","type":"invalid_request_error"}}

for every request that used response_format, regardless of schema shape. Plain, unconstrained chat completions worked fine.

Deployment audit (Phase 1)

• start.sh runs /app/llama-server directly, single model (--hf-repo openbmb/MiniCPM4-0.5B-GGUF:Q8_0 --alias minicpm4-0.5b), with --parallel 1 --cont-batching --flash-attn auto, no --jinja, no router flags, no grammar-related flags. Router/model-preset mode is not in use — this is llama.cpp's plain single-model server. This was verified both by reading start.sh/Dockerfile and by reproducing the exact same failure against a from-source build run the same way, single model, no router.
• The Docker image is FROM ghcr.io/ggml-org/llama.cpp:server — a floating tag. The public Space's system_fingerprint on a plain completion reported b11118-e6ab7c1a4, i.e. build b11118 at commit e6ab7c1a4....
• gateway.py forwarded response_format/grammar/tools/stream fields to llama.cpp unmodified, and turned upstream HTTP errors into an HTTPException with the same status code — it did not swallow or fake success, but it re-wrapped llama.cpp's JSON error body as an escaped string inside {"detail": "..."}", which is awkward for API clients (see "Gateway changes" below).

Root cause (Phase 3)

Reproduced directly against llama-server (bypassing the gateway), including against a from-source build of current llama.cpp master (commit 84e76d8, ~12 days newer than the deployed build), so this is not a stale/floating-tag regression — it is a live upstream bug.

Two distinct bugs, found by attaching gdb (catch throw) to a from-source debug build and by adding a temporary diagnostic print of the caught exception's typeid and what():

Bug 1 — the "std::exception" message is a symptom, not the disease

common/sampling.cpp, in common_sampler_init, prefills the grammar sampler with the chat template's generation-prompt text before generation starts:

try {
    for (const auto & token : prefill_tokens) {
        llama_sampler_accept(grmr, token);
    }
} catch (std::exception &e) {
    LOG_ERR(...);
    throw e;      // <-- BUG: copies/slices `e` down to plain std::exception
}

throw e; copies the caught exception object using the static type of the catch clause (std::exception), discarding whatever real subclass (here std::runtime_error) and message it actually was. That's why every failure surfaced as the useless "std::exception" instead of the real message. The fix is the standard one-line correction: bare throw; (verified locally — see below).

Bug 2 — the real error: a tokenizer/grammar-literal mismatch

With bug 1 patched, the real message is:

Failed to initialize samplers: Unexpected empty grammar stack after accepting piece:  assistant (16434)

llama.cpp's generic auto-parser (used because minicpm4 has no dedicated tool-call parser, only minicpm5 does) builds response_format's grammar wrapped in a literal chat-template prefix, e.g.:

root ::= "<|im_start|>assistant\n" space space ( "```json" space response-format space "```" | space response-format space )

and then "prefills" that literal into the grammar sampler by tokenizing the prefix text and feeding the tokens through llama_sampler_accept (common/sampling.cpp). For minicpm4-0.5b, the token for the word "assistant" immediately after the <|im_start|> control token decodes to " assistant" (with a spurious leading space) — the tokenizer has no space-less "assistant" token, because in ordinary text that word is always preceded by real whitespace. That leading space doesn't match the raw literal text the grammar rule was built from ("assistant", no space), so the grammar's internal stack empties out and llama_grammar_accept_token (src/llama-grammar.cpp) throws.

This reproduces for every response_format request for this model — json_object with no schema, and any json_schema, simple or complex — because the broken step is the literal-prefix prefill, not anything schema-shape-specific. (A separate, already-fixed upstream bug — json-schema-to-grammar's "empty object schema" "{" space space "}" GBNF bug, ggml-org/llama.cpp#25923, fixed in PR #28736 on 2026-09-12 — is not the cause here: the deployed build (e6ab7c1a4, 2026-09-22) already contains that fix.)

A plain "grammar": "<gbnf>" request (bypassing response_format and the auto-parser's literal-prefix/prefill path entirely) works reliably — confirmed directly against the exact deployed llama-server build. That is the basis for the fix shipped here.

Fix (Phase 4/5)

Per the task's preferred order, option A (server/request configuration) turned out to be sufficient and lowest-risk — no image rebuild, no llama.cpp version change needed:

• schema_grammar.py deterministically compiles the JSON Schemas this service needs (object/boolean/string(+enum), required, additionalProperties: false, and untyped/empty-object placeholders) into GBNF, and gateway.py sends that as grammar instead of forwarding response_format to llama.cpp. This avoids the buggy literal-prefix prefill path entirely.
• The converter accepts the OpenAI-style nested {"type": "json_schema", "json_schema": {"schema": {...}}} shape, the flatter {"type": "json_schema", "schema": {...}} shape, and {"type": "json_object"} (with or without a schema). It never modifies the schema's semantics — it raises SchemaToGrammarError for anything outside its supported subset (arrays, pattern, $ref, oneOf, mixed required/optional ordering it can't express as a simple sequential grammar, etc.) rather than silently dropping a constraint. The gateway turns that into an HTTP 422.
• Unit tests: test_schema_grammar.py (23 tests, python3 -m unittest test_schema_grammar.py).
• Also fixed, and verified against a from-source build, but not shipped in this Docker image (which uses the prebuilt ghcr.io/ggml-org/llama.cpp:server image, not a source build): the throw e; → throw; slicing bug in common/sampling.cpp described above. Worth upstreaming separately; tracked here for anyone who does rebuild llama.cpp from source for this Space.

Why not upgrade or patch llama.cpp itself?

The bug is present in current llama.cpp master (verified by building commit 84e76d8 from source), so upgrading the floating tag would not have fixed it. Patching the tokenizer/grammar-literal mismatch correctly (so the literal-prefix grammar tolerates the model's real detokenization) touches shared, security-and-correctness-sensitive code used by every model's response_format handling, and doing that safely needs llama.cpp's own test suite and review — out of scope for a single-model gateway fix, and unnecessary given the request-level workaround fully resolves the problem for this deployment.

Remaining limitations

• The GBNF converter in schema_grammar.py only supports the schema shapes this service's tool-repair verifier flow needs (see its module docstring). Anything else is rejected with a clear 422, not silently passed through — if broader JSON Schema support is needed later, extend _compile_node/_compile_object and their tests.
• minicpm4-0.5b is small; at nonzero temperature its content judgement (e.g. the repairable boolean) can vary run-to-run even though the JSON is always syntactically valid and schema-conformant. The repair test in scripts/validate_structured_output.py uses temperature: 0 for repeatable content, which is also the right default for a verifier/repair role.
• start.sh's default MODEL_REPO (openbmb/MiniCPM4-0.5B-GGUF) does not currently resolve on the Hugging Face Hub (only openbmb/MiniCPM4-0.5B-QAT-Int4-GGUF and the non-GGUF openbmb/MiniCPM4-0.5B exist there today). The running Space still works because the model is already cached, but a cold rebuild/restart that has to re-download the model would fail to start. This is outside this task's scope (structured output) but worth fixing separately.