docs/structured-output-postmortem.md

# Structured output (`response_format`) postmortem

## Symptom

`POST /v1/chat/completions` with `response_format: {"type": "json_object"}` or
`{"type": "json_schema", ...}` returned:

```
HTTP 400
{"error":{"code":400,"message":"Failed to initialize samplers: std::exception","type":"invalid_request_error"}}
```

for **every** request that used `response_format`, regardless of schema
shape. Plain, unconstrained chat completions worked fine.

## Deployment audit (Phase 1)

- `start.sh` runs `/app/llama-server` directly, single model
  (`--hf-repo openbmb/MiniCPM4-0.5B-GGUF:Q8_0 --alias minicpm4-0.5b`), with
  `--parallel 1 --cont-batching --flash-attn auto`, no `--jinja`, no router
  flags, no grammar-related flags. **Router/model-preset mode is not in
  use** — this is llama.cpp's plain single-model server. This was verified
  both by reading `start.sh`/`Dockerfile` and by reproducing the exact same
  failure against a from-source build run the same way, single model, no
  router.
- The Docker image is `FROM ghcr.io/ggml-org/llama.cpp:server` — a floating
  tag. The public Space's `system_fingerprint` on a plain completion
  reported `b11118-e6ab7c1a4`, i.e. build b11118 at commit `e6ab7c1a4...`.
- `gateway.py` forwarded `response_format`/`grammar`/`tools`/`stream`
  fields to llama.cpp unmodified, and turned upstream HTTP errors into an
  `HTTPException` with the same status code — it did not swallow or fake
  success, but it re-wrapped llama.cpp's JSON error body as an escaped
  string inside `{"detail": "..."}"`, which is awkward for API clients (see
  "Gateway changes" below).

## Root cause (Phase 3)

Reproduced directly against llama-server (bypassing the gateway), including
against a **from-source build of current llama.cpp master** (commit
`84e76d8`, ~12 days newer than the deployed build), so this is **not** a
stale/floating-tag regression — it is a live upstream bug.

Two distinct bugs, found by attaching `gdb` (`catch throw`) to a from-source
debug build and by adding a temporary diagnostic print of the caught
exception's `typeid` and `what()`:

### Bug 1 — the "std::exception" message is a symptom, not the disease

`common/sampling.cpp`, in `common_sampler_init`, prefills the grammar
sampler with the chat template's generation-prompt text before generation
starts:

```cpp
try {
    for (const auto & token : prefill_tokens) {
        llama_sampler_accept(grmr, token);
    }
} catch (std::exception &e) {
    LOG_ERR(...);
    throw e;      // <-- BUG: copies/slices `e` down to plain std::exception
}
```

`throw e;` **copies** the caught exception object using the **static** type
of the catch clause (`std::exception`), discarding whatever real subclass
(here `std::runtime_error`) and message it actually was. That's why every
failure surfaced as the useless `"std::exception"` instead of the real
message. The fix is the standard one-line correction: bare `throw;`
(verified locally — see below).

### Bug 2 — the real error: a tokenizer/grammar-literal mismatch

With bug 1 patched, the real message is:

```
Failed to initialize samplers: Unexpected empty grammar stack after accepting piece:  assistant (16434)
```

llama.cpp's generic auto-parser (used because minicpm4 has no dedicated
tool-call parser, only `minicpm5` does) builds `response_format`'s grammar
wrapped in a literal chat-template prefix, e.g.:

```
root ::= "<|im_start|>assistant\n" space space ( "```json" space response-format space "```" | space response-format space )
```

and then "prefills" that literal into the grammar sampler by tokenizing the
prefix text and feeding the tokens through `llama_sampler_accept`
(`common/sampling.cpp`). For minicpm4-0.5b, the token for the word
"assistant" immediately after the `<|im_start|>` control token decodes to
`" assistant"` (**with a spurious leading space**) — the tokenizer has no
space-less "assistant" token, because in ordinary text that word is always
preceded by real whitespace. That leading space doesn't match the raw
literal text the grammar rule was built from (`"assistant"`, no space), so
the grammar's internal stack empties out and
`llama_grammar_accept_token` (`src/llama-grammar.cpp`) throws.

This reproduces for **every** `response_format` request for this model —
`json_object` with no schema, and any `json_schema`, simple or complex —
because the broken step is the literal-prefix prefill, not anything
schema-shape-specific. (A separate, already-fixed upstream bug —
`json-schema-to-grammar`'s "empty object schema" `"{" space space "}"`
GBNF bug, ggml-org/llama.cpp#25923, fixed in PR #28736 on 2026-09-12 — is
**not** the cause here: the deployed build (`e6ab7c1a4`, 2026-09-22)
already contains that fix.)

A plain `"grammar": "<gbnf>"` request (bypassing `response_format` and
the auto-parser's literal-prefix/prefill path entirely) works reliably —
confirmed directly against the exact deployed llama-server build. That is
the basis for the fix shipped here.

## Fix (Phase 4/5)

Per the task's preferred order, option **A** (server/request configuration)
turned out to be sufficient and lowest-risk — no image rebuild, no llama.cpp
version change needed:

- `schema_grammar.py` deterministically compiles the JSON Schemas this
  service needs (object/boolean/string(+enum), `required`,
  `additionalProperties: false`, and untyped/empty-object placeholders)
  into GBNF, and `gateway.py` sends that as `grammar` instead of forwarding
  `response_format` to llama.cpp. This avoids the buggy literal-prefix
  prefill path entirely.
- The converter accepts the OpenAI-style nested
  `{"type": "json_schema", "json_schema": {"schema": {...}}}` shape, the
  flatter `{"type": "json_schema", "schema": {...}}` shape, and
  `{"type": "json_object"}` (with or without a `schema`).
  It never modifies the schema's semantics — it raises
  `SchemaToGrammarError` for anything outside its supported subset (arrays,
  `pattern`, `$ref`, `oneOf`, mixed required/optional ordering it can't
  express as a simple sequential grammar, etc.) rather than silently
  dropping a constraint. The gateway turns that into an HTTP 422.
- Unit tests: `test_schema_grammar.py` (23 tests, `python3 -m unittest
  test_schema_grammar.py`).
- Also fixed, and verified against a from-source build, but **not** shipped
  in this Docker image (which uses the prebuilt
  `ghcr.io/ggml-org/llama.cpp:server` image, not a source build): the
  `throw e;` → `throw;` slicing bug in `common/sampling.cpp` described
  above. Worth upstreaming separately; tracked here for anyone who does
  rebuild llama.cpp from source for this Space.

### Why not upgrade or patch llama.cpp itself?

The bug is present in current llama.cpp master (verified by building
commit `84e76d8` from source), so upgrading the floating tag would not have
fixed it. Patching the tokenizer/grammar-literal mismatch correctly (so the
literal-prefix grammar tolerates the model's real detokenization) touches
shared, security-and-correctness-sensitive code used by every model's
`response_format` handling, and doing that safely needs llama.cpp's own
test suite and review — out of scope for a single-model gateway fix, and
unnecessary given the request-level workaround fully resolves the problem
for this deployment.

## Remaining limitations

- The GBNF converter in `schema_grammar.py` only supports the schema
  shapes this service's tool-repair verifier flow needs (see its module
  docstring). Anything else is rejected with a clear 422, not silently
  passed through — if broader JSON Schema support is needed later, extend
  `_compile_node`/`_compile_object` and their tests.
- `minicpm4-0.5b` is small; at nonzero temperature its *content* judgement
  (e.g. the `repairable` boolean) can vary run-to-run even though the JSON
  is always syntactically valid and schema-conformant. The repair test in
  `scripts/validate_structured_output.py` uses `temperature: 0` for
  repeatable content, which is also the right default for a
  verifier/repair role.
- `start.sh`'s default `MODEL_REPO` (`openbmb/MiniCPM4-0.5B-GGUF`) does not
  currently resolve on the Hugging Face Hub (only
  `openbmb/MiniCPM4-0.5B-QAT-Int4-GGUF` and the non-GGUF
  `openbmb/MiniCPM4-0.5B` exist there today). The running Space still works
  because the model is already cached, but a cold rebuild/restart that has
  to re-download the model would fail to start. This is outside this
  task's scope (structured output) but worth fixing separately.