Commit History
24 days ago
-
Fix schema-constrained JSON generation via response_format->grammar normalization
ClaudeUnverified signaturellama.cpp's own response_format handling throws "Failed to initialize samplers: std::exception" on every request for this model (not just complex schemas): the tokenizer emits a spurious leading space on the token right after a special/control token, which breaks the literal chat-template prefix llama.cpp prefills into the grammar sampler before generation. Not router mode (this Space runs single-model llama-server) and not a stale image (reproduces on a from-source build of current llama.cpp master too). See docs/structured-output-postmortem.md for the full root-cause writeup. gateway.py now deterministically compiles response_format's JSON Schema into GBNF (schema_grammar.py) and sends `grammar` to llama-server instead, which has no such prefix/prefill step and works reliably. Unsupported schema shapes get a clear 422 instead of being silently dropped. Also tightens error handling: malformed client JSON now returns 400 instead of a bare 500, and upstream llama.cpp errors are passed …664fbed