The full Pydantic schema grammar made per-token sampling unusably slow
on Space CPUs, and Qwen3 thinking mode collapsed grammar-constrained
output to bare "{}".
- Send a compact response shape instead of the full JSON schema
(~2.7k -> ~1.4k prompt tokens); applies to llama.cpp and transformers
- Build the ChatML prompt by hand, prefill an empty <think> block and
add /no_think so JSON-constrained generation stays on-distribution
- Constrain output with a small whitespace-free generic JSON grammar
via create_completion instead of the schema grammar
- Add deterministic repairs in parse_model_response: lift top-level
keys misplaced inside check_in, null invalid enum values, prune
missing_fields/follow-ups already answered by the check-in, drop
bare field names from follow_up_questions
- Log finish_reason and token usage, warn on truncation
- Default LLAMA_CPP_MAX_TOKENS to 512
Revert "Disable llama.cpp schema grammar by default"
Lucas
675478d
This reverts commit 5e5caa6edde48c436a1fa4e4f89de31ea7860569.
