Commit History
23 days ago
-
Optimize MiniCPM4 for the code-review role: 8K ctx, compact grammar contract Running
ClaudeUnverified signatureBenchmarked the actual deployed QAT-Int4 model against Q8_0 (the requested quality baseline) on an 8-case code-review accuracy benchmark: QAT-Int4 won (6/8 vs 5/8 correct, 0 vs 1 invalid outputs) and is smaller/faster, so kept. - CTX_SIZE 32768 -> 8192: prefill throughput on this CPU degrades sharply with sequence length (218 tok/s at 1K input down to ~50 tok/s by 9-10K), so an 8K+ review is impractically slow here; 8192 covers realistic reviews and uses ~12% less resident memory. Threads/batch/ubatch swept and left unchanged (2 matches the actual 2-vCPU host; batch/ubatch showed no measurable difference on this CPU-bound workload). - New review_contract.py: a compact, stable {"issues": [...], "verdict": "pass"|"issue"} contract with a hand-built discriminated-union grammar (pass/issue are separate branches, so a valid response can't have a mismatched verdict/issues pair) and issues-before-verdict field order, which raised review accuracy from 2/8 to 6/8 correct with…38a5274
24 days ago
-
Fix schema-constrained JSON generation via response_format->grammar normalization
ClaudeUnverified signaturellama.cpp's own response_format handling throws "Failed to initialize samplers: std::exception" on every request for this model (not just complex schemas): the tokenizer emits a spurious leading space on the token right after a special/control token, which breaks the literal chat-template prefix llama.cpp prefills into the grammar sampler before generation. Not router mode (this Space runs single-model llama-server) and not a stale image (reproduces on a from-source build of current llama.cpp master too). See docs/structured-output-postmortem.md for the full root-cause writeup. gateway.py now deterministically compiles response_format's JSON Schema into GBNF (schema_grammar.py) and sends `grammar` to llama-server instead, which has no such prefix/prefill step and works reliably. Unsupported schema shapes get a clear 422 instead of being silently dropped. Also tightens error handling: malformed client JSON now returns 400 instead of a bare 500, and upstream llama.cpp errors are passed …664fbed