docs(wiki): record response_format fix, Qwen2.5-3B default, ZeroGPU quota behaviour
Browse files- response_format must be {"type":"json_object","schema":...}; the OpenAI-style
"json_schema" is silently ignored by llama-cpp-python, leaving fuse/tweak
unconstrained and forcing every edit onto the keyword editor.
- Default model is now Qwen2.5-3B (6/7 schema incl. Chinese on the pinned
0.3.19; Llama-3.2-3B scored 0/7 even after the fix). Recorded the 3B-vs-7B
trade-off and that Qwen2.5 runs on 0.3.19 with no llama-cpp upgrade.
- Documented ZeroGPU quota-exhaustion symptoms (warmup failures, CPU drop to
~1.9 tok/s). Verified against live Space run logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- wiki/deployment-strategy.md +54 -13
- wiki/log.md +16 -0
wiki/deployment-strategy.md
CHANGED
|
@@ -5,6 +5,7 @@ last_updated: 2026-06-14
|
|
| 5 |
confidence: high
|
| 6 |
sources:
|
| 7 |
- src/buddy_fusion/runtime.py
|
|
|
|
| 8 |
- requirements.txt
|
| 9 |
- README.md
|
| 10 |
- scripts/start-local.sh
|
|
@@ -56,6 +57,20 @@ background while the player reads the intro. It runs once per container
|
|
| 56 |
These are recipe-level JSON, not prose, so the budgets are small β keeping a
|
| 57 |
playthrough's GPU seconds lean so a judge's first splice isn't spent on the JIT.
|
| 58 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
## ZeroGPU vs CPU Basic β reference baseline
|
| 60 |
|
| 61 |
Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
|
|
@@ -84,6 +99,15 @@ ZeroGPU operational constraints worth remembering:
|
|
| 84 |
hence the 30/90 split.
|
| 85 |
- **Hosting needs a ZeroGPU grant** β the `build-small-hackathon` org has one
|
| 86 |
(a personal free account gets a `403`).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 87 |
|
| 88 |
## Space build: the wheel pins (load-bearing)
|
| 89 |
|
|
@@ -117,22 +141,39 @@ Verification signal in the build log: it should print
|
|
| 117 |
`Downloading llama_cpp_python-0.3.19-...-manylinux...whl`, **not** a
|
| 118 |
`.tar.gz` followed by `Building wheel ... still running`.
|
| 119 |
|
| 120 |
-
**chat-template ceiling:**
|
| 121 |
-
|
| 122 |
-
|
| 123 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
|
| 125 |
## Model strategy
|
| 126 |
|
| 127 |
-
- Default real config (`runtime.py`): `
|
| 128 |
-
primary, `SmolLM2-1.7B-Instruct` fallback
|
| 129 |
-
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
-
|
| 134 |
-
|
| 135 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 136 |
[hackathon-context](hackathon-context.md)).
|
| 137 |
|
| 138 |
## UX constraint
|
|
|
|
| 5 |
confidence: high
|
| 6 |
sources:
|
| 7 |
- src/buddy_fusion/runtime.py
|
| 8 |
+
- src/buddy_fusion/prompts.py
|
| 9 |
- requirements.txt
|
| 10 |
- README.md
|
| 11 |
- scripts/start-local.sh
|
|
|
|
| 57 |
These are recipe-level JSON, not prose, so the budgets are small β keeping a
|
| 58 |
playthrough's GPU seconds lean so a judge's first splice isn't spent on the JIT.
|
| 59 |
|
| 60 |
+
### Schema enforcement (response_format) β load-bearing
|
| 61 |
+
|
| 62 |
+
`_blocking_completion` constrains output with
|
| 63 |
+
`response_format={"type": "json_object", "schema": ...}`. **The key must be
|
| 64 |
+
`json_object`, not the OpenAI-style `json_schema`** β llama-cpp-python silently
|
| 65 |
+
ignores `json_schema`, leaving the model completely unconstrained: it returns
|
| 66 |
+
`type` instead of `kind`, bare strings like `["arm", "arm"]`, or a
|
| 67 |
+
`{"genome": {...}}` wrapper. None of those survive `repair_genome`, so every
|
| 68 |
+
edit falls through to the deterministic keyword editor β which is why tweaks
|
| 69 |
+
looked like they "ignored the model" for so long. With `json_object` the schema
|
| 70 |
+
is compiled to a GBNF grammar and enforced, so fuse / tweak output actually
|
| 71 |
+
matches `GENOME_SCHEMA` / `GENOME_EDIT_SCHEMA`. This bug masked the real model
|
| 72 |
+
behaviour; fix it before judging any model's capability.
|
| 73 |
+
|
| 74 |
## ZeroGPU vs CPU Basic β reference baseline
|
| 75 |
|
| 76 |
Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
|
|
|
|
| 99 |
hence the 30/90 split.
|
| 100 |
- **Hosting needs a ZeroGPU grant** β the `build-small-hackathon` org has one
|
| 101 |
(a personal free account gets a `403`).
|
| 102 |
+
- **Quota exhaustion is visible in the run log** β when the org's daily ZeroGPU
|
| 103 |
+
quota runs out, `warmup()` logs `[zerogpu-warmup] failed: ... exceeded ZeroGPU
|
| 104 |
+
quota (... vs. 0s left)` and inference drops to CPU: `[zerogpu-bench] ... 1.9
|
| 105 |
+
tok/s` (vs ~200 on GPU), ~45s per tweak, fuse far worse. At that point tweaks
|
| 106 |
+
also fall to the keyword editor, so the model's real value (Chinese, complex,
|
| 107 |
+
out-of-schema approximations) can't be shown. Repeatedly opening the Space /
|
| 108 |
+
re-warming burns the budget β keep it in reserve before a demo. The
|
| 109 |
+
`duration=90` warmup reserves ~135s and retries on every page load while quota
|
| 110 |
+
is exhausted, a pure cost worth trimming or skipping when quota is low.
|
| 111 |
|
| 112 |
## Space build: the wheel pins (load-bearing)
|
| 113 |
|
|
|
|
| 141 |
`Downloading llama_cpp_python-0.3.19-...-manylinux...whl`, **not** a
|
| 142 |
`.tar.gz` followed by `Building wheel ... still running`.
|
| 143 |
|
| 144 |
+
**chat-template ceiling (narrower than feared):** the cu124 cp312 wheel index
|
| 145 |
+
stops at `0.3.19`, and 0.3.19's Jinja engine is old enough that *some* newer
|
| 146 |
+
model families throw an `unknown tag` error. But **Qwen2.5 (3B/7B/14B/32B) runs
|
| 147 |
+
fine on 0.3.19** β verified locally against the Space-pinned version: the chat
|
| 148 |
+
template loads and `json_object` schema enforcement works. So adopting Qwen2.5
|
| 149 |
+
needs **no** llama-cpp-python upgrade and no wheel / README pin change. (If a
|
| 150 |
+
future model ever does need a newer build: the cu124 index now ships a
|
| 151 |
+
`py3-none-manylinux_2_35_x86_64` wheel for `0.3.29`, Linux-installable β the
|
| 152 |
+
old "cp312 stops at 0.3.19" pin is no longer the only option.)
|
| 153 |
|
| 154 |
## Model strategy
|
| 155 |
|
| 156 |
+
- Default real config (`runtime.py`): **`Qwen2.5-3B-Instruct`** GGUF Q4_K_M
|
| 157 |
+
primary, `SmolLM2-1.7B-Instruct` fallback (the smaller download-failure
|
| 158 |
+
fallback, not a quality fallback).
|
| 159 |
+
- Why Qwen2.5-3B β measured locally on the genome edit schema (0.3.19, *after*
|
| 160 |
+
the `json_object` fix above; before the fix every model scored ~0 because the
|
| 161 |
+
schema never applied):
|
| 162 |
+
- `Qwen2.5-7B` 7/7 (incl. Chinese), `Qwen2.5-3B` 6/7 β both follow the schema
|
| 163 |
+
and produce genuinely model-driven tweaks, in English and Chinese.
|
| 164 |
+
- `Llama-3.2-3B` (the previous default) **0/7 even with the fix** β it overran
|
| 165 |
+
the 200-token edit budget and produced truncated JSON. `SmolLM2-1.7B` 3/7.
|
| 166 |
+
- 3B is the cheapest that works and matches the old footprint (~2GB GGUF); 7B
|
| 167 |
+
is the safest (7/7, ~4.7GB). Chose 3B for cost parity; 7B is the upgrade if
|
| 168 |
+
schema misses ever matter.
|
| 169 |
+
- Out-of-schema instructions ("build a house", "turn it into a robot") still
|
| 170 |
+
no-op or approximate on every model β that is a schema/part-kind limit, not
|
| 171 |
+
a model limit.
|
| 172 |
+
- `scripts/start-local-real-model.sh` presets still select SmolLM2 / Llama for
|
| 173 |
+
local iteration. Model loading is lazy β the first splice downloads the GGUF
|
| 174 |
+
(or pre-cache via `BUDDY_MODEL_PATH`).
|
| 175 |
+
- The 32B competition cap is an upper bound, not a target: a small model keeps
|
| 176 |
+
the `Tiny Titan` / `Off the Grid` bonus quests in reach (see
|
| 177 |
[hackathon-context](hackathon-context.md)).
|
| 178 |
|
| 179 |
## UX constraint
|
wiki/log.md
CHANGED
|
@@ -2,6 +2,22 @@
|
|
| 2 |
|
| 3 |
## 2026-06-14
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
### Theme: switched from light-redirect lock to theme-independence
|
| 6 |
|
| 7 |
- Rewrote `gradio-theme-locking.md` (and its `index.md` line): the app no longer
|
|
|
|
| 2 |
|
| 3 |
## 2026-06-14
|
| 4 |
|
| 5 |
+
### deployment-strategy: response_format bug, Qwen2.5-3B default, ZeroGPU quota
|
| 6 |
+
|
| 7 |
+
- Recorded the root-cause fix: `response_format` must be
|
| 8 |
+
`{"type":"json_object","schema":...}`; the OpenAI-style `"json_schema"` is
|
| 9 |
+
silently ignored by llama-cpp-python, so fuse/tweak ran unconstrained and every
|
| 10 |
+
edit fell to the keyword editor. Added a "Schema enforcement" subsection.
|
| 11 |
+
- Updated Model strategy: default is now `Qwen2.5-3B-Instruct` (6/7 schema, incl.
|
| 12 |
+
Chinese; Llama-3.2-3B scored 0/7 even after the fix β truncated JSON). Recorded
|
| 13 |
+
the local 0.3.19 measurements and the 3B-vs-7B trade-off.
|
| 14 |
+
- Corrected the chat-template ceiling: Qwen2.5 (3Bβ32B) runs on the Space-pinned
|
| 15 |
+
0.3.19, so adopting it needs no llama-cpp-python upgrade; noted the new
|
| 16 |
+
`py3-none-manylinux` 0.3.29 wheel as a future option.
|
| 17 |
+
- Added ZeroGPU quota-exhaustion symptoms (warmup failures, CPU drop to ~1.9
|
| 18 |
+
tok/s) to the operational constraints. Verified against live Space run logs;
|
| 19 |
+
fix is committed (`d9eb5a7`) and deployed.
|
| 20 |
+
|
| 21 |
### Theme: switched from light-redirect lock to theme-independence
|
| 22 |
|
| 23 |
- Rewrote `gradio-theme-locking.md` (and its `index.md` line): the app no longer
|