arkai2025 Claude Opus 4.8 (1M context) commited on
Commit
2a50cd7
Β·
1 Parent(s): d9eb5a7

docs(wiki): record response_format fix, Qwen2.5-3B default, ZeroGPU quota behaviour

Browse files

- response_format must be {"type":"json_object","schema":...}; the OpenAI-style
"json_schema" is silently ignored by llama-cpp-python, leaving fuse/tweak
unconstrained and forcing every edit onto the keyword editor.
- Default model is now Qwen2.5-3B (6/7 schema incl. Chinese on the pinned
0.3.19; Llama-3.2-3B scored 0/7 even after the fix). Recorded the 3B-vs-7B
trade-off and that Qwen2.5 runs on 0.3.19 with no llama-cpp upgrade.
- Documented ZeroGPU quota-exhaustion symptoms (warmup failures, CPU drop to
~1.9 tok/s). Verified against live Space run logs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files changed (2) hide show
  1. wiki/deployment-strategy.md +54 -13
  2. wiki/log.md +16 -0
wiki/deployment-strategy.md CHANGED
@@ -5,6 +5,7 @@ last_updated: 2026-06-14
5
  confidence: high
6
  sources:
7
  - src/buddy_fusion/runtime.py
 
8
  - requirements.txt
9
  - README.md
10
  - scripts/start-local.sh
@@ -56,6 +57,20 @@ background while the player reads the intro. It runs once per container
56
  These are recipe-level JSON, not prose, so the budgets are small β€” keeping a
57
  playthrough's GPU seconds lean so a judge's first splice isn't spent on the JIT.
58
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
  ## ZeroGPU vs CPU Basic β€” reference baseline
60
 
61
  Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
@@ -84,6 +99,15 @@ ZeroGPU operational constraints worth remembering:
84
  hence the 30/90 split.
85
  - **Hosting needs a ZeroGPU grant** β€” the `build-small-hackathon` org has one
86
  (a personal free account gets a `403`).
 
 
 
 
 
 
 
 
 
87
 
88
  ## Space build: the wheel pins (load-bearing)
89
 
@@ -117,22 +141,39 @@ Verification signal in the build log: it should print
117
  `Downloading llama_cpp_python-0.3.19-...-manylinux...whl`, **not** a
118
  `.tar.gz` followed by `Building wheel ... still running`.
119
 
120
- **chat-template ceiling:** `llama-cpp-python==0.3.19`'s template engine is old;
121
- newer model families need β‰₯0.3.26 or they throw a Jinja `unknown tag` error β€”
122
- but the cu124 cp312 index stops at 0.3.19. `Llama-3.2-3B` and `SmolLM2-1.7B`
123
- work; this caps model upgrades.
 
 
 
 
 
124
 
125
  ## Model strategy
126
 
127
- - Default real config (`runtime.py`): `Llama-3.2-3B-Instruct` GGUF Q4_K_M
128
- primary, `SmolLM2-1.7B-Instruct` fallback.
129
- - `scripts/start-local-real-model.sh` preset `fast` swaps to SmolLM2-1.7B
130
- primary for quicker local iteration; preset `llama` keeps Llama-3.2-3B
131
- primary. Model loading is lazy β€” starting the app does not download a GGUF;
132
- the first splice pays that cost (or pre-cache via `BUDDY_MODEL_PATH`).
133
- - The 32B competition cap is an upper bound, not a target: response speed and
134
- demo stability matter more than parameter count, and a small model keeps the
135
- `Tiny Titan` / `Off the Grid` bonus quests in reach (see
 
 
 
 
 
 
 
 
 
 
 
 
136
  [hackathon-context](hackathon-context.md)).
137
 
138
  ## UX constraint
 
5
  confidence: high
6
  sources:
7
  - src/buddy_fusion/runtime.py
8
+ - src/buddy_fusion/prompts.py
9
  - requirements.txt
10
  - README.md
11
  - scripts/start-local.sh
 
57
  These are recipe-level JSON, not prose, so the budgets are small β€” keeping a
58
  playthrough's GPU seconds lean so a judge's first splice isn't spent on the JIT.
59
 
60
+ ### Schema enforcement (response_format) β€” load-bearing
61
+
62
+ `_blocking_completion` constrains output with
63
+ `response_format={"type": "json_object", "schema": ...}`. **The key must be
64
+ `json_object`, not the OpenAI-style `json_schema`** β€” llama-cpp-python silently
65
+ ignores `json_schema`, leaving the model completely unconstrained: it returns
66
+ `type` instead of `kind`, bare strings like `["arm", "arm"]`, or a
67
+ `{"genome": {...}}` wrapper. None of those survive `repair_genome`, so every
68
+ edit falls through to the deterministic keyword editor β€” which is why tweaks
69
+ looked like they "ignored the model" for so long. With `json_object` the schema
70
+ is compiled to a GBNF grammar and enforced, so fuse / tweak output actually
71
+ matches `GENOME_SCHEMA` / `GENOME_EDIT_SCHEMA`. This bug masked the real model
72
+ behaviour; fix it before judging any model's capability.
73
+
74
  ## ZeroGPU vs CPU Basic β€” reference baseline
75
 
76
  Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
 
99
  hence the 30/90 split.
100
  - **Hosting needs a ZeroGPU grant** β€” the `build-small-hackathon` org has one
101
  (a personal free account gets a `403`).
102
+ - **Quota exhaustion is visible in the run log** β€” when the org's daily ZeroGPU
103
+ quota runs out, `warmup()` logs `[zerogpu-warmup] failed: ... exceeded ZeroGPU
104
+ quota (... vs. 0s left)` and inference drops to CPU: `[zerogpu-bench] ... 1.9
105
+ tok/s` (vs ~200 on GPU), ~45s per tweak, fuse far worse. At that point tweaks
106
+ also fall to the keyword editor, so the model's real value (Chinese, complex,
107
+ out-of-schema approximations) can't be shown. Repeatedly opening the Space /
108
+ re-warming burns the budget β€” keep it in reserve before a demo. The
109
+ `duration=90` warmup reserves ~135s and retries on every page load while quota
110
+ is exhausted, a pure cost worth trimming or skipping when quota is low.
111
 
112
  ## Space build: the wheel pins (load-bearing)
113
 
 
141
  `Downloading llama_cpp_python-0.3.19-...-manylinux...whl`, **not** a
142
  `.tar.gz` followed by `Building wheel ... still running`.
143
 
144
+ **chat-template ceiling (narrower than feared):** the cu124 cp312 wheel index
145
+ stops at `0.3.19`, and 0.3.19's Jinja engine is old enough that *some* newer
146
+ model families throw an `unknown tag` error. But **Qwen2.5 (3B/7B/14B/32B) runs
147
+ fine on 0.3.19** β€” verified locally against the Space-pinned version: the chat
148
+ template loads and `json_object` schema enforcement works. So adopting Qwen2.5
149
+ needs **no** llama-cpp-python upgrade and no wheel / README pin change. (If a
150
+ future model ever does need a newer build: the cu124 index now ships a
151
+ `py3-none-manylinux_2_35_x86_64` wheel for `0.3.29`, Linux-installable β€” the
152
+ old "cp312 stops at 0.3.19" pin is no longer the only option.)
153
 
154
  ## Model strategy
155
 
156
+ - Default real config (`runtime.py`): **`Qwen2.5-3B-Instruct`** GGUF Q4_K_M
157
+ primary, `SmolLM2-1.7B-Instruct` fallback (the smaller download-failure
158
+ fallback, not a quality fallback).
159
+ - Why Qwen2.5-3B β€” measured locally on the genome edit schema (0.3.19, *after*
160
+ the `json_object` fix above; before the fix every model scored ~0 because the
161
+ schema never applied):
162
+ - `Qwen2.5-7B` 7/7 (incl. Chinese), `Qwen2.5-3B` 6/7 β€” both follow the schema
163
+ and produce genuinely model-driven tweaks, in English and Chinese.
164
+ - `Llama-3.2-3B` (the previous default) **0/7 even with the fix** β€” it overran
165
+ the 200-token edit budget and produced truncated JSON. `SmolLM2-1.7B` 3/7.
166
+ - 3B is the cheapest that works and matches the old footprint (~2GB GGUF); 7B
167
+ is the safest (7/7, ~4.7GB). Chose 3B for cost parity; 7B is the upgrade if
168
+ schema misses ever matter.
169
+ - Out-of-schema instructions ("build a house", "turn it into a robot") still
170
+ no-op or approximate on every model β€” that is a schema/part-kind limit, not
171
+ a model limit.
172
+ - `scripts/start-local-real-model.sh` presets still select SmolLM2 / Llama for
173
+ local iteration. Model loading is lazy β€” the first splice downloads the GGUF
174
+ (or pre-cache via `BUDDY_MODEL_PATH`).
175
+ - The 32B competition cap is an upper bound, not a target: a small model keeps
176
+ the `Tiny Titan` / `Off the Grid` bonus quests in reach (see
177
  [hackathon-context](hackathon-context.md)).
178
 
179
  ## UX constraint
wiki/log.md CHANGED
@@ -2,6 +2,22 @@
2
 
3
  ## 2026-06-14
4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  ### Theme: switched from light-redirect lock to theme-independence
6
 
7
  - Rewrote `gradio-theme-locking.md` (and its `index.md` line): the app no longer
 
2
 
3
  ## 2026-06-14
4
 
5
+ ### deployment-strategy: response_format bug, Qwen2.5-3B default, ZeroGPU quota
6
+
7
+ - Recorded the root-cause fix: `response_format` must be
8
+ `{"type":"json_object","schema":...}`; the OpenAI-style `"json_schema"` is
9
+ silently ignored by llama-cpp-python, so fuse/tweak ran unconstrained and every
10
+ edit fell to the keyword editor. Added a "Schema enforcement" subsection.
11
+ - Updated Model strategy: default is now `Qwen2.5-3B-Instruct` (6/7 schema, incl.
12
+ Chinese; Llama-3.2-3B scored 0/7 even after the fix β€” truncated JSON). Recorded
13
+ the local 0.3.19 measurements and the 3B-vs-7B trade-off.
14
+ - Corrected the chat-template ceiling: Qwen2.5 (3B–32B) runs on the Space-pinned
15
+ 0.3.19, so adopting it needs no llama-cpp-python upgrade; noted the new
16
+ `py3-none-manylinux` 0.3.29 wheel as a future option.
17
+ - Added ZeroGPU quota-exhaustion symptoms (warmup failures, CPU drop to ~1.9
18
+ tok/s) to the operational constraints. Verified against live Space run logs;
19
+ fix is committed (`d9eb5a7`) and deployed.
20
+
21
  ### Theme: switched from light-redirect lock to theme-independence
22
 
23
  - Rewrote `gradio-theme-locking.md` (and its `index.md` line): the app no longer