docs(wiki): record measured Gemma-4-12B freeform timing on ZeroGPU
Browse filesdeployment-strategy.md: add "Freeform 12B timing" β reload ~3s, ~17-22 tok/s
with grammar (25-41s/call), grammar ~2x cost (16.7->35.9 on/off, intrinsic to
grammar-constrained decode), the duration=30->60 abort fix, the 1500-token >60s
edge, and that ZeroGPU aborts a 3rd GPU acquisition stacked in one demo.load
handler. Bump wrapper list to three @spaces.GPU wrappers.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- wiki/deployment-strategy.md +48 -4
- wiki/log.md +15 -0
wiki/deployment-strategy.md
CHANGED
|
@@ -28,10 +28,16 @@ interprets voice edits (`interpret_genome_command`); a deterministic fallback in
|
|
| 28 |
## Runtime shape (ZeroGPU)
|
| 29 |
|
| 30 |
All model calls funnel through `_blocking_completion` in
|
| 31 |
-
`src/buddy_fusion/runtime.py`, exposed as
|
| 32 |
-
|
| 33 |
-
- `_run_completion` at `duration=30` β
|
| 34 |
-
queues better.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
- `_warmup_completion` at `duration=90` β covers the one-time cold CUDA kernel
|
| 36 |
JIT compile.
|
| 37 |
|
|
@@ -86,6 +92,44 @@ is compiled to a GBNF grammar and enforced, so fuse / tweak output actually
|
|
| 86 |
matches `GENOME_SCHEMA` / `GENOME_EDIT_SCHEMA`. This bug masked the real model
|
| 87 |
behaviour; fix it before judging any model's capability.
|
| 88 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 89 |
## ZeroGPU vs CPU Basic β reference baseline
|
| 90 |
|
| 91 |
Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
|
|
|
|
| 28 |
## Runtime shape (ZeroGPU)
|
| 29 |
|
| 30 |
All model calls funnel through `_blocking_completion` in
|
| 31 |
+
`src/buddy_fusion/runtime.py`, exposed as three `@spaces.GPU` wrappers:
|
| 32 |
+
|
| 33 |
+
- `_run_completion` at `duration=30` β recipe-level Qwen-3B genome/edit calls;
|
| 34 |
+
warm calls are ~1β2s, so a small budget queues better.
|
| 35 |
+
- `_run_freeform_completion` at `duration=60` (`BUDDY_FREEFORM_GPU_DURATION`) β
|
| 36 |
+
the Gemma-4-12B box-geometry calls (generate / fuse / edit). These generate
|
| 37 |
+
300β800+ tokens at ~20 tok/s and hit 25β41s total; on the old shared
|
| 38 |
+
`duration=30` they overran the budget and ZeroGPU killed them mid-decode
|
| 39 |
+
(`ggml_cuda_mul_mat_q` CUDA abort β the call dies and the user gets the
|
| 40 |
+
fallback after a full wait). See "Freeform 12B timing" below.
|
| 41 |
- `_warmup_completion` at `duration=90` β covers the one-time cold CUDA kernel
|
| 42 |
JIT compile.
|
| 43 |
|
|
|
|
| 92 |
matches `GENOME_SCHEMA` / `GENOME_EDIT_SCHEMA`. This bug masked the real model
|
| 93 |
behaviour; fix it before judging any model's capability.
|
| 94 |
|
| 95 |
+
### Freeform 12B timing β measured on the Space (2026-06-15)
|
| 96 |
+
|
| 97 |
+
Gemma-4-12B-it Q4_K_M on ZeroGPU, from the `[zerogpu-bench]` log lines (the
|
| 98 |
+
`_blocking_completion` print of `reload` / `gen` / `tok/s`):
|
| 99 |
+
|
| 100 |
+
- **reload** (rebuild `Llama` + upload ~7GB GGUF to VRAM each call): ~3s β the
|
| 101 |
+
per-call cost is small and constant; it is **not** the bottleneck.
|
| 102 |
+
- **decode**: **~17β22 tok/s with the grammar on**, generating 300β800 tokens β
|
| 103 |
+
**25β41s total** per generate / fuse / edit. The wait is essentially all
|
| 104 |
+
`gen`; cutting it means cutting tokens or raising tok/s.
|
| 105 |
+
- **warmup** correctly pays the one-time JIT: an 8-token warmup call logs
|
| 106 |
+
`genβ73s` once per container, so the JIT is **not** a per-call cost afterward.
|
| 107 |
+
|
| 108 |
+
**Grammar costs ~2Γ.** Same 400-token prompt, identical config, grammar on vs
|
| 109 |
+
off (`BUDDY_GRAMMAR_BENCH=1` runs the unconstrained `[grammar-bench-off]` call):
|
| 110 |
+
**16.7 tok/s on β 35.9 tok/s off**. The GBNF per-token allowed-set evaluation is
|
| 111 |
+
serial CPU work that does not ride the GPU, so it roughly halves throughput.
|
| 112 |
+
This is **intrinsic to grammar-constrained decode, not schema bloat** β `BOX_SCHEMA`
|
| 113 |
+
is already lean (plain-int/str fields, no regex, no enum on the box path), so
|
| 114 |
+
simplifying the schema buys little. The lever is to drop the grammar on the
|
| 115 |
+
freeform box path and lean on `parse_json_object` + `repair_boxes` (keep it on
|
| 116 |
+
the recipe `GENOME_*` path, which is fragile without it β see above). Even
|
| 117 |
+
unconstrained, 35.9 tok/s is still ~3Γ under what a 12B Q4 should hit on the
|
| 118 |
+
ZeroGPU Blackwell, so a second, harder lever (Gemma iSWA `full-size SWA cache`
|
| 119 |
+
path / `flash_attn` / quant) remains.
|
| 120 |
+
|
| 121 |
+
**Residual crash edge:** `duration=60` covers the observed 25β41s calls, but the
|
| 122 |
+
`BUDDY_FREEFORM_MAX_TOKENS=1500` cap at ~20 tok/s is ~75s > 60s β a max-length
|
| 123 |
+
generation can still overrun and abort. Lower the token cap (~900β1000, which
|
| 124 |
+
also trims the typical wait) or raise the duration.
|
| 125 |
+
|
| 126 |
+
**ZeroGPU aborts a 3rd GPU acquisition stacked in one request.** Inside a single
|
| 127 |
+
`demo.load` handler, `warmup` (90s slot) β freeform (60s) β freeform (60s) ran
|
| 128 |
+
the first two fine but ZeroGPU killed the third with `GPU task aborted`. Separate
|
| 129 |
+
user-action calls (each its own request) are unaffected β so production is fine,
|
| 130 |
+
but a diagnostic that fires several GPU calls from one handler must keep to β€2
|
| 131 |
+
acquisitions (why `_grammar_bench` measures only the off run).
|
| 132 |
+
|
| 133 |
## ZeroGPU vs CPU Basic β reference baseline
|
| 134 |
|
| 135 |
Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
|
wiki/log.md
CHANGED
|
@@ -2,6 +2,21 @@
|
|
| 2 |
|
| 3 |
## 2026-06-15
|
| 4 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5 |
### voxel-rendering + deployment-strategy: Figma canvas (pan + pooled live 3D) and one-call bench tweak
|
| 6 |
|
| 7 |
- `voxel-rendering.md`: added "Lab canvas: pan + viewport-pooled live 3D" β tree
|
|
|
|
| 2 |
|
| 3 |
## 2026-06-15
|
| 4 |
|
| 5 |
+
### deployment-strategy: Freeform 12B timing measured on the Space
|
| 6 |
+
|
| 7 |
+
- Added a "Freeform 12B timing" subsection: Gemma-4-12B Q4 on ZeroGPU reloads
|
| 8 |
+
~3s and decodes ~17β22 tok/s with the grammar (25β41s per generate/fuse/edit);
|
| 9 |
+
the wait is essentially all `gen`. warmup correctly pays the one-time ~73s JIT.
|
| 10 |
+
- Grammar costs ~2Γ (16.7 β 35.9 tok/s on/off, identical 400-token bench) and the
|
| 11 |
+
cost is intrinsic to grammar-constrained decode β `BOX_SCHEMA` is already lean
|
| 12 |
+
(no regex/enum), so the lever is dropping the grammar on the freeform box path
|
| 13 |
+
(keep it on the recipe `GENOME_*` path) + leaning on `repair_boxes`. Even
|
| 14 |
+
unconstrained, 35.9 tok/s is ~3Γ under the Blackwell ceiling (iSWA/flash_attn).
|
| 15 |
+
- Documented the `duration=30`β`60` (`_run_freeform_completion`) fix for the
|
| 16 |
+
`ggml_cuda_mul_mat_q` mid-decode abort, the residual 1500-token > 60s edge, and
|
| 17 |
+
that ZeroGPU aborts a 3rd GPU acquisition stacked in one `demo.load` handler
|
| 18 |
+
(`GPU task aborted`). Updated the "Runtime shape" wrapper list to three.
|
| 19 |
+
|
| 20 |
### voxel-rendering + deployment-strategy: Figma canvas (pan + pooled live 3D) and one-call bench tweak
|
| 21 |
|
| 22 |
- `voxel-rendering.md`: added "Lab canvas: pan + viewport-pooled live 3D" β tree
|