arkai2025 Claude Opus 4.8 (1M context) commited on
Commit
5be36f9
Β·
1 Parent(s): edd5989

docs(wiki): record measured Gemma-4-12B freeform timing on ZeroGPU

Browse files

deployment-strategy.md: add "Freeform 12B timing" β€” reload ~3s, ~17-22 tok/s
with grammar (25-41s/call), grammar ~2x cost (16.7->35.9 on/off, intrinsic to
grammar-constrained decode), the duration=30->60 abort fix, the 1500-token >60s
edge, and that ZeroGPU aborts a 3rd GPU acquisition stacked in one demo.load
handler. Bump wrapper list to three @spaces.GPU wrappers.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Files changed (2) hide show
  1. wiki/deployment-strategy.md +48 -4
  2. wiki/log.md +15 -0
wiki/deployment-strategy.md CHANGED
@@ -28,10 +28,16 @@ interprets voice edits (`interpret_genome_command`); a deterministic fallback in
28
  ## Runtime shape (ZeroGPU)
29
 
30
  All model calls funnel through `_blocking_completion` in
31
- `src/buddy_fusion/runtime.py`, exposed as two `@spaces.GPU` wrappers:
32
-
33
- - `_run_completion` at `duration=30` β€” warm calls are ~1–2s; a small budget
34
- queues better.
 
 
 
 
 
 
35
  - `_warmup_completion` at `duration=90` β€” covers the one-time cold CUDA kernel
36
  JIT compile.
37
 
@@ -86,6 +92,44 @@ is compiled to a GBNF grammar and enforced, so fuse / tweak output actually
86
  matches `GENOME_SCHEMA` / `GENOME_EDIT_SCHEMA`. This bug masked the real model
87
  behaviour; fix it before judging any model's capability.
88
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
  ## ZeroGPU vs CPU Basic β€” reference baseline
90
 
91
  Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
 
28
  ## Runtime shape (ZeroGPU)
29
 
30
  All model calls funnel through `_blocking_completion` in
31
+ `src/buddy_fusion/runtime.py`, exposed as three `@spaces.GPU` wrappers:
32
+
33
+ - `_run_completion` at `duration=30` β€” recipe-level Qwen-3B genome/edit calls;
34
+ warm calls are ~1–2s, so a small budget queues better.
35
+ - `_run_freeform_completion` at `duration=60` (`BUDDY_FREEFORM_GPU_DURATION`) β€”
36
+ the Gemma-4-12B box-geometry calls (generate / fuse / edit). These generate
37
+ 300–800+ tokens at ~20 tok/s and hit 25–41s total; on the old shared
38
+ `duration=30` they overran the budget and ZeroGPU killed them mid-decode
39
+ (`ggml_cuda_mul_mat_q` CUDA abort β†’ the call dies and the user gets the
40
+ fallback after a full wait). See "Freeform 12B timing" below.
41
  - `_warmup_completion` at `duration=90` β€” covers the one-time cold CUDA kernel
42
  JIT compile.
43
 
 
92
  matches `GENOME_SCHEMA` / `GENOME_EDIT_SCHEMA`. This bug masked the real model
93
  behaviour; fix it before judging any model's capability.
94
 
95
+ ### Freeform 12B timing β€” measured on the Space (2026-06-15)
96
+
97
+ Gemma-4-12B-it Q4_K_M on ZeroGPU, from the `[zerogpu-bench]` log lines (the
98
+ `_blocking_completion` print of `reload` / `gen` / `tok/s`):
99
+
100
+ - **reload** (rebuild `Llama` + upload ~7GB GGUF to VRAM each call): ~3s β€” the
101
+ per-call cost is small and constant; it is **not** the bottleneck.
102
+ - **decode**: **~17–22 tok/s with the grammar on**, generating 300–800 tokens β†’
103
+ **25–41s total** per generate / fuse / edit. The wait is essentially all
104
+ `gen`; cutting it means cutting tokens or raising tok/s.
105
+ - **warmup** correctly pays the one-time JIT: an 8-token warmup call logs
106
+ `genβ‰ˆ73s` once per container, so the JIT is **not** a per-call cost afterward.
107
+
108
+ **Grammar costs ~2Γ—.** Same 400-token prompt, identical config, grammar on vs
109
+ off (`BUDDY_GRAMMAR_BENCH=1` runs the unconstrained `[grammar-bench-off]` call):
110
+ **16.7 tok/s on β†’ 35.9 tok/s off**. The GBNF per-token allowed-set evaluation is
111
+ serial CPU work that does not ride the GPU, so it roughly halves throughput.
112
+ This is **intrinsic to grammar-constrained decode, not schema bloat** β€” `BOX_SCHEMA`
113
+ is already lean (plain-int/str fields, no regex, no enum on the box path), so
114
+ simplifying the schema buys little. The lever is to drop the grammar on the
115
+ freeform box path and lean on `parse_json_object` + `repair_boxes` (keep it on
116
+ the recipe `GENOME_*` path, which is fragile without it β€” see above). Even
117
+ unconstrained, 35.9 tok/s is still ~3Γ— under what a 12B Q4 should hit on the
118
+ ZeroGPU Blackwell, so a second, harder lever (Gemma iSWA `full-size SWA cache`
119
+ path / `flash_attn` / quant) remains.
120
+
121
+ **Residual crash edge:** `duration=60` covers the observed 25–41s calls, but the
122
+ `BUDDY_FREEFORM_MAX_TOKENS=1500` cap at ~20 tok/s is ~75s > 60s β€” a max-length
123
+ generation can still overrun and abort. Lower the token cap (~900–1000, which
124
+ also trims the typical wait) or raise the duration.
125
+
126
+ **ZeroGPU aborts a 3rd GPU acquisition stacked in one request.** Inside a single
127
+ `demo.load` handler, `warmup` (90s slot) β†’ freeform (60s) β†’ freeform (60s) ran
128
+ the first two fine but ZeroGPU killed the third with `GPU task aborted`. Separate
129
+ user-action calls (each its own request) are unaffected β€” so production is fine,
130
+ but a diagnostic that fires several GPU calls from one handler must keep to ≀2
131
+ acquisitions (why `_grammar_bench` measures only the off run).
132
+
133
  ## ZeroGPU vs CPU Basic β€” reference baseline
134
 
135
  Measured on the sibling project (`build-a-buddy-hf`) with the **identical**
wiki/log.md CHANGED
@@ -2,6 +2,21 @@
2
 
3
  ## 2026-06-15
4
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
5
  ### voxel-rendering + deployment-strategy: Figma canvas (pan + pooled live 3D) and one-call bench tweak
6
 
7
  - `voxel-rendering.md`: added "Lab canvas: pan + viewport-pooled live 3D" β€” tree
 
2
 
3
  ## 2026-06-15
4
 
5
+ ### deployment-strategy: Freeform 12B timing measured on the Space
6
+
7
+ - Added a "Freeform 12B timing" subsection: Gemma-4-12B Q4 on ZeroGPU reloads
8
+ ~3s and decodes ~17–22 tok/s with the grammar (25–41s per generate/fuse/edit);
9
+ the wait is essentially all `gen`. warmup correctly pays the one-time ~73s JIT.
10
+ - Grammar costs ~2Γ— (16.7 β†’ 35.9 tok/s on/off, identical 400-token bench) and the
11
+ cost is intrinsic to grammar-constrained decode β€” `BOX_SCHEMA` is already lean
12
+ (no regex/enum), so the lever is dropping the grammar on the freeform box path
13
+ (keep it on the recipe `GENOME_*` path) + leaning on `repair_boxes`. Even
14
+ unconstrained, 35.9 tok/s is ~3Γ— under the Blackwell ceiling (iSWA/flash_attn).
15
+ - Documented the `duration=30`β†’`60` (`_run_freeform_completion`) fix for the
16
+ `ggml_cuda_mul_mat_q` mid-decode abort, the residual 1500-token > 60s edge, and
17
+ that ZeroGPU aborts a 3rd GPU acquisition stacked in one `demo.load` handler
18
+ (`GPU task aborted`). Updated the "Runtime shape" wrapper list to three.
19
+
20
  ### voxel-rendering + deployment-strategy: Figma canvas (pan + pooled live 3D) and one-call bench tweak
21
 
22
  - `voxel-rendering.md`: added "Lab canvas: pan + viewport-pooled live 3D" β€” tree