Spaces:
Running
Running
Stefatorus Claude Opus 5.5 commited on
Commit ·
146ba97
1
Parent(s): 2dc8d66
Faster WebGPU sampling: fp16 compute, GPU-resident steps, non-blocking thinker preview
Browse files- 002/003 and 001 load the v2 build from webgpu/v2/ in each model repo: one graph run per Euler step (CFG mix +
update inside the graph), z stays on the GPU between steps, fp16 compute (fp32 for graph I/O, CFG/Euler,
timestep/resolution/count embeddings and Range), 1x1 conv + depth-to-space instead of ConvTranspose.
- 003 @ 512: 275 -> 161 ms/step, 14 -> 8.1 s/image; 256 px and 001/002: ~6 -> ~3.4 s (RTX 4060, Edge).
- Show thinker: async readback, colouring in a Web Worker, frames dropped while busy; no cost to sampling.
- "Fast" toggle: 25 steps (~4.1 s at 512); default stays 50.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018EtDckYfNgMbkwqVmTsdha
- README.md +18 -12
- css/style.css +2 -0
- index.html +2 -1
- js/agate.js +76 -10
- js/app.js +12 -9
- js/thinker-worker.js +11 -0
- js/thinker.js +65 -32
README.md
CHANGED
|
@@ -37,14 +37,14 @@ this browser.
|
|
| 37 |
| [Preview 001](https://huggingface.co/Logolabs/agate-preview-001) | 256 px | the first preview | 530 MB |
|
| 38 |
|
| 39 |
Each model is downloaded once and kept in the browser's Cache Storage. The files live in each release's model repo
|
| 40 |
-
under `webgpu/` (e.g. [Logolabs/agate-preview-003/webgpu](https://huggingface.co/Logolabs/agate-preview-003/tree/main/webgpu)),
|
| 41 |
with a `manifest.json` holding sizes and sha256s (the cache keys). `models/` in this Space is the original 001 build
|
| 42 |
(without the plan output), kept for `?models=./models/`.
|
| 43 |
|
| 44 |
## Show thinker
|
| 45 |
|
| 46 |
Agate's thinker lays the picture out on a 16 × 16 grid (the plan) that steers the renderer. With **Show thinker**
|
| 47 |
-
(on by default; it
|
| 48 |
colours and the image the model currently expects (x₁ = z + (1 − t)·v, with the SD latent→RGB approximation) —
|
| 49 |
the same live preview as the [ComfyUI nodes](https://github.com/logolabs/agate-comfyui): the plan's cells are
|
| 50 |
projected on its top three principal components, fitted at the first step with the 2% / 98% quantiles fixed then,
|
|
@@ -55,18 +55,24 @@ so a colour keeps its meaning while the plan evolves.
|
|
| 55 |
A browser with WebGPU: a recent Chrome or Edge (113+). Safari and Firefox may lack WebGPU or need it enabled in
|
| 56 |
their settings; without it the page offers a WebAssembly (CPU) path that works but takes minutes per image.
|
| 57 |
|
| 58 |
-
## Speed and accuracy (RTX 4060 laptop, Edge, WebGPU, warm)
|
| 59 |
|
| 60 |
-
| model | per step (CFG batch of 2) |
|
| 61 |
|---|---|---|---|
|
| 62 |
-
| 003 at 512 px | ~
|
| 63 |
-
| 003 at 256 px | ~
|
| 64 |
-
| 002 / 001 | ~
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
|
| 68 |
-
|
| 69 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
Seeds use a JavaScript PRNG, so a seed gives the same image in every browser, but not the same image as the
|
| 71 |
PyTorch pipeline with that seed. The Python packages decode with the full SD-VAE by default; this page uses TAESD.
|
| 72 |
|
|
|
|
| 37 |
| [Preview 001](https://huggingface.co/Logolabs/agate-preview-001) | 256 px | the first preview | 530 MB |
|
| 38 |
|
| 39 |
Each model is downloaded once and kept in the browser's Cache Storage. The files live in each release's model repo
|
| 40 |
+
under `webgpu/v2/` (e.g. [Logolabs/agate-preview-003/webgpu/v2](https://huggingface.co/Logolabs/agate-preview-003/tree/main/webgpu/v2)),
|
| 41 |
with a `manifest.json` holding sizes and sha256s (the cache keys). `models/` in this Space is the original 001 build
|
| 42 |
(without the plan output), kept for `?models=./models/`.
|
| 43 |
|
| 44 |
## Show thinker
|
| 45 |
|
| 46 |
Agate's thinker lays the picture out on a 16 × 16 grid (the plan) that steers the renderer. With **Show thinker**
|
| 47 |
+
(on by default; it does not slow sampling: previews are read back asynchronously and coloured in a Web Worker) the page shows, at every step, the plan of the conditional branch as
|
| 48 |
colours and the image the model currently expects (x₁ = z + (1 − t)·v, with the SD latent→RGB approximation) —
|
| 49 |
the same live preview as the [ComfyUI nodes](https://github.com/logolabs/agate-comfyui): the plan's cells are
|
| 50 |
projected on its top three principal components, fitted at the first step with the 2% / 98% quantiles fixed then,
|
|
|
|
| 55 |
A browser with WebGPU: a recent Chrome or Edge (113+). Safari and Firefox may lack WebGPU or need it enabled in
|
| 56 |
their settings; without it the page offers a WebAssembly (CPU) path that works but takes minutes per image.
|
| 57 |
|
| 58 |
+
## Speed and accuracy (RTX 4060 laptop, Edge, WebGPU, warm, 50 steps)
|
| 59 |
|
| 60 |
+
| model | per step (CFG batch of 2) | image | before (2026-09-29 morning) |
|
| 61 |
|---|---|---|---|
|
| 62 |
+
| 003 at 512 px | ~161 ms | **~8.1 s** (Fast, 25 steps: ~4.1 s) | 275 ms, 14 s |
|
| 63 |
+
| 003 at 256 px | ~70 ms | ~3.5 s | 120 ms, 6.1 s |
|
| 64 |
+
| 002 / 001 | ~65 ms | ~3.3 s | 105 ms, 5.3 s |
|
| 65 |
+
|
| 66 |
+
How: each step is one graph run that also does the CFG mix and the Euler update, so the latent never leaves the
|
| 67 |
+
GPU between steps (the only readback is the final latent); the network computes in **fp16** (WebGPU `shader-f16`;
|
| 68 |
+
WGSL has no bf16), with fp32 kept for the graph I/O, the CFG/Euler arithmetic, the timestep / resolution / count
|
| 69 |
+
embeddings (sin(1000·t·f) needs fp32) and `Range`; the transposed-conv upsamplers run as 1x1 conv + depth-to-space.
|
| 70 |
+
The thinker preview never blocks sampling: its tensors are downloaded asynchronously, coloured in a Web Worker,
|
| 71 |
+
and frames are dropped while one is pending (measured cost: none).
|
| 72 |
+
|
| 73 |
+
fp16 compute moves the result slightly: against the fp32 PyTorch reference with the same initial noise, 001 is
|
| 74 |
+
within 16/255 (PSNR 48 dB), 002 36 dB, 003 at 256 px 42 dB and 003 at 512 px 31 dB (mean 0.6-1.8/255). For scale,
|
| 75 |
+
the Python package's own CUDA path (bf16) is 14 dB from that fp32 reference for the same noise. Run `?parity=1`.
|
| 76 |
Seeds use a JavaScript PRNG, so a seed gives the same image in every browser, but not the same image as the
|
| 77 |
PyTorch pipeline with that seed. The Python packages decode with the full SD-VAE by default; this page uses TAESD.
|
| 78 |
|
css/style.css
CHANGED
|
@@ -130,3 +130,5 @@ input[type=text] { width: 100%; font: 15px/1.45 var(--sans); color: var(--ink);
|
|
| 130 |
.thinker #pred-canvas { image-rendering: auto; }
|
| 131 |
.thinker figcaption { margin-top: 6px; }
|
| 132 |
.thinker-note { grid-column: 1 / -1; margin: 0; font-size: 13px; color: var(--mute); }
|
|
|
|
|
|
|
|
|
| 130 |
.thinker #pred-canvas { image-rendering: auto; }
|
| 131 |
.thinker figcaption { margin-top: 6px; }
|
| 132 |
.thinker-note { grid-column: 1 / -1; margin: 0; font-size: 13px; color: var(--mute); }
|
| 133 |
+
.label .toggle.fast { margin-left: 10px; text-transform: uppercase; letter-spacing: .1em; color: var(--ink); }
|
| 134 |
+
.label .toggle.fast input { accent-color: var(--red); margin: 0 4px 0 0; vertical-align: -1px; }
|
index.html
CHANGED
|
@@ -68,7 +68,8 @@
|
|
| 68 |
</div>
|
| 69 |
</div>
|
| 70 |
<div class="field">
|
| 71 |
-
<label class="label" for="steps">Steps <b id="steps-v">50</b>
|
|
|
|
| 72 |
<input id="steps" type="range" min="4" max="50" step="1" value="50">
|
| 73 |
</div>
|
| 74 |
<div class="field">
|
|
|
|
| 68 |
</div>
|
| 69 |
</div>
|
| 70 |
<div class="field">
|
| 71 |
+
<label class="label" for="steps">Steps <b id="steps-v">50</b>
|
| 72 |
+
<span class="toggle fast" title="25 steps: about twice as fast; detail holds, the layout can change"><input type="checkbox" id="fast"> Fast</span></label>
|
| 73 |
<input id="steps" type="range" min="4" max="50" step="1" value="50">
|
| 74 |
</div>
|
| 75 |
<div class="field">
|
js/agate.js
CHANGED
|
@@ -15,7 +15,7 @@ export { ort };
|
|
| 15 |
// (001 without the plan output) is used only through ?models=. dir: the layout under a local
|
| 16 |
// models base (?models=<url> or window.AGATE_MODEL_BASE given explicitly, e.g. for local testing). cache: Cache
|
| 17 |
// Storage name (001 keeps the name it always had, so returning visitors do not download it again).
|
| 18 |
-
const HUB = (v) => `https://huggingface.co/Logolabs/agate-preview-${v}/resolve/main/webgpu/`;
|
| 19 |
export const VERSIONS = {
|
| 20 |
"003": { remote: HUB("003"), dir: "003/", cache: "agate-preview-003-web", res: [512, 256], note: "Multi-resolution model, 512 px (or 256). Quoted text is spelled out, counts are coded, “no X” becomes a negative prompt." },
|
| 21 |
"002": { remote: HUB("002"), dir: "002/", cache: "agate-preview-002-web", res: [256], note: "Same network as 001, trained 38,710 steps longer before the same anneal. 256 px, plain prompts." },
|
|
@@ -99,6 +99,9 @@ export class Agate {
|
|
| 99 |
|
| 100 |
get mr() { return this.manifest?.arch === "fcdm_t2mr"; }
|
| 101 |
|
|
|
|
|
|
|
|
|
|
| 102 |
fileKey(f) { return new Request(`${this.base}${f}?sha256=${this.manifest.files[f].sha256}`); }
|
| 103 |
|
| 104 |
async getFile(f, onBytes = () => {}) {
|
|
@@ -115,7 +118,7 @@ export class Agate {
|
|
| 115 |
this.resolution = this.mr ? Number(resolution || manifest.resolution) : 256;
|
| 116 |
// 003: download everything (the two graphs are ~1 MB each and share one weights file), so a resolution
|
| 117 |
// switch needs no network
|
| 118 |
-
const need = this.mr ? Object.keys(manifest.files) : FILES_T2;
|
| 119 |
const total = need.reduce((s, f) => s + manifest.files[real(f)].bytes, 0);
|
| 120 |
let loaded = 0, fromCache = 0;
|
| 121 |
const bufs = {};
|
|
@@ -151,7 +154,10 @@ export class Agate {
|
|
| 151 |
}
|
| 152 |
onProgress({ phase: "init", file: "generator", loaded: total, total });
|
| 153 |
const ts = performance.now();
|
| 154 |
-
if (this.
|
|
|
|
|
|
|
|
|
|
| 155 |
this.weights = bufs["generator.weights"]; // kept for a resolution switch (no re-read)
|
| 156 |
await this.createGenerator(this.resolution, bufs);
|
| 157 |
} else {
|
|
@@ -166,12 +172,20 @@ export class Agate {
|
|
| 166 |
|
| 167 |
// 003: the generator graph for one resolution; the weights file is shared by both graphs.
|
| 168 |
async createGenerator(res, bufs = {}) {
|
| 169 |
-
|
| 170 |
-
if (
|
| 171 |
-
|
| 172 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 173 |
if (this.sessions.gen) { try { await this.sessions.gen.release(); } catch { /* */ } this.sessions.gen = null; }
|
| 174 |
-
this.sessions.gen = await ort.InferenceSession.create(graph, this.sessionOptions({ externalData: [{ path:
|
| 175 |
this.resolution = Number(res);
|
| 176 |
}
|
| 177 |
|
|
@@ -240,9 +254,57 @@ export class Agate {
|
|
| 240 |
// -> { rgba (watermarked unless watermark=false), width, height, latent, timings, prepared }
|
| 241 |
get hasPlan() { return !!this.sessions?.gen?.outputNames?.includes("plan"); }
|
| 242 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 243 |
// onPreview({ step, steps, plan, x1, hw }) after every step, when given and the graph outputs the plan: plan =
|
| 244 |
// the thinker output for the conditional branch (Float32Array 640 x 16 x 16), x1 = z + (1 - t) v (guided).
|
| 245 |
-
async generate({ prompt, negative = "", seed = 0, steps = 50, cfg = 3.0, noise = null, watermark = true, onStep = () => {}, onPreview = null, shouldStop = () => false }) {
|
| 246 |
const C = this.manifest.latent_ch, D = this.manifest.ctx_dim;
|
| 247 |
const rc = this.mr ? this.manifest.resolutions[String(this.resolution)] : { latent_hw: this.manifest.latent_hw, shift: 1 };
|
| 248 |
const hw = rc.latent_hw, shift = Number(rc.shift || 1);
|
|
@@ -268,10 +330,13 @@ export class Agate {
|
|
| 268 |
}
|
| 269 |
|
| 270 |
let z = noise ? Float32Array.from(noise) : Agate.gaussianNoise(seed, per);
|
| 271 |
-
const zz = new Float32Array(2 * per), tt = new Float32Array(2);
|
| 272 |
const grid = Array.from({ length: steps + 1 }, (_, i) => Agate.shiftT(i / steps, shift));
|
| 273 |
t0 = performance.now();
|
| 274 |
const stepMs = [];
|
|
|
|
|
|
|
|
|
|
|
|
|
| 275 |
for (let i = 0; i < steps; i++) {
|
| 276 |
if (shouldStop()) throw new Error("stopped");
|
| 277 |
const ts = performance.now();
|
|
@@ -296,6 +361,7 @@ export class Agate {
|
|
| 296 |
onStep(i + 1, steps, z);
|
| 297 |
await new Promise((r) => setTimeout(r, 0)); // let the progress bar paint
|
| 298 |
}
|
|
|
|
| 299 |
T.samplerMs = performance.now() - t0;
|
| 300 |
T.firstStepMs = stepMs[0];
|
| 301 |
T.stepMs = stepMs.length > 1 ? (T.samplerMs - stepMs[0]) / (stepMs.length - 1) : stepMs[0];
|
|
|
|
| 15 |
// (001 without the plan output) is used only through ?models=. dir: the layout under a local
|
| 16 |
// models base (?models=<url> or window.AGATE_MODEL_BASE given explicitly, e.g. for local testing). cache: Cache
|
| 17 |
// Storage name (001 keeps the name it always had, so returning visitors do not download it again).
|
| 18 |
+
const HUB = (v) => `https://huggingface.co/Logolabs/agate-preview-${v}/resolve/main/webgpu/v2/`; // v2: fast fp16 step graphs
|
| 19 |
export const VERSIONS = {
|
| 20 |
"003": { remote: HUB("003"), dir: "003/", cache: "agate-preview-003-web", res: [512, 256], note: "Multi-resolution model, 512 px (or 256). Quoted text is spelled out, counts are coded, “no X” becomes a negative prompt." },
|
| 21 |
"002": { remote: HUB("002"), dir: "002/", cache: "agate-preview-002-web", res: [256], note: "Same network as 001, trained 38,710 steps longer before the same anneal. 256 px, plain prompts." },
|
|
|
|
| 99 |
|
| 100 |
get mr() { return this.manifest?.arch === "fcdm_t2mr"; }
|
| 101 |
|
| 102 |
+
// the fast build (2026-09-29): one Euler step per run, fp16 compute, everything stays on the GPU between steps
|
| 103 |
+
get fast() { return !!this.manifest?.step; }
|
| 104 |
+
|
| 105 |
fileKey(f) { return new Request(`${this.base}${f}?sha256=${this.manifest.files[f].sha256}`); }
|
| 106 |
|
| 107 |
async getFile(f, onBytes = () => {}) {
|
|
|
|
| 118 |
this.resolution = this.mr ? Number(resolution || manifest.resolution) : 256;
|
| 119 |
// 003: download everything (the two graphs are ~1 MB each and share one weights file), so a resolution
|
| 120 |
// switch needs no network
|
| 121 |
+
const need = this.mr || manifest.step ? Object.keys(manifest.files) : FILES_T2;
|
| 122 |
const total = need.reduce((s, f) => s + manifest.files[real(f)].bytes, 0);
|
| 123 |
let loaded = 0, fromCache = 0;
|
| 124 |
const bufs = {};
|
|
|
|
| 154 |
}
|
| 155 |
onProgress({ phase: "init", file: "generator", loaded: total, total });
|
| 156 |
const ts = performance.now();
|
| 157 |
+
if (this.fast) {
|
| 158 |
+
this.weights = bufs[manifest.step.external_data]; // kept for a resolution switch (no re-read)
|
| 159 |
+
await this.createGenerator(this.resolution, bufs);
|
| 160 |
+
} else if (this.mr) {
|
| 161 |
this.weights = bufs["generator.weights"]; // kept for a resolution switch (no re-read)
|
| 162 |
await this.createGenerator(this.resolution, bufs);
|
| 163 |
} else {
|
|
|
|
| 172 |
|
| 173 |
// 003: the generator graph for one resolution; the weights file is shared by both graphs.
|
| 174 |
async createGenerator(res, bufs = {}) {
|
| 175 |
+
let graphName, ext, extra = {};
|
| 176 |
+
if (this.fast) {
|
| 177 |
+
graphName = this.manifest.step.graphs[String(res)];
|
| 178 |
+
ext = this.manifest.step.external_data;
|
| 179 |
+
extra = { preferredOutputLocation: "gpu-buffer" }; // z_next / x1 / plan stay on the GPU
|
| 180 |
+
} else {
|
| 181 |
+
const r = this.manifest.resolutions[String(res)];
|
| 182 |
+
graphName = r?.graph; ext = r?.external_data;
|
| 183 |
+
}
|
| 184 |
+
if (!graphName) throw new Error(`resolution ${res} not available`);
|
| 185 |
+
const graph = bufs[graphName] || await this.getFile(graphName);
|
| 186 |
+
const weights = this.weights || await this.getFile(ext);
|
| 187 |
if (this.sessions.gen) { try { await this.sessions.gen.release(); } catch { /* */ } this.sessions.gen = null; }
|
| 188 |
+
this.sessions.gen = await ort.InferenceSession.create(graph, this.sessionOptions({ externalData: [{ path: ext, data: weights }], ...extra }));
|
| 189 |
this.resolution = Number(res);
|
| 190 |
}
|
| 191 |
|
|
|
|
| 254 |
// -> { rgba (watermarked unless watermark=false), width, height, latent, timings, prepared }
|
| 255 |
get hasPlan() { return !!this.sessions?.gen?.outputNames?.includes("plan"); }
|
| 256 |
|
| 257 |
+
// GPU-resident sampler for the step graphs: z_next of one run is the z input of the next, ctx / mask / counts
|
| 258 |
+
// are uploaded once, and nothing is read back until the end. The preview (plan + x1) is requested only when
|
| 259 |
+
// onPreview is set, previewReady() says the viewer is free and no earlier readback is still in flight; its
|
| 260 |
+
// download is started but never awaited by the loop (frames are dropped instead).
|
| 261 |
+
async sampleFast({ z, grid, cfg, hw, C, L, D, ctx, mask, cnt, onStep, onPreview, previewReady, shouldStop, stepMs }) {
|
| 262 |
+
const dev = ort.env.webgpu?.device;
|
| 263 |
+
const bufs = [];
|
| 264 |
+
const gpuT = (data, dims) => {
|
| 265 |
+
if (!dev) return new ort.Tensor("float32", data, dims);
|
| 266 |
+
const buf = dev.createBuffer({ size: Math.ceil(data.byteLength / 16) * 16, usage: GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_SRC | GPUBufferUsage.COPY_DST });
|
| 267 |
+
dev.queue.writeBuffer(buf, 0, data);
|
| 268 |
+
bufs.push(buf);
|
| 269 |
+
return ort.Tensor.fromGpuBuffer(buf, { dataType: "float32", dims });
|
| 270 |
+
};
|
| 271 |
+
const fixed = { ctx: gpuT(ctx, [2, L, D]), mask: gpuT(mask, [2, L]) };
|
| 272 |
+
if (this.sessions.gen.inputNames.includes("counts")) fixed.counts = gpuT(cnt || new Float32Array(2 * L), [2, L]);
|
| 273 |
+
const steps = grid.length - 1;
|
| 274 |
+
let zT = new ort.Tensor("float32", z, [1, C, hw, hw]), inflight = false;
|
| 275 |
+
const one = (v) => new ort.Tensor("float32", new Float32Array([v]), [1]);
|
| 276 |
+
const cfgT = one(cfg);
|
| 277 |
+
try {
|
| 278 |
+
for (let i = 0; i < steps; i++) {
|
| 279 |
+
if (shouldStop()) throw new Error("stopped");
|
| 280 |
+
const ts = performance.now();
|
| 281 |
+
const want = !!onPreview && this.hasPlan && !inflight && previewReady();
|
| 282 |
+
const out = await this.sessions.gen.run({ z: zT, t: one(grid[i]), dt: one(grid[i + 1] - grid[i]), cfg: cfgT, ...fixed },
|
| 283 |
+
want ? ["z_next", "x1", "plan"] : ["z_next"]);
|
| 284 |
+
if (i > 0) zT.dispose?.();
|
| 285 |
+
zT = out.z_next;
|
| 286 |
+
if (want) {
|
| 287 |
+
inflight = true;
|
| 288 |
+
const step = i + 1;
|
| 289 |
+
Promise.all([out.plan.getData(true), out.x1.getData(true)])
|
| 290 |
+
.then(([plan, x1]) => onPreview({ step, steps, plan, x1, hw }))
|
| 291 |
+
.catch((e) => console.warn("preview", e))
|
| 292 |
+
.finally(() => { inflight = false; });
|
| 293 |
+
}
|
| 294 |
+
stepMs.push(performance.now() - ts);
|
| 295 |
+
onStep(i + 1, steps, null);
|
| 296 |
+
if ((i & 3) === 3) await new Promise((r) => setTimeout(r, 0)); // let the page paint now and then
|
| 297 |
+
}
|
| 298 |
+
const zf = await zT.getData(true); // the only synchronising readback
|
| 299 |
+
return Float32Array.from(zf);
|
| 300 |
+
} finally {
|
| 301 |
+
for (const b of bufs) { try { b.destroy(); } catch { /* */ } }
|
| 302 |
+
}
|
| 303 |
+
}
|
| 304 |
+
|
| 305 |
// onPreview({ step, steps, plan, x1, hw }) after every step, when given and the graph outputs the plan: plan =
|
| 306 |
// the thinker output for the conditional branch (Float32Array 640 x 16 x 16), x1 = z + (1 - t) v (guided).
|
| 307 |
+
async generate({ prompt, negative = "", seed = 0, steps = 50, cfg = 3.0, noise = null, watermark = true, onStep = () => {}, onPreview = null, previewReady = () => true, shouldStop = () => false }) {
|
| 308 |
const C = this.manifest.latent_ch, D = this.manifest.ctx_dim;
|
| 309 |
const rc = this.mr ? this.manifest.resolutions[String(this.resolution)] : { latent_hw: this.manifest.latent_hw, shift: 1 };
|
| 310 |
const hw = rc.latent_hw, shift = Number(rc.shift || 1);
|
|
|
|
| 330 |
}
|
| 331 |
|
| 332 |
let z = noise ? Float32Array.from(noise) : Agate.gaussianNoise(seed, per);
|
|
|
|
| 333 |
const grid = Array.from({ length: steps + 1 }, (_, i) => Agate.shiftT(i / steps, shift));
|
| 334 |
t0 = performance.now();
|
| 335 |
const stepMs = [];
|
| 336 |
+
if (this.fast) {
|
| 337 |
+
z = await this.sampleFast({ z, grid, cfg, hw, C, L, D, ctx, mask, cnt: feeds.counts?.data, onStep, onPreview, previewReady, shouldStop, stepMs });
|
| 338 |
+
} else {
|
| 339 |
+
const zz = new Float32Array(2 * per), tt = new Float32Array(2);
|
| 340 |
for (let i = 0; i < steps; i++) {
|
| 341 |
if (shouldStop()) throw new Error("stopped");
|
| 342 |
const ts = performance.now();
|
|
|
|
| 361 |
onStep(i + 1, steps, z);
|
| 362 |
await new Promise((r) => setTimeout(r, 0)); // let the progress bar paint
|
| 363 |
}
|
| 364 |
+
}
|
| 365 |
T.samplerMs = performance.now() - t0;
|
| 366 |
T.firstStepMs = stepMs[0];
|
| 367 |
T.stepMs = stepMs.length > 1 ? (T.samplerMs - stepMs[0]) / (stepMs.length - 1) : stepMs[0];
|
js/app.js
CHANGED
|
@@ -116,7 +116,8 @@ async function pickRes(r) {
|
|
| 116 |
}
|
| 117 |
}
|
| 118 |
|
| 119 |
-
ui.steps.oninput = () => { $("steps-v").textContent = ui.steps.value; };
|
|
|
|
| 120 |
ui.cfg.oninput = () => { $("cfg-v").textContent = Number(ui.cfg.value).toFixed(1); };
|
| 121 |
ui.dice.onclick = () => { ui.seed.value = Math.floor(Math.random() * 2 ** 31); };
|
| 122 |
ui.crisp.checked = store.get("agate-crisp") === "1";
|
|
@@ -169,18 +170,19 @@ async function pngWithMetadata() {
|
|
| 169 |
return new Blob([bytes], { type: "image/png" });
|
| 170 |
}
|
| 171 |
|
|
|
|
|
|
|
| 172 |
function previewFn() {
|
| 173 |
thinkerView.reset();
|
| 174 |
-
let
|
| 175 |
-
window.__agate.thinker = {
|
| 176 |
-
|
| 177 |
-
const t0 = performance.now();
|
| 178 |
ui.thinker.hidden = false;
|
| 179 |
-
thinkerView.drawPlan(plan);
|
| 180 |
-
thinkerView.drawPrediction(x1, hw);
|
| 181 |
ui.thinkerStep.textContent = `${step} / ${steps}`;
|
| 182 |
-
|
| 183 |
-
|
|
|
|
|
|
|
| 184 |
};
|
| 185 |
}
|
| 186 |
|
|
@@ -197,6 +199,7 @@ async function generate(opts = {}) {
|
|
| 197 |
watermark: opts.watermark ?? true,
|
| 198 |
onStep: (i, n) => setProgress(i / n, `Step ${i} / ${n}`, ""),
|
| 199 |
onPreview: (opts.thinker ?? ui.showThinker.checked) ? previewFn() : null,
|
|
|
|
| 200 |
shouldStop: () => stop,
|
| 201 |
});
|
| 202 |
const ctx = ui.canvas.getContext("2d");
|
|
|
|
| 116 |
}
|
| 117 |
}
|
| 118 |
|
| 119 |
+
ui.steps.oninput = () => { $("steps-v").textContent = ui.steps.value; $("fast").checked = Number(ui.steps.value) === 25; };
|
| 120 |
+
$("fast").onchange = () => { ui.steps.value = $("fast").checked ? 25 : 50; $("steps-v").textContent = ui.steps.value; };
|
| 121 |
ui.cfg.oninput = () => { $("cfg-v").textContent = Number(ui.cfg.value).toFixed(1); };
|
| 122 |
ui.dice.onclick = () => { ui.seed.value = Math.floor(Math.random() * 2 ** 31); };
|
| 123 |
ui.crisp.checked = store.get("agate-crisp") === "1";
|
|
|
|
| 170 |
return new Blob([bytes], { type: "image/png" });
|
| 171 |
}
|
| 172 |
|
| 173 |
+
// The previews: frames go to thinkerView, which colours them in a Web Worker and drops frames while busy. The
|
| 174 |
+
// sampler never waits for any of it (agate.js sampleFast); with Show thinker off nothing is requested at all.
|
| 175 |
function previewFn() {
|
| 176 |
thinkerView.reset();
|
| 177 |
+
let n = 0, t0 = performance.now();
|
| 178 |
+
window.__agate.thinker = { submitted: 0 };
|
| 179 |
+
thinkerView.onDrawn = ({ step, steps, workerMs, dropped, drawn }) => {
|
|
|
|
| 180 |
ui.thinker.hidden = false;
|
|
|
|
|
|
|
| 181 |
ui.thinkerStep.textContent = `${step} / ${steps}`;
|
| 182 |
+
window.__agate.thinker = { submitted: n, drawn, dropped, lastWorkerMs: workerMs, lastStep: step };
|
| 183 |
+
};
|
| 184 |
+
return ({ step, steps, plan, x1, hw }) => {
|
| 185 |
+
if (thinkerView.submit(plan, x1, hw, { step, steps })) n++;
|
| 186 |
};
|
| 187 |
}
|
| 188 |
|
|
|
|
| 199 |
watermark: opts.watermark ?? true,
|
| 200 |
onStep: (i, n) => setProgress(i / n, `Step ${i} / ${n}`, ""),
|
| 201 |
onPreview: (opts.thinker ?? ui.showThinker.checked) ? previewFn() : null,
|
| 202 |
+
previewReady: () => !thinkerView.pending,
|
| 203 |
shouldStop: () => stop,
|
| 204 |
});
|
| 205 |
const ctx = ui.canvas.getContext("2d");
|
js/thinker-worker.js
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Web Worker: colours the thinker previews off the main thread (see thinker.js).
|
| 2 |
+
import { planPixels, predPixels } from "./thinker.js";
|
| 3 |
+
|
| 4 |
+
let gen = -1, state = {};
|
| 5 |
+
self.onmessage = (e) => {
|
| 6 |
+
const { gen: g, plan, x1, hw, meta } = e.data;
|
| 7 |
+
if (g !== gen) { gen = g; state = {}; } // a new generation: fit a new basis at its first frame
|
| 8 |
+
const t0 = performance.now();
|
| 9 |
+
const p = planPixels(plan, state), q = predPixels(x1, hw);
|
| 10 |
+
self.postMessage({ gen: g, plan: p, pred: q, hw, meta, ms: performance.now() - t0 }, [p.buffer, q.buffer]);
|
| 11 |
+
};
|
js/thinker.js
CHANGED
|
@@ -75,42 +75,75 @@ function quantile(sorted, q) { // torch.quantile, linear interpo
|
|
| 75 |
return sorted[lo] + (sorted[hi] - sorted[lo]) * (pos - lo);
|
| 76 |
}
|
| 77 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
export class ThinkerView {
|
| 79 |
-
constructor(planCanvas, predCanvas
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 80 |
|
| 81 |
-
reset() { this.
|
| 82 |
|
| 83 |
-
|
| 84 |
-
|
| 85 |
-
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
|
| 89 |
-
const
|
| 90 |
-
if (
|
| 91 |
-
|
| 92 |
-
|
| 93 |
-
}
|
| 94 |
-
const img = new ImageData(g, g);
|
| 95 |
-
for (let p = 0; p < n; p++) {
|
| 96 |
-
for (let k = 0; k < 3; k++) img.data[p * 4 + k] = Math.round(Math.min(1, Math.max(0, (Y[k][p] - this.lo[k]) / (this.hi[k] - this.lo[k] + 1e-8))) * 255);
|
| 97 |
-
img.data[p * 4 + 3] = 255;
|
| 98 |
-
}
|
| 99 |
-
const cv = this.planCanvas; cv.width = g; cv.height = g;
|
| 100 |
-
cv.getContext("2d").putImageData(img, 0, 0);
|
| 101 |
}
|
| 102 |
|
| 103 |
-
|
| 104 |
-
|
| 105 |
-
|
| 106 |
-
|
| 107 |
-
|
| 108 |
-
|
| 109 |
-
|
| 110 |
-
|
| 111 |
-
img.data[p * 4 + 3] = 255;
|
| 112 |
-
}
|
| 113 |
-
const cv = this.predCanvas; cv.width = hw; cv.height = hw;
|
| 114 |
-
cv.getContext("2d").putImageData(img, 0, 0);
|
| 115 |
}
|
|
|
|
|
|
|
|
|
|
| 116 |
}
|
|
|
|
| 75 |
return sorted[lo] + (sorted[hi] - sorted[lo]) * (pos - lo);
|
| 76 |
}
|
| 77 |
|
| 78 |
+
// state: { basis, lo, hi } (filled at the first call of a generation). -> RGBA Uint8ClampedArray (g * g * 4)
|
| 79 |
+
export function planPixels(plan, state, C = 640, g = 16) {
|
| 80 |
+
const n = g * g, X = new Float64Array(n * C);
|
| 81 |
+
for (let c = 0; c < C; c++) for (let p = 0; p < n; p++) X[p * C + c] = plan[c * n + p];
|
| 82 |
+
for (let c = 0; c < C; c++) { let m = 0; for (let p = 0; p < n; p++) m += X[p * C + c]; m /= n; for (let p = 0; p < n; p++) X[p * C + c] -= m; }
|
| 83 |
+
if (!state.basis) state.basis = topComponents(X, n, C);
|
| 84 |
+
const Y = state.basis.map((v) => { const y = new Float64Array(n); for (let p = 0; p < n; p++) { let s = 0; const r = p * C; for (let c = 0; c < C; c++) s += X[r + c] * v[c]; y[p] = s; } return y; });
|
| 85 |
+
if (!state.lo) {
|
| 86 |
+
state.lo = Y.map((y) => quantile(Float64Array.from(y).sort(), 0.02));
|
| 87 |
+
state.hi = Y.map((y) => quantile(Float64Array.from(y).sort(), 0.98));
|
| 88 |
+
}
|
| 89 |
+
const px = new Uint8ClampedArray(n * 4);
|
| 90 |
+
for (let p = 0; p < n; p++) {
|
| 91 |
+
for (let k = 0; k < 3; k++) px[p * 4 + k] = Math.round(Math.min(1, Math.max(0, (Y[k][p] - state.lo[k]) / (state.hi[k] - state.lo[k] + 1e-8))) * 255);
|
| 92 |
+
px[p * 4 + 3] = 255;
|
| 93 |
+
}
|
| 94 |
+
return px;
|
| 95 |
+
}
|
| 96 |
+
|
| 97 |
+
// x1: Float32Array (4 * hw * hw) in the model's (scaled SD-VAE) latent space -> RGBA (hw * hw * 4)
|
| 98 |
+
export function predPixels(x1, hw) {
|
| 99 |
+
const n = hw * hw, px = new Uint8ClampedArray(n * 4);
|
| 100 |
+
for (let p = 0; p < n; p++) {
|
| 101 |
+
for (let k = 0; k < 3; k++) {
|
| 102 |
+
let s = 0; for (let c = 0; c < 4; c++) s += x1[c * n + p] * RGB_FACTORS[c][k];
|
| 103 |
+
px[p * 4 + k] = Math.floor(Math.min(1, Math.max(0, (s + 1) / 2)) * 255);
|
| 104 |
+
}
|
| 105 |
+
px[p * 4 + 3] = 255;
|
| 106 |
+
}
|
| 107 |
+
return px;
|
| 108 |
+
}
|
| 109 |
+
|
| 110 |
+
// Draws previews; the colouring runs in a Web Worker (thinker-worker.js) when available, so the page and the
|
| 111 |
+
// sampler never wait for it. A frame arriving while the previous one is still being coloured is dropped.
|
| 112 |
export class ThinkerView {
|
| 113 |
+
constructor(planCanvas, predCanvas, onDrawn = () => {}) {
|
| 114 |
+
this.planCanvas = planCanvas; this.predCanvas = predCanvas; this.onDrawn = onDrawn;
|
| 115 |
+
this.gen = 0; this.busy = false; this.dropped = 0; this.drawn = 0;
|
| 116 |
+
try {
|
| 117 |
+
this.worker = new Worker(new URL("./thinker-worker.js", import.meta.url), { type: "module" });
|
| 118 |
+
this.worker.onmessage = (e) => this._paint(e.data);
|
| 119 |
+
} catch { this.worker = null; }
|
| 120 |
+
this.reset();
|
| 121 |
+
}
|
| 122 |
|
| 123 |
+
reset() { this.gen++; this.state = {}; this.busy = false; this.dropped = 0; this.drawn = 0; }
|
| 124 |
|
| 125 |
+
get pending() { return this.busy; }
|
| 126 |
+
|
| 127 |
+
// plan: Float32Array (640 * 16 * 16), x1: Float32Array (4 * hw * hw); both are transferred to the worker
|
| 128 |
+
submit(plan, x1, hw, meta = {}) {
|
| 129 |
+
if (this.busy) { this.dropped++; return false; }
|
| 130 |
+
this.busy = true;
|
| 131 |
+
const job = { gen: this.gen, plan, x1, hw, meta };
|
| 132 |
+
if (this.worker) this.worker.postMessage(job, [plan.buffer, x1.buffer]);
|
| 133 |
+
else setTimeout(() => this._paint({ gen: job.gen, hw, meta, plan: planPixels(plan, this.state), pred: predPixels(x1, hw) }), 0);
|
| 134 |
+
return true;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 135 |
}
|
| 136 |
|
| 137 |
+
_paint({ gen, plan, pred, hw, meta, ms }) {
|
| 138 |
+
this.busy = false;
|
| 139 |
+
if (gen !== this.gen) return;
|
| 140 |
+
const put = (cv, px, s) => { cv.width = s; cv.height = s; cv.getContext("2d").putImageData(new ImageData(px, s, s), 0, 0); };
|
| 141 |
+
put(this.planCanvas, plan, 16);
|
| 142 |
+
put(this.predCanvas, pred, hw);
|
| 143 |
+
this.drawn++;
|
| 144 |
+
this.onDrawn({ ...meta, workerMs: ms, dropped: this.dropped, drawn: this.drawn });
|
|
|
|
|
|
|
|
|
|
|
|
|
| 145 |
}
|
| 146 |
+
|
| 147 |
+
// synchronous path (tests): colour on this thread
|
| 148 |
+
drawPlan(plan) { const px = planPixels(plan, this.state); const cv = this.planCanvas; cv.width = 16; cv.height = 16; cv.getContext("2d").putImageData(new ImageData(px, 16, 16), 0, 0); }
|
| 149 |
}
|