Spaces:
Running
Running
Sync from GitHub
Browse files
classic/engines/NOTICE
CHANGED
|
@@ -10,7 +10,10 @@ fork of gemma4-webgpu:
|
|
| 10 |
|
| 11 |
Changes from the original: rewritten for mixture-of-experts models whose experts stream from OPFS (Q8_0 and Q4_0
|
| 12 |
split planes, slot-indexed expert GEMVs, GPU router top-k, chunked prefill, Q6_K head planes). `ternary_bonsai_2_dflash.js` is built from NakliTechie/dflash-mlx-bonsai2 (Apache-2.0). `lfm2_5.js`,
|
| 13 |
-
`gemma-4-e2b.js` and `ternary_bonsai_2_27b.js` are built from webml-community Hugging Face Spaces by the same author, MIT.
|
|
|
|
|
|
|
|
|
|
| 14 |
this folder is MIT (../LICENSE), Copyright (c) 2026 Chirag Patnaik.
|
| 15 |
|
| 16 |
The engine files are copies of LocalMind's (https://github.com/NakliTechie/LocalMind), kept in step by
|
|
|
|
| 10 |
|
| 11 |
Changes from the original: rewritten for mixture-of-experts models whose experts stream from OPFS (Q8_0 and Q4_0
|
| 12 |
split planes, slot-indexed expert GEMVs, GPU router top-k, chunked prefill, Q6_K head planes). `ternary_bonsai_2_dflash.js` is built from NakliTechie/dflash-mlx-bonsai2 (Apache-2.0). `lfm2_5.js`,
|
| 13 |
+
`gemma-4-e2b.js` and `ternary_bonsai_2_27b.js` are built from webml-community Hugging Face Spaces by the same author, MIT.
|
| 14 |
+
`clover_moe_ssd.js` follows the math of Diffuspace's modeling_clover_moe.py (Apache-2.0, Clover-35B-A3B-Preview); no
|
| 15 |
+
code is copied. `vendor/kohra-denoise.js` is kohra.js's denoise.js, unchanged (NakliTechie/kohra.js, Apache-2.0,
|
| 16 |
+
engines/LICENSE-APACHE), checked by scripts/sync-kohra.mjs. Everything else in
|
| 17 |
this folder is MIT (../LICENSE), Copyright (c) 2026 Chirag Patnaik.
|
| 18 |
|
| 19 |
The engine files are copies of LocalMind's (https://github.com/NakliTechie/LocalMind), kept in step by
|
classic/engines/clover_moe_ssd.js
ADDED
|
@@ -0,0 +1,498 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
/* clover_moe_ssd.js — Clover-35B-A3B-Preview (Diffuspace, Apache-2.0; GGUF arch `clover_moe`) in a browser tab,
|
| 2 |
+
* larger than RAM. Clover is a block-diffusion model on Qwen3.6-35B-A3B: it fills a 32-token block of mask tokens
|
| 3 |
+
* over several denoising passes, then commits the block into a causal prefix. The backbone, the expert paging and
|
| 4 |
+
* the commit pass are Qwen35MoeSsd's (a commit is one prefillChunk); what is new is the preview pass:
|
| 5 |
+
* - DeltaNet layers: the 4-tap conv also looks right, inside the block (conv1d.weight_right, 3 taps), and the
|
| 6 |
+
* delta rule becomes a softmax over the block's 32 keys plus one "history" slot, whose value is the read-only
|
| 7 |
+
* clean recurrent state read with the query and decayed by the position's gate. Nothing is written back.
|
| 8 |
+
* - full-attention layers: every block token sees the whole prefix and the whole block (no causal mask). The
|
| 9 |
+
* block's K/V go into the cache rows after the prefix as scratch; a commit overwrites them.
|
| 10 |
+
* - the MoE: one shared expert subset per block (the coreset), the smallest set of experts, by summed routing mass
|
| 11 |
+
* of every token's top 8, that covers moe_block_coreset_top_p (0.9); each token then routes inside it.
|
| 12 |
+
* - logits come back for every block row, and row i scores slot i (not the next token).
|
| 13 |
+
* The math follows the authors' reference (modeling_clover_moe.py: bd_gdn_conv1d, bd_gdn_local_memory, coreset_torch).
|
| 14 |
+
* The denoising loop is not here: kohra's denoise module drives forward() below.
|
| 15 |
+
*
|
| 16 |
+
* const m = await CloverMoeSsd.load(null, { url: '…/Clover-35B-A3B-Preview-Q4_0.gguf', poolBytes });
|
| 17 |
+
* await m.forward({ mode: 'commit', start: 0, ids: prompt, wantLogits: 'none' });
|
| 18 |
+
* const logits = await m.forward({ mode: 'preview', start: m.position, ids: block, wantLogits: 'all' });
|
| 19 |
+
*/
|
| 20 |
+
|
| 21 |
+
import { Qwen35MoeSsd } from './qwen35_moe_ssd.js';
|
| 22 |
+
import { STORAGE, COPY_SRC, COPY_DST, MAP_READ } from './qwen3_moe_ssd.js';
|
| 23 |
+
import { denoise, PRESETS } from './vendor/kohra-denoise.js';
|
| 24 |
+
|
| 25 |
+
// The authors' Q4_0 file: Q4_0 matrices, experts, embedding and output, F32 norms and scalars (the kernels
|
| 26 |
+
// Qwen35MoeSsd already has). Their Q8_0 file keeps the embedding, output and alpha/beta in BF16, which no kernel here reads.
|
| 27 |
+
export const CLOVER_35B_A3B = {
|
| 28 |
+
repo: 'Diffuspace/Clover-35B-A3B-Preview-GGUF',
|
| 29 |
+
file: 'Clover-35B-A3B-Preview-Q4_0.gguf',
|
| 30 |
+
revision: '32e22a02b885e173177b7ce7661ad13448b4f89e',
|
| 31 |
+
sha256: '3e2cf14ca47f6e41c251f08237ac79cd37ad81a2ea2a220fff731e34c66930d5',
|
| 32 |
+
size: 19586909600,
|
| 33 |
+
};
|
| 34 |
+
|
| 35 |
+
const QB = 'struct Q { T: u32, pos0: u32, S: u32, pad: u32 }';
|
| 36 |
+
const NEG = '-3.4e38';
|
| 37 |
+
|
| 38 |
+
const PREVIEW_KERNELS = {
|
| 39 |
+
// bd_gdn_conv1d: per channel, the causal 4-tap conv over [3 committed inputs | the block] plus the right taps
|
| 40 |
+
// r[d-1]·x[t+d] (d = 1..3, inside the block), then SiLU. The committed inputs are only read.
|
| 41 |
+
bdConvB: `
|
| 42 |
+
struct P { n: u32 }
|
| 43 |
+
${QB}
|
| 44 |
+
@group(0) @binding(0) var<storage, read> mixed: array<f32>;
|
| 45 |
+
@group(0) @binding(1) var<storage, read> w: array<f32>;
|
| 46 |
+
@group(0) @binding(2) var<storage, read> wr: array<f32>;
|
| 47 |
+
@group(0) @binding(3) var<storage, read> st: array<f32>;
|
| 48 |
+
@group(0) @binding(4) var<storage, read_write> y: array<f32>;
|
| 49 |
+
@group(0) @binding(5) var<uniform> p: P;
|
| 50 |
+
@group(0) @binding(6) var<uniform> qd: Q;
|
| 51 |
+
fn xin(c: u32, i: u32) -> f32 { if (i < 3u) { return st[c * 3u + i]; } return mixed[(i - 3u) * p.n + c]; }
|
| 52 |
+
@compute @workgroup_size(256) fn main(@builtin(global_invocation_id) gi: vec3<u32>) {
|
| 53 |
+
let c = gi.x; if (c >= p.n) { return; }
|
| 54 |
+
let w0 = w[c * 4u]; let w1 = w[c * 4u + 1u]; let w2 = w[c * 4u + 2u]; let w3 = w[c * 4u + 3u];
|
| 55 |
+
for (var t = 0u; t < qd.T; t++) {
|
| 56 |
+
var sum = 0.0;
|
| 57 |
+
sum += xin(c, t) * w0; sum += xin(c, t + 1u) * w1; sum += xin(c, t + 2u) * w2; sum += xin(c, t + 3u) * w3;
|
| 58 |
+
for (var d = 1u; d < 4u; d++) { if (t + d < qd.T) { sum += mixed[(t + d) * p.n + c] * wr[c * 3u + d - 1u]; } }
|
| 59 |
+
y[t * p.n + c] = sum / (1.0 + exp(-sum));
|
| 60 |
+
}
|
| 61 |
+
}`,
|
| 62 |
+
// L2-normalised q and k (x / sqrt(Σx² + eps)) for (key head wg.x, token wg.y), into one buffer: q rows [T][kH·dS]
|
| 63 |
+
// from 0, k rows from kOff (one buffer keeps bdGdnB within 10 storage bindings).
|
| 64 |
+
bdQkNormB: `
|
| 65 |
+
struct P { kH: u32, dS: u32, keyDim: u32, convDim: u32, kOff: u32, eps: f32 }
|
| 66 |
+
@group(0) @binding(0) var<storage, read> cv: array<f32>;
|
| 67 |
+
@group(0) @binding(1) var<storage, read_write> qk: array<f32>;
|
| 68 |
+
@group(0) @binding(2) var<uniform> p: P;
|
| 69 |
+
var<workgroup> rq: array<f32, 128>;
|
| 70 |
+
var<workgroup> rk: array<f32, 128>;
|
| 71 |
+
@compute @workgroup_size(128) fn main(@builtin(workgroup_id) wg: vec3<u32>, @builtin(local_invocation_id) l: vec3<u32>) {
|
| 72 |
+
let kh = wg.x; let t = wg.y; let j = l.x; let cb = t * p.convDim;
|
| 73 |
+
let qv = cv[cb + kh * p.dS + j]; let kv = cv[cb + p.keyDim + kh * p.dS + j];
|
| 74 |
+
rq[j] = qv * qv; rk[j] = kv * kv; workgroupBarrier();
|
| 75 |
+
for (var s = 64u; s > 0u; s >>= 1u) { if (j < s) { rq[j] += rq[j + s]; rk[j] += rk[j + s]; } workgroupBarrier(); }
|
| 76 |
+
let o = t * p.keyDim + kh * p.dS + j;
|
| 77 |
+
qk[o] = qv / sqrt(rq[0] + p.eps); qk[p.kOff + o] = kv / sqrt(rk[0] + p.eps);
|
| 78 |
+
}`,
|
| 79 |
+
// bd_gdn_local_memory for (value head wg.x, query token wg.y); thread j = value row j. k-head = h % kH (the GGUF's
|
| 80 |
+
// tiled head order, as gdnStep). Logits over the block's keys: (qn·kn)·logitScale[h]; the history slot's logit:
|
| 81 |
+
// bias[h] + c[h]·log(T/32); its value: the clean state read with qn·scale, times exp(g), g = softplus(a + dt)·A.
|
| 82 |
+
bdGdnB: `
|
| 83 |
+
struct P { kH: u32, dS: u32, keyDim: u32, convDim: u32, vH: u32, kOff: u32, scale: f32 }
|
| 84 |
+
${QB}
|
| 85 |
+
@group(0) @binding(0) var<storage, read> qk: array<f32>;
|
| 86 |
+
@group(0) @binding(1) var<storage, read> cv: array<f32>;
|
| 87 |
+
@group(0) @binding(2) var<storage, read> al: array<f32>;
|
| 88 |
+
@group(0) @binding(3) var<storage, read> dt: array<f32>;
|
| 89 |
+
@group(0) @binding(4) var<storage, read> aa: array<f32>;
|
| 90 |
+
@group(0) @binding(5) var<storage, read> ls: array<f32>;
|
| 91 |
+
@group(0) @binding(6) var<storage, read> hb: array<f32>;
|
| 92 |
+
@group(0) @binding(7) var<storage, read> hc: array<f32>;
|
| 93 |
+
@group(0) @binding(8) var<storage, read> M: array<f32>;
|
| 94 |
+
@group(0) @binding(9) var<storage, read_write> o: array<f32>;
|
| 95 |
+
@group(0) @binding(10) var<uniform> p: P;
|
| 96 |
+
@group(0) @binding(11) var<uniform> qd: Q;
|
| 97 |
+
var<workgroup> qs: array<f32, 128>;
|
| 98 |
+
var<workgroup> wt: array<f32, 33>;
|
| 99 |
+
fn log1p_(u: f32) -> f32 { let y = 1.0 + u; return log(y) - ((y - 1.0) - u) / y; }
|
| 100 |
+
@compute @workgroup_size(128) fn main(@builtin(workgroup_id) wg: vec3<u32>, @builtin(local_invocation_id) l: vec3<u32>) {
|
| 101 |
+
let h = wg.x; let t = wg.y; let j = l.x; let dS = p.dS; let kh = h % p.kH; let T = qd.T;
|
| 102 |
+
let qb = t * p.keyDim + kh * dS;
|
| 103 |
+
qs[j] = qk[qb + j];
|
| 104 |
+
workgroupBarrier();
|
| 105 |
+
if (j < T) {
|
| 106 |
+
let kb = j * p.keyDim + kh * dS; var acc = 0.0;
|
| 107 |
+
for (var i = 0u; i < dS; i++) { acc += qs[i] * qk[p.kOff + kb + i]; }
|
| 108 |
+
wt[j] = acc * ls[h];
|
| 109 |
+
}
|
| 110 |
+
if (j == 32u) { wt[T] = hb[h] + hc[h] * log(f32(T) / 32.0); }
|
| 111 |
+
workgroupBarrier();
|
| 112 |
+
if (j == 0u) {
|
| 113 |
+
var mx = wt[0];
|
| 114 |
+
for (var s = 1u; s <= T; s++) { mx = max(mx, wt[s]); }
|
| 115 |
+
var sum = 0.0;
|
| 116 |
+
for (var s = 0u; s <= T; s++) { let e = exp(wt[s] - mx); wt[s] = e; sum += e; }
|
| 117 |
+
for (var s = 0u; s <= T; s++) { wt[s] = wt[s] / sum; }
|
| 118 |
+
}
|
| 119 |
+
workgroupBarrier();
|
| 120 |
+
let row = (h * dS + j) * dS; var rd = 0.0;
|
| 121 |
+
for (var i = 0u; i < dS; i++) { rd += M[row + i] * (qs[i] * p.scale); }
|
| 122 |
+
let x = al[t * p.vH + h] + dt[h];
|
| 123 |
+
let sp = select(log1p_(exp(x)), x, x > 20.0);
|
| 124 |
+
var acc = wt[T] * (rd * exp(sp * aa[h]));
|
| 125 |
+
var loc = 0.0;
|
| 126 |
+
for (var s = 0u; s < T; s++) { loc += wt[s] * cv[s * p.convDim + 2u * p.keyDim + h * dS + j]; }
|
| 127 |
+
o[t * p.vH * dS + h * dS + j] = acc + loc;
|
| 128 |
+
}`,
|
| 129 |
+
// Scores for T queries over all S = pos0 + T keys, unmasked: the block sees the prefix and itself both ways.
|
| 130 |
+
attnScoreAllB: `enable f16;
|
| 131 |
+
struct P { heads: u32, kvHeads: u32, hd: u32, scale: f32 }
|
| 132 |
+
${QB}
|
| 133 |
+
@group(0) @binding(0) var<storage, read> q: array<f32>;
|
| 134 |
+
@group(0) @binding(1) var<storage, read> kc: array<f16>;
|
| 135 |
+
@group(0) @binding(2) var<storage, read_write> sc: array<f32>;
|
| 136 |
+
@group(0) @binding(3) var<uniform> p: P;
|
| 137 |
+
@group(0) @binding(4) var<uniform> qd: Q;
|
| 138 |
+
@compute @workgroup_size(256) fn main(@builtin(global_invocation_id) g: vec3<u32>, @builtin(num_workgroups) ng: vec3<u32>) {
|
| 139 |
+
let idx = g.y * ng.x * 256u + g.x; let S = qd.S; let T = qd.T;
|
| 140 |
+
if (idx >= p.heads * T * S) { return; }
|
| 141 |
+
let h = idx / (T * S); let rem = idx % (T * S); let t = rem / S; let pos = rem % S;
|
| 142 |
+
let kvh = h / (p.heads / p.kvHeads);
|
| 143 |
+
let qo = t * p.heads * p.hd + h * p.hd; let ko = pos * p.kvHeads * p.hd + kvh * p.hd;
|
| 144 |
+
var acc = 0.0;
|
| 145 |
+
for (var d = 0u; d < p.hd; d++) { acc += q[qo + d] * f32(kc[ko + d]); }
|
| 146 |
+
sc[idx] = acc * p.scale;
|
| 147 |
+
}`,
|
| 148 |
+
// coreset_torch, one workgroup, thread e = expert (E ≤ 256). The block's mass on expert e: Σ over tokens and their
|
| 149 |
+
// top-k picks of e of the pick's full-softmax probability; normalised, sorted (descending, ties by index), kept
|
| 150 |
+
// while the mass before it is under topP (the first always). Router logits of every other expert become -3.4e38,
|
| 151 |
+
// so topkB, run again, routes each token inside the subset (picks past the subset get weight 0).
|
| 152 |
+
coresetB: `
|
| 153 |
+
struct P { E: u32, K: u32, topP: f32 }
|
| 154 |
+
${QB}
|
| 155 |
+
@group(0) @binding(0) var<storage, read_write> lg: array<f32>;
|
| 156 |
+
@group(0) @binding(1) var<storage, read> sel: array<u32>;
|
| 157 |
+
@group(0) @binding(2) var<uniform> p: P;
|
| 158 |
+
@group(0) @binding(3) var<uniform> qd: Q;
|
| 159 |
+
var<workgroup> red: array<f32, 256>;
|
| 160 |
+
var<workgroup> pr: array<f32, 256>;
|
| 161 |
+
var<workgroup> srt: array<f32, 256>;
|
| 162 |
+
var<workgroup> keep: array<u32, 256>;
|
| 163 |
+
var<workgroup> stat: array<f32, 2>;
|
| 164 |
+
@compute @workgroup_size(256) fn main(@builtin(local_invocation_id) l: vec3<u32>) {
|
| 165 |
+
let e = l.x; let E = p.E; let on = e < E;
|
| 166 |
+
var mine = 0.0;
|
| 167 |
+
for (var t = 0u; t < qd.T; t++) {
|
| 168 |
+
let v = select(${NEG}, lg[t * E + min(e, E - 1u)], on);
|
| 169 |
+
red[e] = v; workgroupBarrier();
|
| 170 |
+
for (var s = 128u; s > 0u; s >>= 1u) { if (e < s) { red[e] = max(red[e], red[e + s]); } workgroupBarrier(); }
|
| 171 |
+
if (e == 0u) { stat[0] = red[0]; } workgroupBarrier();
|
| 172 |
+
let ex = select(0.0, exp(v - stat[0]), on);
|
| 173 |
+
red[e] = ex; workgroupBarrier();
|
| 174 |
+
for (var s = 128u; s > 0u; s >>= 1u) { if (e < s) { red[e] += red[e + s]; } workgroupBarrier(); }
|
| 175 |
+
if (e == 0u) { stat[1] = red[0]; } workgroupBarrier();
|
| 176 |
+
for (var j = 0u; j < p.K; j++) { if (sel[t * 2u * p.K + j] == e) { mine += ex / stat[1]; } }
|
| 177 |
+
workgroupBarrier();
|
| 178 |
+
}
|
| 179 |
+
red[e] = mine; workgroupBarrier();
|
| 180 |
+
for (var s = 128u; s > 0u; s >>= 1u) { if (e < s) { red[e] += red[e + s]; } workgroupBarrier(); }
|
| 181 |
+
let tot = max(red[0], 1e-9);
|
| 182 |
+
let pe = select(-1.0, mine / tot, on);
|
| 183 |
+
pr[e] = pe; workgroupBarrier();
|
| 184 |
+
var rank = 0u;
|
| 185 |
+
for (var f = 0u; f < 256u; f++) { let pf = pr[f]; if (pf > pe || (pf == pe && f < e)) { rank++; } }
|
| 186 |
+
srt[rank] = pe; workgroupBarrier();
|
| 187 |
+
if (e == 0u) {
|
| 188 |
+
var cum = 0.0;
|
| 189 |
+
for (var r = 0u; r < E; r++) { keep[r] = select(0u, 1u, cum < p.topP || r == 0u); cum += srt[r]; }
|
| 190 |
+
}
|
| 191 |
+
workgroupBarrier();
|
| 192 |
+
if (on && keep[rank] == 0u) { for (var t = 0u; t < qd.T; t++) { lg[t * E + e] = ${NEG}; } }
|
| 193 |
+
}`,
|
| 194 |
+
// Per block row: the argmax with the mask id excluded (lowest id on a tie) and its softmax probability at T = 1
|
| 195 |
+
// over the rest of the vocabulary: out[2t] = id, out[2t + 1] = probability bits. One workgroup per row.
|
| 196 |
+
// A = -exp(A_log) in place: clover_moe GGUFs store ssm_a as A_log itself (checked against the BF16 checkpoint,
|
| 197 |
+
// 2026-10-30), where the qwen35moe files store -exp(A_log), which gdnStep and bdGdnB read.
|
| 198 |
+
negExp: `
|
| 199 |
+
struct P { n: u32 }
|
| 200 |
+
@group(0) @binding(0) var<storage, read_write> a: array<f32>;
|
| 201 |
+
@group(0) @binding(1) var<uniform> p: P;
|
| 202 |
+
@compute @workgroup_size(64) fn main(@builtin(global_invocation_id) gi: vec3<u32>) { if (gi.x < p.n) { a[gi.x] = -exp(a[gi.x]); } }`,
|
| 203 |
+
top1B: `
|
| 204 |
+
struct P { V: u32, mask: u32 }
|
| 205 |
+
@group(0) @binding(0) var<storage, read> lg: array<f32>;
|
| 206 |
+
@group(0) @binding(1) var<storage, read_write> out: array<u32>;
|
| 207 |
+
@group(0) @binding(2) var<uniform> p: P;
|
| 208 |
+
var<workgroup> rv: array<f32, 256>;
|
| 209 |
+
var<workgroup> ri: array<u32, 256>;
|
| 210 |
+
@compute @workgroup_size(256) fn main(@builtin(workgroup_id) wg: vec3<u32>, @builtin(local_invocation_id) l: vec3<u32>) {
|
| 211 |
+
let b = wg.x * p.V; let t = l.x;
|
| 212 |
+
var bv = ${NEG}; var bi = 0xffffffffu;
|
| 213 |
+
for (var i = t; i < p.V; i += 256u) { if (i != p.mask) { let v = lg[b + i]; if (v > bv) { bv = v; bi = i; } } }
|
| 214 |
+
rv[t] = bv; ri[t] = bi; workgroupBarrier();
|
| 215 |
+
for (var s = 128u; s > 0u; s >>= 1u) {
|
| 216 |
+
if (t < s) { let o = rv[t + s]; let oi = ri[t + s]; if (o > rv[t] || (o == rv[t] && oi < ri[t])) { rv[t] = o; ri[t] = oi; } }
|
| 217 |
+
workgroupBarrier();
|
| 218 |
+
}
|
| 219 |
+
let mx = rv[0]; let id = ri[0]; workgroupBarrier();
|
| 220 |
+
var sum = 0.0;
|
| 221 |
+
for (var i = t; i < p.V; i += 256u) { if (i != p.mask) { sum += exp(lg[b + i] - mx); } }
|
| 222 |
+
rv[t] = sum; workgroupBarrier();
|
| 223 |
+
for (var s = 128u; s > 0u; s >>= 1u) { if (t < s) { rv[t] += rv[t + s]; } workgroupBarrier(); }
|
| 224 |
+
if (t == 0u) { out[2u * wg.x] = id; out[2u * wg.x + 1u] = bitcast<u32>(1.0 / rv[0]); }
|
| 225 |
+
}`,
|
| 226 |
+
};
|
| 227 |
+
|
| 228 |
+
export class CloverMoeSsd extends Qwen35MoeSsd {
|
| 229 |
+
static archs = ['clover_moe'];
|
| 230 |
+
|
| 231 |
+
static async load(modelId = null, opts = {}) {
|
| 232 |
+
const source = opts.source || (opts.url ? { repo: null, file: opts.url.split('/').pop(), revision: null } : CLOVER_35B_A3B);
|
| 233 |
+
return super.load(modelId, { ...opts, source });
|
| 234 |
+
}
|
| 235 |
+
|
| 236 |
+
constructor(device, manifest, gguf, opts) {
|
| 237 |
+
super(device, manifest, gguf, opts);
|
| 238 |
+
const kv = gguf.kv, g = (k) => kv[`clover_moe.${k}`];
|
| 239 |
+
this.block = g('block_size') || 32;
|
| 240 |
+
if (this.block > 32) throw new Error(`bdGdnB holds at most 32 block keys, got a block of ${this.block}`);
|
| 241 |
+
this.coreset = g('moe_block_coreset_enabled') !== false;
|
| 242 |
+
this.coresetTopP = g('moe_block_coreset_top_p') ?? 0.9;
|
| 243 |
+
this.maskId = this.tokenizer.ids.get('<|mask|>');
|
| 244 |
+
if (this.maskId == null) throw new Error('no <|mask|> token in the vocabulary');
|
| 245 |
+
if (this.cfg.experts > 256) throw new Error('coresetB handles at most 256 experts');
|
| 246 |
+
// capture: per-layer block rows of the last preview (test/browser/clover-states.html, the oracle gate)
|
| 247 |
+
this.capture = !!opts.capture;
|
| 248 |
+
}
|
| 249 |
+
|
| 250 |
+
get kernels() {
|
| 251 |
+
if (!this._ckernels) this._ckernels = { ...super.kernels, ...PREVIEW_KERNELS };
|
| 252 |
+
return this._ckernels;
|
| 253 |
+
}
|
| 254 |
+
isGuarded(name) { return !(name in PREVIEW_KERNELS) && super.isGuarded(name); }
|
| 255 |
+
|
| 256 |
+
async uploadDense(onProgress) {
|
| 257 |
+
await super.uploadDense(onProgress);
|
| 258 |
+
const s = this.cfg.ssm, enc = this.device.createCommandEncoder(), pass = enc.beginComputePass(), u = this.uniform([s.vHeads]);
|
| 259 |
+
for (let l = 0; l < this.cfg.layers; l++) {
|
| 260 |
+
if ((l + 1) % this.cfg.attnInterval === 0) continue;
|
| 261 |
+
this.dispatch(pass, 'negExp', this.bind('negExp', [this.w[`blk.${l}.ssm_a`].raw, u]), Math.ceil(s.vHeads / 64));
|
| 262 |
+
}
|
| 263 |
+
pass.end(); this.device.queue.submit([enc.finish()]);
|
| 264 |
+
await this.device.queue.onSubmittedWorkDone();
|
| 265 |
+
}
|
| 266 |
+
|
| 267 |
+
initBuffers() {
|
| 268 |
+
super.initBuffers();
|
| 269 |
+
if (!this.batchPrefill) throw new Error('CloverMoeSsd needs batchPrefill');
|
| 270 |
+
if (this.chunkTokens < this.block) throw new Error(`a chunk of ${this.chunkTokens} tokens < block ${this.block}`);
|
| 271 |
+
this.initPreview();
|
| 272 |
+
}
|
| 273 |
+
|
| 274 |
+
// Buffers and bind groups of the preview pass. It shares the chunk buffers (this.b) with prefillChunk: the two
|
| 275 |
+
// never run at once (both hold the engine lock through forward()).
|
| 276 |
+
initPreview() {
|
| 277 |
+
const c = this.cfg, s = c.ssm, B = this.block, b = this.b, W = this.w, H = c.hidden;
|
| 278 |
+
const keyDim = s.kHeads * s.dState, convDim = 2 * keyDim + s.dInner;
|
| 279 |
+
const A = (n, cat = 'batch') => this.buffer(n * 4, STORAGE | COPY_SRC | COPY_DST, cat);
|
| 280 |
+
this.p = { qk: A(2 * B * keyDim), logits: A(B * c.vocab), top1: A(2 * B) };
|
| 281 |
+
this.rbTop1 = this.buffer(2 * B * 4, MAP_READ | COPY_DST, 'batch');
|
| 282 |
+
this.rbBlockLogits = this.buffer(B * c.vocab * 4, MAP_READ | COPY_DST, 'batch');
|
| 283 |
+
if (this.capture) { this.p.cap = A(c.layers * B * H); this.rbCap = this.buffer(c.layers * B * H * 4, MAP_READ | COPY_DST, 'batch'); }
|
| 284 |
+
const U = (w) => this.uniform(w), qb = this.qb, raw = (name) => {
|
| 285 |
+
const t = W[name]; if (!t || !t.raw) throw new Error(`missing F32 tensor ${name}`); return t.raw;
|
| 286 |
+
};
|
| 287 |
+
const u = {
|
| 288 |
+
conv: U([convDim]), qkn: U([s.kHeads, s.dState, keyDim, convDim, B * keyDim, { f: c.eps }]),
|
| 289 |
+
gdn: U([s.kHeads, s.dState, keyDim, convDim, s.vHeads, B * keyDim, { f: 1 / Math.sqrt(s.dState) }]),
|
| 290 |
+
att: U([c.heads, c.kvHeads, c.headDim, { f: 1 / Math.sqrt(c.headDim) }]),
|
| 291 |
+
core: U([c.experts, c.topK, { f: this.coresetTopP }]), lm: U([c.vocab, H]), top1: U([c.vocab, this.maskId]),
|
| 292 |
+
};
|
| 293 |
+
for (let l = 0; l < c.layers; l++) {
|
| 294 |
+
const n = (t) => `blk.${l}.${t}`, g = {};
|
| 295 |
+
if (this.recurrent[l]) {
|
| 296 |
+
g.conv = this.bind('bdConvB', [b.mixed, raw(n('ssm_conv1d.weight')), raw(n('ssm_conv1d.weight_right')), this.conv[l], b.cv, u.conv, qb]);
|
| 297 |
+
g.qkn = this.bind('bdQkNormB', [b.cv, this.p.qk, u.qkn]);
|
| 298 |
+
g.gdn = this.bind('bdGdnB', [this.p.qk, b.cv, b.al, raw(n('ssm_dt.bias')), raw(n('ssm_a')),
|
| 299 |
+
raw(n('ssm_bd_logit_scale.weight')), raw(n('ssm_bd_history.bias')), raw(n('ssm_bd_history.weight')), this.ssm[l], b.go, u.gdn, qb]);
|
| 300 |
+
} else {
|
| 301 |
+
g.score = this.bind('attnScoreAllB', [b.q, this.kc[l], b.sc, u.att, qb]);
|
| 302 |
+
}
|
| 303 |
+
g.core = this.bind('coresetB', [b.rl, b.sel, u.core, qb]);
|
| 304 |
+
this.layers[l].P = g;
|
| 305 |
+
}
|
| 306 |
+
this.gP = {
|
| 307 |
+
rmsOut: this.bind('rmsnormB', [b.x, raw('output_norm.weight'), b.xn, U([H, { f: c.eps }])]),
|
| 308 |
+
lm: this.bind(this.qk.mmT, [b.xn, W['output.weight'].q, W['output.weight'].s, this.p.logits, u.lm, qb]),
|
| 309 |
+
top1: this.bind('top1B', [this.p.logits, this.p.top1, u.top1]),
|
| 310 |
+
};
|
| 311 |
+
if (this.qk.mmT !== 'matmulQ4T' && this.qk.mmT !== 'matmulQ8T') throw new Error('the batched LM head needs a Q4_0 or Q8_0 output matrix');
|
| 312 |
+
if (W['output.weight'].q == null) throw new Error('the batched LM head needs a Q4_0 or Q8_0 output matrix');
|
| 313 |
+
}
|
| 314 |
+
|
| 315 |
+
// Layer l of the preview over the block's T rows, up to its routing (encodeTrunkB with the preview kernels).
|
| 316 |
+
encodePreviewTrunk(pass, l, T, S) {
|
| 317 |
+
const c = this.cfg, s = c.ssm, g = this.layers[l].B, gp = this.layers[l].P, d = (nm, gr, x, y, z) => this.dispatch(pass, nm, gr, x, y, z);
|
| 318 |
+
const H = c.hidden, QN = c.heads * c.headDim, kvn = c.kvHeads * c.headDim, convDim = 2 * s.kHeads * s.dState + s.dInner;
|
| 319 |
+
const tt = Math.ceil(T / 16), mm = (n) => Math.ceil(n / 16), QK = this.qk;
|
| 320 |
+
d('rmsnormB', g.rmsA, T);
|
| 321 |
+
if (this.recurrent[l]) {
|
| 322 |
+
d(QK.mmT, g.qkv, mm(convDim), tt); d(QK.mmT, g.z, mm(s.dInner), tt); d(QK.mmT, g.alpha, mm(s.vHeads), tt);
|
| 323 |
+
d('bdConvB', gp.conv, Math.ceil(convDim / 256));
|
| 324 |
+
d('bdQkNormB', gp.qkn, s.kHeads, T);
|
| 325 |
+
d('bdGdnB', gp.gdn, s.vHeads, T);
|
| 326 |
+
d('gdnNormGateB', g.norm, s.vHeads, T);
|
| 327 |
+
d(QK.mmT, g.out, mm(H), tt);
|
| 328 |
+
} else {
|
| 329 |
+
d(QK.mmT, g.q, mm(2 * QN), tt); d(QK.mmT, g.k, mm(kvn), tt); d(QK.mmT, g.v, mm(kvn), tt);
|
| 330 |
+
d('qkNormRopePB', g.ropeQ, c.heads, T); d('qkNormRopePB', g.ropeK, c.kvHeads, T);
|
| 331 |
+
d('kvStoreB', g.kv, Math.ceil(kvn / 256), T); // scratch rows pos0.. after the prefix
|
| 332 |
+
d('attnScoreAllB', gp.score, Math.ceil(c.heads * T * S / 256));
|
| 333 |
+
d('softmaxB', g.soft, c.heads * T);
|
| 334 |
+
d('attnOutPosB', g.attOut, Math.ceil(QN / 256), T);
|
| 335 |
+
d('sigmoidMulB', g.gate, Math.ceil(QN / 256), T);
|
| 336 |
+
d(QK.mmT, g.o, mm(H), tt);
|
| 337 |
+
}
|
| 338 |
+
d('addInPlaceB', g.addO, Math.ceil(H / 256), T);
|
| 339 |
+
d('rmsnormB', g.rmsF, T);
|
| 340 |
+
d('matmulF32T', g.router, c.experts, T);
|
| 341 |
+
d('topkB', g.topk, T);
|
| 342 |
+
if (this.coreset) { d('coresetB', gp.core, 1); d('topkB', g.topk, T); }
|
| 343 |
+
d(QK.mmT, g.shg, mm(c.shexpFf), tt); d(QK.mmT, g.shu, mm(c.shexpFf), tt);
|
| 344 |
+
d('siluMul2B', g.shact, Math.ceil(c.shexpFf / 256), T);
|
| 345 |
+
d(QK.mmT, g.shd, mm(H), tt);
|
| 346 |
+
d('matmulF32T', g.shgate, 1, T);
|
| 347 |
+
}
|
| 348 |
+
|
| 349 |
+
// One denoising pass over the block `ids` (mask slots hold this.maskId) at positions position..position+T-1.
|
| 350 |
+
// Reads the committed cache and writes nothing to it (KV rows past the prefix are scratch). want: 'all' → T × vocab
|
| 351 |
+
// logits (Float32Array, row t scores slot t); 'top1' → { token: Int32Array(T), conf: Float32Array(T) }, the argmax
|
| 352 |
+
// over the vocabulary without the mask id and its probability at T = 1.
|
| 353 |
+
async preview(ids, want = 'all') {
|
| 354 |
+
const c = this.cfg, dev = this.device, K = c.topK, H = c.hidden, T = ids.length, pos0 = this.position, S = pos0 + T;
|
| 355 |
+
if (T < 1 || T > this.block || this.block % T) throw new Error(`a preview block of ${T} must divide ${this.block}`);
|
| 356 |
+
if (S > this.maxCtx) throw this.contextFull();
|
| 357 |
+
const tStart = performance.now();
|
| 358 |
+
dev.queue.writeBuffer(this.a.stop, 0, new Uint32Array(4));
|
| 359 |
+
dev.queue.writeBuffer(this.b.ids, 0, Uint32Array.from(ids));
|
| 360 |
+
dev.queue.writeBuffer(this.qb, 0, new Uint32Array([T, pos0, S, 0]));
|
| 361 |
+
dev.queue.writeBuffer(this.b.rope, 0, this.ropeRows(this.textPositions(T), T));
|
| 362 |
+
const selBytes = T * 2 * K * 4, rows = T * H * 4;
|
| 363 |
+
let enc = dev.createCommandEncoder(), pass = enc.beginComputePass();
|
| 364 |
+
this.dispatch(pass, this.qk.embB, this.gB.embed, Math.ceil(H / 256), T);
|
| 365 |
+
this.encodePreviewTrunk(pass, 0, T, S);
|
| 366 |
+
pass.end();
|
| 367 |
+
enc.copyBufferToBuffer(this.b.sel, 0, this.rbSelB, 0, selBytes);
|
| 368 |
+
dev.queue.submit([enc.finish()]);
|
| 369 |
+
const units = new Set();
|
| 370 |
+
for (let l = 0; l < c.layers; l++) {
|
| 371 |
+
const t0 = performance.now();
|
| 372 |
+
await this.rbSelB.mapAsync(1, 0, selBytes);
|
| 373 |
+
this.counters.gpuWaitMs += performance.now() - t0;
|
| 374 |
+
const sel = new Uint32Array(this.rbSelB.getMappedRange(0, selBytes).slice(0)), wts = new Float32Array(sel.buffer);
|
| 375 |
+
this.rbSelB.unmap();
|
| 376 |
+
// Picks with weight 0 (a token routed past the coreset) need no load: they take any loaded slot.
|
| 377 |
+
const pairs = new Uint32Array(T * K), live = new Set();
|
| 378 |
+
for (let t = 0; t < T; t++) for (let j = 0; j < K; j++) {
|
| 379 |
+
const e = sel[t * 2 * K + j]; pairs[t * K + j] = e;
|
| 380 |
+
if (wts[t * 2 * K + K + j] > 0) live.add(e);
|
| 381 |
+
}
|
| 382 |
+
const unique = [...live];
|
| 383 |
+
for (const e of unique) units.add(l * c.experts + e);
|
| 384 |
+
const tq = performance.now();
|
| 385 |
+
const slotsU = await this.xs.ensure(l, unique);
|
| 386 |
+
this.counters.ensureMs += performance.now() - tq;
|
| 387 |
+
const slotOf = new Map(unique.map((e, i) => [e, slotsU[i]]));
|
| 388 |
+
dev.queue.writeBuffer(this.b.slots, 0, pairs.map((e) => slotOf.get(e) ?? slotsU[0]));
|
| 389 |
+
enc = dev.createCommandEncoder(); pass = enc.beginComputePass();
|
| 390 |
+
this.encodeExpertsB(pass, l, T);
|
| 391 |
+
pass.end();
|
| 392 |
+
if (this.capture) enc.copyBufferToBuffer(this.b.x, 0, this.p.cap, l * this.block * H * 4, rows);
|
| 393 |
+
if (l + 1 < c.layers) {
|
| 394 |
+
pass = enc.beginComputePass();
|
| 395 |
+
this.encodePreviewTrunk(pass, l + 1, T, S);
|
| 396 |
+
pass.end();
|
| 397 |
+
enc.copyBufferToBuffer(this.b.sel, 0, this.rbSelB, 0, selBytes);
|
| 398 |
+
} else {
|
| 399 |
+
pass = enc.beginComputePass();
|
| 400 |
+
this.dispatch(pass, 'rmsnormB', this.gP.rmsOut, T);
|
| 401 |
+
this.dispatch(pass, this.qk.mmT, this.gP.lm, Math.ceil(c.vocab / 16), Math.ceil(T / 16));
|
| 402 |
+
if (want === 'top1') this.dispatch(pass, 'top1B', this.gP.top1, T);
|
| 403 |
+
pass.end();
|
| 404 |
+
if (want === 'top1') enc.copyBufferToBuffer(this.p.top1, 0, this.rbTop1, 0, 2 * T * 4);
|
| 405 |
+
else enc.copyBufferToBuffer(this.p.logits, 0, this.rbBlockLogits, 0, T * c.vocab * 4);
|
| 406 |
+
}
|
| 407 |
+
dev.queue.submit([enc.finish()]);
|
| 408 |
+
this.xs.release(slotsU);
|
| 409 |
+
}
|
| 410 |
+
let result;
|
| 411 |
+
if (want === 'top1') {
|
| 412 |
+
await this.rbTop1.mapAsync(1, 0, 2 * T * 4);
|
| 413 |
+
const u = new Uint32Array(this.rbTop1.getMappedRange(0, 2 * T * 4).slice(0)), f = new Float32Array(u.buffer);
|
| 414 |
+
this.rbTop1.unmap();
|
| 415 |
+
result = { token: new Int32Array(T), conf: new Float32Array(T) };
|
| 416 |
+
for (let t = 0; t < T; t++) { result.token[t] = u[2 * t]; result.conf[t] = f[2 * t + 1]; }
|
| 417 |
+
} else {
|
| 418 |
+
await this.rbBlockLogits.mapAsync(1, 0, T * c.vocab * 4);
|
| 419 |
+
result = new Float32Array(this.rbBlockLogits.getMappedRange(0, T * c.vocab * 4).slice(0));
|
| 420 |
+
this.rbBlockLogits.unmap();
|
| 421 |
+
}
|
| 422 |
+
this.counters.previews = (this.counters.previews || 0) + 1;
|
| 423 |
+
this.counters.previewExperts = (this.counters.previewExperts || 0) + units.size;
|
| 424 |
+
this.counters.previewMs = (this.counters.previewMs || 0) + performance.now() - tStart;
|
| 425 |
+
return result;
|
| 426 |
+
}
|
| 427 |
+
|
| 428 |
+
// The last preview's per-layer residual rows: layers × T × hidden (capture: true only).
|
| 429 |
+
async readCapture(T = this.block) {
|
| 430 |
+
const c = this.cfg, n = c.layers * this.block * c.hidden * 4, enc = this.device.createCommandEncoder();
|
| 431 |
+
enc.copyBufferToBuffer(this.p.cap, 0, this.rbCap, 0, n);
|
| 432 |
+
this.device.queue.submit([enc.finish()]);
|
| 433 |
+
await this.rbCap.mapAsync(1);
|
| 434 |
+
const all = new Float32Array(this.rbCap.getMappedRange().slice(0));
|
| 435 |
+
this.rbCap.unmap();
|
| 436 |
+
return Array.from({ length: c.layers }, (_, l) => all.subarray(l * this.block * c.hidden, (l * this.block + T) * c.hidden));
|
| 437 |
+
}
|
| 438 |
+
|
| 439 |
+
// The callback kohra's denoise module drives (one call at a time, under the engine lock):
|
| 440 |
+
// commit: appends ids at positions [start, start + ids.length); start is the cache length, or 0 for a new prompt
|
| 441 |
+
// (which reuses the cached prefix it shares). Returns null (Clover commits never need logits).
|
| 442 |
+
// preview: a denoising pass over the block at the cache end; see preview().
|
| 443 |
+
async forward(args) {
|
| 444 |
+
const release = await this.acquire();
|
| 445 |
+
try { return await this.forwardLocked(args); } catch (e) { throw this.stepError(e); } finally { release(); }
|
| 446 |
+
}
|
| 447 |
+
async forwardLocked({ mode, start, ids, wantLogits = mode === 'preview' ? 'all' : 'none' }) {
|
| 448 |
+
ids = Array.from(ids);
|
| 449 |
+
if (mode === 'preview') {
|
| 450 |
+
if (start !== this.position) throw new Error(`preview at ${start}, but the cache holds ${this.position} tokens`);
|
| 451 |
+
if (wantLogits !== 'all' && wantLogits !== 'top1') throw new Error(`preview returns 'all' or 'top1', not ${wantLogits}`);
|
| 452 |
+
return await this.preview(ids, wantLogits);
|
| 453 |
+
}
|
| 454 |
+
if (mode !== 'commit') throw new Error(`forward mode ${mode}`);
|
| 455 |
+
if (wantLogits !== 'none') throw new Error('Clover commits return no logits');
|
| 456 |
+
if (start === 0) { await this.prefillSegments([{ ids }], 'none', { sysEnd: this.checkpointEnd(ids) }); return null; }
|
| 457 |
+
if (start !== this.position) throw new Error(`commit at ${start}, but the cache holds ${this.position} tokens`);
|
| 458 |
+
for (let i = 0; i < ids.length; i += this.chunkTokens) {
|
| 459 |
+
const chunk = ids.slice(i, i + this.chunkTokens);
|
| 460 |
+
if (chunk.length < 2) await this.step(chunk[0], 'none'); else await this.prefillChunk(chunk, 'none');
|
| 461 |
+
}
|
| 462 |
+
return null;
|
| 463 |
+
}
|
| 464 |
+
|
| 465 |
+
// A chat turn (generate() holds the engine lock around it): kohra's denoise loop with Clover's decoding rules
|
| 466 |
+
// (PRESETS.clover: threshold 0.95, the reference generation config's penalties), yielding text block by block.
|
| 467 |
+
// opts beyond maxNewTokens / signal / enableThinking go to denoise() (e.g. penalties: null, top1: true).
|
| 468 |
+
async *generateTurn(messages, { maxNewTokens = 512, signal, enableThinking = true, ...loop } = {}) {
|
| 469 |
+
const promptIds = this.tokenizer.encode(this.chatPrompt(messages, { enableThinking }));
|
| 470 |
+
const blocks = [], out = [];
|
| 471 |
+
let wake = null, done = false, failed = null, stop = false;
|
| 472 |
+
const STOP = new Error('stopped');
|
| 473 |
+
const forward = (a) => { if (stop || (signal && signal.aborted)) throw STOP; return this.forwardLocked(a); };
|
| 474 |
+
const run = denoise({ ...PRESETS.clover, top1: true, ...loop, forward, promptIds, maxNewTokens, maskId: this.maskId, blockSize: this.block,
|
| 475 |
+
onBlock: ({ tokens }) => { blocks.push(tokens); if (wake) wake(); } })
|
| 476 |
+
.then((r) => { this.lastDenoise = { previews: r.previews, commits: r.commits, seconds: r.seconds, tokens: r.tokenIds.length }; },
|
| 477 |
+
(e) => { if (e !== STOP) failed = e; })
|
| 478 |
+
.finally(() => { done = true; if (wake) wake(); });
|
| 479 |
+
try {
|
| 480 |
+
for (;;) {
|
| 481 |
+
while (blocks.length) {
|
| 482 |
+
for (const token of blocks.shift()) { out.push(token); yield { text: this.tokenizer.decode(out), token, tokens: out.length }; }
|
| 483 |
+
}
|
| 484 |
+
if (done) break;
|
| 485 |
+
await new Promise((r) => { wake = r; });
|
| 486 |
+
wake = null;
|
| 487 |
+
}
|
| 488 |
+
if (failed) throw failed;
|
| 489 |
+
} finally { stop = true; await run; }
|
| 490 |
+
}
|
| 491 |
+
|
| 492 |
+
stats() {
|
| 493 |
+
const n = Math.max(1, this.counters.previews || 0);
|
| 494 |
+
return { ...super.stats(), previews: this.counters.previews || 0, previewMsEach: (this.counters.previewMs || 0) / n, expertsPerPreview: (this.counters.previewExperts || 0) / n };
|
| 495 |
+
}
|
| 496 |
+
}
|
| 497 |
+
|
| 498 |
+
export { PREVIEW_KERNELS as CLOVER_PREVIEW_KERNELS };
|
classic/engines/qwen35_moe_ssd.js
CHANGED
|
@@ -741,6 +741,7 @@ function guarded(src) {
|
|
| 741 |
}
|
| 742 |
|
| 743 |
export class Qwen35MoeSsd extends Qwen3MoeSsd {
|
|
|
|
| 744 |
// Without a url, the pinned Hugging Face file; with one (a local serve, a layer cut), the OPFS
|
| 745 |
// directory is named after the file, never after rung 2a's model.
|
| 746 |
static async load(modelId = null, opts = {}) {
|
|
@@ -751,7 +752,7 @@ export class Qwen35MoeSsd extends Qwen3MoeSsd {
|
|
| 751 |
constructor(device, manifest, gguf, opts) {
|
| 752 |
super(device, manifest, gguf, opts);
|
| 753 |
const c = this.cfg;
|
| 754 |
-
if (c.arch
|
| 755 |
if (c.ssm.dConv !== 4) throw new Error(`gdnConv assumes a 4-tap conv, got ${c.ssm.dConv}`);
|
| 756 |
if (c.ssm.dState !== 128 || c.ssm.dInner !== c.ssm.vHeads * c.ssm.dState) throw new Error('gdnStep assumes 128-wide heads');
|
| 757 |
// Weight type: Q8_0 (Qwen3.6, MiniCPM-V Q8_0) or Q4_0 (MiniCPM-V Q4_0: Q4_0 matrices and experts, Q6_K output).
|
|
|
|
| 741 |
}
|
| 742 |
|
| 743 |
export class Qwen35MoeSsd extends Qwen3MoeSsd {
|
| 744 |
+
static archs = ['qwen35moe'];
|
| 745 |
// Without a url, the pinned Hugging Face file; with one (a local serve, a layer cut), the OPFS
|
| 746 |
// directory is named after the file, never after rung 2a's model.
|
| 747 |
static async load(modelId = null, opts = {}) {
|
|
|
|
| 752 |
constructor(device, manifest, gguf, opts) {
|
| 753 |
super(device, manifest, gguf, opts);
|
| 754 |
const c = this.cfg;
|
| 755 |
+
if (!this.constructor.archs.includes(c.arch)) throw new Error(`${this.constructor.name} needs a ${this.constructor.archs.join(' or ')} GGUF, got ${c.arch}`);
|
| 756 |
if (c.ssm.dConv !== 4) throw new Error(`gdnConv assumes a 4-tap conv, got ${c.ssm.dConv}`);
|
| 757 |
if (c.ssm.dState !== 128 || c.ssm.dInner !== c.ssm.vHeads * c.ssm.dState) throw new Error('gdnStep assumes 128-wide heads');
|
| 758 |
// Weight type: Q8_0 (Qwen3.6, MiniCPM-V Q8_0) or Q4_0 (MiniCPM-V Q4_0: Q4_0 matrices and experts, Q6_K output).
|
classic/engines/qwen3_moe_ssd.js
CHANGED
|
@@ -43,12 +43,13 @@ export { FORMAT as STORE_FORMAT, OPFS_ROOT as STORE_ROOT };
|
|
| 43 |
|
| 44 |
// ── GGUF ────────────────────────────────────────────────────────────────────
|
| 45 |
// qwen3moe (rung 2a) and qwen35moe (Qwen3.5/3.6 MoE, rung 2b: qwen35_moe_ssd.js) share the
|
| 46 |
-
// expert layout; qwen35moe adds the Gated DeltaNet (ssm.*), partial RoPE and a shared expert.
|
|
|
|
| 47 |
export function configFromGguf(kv) {
|
| 48 |
const a = kv['general.architecture'];
|
| 49 |
-
if (a !== 'qwen3moe' && a !== 'qwen35moe') throw new Error(`expected a qwen3moe or
|
| 50 |
const g = (k) => kv[`${a}.${k}`];
|
| 51 |
-
const extra = a
|
| 52 |
ropeDims: g('rope.dimension_count'), shexpFf: g('expert_shared_feed_forward_length'),
|
| 53 |
attnInterval: g('full_attention_interval') || 4,
|
| 54 |
ssm: { dConv: g('ssm.conv_kernel'), dInner: g('ssm.inner_size'), dState: g('ssm.state_size'), vHeads: g('ssm.time_step_rank'), kHeads: g('ssm.group_count') },
|
|
|
|
| 43 |
|
| 44 |
// ── GGUF ────────────────────────────────────────────────────────────────────
|
| 45 |
// qwen3moe (rung 2a) and qwen35moe (Qwen3.5/3.6 MoE, rung 2b: qwen35_moe_ssd.js) share the
|
| 46 |
+
// expert layout; qwen35moe adds the Gated DeltaNet (ssm.*), partial RoPE and a shared expert. clover_moe (Clover,
|
| 47 |
+
// block diffusion on Qwen3.6: clover_moe_ssd.js) is qwen35moe plus per-DeltaNet-layer preview tensors.
|
| 48 |
export function configFromGguf(kv) {
|
| 49 |
const a = kv['general.architecture'];
|
| 50 |
+
if (a !== 'qwen3moe' && a !== 'qwen35moe' && a !== 'clover_moe') throw new Error(`expected a qwen3moe, qwen35moe or clover_moe GGUF, got ${a}`);
|
| 51 |
const g = (k) => kv[`${a}.${k}`];
|
| 52 |
+
const extra = a === 'qwen3moe' ? {} : {
|
| 53 |
ropeDims: g('rope.dimension_count'), shexpFf: g('expert_shared_feed_forward_length'),
|
| 54 |
attnInterval: g('full_attention_interval') || 4,
|
| 55 |
ssm: { dConv: g('ssm.conv_kernel'), dInner: g('ssm.inner_size'), dState: g('ssm.state_size'), vHeads: g('ssm.time_step_rank'), kHeads: g('ssm.group_count') },
|
classic/engines/vendor/kohra-denoise.js
ADDED
|
@@ -0,0 +1,308 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// denoise.js: the block-diffusion decoding loop, with no runtime inside.
|
| 2 |
+
//
|
| 3 |
+
// The caller supplies one async forward pass:
|
| 4 |
+
//
|
| 5 |
+
// forward({ mode, start, ids, wantLogits }) -> Float32Array | null
|
| 6 |
+
//
|
| 7 |
+
// mode 'preview' the denoise pass over the current block: reads the committed cache, writes
|
| 8 |
+
// nothing, returns [ids.length * V] fp32 logits (row-major). wantLogits = 'all'.
|
| 9 |
+
// mode 'commit' the causal pass: appends ids at absolute positions [start, start + ids.length).
|
| 10 |
+
// Prefill is the first commit; commits arrive in order with no gaps.
|
| 11 |
+
// wantLogits 'last' returns the last position's V logits; 'none' returns null.
|
| 12 |
+
// Optional fast path (option top1): a greedy, penalty-free preview asks wantLogits 'top1' and gets
|
| 13 |
+
// { token: Int32Array(n), conf: Float32Array(n) }: the per-row argmax with the mask id excluded and
|
| 14 |
+
// its softmax probability at T = 1 (mask logit at -inf). A runtime without it throws, and the loop
|
| 15 |
+
// falls back to 'all' for the rest of the run.
|
| 16 |
+
//
|
| 17 |
+
// Logits are same-position: row i scores slot start + i (not the next token).
|
| 18 |
+
//
|
| 19 |
+
// `ids` is an Int32Array; masked slots hold `maskId`. The loop reads the returned array during the
|
| 20 |
+
// step and never keeps it, so the caller owns buffers and disposal.
|
| 21 |
+
// kohra's ONNX path (kohra.js) and diskformer's clover_moe engine both drive this loop.
|
| 22 |
+
|
| 23 |
+
const DEFAULTS = {
|
| 24 |
+
maskId: 151669,
|
| 25 |
+
eosIds: [], // stop tokens; the response is trimmed before the first one
|
| 26 |
+
blockSize: 32,
|
| 27 |
+
maxNewTokens: 128,
|
| 28 |
+
threshold: null, // number: fill every masked position past it per preview (min 1).
|
| 29 |
+
// null: fixed reveal schedule over `steps` previews.
|
| 30 |
+
strict: false, // threshold compare: true uses conf > threshold, false conf >= threshold
|
| 31 |
+
steps: 128, // fixed-schedule mode only: total previews, split evenly over blocks
|
| 32 |
+
maxStepsPerBlock: null, // threshold mode: throw if a block still has masks after this many previews
|
| 33 |
+
temperature: 0, // 0 = argmax; > 0 = sample softmax(logits / temperature)
|
| 34 |
+
topK: 0, // sampling filters (temperature > 0 only); 0 / 1 = off
|
| 35 |
+
topP: 1,
|
| 36 |
+
remasking: 'low_confidence', // or 'random'
|
| 37 |
+
suppressMask: false, // set the mask token's logit to -inf before scoring
|
| 38 |
+
seed: false, // a commit's last logit picks the next block's first token
|
| 39 |
+
alignToGrid: false, // blocks on the absolute pos // blockSize grid (first block holds the prompt tail)
|
| 40 |
+
fullLastBlock: false, // denoise the last block at full blockSize, then trim to maxNewTokens
|
| 41 |
+
stopAtEos: false, // stop after a resolved block that holds a stop token
|
| 42 |
+
penalties: null, // see applyPenalties
|
| 43 |
+
top1: false, // try the 'top1' preview fast path when greedy and penalty-free
|
| 44 |
+
onStep: null, // ({ canvas, P, start, end, fresh, block, numBlocks, previews, elapsedMs })
|
| 45 |
+
onBlock: null, // ({ tokens, block }) after each resolved block (response tokens only)
|
| 46 |
+
yieldEvery: 1, // release the event loop every N previews
|
| 47 |
+
};
|
| 48 |
+
|
| 49 |
+
// Clover-35B-A3B-Preview: generation_config.json plus the denoising config in modeling_clover_moe.py.
|
| 50 |
+
export const PRESETS = {
|
| 51 |
+
clover: {
|
| 52 |
+
maskId: 151669,
|
| 53 |
+
eosIds: [248046, 248044],
|
| 54 |
+
blockSize: 32,
|
| 55 |
+
threshold: 0.95,
|
| 56 |
+
strict: true,
|
| 57 |
+
maxStepsPerBlock: 32,
|
| 58 |
+
suppressMask: true,
|
| 59 |
+
fullLastBlock: true,
|
| 60 |
+
stopAtEos: true,
|
| 61 |
+
penalties: {
|
| 62 |
+
presence: 0.9, frequency: 0, repetition: 1.05,
|
| 63 |
+
length: 10, lengthStart: 8192, lengthWindow: 8192,
|
| 64 |
+
thinkEnd: 10, thinkEndStart: 4096, thinkEndWindow: 4096, thinkEndId: 248069,
|
| 65 |
+
stopIds: [248044, 248046], // eos ids plus pad (248044)
|
| 66 |
+
},
|
| 67 |
+
},
|
| 68 |
+
};
|
| 69 |
+
|
| 70 |
+
// Cooperative yield that lets the UI paint without waiting on the compositor.
|
| 71 |
+
// requestAnimationFrame stalls in a hidden tab; a MessageChannel macrotask always fires.
|
| 72 |
+
export const yieldToEventLoop = (() => {
|
| 73 |
+
if (typeof MessageChannel === 'undefined') {
|
| 74 |
+
return () => new Promise((resolve) => setTimeout(resolve, 0));
|
| 75 |
+
}
|
| 76 |
+
const ch = new MessageChannel();
|
| 77 |
+
const queue = [];
|
| 78 |
+
// ref/unref exist only in Node: hold the port open while a yield is pending, so an
|
| 79 |
+
// idle port never keeps the process alive and a pending one never lets it exit.
|
| 80 |
+
ch.port1.onmessage = () => { queue.shift()?.(); if (!queue.length) ch.port1.unref?.(); };
|
| 81 |
+
ch.port1.unref?.();
|
| 82 |
+
return () => new Promise((resolve) => { queue.push(resolve); ch.port1.ref?.(); ch.port2.postMessage(0); });
|
| 83 |
+
})();
|
| 84 |
+
|
| 85 |
+
// Per-block reveal counts for the linear-alpha (MDLM) scheduler: round remaining/(steps-i)
|
| 86 |
+
// per step, skip zero entries. Mirrors dllm get_num_transfer_tokens.
|
| 87 |
+
export function transferSchedule(nMasked, steps) {
|
| 88 |
+
const ks = [];
|
| 89 |
+
let remaining = nMasked;
|
| 90 |
+
for (let i = 0; i < steps && remaining > 0; i++) {
|
| 91 |
+
const k = Math.min(Math.round(remaining / (steps - i)), remaining);
|
| 92 |
+
if (k > 0) { ks.push(k); remaining -= k; }
|
| 93 |
+
}
|
| 94 |
+
return ks;
|
| 95 |
+
}
|
| 96 |
+
|
| 97 |
+
// Response-only penalties, in CloverPenaltyState.apply order (sampling_clover_moe.py):
|
| 98 |
+
// frequency, presence, repetition over committed response tokens, then the length ramp on
|
| 99 |
+
// stop ids and the think-end ramp. `counts` holds committed response tokens only; tokens still
|
| 100 |
+
// in the block never penalise themselves. Mutates rows of `logits` in place.
|
| 101 |
+
export function applyPenalties(logits, rows, V, { counts, length, block, maskId, p }) {
|
| 102 |
+
// Every operation rounds to fp32 the way PyTorch's fp32 kernels do, with scalars cast to fp32.
|
| 103 |
+
const fp = Math.fround;
|
| 104 |
+
const ramp = (row, start, window, strength) =>
|
| 105 |
+
fp(fp(Math.min(Math.max(fp(fp(length + row + 1 - start) / fp(window)), 0), 1)) * fp(strength));
|
| 106 |
+
let closeOpen = false;
|
| 107 |
+
if (p.thinkEnd) {
|
| 108 |
+
closeOpen = !counts.has(p.thinkEndId);
|
| 109 |
+
for (let i = 0; i < block.length && closeOpen; i++) {
|
| 110 |
+
if (block[i] !== maskId && block[i] === p.thinkEndId) closeOpen = false;
|
| 111 |
+
}
|
| 112 |
+
}
|
| 113 |
+
const freq = fp(p.frequency ?? 0), pres = fp(p.presence ?? 0), rep = fp(p.repetition ?? 1);
|
| 114 |
+
for (let r = 0; r < rows; r++) {
|
| 115 |
+
const off = r * V;
|
| 116 |
+
for (const [tok, n] of counts) {
|
| 117 |
+
let l = logits[off + tok];
|
| 118 |
+
if (freq) l = fp(l - fp(n * freq));
|
| 119 |
+
if (pres) l = fp(l - pres);
|
| 120 |
+
if (rep !== 1) l = fp(l < 0 ? l * rep : l / rep);
|
| 121 |
+
logits[off + tok] = l;
|
| 122 |
+
}
|
| 123 |
+
if (p.length && p.stopIds?.length) {
|
| 124 |
+
const add = ramp(r, p.lengthStart, p.lengthWindow, p.length);
|
| 125 |
+
for (const s of p.stopIds) logits[off + s] = fp(logits[off + s] + add);
|
| 126 |
+
}
|
| 127 |
+
if (closeOpen) {
|
| 128 |
+
logits[off + p.thinkEndId] = fp(logits[off + p.thinkEndId] + ramp(r, p.thinkEndStart, p.thinkEndWindow, p.thinkEnd));
|
| 129 |
+
}
|
| 130 |
+
}
|
| 131 |
+
}
|
| 132 |
+
|
| 133 |
+
// Pick a token for one logits row and its confidence (softmax at T = 1 of the same row).
|
| 134 |
+
function pick(logits, off, V, cfg) {
|
| 135 |
+
let maxL = -Infinity;
|
| 136 |
+
for (let v = 0; v < V; v++) if (logits[off + v] > maxL) maxL = logits[off + v];
|
| 137 |
+
let sumExp = 0;
|
| 138 |
+
for (let v = 0; v < V; v++) sumExp += Math.exp(logits[off + v] - maxL);
|
| 139 |
+
let best = 0;
|
| 140 |
+
if (cfg.temperature > 0) {
|
| 141 |
+
const keep = samplingFilter(logits, off, V, cfg);
|
| 142 |
+
let bestScore = -Infinity;
|
| 143 |
+
for (let v = 0; v < V; v++) {
|
| 144 |
+
if (keep && !keep(logits[off + v], v)) continue;
|
| 145 |
+
// Gumbel-max on logits / temperature == a draw from softmax(logits / temperature).
|
| 146 |
+
const s = logits[off + v] / cfg.temperature - Math.log(-Math.log(cfg.random() + 1e-12) + 1e-12);
|
| 147 |
+
if (s > bestScore) { bestScore = s; best = v; }
|
| 148 |
+
}
|
| 149 |
+
} else {
|
| 150 |
+
for (let v = 1; v < V; v++) if (logits[off + v] > logits[off + best]) best = v;
|
| 151 |
+
}
|
| 152 |
+
return { tok: best, conf: Math.exp(logits[off + best] - maxL) / sumExp };
|
| 153 |
+
}
|
| 154 |
+
|
| 155 |
+
// top-k then top-p over logits / temperature, as predict_with_confidence does. Returns null when off.
|
| 156 |
+
function samplingFilter(logits, off, V, cfg) {
|
| 157 |
+
const k = cfg.topK > 0 && cfg.topK < V ? cfg.topK : 0;
|
| 158 |
+
const topP = cfg.topP != null && cfg.topP < 1 ? cfg.topP : 0;
|
| 159 |
+
if (!k && !topP) return null;
|
| 160 |
+
const order = Array.from({ length: V }, (_, v) => v).sort((a, b) => logits[off + b] - logits[off + a]);
|
| 161 |
+
let n = k || V;
|
| 162 |
+
if (topP) {
|
| 163 |
+
const T = cfg.temperature, top = logits[off + order[0]] / T;
|
| 164 |
+
let z = 0;
|
| 165 |
+
for (let i = 0; i < n; i++) z += Math.exp(logits[off + order[i]] / T - top);
|
| 166 |
+
let cum = 0;
|
| 167 |
+
for (let i = 0; i < n; i++) {
|
| 168 |
+
const pr = Math.exp(logits[off + order[i]] / T - top) / z;
|
| 169 |
+
if (cum > topP) { n = i; break; } // drop once the mass before this token passes top_p
|
| 170 |
+
cum += pr;
|
| 171 |
+
}
|
| 172 |
+
}
|
| 173 |
+
const allowed = new Set(order.slice(0, Math.max(n, 1)));
|
| 174 |
+
return (_, v) => allowed.has(v);
|
| 175 |
+
}
|
| 176 |
+
|
| 177 |
+
// Run the loop. `promptIds` is an array of token ids. Returns
|
| 178 |
+
// { tokenIds, stopToken, canvas, P, previews, commits, seconds }.
|
| 179 |
+
export async function denoise({ forward, promptIds, ...options }) {
|
| 180 |
+
const cfg = { random: Math.random, ...DEFAULTS, ...options };
|
| 181 |
+
const MASK = cfg.maskId, bs = cfg.blockSize;
|
| 182 |
+
const eos = new Set(cfg.eosIds);
|
| 183 |
+
const P = promptIds.length;
|
| 184 |
+
const maxNew = cfg.maxNewTokens;
|
| 185 |
+
|
| 186 |
+
// Blocks to denoise, left to right. Prompt-aligned: blocks start at P. Grid-aligned:
|
| 187 |
+
// blocks live on pos // blockSize, so the first one holds the prompt tail.
|
| 188 |
+
const first = cfg.alignToGrid ? Math.floor(P / bs) * bs : P;
|
| 189 |
+
const limit = P + maxNew;
|
| 190 |
+
const blocks = [];
|
| 191 |
+
for (let s = first; s < limit; s += bs) blocks.push([s, cfg.fullLastBlock ? s + bs : Math.min(s + bs, limit)]);
|
| 192 |
+
const numBlocks = blocks.length;
|
| 193 |
+
const T = numBlocks ? blocks[numBlocks - 1][1] : P;
|
| 194 |
+
const stepsPerBlock = Math.ceil(cfg.steps / Math.max(numBlocks, 1));
|
| 195 |
+
|
| 196 |
+
const canvas = new Int32Array(Math.max(T, P)).fill(MASK);
|
| 197 |
+
canvas.set(promptIds, 0);
|
| 198 |
+
|
| 199 |
+
const counts = new Map(); // committed response tokens, for penalties
|
| 200 |
+
let responseLength = 0;
|
| 201 |
+
let previews = 0, commits = 0;
|
| 202 |
+
let top1 = cfg.top1 && !(cfg.temperature > 0) && !cfg.penalties && cfg.remasking !== 'random';
|
| 203 |
+
const t0 = performance.now();
|
| 204 |
+
|
| 205 |
+
const commit = async (s, e, next) => {
|
| 206 |
+
const wantLogits = cfg.seed && next ? 'last' : 'none';
|
| 207 |
+
const out = await forward({ mode: 'commit', start: s, ids: canvas.slice(s, e), wantLogits });
|
| 208 |
+
commits++;
|
| 209 |
+
if (wantLogits === 'last' && canvas[next] === MASK) {
|
| 210 |
+
const V = out.length;
|
| 211 |
+
let best = 0;
|
| 212 |
+
for (let v = 1; v < V; v++) if (out[v] > out[best]) best = v;
|
| 213 |
+
canvas[next] = best;
|
| 214 |
+
}
|
| 215 |
+
};
|
| 216 |
+
|
| 217 |
+
if (numBlocks && first > 0) await commit(0, first, first); // prefill
|
| 218 |
+
|
| 219 |
+
let stopToken = null, genEnd = limit;
|
| 220 |
+
for (let b = 0; b < numBlocks; b++) {
|
| 221 |
+
const [start, end] = blocks[b];
|
| 222 |
+
const masked = () => { let n = 0; for (let i = start; i < end; i++) if (canvas[i] === MASK) n++; return n; };
|
| 223 |
+
const nMasked = masked();
|
| 224 |
+
const schedule = cfg.threshold == null ? transferSchedule(nMasked, stepsPerBlock) : null;
|
| 225 |
+
let step = 0;
|
| 226 |
+
|
| 227 |
+
while (masked() > 0) {
|
| 228 |
+
if (schedule && step >= schedule.length) break;
|
| 229 |
+
if (!schedule && cfg.maxStepsPerBlock != null && step >= cfg.maxStepsPerBlock) {
|
| 230 |
+
throw new Error(`denoise: block ${b} still has masks after ${step} previews`);
|
| 231 |
+
}
|
| 232 |
+
const ids = canvas.slice(start, end);
|
| 233 |
+
const cand = [];
|
| 234 |
+
let fast = null;
|
| 235 |
+
if (top1) {
|
| 236 |
+
try { fast = await forward({ mode: 'preview', start, ids, wantLogits: 'top1' }); } catch { top1 = false; }
|
| 237 |
+
}
|
| 238 |
+
if (fast) {
|
| 239 |
+
previews++;
|
| 240 |
+
for (let p = start; p < end; p++) {
|
| 241 |
+
if (canvas[p] === MASK) cand.push({ p, tok: fast.token[p - start], conf: fast.conf[p - start] });
|
| 242 |
+
}
|
| 243 |
+
} else {
|
| 244 |
+
const logits = await forward({ mode: 'preview', start, ids, wantLogits: 'all' });
|
| 245 |
+
previews++;
|
| 246 |
+
const rows = end - start, V = logits.length / rows;
|
| 247 |
+
if (cfg.suppressMask) for (let r = 0; r < rows; r++) logits[r * V + MASK] = -Infinity;
|
| 248 |
+
if (cfg.penalties) {
|
| 249 |
+
applyPenalties(logits, rows, V, { counts, length: responseLength, block: ids, maskId: MASK, p: cfg.penalties });
|
| 250 |
+
}
|
| 251 |
+
for (let p = start; p < end; p++) {
|
| 252 |
+
if (canvas[p] !== MASK) continue;
|
| 253 |
+
const { tok, conf } = pick(logits, (p - start) * V, V, cfg);
|
| 254 |
+
cand.push({ p, tok, conf: cfg.remasking === 'random' ? cfg.random() : conf });
|
| 255 |
+
}
|
| 256 |
+
}
|
| 257 |
+
|
| 258 |
+
let chosen;
|
| 259 |
+
if (schedule) {
|
| 260 |
+
cand.sort((x, y) => y.conf - x.conf);
|
| 261 |
+
chosen = cand.slice(0, schedule[step]);
|
| 262 |
+
} else {
|
| 263 |
+
const thr = cfg.threshold;
|
| 264 |
+
chosen = cand.filter((c) => (cfg.strict ? c.conf > thr : c.conf >= thr));
|
| 265 |
+
if (chosen.length === 0) { // progress floor: the single most confident
|
| 266 |
+
let top = cand[0];
|
| 267 |
+
for (const c of cand) if (c.conf > top.conf) top = c;
|
| 268 |
+
chosen = [top];
|
| 269 |
+
}
|
| 270 |
+
}
|
| 271 |
+
step++;
|
| 272 |
+
|
| 273 |
+
const fresh = new Set();
|
| 274 |
+
for (const { p, tok } of chosen) { canvas[p] = tok; fresh.add(p); }
|
| 275 |
+
cfg.onStep?.({ canvas, P, start, end, fresh, block: b, numBlocks, previews, elapsedMs: performance.now() - t0 });
|
| 276 |
+
if (cfg.yieldEvery && previews % cfg.yieldEvery === 0) await yieldToEventLoop();
|
| 277 |
+
}
|
| 278 |
+
|
| 279 |
+
// Response tokens of this block, in position order, cut at the first stop token.
|
| 280 |
+
// Stop only after the whole block resolves: an earlier slot can fill after a later EOS.
|
| 281 |
+
const respStart = Math.max(start, P), respEnd = Math.min(end, limit);
|
| 282 |
+
let cut = respEnd;
|
| 283 |
+
if (cfg.stopAtEos) {
|
| 284 |
+
for (let i = respStart; i < respEnd; i++) if (eos.has(canvas[i])) { cut = i + 1; stopToken = canvas[i]; break; }
|
| 285 |
+
}
|
| 286 |
+
if (cfg.penalties) {
|
| 287 |
+
for (let i = respStart; i < cut; i++) counts.set(canvas[i], (counts.get(canvas[i]) ?? 0) + 1);
|
| 288 |
+
}
|
| 289 |
+
responseLength += cut - respStart;
|
| 290 |
+
cfg.onBlock?.({ tokens: Array.from(canvas.subarray(respStart, stopToken == null ? cut : cut - 1)), block: b });
|
| 291 |
+
if (stopToken != null) { genEnd = cut - 1; break; }
|
| 292 |
+
if (b + 1 < numBlocks && end < limit) await commit(start, end, blocks[b + 1][0]);
|
| 293 |
+
}
|
| 294 |
+
|
| 295 |
+
// Trim at the first stop token after the prompt (fixed-schedule runs fill every block first).
|
| 296 |
+
if (stopToken == null) {
|
| 297 |
+
for (let i = P; i < limit; i++) if (eos.has(canvas[i])) { genEnd = i; stopToken = canvas[i]; break; }
|
| 298 |
+
}
|
| 299 |
+
return {
|
| 300 |
+
tokenIds: Array.from(canvas.subarray(P, Math.min(genEnd, limit))),
|
| 301 |
+
stopToken,
|
| 302 |
+
canvas,
|
| 303 |
+
P,
|
| 304 |
+
previews,
|
| 305 |
+
commits,
|
| 306 |
+
seconds: (performance.now() - t0) / 1000,
|
| 307 |
+
};
|
| 308 |
+
}
|