Reductions refuse what they used to accept: GPU row-max non-finite guard, averageGrads length/NaN refusal, int32 accumulator bound, capacity-weighted cluster loss, finite reduced-gradient check, empty CUDA_VISIBLE_DEVICES

README.md

@@ -303,7 +303,7 @@ the gate; a stalled roster gradient is a bug in the *protocol*, where every
 computed value on every peer was correct, so no data oracle could fire. Those
 needed different instruments, not better oracles.
 
-All twelve test suites (`cd web && npm test`) pass; results with methodology in
+All thirteen test suites (`cd web && npm test`) pass; results with methodology in
 [`web/TEST_RESULTS.md`](web/TEST_RESULTS.md).
 
 ---
@@ -372,6 +372,30 @@ web/                         DaisyChain-Web: P2P browser training (WebRTC + WebG
 export_luts_web.py           regenerates web/public LUTs from the trained units
 ```
 
+## Recent updates (30 September 2026) — reductions refuse what they used to accept
+
+Six fixes, each found by applying a lesson from differential-testing an RDNA2 shader
+translator: *a second implementation nobody checks is not verified*, *inactive or absent
+participants must not leak into a reduction*, *a max over NaN is undefined*, and *know your
+accumulator width*. Every one is covered by a check that fails on the old code.
+`python test_verified_units.py` + `python test_cluster_reductions.py` and
+`cd web && npm test` (thirteen suites) reproduce all of it.
+
+| # | where | what was wrong | now |
+| --- | --- | --- | --- |
+| 1 | browser, WebGPU MLP chain | the row max comes from an `atomicMax` over `bitcast(abs(h))` and never passed the non-finite guard the CPU path has; a NaN/Inf `h1` became `inv = 0`/NaN and the GPU quantized `floor(NaN)` (implementation-defined) where a CPU peer threw | `scalesFromAbsMax` refuses a non-finite max. NaN and Inf bits order above every finite value as u32, so the row max is non-finite **iff** some element is — one check covers the whole row (checked on 2000 random rows) |
+| 2 | browser, `averageGrads` | a gradient of a different length was silently averaged over a prefix (typed arrays ignore out-of-range writes); a NaN/Inf gradient went straight into DaisyAdam, poisoning `m`/`v` on **every** peer identically, so no fork check could see it | refuses both with `GradientError`; the training loop halts at that step with weights and Adam state untouched (still a valid checkpoint) |
+| 3 | browser, int8 GEMMs (JS and WGSL) | the `Int32Array` and WGSL `i32` accumulators wrap past K = 131,071 — CPU and GPU agree, on a wrong sum | refused with `AccumulatorBoundError` at every GEMM entry point; the same bound as the Python kernel's `MAX_INT32_K` |
+| 4 | Python, logged cluster loss | `sum(loss_i) / world` — an equal-weight mean of **unequal**, capacity-weighted batches | `cluster_mean_loss` = Σ wᵢ·lossᵢ with the gradient's own weights. Measured on 3- and 13-sample nodes: old **1.418**, true full-batch loss **0.959**, new 0.959 |
+| 5 | Python, gradient all-reduce | one node's NaN was summed into every replica and applied | `check_finite_grads` refuses the reduced gradient before `opt.step()`; every rank holds the same sum, so every rank stops at the same step with no extra communication |
+| 6 | Python, `pick_device` | with `CUDA_VISIBLE_DEVICES` set but **empty** (a common "CPU only"), torch reports `is_available() = True` and `device_count() = 0`, and the node crashed at startup | a device must actually exist; falls back to CPU |
+
+Unchanged for valid inputs: finite gradients, finite activations, and K ≤ 131,071 produce
+bit-identical results to before, so updated and older peers still co-train. A 2-rank
+end-to-end `DaisyCluster.fit` run stays bit-identical across replicas (`replica_diff 0.0`).
+The WebGPU-side changes are host-side JavaScript (checked by `node --check` and by the
+Node suites through the shared mirror functions); the WGSL kernels themselves are unchanged.
+
 ## Recent updates (September 2026) — requant rounding, bounded units, non-finite guards
 
 Everything below is measured, and `python test_verified_units.py` (25 checks) plus

daisychain/cluster.py

@@ -31,14 +31,21 @@ def configure_cpu(fraction: float = 0.9) -> int:
     return n
 
 
+def _cuda_usable() -> bool:
+    """is_available() alone is not enough: with CUDA_VISIBLE_DEVICES set but EMPTY (a common way
+    to say "CPU only") torch reports available=True and device_count()=0, and the node crashed
+    at startup in get_device_properties(0). Require a device to actually exist (2026-09-30)."""
+    return torch.cuda.is_available() and torch.cuda.device_count() > 0
+
+
 def pick_device() -> "torch.device":
     if os.environ.get("DAISY_FORCE_CPU") == "1":
         return torch.device("cpu")
-    return torch.device("cuda" if torch.cuda.is_available() else "cpu")
+    return torch.device("cuda" if _cuda_usable() else "cpu")
 
 
 def _gpu_info():
-    if not torch.cuda.is_available():
+    if not _cuda_usable():
         return None
     p = torch.cuda.get_device_properties(0)
     return {"name": p.name, "vram_gb": round(p.total_memory / 1e9, 1),
@@ -130,6 +137,35 @@ def broadcast_params(model, src: int = 0):
         p.data.copy_(cpu.to(p.data.device))
 
 
+@torch.no_grad()
+def check_finite_grads(model, step: int) -> None:
+    """Refuse a non-finite REDUCED gradient before the optimizer applies it.
+
+    After the all-reduce every rank holds the identical summed gradient, so every rank reaches
+    the same verdict with no extra communication and they all stop at the same step. Before
+    2026-09-30 one node's NaN was summed into every replica and opt.step() wrote it into the
+    weights (and Adam's moments) everywhere -- the replicas still agreed, so replica_diff
+    could not see it."""
+    for name, p in model.named_parameters():
+        if p.grad is not None and not bool(torch.isfinite(p.grad).all()):
+            bad = int((~torch.isfinite(p.grad)).sum())
+            raise FloatingPointError(
+                "step %d: reduced gradient of %s has %d non-finite value(s); refusing to apply "
+                "it (weights and optimizer state unchanged)" % (step, name, bad))
+
+
+@torch.no_grad()
+def cluster_mean_loss(loss, weight: float) -> float:
+    """The full-batch mean loss: sum_i w_i * loss_i with the SAME capacity weights
+    (w_i = n_i / sum n) the gradient all-reduce uses. A collective -- call it on every rank.
+
+    It used to be sum(loss_i) / world: an equal-weight mean of UNEQUAL batches, so the logged
+    cluster loss leaned toward the small (slow) nodes whenever capacities differed (2026-09-30)."""
+    lt = loss.detach().to("cpu").clone().double() * weight
+    dist.all_reduce(lt, op=dist.ReduceOp.SUM)
+    return lt.item()
+
+
 @torch.no_grad()
 def capacity_weighted_allreduce_grads(model, weight: float):
     """Σ_i w_i g_i with w_i = n_i/Σn_j == the true full-batch mean gradient.
@@ -185,14 +221,13 @@ class DaisyCluster:
             loss = task.loss(model, X, y)
             loss.backward()
             capacity_weighted_allreduce_grads(model, w)
+            check_finite_grads(model, s)
             opt.step()
             if step_delay:
                 time.sleep(step_delay)
             if s % max(1, steps // 20) == 0 or s == steps - 1:
-                lt = loss.detach().to("cpu").clone()
-                dist.all_reduce(lt, op=dist.ReduceOp.SUM)
+                cl = cluster_mean_loss(loss, w)             # collective: every rank calls it
                 if self.is_master():
-                    cl = lt.item() / self.world
                     print(f"  step {s:5d}  cluster-avg loss {cl:.6f}", flush=True)
                     if status_path:
                         self._write_status(status_path, step=s, total_steps=steps,

test_cluster_reductions.py

@@ -0,0 +1,111 @@
+"""Cluster reductions (2026-09-30). Run: python test_cluster_reductions.py  (CPU only, ~10 s).
+
+1. cluster_mean_loss is the FULL-BATCH mean loss: sum_i w_i * loss_i with the capacity weights
+   the gradient all-reduce uses. It was sum(loss_i) / world -- checked on a real 2-rank gloo
+   group with unequal batches (3 and 13 samples), against the loss of the concatenated batch.
+2. check_finite_grads refuses a non-finite reduced gradient BEFORE opt.step(), and leaves the
+   weights untouched; a finite gradient passes.
+"""
+from __future__ import annotations
+
+import os
+import sys
+import tempfile
+
+import torch
+import torch.distributed as dist
+import torch.multiprocessing as mp
+
+from daisychain.cluster import (capacity_weighted_allreduce_grads, check_finite_grads,
+                                cluster_mean_loss)
+
+FAILURES = []
+
+
+def ck(name, cond, detail=""):
+    print("  %s  %s%s" % ("ok  " if cond else "FAIL", name, ("  " + str(detail)) if detail else ""))
+    if not cond:
+        FAILURES.append(name)
+
+
+SIZES = (3, 13)
+
+
+def data(rank):
+    g = torch.Generator().manual_seed(100 + rank)
+    X = torch.randn(SIZES[rank], 4, generator=g)
+    y = torch.randn(SIZES[rank], 1, generator=g)
+    return X, y
+
+
+def model0():
+    torch.manual_seed(7)
+    return torch.nn.Linear(4, 1)
+
+
+def worker(rank, world, init_file, out_file):
+    dist.init_process_group("gloo", init_method="file://" + init_file, rank=rank, world_size=world)
+    w = SIZES[rank] / sum(SIZES)
+    m = model0()
+    X, y = data(rank)
+    loss = torch.nn.functional.mse_loss(m(X), y)
+    loss.backward()
+    cl = cluster_mean_loss(loss, w)
+    old = loss.detach().clone()
+    dist.all_reduce(old, op=dist.ReduceOp.SUM)                 # the previous formula
+    capacity_weighted_allreduce_grads(m, w)
+    if rank == 0:
+        torch.save({"cl": cl, "old": old.item() / world,
+                    "grad": [p.grad.clone() for p in m.parameters()]}, out_file)
+    dist.barrier()
+    dist.destroy_process_group()
+
+
+def main():
+    print("cluster_mean_loss -- 2 ranks, batches of %d and %d samples" % SIZES)
+    with tempfile.TemporaryDirectory() as td:
+        init_file, out_file = os.path.join(td, "init"), os.path.join(td, "out.pt")
+        mp.spawn(worker, args=(2, init_file, out_file), nprocs=2, join=True)
+        got = torch.load(out_file)
+    m = model0()
+    Xa = torch.cat([data(0)[0], data(1)[0]])
+    ya = torch.cat([data(0)[1], data(1)[1]])
+    full = torch.nn.functional.mse_loss(m(Xa), ya)
+    full.backward()
+    ck("equals the loss of the concatenated batch (the full-batch mean)",
+       abs(got["cl"] - full.item()) < 1e-6, (got["cl"], full.item()))
+    ck("the old sum/world formula did NOT (it is the equal-weight mean of unequal batches)",
+       abs(got["old"] - full.item()) > 1e-4, (got["old"], full.item()))
+    ck("and the gradient all-reduce agrees with the same weighting (consistency check)",
+       all(torch.allclose(g, p.grad, atol=1e-6) for g, p in zip(got["grad"], m.parameters())))
+
+    print("check_finite_grads -- refuse a non-finite reduced gradient before opt.step()")
+    lin = model0()
+    before = [p.detach().clone() for p in lin.parameters()]
+    lin(torch.ones(2, 4)).sum().backward()
+    try:
+        check_finite_grads(lin, 0)
+        ck("a finite gradient passes", True)
+    except FloatingPointError as e:
+        ck("a finite gradient passes", False, e)
+    for bad in (float("nan"), float("inf")):
+        lin.weight.grad[0, 1] = bad
+        try:
+            check_finite_grads(lin, 5)
+            ck("%r in a reduced gradient raises" % bad, False, "applied silently")
+        except FloatingPointError as e:
+            ck("%r in a reduced gradient raises, naming the step and parameter" % bad,
+               "step 5" in str(e) and "weight" in str(e), str(e)[:80])
+    ck("the weights were never touched",
+       all(torch.equal(a, b) for a, b in zip(before, lin.parameters())))
+
+    print()
+    if FAILURES:
+        print("FAILED: %d" % len(FAILURES))
+        return 1
+    print("all checks passed")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

web/TEST_RESULTS.md

@@ -1,6 +1,6 @@
 # Test results — updated 2026-07-17
 
-All suites run with `npm test` (chains all twelve). Every suite exits 0.
+All suites run with `npm test` (chains all thirteen). Every suite exits 0.
 Hardware for GPU numbers: NVIDIA via WebGPU, DP4A int8 dot path, exact-gated
 against the verified units at init.
 
@@ -16,7 +16,9 @@ against the verified units at init.
 | `test_optimizer.js` | DaisyAdam beats SGD through the units, deterministic replicas | PASS — 1.59 vs 1.95, replica diff 0.000e+0 |
 | `test_transformer.js` | the transformer LM trains through the units end to end | PASS — loss 4.75 → 1.26 (baseline 4.56), replica diff 0.000e+0 |
 | `test_unit_backward.js` | int8 STE gradients do not damage convergence | PASS — units/float loss ratio 1.007 |
+| `test_b2b.js` | the chained GPU MLP's CPU mirror: gemm1 byte-identical to the un-chained GEMM, deterministic, +0.5 quantize FMA-contraction-immune | PASS — chained transformer converges 4.7384 → 2.0512 |
 | `test_nonfinite.js` | every quantizer refuses NaN/Inf instead of zeroing the layer | PASS — 13/13; the old quantizers fail 12 (added 2026-09-27) |
+| `test_reductions.js` | the GPU row-max guard is complete, `averageGrads` refuses wrong-length / non-finite gradients, int32 accumulators refuse K > 131,071 | PASS — 17/17; the old code fails 11 (added 2026-09-30) |
 
 ## The IEEE-754 oracle (`test_ieee.js`)
 

web/package.json

@@ -6,7 +6,7 @@
   "author": "Dean Byrne (Quazim0t0) / DaisyChainAI",
   "scripts": {
     "start": "node server.js",
-    "test": "node test_core.js && node test_verified.js && node test_ieee.js && node test_gates.js && node test_metamorphic.js && node test_corpus.js && node test_selfcorpus.js && node test_b2b.js && node test_optimizer.js && node test_transformer.js && node test_unit_backward.js && node test_nonfinite.js"
+    "test": "node test_core.js && node test_verified.js && node test_ieee.js && node test_gates.js && node test_metamorphic.js && node test_corpus.js && node test_selfcorpus.js && node test_b2b.js && node test_optimizer.js && node test_transformer.js && node test_unit_backward.js && node test_nonfinite.js && node test_reductions.js"
   },
   "dependencies": {
     "hyparquet": "^1.26.2",

web/public/app.js

@@ -914,7 +914,15 @@ async function train(cfg, resume) {
       else if (hs && hs.has(id)) { lossSum += hs.get(id).loss; lossN++; contrib.set(id, (contrib.get(id) || 0) + 1); }
     }
     const clusterLoss = lossSum / Math.max(1, lossN);
-    const avg = TrainCore.averageGrads(all);
+    let avg;
+    try { avg = TrainCore.averageGrads(all); }
+    catch (e) {
+      // every device averages the same roster, so every device stops HERE, at the same step,
+      // with weights and Adam state untouched (still a valid checkpoint to resume from)
+      if (e.name !== "GradientError") throw e;
+      halted = `step ${s + 1}: ${e.message} -- refusing to apply it (weights and Adam state unchanged)`;
+      break;
+    }
     const upd = opt.step(avg);                      // DaisyAdam on the cluster-avg grad
     Transformer.applyUpdate(model, upd);            // W -= upd (lr folded into upd)
     // retain a few applied steps so stalled peers can repair (evict beyond that)

web/public/traincore.js

@@ -54,13 +54,29 @@
   // ORDER MATTERS: float addition is not associative, so every replica MUST
   // pass the gradients in the same order (the leader's roster order) or their
   // averages differ in the last bits and the weights fork. Never self-first.
+  //
+  // REFUSES (throws GradientError) rather than averaging when (a) the gradients differ in
+  // length -- typed arrays silently ignore out-of-range writes, so a truncated gradient used
+  // to be averaged over a prefix with no error -- or (b) any value is non-finite: one NaN
+  // would enter DaisyAdam's m/v on EVERY peer (and every later checkpoint) identically, so no
+  // fork check could see it. Refusing BEFORE the optimizer keeps weights and moments clean.
   function averageGrads(grads) {
-    const out = new Float32Array(grads[0].length);
+    const len = grads[0].length;
+    for (let r = 0; r < grads.length; r++) {
+      const g = grads[r];
+      if (!g || g.length !== len) throw gradError("gradient " + r + " has " +
+        (g ? g.length : "no") + " values, gradient 0 has " + len);
+      for (let i = 0; i < len; i++) if (!Number.isFinite(g[i]))
+        throw gradError("gradient " + r + " is non-finite at index " + i + " (" + g[i] + ")");
+    }
+    const out = new Float32Array(len);
     for (const g of grads) for (let i = 0; i < g.length; i++) out[i] += g[i];
     for (let i = 0; i < out.length; i++) out[i] /= grads.length;
     return out;
   }
 
+  function gradError(msg) { const e = new Error(msg); e.name = "GradientError"; return e; }
+
   // DaisyAdam — Adam with bias correction, applied to the cluster-averaged
   // gradient. State is a pure function of the gradient sequence, so every peer
   // that averages the same gradients keeps bit-identical moments: no optimizer

web/public/verified_core.js

@@ -35,8 +35,22 @@
     return { q, scale };
   }
 
+  // int32 accumulator headroom: |a*b| <= 128*128, so K*16384 must fit in int32. Same bound as
+  // the Python kernel (kernel.MAX_INT32_K). The Int32Array accumulators here WRAP exactly like
+  // WGSL i32, so CPU and GPU peers would agree -- on a silently wrong sum. Refuse instead.
+  const MAX_INT32_K = Math.floor((2 ** 31 - 1) / (128 * 128));      // 131071
+  function accBound(k, where) {
+    if (k > MAX_INT32_K) {
+      const e = new Error(where + ": contraction length " + k + " exceeds the int32 accumulator " +
+                          "headroom " + MAX_INT32_K + " (the sum would wrap)");
+      e.name = "AccumulatorBoundError";
+      throw e;
+    }
+  }
+
   // int8 matmul via the verified multiply LUT: acc(m×n) = sum_k mulLUT[Xq,Wq]
   function lutMatmulJS(Xq, Wq, m, k, n, L) {
+    accBound(k, "lutMatmulJS");
     const C = new Int32Array(m * n), mul = L.mul;
     for (let i = 0; i < m; i++) {
       for (let p = 0; p < k; p++) {
@@ -151,6 +165,12 @@
     return mx;
   }
   function scalesFromAbsMax(mx) {                // f64 divisions: exactly rounded, device-identical
+    // The GPU chain gets mx from an atomicMax over bitcast(abs(h)) and never passes through
+    // rowAbsMax, so this is where ITS non-finite guard lives: NaN bits (0x7FC00000) and Inf
+    // (0x7F800000) both order above every finite |h| as u32, so the row max is non-finite iff
+    // some element is. Before 2026-09-30 a GPU peer turned that into inv = 0/NaN and quantized
+    // floor(NaN) on the device (implementation-defined) where a CPU peer threw.
+    assertFinite(mx, "scalesFromAbsMax");
     const scale = new Float32Array(mx.length), inv = new Float32Array(mx.length);
     for (let i = 0; i < mx.length; i++) {
       scale[i] = Math.max(mx[i] / 127, 1e-8);
@@ -221,6 +241,7 @@
   function bgemmJS(Xq, Wq, rs, cs, d, L) {
     const { m, k, n } = d, batch = d.batch || 1, relu = !!d.relu, mul = L.mul;
     const raw = !!d.acc;
+    accBound(k, "bgemmJS");
     const out = raw ? new Int32Array(batch * m * n) : new Float32Array(batch * m * n);
     const acc = new Int32Array(n);
     for (let bz = 0; bz < batch; bz++) {
@@ -510,7 +531,7 @@
     for (let j = 0; j < W2.length; j++) W2[j] -= lr * gAvg[W1.length + j];
   }
 
-  const api = { assertFinite, quantize, quantize2, quantizeRows, quantizeCols, quantizeHeadCols, lutMatmulJS, lutMatmul3JS, lutMatmul3,
+  const api = { assertFinite, accBound, MAX_INT32_K, quantize, quantize2, quantizeRows, quantizeCols, quantizeHeadCols, lutMatmulJS, lutMatmul3JS, lutMatmul3,
                 bgemmJS, vgemmBlock, auditTile, epi, attScoresJS, attCtxJS, linearFwd, forward, backward, splitApply,
                 rowAbsMax, scalesFromAbsMax, quantizeRowsInv, vmlpBlock, bitDiff,
                 auditAttScores, auditAttCtx, fgemmMirror };

web/public/webgpu.js

@@ -611,6 +611,7 @@
   }
 
   async function gpuMatmulLUT(device, pipeline, lutBuf, Xq, Wq, m, k, n) {
+    root.Verified.accBound(k, "gpuMatmulLUT");                  // WGSL `var s : i32` wraps past this
     const X32 = Int32Array.from(Xq), W32 = Int32Array.from(Wq);   // byte -> i32
     const bufX = up(device, X32, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufW = up(device, W32, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
@@ -626,6 +627,7 @@
   // attention kernels: int8 (widened i32) in, f32 out, strided head indexing
   async function gpuAttScores(device, pipeline, qq, kq, qs, ks, d) {
     const { B, T, heads } = d;
+    root.Verified.accBound(d.hd, "gpuAttScores");
     const bufQ = up(device, Int32Array.from(qq), GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufK = up(device, Int32Array.from(kq), GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufQs = up(device, qs, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
@@ -640,6 +642,7 @@
   }
   async function gpuAttCtx(device, pipeline, aq, vq, as, vs, d) {
     const { B, T, heads, hd } = d;
+    root.Verified.accBound(T, "gpuAttCtx");
     const bufA = up(device, Int32Array.from(aq), GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufV = up(device, Int32Array.from(vq), GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufAs = up(device, as, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
@@ -759,6 +762,7 @@
 
   async function gpuBgemmLUT(device, pipeline, lutBuf, Xq, Wq, rs, cs, d) {
     const { m, k, n } = d, batch = d.batch || 1, flags = d.relu ? 1 : 0;
+    root.Verified.accBound(k, "gpuBgemmLUT");
     const bufX = up(device, Int32Array.from(Xq), GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufW = up(device, Int32Array.from(Wq), GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
     const bufR = up(device, rs, GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST);
@@ -775,6 +779,7 @@
 
   async function gpuBgemmDP4(device, pipeline, Xq, Wq, rs, cs, d) {
     const { m, k, n } = d, batch = d.batch || 1, flags = d.relu ? 1 : 0;
+    root.Verified.accBound(k, "gpuBgemmDP4");
     const kw = Math.ceil(k / 4);
     const Xp = packRows(Xq, batch * m, k, kw);            // rows are (batch·m)
     const Wp = new Uint32Array(batch * n * kw);           // per-batch Wᵀ, packed
@@ -801,6 +806,7 @@
   // left operand is produced and consumed entirely on the GPU.
   async function gpuMlpChain(device, env, xq, w1q, w2q, xs, w1s, w2s, d) {
     const { m, k, h, n } = d, dp4 = !!env.dp4;
+    root.Verified.accBound(k, "gpuMlpChain gemm1"); root.Verified.accBound(h, "gpuMlpChain gemm2");
     const SU = GPUBufferUsage.STORAGE | GPUBufferUsage.COPY_DST;
     let bufX, bufW1, bufW2, kw1, hw;
     if (dp4) {

web/test_reductions.js

@@ -0,0 +1,85 @@
+// Reductions and accumulators that used to accept bad input silently (2026-09-30). Each check
+// fails on the previous code:
+//  1. the GPU MLP chain's row max comes from an atomicMax over bitcast(abs(h)) and never passed a
+//     non-finite guard -- scalesFromAbsMax now refuses, and the bit-order argument that makes
+//     that ONE check complete is checked here too;
+//  2. averageGrads averaged a truncated gradient over a prefix, and fed NaN/Inf into DaisyAdam
+//     on every peer at once -- it now refuses both;
+//  3. the Int32Array / WGSL i32 accumulators WRAP past K = 131071 (CPU and GPU agree on the wrong
+//     sum) -- the GEMMs now refuse, the same bound as the Python kernel's MAX_INT32_K.
+"use strict";
+const V = require("./public/verified_core.js");
+const TC = require("./public/traincore.js");
+
+let fails = 0;
+function ck(name, cond) { console.log((cond ? "  ok    " : "  FAIL  ") + name); if (!cond) fails++; }
+function throwsName(fn, name) { try { fn(); return false; } catch (e) { return e.name === name; } }
+
+// ---- 1. the GPU row-max readback ------------------------------------------------------------
+for (const bad of [NaN, Infinity, -Infinity]) {
+  ck("scalesFromAbsMax refuses a row max of " + bad,
+     throwsName(() => V.scalesFromAbsMax(Float32Array.from([1.5, bad, 0.25])), "NonFiniteError"));
+}
+const fin = V.scalesFromAbsMax(Float32Array.from([127, 0, 1e-12]));
+ck("finite row maxes give the same scales as before (1, 1e-8 floor)",
+   fin.scale[0] === 1 && Math.abs(fin.scale[1] - 1e-8) < 1e-15 && fin.inv[0] === 1);
+
+// What the WGSL kernel computes: atomicMax over the u32 bits of |h|. Emulated exactly with
+// typed-array views. The guard above is complete iff "max bits non-finite <=> some h non-finite".
+function gpuRowMax(h) {
+  const f = new Float32Array(1), u = new Uint32Array(f.buffer);
+  let mx = 0;
+  for (const v of h) { f[0] = Math.abs(v); if (u[0] > mx) mx = u[0]; }
+  u[0] = mx; return f[0];
+}
+let rng = 12345;
+const rnd = () => ((rng = (rng * 1103515245 + 12345) >>> 0) / 4294967296);
+let complete = true, exact = true;
+for (let trial = 0; trial < 2000; trial++) {
+  const h = new Float32Array(16).map(() => (rnd() - 0.5) * Math.pow(2, Math.floor(rnd() * 200) - 100));
+  const want = h.reduce((a, v) => Math.max(a, Math.abs(v)), 0);
+  if (gpuRowMax(h) !== Math.fround(want)) exact = false;
+  if (trial % 2) h[Math.floor(rnd() * 16)] = [NaN, Infinity, -Infinity, -NaN][trial % 4];
+  const nonfinite = h.some(v => !Number.isFinite(v));
+  if (Number.isFinite(gpuRowMax(h)) === nonfinite) complete = false;
+}
+ck("bit-order row max equals the true |max| on finite rows (2000 rows, wide exponents)", exact);
+ck("bit-order row max is non-finite exactly when some element is (NaN/Inf/-Inf injected)", complete);
+
+// ---- 2. averageGrads ------------------------------------------------------------------------
+const g1 = Float32Array.from([1, 2, 3, 4]), g2 = Float32Array.from([3, 2, 1, 0]);
+const avg = TC.averageGrads([g1, g2]);
+ck("equal-length finite gradients average as before", Array.from(avg).join() === "2,2,2,2");
+ck("a truncated gradient is refused (was averaged over a prefix)",
+   throwsName(() => TC.averageGrads([g1, Float32Array.from([1, 2])]), "GradientError"));
+ck("a longer gradient is refused (its tail was silently dropped)",
+   throwsName(() => TC.averageGrads([g1, Float32Array.from([1, 2, 3, 4, 5])]), "GradientError"));
+for (const bad of [NaN, Infinity]) {
+  ck("a gradient containing " + bad + " is refused before DaisyAdam sees it",
+     throwsName(() => TC.averageGrads([g1, Float32Array.from([0, bad, 0, 0])]), "GradientError"));
+}
+// why refusing matters: one NaN step poisons Adam's moments for good
+const adam = TC.makeAdam(1, { lr: 0.1 });
+adam.step(Float32Array.from([NaN]));
+ck("(context) one NaN step leaves Adam's moments NaN on a finite gradient afterwards",
+   Number.isNaN(adam.step(Float32Array.from([1]))[0]));
+
+// ---- 3. int32 accumulator headroom ----------------------------------------------------------
+ck("MAX_INT32_K matches the Python kernel: floor((2^31-1) / 16384) = 131071", V.MAX_INT32_K === 131071);
+const L = { mul: new Int32Array(65536) };
+for (let a = 0; a < 256; a++) for (let b = 0; b < 256; b++) L.mul[a * 256 + b] = (a << 24 >> 24) * (b << 24 >> 24);
+const K = V.MAX_INT32_K + 1;
+const worst = new Int8Array(K).fill(-128);
+ck("lutMatmulJS refuses K = 131072 (all -128*-128: the sum 2^31 wraps to -2^31)",
+   throwsName(() => V.lutMatmulJS(worst, worst, 1, K, 1, L), "AccumulatorBoundError"));
+const acc = new Int32Array(1); acc[0] = 2 ** 31;
+ck("(context) Int32Array really wraps 2^31 to -2^31 -- the silent failure being refused", acc[0] === -(2 ** 31));
+ck("bgemmJS refuses the same K",
+   throwsName(() => V.bgemmJS(worst, worst, new Float32Array(1), new Float32Array(1),
+                              { m: 1, k: K, n: 1 }, L), "AccumulatorBoundError"));
+const okK = worst.subarray(0, V.MAX_INT32_K);
+ck("K = 131071 still runs and is exact (131071 * 16384)",
+   V.lutMatmulJS(okK, okK, 1, V.MAX_INT32_K, 1, L)[0] === 131071 * 16384);
+
+if (fails) { console.log("REDUCTIONS TEST FAILED: " + fails); process.exit(1); }
+console.log("REDUCTIONS TEST PASSED — non-finite row maxes, bad gradients and int32 overflow are refused.");