One arithmetic on every device: mixed CPU+GPU optimizer, replica hash check, FTZ pin, subnormal gate, live f32 audit, probe/fragment fixes (2026-10-14)

README.md

@@ -1,322 +1,588 @@
----
-license: mit
-tags:
-  - distributed-training
-  - old-hardware
-  - int8
-  - webgpu
-  - webrtc
----
-
-# DaisyChain-Train - old hardware training pipeline
-
-Part of DaisyChain on Hugging Face: https://huggingface.co/DaisyChainAI
-
-Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train
-
-I built this so spare / old machines can train a shared model. The training runs through emulated GPU logic: verified INT8 units (GUDA-style) that stand in for a GPU's math. Machines without a modern GPU still do the work. Chain several together and they train one model as a cluster.
-
-Before you rely on it, read what it cannot do: [Limitations](docs/LIMITS.md).
-
-Use the hardware you already have. Each machine runs the emulated GPU logic (verified INT8 units: multiply / requantize / ReLU). DaisyChain pools the machines data-parallel: device selection, capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard. Two ways to run: **Docker** or **Python**.
-
-## Read this first
-
-DaisyChain-Train is for **small models on spare hardware**. It **pools compute, not memory** (the model must fit on one machine). Scaling is **sublinear**. It is **not** a substitute for a real GPU on large models. Full envelope: **[docs/LIMITS.md](docs/LIMITS.md)**. Read that before you depend on it.
-
-## Feature list
-
-### Python cluster trainer (`daisychain/`)
-
-- **Data-parallel training across mixed machines.** Each node trains its own shard. Gradients combine into the exact full-batch gradient. Replicas stay bit-identical.
-- **Capacity-weighted sharding.** Faster machines automatically take a bigger share of the batch.
-- **Emulated GPU compute (verified INT8 units).** `VerifiedLinear` layers run every forward multiply / requantize / ReLU through the bundled trained units. Rank 0 prints cluster-wide unit-invocation counts.
-- **Bring your own model.** Any `Task` (`build_model` / `sample` / `loss`) via `DAISY_TASK`. Template in `examples/my_task_template.py`.
-- **Plain-float alternative task.** Same cluster and pooling with ordinary float math.
-- **Live dashboard** (`daisychain-dashboard`). Readiness banner, P2P connectivity scan, pooled cores/RAM, per-node capacity plan, live loss.
-- **SpikeWhale control panel** (`spikewhale_panel`, `localhost:8899`). Sliders for model size / training settings, any HF dataset you can access (default streamed FineWeb-Edu), start/stop/re-adjust, live loss.
-- **Docker demo cluster.** 3 nodes + dashboard in one command.
-- **Windows helper** (`scripts\setup.bat`) and Tailscale mesh guide.
-
-### DaisyChain-Web (`web/`) - browser P2P training
-
-- **Zero-install nodes.** Opening the page IS joining. Devices on one network auto-group (Snapdrop-style, by public IP).
-- **Private cross-network rooms.** `?room=CODE` with **host approval** for every join.
-- **Full WebRTC mesh.** Gradients travel peer-to-peer. The server only signals and serves static files. It never sees weights or gradients.
-- **Leader-follower runs.** Whoever presses Start sets width / sequence / batch-per-device / steps / learning rate for the whole group. Config is broadcast on the wire.
-- **Mid-run join.** Late devices are synced in (weights + step) and contribute from the next step.
-- **Bit-identical replicas.** Same seeded init, strict roster-order gradient averaging, deterministic Adam with identical state on every peer. Verified live by per-step weight hashes.
-- **Sync guard.** Any weight-hash mismatch stops the run instead of training past a fork. The step roster forbids silent partial averages.
-- **Gradient repair.** A follower missing a roster gradient re-requests it from the leader (8 steps retained), bit-exact, and the run continues.
-- **Cross-device kernel probe.** Every step, every device re-hashes a fixed seeded int8 GEMM through its live kernel. Catches broken arithmetic that weight hashes cannot see.
-- **Hardcoded FineWeb-Edu streaming.** The server reads random slices of the 10BT parquet shards straight off the HF CDN via HTTP range requests (pure-JS `hyparquet`). Built-in corpus fallback offline.
-- **Checkpoints.** Download `.pt`. Upload -> **broadcast to the whole group**. Validated (magic, dims, tokenizer vocab) before accepting.
-- **Inference kit.** One self-contained HTML file with the trained weights baked in. Generations offline, anywhere.
-- **In-page generation.** Prompt box on the trained model.
-- **Old-hardware tier.** No WebGPU? The identical units run on CPU (same bits, so CPU and GPU devices co-train in one group). There is no plain-float path.
-- **Large-message fragmentation.** Multi-MB gradients/checkpoints chunked at 48 KB over the data channels.
-
-### Verified compute and kernels (web)
-
-- **Verified INT8 units everywhere.** Block-scaled int8 GEMM: exact LUT products, exact int32 accumulation, bit-exact f32 epilogue with a pinned rounding schedule. Scales derived in JS f64 (division never runs on GPU).
-- **Backends, best-first.** DP4A hardware int8 dot -> LUT compute shader -> CPU mirror. Every kernel is **exact-gated at init** (bit-level compares) and demoted to the mirror on any mismatch.
-- **Continuous random-cell audit** at live training shapes.
-- **Fused attention kernels.** Gather/scatter head-strided q·kᵀ and a·v straight from BT×C layout (CUTLASS ex. 36/52 style).
-- **QKV dual-GEMM fusion.** Shared left operand quantized once, one batch-3 dispatch (ex. 45). Bit-identical.
-- **B2B MLP chain.** Both MLP GEMMs back-to-back on GPU with fused per-row absmax reduction and on-device quantize (ex. 13 + 23). WGSL-exact respec with a fround-stepped JS mirror. FMA-contraction-immune by construction.
-- **Dispatch-optimized backward.** Overlapped independent GEMMs, batch-3 sibling fusions (ex. 05/24). Bit-identical gradients. Optional int8 STE backward path (dormant, 1.21x vs float).
-
-### Verification stack (web)
-
-- **Exact init gates** on every kernel, every device, every boot. Including gates that "gate the gate" with discriminating boundary inputs.
-- **IEEE-754 binary32 oracle** in exact BigInt arithmetic. Proves the JS epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs).
-- **Metamorphic property suite.** Reference-free relations + definitional absolutes. **4/4** on an externally-authored bug corpus, matching the differential gate.
-- **RDNA2 ISA audit hardenings.** Bit-level (-0-aware) gate comparisons. Proof that FMA contraction cannot change the quantize.
-- **Twelve-suite test chain** (`cd web && npm test`): convergence, replicas, oracle, gates, properties, external corpus, **self-corpus** (the instruments scored against my own bugs), B2B, optimizer, transformer LM, int8 backward, non-finite quantize refusal. Results in [web/TEST_RESULTS.md](web/TEST_RESULTS.md).
-- **Dirty-buffer gate.** The pool is poisoned before a re-sweep so state bugs (a kernel assuming zeroed memory) are caught deterministically rather than by ordering luck.
-
-### Documentation
-
-- Python: [QUICKSTART](docs/QUICKSTART.md), [LIMITS](docs/LIMITS.md), [CUSTOM_TASK](docs/CUSTOM_TASK.md), [TAILSCALE](docs/TAILSCALE.md).
-- Web: [Getting started](web/docs/GETTING_STARTED.md), [Architecture](web/docs/ARCHITECTURE.md), [Verification](web/docs/VERIFICATION.md), [Troubleshooting](web/docs/TROUBLESHOOTING.md).
-
----
-
-## Quick start
-
-### Docker (most reliable - one command)
-
-```bash
-docker compose -f docker/docker-compose.yml up --build
-# open http://localhost:8080
-```
-
-Brings up a 3-node demo cluster + dashboard on one machine.
-
-### Python (real machines)
-
-On every machine (`pip install -e .`):
-
-```bash
-export MASTER_ADDR=100.101.102.10   # coordinator IP (Tailscale 100.x recommended)
-export MASTER_PORT=29560
-export WORLD_SIZE=3
-export RANK=0                        # 1, 2, ... on the others
-export GLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC
-daisychain-train
-```
-
-### Windows helper
-
-```bat
-scripts\setup.bat
-```
-
-An interactive menu: Docker, Python node, or just install deps.
-
-Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**.
-
-### SpikeWhale control panel (sliders -> real training)
-
-```bash
-python -m daisychain.spikewhale_panel
-# open http://localhost:8899
-```
-
-A web control panel: pick model size / training settings with sliders, choose any Hugging Face dataset you have access to (default: streamed FineWeb-Edu), hit Start, and watch the live loss. Stop and re-adjust any time with **<- Back to settings**. Launches the real DaisyChain training underneath.
-
-### DaisyChain-Web (train by opening a browser tab)
-
-```bash
-cd web && npm install && node server.js
-# open http://localhost:8787 on every device
-```
-
-Zero-install browser training. Devices on the same network auto-group (Snapdrop-style) and train a shared model **peer-to-peer over WebRTC**, computing through the same verified INT8 units (WebGPU, with the identical units on CPU for machines without it - there is no plain-float path). Private cross-network rooms via `?room=CODE` with **host approval**. The room creator accepts each device before it can join. Includes gradient averaging with a deterministic Adam optimizer (identical state on every peer, nothing extra over the wire), checkpoint **download** (`.pt`) and **upload -> broadcast** so one device can restore the whole group after a failure.
-
-**Live demo:** https://huggingface.co/spaces/Quazim0t0/DaisyChain-Web
-
----
-
-## How it works
-
-Each machine runs the **same** command. They form a cluster and train one shared model. Two things happen:
-
-1. **The compute runs through the emulated GPU logic.** By default the model is built from `VerifiedLinear` layers, so every forward multiply / requantize / ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`) - the emulated GPU math. Rank 0 prints **cluster-wide unit-invocation counts** so you can see the emulated logic doing the work.
-2. **The machines are pooled data-parallel.** Each node trains on its own shard. Gradients are capacity-weighted and combined into the exact full-batch gradient, so replicas stay **bit-identical**. Faster machines automatically take a bigger share.
-
-```
-  old machine A -+
-  old machine B -+- each runs the emulated GPU logic on its shard  ->  one model
-  old machine C -+      (gradients combined across the cluster)
-```
-
-## Bring your own model
-
-DaisyChain-Train trains any **Task** (`build_model` / `sample` / `loss`). Copy `examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use `VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**.
-
-## Plain-float alternative
-
-To skip the emulated units and train with normal float math on each machine, set `DAISY_TASK=daisychain.example_task:ExampleTask`. Same cluster, same pooling. The model math just runs as ordinary float instead of through the verified units.
-
-## The dashboard
-
-`daisychain-dashboard` (or the Docker service) serves a Tailwind page at `:8080`: readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan (per-node device, weight, batch), and live training loss.
-
-## Networking
-
-Use **Tailscale** for a P2P mesh so machines on different networks get stable IPs on one interface. **[docs/TAILSCALE.md](docs/TAILSCALE.md)**.
-
----
-
-## Layout
-
-```
-daisychain/cluster.py        capacity-weighted CPU/GPU data-parallel trainer
-daisychain/train.py          entry point (daisychain-train)
-daisychain/verified/         bundled trained N/N units + VerifiedLinear (train through them)
-daisychain/verified_task.py  default task: forward runs on the verified units
-daisychain/example_task.py   plain-float alternative task
-daisychain/task.py           the Task interface + loader
-daisychain/dashboard/        agent + P2P scanner + Tailwind server
-docker/                      Dockerfile, dashboard image, compose (demo cluster)
-scripts/setup.bat / setup.sh interactive setup helpers
-config/                      nodes + cluster env examples
-examples/my_task_template.py starting point for your own model
-test_verified_units.py       full-domain checks of every shipped verified path (25)
-docs/                        QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
-daisychain/spikewhale_task.py   trains the real SpikeWhale on streamed HF datasets
-daisychain/spikewhale_panel.py  slider control panel (localhost:8899)
-web/                         DaisyChain-Web: P2P browser training (WebRTC + WebGPU)
-export_luts_web.py           regenerates web/public LUTs from the trained units
-```
-
-## Recent updates (30 September 2026) - reductions refuse what they used to accept
-
-Six fixes, each found by applying a lesson from differential-testing an RDNA2 shader translator: *a second implementation nobody checks is not verified*, *inactive or absent participants must not leak into a reduction*, *a max over NaN is undefined*, and *know your accumulator width*. Every one is covered by a check that fails on the old code. `python test_verified_units.py` + `python test_cluster_reductions.py` and `cd web && npm test` (thirteen suites) reproduce all of it.
-
-| # | where | what was wrong | now |
-| --- | --- | --- | --- |
-| 1 | browser, WebGPU MLP chain | the row max comes from an `atomicMax` over `bitcast(abs(h))` and never passed the non-finite guard the CPU path has; a NaN/Inf `h1` became `inv = 0`/NaN and the GPU quantized `floor(NaN)` (implementation-defined) where a CPU peer threw | `scalesFromAbsMax` refuses a non-finite max. NaN and Inf bits order above every finite value as u32, so the row max is non-finite **iff** some element is - one check covers the whole row (checked on 2000 random rows) |
-| 2 | browser, `averageGrads` | a gradient of a different length was silently averaged over a prefix (typed arrays ignore out-of-range writes); a NaN/Inf gradient went straight into DaisyAdam, poisoning `m`/`v` on **every** peer identically, so no fork check could see it | refuses both with `GradientError`; the training loop halts at that step with weights and Adam state untouched (still a valid checkpoint) |
-| 3 | browser, int8 GEMMs (JS and WGSL) | the `Int32Array` and WGSL `i32` accumulators wrap past K = 131,071 - CPU and GPU agree, on a wrong sum | refused with `AccumulatorBoundError` at every GEMM entry point; the same bound as the Python kernel's `MAX_INT32_K` |
-| 4 | Python, logged cluster loss | `sum(loss_i) / world` - an equal-weight mean of **unequal**, capacity-weighted batches | `cluster_mean_loss` = Σ wᵢ·lossᵢ with the gradient's own weights. Measured on 3- and 13-sample nodes: old **1.418**, true full-batch loss **0.959**, new 0.959 |
-| 5 | Python, gradient all-reduce | one node's NaN was summed into every replica and applied | `check_finite_grads` refuses the reduced gradient before `opt.step()`; every rank holds the same sum, so every rank stops at the same step with no extra communication |
-| 6 | Python, `pick_device` | with `CUDA_VISIBLE_DEVICES` set but **empty** (a common "CPU only"), torch reports `is_available() = True` and `device_count() = 0`, and the node crashed at startup | a device must actually exist; falls back to CPU |
-
-Unchanged for valid inputs: finite gradients, finite activations, and K <= 131,071 produce bit-identical results to before, so updated and older peers still co-train. A 2-rank end-to-end `DaisyCluster.fit` run stays bit-identical across replicas (`replica_diff 0.0`). The WebGPU-side changes are host-side JavaScript (checked by `node --check` and by the Node suites through the shared mirror functions). The WGSL kernels themselves are unchanged.
-
-## Recent updates (September 2026) - requant rounding, bounded units, non-finite guards
-
-Everything below is measured. `python test_verified_units.py` (25 checks) plus `cd web && npm test` (twelve suites) reproduce it.
-
-**The requant rounds instead of truncating.** `NeuralRequant16` was `sat_int8(x >> 8)`, and Python's `>>` on negatives is an arithmetic floor, so every requantized activation carried a systematic bias. Over the full int16 domain:
-
-| requant | mean error | disagrees with the other on |
-| --- | --- | --- |
-| floor `x >> 8` (old) | **-0.4981 LSB** | 32,640 / 65,536 |
-| round-half-up `(x + 128) >> 8` (now) | +0.0020 LSB | |
-
-The shipped `requant16.pt` implemented the floor exactly, so this was real behaviour, not a stale docstring. It was **retrained** against the new reference and saved only at 65,536/65,536 (a 99.9% unit would silently break the LUT equivalence the browser relies on), and `web/public/requant_lut.bin` was **re-exported** to match. The invariant is *neural forward == LUT == native integer op*, so a table that lags its unit is a fork. Tie rule, stated precisely: this is half-UP. Python/numpy/torch `round` is half-to-even. The two differ on exactly the 128 exact ties (0.2% of inputs) and both are unbiased to within 0.002 LSB. The Python fleet quantizes with `np.round` and the browser with `floor(x + 0.5)`. The two fleets never co-train, so that cannot fork a group.
-
-**Bounded elementwise units.** The GEMM backends already capped their temporaries; the units right after them did not. `relu_array` / `requant_array` received the whole activation matrix in the proof path at ~1.5-2 KB per element (262,144 elements = **407 MB**), and the multiply-LUT build pushed 262,144 rows through the atom net at startup (**277 MB RSS on every node**). Both are now blocked: 407 MB -> **2 MB, flat with N**; 277 MB -> **1 MB**. Elementwise blocking is exact (no accumulation order), verified bit-identical at N = 1 / 999 / 32768 / 32769 / 100000. `dataset()` builders are vectorised (65,536 Python iterations -> one numpy pass, 0.015 s), which is what makes retraining a unit practical at all.
-
-**A NaN or Inf no longer quantizes to zeros.** A float -> int8 conversion has no answer for NaN/Inf, and both trainers gave a silent wrong one:
-
-| path | input | old result |
-| --- | --- | --- |
-| Python `qat.py` | `[0.5, nan, 3.0]` | scale = NaN -> **`[0, 0, 0]`** (only a RuntimeWarning) |
-| Python `qat.py` | `[0.5, inf, 3.0]` | scale = Inf -> **`[0, 0, 0]`** |
-| browser quantizers | `[0.5, Inf, 3]` | scale = Infinity -> **`[0, 0, 0]`** |
-| browser quantizers | `[0.5, NaN, 3]` | NaN skipped by the \|max\| scan, stored as **0** |
-
-One bad value zeroed the whole layer's int8 input and training carried on. Now `VerifiedLinear` raises `FloatingPointError`, and every browser quantizer (`quantize`, `quantizeRows`, `quantizeCols`, `rowAbsMax`, `quantizeHeadCols`, and the transformer's `quantizeColsAsRows`) throws `NonFiniteError`, naming how many values were bad. Finite inputs are untouched, so builds with and without the guard still co-train. Both new tests fail against the old code (Python: 2 checks; web: `test_nonfinite.js`, 12) and pass now.
-
-## Recent updates (August 2026) - verified compute path
-
-Three changes to `daisychain/verified/`, each measured rather than asserted. `python test_verified_units.py` (24 checks) reproduces all of it.
-
-**Bounded GEMM memory.** `LUTBackend.gemm` and `NeuralBackend.gemm` materialized the whole `(m, n, k)` product block before reducing it, so peak memory was cubic in layer width. At the SpikeWhale panel's maximum (hidden 768, sequence 512) a single layer's forward allocated **2.4 GB** - on the spare hardware this project exists to use. Both now block the `m` axis. The contraction axis `k` is untouched, so the sum and its order are unchanged and results stay bit-identical:
-
-| GEMM | before | after |
-| --- | --- | --- |
-| 64x64x64 | 2.3 MB | 2.3 MB (unchanged path) |
-| 256x256x256 | 135.8 MB | 68.9 MB |
-| 512x768x768 (panel max) | 2426.9 MB | **77.2 MB** |
-| neural backend, 64x96x96 | 519.2 MB | **57.5 MB** |
-
-Caps are class attributes (`LUTBackend.max_block_bytes`, `NeuralBackend.max_products`), tunable per instance. Below the cap the LUT path takes the original single-block branch unchanged.
-
-The two backends need *different* accounting, which is worth knowing before tuning them: a LUT product costs 8 bytes, but a neural product is split into four nibble pairs through a 128-wide net and costs ~880 bytes. Applying the LUT's byte budget to the neural path computes a block 110x too large and never splits at all.
-
-**The multiply LUT is certified, not trusted.** `build_luts()` returned its tables unchecked while the docs described a self-certify gate. Because the table *is* the complete finite domain, certifying it **is** the exhaustive verification: `LUTBackend.__init__` now checks all 65536 entries against signed integer multiply and raises on mismatch. Cost: **0.5 ms**.
-
-**Full-domain tests for the paths that ship.** The units' exhaustive verification covers the scalar entry points (`NeuralMul8.verify()` walks `mul()`). Production runs the batched ones: `mul_array`, `relu_array`, `requant_array`, and the LUTs built from them, which are different code. `test_verified_units.py` checks every shipped path against golden integer arithmetic over its complete domain. All pass today. Nothing would have caught a regression before.
-
-**Two guards against silent no-ops.** `gemm_int8` enforces the int32 accumulator bound (`MAX_INT32_K = 131071`) instead of asserting it in a comment, since layer width is user-selectable and an overflow would produce wrong numbers rather than raise. `instrument.require(**minimums)` fails when a verified unit was under-invoked, and refuses to report at all while counting is disabled. A zero from a probe that was switched off is not evidence that the units did not run.
-
-## Recent updates (July 2026) - DaisyChain-Web
-
-**Verification stack.** The browser trainer's correctness is now checked by things that run, not argued. Full results: [web/TEST_RESULTS.md](web/TEST_RESULTS.md).
-
-- **IEEE-754 oracle** (`web/test_ieee.js`). A binary32 oracle built from the standard in exact BigInt arithmetic proves the JS epilogue mirror is spec-correct, and rejects the old round-once mirror on 34% of inputs.
-- **Metamorphic properties + oracle mutation scoring** (`test_metamorphic.js`, `test_corpus.js`). Properties needing no reference implementation, scored against an externally-authored bug taxonomy. **4/4**, matching the exact differential gate's 4/4. Relations own the loop bugs. Two definitional absolutes (ReLU output range, a unit-scale integer anchor) own the value bugs no relation can see.
-- **Exact kernel gates on every live kernel**, a continuous audit at live shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on every honest device, any backend). The audit's sampling was rebuilt against a named bug class. Its old constants (6 cells, 2% of GEMMs) bounded an overhead that had never been measured (auditing *every* GEMM costs <0.01% of a step), and uniform random cells cannot see a last-row/column bug at a 16512-wide output. Sampling is now **stratified**: the first cells are the structural danger points, chosen deliberately. On a last-column bug, same cell budget: uniform caught **5/300** audits, stratified **300/300**, with zero false positives.
-- **RDNA2 ISA audit.** Reading a real GPU's shader ISA against our determinism assumptions confirmed three of them on silicon (exact packed int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal - division stays off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE variants that **flush -0 to +0**. JS `!==` cannot see that (`-0 !== 0` is false), so all gates and audits now compare **bit patterns** - exactly what the replica hash sees. (2) FMA contraction of the quantize's `x·inv + 0.5` (one rounding instead of two) turned out to be **floor-invisible by construction** - proven in `test_b2b.js` with 175k+ last-ulp anomalies at binade edges, zero surviving `floor()`. Rounding mode and denorm flushing are runtime driver state on real hardware, which is why every device re-runs the exact gates at every init.
-
-**Training data.** FineWeb-Edu (10BT sample) is the hardcoded dataset. The Space reads random slices of the parquet shards straight off the HF CDN with range requests (pure-JS `hyparquet`, SNAPPY) and serves plain text at `/data`. No dependency on the datasets-server rows API and its 503s.
-
-**Resilience.** The sync guard now *repairs* instead of halting. A roster gradient that reached the leader but not some follower (asymmetric WebRTC mesh) is re-requested from the leader, bit-exact, and the run continues. The guard still stops anything that would fork the weights.
-
-**CUTLASS-style kernel work**, each step proven bit-identical or exact-gated:
-
-- **Dispatch-optimized backward** (ex. 05/24): independent GEMMs overlapped, sibling trios fused into batch-3 dispatches. Bit-identical gradients. Dormant int8-backward path down from 1.63x to 1.21x vs float.
-- **QKV dual-GEMM fusion** (ex. 45): q/k/v share one left operand. Quantized once, one batched dispatch, zero changed bits.
-- **B2B MLP chain** (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with a fused per-row absmax reduction. The intermediate is quantized on-device via a WGSL-exact respec (`floor(f32(x·invScale)+0.5)` - no GPU division) whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical.
-
-**Profile-driven speed work.** Every change below is bit-identical (gradient and loss hashes unchanged), so none of it trades correctness for wall clock:
-
-- **Buffer pooling.** GPU buffers are recycled by size bucket instead of being created and destroyed per dispatch (~19 per MLP call, per layer, per step). **6-10% faster**, every hash unchanged.
-- **Shared-operand embedding GEMMs.** Profiling put two f32 backward GEMMs at **55% of the entire step**, and both consumed the same `dlogits` operand: ~17 MB at the 16512-token vocab, uploaded *twice*. One upload, one encoder, one submit: that pair went 205 -> 90 ms and the step **12% faster**. The fusion is gated bit-for-bit against the two calls it replaces.
-- **A negative result, kept on purpose.** The remaining hot kernel looked cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes contiguous changed nothing. Two probes explain why: holding the output at 17 MB while cutting compute 32x barely moved the time. The logits GEMM is **transfer-bound, not compute-bound**, and the readback cannot be removed because softmax must stay in JS (WGSL's `exp` is not correctly rounded, and a per-vendor `exp` would fork replicas). The real lever there is the vocabulary, not the kernel.
-- **An init backend race was tried and removed.** It tied on the shipped path, cost ~430 ms of init, and made the backend vary between page loads, which silently invalidated three A/B comparisons before it was caught. A knob that changes what you are measuring is worse than a fixed choice.
-
-**Dirty-buffer gate - and the assumption it falsified.** Pooling introduced a bug class the gates predate: a pooled buffer is *not* zero-initialized, so a kernel that assumes zeros is right on step one and wrong on step two. That is a **state** bug, where no single call is wrong and the *sequence* is, which is the family no oracle can reach. The assumption was that the gates were blind to it. Mutation-testing the gate proved otherwise: deleting the zeroing made the plain gate fail at its *second* shape, because the sweep's own shapes recycle each other's buffers. The suite had **incidental** coverage nobody designed, which is coverage nobody can rely on. Shorten the shape list and it evaporates with the gate still green. It is now deliberate: the pool is poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer depends on ordering luck. ~90 ms one-time.
-
-**Scoring the oracles against my OWN bugs** (`web/test_selfcorpus.js`). The external corpus measures kernel bugs someone else wrote down, so this suite asks the harder question: what do the instruments score against the four real bugs of the month? Properties **0/2** on the data-plane pair (the `c·out` theorem again), differential **2/2** - but half the bugs were not in the kernels at all. A dead gate is a bug in a *checker*, caught only by mutating the gate. A stalled roster gradient is a bug in the *protocol*, where every computed value on every peer was correct, so no data oracle could fire. Those needed different instruments, not better oracles.
-
-All thirteen test suites (`cd web && npm test`) pass. Results with methodology in [`web/TEST_RESULTS.md`](web/TEST_RESULTS.md).
-
----
-
-## Install
-
-```bash
-pip install torch numpy psutil
-pip install -e .          # exposes: daisychain-train, daisychain-agent, daisychain-dashboard
-```
-
-Requires Python >= 3.9, PyTorch >= 2.0. Multi-node is reliable on **Linux/macOS**. On **Windows use Docker/WSL** (see [Limitations](docs/LIMITS.md)).
-
----
-
-## Links
-
-- **DaisyChain on Hugging Face:** https://huggingface.co/DaisyChainAI
-- **This model:** https://huggingface.co/DaisyChainAI/DaisyChain-Train
-
-**License:** MIT · **Author:** Dean Byrne (Quazim0t0) · **Org:** DaisyChainAI
-
-## Citation
-
-```bibtex
-@misc{byrne2026daisychain,
-  title        = {DaisyChain-Train: An Old Hardware Training Pipeline},
-  author       = {Byrne, Dean (Quazim0t0)},
-  year         = {2026},
-  howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}},
-  note         = {Chain spare/old machines into a data-parallel training cluster}
-}
-```
-
-**Dean Byrne (Quazim0t0)** · 2026
+---
+license: mit
+tags:
+  - distributed-training
+  - old-hardware
+  - int8
+  - webgpu
+  - webrtc
+---
+
+# 🌼 DaisyChain-Train — Old Hardware Training Pipeline
+
+**Part of DaisyChain on 🤗 Hugging Face → https://huggingface.co/DaisyChainAI**
+Model page (weights + card): https://huggingface.co/DaisyChainAI/DaisyChain-Train
+
+---
+
+> **In plain terms:** DaisyChain-Train lets you use **old / spare machines** to
+> train neural networks. The training runs through **emulated GPU logic** —
+> verified INT8 units (GUDA-style) that stand in for a GPU's math — so machines
+> *without* a modern GPU can still do the work. Chain several together and they
+> train one shared model as a cluster.
+> Before you rely on it, see what it **can't** do → [Limitations](docs/LIMITS.md).
+
+**Use the hardware you already have to train.** Each machine runs the emulated
+GPU logic (verified INT8 units — multiply / requantize / ReLU) to compute the
+model, and DaisyChain pools the machines data-parallel: device selection,
+capacity-weighted sharding, gradient sync, a P2P setup, and a live dashboard.
+Two ways to run — **Docker** or **Python**.
+
+---
+
+## ⚠️ Read this first
+DaisyChain-Train is for **small models on spare hardware**. It **pools compute,
+not memory** (the model must fit on one machine), scaling is **sublinear**, and
+it is **not** a substitute for a real GPU on large models. Full envelope in
+**[docs/LIMITS.md](docs/LIMITS.md)** — please read it before relying on it.
+
+---
+
+## Feature list
+
+### Python cluster trainer (`daisychain/`)
+- **Data-parallel training across mixed machines** — each node trains its own
+  shard; gradients combine into the exact full-batch gradient, replicas stay
+  bit-identical.
+- **Capacity-weighted sharding** — faster machines automatically take a
+  bigger share of the batch.
+- **Emulated GPU compute (verified INT8 units)** — `VerifiedLinear` layers
+  run every forward multiply / requantize / ReLU through the bundled trained
+  units; cluster-wide unit-invocation counts printed by rank 0.
+- **Bring your own model** — any `Task` (`build_model` / `sample` / `loss`)
+  via `DAISY_TASK`; template in `examples/my_task_template.py`.
+- **Plain-float alternative task** — same cluster and pooling with ordinary
+  float math.
+- **Live dashboard** (`daisychain-dashboard`) — readiness banner, P2P
+  connectivity scan, pooled cores/RAM, per-node capacity plan, live loss.
+- **SpikeWhale control panel** (`spikewhale_panel`, `localhost:8899`) —
+  sliders for model size / training settings, any HF dataset you can access
+  (default streamed FineWeb-Edu), start/stop/re-adjust, live loss.
+- **Docker demo cluster** — 3 nodes + dashboard in one command.
+- **Windows helper** (`scripts\setup.bat`) and Tailscale mesh guide.
+
+### DaisyChain-Web (`web/`) — browser P2P training
+- **Zero-install nodes** — opening the page IS joining; devices on one
+  network auto-group (Snapdrop-style, by public IP).
+- **Private cross-network rooms** — `?room=CODE` with **host approval** for
+  every join.
+- **Full WebRTC mesh** — gradients travel peer-to-peer; the server only
+  signals and serves static files, it never sees weights or gradients.
+- **Leader-follower runs** — whoever presses Start sets width / sequence /
+  batch-per-device / steps / learning rate for the whole group; config
+  broadcast on the wire.
+- **Mid-run join** — late devices are synced in (weights + step) and
+  contribute from the next step.
+- **Bit-identical replicas** — same seeded init, strict roster-order gradient
+  averaging, deterministic Adam with identical state on every peer; verified
+  live by per-step weight hashes.
+- **Sync guard** — any weight-hash mismatch stops the run instead of
+  training past a fork; the step roster forbids silent partial averages.
+- **Gradient repair** — a follower missing a roster gradient re-requests it
+  from the leader (8 steps retained), bit-exact, and the run continues.
+- **Cross-device kernel probe** — every step, every device re-hashes a fixed
+  seeded int8 GEMM through its live kernel; catches broken arithmetic that
+  weight hashes cannot see.
+- **Hardcoded FineWeb-Edu streaming** — the server reads random slices of
+  the 10BT parquet shards straight off the HF CDN via HTTP range requests
+  (pure-JS `hyparquet`); built-in corpus fallback offline.
+- **Checkpoints** — download `.pt`, upload → **broadcast to the whole
+  group**; validated (magic, dims, tokenizer vocab) before accepting.
+- **Inference kit** — one self-contained HTML file with the trained weights
+  baked in; generations offline, anywhere.
+- **In-page generation** — prompt box on the trained model.
+- **Old-hardware tier** — no WebGPU? The identical units run on CPU (same
+  bits, so CPU and GPU devices co-train in one group). There is no
+  plain-float path.
+- **Large-message fragmentation** — multi-MB gradients/checkpoints chunked
+  at 48 KB over the data channels.
+
+### Verified compute & kernels (web)
+- **Verified INT8 units everywhere** — block-scaled int8 GEMM: exact LUT
+  products, exact int32 accumulation, bit-exact f32 epilogue with a pinned
+  rounding schedule; scales derived in JS f64 (division never runs on GPU).
+- **Backends, best-first** — DP4A hardware int8 dot → LUT compute shader →
+  CPU mirror; every kernel **exact-gated at init** (bit-level compares) and
+  demoted to the mirror on any mismatch.
+- **Continuous random-cell audit** at live training shapes.
+- **Fused attention kernels** — gather/scatter head-strided q·kᵀ and a·v
+  straight from BT×C layout (CUTLASS ex. 36/52 style).
+- **QKV dual-GEMM fusion** — shared left operand quantized once, one batch-3
+  dispatch (ex. 45); bit-identical.
+- **B2B MLP chain** — both MLP GEMMs back-to-back on GPU with fused per-row
+  absmax reduction and on-device quantize (ex. 13 + 23); WGSL-exact respec
+  with a fround-stepped JS mirror; fma-contraction-immune by construction.
+- **Dispatch-optimized backward** — overlapped independent GEMMs, batch-3
+  sibling fusions (ex. 05/24); bit-identical gradients; optional int8 STE
+  backward path (dormant, 1.21× vs float).
+
+### Verification stack (web)
+- **Exact init gates** on every kernel, every device, every boot — including
+  gates that "gate the gate" with discriminating boundary inputs.
+- **IEEE-754 binary32 oracle** in exact BigInt arithmetic — proves the JS
+  epilogue mirror is spec-correct (rejects the old mirror on 34% of inputs).
+- **Metamorphic property suite** — reference-free relations + definitional
+  absolutes; **4/4** on an externally-authored bug corpus, matching the
+  differential gate.
+- **RDNA2 ISA audit hardenings** — bit-level (−0-aware) gate comparisons;
+  proof that FMA contraction cannot change the quantize.
+- **Twelve-suite test chain** (`cd web && npm test`) — convergence, replicas,
+  oracle, gates, properties, external corpus, **self-corpus** (the instruments
+  scored against my own bugs), B2B, optimizer, transformer LM, int8 backward,
+  non-finite quantize refusal;
+  results in [web/TEST_RESULTS.md](web/TEST_RESULTS.md).
+- **Dirty-buffer gate** — the pool is poisoned before a re-sweep so state bugs
+  (a kernel assuming zeroed memory) are caught deterministically rather than
+  by ordering luck.
+
+### Documentation
+- Python: [QUICKSTART](docs/QUICKSTART.md), [LIMITS](docs/LIMITS.md),
+  [CUSTOM_TASK](docs/CUSTOM_TASK.md), [TAILSCALE](docs/TAILSCALE.md).
+- Web: [Getting started](web/docs/GETTING_STARTED.md),
+  [Architecture](web/docs/ARCHITECTURE.md),
+  [Verification](web/docs/VERIFICATION.md),
+  [Troubleshooting](web/docs/TROUBLESHOOTING.md).
+
+---
+
+## Quick start
+
+### Docker (most reliable — one command)
+```bash
+docker compose -f docker/docker-compose.yml up --build
+# open http://localhost:8080
+```
+Brings up a 3-node demo cluster + dashboard on one machine.
+
+### Python (real machines)
+On every machine (`pip install -e .`):
+```bash
+export MASTER_ADDR=100.101.102.10   # coordinator IP (Tailscale 100.x recommended)
+export MASTER_PORT=29560
+export WORLD_SIZE=3
+export RANK=0                        # 1, 2, ... on the others
+export GLOO_SOCKET_IFNAME=tailscale0 # your mesh / LAN NIC
+daisychain-train
+```
+
+### Windows helper
+```bat
+scripts\setup.bat
+```
+An interactive menu: Docker, Python node, or just install deps.
+
+Full walkthrough: **[docs/QUICKSTART.md](docs/QUICKSTART.md)**.
+
+### 🐋 SpikeWhale control panel (sliders → real training)
+```bash
+python -m daisychain.spikewhale_panel
+# open http://localhost:8899
+```
+A web control panel: pick model size / training settings with sliders, choose any
+HuggingFace dataset you have access to (default: streamed FineWeb-Edu), hit
+Start, and watch the live loss. Stop and re-adjust any time with
+**← Back to settings**. Launches the real DaisyChain training underneath.
+
+### 🌐 DaisyChain-Web (train by opening a browser tab)
+```bash
+cd web && npm install && node server.js
+# open http://localhost:8787 on every device
+```
+Zero-install browser training: devices on the same network auto-group
+(Snapdrop-style) and train a shared model **peer-to-peer over WebRTC**, computing
+through the same verified INT8 units (WebGPU, with the identical units on CPU
+for machines without it — there is no plain-float path). Private cross-network
+rooms via `?room=CODE` with **host approval** — the room creator accepts each
+device before it can join. Includes gradient averaging with a deterministic
+Adam optimizer (identical state on every peer, nothing extra over the wire),
+checkpoint **download** (.pt) and **upload → broadcast** so one device can
+restore the whole group after a failure.
+
+**Live demo:** https://huggingface.co/spaces/Quazim0t0/DaisyChain-Web
+
+### Recent updates (July 2026) — DaisyChain-Web
+
+**Verification stack** — the browser trainer's correctness is now *checked by
+things that run*, not argued ([full results](web/TEST_RESULTS.md)):
+- **IEEE-754 oracle** (`web/test_ieee.js`): a binary32 oracle built from the
+  standard in exact BigInt arithmetic proves the JS epilogue mirror is
+  spec-correct — and rejects the old round-once mirror on 34% of inputs.
+- **Metamorphic properties + oracle mutation scoring** (`test_metamorphic.js`,
+  `test_corpus.js`): properties needing no reference implementation, scored
+  against an externally-authored bug taxonomy — **4/4**, matching the exact
+  differential gate's 4/4. Relations own the loop bugs; two definitional
+  absolutes (ReLU output range, a unit-scale integer anchor) own the value
+  bugs no relation can see.
+- **Exact kernel gates on every live kernel**, a continuous audit at live
+  shapes, and a cross-device kernel probe (same seeded int8 GEMM, same hash on
+  every honest device, any backend). The audit's sampling was rebuilt against
+  a named bug class: its old constants (6 cells, 2% of GEMMs) bounded an
+  overhead that had never been measured (auditing *every* GEMM costs <0.01% of
+  a step), and uniform random cells cannot see a last-row/column bug at a
+  16512-wide output. Sampling is now **stratified** — the first cells are the
+  structural danger points, chosen deliberately. On a last-column bug, same
+  cell budget: uniform caught **5/300** audits, stratified **300/300**, with
+  zero false positives.
+- **RDNA2 ISA audit**: reading a real GPU's shader ISA against our
+  determinism assumptions confirmed three of them on silicon (exact packed
+  int8 dot; correctly-rounded f32 add/mul; 1-ULP reciprocal — division stays
+  off the GPU) and produced two hardenings. (1) Real ISAs have non-IEEE
+  variants that **flush −0 to +0**; JS `!==` can't see that (`-0 !== 0` is
+  false), so all gates and audits now compare **bit patterns** — exactly what
+  the replica hash sees. (2) FMA contraction of the quantize's `x·inv + 0.5`
+  (one rounding instead of two) turned out to be **floor-invisible by
+  construction** — proven in `test_b2b.js` with 175k+ last-ulp anomalies at
+  binade edges, zero surviving `floor()`. Rounding mode and denorm flushing
+  are runtime driver state on real hardware, which is why every device
+  re-runs the exact gates at every init.
+
+**Training data** — FineWeb-Edu (10BT sample) is the hardcoded dataset. The
+Space reads random slices of the parquet shards straight off the HF CDN with
+range requests (pure-JS `hyparquet`, SNAPPY) and serves plain text at `/data`
+— no dependency on the datasets-server rows API and its 503s.
+
+**Resilience** — the sync guard now *repairs* instead of halting: a roster
+gradient that reached the leader but not some follower (asymmetric WebRTC
+mesh) is re-requested from the leader, bit-exact, and the run continues. The
+guard still stops anything that would fork the weights.
+
+**CUTLASS-style kernel work**, each step proven bit-identical or exact-gated:
+- **Dispatch-optimized backward** (ex. 05/24): independent GEMMs overlapped,
+  sibling trios fused into batch-3 dispatches — bit-identical gradients,
+  dormant int8-backward path down from 1.63× to 1.21× vs float.
+- **QKV dual-GEMM fusion** (ex. 45): q/k/v share one left operand — quantized
+  once, one batched dispatch, zero changed bits.
+- **B2B MLP chain** (ex. 13 + 23): both MLP GEMMs back-to-back on the GPU with
+  a fused per-row absmax reduction; the intermediate is quantized on-device
+  via a WGSL-exact respec (`floor(f32(x·invScale)+0.5)` — no GPU division)
+  whose fround-stepped JS mirror keeps mixed GPU/CPU fleets bit-identical.
+
+**Profile-driven speed work** — every change below is bit-identical (gradient
+and loss hashes unchanged), so none of it trades correctness for wall clock:
+- **Buffer pooling**: GPU buffers are recycled by size bucket instead of being
+  created and destroyed per dispatch (~19 per MLP call, per layer, per step).
+  **6–10% faster**, every hash unchanged.
+- **Shared-operand embedding GEMMs**: profiling put two f32 backward GEMMs at
+  **55% of the entire step**, and both consumed the same `dlogits` operand —
+  ~17 MB at the 16512-token vocab, uploaded *twice*. One upload, one encoder,
+  one submit: that pair went 205 → 90 ms and the step **12% faster**. The
+  fusion is gated bit-for-bit against the two calls it replaces.
+- **A negative result, kept on purpose**: the remaining hot kernel looked
+  cache-hostile (adjacent lanes wrote 66 KB apart), but making the writes
+  contiguous changed nothing. Two probes explain why — holding the output at
+  17 MB while cutting compute 32× barely moved the time. The logits GEMM is
+  **transfer-bound, not compute-bound**, and the readback cannot be removed
+  because softmax must stay in JS (WGSL's `exp` is not correctly rounded, and
+  a per-vendor `exp` would fork replicas). The real lever there is the
+  vocabulary, not the kernel.
+- **An init backend race was tried and removed**: it tied on the shipped path,
+  cost ~430 ms of init, and made the backend vary between page loads, which
+  silently invalidated three A/B comparisons before it was caught. A knob that
+  changes what you are measuring is worse than a fixed choice.
+
+**Dirty-buffer gate — and the assumption it falsified.** Pooling introduced a
+bug class the gates predate: a pooled buffer is *not* zero-initialized, so a
+kernel that assumes zeros is right on step one and wrong on step two. That is
+a **state** bug, where no single call is wrong and the *sequence* is, which is
+the family no oracle can reach. The assumption was that the gates were blind
+to it. Mutation-testing the gate proved otherwise: deleting the zeroing made
+the plain gate fail at its *second* shape, because the sweep's own shapes
+recycle each other's buffers. The suite had **incidental** coverage nobody
+designed, which is coverage nobody can rely on — shorten the shape list and it
+evaporates with the gate still green. It is now deliberate: the pool is
+poisoned with 1e4-magnitude residue before a re-sweep, so detection no longer
+depends on ordering luck. ~90 ms one-time.
+
+**Scoring the oracles against my OWN bugs** (`web/test_selfcorpus.js`) — the
+external corpus measures kernel bugs someone else wrote down, so this suite
+asks the harder question: what do the instruments score against the four real
+bugs of the month? Properties **0/2** on the data-plane pair (the `c·out`
+theorem again), differential **2/2** — but half the bugs were not in the
+kernels at all. A dead gate is a bug in a *checker*, caught only by mutating
+the gate; a stalled roster gradient is a bug in the *protocol*, where every
+computed value on every peer was correct, so no data oracle could fire. Those
+needed different instruments, not better oracles.
+
+All thirteen test suites (`cd web && npm test`) pass; results with methodology in
+[`web/TEST_RESULTS.md`](web/TEST_RESULTS.md).
+
+---
+
+## How it works
+
+Each machine runs the **same** command; they form a cluster and train one shared
+model. Two things happen:
+
+1. **The compute runs through the emulated GPU logic.** By default the model is
+   built from `VerifiedLinear` layers, so every forward multiply / requantize /
+   ReLU is done by the **bundled verified INT8 units** (`daisychain/verified/`) —
+   the emulated GPU math. Rank 0 prints **cluster-wide unit-invocation counts**
+   so you can see the emulated logic doing the work.
+2. **The machines are pooled data-parallel.** Each node trains on its own shard;
+   gradients are capacity-weighted and combined into the exact full-batch
+   gradient, so replicas stay **bit-identical**. Faster machines automatically
+   take a bigger share.
+
+```
+  old machine A ─┐
+  old machine B ─┼─►  each runs the emulated GPU logic on its shard  ─►  one model
+  old machine C ─┘        (gradients combined across the cluster)
+```
+
+## Bring your own model
+DaisyChain-Train trains any **Task** (`build_model` / `sample` / `loss`). Copy
+`examples/my_task_template.py`, set `DAISY_TASK=your_module:YourTask`. Use
+`VerifiedLinear` (see `daisychain/verified_task.py`) to run your model's compute
+through the emulated units. See **[docs/CUSTOM_TASK.md](docs/CUSTOM_TASK.md)**.
+
+## Plain-float alternative
+To skip the emulated units and train with normal float math on each machine, set
+`DAISY_TASK=daisychain.example_task:ExampleTask`. Same cluster, same pooling —
+the model math just runs as ordinary float instead of through the verified units.
+
+## The dashboard
+`daisychain-dashboard` (or the Docker service) serves a Tailwind page at `:8080`
+— readiness banner, P2P connectivity scan, pooled cores/RAM + capacity plan
+(per-node device, weight, batch), and live training loss.
+
+## Networking
+Use **Tailscale** for a P2P mesh so machines on different networks get stable IPs
+on one interface — **[docs/TAILSCALE.md](docs/TAILSCALE.md)**.
+
+---
+
+## Layout
+```
+daisychain/cluster.py        capacity-weighted CPU/GPU data-parallel trainer
+daisychain/train.py          entry point (daisychain-train)
+daisychain/verified/         bundled trained N/N units + VerifiedLinear (train through them)
+daisychain/verified_task.py  default task: forward runs on the verified units
+daisychain/example_task.py   plain-float alternative task
+daisychain/task.py           the Task interface + loader
+daisychain/dashboard/        agent + P2P scanner + Tailwind server
+docker/                      Dockerfile, dashboard image, compose (demo cluster)
+scripts/setup.bat / setup.sh interactive setup helpers
+config/                      nodes + cluster env examples
+examples/my_task_template.py starting point for your own model
+test_verified_units.py       full-domain checks of every shipped verified path (25)
+docs/                        QUICKSTART, LIMITS, CUSTOM_TASK, TAILSCALE
+daisychain/spikewhale_task.py   trains the real SpikeWhale on streamed HF datasets
+daisychain/spikewhale_panel.py  slider control panel (localhost:8899)
+web/                         DaisyChain-Web: P2P browser training (WebRTC + WebGPU)
+export_luts_web.py           regenerates web/public LUTs from the trained units
+```
+
+## Recent updates (14 October 2026) — one arithmetic, whatever the device
+
+Lessons from making an emulator give one set of semantics on two backends, applied here and
+measured on real hardware:
+
+- **Mixed CPU + GPU clusters forked at step 1.** On identical gradients, CUDA Adam and CPU Adam
+  differ in about 2% of weights after 20 steps. That holds for the default, `foreach=False` and
+  `fused=True` alike: same rule, two implementations, two roundings. `replica_diff` only ran
+  after training, so nothing noticed. Now:
+  - In a mixed fleet, GPU ranks step the optimizer on CPU master copies, so every rank runs one
+    implementation (`DAISY_CPU_OPT=0/1` overrides the choice).
+  - Every rank compares a hash of its parameter bits at the logging cadence and stops on any
+    difference.
+  - `test_cluster_mixed.py` (real 2-rank gloo, one CUDA rank and one CPU rank) checks that the
+    old behaviour is stopped by the check and that the new one ends bit-identical.
+- **Host floating-point mode is pinned.** With flush-to-zero on, the same seed gives different
+  weights (measured), and any library built with `-ffast-math` can switch it on just by loading.
+  `pin_fp_mode()` turns it off in `configure_cpu` and `fit`.
+- **Subnormals in WebGPU.** WGSL permits flushing f32 subnormals, and the JS mirror never
+  flushes. The f32 GEMM gate's inputs (~0.1) never produced one. Measured in a real browser: the
+  Intel gen-9 iGPU **flushes**, and the old gate called it "bit-exact". The gate now includes a
+  subnormal case, so a flushing GPU keeps its backward in JS instead of forking from
+  non-flushing peers. The app also asks for the `high-performance` adapter now; the default
+  picked that iGPU over a discrete GPU.
+- **Capacity in one unit.** A node whose benchmark failed reported its *core count* as
+  capacity, which was then weighed against other nodes' matmuls/sec. Mixed units now fall back
+  to core counts for every node, with a warning.
+- **The kernel probe can no longer pass by not running.** If a device's probe threw, it kept its
+  previous hash (or 0), and a `phash` of 0 made the peer comparison *skip* that device. A failed
+  probe now reports 0, and 0 halts the run (`web/test_probe_unproven.js`).
+- **Partial fragments of a closed channel are dropped.** Before, they were kept on the side that
+  doesn't start the reconnect. Repeated or out-of-range fragment numbers are rejected instead of
+  being counted.
+
+Verified in-browser: 2 peers over WebRTC, 100/100 steps, loss 9.76 → 6.26, no weight, kernel or
+audit halts.
+
+### The f32 backward GEMM is audited too
+
+Three gaps from a review of which checks actually cover each live WebGPU kernel:
+
+- **The transposed f32 GEMM was compared only with itself.** The init gate checked the plain
+  split-K shape against the exact mirror. The transposed index pattern (`A[p*m+row]`, used for
+  the embedding gradient) was only ever compared as `fgemm2`'s leg against an `fgemm` call that
+  runs the same shader, which is circular. It is now gated against `fgemmMirror`, including a
+  transposed split-K shape (k > 4096).
+- **The f32 GEMM had no live-shape audit.** Every int8 kernel samples cells at live shapes. The
+  f32 pair, whose output *is* the embedding gradient, had been checked only at 7×4500×5 and
+  9×300×6. It is now audited by `auditFgemm` (structural corners first, then random cells) on
+  the mirror's partition and rounding schedule. The device's schedule (`fgemmFma`) is recorded
+  at init and passed through.
+- **Dirty-pool re-gates for the attention kernels and the f32 GEMM.** The bgemm and MLP gates
+  already re-ran on pooled buffers poisoned with large residue. The attention and f32 kernels
+  were only ever gated on fresh buffers; now they are re-gated on poisoned ones too.
+
+`web/test_fgemm_audit.js` (in `npm test`) checks that the audit accepts the mirror at live-like
+shapes under both schedules and catches a 1-ulp error in a corner and in an interior cell. It
+also checks that the two schedules really differ on split-K data, so the flag is load-bearing.
+
+## Recent updates (30 September 2026) — reductions refuse what they used to accept
+
+Six fixes, each found by applying a lesson from differential-testing an RDNA2 shader
+translator: *a second implementation nobody checks is not verified*, *inactive or absent
+participants must not leak into a reduction*, *a max over NaN is undefined*, and *know your
+accumulator width*. Every one is covered by a check that fails on the old code.
+`python test_verified_units.py` + `python test_cluster_reductions.py` and
+`cd web && npm test` (thirteen suites) reproduce all of it.
+
+| # | where | what was wrong | now |
+| --- | --- | --- | --- |
+| 1 | browser, WebGPU MLP chain | the row max comes from an `atomicMax` over `bitcast(abs(h))` and never passed the non-finite guard the CPU path has; a NaN/Inf `h1` became `inv = 0`/NaN and the GPU quantized `floor(NaN)` (implementation-defined) where a CPU peer threw | `scalesFromAbsMax` refuses a non-finite max. NaN and Inf bits order above every finite value as u32, so the row max is non-finite **iff** some element is — one check covers the whole row (checked on 2000 random rows) |
+| 2 | browser, `averageGrads` | a gradient of a different length was silently averaged over a prefix (typed arrays ignore out-of-range writes); a NaN/Inf gradient went straight into DaisyAdam, poisoning `m`/`v` on **every** peer identically, so no fork check could see it | refuses both with `GradientError`; the training loop halts at that step with weights and Adam state untouched (still a valid checkpoint) |
+| 3 | browser, int8 GEMMs (JS and WGSL) | the `Int32Array` and WGSL `i32` accumulators wrap past K = 131,071 — CPU and GPU agree, on a wrong sum | refused with `AccumulatorBoundError` at every GEMM entry point; the same bound as the Python kernel's `MAX_INT32_K` |
+| 4 | Python, logged cluster loss | `sum(loss_i) / world` — an equal-weight mean of **unequal**, capacity-weighted batches | `cluster_mean_loss` = Σ wᵢ·lossᵢ with the gradient's own weights. Measured on 3- and 13-sample nodes: old **1.418**, true full-batch loss **0.959**, new 0.959 |
+| 5 | Python, gradient all-reduce | one node's NaN was summed into every replica and applied | `check_finite_grads` refuses the reduced gradient before `opt.step()`; every rank holds the same sum, so every rank stops at the same step with no extra communication |
+| 6 | Python, `pick_device` | with `CUDA_VISIBLE_DEVICES` set but **empty** (a common "CPU only"), torch reports `is_available() = True` and `device_count() = 0`, and the node crashed at startup | a device must actually exist; falls back to CPU |
+
+Unchanged for valid inputs: finite gradients, finite activations, and K ≤ 131,071 produce
+bit-identical results to before, so updated and older peers still co-train. A 2-rank
+end-to-end `DaisyCluster.fit` run stays bit-identical across replicas (`replica_diff 0.0`).
+The WebGPU-side changes are host-side JavaScript (checked by `node --check` and by the
+Node suites through the shared mirror functions); the WGSL kernels themselves are unchanged.
+
+## Recent updates (September 2026) — requant rounding, bounded units, non-finite guards
+
+Everything below is measured, and `python test_verified_units.py` (25 checks) plus
+`cd web && npm test` (twelve suites) reproduce it.
+
+**The requant rounds instead of truncating.** `NeuralRequant16` was
+`sat_int8(x >> 8)`, and Python's `>>` on negatives is an arithmetic floor, so every
+requantized activation carried a systematic bias. Over the full int16 domain:
+
+| requant | mean error | disagrees with the other on |
+| --- | --- | --- |
+| floor `x >> 8` (old) | **-0.4981 LSB** | 32,640 / 65,536 |
+| round-half-up `(x + 128) >> 8` (now) | +0.0020 LSB | |
+
+The shipped `requant16.pt` implemented the floor exactly, so this was real behaviour,
+not a stale docstring. It was **retrained** against the new reference and saved only
+at 65,536/65,536 (a 99.9% unit would silently break the LUT equivalence the browser
+relies on), and `web/public/requant_lut.bin` was **re-exported** to match: the
+invariant is *neural forward == LUT == native integer op*, so a table that lags its
+unit is a fork. Tie rule, stated precisely: this is half-UP; Python/numpy/torch
+`round` is half-to-even. The two differ on exactly the 128 exact ties (0.2% of
+inputs) and both are unbiased to within 0.002 LSB. The Python fleet quantizes with
+`np.round` and the browser with `floor(x + 0.5)`; the two fleets never co-train, so
+that cannot fork a group.
+
+**Bounded elementwise units.** The GEMM backends already capped their temporaries;
+the units right after them did not. `relu_array` / `requant_array` received the
+whole activation matrix in the proof path at ~1.5–2 KB per element (262,144
+elements = **407 MB**), and the multiply-LUT build pushed 262,144 rows through the
+atom net at startup (**277 MB RSS on every node**). Both are now blocked:
+407 MB → **2 MB, flat with N**; 277 MB → **1 MB**. Elementwise blocking is exact
+(no accumulation order), verified bit-identical at N = 1 / 999 / 32768 / 32769 /
+100000. `dataset()` builders are vectorised (65,536 Python iterations → one numpy
+pass, 0.015 s), which is what makes retraining a unit practical at all.
+
+**A NaN or Inf no longer quantizes to zeros.** A float → int8 conversion has no
+answer for NaN/Inf, and both trainers gave a silent wrong one:
+
+| path | input | old result |
+| --- | --- | --- |
+| Python `qat.py` | `[0.5, nan, 3.0]` | scale = NaN → **`[0, 0, 0]`** (only a RuntimeWarning) |
+| Python `qat.py` | `[0.5, inf, 3.0]` | scale = Inf → **`[0, 0, 0]`** |
+| browser quantizers | `[0.5, Inf, 3]` | scale = Infinity → **`[0, 0, 0]`** |
+| browser quantizers | `[0.5, NaN, 3]` | NaN skipped by the \|max\| scan, stored as **0** |
+
+One bad value zeroed the whole layer's int8 input and training carried on. Now
+`VerifiedLinear` raises `FloatingPointError`, and every browser quantizer
+(`quantize`, `quantizeRows`, `quantizeCols`, `rowAbsMax`, `quantizeHeadCols`, and
+the transformer's `quantizeColsAsRows`) throws `NonFiniteError`, naming how many
+values were bad. Finite inputs are untouched, so builds with and without the guard
+still co-train. Both new tests fail against the old code (Python: 2 checks; web:
+`test_nonfinite.js`, 12) and pass now.
+
+## Recent updates (August 2026) — verified compute path
+
+Three changes to `daisychain/verified/`, each measured rather than asserted.
+`python test_verified_units.py` (24 checks) reproduces all of it.
+
+**Bounded GEMM memory.** `LUTBackend.gemm` and `NeuralBackend.gemm` materialized
+the whole `(m, n, k)` product block before reducing it, so peak memory was cubic
+in layer width. At the SpikeWhale panel's maximum (hidden 768, sequence 512) a
+single layer's forward allocated **2.4 GB** — on the spare hardware this project
+exists to use. Both now block the `m` axis. The contraction axis `k` is untouched,
+so the sum and its order are unchanged and results stay bit-identical:
+
+| GEMM | before | after |
+| --- | --- | --- |
+| 64x64x64 | 2.3 MB | 2.3 MB (unchanged path) |
+| 256x256x256 | 135.8 MB | 68.9 MB |
+| 512x768x768 (panel max) | 2426.9 MB | **77.2 MB** |
+| neural backend, 64x96x96 | 519.2 MB | **57.5 MB** |
+
+Caps are class attributes (`LUTBackend.max_block_bytes`,
+`NeuralBackend.max_products`), tunable per instance. Below the cap the LUT path
+takes the original single-block branch unchanged.
+
+The two backends need *different* accounting, which is worth knowing before tuning
+them: a LUT product costs 8 bytes, but a neural product is split into four nibble
+pairs through a 128-wide net and costs ~880 bytes. Applying the LUT's byte budget
+to the neural path computes a block 110x too large and never splits at all.
+
+**The multiply LUT is certified, not trusted.** `build_luts()` returned its tables
+unchecked while the docs described a self-certify gate. Because the table *is* the
+complete finite domain, certifying it **is** the exhaustive verification:
+`LUTBackend.__init__` now checks all 65536 entries against signed integer multiply
+and raises on mismatch. Cost: **0.5 ms**.
+
+**Full-domain tests for the paths that ship.** The units' exhaustive verification
+covers the scalar entry points (`NeuralMul8.verify()` walks `mul()`); production
+runs the batched ones — `mul_array`, `relu_array`, `requant_array` — and the LUTs
+built from them, which are different code. `test_verified_units.py` checks every
+shipped path against golden integer arithmetic over its complete domain. All pass
+today; nothing would have caught a regression before.
+
+**Two guards against silent no-ops.** `gemm_int8` enforces the int32 accumulator
+bound (`MAX_INT32_K = 131071`) instead of asserting it in a comment, since layer
+width is user-selectable and an overflow would produce wrong numbers rather than
+raise. `instrument.require(**minimums)` fails when a verified unit was
+under-invoked, and refuses to report at all while counting is disabled — a zero
+from a probe that was switched off is not evidence that the units did not run.
+
+---
+
+## Install
+```bash
+pip install torch numpy psutil
+pip install -e .          # exposes: daisychain-train, daisychain-agent, daisychain-dashboard
+```
+Requires Python ≥ 3.9, PyTorch ≥ 2.0. Multi-node is reliable on **Linux/macOS**;
+on **Windows use Docker/WSL** (see [Limitations](docs/LIMITS.md)).
+
+---
+
+## Links
+- **DaisyChain on Hugging Face:** https://huggingface.co/DaisyChainAI
+- **This model:** https://huggingface.co/DaisyChainAI/DaisyChain-Train
+
+**License:** MIT · **Author:** Dean Byrne (Quazim0t0) · **Org:** DaisyChainAI
+
+## Citation
+```bibtex
+@misc{byrne2026daisychain,
+  title        = {DaisyChain-Train: An Old Hardware Training Pipeline},
+  author       = {Byrne, Dean (Quazim0t0)},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/DaisyChainAI/DaisyChain-Train}},
+  note         = {Chain spare/old machines into a data-parallel training cluster}
+}
+```
+
+**Dean Byrne (Quazim0t0)** · 2026

daisychain/cluster.py

@@ -28,9 +28,20 @@ def configure_cpu(fraction: float = 0.9) -> int:
     cores = os.cpu_count() or 1
     n = max(1, int(round(cores * fraction)))
     torch.set_num_threads(n)
+    pin_fp_mode()
     return n
 
 
+def pin_fp_mode() -> None:
+    """Pin the host floating-point mode: subnormals ON (no flush-to-zero / denormals-are-zero).
+
+    FTZ/DAZ live in the thread's MXCSR, and any library built with -ffast-math can switch them
+    on just by being loaded (the emulator found exactly this leaking into Python, 2026-09).
+    MEASURED 2026-10-14: the same seed and gradients through 5 Adam steps give DIFFERENT weights
+    with torch.set_flush_denormal(True) -- so one node with a leaked FTZ forks its replica."""
+    torch.set_flush_denormal(False)
+
+
 def _cuda_usable() -> bool:
     """is_available() alone is not enough: with CUDA_VISIBLE_DEVICES set but EMPTY (a common way
     to say "CPU only") torch reports available=True and device_count()=0, and the node crashed
@@ -60,9 +71,14 @@ def _available_ram_gb():
         return None
 
 
-def capacity_score(device=None, secs: float = 0.3) -> float:
+def capacity_score(device=None, secs: float = 0.3):
     """Measured throughput: fixed matmuls/sec on the local device. Self-calibrating
-    (a GPU scores far higher), so capacity weighting hands it a bigger batch."""
+    (a GPU scores far higher), so capacity weighting hands it a bigger batch.
+
+    Returns None if the measurement fails. It used to return the core count instead -- a number
+    in a DIFFERENT UNIT that cluster_plan then weighed against other nodes' matmuls/sec, so a node
+    whose benchmark threw got ~8 "capacity" next to a peer's ~200 and a sliver of the batch, with
+    nothing in the plan saying why (2026-10-14: a fallback that looks like a result)."""
     dev = device or pick_device()
     try:
         a = torch.randn(512, 512, device=dev)
@@ -78,7 +94,7 @@ def capacity_score(device=None, secs: float = 0.3) -> float:
             torch.cuda.synchronize()
         return it / (time.time() - t0)
     except Exception:
-        return float(os.cpu_count() or 1)
+        return None
 
 
 def survey_node(cpu_fraction: float = 0.9, measure: bool = True) -> dict:
@@ -86,15 +102,36 @@ def survey_node(cpu_fraction: float = 0.9, measure: bool = True) -> dict:
     dev = pick_device()
     gpu = _gpu_info() if dev.type == "cuda" else None
     if "DAISY_CAPACITY" in os.environ:
-        cap = float(os.environ["DAISY_CAPACITY"])
+        cap, unit = float(os.environ["DAISY_CAPACITY"]), "user"
     elif measure:
-        cap = capacity_score(dev)
+        cap, unit = capacity_score(dev), "matmul/s"
+        if cap is None:
+            cap, unit = float(cores), "cores"
     else:
-        cap = float(cores)
+        cap, unit = float(cores), "cores"
     return {"host": socket.gethostname(), "cores": cores,
             "threads": max(1, int(round(cores * cpu_fraction))),
             "ram_gb": _available_ram_gb(), "device": dev.type,
-            "gpu": gpu, "capacity": cap}
+            "gpu": gpu, "capacity": cap, "capacity_unit": unit}
+
+
+def plan_capacities(nodes):
+    """Per-node capacities in ONE unit, plus the unit used.
+
+    Measured matmuls/sec and core counts are not comparable. If the measured nodes do not all
+    share a unit, every node falls back to its core count (a coarser but consistent weighting)
+    instead of mixing the two. A DAISY_CAPACITY override ("user") is taken as stated: the
+    operator chose its scale. Nodes from an older survey without a unit count as "cores" if
+    their capacity equals their core count, else as measured."""
+    def unit_of(g):
+        u = g.get("capacity_unit")
+        if u:
+            return u
+        return "cores" if float(g.get("capacity") or 0) == float(g["cores"]) else "matmul/s"
+    units = {unit_of(g) for g in nodes} - {"user"}
+    if len(units) > 1:
+        return [float(g["cores"]) for g in nodes], "cores (mixed units: %s)" % ", ".join(sorted(units))
+    return [float(g.get("capacity") or g["cores"]) for g in nodes], (units.pop() if units else "user")
 
 
 # ------------------------------------------------------------------ cluster ---
@@ -111,7 +148,10 @@ def cluster_plan(cpu_fraction: float = 0.9, base_batch: int = 32) -> dict:
     gathered = [None] * world
     dist.all_gather_object(gathered, me)
 
-    caps = [float(g.get("capacity") or g["cores"]) for g in gathered]
+    caps, cap_unit = plan_capacities(gathered)
+    if cap_unit.startswith("cores (mixed") and rank == 0:
+        print("  [plan] capacity measured in different units across nodes -- weighting every node "
+              "by core count instead (%s)" % cap_unit, flush=True)
     total_cap = sum(caps) or world
     global_batch = base_batch * world
     batches = [max(1, round(global_batch * c / total_cap)) for c in caps]
@@ -125,10 +165,39 @@ def cluster_plan(cpu_fraction: float = 0.9, base_batch: int = 32) -> dict:
             "total_ram_gb": (sum(rams) if rams else None),
             "global_batch": sum(batches),
             "devices": [g.get("device", "cpu") for g in gathered],
-            "capacities": [round(c, 1) for c in caps],
+            "capacities": [round(c, 1) for c in caps], "capacity_unit": cap_unit,
             "gpus": [g.get("gpu") for g in gathered]}
 
 
+@torch.no_grad()
+def param_hash(params) -> int:
+    """64-bit BLAKE2b digest of the exact parameter BITS (CPU copies), for replica checks."""
+    import hashlib
+    h = hashlib.blake2b(digest_size=8)
+    for p in params:
+        h.update(p.detach().to("cpu").contiguous().view(-1).numpy().tobytes())
+    return int.from_bytes(h.digest(), "little", signed=True)
+
+
+@torch.no_grad()
+def check_replicas(params, step: int) -> None:
+    """Refuse to continue if any replica's parameter bits differ from rank 0's. A collective.
+
+    replica_diff used to run once, AFTER training: a fork at step 1 surfaced only as a final
+    number. The web client hashes weights every step; this is the same check for the cluster."""
+    mine = torch.tensor([param_hash(params)], dtype=torch.int64)
+    allh = [torch.zeros_like(mine) for _ in range(dist.get_world_size())]
+    dist.all_gather(allh, mine)
+    bad = [r for r, h in enumerate(allh) if int(h) != int(allh[0])]
+    if bad:
+        raise RuntimeError("step %d: replica(s) %s diverged from rank 0 (parameter bits differ); "
+                           "refusing to continue on forked weights" % (step, bad))
+
+
+def mixed_devices(plan) -> bool:
+    return len(set(plan.get("devices") or [])) > 1
+
+
 @torch.no_grad()
 def broadcast_params(model, src: int = 0):
     for p in model.parameters():
@@ -204,15 +273,38 @@ class DaisyCluster:
             pass
 
     def fit(self, model, task, steps=500, lr=1e-2, optimizer="sgd",
-            status_path=None, step_delay=0.0):
+            status_path=None, step_delay=0.0, check_every=None):
         """Train `model` on `task` (build_model already called). task.sample(n)
-        draws this node's shard; task.loss(model, X, y) returns a scalar."""
+        draws this node's shard; task.loss(model, X, y) returns a scalar.
+
+        Mixed CPU/GPU fleets: the optimizer runs on CPU MASTER copies on every rank. MEASURED
+        2026-10-14 (RTX, torch 2.x): CUDA Adam vs CPU Adam on identical gradients differ in ~2% of
+        weights after 20 steps (default, foreach=False and fused=True alike) -- the same update
+        rule, two implementations, two roundings. Replicas forked at step 1 and nothing checked
+        until replica_diff after training. One implementation everywhere, like an emulator that
+        must give one semantics on two backends. Homogeneous fleets keep the on-device optimizer.
+        Every `check_every` steps (default: the logging cadence) all ranks compare a hash of
+        their parameter bits and stop on any difference (DAISY_CPU_OPT=0/1 overrides the choice)."""
         model.to(self.device)
+        pin_fp_mode()                            # fit may run on a thread configure_cpu never saw
         broadcast_params(model)
+        params = list(model.parameters())
+        force = os.environ.get("DAISY_CPU_OPT")
+        cpu_opt = (force == "1") if force in ("0", "1") else \
+            (mixed_devices(self.plan) and self.device.type != "cpu")
+        if cpu_opt:
+            master = [torch.nn.Parameter(p.detach().to("cpu").clone()) for p in params]
+            opt_params = master
+        else:
+            master, opt_params = None, params
         if optimizer == "adam":
-            opt = torch.optim.Adam(model.parameters(), lr=lr)
+            opt = torch.optim.Adam(opt_params, lr=lr)
         else:
-            opt = torch.optim.SGD(model.parameters(), lr=lr)
+            opt = torch.optim.SGD(opt_params, lr=lr)
+        if self.is_master() and mixed_devices(self.plan):
+            print("  [fit] mixed devices %s: optimizer on CPU master copies on GPU ranks"
+                  % sorted(set(self.plan["devices"])), flush=True)
+        every = check_every or max(1, steps // 20)
         w, nb = self.plan["my_weight"], self.plan["my_local_batch"]
         for s in range(steps):
             X, y = task.sample(nb)
@@ -222,7 +314,17 @@ class DaisyCluster:
             loss.backward()
             capacity_weighted_allreduce_grads(model, w)
             check_finite_grads(model, s)
-            opt.step()
+            if master is not None:
+                with torch.no_grad():
+                    for mp, p in zip(master, params):
+                        mp.grad = p.grad.detach().to("cpu")
+                    opt.step()
+                    for mp, p in zip(master, params):
+                        p.data.copy_(mp.data.to(p.data.device))
+            else:
+                opt.step()
+            if s % every == 0 or s == steps - 1:
+                check_replicas(params, s)              # collective: every rank calls it
             if step_delay:
                 time.sleep(step_delay)
             if s % max(1, steps // 20) == 0 or s == steps - 1:

test_cluster_mixed.py

@@ -0,0 +1,98 @@
+"""Mixed CPU + GPU fleet: replicas must stay bit-identical (2026-10-14).
+
+Real 2-rank gloo run: rank 0 trains on CUDA, rank 1 on CPU, with Adam.
+  * MEASURED first: CUDA Adam and CPU Adam on identical gradients differ in ~2% of weights after
+    20 steps, so with the optimizer on each rank's own device the replicas fork at step 1.
+    With DAISY_CPU_OPT=0 (the old behaviour) the new per-interval replica check must STOP the run.
+  * Default: GPU ranks step the optimizer on CPU master copies, so every rank runs the same
+    implementation; the run completes and the final parameter bits agree exactly.
+Also: pin_fp_mode turns flush-to-zero off even if something switched it on.
+Needs a CUDA device; skips (exit 0, says so) without one.
+"""
+import os
+import sys
+import tempfile
+
+import torch
+import torch.distributed as dist
+import torch.multiprocessing as mp
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from daisychain import cluster as C          # noqa: E402
+
+FAILURES = []
+
+
+def ck(name, cond, detail=""):
+    print(("  ok    " if cond else "  FAIL  ") + name + ("" if cond else "  [%s]" % (detail,)))
+    if not cond:
+        FAILURES.append(name)
+
+
+class Task:
+    def __init__(self, rank):
+        self.g = torch.Generator().manual_seed(1000 + rank)
+
+    def sample(self, n):
+        X = torch.randn(n, 32, generator=self.g)
+        return X, X[:, :1] * 0.5 - X[:, 1:2]
+
+    def loss(self, model, X, y):
+        return torch.nn.functional.mse_loss(model(X), y)
+
+
+def worker(rank, world, init_file, out_dir, cpu_opt):
+    if cpu_opt is not None:
+        os.environ["DAISY_CPU_OPT"] = cpu_opt
+    dist.init_process_group("gloo", init_method="file://" + init_file, rank=rank, world_size=world)
+    cl = C.DaisyCluster.__new__(C.DaisyCluster)
+    cl.threads, cl.rank, cl.world = 2, rank, world
+    cl.device = torch.device("cuda" if rank == 0 else "cpu")
+    cl.plan = {"devices": ["cuda", "cpu"], "my_weight": 0.5, "my_local_batch": 64,
+               "weights": [0.5, 0.5], "local_batches": [64, 64], "total_cores": 2, "total_ram_gb": None}
+    torch.manual_seed(3)
+    model = torch.nn.Sequential(torch.nn.Linear(32, 64), torch.nn.ReLU(), torch.nn.Linear(64, 1))
+    err = None
+    try:
+        cl.fit(model, Task(rank), steps=40, lr=1e-3, optimizer="adam", check_every=5)
+    except RuntimeError as e:
+        err = str(e)
+    torch.save({"err": err, "hash": C.param_hash(list(model.parameters()))},
+               os.path.join(out_dir, "r%d.pt" % rank))
+    dist.destroy_process_group()
+
+
+def run(cpu_opt):
+    with tempfile.TemporaryDirectory() as td:
+        mp.spawn(worker, args=(2, os.path.join(td, "init"), td, cpu_opt), nprocs=2, join=True)
+        return [torch.load(os.path.join(td, "r%d.pt" % r)) for r in range(2)]
+
+
+def main():
+    if not C._cuda_usable():
+        print("SKIPPED: no CUDA device (this test needs one GPU rank and one CPU rank)")
+        return 0
+    print("old behaviour (DAISY_CPU_OPT=0): optimizer on each rank's own device")
+    old = run("0")
+    ck("the replica check stops the run on both ranks", all(r["err"] and "diverged" in r["err"] for r in old),
+       [r["err"] for r in old])
+    print("default: GPU ranks step on CPU master copies")
+    new = run(None)
+    ck("the run completes on both ranks", all(r["err"] is None for r in new), [r["err"] for r in new])
+    ck("final parameter bits are identical across the CUDA and CPU ranks", new[0]["hash"] == new[1]["hash"])
+
+    print("pin_fp_mode")
+    torch.set_flush_denormal(True)
+    C.pin_fp_mode()
+    ck("subnormals survive after pin_fp_mode (FTZ off)",
+       float(torch.tensor([1e-39], dtype=torch.float32) * 1.0) != 0.0)
+    print()
+    if FAILURES:
+        print("FAILED: %d" % len(FAILURES))
+        return 1
+    print("all checks passed")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

test_cluster_reductions.py

@@ -99,6 +99,19 @@ def main():
     ck("the weights were never touched",
        all(torch.equal(a, b) for a, b in zip(before, lin.parameters())))
 
+    print("plan_capacities -- never weigh matmuls/sec against core counts (2026-10-14)")
+    from daisychain.cluster import plan_capacities
+    meas = {"cores": 8, "capacity": 210.0, "capacity_unit": "matmul/s"}
+    fell = {"cores": 8, "capacity": 8.0, "capacity_unit": "cores"}      # its benchmark threw
+    caps, unit = plan_capacities([meas, dict(meas, capacity=190.0)])
+    ck("all measured: the measured capacities are used", caps == [210.0, 190.0] and unit == "matmul/s")
+    caps, unit = plan_capacities([meas, fell])
+    ck("one fallback: EVERY node weighs by cores, not 210 vs 8", caps == [8.0, 8.0] and "mixed" in unit, (caps, unit))
+    caps, unit = plan_capacities([meas, {"cores": 4, "capacity": 999.0, "capacity_unit": "user"}])
+    ck("a DAISY_CAPACITY override is taken as stated", caps == [210.0, 999.0], caps)
+    caps, unit = plan_capacities([{"cores": 8, "capacity": 8.0}, {"cores": 8, "capacity": 150.0}])
+    ck("old-format surveys (no unit) are classified, and mixed -> cores", caps == [8.0, 8.0], caps)
+
     print()
     if FAILURES:
         print("FAILED: %d" % len(FAILURES))

web/package.json

@@ -6,7 +6,7 @@
   "author": "Dean Byrne (Quazim0t0) / DaisyChainAI",
   "scripts": {
     "start": "node server.js",
-    "test": "node test_core.js && node test_verified.js && node test_ieee.js && node test_gates.js && node test_metamorphic.js && node test_corpus.js && node test_selfcorpus.js && node test_b2b.js && node test_optimizer.js && node test_transformer.js && node test_unit_backward.js && node test_nonfinite.js && node test_reductions.js"
+    "test": "node test_core.js && node test_verified.js && node test_ieee.js && node test_gates.js && node test_metamorphic.js && node test_corpus.js && node test_selfcorpus.js && node test_b2b.js && node test_optimizer.js && node test_transformer.js && node test_unit_backward.js && node test_nonfinite.js && node test_reductions.js && node test_fgemm_audit.js && node test_probe_unproven.js"
   },
   "dependencies": {
     "hyparquet": "^1.26.2",

web/public/app.js

@@ -268,15 +268,22 @@ function setupChannel(peerId, dc) {
     // the next step boundary instead of making them wait for the next run
     if (training && leaderId === null) { pendingJoins.add(peerId); log(`${nmeOf(peerId)} will be synced into the running training`); }
   };
-  dc.onclose = () => { chans.delete(peerId); updatePeers(); wake(); scheduleReconnect(peerId); };
+  dc.onclose = () => { chans.delete(peerId); dropFragments(peerId); updatePeers(); wake(); scheduleReconnect(peerId); };
   dc.onmessage = (e) => onGrad(peerId, e.data);
 }
 function cleanupPeer(id) {
   const pc = pcs.get(id); if (pc) pc.close();
   pcs.delete(id); chans.delete(id); pendingCand.delete(id); pendingJoins.delete(id);
-  for (const k of fragIn.keys()) if (k.startsWith(id + ":")) fragIn.delete(k);
+  dropFragments(id);
   wake();
 }
+// A message cut off by a closed channel is never completed (the sender does not
+// resume it), so its parts must go when the channel does. Before 2026-10-14 only
+// cleanupPeer dropped them, and only the reconnect INITIATOR calls that: the
+// other side kept up to a resume bundle's worth of dead parts per drop.
+function dropFragments(id) {
+  for (const k of [...fragIn.keys()]) if (k.startsWith(id + ":")) fragIn.delete(k);
+}
 
 // ---- checkpoints ------------------------------------------------------------
 // File layout (also the broadcast payload after the sentinel):
@@ -555,8 +562,13 @@ function onFragment(peerId, buf) {               // returns full message when co
   const key = peerId + ":" + id;
   let st = fragIn.get(key);
   if (!st) { st = { id, parts: [], got: 0, total }; fragIn.set(key, st); }
+  if (seq < 0 || seq >= st.total || total !== st.total) {     // malformed: drop the whole message, loudly
+    log(`dropped malformed message ${id} from ${nmeOf(peerId)} (fragment ${seq}/${total}, expected /${st.total})`);
+    fragIn.delete(key);
+    return null;
+  }
+  if (!st.parts[seq]) st.got++;                  // a repeated part must not count twice
   st.parts[seq] = new Uint8Array(buf, 16).slice(0);
-  st.got++;
   if (st.got < st.total) return null;
   fragIn.delete(key);
   let len = 0; for (const p of st.parts) len += p.length;
@@ -612,8 +624,12 @@ async function refreshProbe() {
     mathHash = Compute.mathProbe();
     log(`JS math library hash: ${mathHash.toString(16)} (peers must match; ECMA-262 does not require exp/log/cos/sin to be correctly rounded)`);
   }
+  // A failed probe must not leave the PREVIOUS hash in place: that hash would
+  // go out with every gradient as a proof of kernels this run never re-checked.
+  // 0 means "unproven", and the roster check halts on it (2026-10-14; the same
+  // shape as an emulator call that reported success without doing the work).
   try { probeHash = await Compute.kernelProbe(compute, L); }
-  catch (e) { log(`kernel probe failed to run: ${e.message}`); }
+  catch (e) { probeHash = 0; log(`kernel probe failed to run: ${e.message} — this device's kernels are UNPROVEN`); }
   return probeHash;
 }
 function hashWeights() {                        // FNV-1a over the flat param bytes
@@ -898,8 +914,16 @@ async function train(cfg, resume) {
       if (bad) { halted = `weights diverged from ${nmeOf(bad)} (detected at step ${s + 1})`; break; }
       // kernel disagreement: same seeded probe, same exact integer arithmetic
       // expected on every backend. This is the check the weight hash can't make.
-      const badK = roster.find(id => id !== myId && hs.has(id) && hs.get(id).phash &&
-                                     probeHash && hs.get(id).phash !== probeHash);
+      // phash 0 = that device's probe did not run. It used to make this
+      // comparison skip the peer (`phash && ...`), so a device whose kernels
+      // could not be proven trained on unchecked; now it stops the run.
+      const unproven = roster.find(id => hs.has(id) && !hs.get(id).phash);
+      if (unproven) {
+        halted = `${unproven === myId ? "this device's" : nmeOf(unproven) + "'s"} kernel probe failed to run ` +
+                 `at step ${s + 1} — its arithmetic is unproven, so its gradients cannot be trusted`;
+        break;
+      }
+      const badK = roster.find(id => id !== myId && hs.has(id) && hs.get(id).phash !== probeHash);
       if (badK) {
         halted = `kernel disagreement with ${nmeOf(badK)} at step ${s + 1} — their verified-unit probe ` +
                  `hashes ${hs.get(badK).phash} vs mine ${probeHash}. One of us is computing different ` +

web/public/transformer.js

@@ -182,7 +182,7 @@
       cfg: { ...cfg, layers, heads, hidden, vocab: vocabSize() },
       ctx: { L, bgemm: (engine && engine.bgemm) || null,
              att: (engine && engine.att) || null, fgemm: (engine && engine.fgemm) || null,
-             fgemm2: (engine && engine.fgemm2) || null,
+             fgemm2: (engine && engine.fgemm2) || null, fgemmFma: !!(engine && engine.fgemmFma),
              mlp: (engine && engine.mlp) || null,
              audit: audit || null, unitBackward: !!cfg.unitBackward },
       emb: add("emb", mk(vocabSize() * c, 0.08)),
@@ -425,6 +425,14 @@
         bmm(cache.dlogits, m.emb, BT, vocab, C),
       ]);
     }
+    // live-shape audit of the f32 GEMM pair (both legs; gated only at small shapes)
+    if ((m.ctx.fgemm2 || m.ctx.fgemm) && !units && m.ctx.audit && m.ctx.audit.due()) {
+      const bad = V.auditFgemm(cache.dlogits, cache.lnf.y, { m: vocab, k: BT, n: C, transA: true },
+                               g[gi.emb], m.ctx.fgemmFma, m.ctx.audit.cells) ||
+                  V.auditFgemm(cache.dlogits, m.emb, { m: BT, k: vocab, n: C },
+                               dlnfIn, m.ctx.fgemmFma, m.ctx.audit.cells);
+      if (bad) m.ctx.audit.fail(bad);
+    }
     let dx = lnBwd(dlnfIn, cache.lnf.y, cache.lnf.sig, BT, C);
     const scale = 1 / Math.sqrt(hd);
     // concat helper for fusing sibling GEMMs into one batched dispatch

web/public/verified_core.js

@@ -334,6 +334,40 @@
     return out;
   }
 
+  // ---- live audit for the split-K f32 GEMM -----------------------------------
+  // Every int8 kernel had a live-shape audit; the f32 backward GEMM -- whose
+  // output IS the embedding gradient -- had only its init gate at 7x4500x5 and
+  // 9x300x6, while training runs it at vocab x BT x C. Same remedy as auditTile:
+  // recompute a few cells (structural corners first) with the mirror's exact
+  // partition and rounding schedule, compare bit-for-bit (2026-10-14 review).
+  function fgemmCell(A, Bm, d, fma, row, col) {
+    const { m, k, n } = d, transA = !!d.transA;
+    const S = k > 4096 ? Math.min(16, Math.ceil(k / 2048)) : 1;
+    const ks = Math.ceil(k / S);
+    let acc = 0;
+    for (let z = 0; z < S; z++) {
+      const p0 = z * ks, p1 = Math.min(k, p0 + ks);
+      let s = 0;
+      for (let p = p0; p < p1; p++) {
+        const a = transA ? A[p * m + row] : A[row * k + p];
+        s = fma ? f32(s + a * Bm[p * n + col]) : f32(s + f32(a * Bm[p * n + col]));
+      }
+      acc = f32(acc + s);
+    }
+    return acc;
+  }
+  function auditFgemm(A, Bm, d, got, fma, nCells) {
+    const { m, n } = d, N = nCells || 8;
+    const edges = [[m - 1, n - 1], [0, n - 1], [m - 1, 0], [0, 0]];
+    for (let t = 0; t < N; t++) {
+      const [i, j] = t < edges.length ? edges[t] : [(Math.random() * m) | 0, (Math.random() * n) | 0];
+      const v = fgemmCell(A, Bm, d, fma, i, j);
+      if (bitDiff(got[i * n + j], v))
+        return `f32 GEMM audit failed at [${i},${j}] shape ${m}x${d.k}x${n}${d.transA ? " transA" : ""}: kernel ${got[i * n + j]} vs mirror ${v}`;
+    }
+    return null;
+  }
+
   // ---- live audits for the fused attention kernels ---------------------------
   // The attention kernels had exact INIT gates but nothing at live shapes —
   // the exact gap the GEMM audit exists to close, left open on the kernels with
@@ -538,7 +572,7 @@
   const api = { assertFinite, accBound, MAX_INT32_K, quantize, quantize2, quantizeRows, quantizeCols, quantizeHeadCols, lutMatmulJS, lutMatmul3JS, lutMatmul3,
                 bgemmJS, vgemmBlock, auditTile, epi, attScoresJS, attCtxJS, linearFwd, forward, backward, splitApply,
                 rowAbsMax, scalesFromAbsMax, quantizeRowsInv, vmlpBlock, bitDiff,
-                auditAttScores, auditAttCtx, fgemmMirror };
+                auditAttScores, auditAttCtx, fgemmMirror, auditFgemm };
   if (typeof module !== "undefined" && module.exports) { TC = require("./traincore.js"); module.exports = api; }
   else { TC = root.TrainCore; root.Verified = api; }
 })(typeof self !== "undefined" ? self : this);

web/public/webgpu.js

@@ -286,7 +286,11 @@
                   matmulInt8: (Xq, Wq, m, k, n, LL) => root.Verified.lutMatmulJS(Xq, Wq, m, k, n, LL) };
     if (!(root.navigator && navigator.gpu)) return cpu;
     try {
-      const adapter = await navigator.gpu.requestAdapter();
+      // Ask for the discrete GPU: the default adapter on a laptop is the iGPU
+      // (measured 2026-10-14: Intel gen-9 picked while an RTX sat idle). The
+      // browser may still hand back the iGPU; the gates decide what runs.
+      const adapter = (await navigator.gpu.requestAdapter({ powerPreference: "high-performance" })) ||
+                      (await navigator.gpu.requestAdapter());
       if (!adapter) return cpu;
       const device = await adapter.requestDevice();
       const info = adapter.info || {};
@@ -330,9 +334,27 @@
       // because head-strided addressing is where these kernels can go wrong.
       let att = { scores: (qq, kq, qs, ks, d) => gpuAttScores(device, d.acc ? scoresVPipe : scoresPipe, qq, kq, qs, ks, d),
                   ctx: (aq, vq, as, vs, d) => gpuAttCtx(device, d.acc ? ctxVPipe : ctxPipe, aq, vq, as, vs, d) };
+      // Two sweeps: the second runs on RECYCLED pool buffers left full of
+      // large-magnitude residue (same shapes => same buckets, LIFO free list),
+      // the dirty-buffer gate the bgemm and MLP kernels already had. Without it
+      // a kernel that skips writing some output cell would pass on fresh
+      // buffers and corrupt live steps (2026-10-14 review).
+      const ATT_SHAPES = [{ B: 2, T: 8, heads: 2, hd: 8 }, { B: 1, T: 32, heads: 2, hd: 16 },
+                          { B: 3, T: 7, heads: 3, hd: 5 }, { B: 2, T: 33, heads: 4, hd: 8 }];
       try {
-        for (const d0 of [{ B: 2, T: 8, heads: 2, hd: 8 }, { B: 1, T: 32, heads: 2, hd: 16 },
-                          { B: 3, T: 7, heads: 3, hd: 5 }, { B: 2, T: 33, heads: 4, hd: 8 }]) {
+        for (const dirty of [false, true]) {
+          if (dirty) for (const d0 of ATT_SHAPES) {
+            const nQ = d0.B * d0.T * d0.heads * d0.hd, nA = d0.B * d0.heads * d0.T * d0.T;
+            const big = (n) => new Float32Array(n).fill(1e4);
+            for (const acc of [false, true]) {
+              const dv = { ...d0, acc };
+              await att.scores(new Int8Array(nQ).fill(127), new Int8Array(nQ).fill(127),
+                               big(d0.B * d0.T * d0.heads), big(d0.B * d0.T * d0.heads), dv);
+              await att.ctx(new Int8Array(nA).fill(127), new Int8Array(nQ).fill(127),
+                            big(d0.B * d0.heads * d0.T), big(d0.B * d0.heads * d0.hd), dv);
+            }
+          }
+        for (const d0 of ATT_SHAPES) {
           const nQ = d0.B * d0.T * d0.heads * d0.hd;
           const qq = new Int8Array(nQ), kq = new Int8Array(nQ), vq = new Int8Array(nQ);
           for (let i = 0; i < nQ; i++) { qq[i] = (Math.random() * 256 - 128) | 0; kq[i] = (Math.random() * 256 - 128) | 0; vq[i] = (Math.random() * 256 - 128) | 0; }
@@ -353,7 +375,8 @@
           const refS = root.Verified.attScoresJS(qq, kq, qs, ks, d0, L);
           const refC = root.Verified.attCtxJS(aq, vq, as, vs, d0, L);
           for (let i = 0; i < refS.length; i++) if (bitDiff(hwS[i], refS[i])) throw new Error(`scores epilogue mismatch @${i}`);
-          for (let i = 0; i < refC.length; i++) if (bitDiff(hwC[i], refC[i])) throw new Error(`ctx epilogue mismatch @${i}`);
+          for (let i = 0; i < refC.length; i++) if (bitDiff(hwC[i], refC[i])) throw new Error(`ctx epilogue mismatch @${i}${dirty ? " (dirty pool)" : ""}`);
+        }
         }
       } catch (e) {
         console.warn("fused attention kernels failed verification — using CPU LUT mirrors:", e.message);
@@ -398,6 +421,39 @@
         const Bt = Float32Array.from({ length: mT * nT }, () => Math.random() - 0.5);
         const Bn = Float32Array.from({ length: kT * nT }, () => Math.random() - 0.5);
         const dT = { m: kT, k: mT, n: nT, transA: true }, dN = { m: mT, k: kT, n: nT };
+        // transA against the MIRROR, not just against fgemm: before 2026-10-14
+        // the transA index pattern was only ever compared with itself (fgemm2
+        // leg vs fgemm call, same shader), so a wrong A[p*m+row] passed. The
+        // second shape puts transA through split-K (k > 4096) as well.
+        const fma = fgemmFma;
+        for (const [mA, kA, nA] of [[kT, mT, nT], [6, 4500, 3]]) {
+          const At = Float32Array.from({ length: mA * kA }, () => Math.random() - 0.5);
+          const Bx = Float32Array.from({ length: kA * nA }, () => Math.random() - 0.5);
+          const dX = { m: mA, k: kA, n: nA, transA: true };
+          const got = await fgemm(At, Bx, dX), want = root.Verified.fgemmMirror(At, Bx, dX, fma);
+          for (let i = 0; i < want.length; i++) if (bitDiff(got[i], want[i])) throw new Error(`fgemm transA differs from the mirror @${i} (${mA}x${kA}x${nA})`);
+        }
+        // SUBNORMALS: WGSL permits an implementation to flush f32 subnormals to
+        // zero, and the JS mirror never flushes. The random inputs above are
+        // ~0.1 in magnitude and never produce one, so a flushing GPU passed this
+        // gate and then forked from non-flushing peers on tiny gradients
+        // (2026-10-14; the emulator's FTZ/DAZ lesson). Products here are ~1e-40.
+        {
+          const dS = { m: 3, k: 64, n: 2 };
+          const As = Float32Array.from({ length: dS.m * dS.k }, (_, i) => (i % 2 ? 1 : -1) * (1 + (i % 7)) * 1e-20);
+          const Bs = Float32Array.from({ length: dS.k * dS.n }, (_, i) => (1 + (i % 5)) * 1e-20);
+          const got = await fgemm(As, Bs, dS), want = root.Verified.fgemmMirror(As, Bs, dS, fma);
+          let sub = 0;
+          for (let i = 0; i < want.length; i++) if (want[i] !== 0 && Math.abs(want[i]) < 1.1754943508222875e-38) sub++;
+          if (!sub) throw new Error("subnormal gate case produced no subnormal (test is vacuous)");
+          for (let i = 0; i < want.length; i++) if (bitDiff(got[i], want[i]))
+            throw new Error(`fgemm flushes or rounds subnormals differently @${i}: ${got[i]} vs mirror ${want[i]}`);
+        }
+        // dirty re-gate: poison the partial/output buckets at the gate's shapes, then re-check
+        await fgemm(new Float32Array(m0 * k0).fill(1e4), new Float32Array(k0 * n0).fill(1e4), dG);
+        const hw2 = await fgemm(A, Bm, dG);
+        const mRef = fma ? mFma : mStep;
+        for (let i = 0; i < hw2.length; i++) if (bitDiff(hw2[i], mRef[i])) throw new Error(`fgemm differs on a recycled (dirty) pool @${i}`);
         const [f1, f2] = await fgemm2(A2, Bt, dT, Bn, dN);
         const [r1, r2] = [await fgemm(A2, Bt, dT), await fgemm(A2, Bn, dN)];
         for (let i = 0; i < r1.length; i++) if (bitDiff(f1[i], r1[i])) throw new Error("fgemm2 transA leg differs");
@@ -559,7 +615,7 @@
 
       const viaLUT = { backend: "webgpu", label: `${gpuName} (LUT shader · exact-gated)`,
                        matmulInt8: (Xq, Wq, m, k, n) => gpuMatmulLUT(device, lutPipe, lutBuf, Xq, Wq, m, k, n),
-                       bgemm: bgLut, att, fgemm, fgemm2, mlp: mlpLut };
+                       bgemm: bgLut, att, fgemm, fgemm2, fgemmFma, mlp: mlpLut };
 
       // DP4A pipeline — only if the WGSL feature exists AND its batched kernel
       // reproduces the verified units exactly across the shape sweep
@@ -587,7 +643,7 @@
       // worse than a fixed choice. DP4A ships: it wins the int8-backward mode
       // by ~12% and ties elsewhere.
       return { backend: "webgpu", label: `${gpuName} (DP4A int8 dot · exact-gated vs units)`,
-               bgemm: bg, att, fgemm, fgemm2, mlp: mlpDp4 };
+               bgemm: bg, att, fgemm, fgemm2, fgemmFma, mlp: mlpDp4 };
     } catch (e) { console.warn("WebGPU init failed, CPU fallback:", e); return cpu; }
   }
 

web/test_fgemm_audit.js

@@ -0,0 +1,46 @@
+// Live audit of the split-K f32 GEMM (2026-10-14 review).
+// auditFgemm recomputes single cells with the mirror's partition and rounding
+// schedule; it must (a) accept the mirror's own output at the LIVE shapes
+// (transA embedding gradient, split-K dlnf), and (b) reject a one-ulp error
+// in a corner cell and in an interior cell, under both rounding schedules.
+const V = require("./public/verified_core.js");
+
+let pass = 0, fail = 0;
+const t = (name, ok, detail) => {
+  if (ok) { pass++; console.log("  PASS  " + name); }
+  else { fail++; console.log(`  FAIL  ${name}  [${detail || ""}]`); }
+};
+const rnd = (n) => Float32Array.from({ length: n }, () => Math.random() - 0.5);
+const bumpUlp = (x) => { const f = new Float32Array([x]), u = new Uint32Array(f.buffer); u[0] += 1; return f[0]; };
+
+// live-like shapes, scaled down: transA leg (m=vocab, k=BT) and the split-K leg (k=vocab > 4096)
+const shapes = [{ m: 600, k: 64, n: 12, transA: true }, { m: 64, k: 5000, n: 12 }, { m: 3, k: 9000, n: 2, transA: true }];
+for (const fma of [false, true]) {
+  for (const d of shapes) {
+    const A = rnd(d.m * d.k), B = rnd(d.k * d.n);
+    const out = V.fgemmMirror(A, B, d, fma);
+    const tag = `${d.m}x${d.k}x${d.n}${d.transA ? " transA" : ""} fma=${fma}`;
+    t(`audit accepts the mirror (${tag})`, V.auditFgemm(A, B, d, out, fma, 16) === null);
+    const corner = out.slice(); corner[d.m * d.n - 1] = bumpUlp(corner[d.m * d.n - 1]);
+    t(`audit catches a 1-ulp error in the last cell (${tag})`, V.auditFgemm(A, B, d, corner, fma, 4) !== null);
+    // interior: the random cells must eventually find it; 0 corners, 400 random draws on a small grid
+    const small = { m: 5, k: d.k, n: 3, transA: d.transA };
+    const As = rnd(small.m * small.k), Bs = rnd(small.k * small.n);
+    const o2 = V.fgemmMirror(As, Bs, small, fma); o2[1 * 3 + 1] = bumpUlp(o2[1 * 3 + 1]);
+    t(`audit's random cells find an interior error (${tag})`, V.auditFgemm(As, Bs, small, o2, fma, 400) !== null);
+  }
+}
+// The schedule matters: on split-K data the two schedules differ somewhere, so auditing
+// with the wrong schedule must be able to fail -- otherwise the fma flag is untested.
+{
+  const d = { m: 4, k: 6000, n: 4 };
+  const A = Float32Array.from({ length: d.m * d.k }, () => (Math.random() - 0.5) * 3);
+  const B = Float32Array.from({ length: d.k * d.n }, () => (Math.random() - 0.5) * 3);
+  const step = V.fgemmMirror(A, B, d, false), fused = V.fgemmMirror(A, B, d, true);
+  let differ = false;
+  for (let i = 0; i < step.length; i++) if (step[i] !== fused[i]) { differ = true; break; }
+  t("the two rounding schedules differ on split-K data (fma flag is load-bearing)", differ);
+  t("auditing fused output with the two-rounding schedule fails", V.auditFgemm(A, B, d, fused, false, 200) !== null);
+}
+console.log(`test_fgemm_audit.js: ${pass} passed, ${fail} failed`);
+process.exit(fail ? 1 : 0);

web/test_probe_unproven.js

@@ -0,0 +1,36 @@
+// "0 = unproven" for the kernel probe hash (2026-10-14).
+// app.js now treats phash 0 as "this device's probe did not run" and halts on it
+// (the comparison used to SKIP such a peer). That is only sound if a probe that
+// DID run can never hash to 0: check the canonical probe on the CPU backend and
+// on a fake bgemm that returns the same result, and that a wrong kernel moves it.
+const fs = require("fs");
+const path = require("path");
+const V = require("./public/verified_core.js");
+
+const root = { Verified: V };
+new Function("self", fs.readFileSync(path.join(__dirname, "public", "webgpu.js"), "utf8"))(root);
+const mul = new Int16Array(fs.readFileSync(path.join(__dirname, "public", "mul_lut.bin")).buffer.slice(0));
+const L = { mul };
+
+let pass = 0, fail = 0;
+const t = (name, ok, detail) => {
+  if (ok) { pass++; console.log("  PASS  " + name); }
+  else { fail++; console.log(`  FAIL  ${name}  [${detail || ""}]`); }
+};
+
+(async () => {
+  const cpu = await root.Compute.kernelProbe(null, L);
+  t("the canonical probe on the CPU mirror is non-zero", cpu !== 0, cpu);
+  t("...and deterministic", (await root.Compute.kernelProbe(null, L)) === cpu);
+  const same = { bgemm: async (Xq, Wq, rs, cs, d) => V.bgemmJS(Xq, Wq, rs, cs, d, L) };
+  t("a backend computing the same values hashes the same", (await root.Compute.kernelProbe(same, L)) === cpu);
+  const off = { bgemm: async (Xq, Wq, rs, cs, d) => {
+    const o = V.bgemmJS(Xq, Wq, rs, cs, d, L); o[o.length - 1] += 1; return o; } };
+  const h = await root.Compute.kernelProbe(off, L);
+  t("a kernel wrong in one cell hashes differently (and non-zero)", h !== cpu && h !== 0, h);
+  const app = fs.readFileSync(path.join(__dirname, "public", "app.js"), "utf8");
+  t("app.js resets probeHash to 0 when the probe throws", /catch \(e\) \{ probeHash = 0;/.test(app));
+  t("app.js halts on a roster member with phash 0", app.includes("roster.find(id => hs.has(id) && !hs.get(id).phash)"));
+  console.log(`test_probe_unproven.js: ${pass} passed, ${fail} failed`);
+  process.exit(fail ? 1 : 0);
+})();