WarpStateAI / ops /MATMUL_TUNING.md
zerdovzad's picture
Upload 139 files
c8ba9c6 verified
|
Raw History Blame Contribute Delete
20.9 kB

WarpState's matrix experiments select BF16 kernels by operation, exact matrix dimensions, and operand strides. They are opt-in through --matmul-plan in run_40b.py and ops/benchmark_training.py. The workstation preset selects compiled autograd with communication overlap. Matrix kernels remain explicit experiments; isolated GEMM improvements alone do not select them for production. The complete two-GPU comparisons and validation are recorded below.

The implementation is split across these source files:

File Responsibility
matmul_kernels.py BF16 ATen, tiled, persistent, and persistent TMA matrix launches; tile validation and shape/stride signatures.
matmul_epilogue.py Paired up/gate matrix multiplication with a SwiGLU epilogue; returns both packed projection and activation.
matmul_execution.py Strict plan loading, registered autograd, and selected core-linear/SwiGLU training adapters.
tune_matmul.py Disposable ordinary/fused FFN matrix tuning with warmed controls and generated candidate recipes.
activation_execution.py Existing BF16 SwiGLU derivative used by the fused epilogue's backward.

The adapter is applied before core compilation and DDP construction. Original FP32 parameters remain the autograd leaves. Their differentiable BF16 casts stay inside the visible forward graph; the adapter does not retain external casts across updates or write leaf gradients through hooks. Matrix operands, returned matrix gradients, projections, and activations are BF16. Custom dot products accumulate in FP32. Parameter-gradient accumulation, clipping, DDP reduction, FP32 master parameters, and AdamW state remain under the existing training framework. The token cursor, objective, optimizer schedule, and checkpoint keys retain their existing meanings. CPU and evaluation model execution use the original module paths.

Selected kernel launches use opaque registered custom operations with fake output metadata, surrounded by the adapter's registered autograd. This is necessary because installed PyTorch 2.14 compiled autograd cannot retrace AOT's functionalized Triton launch operator. Unselected ATen products remain visible to the compiler. The CUDA suite exercises this second backward capture as well as ordinary eager/AOT execution; a plain forward compilation test would miss it.

Each JSON plan has format warpstate-matmul-plan-v1, an optional description, and an entries array. For A[M,K] @ B[K,N], m, n, and k describe that operation; strides are in elements. They are essential: a transposed weight and a contiguous matrix with the same dimensions can require different kernels. The complete dispatch key is (kind, m, n, k, a_stride, b_stride). Missing entries use ATen. An empty plan selects no custom kernels. Duplicate JSON keys, duplicate dispatch entries, unknown fields, invalid tiles, and non-BF16 entries are rejected.

This is a candidate example, not a selected production recipe:

{
  "format": "warpstate-matmul-plan-v1",
  "description": "Candidate paired FFN epilogue; requires complete-model validation.",
  "entries": [
    {
      "kind": "swiglu",
      "name": "ffn_up_gate.fwd",
      "m": 8192,
      "n": 14336,
      "k": 2048,
      "a_stride": [2048, 1],
      "b_stride": [1, 2048],
      "dtype": "bfloat16",
      "algorithm": "tiled",
      "tile": {
        "block_m": 128,
        "block_n": 128,
        "block_k": 32,
        "num_warps": 8,
        "num_stages": 3
      }
    }
  ]
}

kind: "mm" selects an ordinary matrix product. Forward and backward products have independent entries. The FFN up/gate forward uses (M,N,K) = (8192,14336,2048) with strides [2048,1] and [1,2048]. Its weight-gradient product uses (14336,2048,8192) with strides [1,14336] and [2048,1], matching the original dY.T @ X orientation. A forward winner therefore does not select that weight-gradient kernel automatically.

kind: "swiglu" selects the fused forward. Here N=2H is the complete packed projection width, while block_n counts columns in each half. Backward uses the saved BF16 packed projection, the existing SwiGLU derivative, and independently selected mm entries for dX and dW.

Algorithm Execution Supported plan kinds
aten Calls the installed PyTorch matrix implementation with its existing process settings. mm, swiglu
tiled Launches one program for each output tile, with grouped row tiles for weight reuse. mm, swiglu
persistent Launches at most one program per SM; each program strides over several output tiles using ordinary loads. mm, swiglu
persistent_tma Persistent scheduling with Triton tensor descriptors and hardware TMA operand loads. mm

Persistent scheduling alone does not imply TMA or warp specialization. The TMA implementation uses an opaque registered custom-op boundary because the installed PyTorch 2.14 AOT path does not preserve the host descriptor metadata through a plain triton_op launch. It does not secretly substitute another algorithm. TMA operands need an inner stride of one and 16-byte-aligned rows and storage offsets. The training selector falls back to ATen for an incompatible layout; the explicit kernel launcher reports unsupported requests. The paired SwiGLU kernel does not currently support TMA.

Tiles specify (block_m, block_n, block_k, num_warps, num_stages). Block sizes must be powers of two from 16 through 256, with 4 or 8 warps and 1 through 4 pipeline stages. A conservative SM120 operand-pipeline estimate rejects ordinary tiles above (BM+BN)*BK*2*stages = 96 KiB and paired tiles above (BM+2*BN)*BK*2*stages = 96 KiB. This is an estimate, not a guarantee about compiler register/shared-memory allocation. The tuner records compilation failures without promoting those candidates.

The paired epilogue arranges adjacent logical columns as up/gate pairs, using their physical packed offsets j and j+H. One wider tensor-core dot reuses the input tile, then splits the accumulator in registers. Both FP32 projection halves are rounded to BF16 before the activation. The kernel stores that rounded packed projection for backward and also writes the BF16 activation. It saves a subsequent packed-projection read and activation launch; it does not remove the projection storage required by backward.

The eager_rounding argument additionally rounds the SiLU result to BF16 before multiplication when following eager execution. Compiled execution follows the existing compiler's fused SiLU/product arithmetic, while still rounding the GEMM projection first. No FP8, FP16, quantization, split-K atomics, or reduced-precision gradient communication is introduced. Exact equality on measured aligned shapes does not promise bitwise equality for every GEMM layout or compiler version. For example, the tested ragged K=193 cuBLAS reference used a different reduction result: its packed relative-L2 difference was 0.269%, while the custom result differed by 0.000394% from an FP32-dot reference. Tests separately check projection accuracy and that activation consumes the saved rounded BF16 projection.

On GPU 1 of the RTX PRO 6000 Blackwell Max-Q workstation, the final paired kernel was compared with torch.compile(fullgraph=True) around the complete BF16 mm -> split -> SiLU -> multiply forward, returning both packed and activation outputs. Inputs had shape [8192,2048] and transposed weights [2048,14336]. With seed 563, 400 warmup calls per variant, and 15 alternating-order batches of 40 CUDA-event-timed calls, the selected microbenchmark tile measured:

Forward implementation Median milliseconds
Compiled ATen matrix multiplication plus SwiGLU 1.950789
Paired single-dot epilogue, tiled (128,128,32,8,3) 1.905561

This is 2.37% greater primitive throughput, or approximately 45 microseconds less per call; paired batch gains ranged from 2.08% to 2.75%. Packed projections and activations matched the compiled reference bitwise for two production-size random input seeds. The final epilogue suite passed 33 tests, covering both schedules, eager/compiled rounding, ragged dimensions, and the default/best tiles. A subsequent CPU-only invocation passed 9 and skipped 24 CUDA fixtures.

Applying the measured 45-microsecond difference to 28 cores and eight microbatches gives about 10 milliseconds per update before interaction with backward, communication, compiler scheduling, and allocator behavior. That arithmetic is not a measured complete-training gain. The earlier two-dot implementation lost to the control; it is superseded by the single-dot source.

Generated evidence remains under work/validation/rtx-opt-20260912/epilogue/:

  • paired-swiglu-confirm.json: final steady-state comparison and second input seed.
  • paired-swiglu-single-dot.json: final single-dot implementation's tile/schedule screen.
  • paired-swiglu.json and paired-swiglu-more-tiles.json: superseded two-dot screens.

The ordinary GEMM screen tested three algorithms and five tiles for each FFN up/gate forward and weight-gradient shape: 30 candidates total. Every candidate matched the ATen BF16 output exactly. All forward candidates lost. The selected weight-gradient candidate used persistent TMA with tile (256,128,32,8,3): median candidate time 1.811495 ms and median paired throughput gain 3.39%. Its weakest of three paired trials gained 1.30%. The first trial's two controls drifted from 1.589741 to 1.765839 ms, so this short screen alone is insufficient to select a production configuration. The complete generated sweep and candidate recipe are in work/benchmark-matmul-tiles-20260912/.

To reproduce the ordinary FFN forward/weight-gradient tuning, reserve an idle GPU first. From the repository root, the following command uses physical GPU 0 and keeps outputs and compiler caches inside the workspace. The output directory must be a new work/benchmark-* directory; existing directories are rejected.

cd /workspace/WarpState/rtx6000_40b
source ops/training-env.sh
CUDA_VISIBLE_DEVICES=0 .server-venv/bin/python -B ops/tune_matmul.py \
  --kind mm --operations fwd dw --algorithms tiled persistent persistent_tma \
  --rows 8192 --seed 1337 --warmup 10 --burnin-seconds 1 \
  --repeats 30 --trials 3 --minimum-gain 0.01 \
  --maximum-relative-rms 0.003 \
  --output-dir work/benchmark-matmul-tuning-reproduction

To recheck the measured fused tile with the same dimensions, seed, and operand scale, use the fused mode on an idle GPU:

CUDA_VISIBLE_DEVICES=1 .server-venv/bin/python -B ops/tune_matmul.py \
  --kind swiglu --operations fwd --algorithms tiled \
  --tile 128,128,32,8,3 --rows 8192 --seed 563 --input-scale 0.1 \
  --warmup 400 --burnin-seconds 1 --repeats 40 --trials 15 \
  --minimum-gain 0.01 --maximum-relative-rms 0.003 \
  --output-dir work/benchmark-matmul-swiglu-reproduction

This uses the tuner's bracketed control/candidate/control protocol. The historical 15-batch report used alternating pairs, so compare the resulting measurements as independent checks rather than expecting identical timing samples. Fused mode compiles the complete matrix-plus-SwiGLU reference and validates both packed projection and activation separately. It supports only --operations fwd with tiled or persistent. Omit --tile to use that mode's bounded tile list, or repeat it to specify several candidates. --input-scale defaults to 1.0 for both operands; the recorded epilogue comparisons used 0.1.

--list-cases prints the requested dimensions and strides without using CUDA. The tuner generates deterministic BF16 random operands from the supplied seed; it does not open a model, checkpoint, or token database. In this disposable process it disables BF16 reduced-precision reduction and TF32 for the ATen reference. Every timed candidate has preceding and following warmed controls. Eligibility requires finite results, the configured relative RMS tolerance, and at least the configured gain in every trial. Compilation failures remain in results.json. The best eligible entry for each tested operation is written to candidate-policy.json; an empty recipe means no candidate passed selection. Results include source hashes, software/device metadata, input signatures, timing triples, per-output numerical error, and GPU telemetry. Generated reports and candidate recipes remain ignored under work/; a recipe selected after complete validation should be copied into tracked configs/ with its provenance.

Use the same checkpoint, input offsets, microbatch, loss/gradient modes, and warmup/measurement counts to compare a candidate with a control. For example, after reserving both workstation GPUs, this disposable command tests the generated recipe against the retained full checkpoint:

CUDA_VISIBLE_DEVICES=0,1 .server-venv/bin/torchrun --standalone --nproc-per-node=2 \
  ops/benchmark_training.py --checkpoint work/run/latest.pt \
  --data work/data/train.bin --micro-batch 4 --seq-len 2048 \
  --global-tokens-per-update 131072 --no-activation-checkpointing \
  --compile-cores --compile-mode default --loss-mode compiled-full \
  --bf16-residuals --gradient-mode compiled-autograd-overlap \
  --optimizer-state resume --warmup-updates 8 --measured-updates 20 \
  --matmul-plan work/benchmark-matmul-tuning-reproduction/candidate-policy.json \
  --output-dir work/benchmark-matmul-training-candidate

Run a matching control with --matmul-plan omitted and a different new output directory. Neither benchmark advances the production checkpoint. Run numerical backward/optimizer and checkpoint/resume tests before selecting a training recipe. The production runner accepts the same --matmul-plan PATH option alongside its existing resume arguments; it is not an instruction to start a new run or change the established global token batch. External core-weight cast caching and the reduce-overhead CUDA-graph mode are explicitly incompatible with this adapter. Standalone --activation-mode swiglu|rmsnorm|all kernels require eager gradient execution; their transparent backward launches have the same compiled-autograd limitation described above. Both CLIs reject that unsupported combination. Use --activation-mode baseline with compiled autograd; matrix-plan fused epilogues still work through their separate opaque boundaries. Combining a matrix plan and compiled autograd also requires activation checkpointing off. The installed compiler cannot trace the saved-tensor unpack hook when the runner wraps an already compiled core in an eager checkpoint. Recomputation captured inside AOT passes the small kernel tests, but does not establish support for that different runner nesting.

Plan loading happens once, before execution. It computes a canonical recipe SHA256 and implementation SHA256 values for matmul_execution.py and matmul_kernels.py; a fused plan also includes matmul_epilogue.py and activation_execution.py. The runner records this metadata in settings and the checkpoint training signature, and the benchmark records it with its source provenance. A changed recipe or implementation therefore changes the execution signature. Legacy checkpoints normalize to no matrix plan. Resuming with an execution change requires the existing explicit --reconfigure mechanism, which still protects the data, context, global token batch, and optimizer schedule. Reformatting JSON without changing its parsed canonical content leaves the recipe identity unchanged; changing a description or an entry changes it.

The complete-model comparisons resume the same step-40,081 checkpoint into disposable GPU model/AdamW copies. Each uses eight warmup updates and 20 timed updates, BF16 residuals, compiled full loss, the same 131,072-token global batch, and compiled autograd with delayed communication waits. They exclude compilation, evaluation and checkpoint pauses. The generated rank logs and summaries are under work/benchmark-opt-20260912/.

Configuration Timed tokens/s Change from control Reserved GiB/rank
Communication-overlap control (matmul-control) 31,022.34 — 66.98
Persistent TMA dW only (matmul-dw) 31,063.57 +0.13% 66.99
Fused forward + TMA dW, initial boundary (matmul-combined) 31,002.83 -0.06% 66.54
Fused forward + TMA dW, shared weight buffer (matmul-combined-native) 31,156.20 +0.43% 66.98

These first comparisons are effectively tied. Over all 28 updates, the initial combined candidate's maximum loss difference from the fresh control was 0.00008535, and the maximum relative gradient-norm difference was 0.124%. Small BF16 reduction/fusion differences do not imply training progress.

After the weight-layout correction, the candidate averaged 4.20693 seconds per update versus 4.22508 for the control. Its maximum loss difference was 0.00010061 and its maximum relative gradient-norm difference was below 0.114%; all observed losses and norms were finite. This single comparison shows a small +0.43% gain, which has not been confirmed across repeated timing windows. The matrix recipe remains opt-in. The resume launcher retains the repeatedly verified communication-overlap configuration from OPTIMIZATION_20260912.md, which measured a 1.53% improvement over eager gradients in the repeated comparison. These percentages come from separate experiments and must not be added into a claimed combined speedup. The generated numerical/timing comparison is work/validation/rtx-opt-20260912/matmul/full-model-comparison.json.

Generated-code inspection verified the selected forward and dW routes, with 224 uses each per rank/update inferred from the compiled graphs and loop counts. It also found an avoidable cost in the initial fused boundary: passing W.T into an opaque operation made Inductor store two identical BF16 weight buffers, while the control shared one between forward and backward. The final adapter passes native-layout BF16 W and forms its transpose inside the launch. The actual-size compiled core now shares one allocation and one cast with backward, removing 58,720,256 extra bytes of stores per core call, or 13,153,337,344 bytes per rank/update. This is generated-code evidence, not a measured bandwidth counter or a guaranteed wall-time saving. The TMA dW route already reuses its gradient buffer and required no corresponding change.

The tracked matmul_rtx6000_bf16_experimental.json contains both exact candidate shapes and tiles. The final native-weight boundary comparison uses this recipe directly. Static dispatch and buffer evidence is recorded in work/validation/rtx-opt-20260912/matmul/compiled-dispatch-evidence.json and dw-dispatch-evidence.json. Production remains paused; these experiments leave the step-40,081 checkpoint and archived weights unchanged.

The integration validation passed 208 CPU tests, with 83 CUDA fixtures skipped and three existing compiler warnings. Separate reserved-GPU runs passed the 33-test epilogue suite and 35 adapter tests, including tiled, persistent, and TMA forward/dX/dW, FP32 gradient accumulation and AdamW, ordinary AOT, and compiled autograd. The adapter suite includes two alignment regressions: shifted inputs use exact ATen fallback in eager mode and match a compiled ATen control when compiled. Inductor can realign an input before the opaque boundary, making TMA valid; the original caller's offset need not survive compilation.

A two-GPU runner test combined the fused persistent epilogue, persistent TMA weight gradients, and delayed communication waits. Continuous execution matched asynchronous save/stop/resume exactly for model weights, full FP32 AdamW state, both ranks' RNG state, losses, and token cursors. Its final partial update also covered a rank with entirely masked targets. The final implementation passed in 73.63 seconds; its log is work/validation/rtx-opt-20260912/pytest-matmul-resume-final.log. The CPU and alignment regression logs are pytest-matmul-final-cpu.log and pytest-offset-guard.log in the same validation directory. These checks prove continuation consistency within the selected execution mode, not bitwise equivalence to a different BF16 reduction/fusion strategy. The final CLI compatibility/signature checks passed 103 tests with eight GPU fixtures skipped (pytest-matmul-cli-final.log). The complete package manifest was verified after updating the runner's SHA256/byte count. After the final benchmark, the production checkpoint's SHA256 still matched the paused record.