Hugging Face's logo

flashrt
/
fp8-gemm

Model card Files
xet
 Community
fp8-gemm / benchmarks /RESULTS.md
liangsu9988's picture
liangsu9988
Promote latest kernel artifacts to main
6a04ab4 verified
preview code
|
Raw History Blame Contribute Delete
7.64 kB

Benchmark Results: fp8-gemm

Validated locally on June 20, 2026.

Environment:

  • GPU: NVIDIA GeForce RTX 5090
  • PyTorch: 2.9.1+cu128
  • CUDA runtime reported by PyTorch: 12.8
  • Build target: sm_120a
  • Backend: source extension
  • Benchmark command:
python fp8-gemm/benchmarks/benchmark.py \
  --backend source --mode headline --warmup 20 --iterations 100 --compile-ref

Correctness gate:

python fp8-gemm/tests/test_fp8_gemm.py --backend source --mode full

Result: 8/8 rows passed. Metrics recorded: max absolute error, mean absolute error, p99 absolute error, cosine similarity, dtype, and tolerance. Public v1 SM120 scope is M=1 decode and 2 <= M <= 64 small-M rows. SM110 uses a separate full-row CUTLASS dispatcher described below.

Headline Rows

Shape Tile FlashRT us Torch eager us Torch compile us Speedup vs eager Speedup vs compile Max abs P99 abs Cosine
M=1,K=4096,N=2048 gemv_fp8_m1_w4 6.188 32.799 41.714 5.30x 6.74x 0.000 0.000 1.000000
M=1,K=4096,N=8192 gemv_fp8_m1_w8 10.290 162.342 156.012 15.78x 15.16x 0.000 0.000 1.000000
M=16,K=4096,N=4096 ld_fp8_gemm_16x128x256_w4 14.391 106.215 96.204 7.38x 6.68x 0.000 0.000 1.000000
M=32,K=4096,N=8192 ld_fp8_gemm_32x128x256_w4 22.581 200.997 189.331 8.90x 8.38x 0.000 0.000 1.000000
M=64,K=512,N=1024 ld_fp8_gemm_64x128x256_w4 8.259 18.085 50.002 2.19x 6.05x 0.000 0.000 1.000000

M=1 Variant Sweep

The dispatcher defaults to variant=0. Explicit variants are retained for benchmarking and tuning; public callers should use variant=0 unless they have measured their exact shape.

Shape Variant Tile FlashRT us Speedup vs eager Status
M=1,K=4096,N=2048 0 gemv_fp8_m1_w4 6.188 5.30x pass
M=1,K=4096,N=2048 4 gemv_fp8_m1_w4 6.186 5.30x pass
M=1,K=4096,N=2048 8 gemv_fp8_m1_w8 6.184 5.30x pass
M=1,K=4096,N=2048 16 gemv_fp8_m1_w16 6.188 5.30x pass
M=1,K=4096,N=8192 0 gemv_fp8_m1_w8 10.290 15.78x pass
M=1,K=4096,N=8192 4 gemv_fp8_m1_w4 10.274 15.81x pass
M=1,K=4096,N=8192 8 gemv_fp8_m1_w8 10.272 15.82x pass
M=1,K=4096,N=8192 16 gemv_fp8_m1_w16 10.278 15.80x pass

Block-128 Scaled GEMM

Measured on RTX 5090 against an independent binding of the original FlashRT pointer API. The wrapper and native columns execute the same production CUTLASS kernel from separate extension modules. PyTorch eager and compile dequantize the block-scaled tensors to FP32, run the GEMM, and cast to BF16. CUTLASS is already the native implementation, so there is no additional contract-equivalent library row.

Workload (M,K,N) Native us Wrapper us Wrapper/native Eager us Compile us Max abs P99 abs Cosine
decode (1,1024,1024) 10.287 10.276 0.999 36.256 42.242 0.000000 0.000000 1.0000001
action (51,1536,1536) 14.364 14.360 1.000 55.595 51.500 0.000061 0.000000 1.0000001
GROOT (277,2048,2048) 28.698 28.692 1.000 104.895 73.989 0.000122 0.000000 1.0000001
vision (1024,1152,1152) 18.456 18.466 1.001 101.103 79.565 0.000122 0.000000 1.0000000
video (2520,3072,3072) 114.188 114.696 1.004 1032.687 930.219 0.000244 0.000000 1.0000000
Qwen MLP (128,4096,12288) 51.258 51.253 1.000 892.744 390.698 0.000244 0.000000 1.0000000

All wrapper outputs were bitwise equal to the original native entry. The PyTorch-reference metrics above use production-scale ranges; the wider full correctness sweep remains the release gate.

Release Status

  • Source correctness: passed.
  • Source benchmark/tile sweep: passed for v1 public scope.
  • Existing SM120 installed artifacts: published.
  • SM110 local installed artifact: correctness, compile, graph, and parity passed.
  • SM110 Hub artifact: pending clean-commit rebuild and upload.

NVIDIA Thor SM110 Results

Validated August 2, 2026 on NVIDIA Thor with PyTorch 2.11.0+cu130, CUDA 13.0, CUTLASS 4.5.2, and a locally installed torch211-cxx11-cu130-aarch64-linux artifact. Timings are CUDA Graph replay latencies. Native is the independently loaded original FlashRT pointer API; the ratio is installed artifact / native, so values above 1 are slower.

Workload (M,K,N) Auto tile Artifact us Native us Artifact/native Correctness
decode (1,4096,2048) T1 17.232 17.344 0.997 pass
decode-wide (1,4096,8192) T1 50.144 50.112 1.001 pass
small-M (16,4096,4096) T1 23.952 24.304 0.997 pass
small-M (32,4096,8192) T1 46.256 46.000 1.013 pass
small-M (64,512,1024) T1 11.296 11.264 0.997 pass
PI0.5 QKV (51,2048,2560) T1 14.080 14.048 1.009 pass
PI0.5 O (51,2048,2048) T1 13.440 13.504 0.993 pass
PI0.5 gate/up (51,2048,16384) Wide 92.496 81.312 1.128 pass
PI0.5 down (51,8192,2048) T1 23.392 23.392 1.003 pass
GROOT DiT QKV (51,1536,4608) T1 15.232 15.104 1.004 pass
GROOT N1.7 O (277,2048,2048) Wide 18.704 18.704 1.002 pass
GROOT N1.7 gate/up (277,2048,16384) Wide 186.432 189.360 0.985 pass
GROOT N1.7 down (277,8192,2048) Sq 50.080 49.984 1.003 pass
GROOT vision O (1024,1024,1024) Sq 15.360 15.424 0.997 pass
Cosmos Edge action (64,2048,9216) T1 25.264 25.312 0.999 pass
LingBot vision O (1024,1280,1280) Wide 17.152 17.216 0.997 pass
LingBot action gate/up (105,2048,16384) Wide 140.080 139.984 1.002 pass

Each graph ratio is the median of paired, per-launch package/native samples; candidate order rotates every round to control Thor DVFS bias. Sixteen rows are within about 1.3% of the original FlashRT pointer extension. PI0.5 gate/up is a reproducible CUTLASS dependency-version outlier: the Hub-buildable package uses CUTLASS 4.5.2 while the original native extension uses 4.4.2, and the paired ratio in the final clean artifact run is 1.128. CUTLASS 4.0 was also tested but failed at runtime on SM110, so it is not a valid packaging fallback. This row is retained explicitly and is not used for a native-parity claim.

The installed artifact selected the fastest validated tactic on every row; worst auto/fastest-valid-tile was 1.0028. Source-to-artifact packaging parity passed with median 0.9986, p95 1.0195, and max 1.0244.

PI0.5 Thor Prefill and BF16 Bias Update

Source gate rerun August 8, 2026 on NVIDIA Thor, PyTorch 2.13.0+cu130:

  • correctness: 39/39, fail_count=0;
  • plain FP8 GEMM outputs were bitwise equal to the reference across the M=65..1024 band;
  • PI0.5 prefill QKV/O/gate-up/down auto dispatch was within 2% of the fastest valid Sq/T1/Wide package tile and the original FlashRT native entry;
  • all 11 BF16 bias, bias+residual, and bias+GELU checks passed;
  • bias-only output was exact; residual p99 was at most one BF16 step; tanh-GELU p99 was at most 3.8147e-6, with cosine at least 0.999995.

The down-projection bias family uses a CUTLASS Wide fused epilogue while other SigLIP shapes use the faster cuBLASLt path. The dispatcher preserves the public row-major (N,K) weight contract.