| --- |
| library_name: kernels |
| license: apache-2.0 |
| tags: |
| - kernel |
| - webgpu |
| - wgsl |
| --- |
| # ai.onnx.QLinearMatMul |
|
|
| `ai.onnx` · standard ONNX operator · ONNX opset ≥ 21 |
|
|
| ## Description |
|
|
| Quantized matrix multiplication of tensors `a` and `b` with supported ONNX `MatMul` vector and batch-broadcasting shapes. Each operand and the output carry their own scale and zero point; the result is rounded to nearest even and saturated to the output type. This package requires single-element rank-1 quantization tensors for `a` and `y`. For `b`, it supports single-element rank-1 tensors and selected per-column layouts; per-row quantization of `a` is unsupported. |
|
|
| See the [ONNX `QLinearMatMul` spec](https://onnx.ai/onnx/operators/onnx__QLinearMatMul.html) for the reference semantics. |
|
|
| ## Inputs |
|
|
| | Name | Logical dtype | Rank | Shape | Description | Presence | |
| | --- | --- | --- | --- | --- | --- | |
| | `a` | `TA` | — | — | N-dimensional quantized matrix a. | required | |
| | `a_scale` | `TF` | `1` | — | Per-tensor scale of quantized input `a`, supplied as a single-element rank-1 tensor. | required | |
| | `a_zero_point` | `TA` | `1` | — | Zero point of quantized input `a`; must have the same shape as `a_scale`. | required | |
| | `b` | `TB` | — | — | N-dimensional quantized matrix b. | required | |
| | `b_scale` | `TF` | — | — | Scale of quantized input `b`: a single-element rank-1 tensor for per-tensor quantization, a length-N vector for a rank-2 matrix, or `[batch, 1, N]` for a rank-3 matrix with per-column quantization. | required | |
| | `b_zero_point` | `TB` | — | — | Zero point of quantized input `b`: a single-element rank-1 tensor, a length-N vector or `[1, N]` for a rank-2 matrix, or `[batch, 1, N]` for a rank-3 matrix. Rank-2 per-column scales and zero points may independently use their per-tensor forms. | required | |
| | `y_scale` | `TF` | `1` | — | Per-tensor scale of quantized output `y`, supplied as a single-element rank-1 tensor. | required | |
| | `y_zero_point` | `TY` | `1` | — | Per-tensor zero point of quantized output `y`, supplied as a single-element rank-1 tensor. | required | |
|
|
| ## Outputs |
|
|
| | Name | Logical dtype | Rank | Shape | Description | Presence | |
| | --- | --- | --- | --- | --- | --- | |
| | `y` | `TY` | derived | ONNX MatMul result of `a` and `b` | Quantized matrix multiply result of a * b. | required | |
|
|
| ## Type constraints |
|
|
| | Variable | Allowed dtypes | |
| | --- | --- | |
| | `TA` | `uint8`, `int8` | |
| | `TB` | `uint8`, `int8` | |
| | `TY` | `uint8`, `int8` | |
| | `TF` | `float32`, `float16` | |
|
|
| ## Implementation variants |
|
|
| One implementation is selected per call from the device capabilities, the request shapes and the dtypes; these notes say what each one covers. |
|
|
| - `sgmat_precast_scalar_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_scalar_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_scalar` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_scalar` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_column_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_column_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_column` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_column` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_column_scale_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_column_scale_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_column_scale` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_column_scale` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_column_zero_f16` — Prepare exact f16 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_column_zero_f16` — Prepare lossless f16 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_precast_column_zero` — Prepare exact f32 representations of adjusted byte integers once, then load them directly into f32-accumulating subgroup matrices. Row groups follow matrix geometry, output rows, invocation limits and workgroup storage; bounded partial sums retain exact modular int32 arithmetic. |
| - `portable_precast_column_zero` — Prepare lossless f32 representations of adjusted byte integers, then use vector workgroup GEMM with exact bounded f32 partials and modular int32 totals. Geometry follows output rows, invocation and storage limits; f32 storage requires enough row reuse to amortize preparation. |
| - `sgmat_exact_scalar` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits. |
| - `sgmat_exact_column` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits. |
| - `sgmat_exact_column_scale` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits. |
| - `sgmat_exact_column_zero` — Exact integer products through bounded f32 matrix partials, fused with scalar or per-column requantization. Tile height follows workgroup memory and invocation limits. |
| - `portable_accurate_dp4a_rank2_scalar_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank2_per_column_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank2_per_column_scale_scalar_zp` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank2_scalar_quantization_tail` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank2_per_column_scale_scalar_zp_tail` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank3_scalar_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank4_scalar_quantization` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank2_scalar_scale_per_column_zp` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
| - `portable_accurate_dp4a_rank2_scalar_scale_per_column_zp_tail` — Uses tiled integer accumulation and accurate requantization when a fused path is unavailable, including devices whose storage-binding limit cannot accommodate it. Native packed dot products are used when available. |
|
|
| ## Device requirements |
|
|
| Some implementation variants require `subgroup-matrix`, `shader-f16`, and `subgroups`. These are route-specific capabilities, not package-wide requirements; availability also depends on the request shape and dtype. |
|
|
| ## Files |
|
|
| - [`metadata.json`](build/webgpu/metadata.json) — kernel metadata (id, digests, per-variant templates, provenance) |
| - [`manifest.json`](build/webgpu/manifest.json) — the op contract (source of truth) |
| - [`test.json`](build/webgpu/test.json) — correctness cases |
| - [`bench.json`](build/webgpu/bench.json) — benchmark cases |
| - [`qlinear-matmul-accumulate.wgsl.jinja`](build/webgpu/qlinear-matmul-accumulate.wgsl.jinja) |
| - [`qlinear-matmul-requantize.wgsl.jinja`](build/webgpu/qlinear-matmul-requantize.wgsl.jinja) |
| - [`quant-dp4a-matmul.wgsl.jinja`](build/webgpu/quant-dp4a-matmul.wgsl.jinja) |
| - [`quant-exact-matrix.wgsl.jinja`](build/webgpu/quant-exact-matrix.wgsl.jinja) |
| - [`quant-exact-portable.wgsl.jinja`](build/webgpu/quant-exact-portable.wgsl.jinja) |
| - [`quant-exact-prepare.wgsl.jinja`](build/webgpu/quant-exact-prepare.wgsl.jinja) |
| - [`quant-matmul-accumulate-rank4.wgsl.jinja`](build/webgpu/quant-matmul-accumulate-rank4.wgsl.jinja) |
|
|
| ## Use with `@huggingface/kernels` |
|
|
| ```sh |
| npm install --save-exact @huggingface/kernels@0.0.1-preview.3 |
| ``` |
|
|
| Required output shapes and logical data types are inferred from the supplied inputs and attributes; result tensors are allocated automatically. |
|
|
| The `version: 1` option selects the published kernel contract; it is independent of any operator opset, contrib `since_version`, or model version. |
| It follows the `v1` branch as fixes land. To pin exact artifact bytes, pass a 40-character commit `revision` instead of `version`. |
|
|
| Replace each `*Data` placeholder with a typed array containing the corresponding input data. |
|
|
| ```js |
| import { getKernel } from "@huggingface/kernels"; |
| |
| const kernel = await getKernel("webgpu-kernels/ai.onnx.QLinearMatMul", { version: 1 }); |
| const { y } = await kernel({ |
| a: { data: aData, shape: [1, 1] }, |
| a_scale: { data: a_scaleData, shape: [1] }, |
| a_zero_point: { data: a_zero_pointData, shape: [1] }, |
| b: { data: bData, shape: [1, 2] }, |
| b_scale: { data: b_scaleData, shape: [1] }, |
| b_zero_point: { data: b_zero_pointData, shape: [1] }, |
| y_scale: { data: y_scaleData, shape: [1] }, |
| y_zero_point: { data: y_zero_pointData, shape: [1] }, |
| }); |
| ``` |
|
|
