GGUF kernels: dequantize + fused gemv over packed blocks

.gitattributes

@@ -42,3 +42,9 @@ build/torch28-cxx11-cu129-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filt
 build/torch29-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
 build/torch29-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
 build/torch29-cxx11-cu130-x86_64-linux/_gguf_kernels_20260805124229.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch211-cxx11-cu126-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch211-cxx11-cu128-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch211-cxx11-cu130-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch212-cxx11-cu126-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch212-cxx11-cu130-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so filter=lfs diff=lfs merge=lfs -text
+build/torch212-cxx11-cu132-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,4 @@
+build/
+ci-out/
+result
+__pycache__/

SKILL.md

@@ -0,0 +1,162 @@
+---
+name: build-kernels
+description: Build a CUDA/Metal kernel into a Hub `kernels` repo with kernel-builder, and load it from transformers. Covers the layout, the build.toml traps, running the builder without nix (podman), and what the Hub and the `kernels` client each require. Use when packaging a kernel, debugging a kernel-builder build, or when `get_kernel` fails to load one.
+---
+
+# Building kernels for the Hub
+
+Learned by packaging llama.cpp's GGUF kernels as `marcsun13/gguf-kernels`. The official guide covers
+the happy path; this is what actually bit me, in the order it bit me.
+
+## 1. Layout
+
+```
+my-kernel/
+├── build.toml              # the only build config you write
+├── flake.nix               # nix entry point
+├── my_kernel_cuda/*.cu     # one directory per backend
+├── torch-ext/
+│   ├── torch_binding.h     # the entry points, same signature for every backend
+│   ├── torch_binding.cpp   # schema + per-backend registration
+│   └── my_kernel/__init__.py   # the Python API (hyphens in the name become underscores)
+└── tests/test_my_kernel.py
+```
+
+**One schema, one implementation per backend.** Do not split the registration per backend — declare
+the ops once and pick the implementation with the macros the builder defines:
+
+```cpp
+TORCH_LIBRARY_EXPAND(TORCH_EXTENSION_NAME, ops) {
+  ops.def("my_op(Tensor x) -> Tensor");
+#if defined(CUDA_KERNEL) || defined(ROCM_KERNEL)
+  ops.impl("my_op", torch::kCUDA, &my_op);
+#elif defined(METAL_KERNEL)
+  ops.impl("my_op", torch::kMPS, &my_op);
+#endif
+}
+REGISTER_EXTENSION(TORCH_EXTENSION_NAME)
+```
+
+Adding a backend is then a `[kernel.*]` section and a directory. Nothing else moves, and callers
+dispatch on the tensor's device as usual.
+
+## 2. build.toml traps
+
+- **Every file must be listed in `src`, headers included.** The builder stages only what is listed,
+  so an unlisted `.cuh` fails as "No such file or directory" even with the right `include` dirs.
+- **`include` needs `torch-ext`** if a backend source includes `torch_binding.h`.
+- **A vendored `.cu` you `#include` must be renamed** (e.g. `mmvq.cu` → `mmvq-impl.cuh`). Anything
+  ending in `.cu` in `src` is compiled as its own translation unit. Watch for collisions when
+  renaming: upstream often ships both `foo.cu` and `foo.cuh`, and clobbering the header gives a
+  baffling "#include nested depth 200 exceeds maximum" — the file now includes itself.
+- **Undo torch's half/bf16 flags** for ggml-style code:
+  `-U__CUDA_NO_HALF_OPERATORS__`, `-U__CUDA_NO_BFLOAT16_CONVERSIONS__` and friends.
+- `cuda-capabilities` are intersected with what each variant's CUDA toolkit supports, so listing
+  `12.0` is harmless on an older toolkit — it is silently dropped.
+
+## 3. Which builder, and why it decides everything
+
+Use `github:huggingface/kernels` (the old `kernel-builder` repo is stale) with
+`lib.genKernelFlakeOutputs`. **Pin it to the same version as the `kernels` client you load with**,
+because the builder version decides two things at once:
+
+| | |
+| --- | --- |
+| which torch variants exist | `v0.15.2` → torch 2.11/2.12; `main` → 2.12/2.13 |
+| the `metadata.json` format | older builders omit `name`, and a current client refuses to load that |
+
+A mismatch surfaces as `Cannot parse metadata from ...: missing field 'name'`, or as a build with no
+variant your interpreter can import. Check what came out before debugging anything else:
+
+```bash
+ls build/ && head -6 build/*/metadata.json
+```
+
+The builder also wants **a git repo with a clean tree** — otherwise it refuses to build
+("non-reproducible"), and uncommitted changes leave the artifact id ending in `_dirty`.
+
+## 4. Running the builder without nix (podman)
+
+The docker image works under rootless podman with two adjustments:
+
+```bash
+export XDG_RUNTIME_DIR=/tmp/podman-run && mkdir -p $XDG_RUNTIME_DIR   # else podman won't start
+podman run --rm --privileged \
+  --mount type=bind,source=$PWD,target=/kernelcode -w /kernelcode \
+  --entrypoint bash ghcr.io/huggingface/kernel-builder:main \
+  -c "nix run .#build-and-copy -L --max-jobs 4 --cores 16"
+```
+
+**`--privileged` is required.** Without it nix's unpack phase dies on
+`cp: setting permissions for 'source': Operation not permitted` — rootless podman lacks the
+capability for that chmod. `--option sandbox false` and `NIX_REMOTE=` do not help; don't bother.
+
+- `nix run .#build-and-copy` → every variant, into `build/`. Budget ~150 s per variant.
+- `nix build .#ci` → this system's variant only, into `result/`. Note **`nix run .#ci` fails**: it is
+  a package, not an app.
+
+## 5. Python side
+
+`_ops.py` is generated. Use relative imports only, and put the `torch.compile` fakes in the kernel —
+they belong with the op, not with the caller:
+
+```python
+from ._ops import add_op_namespace_prefix, ops
+
+def my_op(x): return ops.my_op(x)
+
+@torch.library.register_fake(add_op_namespace_prefix("my_op"))
+def _(x): return x.new_empty(...)
+```
+
+Publish the kernel's own limits as constants (max rows, supported type ids). Callers should read
+capabilities from the kernel instead of hardcoding them — a second backend will differ.
+
+Test against a built variant by putting it on the path, and assert the graph does not break:
+
+```bash
+PYTHONPATH=build/torch211-cxx11-cu128-x86_64-linux pytest tests/ -q
+```
+
+```python
+compiled = torch.compile(lambda x: my_op(x), fullgraph=True)   # fails loudly if the fake is wrong
+```
+
+Prefer a reference implementation that is already installed over writing one. For GGUF,
+`gguf.quants.dequantize` unpacks every quant type and accepts *any* byte pattern, so random bytes
+make a complete test with no checkpoint and no quantizer — just mask the fp16 scale fields so a
+random pattern cannot produce inf/nan (clearing bit 6 of every odd byte does it).
+
+## 6. Publishing, and what the client demands
+
+Upload `build/` plus `README.md` (with `tags: [kernel]`), and keep the sources in the repo so it stays
+reproducible. Then, in increasing order of surprise:
+
+1. **Tag a version.** `get_kernel` refuses an unpinned repo: "A kernel version or revision must be
+   specified". Create a `v<major>.<minor>.<patch>` tag, load with `version=<major>`.
+2. **Trust.** `get_kernel` loads unattended only from a publisher the Hub marks as a trusted kernel
+   publisher (e.g. `kernels-community`). A personal repo needs `trust_remote_code=True`, i.e. the
+   caller opting into running downloaded code — so a library should not pass it silently.
+3. **Indexing.** The client resolves through `https://huggingface.co/api/kernels/<repo>/...`, which
+   401s for a repo the Hub has not indexed as a kernel. A correct repo can still be unfetchable;
+   `kernels-community` repos work, so compare against one before suspecting your own build.
+
+Test without the Hub at all:
+
+```bash
+LOCAL_KERNELS="user/my-kernel=/path/to/my-kernel" python -c \
+  "from kernels import get_kernel; print(get_kernel('user/my-kernel', version=1))"
+```
+
+## 7. Loading from transformers
+
+Resolve once, keep the ops in module-level globals, call them as plain functions:
+
+```python
+module = get_kernel(repo, version=1)
+_MY_OP = module.my_op          # NOT self.kernel.my_op
+```
+
+Reaching an op through an attribute on an object makes dynamo graph-break at every call, which costs
+more than the kernel saves — the same reason the fakes matter. Treat "no kernel" as a normal outcome:
+return `None`, log why, and fall back to the dense path.

build.toml

@@ -8,6 +8,7 @@
 [general]
 name = "gguf-kernels"
 version = 1
+license = "MIT"
 backends = ["cuda"]
 
 [torch]

build/torch210-cxx11-cu126-x86_64-linux/metadata.json

@@ -1,4 +0,0 @@
-{
-  "version": 1,
-  "python-depends": []
-}

build/torch210-cxx11-cu128-x86_64-linux/metadata.json

@@ -1,4 +0,0 @@
-{
-  "version": 1,
-  "python-depends": []
-}

build/torch210-cxx11-cu130-x86_64-linux/metadata.json

@@ -1,4 +0,0 @@
-{
-  "version": 1,
-  "python-depends": []
-}

build/{torch210-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so → torch211-cxx11-cu126-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so}

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:deef7081403683a708ba874d97321243281a1b6e9e1f23ae06c930f06459f3b1
-size 26692640
+oid sha256:4bc09a45aabd708b54f9ce83b5ec9627ed185826b64aeb2b5673f46cad4c508e
+size 24799632

build/{torch210-cxx11-cu128-x86_64-linux → torch211-cxx11-cu126-x86_64-linux}/_ops.py

@@ -1,9 +1,9 @@
 import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
+from . import _gguf_kernels_cuda_e4e2164
+ops = torch.ops._gguf_kernels_cuda_e4e2164
 
 def add_op_namespace_prefix(op_name: str):
     """
     Prefix op by namespace.
     """
-    return f"_gguf_kernels_20260805124229::{op_name}"
+    return f"_gguf_kernels_cuda_e4e2164::{op_name}"

build/{torch210-cxx11-cu128-x86_64-linux → torch211-cxx11-cu126-x86_64-linux}/gguf_kernels/__init__.py

@@ -1,10 +1,10 @@
 import ctypes
+import importlib.util
 import sys
-
-import importlib
 from pathlib import Path
 from types import ModuleType
 
+
 def _import_from_path(file_path: Path) -> ModuleType:
     # We cannot use the module name as-is, after adding it to `sys.modules`,
     # it would also be used for other imports. So, we make a module name that

build/torch211-cxx11-cu126-x86_64-linux/metadata.json

@@ -0,0 +1,17 @@
+{
+  "name": "gguf-kernels",
+  "id": "_gguf_kernels_cuda_e4e2164",
+  "version": 1,
+  "license": "MIT",
+  "python-depends": [],
+  "backend": {
+    "type": "cuda",
+    "archs": [
+      "7.5",
+      "8.0",
+      "8.6",
+      "8.9",
+      "9.0"
+    ]
+  }
+}

build/{torch210-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so → torch211-cxx11-cu128-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so}

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b6cba7d21629a2a2269bd06de30ace18b373266c7e74e75e11f093cc27eec066
-size 39064328
+oid sha256:b71394062f24b9601e17ee704b313494487bbe77b8d20f54adbfd52665d9f300
+size 37124624

build/{torch210-cxx11-cu126-x86_64-linux → torch211-cxx11-cu128-x86_64-linux}/_ops.py

@@ -1,9 +1,9 @@
 import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
+from . import _gguf_kernels_cuda_e4e2164
+ops = torch.ops._gguf_kernels_cuda_e4e2164
 
 def add_op_namespace_prefix(op_name: str):
     """
     Prefix op by namespace.
     """
-    return f"_gguf_kernels_20260805124229::{op_name}"
+    return f"_gguf_kernels_cuda_e4e2164::{op_name}"

build/{torch210-cxx11-cu130-x86_64-linux → torch211-cxx11-cu128-x86_64-linux}/gguf_kernels/__init__.py

@@ -1,10 +1,10 @@
 import ctypes
+import importlib.util
 import sys
-
-import importlib
 from pathlib import Path
 from types import ModuleType
 
+
 def _import_from_path(file_path: Path) -> ModuleType:
     # We cannot use the module name as-is, after adding it to `sys.modules`,
     # it would also be used for other imports. So, we make a module name that

build/torch211-cxx11-cu128-x86_64-linux/metadata.json

@@ -0,0 +1,19 @@
+{
+  "name": "gguf-kernels",
+  "id": "_gguf_kernels_cuda_e4e2164",
+  "version": 1,
+  "license": "MIT",
+  "python-depends": [],
+  "backend": {
+    "type": "cuda",
+    "archs": [
+      "10.0",
+      "12.0",
+      "7.5",
+      "8.0",
+      "8.6",
+      "8.9",
+      "9.0"
+    ]
+  }
+}

build/{torch210-cxx11-cu130-x86_64-linux/_gguf_kernels_20260805124229.abi3.so → torch211-cxx11-cu130-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so}

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4e935328add4b20b3761b3ab018967f2b61030491865b841c455e6a25702e4e
-size 39318224
+oid sha256:5b19aec224f81561100e3133d414393779bacfabf2c703040e64b95fe0b69cef
+size 37378504

build/{torch210-cxx11-cu130-x86_64-linux → torch211-cxx11-cu130-x86_64-linux}/_ops.py

@@ -1,9 +1,9 @@
 import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
+from . import _gguf_kernels_cuda_e4e2164
+ops = torch.ops._gguf_kernels_cuda_e4e2164
 
 def add_op_namespace_prefix(op_name: str):
     """
     Prefix op by namespace.
     """
-    return f"_gguf_kernels_20260805124229::{op_name}"
+    return f"_gguf_kernels_cuda_e4e2164::{op_name}"

build/{torch210-cxx11-cu126-x86_64-linux → torch211-cxx11-cu130-x86_64-linux}/gguf_kernels/__init__.py

@@ -1,10 +1,10 @@
 import ctypes
+import importlib.util
 import sys
-
-import importlib
 from pathlib import Path
 from types import ModuleType
 
+
 def _import_from_path(file_path: Path) -> ModuleType:
     # We cannot use the module name as-is, after adding it to `sys.modules`,
     # it would also be used for other imports. So, we make a module name that

build/torch211-cxx11-cu130-x86_64-linux/metadata.json

@@ -0,0 +1,19 @@
+{
+  "name": "gguf-kernels",
+  "id": "_gguf_kernels_cuda_e4e2164",
+  "version": 1,
+  "license": "MIT",
+  "python-depends": [],
+  "backend": {
+    "type": "cuda",
+    "archs": [
+      "10.0",
+      "12.0",
+      "7.5",
+      "8.0",
+      "8.6",
+      "8.9",
+      "9.0"
+    ]
+  }
+}

build/{torch28-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so → torch212-cxx11-cu126-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so}

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:6708d42e7390b3a6fe316852e3857716defd991e5861031c8d04a1e7b4e0260f
-size 26686512
+oid sha256:0176cb70a486d747de1e4b4aa785760580e7b4dc8afc9efc897621d5fd307639
+size 24805280

build/{torch28-cxx11-cu126-x86_64-linux → torch212-cxx11-cu126-x86_64-linux}/_ops.py

@@ -1,9 +1,9 @@
 import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
+from . import _gguf_kernels_cuda_e4e2164
+ops = torch.ops._gguf_kernels_cuda_e4e2164
 
 def add_op_namespace_prefix(op_name: str):
     """
     Prefix op by namespace.
     """
-    return f"_gguf_kernels_20260805124229::{op_name}"
+    return f"_gguf_kernels_cuda_e4e2164::{op_name}"

build/{torch28-cxx11-cu126-x86_64-linux → torch212-cxx11-cu126-x86_64-linux}/gguf_kernels/__init__.py

@@ -1,10 +1,10 @@
 import ctypes
+import importlib.util
 import sys
-
-import importlib
 from pathlib import Path
 from types import ModuleType
 
+
 def _import_from_path(file_path: Path) -> ModuleType:
     # We cannot use the module name as-is, after adding it to `sys.modules`,
     # it would also be used for other imports. So, we make a module name that

build/torch212-cxx11-cu126-x86_64-linux/metadata.json

@@ -0,0 +1,17 @@
+{
+  "name": "gguf-kernels",
+  "id": "_gguf_kernels_cuda_e4e2164",
+  "version": 1,
+  "license": "MIT",
+  "python-depends": [],
+  "backend": {
+    "type": "cuda",
+    "archs": [
+      "7.5",
+      "8.0",
+      "8.6",
+      "8.9",
+      "9.0"
+    ]
+  }
+}

build/torch212-cxx11-cu130-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4d7e366d102d2194c5df8ab2b96f2c741e4738798e4ce913c5f980eefe0f96d4
+size 37380240

build/torch212-cxx11-cu130-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_cuda_e4e2164
+ops = torch.ops._gguf_kernels_cuda_e4e2164
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_cuda_e4e2164::{op_name}"

build/torch212-cxx11-cu130-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch212-cxx11-cu130-x86_64-linux/metadata.json

@@ -0,0 +1,19 @@
+{
+  "name": "gguf-kernels",
+  "id": "_gguf_kernels_cuda_e4e2164",
+  "version": 1,
+  "license": "MIT",
+  "python-depends": [],
+  "backend": {
+    "type": "cuda",
+    "archs": [
+      "10.0",
+      "12.0",
+      "7.5",
+      "8.0",
+      "8.6",
+      "8.9",
+      "9.0"
+    ]
+  }
+}

build/torch212-cxx11-cu132-x86_64-linux/_gguf_kernels_cuda_e4e2164.abi3.so

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:998f0c3e2e8a77d5e7dbd43ed817a98d46f519e1bcd626cd58d6365d2666edb4
+size 37398744

build/torch212-cxx11-cu132-x86_64-linux/_ops.py

@@ -0,0 +1,9 @@
+import torch
+from . import _gguf_kernels_cuda_e4e2164
+ops = torch.ops._gguf_kernels_cuda_e4e2164
+
+def add_op_namespace_prefix(op_name: str):
+    """
+    Prefix op by namespace.
+    """
+    return f"_gguf_kernels_cuda_e4e2164::{op_name}"

build/torch212-cxx11-cu132-x86_64-linux/gguf_kernels/__init__.py

@@ -0,0 +1,26 @@
+import ctypes
+import importlib.util
+import sys
+from pathlib import Path
+from types import ModuleType
+
+
+def _import_from_path(file_path: Path) -> ModuleType:
+    # We cannot use the module name as-is, after adding it to `sys.modules`,
+    # it would also be used for other imports. So, we make a module name that
+    # depends on the path for it to be unique using the hex-encoded hash of
+    # the path.
+    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
+    module_name = path_hash
+    spec = importlib.util.spec_from_file_location(module_name, file_path)
+    if spec is None:
+        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
+    module = importlib.util.module_from_spec(spec)
+    if module is None:
+        raise ImportError(f"Cannot load module {module_name} from spec")
+    sys.modules[module_name] = module
+    spec.loader.exec_module(module)  # type: ignore
+    return module
+
+
+globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch212-cxx11-cu132-x86_64-linux/metadata.json

@@ -0,0 +1,19 @@
+{
+  "name": "gguf-kernels",
+  "id": "_gguf_kernels_cuda_e4e2164",
+  "version": 1,
+  "license": "MIT",
+  "python-depends": [],
+  "backend": {
+    "type": "cuda",
+    "archs": [
+      "10.0",
+      "12.0",
+      "7.5",
+      "8.0",
+      "8.6",
+      "8.9",
+      "9.0"
+    ]
+  }
+}

build/torch28-cxx11-cu126-x86_64-linux/metadata.json

@@ -1,4 +0,0 @@
-{
-  "version": 1,
-  "python-depends": []
-}

build/torch28-cxx11-cu128-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:964ea6025655311cfebe2fb58eca589ace6d208222d793134a48925309ccf910
-size 39054016

build/torch28-cxx11-cu128-x86_64-linux/_ops.py

@@ -1,9 +0,0 @@
-import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
-
-def add_op_namespace_prefix(op_name: str):
-    """
-    Prefix op by namespace.
-    """
-    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch28-cxx11-cu128-x86_64-linux/gguf_kernels/__init__.py

@@ -1,26 +0,0 @@
-import ctypes
-import sys
-
-import importlib
-from pathlib import Path
-from types import ModuleType
-
-def _import_from_path(file_path: Path) -> ModuleType:
-    # We cannot use the module name as-is, after adding it to `sys.modules`,
-    # it would also be used for other imports. So, we make a module name that
-    # depends on the path for it to be unique using the hex-encoded hash of
-    # the path.
-    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
-    module_name = path_hash
-    spec = importlib.util.spec_from_file_location(module_name, file_path)
-    if spec is None:
-        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
-    module = importlib.util.module_from_spec(spec)
-    if module is None:
-        raise ImportError(f"Cannot load module {module_name} from spec")
-    sys.modules[module_name] = module
-    spec.loader.exec_module(module)  # type: ignore
-    return module
-
-
-globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch28-cxx11-cu128-x86_64-linux/metadata.json

@@ -1,4 +0,0 @@
-{
-  "version": 1,
-  "python-depends": []
-}

build/torch28-cxx11-cu129-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:51b5f28bd29d9373e0a9a5046e7314af5061350b340646ab98dfa391fd4f8670
-size 38993592

build/torch28-cxx11-cu129-x86_64-linux/_ops.py

@@ -1,9 +0,0 @@
-import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
-
-def add_op_namespace_prefix(op_name: str):
-    """
-    Prefix op by namespace.
-    """
-    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch28-cxx11-cu129-x86_64-linux/gguf_kernels/__init__.py

@@ -1,26 +0,0 @@
-import ctypes
-import sys
-
-import importlib
-from pathlib import Path
-from types import ModuleType
-
-def _import_from_path(file_path: Path) -> ModuleType:
-    # We cannot use the module name as-is, after adding it to `sys.modules`,
-    # it would also be used for other imports. So, we make a module name that
-    # depends on the path for it to be unique using the hex-encoded hash of
-    # the path.
-    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
-    module_name = path_hash
-    spec = importlib.util.spec_from_file_location(module_name, file_path)
-    if spec is None:
-        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
-    module = importlib.util.module_from_spec(spec)
-    if module is None:
-        raise ImportError(f"Cannot load module {module_name} from spec")
-    sys.modules[module_name] = module
-    spec.loader.exec_module(module)  # type: ignore
-    return module
-
-
-globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))

build/torch28-cxx11-cu129-x86_64-linux/metadata.json

@@ -1,4 +0,0 @@
-{
-  "version": 1,
-  "python-depends": []
-}

build/torch29-cxx11-cu126-x86_64-linux/__init__.py

@@ -1,55 +0,0 @@
-"""Compute directly on the packed blocks of a GGUF checkpoint.
-
-A GGUF weight is stored as blocks of 32 or 256 values sharing a scale. These kernels read that
-layout as it is, so a quantized model can be loaded and run without ever materializing a dense
-copy — which is the whole memory saving.
-
-Ported from llama.cpp's ggml-cuda; `vendor/UPSTREAM` pins the revision. The ops are named for what
-they do rather than for a backend: each backend registers its own implementation of the same
-schema, so calls dispatch on the tensor's device.
-"""
-
-import torch
-
-from ._ops import add_op_namespace_prefix, ops
-
-
-__all__ = ["GEMV_TYPES", "MAX_GEMV_ROWS", "dequantize", "mul_mat_vec"]
-
-# Upstream's MMVQ_MAX_BATCH_SIZE: `mul_mat_vec` has no implementation beyond this many rows, so a
-# caller with more (prefill) dequantizes and uses an ordinary matmul.
-MAX_GEMV_ROWS = 8
-
-# ggml type ids `mul_mat_vec` implements: Q4_0/Q4_1/Q5_0/Q5_1/Q8_0, the K quants, and the IQ quants.
-# `dequantize` covers more, so check this one before choosing the fused path.
-GEMV_TYPES = frozenset({2, 3, 6, 7, 8, 10, 11, 12, 13, 14, 16, 17, 18, 19, 20, 21, 22, 23, 29, 39, 40, 41, 42})
-
-
-def dequantize(
-    blocks: torch.Tensor, ggml_type: int, rows: int, cols: int, dtype: torch.dtype
-) -> torch.Tensor:
-    """`(rows, bytes_per_row)` uint8 blocks -> `(rows, cols)` values of `dtype`."""
-    return ops.dequantize(blocks, ggml_type, rows, cols, dtype)
-
-
-def mul_mat_vec(
-    blocks: torch.Tensor, x: torch.Tensor, ggml_type: int, out_features: int
-) -> torch.Tensor:
-    """Fused dequantize-gemv: `x @ blocks.T` without unpacking `blocks`.
-
-    `x` is `(rows, in_features)` and `rows` must be at most `MAX_GEMV_ROWS`. The result is f32
-    whatever `x`'s dtype was, since the kernel writes an f32 destination; cast it at the call site,
-    where the cast can be fused into whatever consumes it.
-    """
-    return ops.mul_mat_vec(blocks, x, ggml_type, out_features)
-
-
-# Without these, torch.compile cannot trace the ops and breaks the graph at every call.
-@torch.library.register_fake(add_op_namespace_prefix("dequantize"))
-def _dequantize_fake(blocks, ggml_type, rows, cols, dtype):
-    return blocks.new_empty((rows, cols), dtype=dtype)
-
-
-@torch.library.register_fake(add_op_namespace_prefix("mul_mat_vec"))
-def _mul_mat_vec_fake(blocks, x, ggml_type, out_features):
-    return x.new_empty((x.shape[0], out_features), dtype=torch.float32)

build/torch29-cxx11-cu126-x86_64-linux/_gguf_kernels_20260805124229.abi3.so

@@ -1,3 +0,0 @@
-version https://git-lfs.github.com/spec/v1
-oid sha256:d7435e6aeaab83f149782dd9d4dd926012315c0e9162b2a141dcbc961da18ea9
-size 26690648

build/torch29-cxx11-cu126-x86_64-linux/_ops.py

@@ -1,9 +0,0 @@
-import torch
-from . import _gguf_kernels_20260805124229
-ops = torch.ops._gguf_kernels_20260805124229
-
-def add_op_namespace_prefix(op_name: str):
-    """
-    Prefix op by namespace.
-    """
-    return f"_gguf_kernels_20260805124229::{op_name}"

build/torch29-cxx11-cu126-x86_64-linux/gguf_kernels/__init__.py

@@ -1,26 +0,0 @@
-import ctypes
-import sys
-
-import importlib
-from pathlib import Path
-from types import ModuleType
-
-def _import_from_path(file_path: Path) -> ModuleType:
-    # We cannot use the module name as-is, after adding it to `sys.modules`,
-    # it would also be used for other imports. So, we make a module name that
-    # depends on the path for it to be unique using the hex-encoded hash of
-    # the path.
-    path_hash = "{:x}".format(ctypes.c_size_t(hash(file_path.absolute())).value)
-    module_name = path_hash
-    spec = importlib.util.spec_from_file_location(module_name, file_path)
-    if spec is None:
-        raise ImportError(f"Cannot load spec for {module_name} from {file_path}")
-    module = importlib.util.module_from_spec(spec)
-    if module is None:
-        raise ImportError(f"Cannot load module {module_name} from spec")
-    sys.modules[module_name] = module
-    spec.loader.exec_module(module)  # type: ignore
-    return module
-
-
-globals().update(vars(_import_from_path(Path(__file__).parent.parent / "__init__.py")))