Kernels:
Trusted publisher
Download build/torch213-cpu-x86_64-linux/custom_ops.py from kernels-community/quantization-bitsandbytes: direct link, hf CLI and curl.
- Browser
- Download file 496 Bytes
-
https://huggingface.co/kernels/kernels-community/quantization-bitsandbytes/resolve/v1/build/torch213-cpu-x86_64-linux/custom_ops.py
- Command line
-
hf download hf://kernels-community/quantization-bitsandbytes@v1/build/torch213-cpu-x86_64-linux/custom_ops.py
-
curl -L -o custom_ops.py https://huggingface.co/kernels/kernels-community/quantization-bitsandbytes/resolve/v1/build/torch213-cpu-x86_64-linux/custom_ops.py
496 Bytes
| import torch | |
| from ._ops import ops | |
| def gemm_4bit_forward( | |
| input: torch.Tensor, | |
| weight: torch.Tensor, | |
| absmax: torch.Tensor, | |
| blocksize: int, | |
| quant_type: int, | |
| ) -> torch.Tensor: | |
| original_dtype = input.dtype | |
| if original_dtype != torch.bfloat16: | |
| input = input.to(torch.bfloat16) | |
| output = ops.gemm_4bit_forward(input, weight, absmax, blocksize, quant_type) | |
| if original_dtype != torch.bfloat16: | |
| output = output.to(original_dtype) | |
| return output | |