Parakeet-110M-MNN-INT8 / cuda-low-memory-fix.patch
sujitvasanth's picture
Add experimental fixed-length Parakeet 110M MNN INT8 models and CUDA fix
11f1ac3 verified
Raw History Blame Contribute Delete
567 Bytes
--- a/source/backend/cuda/execution/weight_only_quant/ConvFpAIntBExecution.cu
+++ b/source/backend/cuda/execution/weight_only_quant/ConvFpAIntBExecution.cu
@@ -1792,7 +1792,8 @@
DivModFast d_ow(ow);
DivModFast d_oh(oh);
- const int batch = inputs[0]->batch();
+ // Pointwise convolution has one GEMM row per spatial position, not only per batch item.
+ const int batch = mGemmInfo.elh[0];
// Weight dequantization now happens once in onResize (STATIC buffer).
// GEMV path (batch <= 4) reads packed INT4 weights directly.