aceOfDiamonds commited on
Commit
c5615d7
·
1 Parent(s): 1cd2d0f

Quantize the whole decoder in the int8 variant

Browse files

The int8 build quantized only the tied logit head, leaving the rest of the
decoder in fp32. Quantizing every decode-side linear takes the file from
154 MB to 130 MB with a transcript identical to the fp32 decoder.

Encode is untouched and its output is bit-identical to the fp16 variant:
the cross-attention key and value projections are excluded, since the
encoder hoists them and they run in the Vulkan partition.

vulkan/whisper_tiny_vulkan_int8.pte CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:a9014d2ceb0c6a5bafe94086274556b11575d37840b2ae9d4d9aec31b90821de
3
- size 154213348
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1953ba74947ee23d5f076fa8a198011bd0d6213e5a5e615ee4696ebb418fe20a
3
+ size 129521380