msluszniak commited on
Commit
511fe51
·
verified ·
1 Parent(s): c99612f

perf(coreml): run decode on the CPU, not the Neural Engine

Browse files

decode is a single-token program whose cost on device is dispatch rather than arithmetic, so it was paying Neural Engine overhead on every call under ExecuTorch's ALL default. Pinning only decode to CPU_ONLY (encode stays on ALL) measures 6.7 ms/step against 20.0 on an iPhone 16.

On 14.5s of real speech the whole transcribe is 1.5-3.1x faster depending on size, with a byte-identical transcript: this removes overhead, it does not change numerics.

Exported from export-scripts MR !18.

coreml/whisper_tiny_en_coreml_fp16.pte CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:3f1a64779e891e77df585c8f5610996e8f3f383ee3d52a4eed10003348cadb56
3
- size 76476268
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99ce4f67ba8fb9d01ae08bd9a2e61f2be45caed6117d36a01368ec495d8c7f03
3
+ size 76477664