aceOfDiamonds commited on
Commit
49dcffa
·
1 Parent(s): 0b80b54

Replace the Vulkan fp32 variant with fp16

Browse files

fp16 measures 2.05x XNNPACK on an Adreno 840 against 1.30x for fp32, and 2.85x
on a Mali-G76 against 3.20x. fp32 is the faster arm on Mali, but fp16 wins on
both parts and halves the download, so it is the better single default.

fp16 requires the 30522-row token embedding to stay off texture storage; the
memory planner otherwise moves it onto a kernel that returns silently wrong
values above a 16384-row vocabulary, cosine 0.67 against the CPU reference.
With the embedding exempted it is 0.99996 on Adreno and 0.99999 on Mali.
See pytorch/executorch#22333.

Requires the ExecuTorch 1.4.1 Vulkan runtime from the v0.10.0-libs-1.4.1
artifacts.

.gitattributes CHANGED
@@ -39,3 +39,4 @@ xnnpack/all_minilm_l6_v2_xnnpack_fp32.pte filter=lfs diff=lfs merge=lfs -text
39
  coreml/all_minilm_l6_v2_coreml_fp16.pte filter=lfs diff=lfs merge=lfs -text
40
  coreml/all_minilm_l6_v2_coreml_fp32.pte filter=lfs diff=lfs merge=lfs -text
41
  vulkan/all_minilm_l6_v2_vulkan_fp32.pte filter=lfs diff=lfs merge=lfs -text
 
 
39
  coreml/all_minilm_l6_v2_coreml_fp16.pte filter=lfs diff=lfs merge=lfs -text
40
  coreml/all_minilm_l6_v2_coreml_fp32.pte filter=lfs diff=lfs merge=lfs -text
41
  vulkan/all_minilm_l6_v2_vulkan_fp32.pte filter=lfs diff=lfs merge=lfs -text
42
+ vulkan/all_minilm_l6_v2_vulkan_fp16.pte filter=lfs diff=lfs merge=lfs -text
vulkan/{all_minilm_l6_v2_vulkan_fp32.pte → all_minilm_l6_v2_vulkan_fp16.pte} RENAMED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:5885b4d342315c154466e2379575d9ea7683310081319356ddc0659a0075b826
3
- size 90343684
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:38744f91a7499d5bdf4ba3e9047a1b8a3189f85fcf041dde4cce4eb1783bf4fb
3
+ size 45213698
vulkan/config.json CHANGED
@@ -9,8 +9,8 @@
9
  "license": "apache-2.0",
10
  "variants": [
11
  {
12
- "file": "all_minilm_l6_v2_vulkan_fp32.pte",
13
- "precision": "fp32",
14
  "quantized": false,
15
  "default": true,
16
  "methods": {
 
9
  "license": "apache-2.0",
10
  "variants": [
11
  {
12
+ "file": "all_minilm_l6_v2_vulkan_fp16.pte",
13
+ "precision": "fp16",
14
  "quantized": false,
15
  "default": true,
16
  "methods": {