The published build rejected seq=1 with "Size (1) of dimension (1) is not in
allowed range (2..254)", which surfaced in React Native ExecuTorch as
Error::DelegateInvalidHandle for any single-word input. See
software-mansion/react-native-executorch#1164.
torch.export keeps the declared min of 1; to_edge_transform_and_lower is what
widened the token dim to 2..254. Exporting under
torch.fx.experimental._config.backed_size_oblivious keeps 1 in the range.
schema.json already declared min 1, so the file now matches the contract it
publishes rather than the other way round.
Verified on macOS (ExecuTorch 1.4.1) against the eager model, cosine at every
length from 1 to the maximum:
seq=1 0.99997
seq=2 0.99998
seq=16 0.99998
seq=254 0.99995
and on an iPhone 16 in the demo app, where a single word now embeds instead of
failing.
The file name is unchanged, so nothing moves in the model registry.
quantized duplicated precisions.json, which is authoritative for the
quantized/non-quantized partition. default expressed a per-repository
recommendation nothing consumed: the library resolves its own per-platform
default. Both are gone from the schema.
Adds library_name to the model card, and pipeline_tag where it was missing.
Brings the config and card licenses into agreement, and repairs the variant
declarations that did not match the files actually published here.
Re-export the Vulkan fp16 variant with rank-3 attention
Attention now folds the singleton batch into the head dim and runs as
[H, T, D] batched matmuls. The stock 4-D reshapes each become their own
materialised view dispatch on Vulkan; this drops them and is bit-identical to
the previous export on CPU.
At the published 254-token shape: 10.30 -> 8.82 ms on an Adreno 840 and
107.58 -> 97.94 ms on a Mali-G76, which is 2.32x and 3.11x XNNPACK.
fp16 measures 2.05x XNNPACK on an Adreno 840 against 1.30x for fp32, and 2.85x
on a Mali-G76 against 3.20x. fp32 is the faster arm on Mali, but fp16 wins on
both parts and halves the download, so it is the better single default.
fp16 requires the 30522-row token embedding to stay off texture storage; the
memory planner otherwise moves it onto a kernel that returns silently wrong
values above a 16384-row vocabulary, cosine 0.67 against the CPU reference.
With the embedding exempted it is 0.99996 on Adreno and 0.99999 on Mali.
See pytorch/executorch#22333.
Requires the ExecuTorch 1.4.1 Vulkan runtime from the v0.10.0-libs-1.4.1
artifacts.
Android GPU build of all-MiniLM-L6-v2. 2.4x faster than the XNNPACK build on a Mali-G76 at the published 254 token shape (169.8 ms against 408.4 ms median), bit-identical across 10 executions, cosine 0.9999924 against the CPU reference.
Core ML: pin compute_unit=CPU_ONLY, drop the fp32 variant
These graphs fail the Apple Neural Engine path (coremltools#2825 masked mean-pool over a dynamic sequence dim; clip-text additionally ANECCompile FAILED). Core ML still returned correct output because it silently fell back to CPU, but the failed ANE compile was paid on every call. Pinning compute_unit=CPU_ONLY skips the attempt.
Measured on an iPhone 16, fp16, 3 warmup + 15 timed runs, median:
all-MiniLM 13.7 ms -> 2.5 ms 5.5x
multi-qa 14.1 ms -> 2.5 ms 5.6x
paraphrase-L12 26.9 ms -> 4.2 ms 6.4x
distiluse 33.0 ms -> 8.0 ms 4.1x
clip-text 18.1 ms -> 4.9 ms 3.7x
The fp32 variant is removed. Once fp16 is pinned it is dominated on both axes: slower (all-MiniLM 4.5 vs 2.5 ms, clip-text 9.5 vs 4.9 ms) and twice the size. Output fidelity against fp32 was gated first: relative L2 4e-6 (all-MiniLM, multi-qa), 0.0010 (paraphrase), 0.0046 (distiluse).
v0.9.0 and below are untouched.
