mlboydaisuke commited on
Commit
fdfc2f4
·
verified ·
1 Parent(s): e1de77d

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +20 -0
README.md CHANGED
@@ -37,6 +37,26 @@ portable option and are what runs on Android.
37
  \*Mac arm64, single process, median of 10 — a reference point for relative cost
38
  only, not a device number (torch eager fp32 on the same machine: 34.4 ms).
39
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
40
  ## Verification (executorch 1.4.0, torch 2.13.0)
41
 
42
  Parity is measured against the fp32 eager model on real image input; `corr` is
 
37
  \*Mac arm64, single process, median of 10 — a reference point for relative cost
38
  only, not a device number (torch eager fp32 on the same machine: 34.4 ms).
39
 
40
+ ### Checked in the task's own units
41
+
42
+ Correlation is a first filter, and on an embedding it is a weak one: a build can read
43
+ 0.999 against eager and still move an answer, because what decides is whether the error is
44
+ smaller than the gap between the answer a query gets and the runner-up.
45
+
46
+ The test is **near-duplicate retrieval** over 52 photographs, each also present centre-cropped to 90% and re-encoded as JPEG. Every item has one obviously
47
+ correct nearest neighbour, and each build has to find it.
48
+
49
+ | build | correct neighbour kept | worst score shift | budget spent |
50
+ |---|---|---|---|
51
+ | fp32 | 104 of 104 | 0.0015 | 5% |
52
+ | fp16 | 104 of 104 | 0.0015 | 5% |
53
+ | Core ML (fp16, iOS) | 104 of 104 | 0.0083 | 28% |
54
+
55
+ The closest decision eager makes on this set is **0.0298**, and the bar is half of
56
+ it. An earlier version of this card cleared int8 on "cosine similarity of the embeddings",
57
+ which is correlation wearing a task metric's clothes — it never asked whether a retrieval
58
+ answer moved.
59
+
60
  ## Verification (executorch 1.4.0, torch 2.13.0)
61
 
62
  Parity is measured against the fp32 eager model on real image input; `corr` is