mlboydaisuke commited on
Commit
e8c3d5e
Β·
verified Β·
1 Parent(s): 1175a4e

README: the iPhone 17 Pro rows (201/201, 47 ms per decision, the Neural Engine preference)

Browse files
Files changed (1) hide show
  1. README.md +22 -6
README.md CHANGED
@@ -31,8 +31,8 @@ A **typed decision model** that answers in one forward pass: give it a state (a
31
  JSON record, a conversation) and a typed question β€” `choice` (pick one of named options), `score`
32
  (an ordered scale) or `noul` (yes/no) β€” and it returns a probability for every option, no generation.
33
  [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya), subfolder `multilingual/`
34
- (Apache-2.0, revision `1c5edc17…`, laya 0.3.4), as a static `.aimodel` for macOS 27 and, compiled ahead
35
- of time, for the iPhone 17 Pro (not yet run on a device). It is the catalog's first **encoder-type
36
  decision model**: the other
37
  decision models here are language models read out after a prefill; this one scores every option at its
38
  own marker position in a single encoder call.
@@ -99,8 +99,24 @@ On the CPU, wfp16 and fp32 return bit-identical marker and act logits on all 402
99
  weights in fp16 β€” exact, the checkpoint is F16 β€” and computes in fp32. On this Mac's GPU both stay within
100
  1e-4 of the CPU run as well. **Use the GPU, and request it explicitly**: with the Neural Engine preference
101
  the Mac returned the GPU's results bit for bit for fp32, but for wfp16 it returned different results that
102
- changed from run to run (next section). **iPhone: not measured** β€” the `ios-h18p/` and `ios/` folders have
103
- not run on a device yet.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
104
 
105
  The publisher's own model on the SemIf `authored144` fixture (144 three-option evidence / rule /
106
  candidate questions it was not trained for), as `laya.load(...).predict` answers them: mean family
@@ -217,8 +233,8 @@ at the final norm (0.13 at S=256 in torch), the state that normalizes those larg
217
 
218
  ## License and limits
219
 
220
- Apache-2.0 at the pinned upstream revision. Not tested: any iPhone (the device gate is pending), other
221
- Macs or OS builds, dynamic or batched shapes, windows other than 256 and 512, more than 20 options, the
222
  fitted calibration's quality at S = 512, languages beyond the fixture's English, Japanese and mixed rows,
223
  sustained thermals. The act probability is saturated at 1.0 on every fixture row, in the publisher's
224
  model as here: it is carried through the graph, not evidence of when to escalate.
 
31
  JSON record, a conversation) and a typed question β€” `choice` (pick one of named options), `score`
32
  (an ordered scale) or `noul` (yes/no) β€” and it returns a probability for every option, no generation.
33
  [`convaiinnovations/laya`](https://huggingface.co/convaiinnovations/laya), subfolder `multilingual/`
34
+ (Apache-2.0, revision `1c5edc17…`, laya 0.3.4), as a static `.aimodel` for macOS 27 and for the iPhone
35
+ 17 Pro (run there 2026-09-23: the same 201 rows pass, 47 ms per decision on the GPU). It is the catalog's first **encoder-type
36
  decision model**: the other
37
  decision models here are language models read out after a prefill; this one scores every option at its
38
  own marker position in a single encoder call.
 
99
  weights in fp16 β€” exact, the checkpoint is F16 β€” and computes in fp32. On this Mac's GPU both stay within
100
  1e-4 of the CPU run as well. **Use the GPU, and request it explicitly**: with the Neural Engine preference
101
  the Mac returned the GPU's results bit for bit for fp32, but for wfp16 it returned different results that
102
+ changed from run to run (next section). **iPhone 17 Pro** (iOS 27.0 24A437, the `ios/wfp16-s256` JIT bundle,
103
+ 2026-09-23, a headless harness over CoreAIKit's `EncoderDecider.decideRow` on the 201 fixture rows, the
104
+ fixture's own token ids, T = 1): on the GPU **201/201 rows within 1e-3, argmax 81/81, max |Ξ”p| 8.1e-6**,
105
+ act probability delta 0, no drift over three passes; CPU-only the same verdict at 9.4e-6. The kit's builder
106
+ renders every row identically on the phone (201/201 tokens and markers). One decision on the phone's GPU:
107
+ **53 ms median** (p90 57 ms, 30 warm-up calls, 603 timed calls) with the row's tokens given, 69–70 ms
108
+ through `TypedDecisions` with the state's tokens kept (the question tokenized on the phone, 5 ms median),
109
+ 73 ms CPU-only; the bundle loads in 1.2–2.5 s, the process peaks at 357 MB (670 MB CPU-only). With a Neural
110
+ Engine preference the phone misses the bar the way the Mac does β€” 196/201 within 1e-3, argmax 80/81, max
111
+ |Ξ”p| 0.33 on five rows β€” at 82 ms and 1.4 GB, so the kit keeps refusing that preference for this bundle.
112
+ Thermal state during those runs: "fair" for the GPU rows, "serious" for the CPU-only and Neural Engine
113
+ rows; the GPU gate run again at "nominal" (2026-09-24) gave the same verdict at 54.5 ms median (p90 59.6),
114
+ so the row cost is not thermal-bound. Through the kit's own `decide-cli bench --repeat 3` on the cool phone
115
+ (a 109-token state, eight questions, the measure of the Mac's 11.5 ms): **47.1 ms per decision** with the
116
+ state shared, 46.4 from scratch, 390 / 458 ms for the state and its eight. [`gate-laya-multilingual-iphone-gpu.json`](https://github.com/john-rocky/coreai-model-zoo/blob/main/models/laya-multilingual/gate-laya-multilingual-iphone-gpu.json)
117
+ in the zoo repository is the first GPU run's record. The AOT `ios-h18p/wfp16-s256` bundle, run the same way on the cool phone
118
+ (2026-09-24): the same verdict (201/201, argmax 81/81, max |Ξ”p| 8.1e-6) at **51.0 ms** median (p90 54.4), first call
119
+ 68.6 ms after a 1.7 s load, peak 357 MB β€” no faster to load or to run than the JIT bundle the catalog ships.
120
 
121
  The publisher's own model on the SemIf `authored144` fixture (144 three-option evidence / rule /
122
  candidate questions it was not trained for), as `laya.load(...).predict` answers them: mean family
 
233
 
234
  ## License and limits
235
 
236
+ Apache-2.0 at the pinned upstream revision. Not tested: the 512 window on the phone (the 256 window was, JIT
237
+ and AOT h18p, on one iPhone 17 Pro), other Macs or OS builds, dynamic or batched shapes, windows other than 256 and 512, more than 20 options, the
238
  fitted calibration's quality at S = 512, languages beyond the fixture's English, Japanese and mixed rows,
239
  sustained thermals. The act probability is saturated at 1.0 on every fixture row, in the publisher's
240
  model as here: it is carried through the graph, not evidence of when to escalate.