NagaYu commited on
Commit
9831eca
Β·
verified Β·
1 Parent(s): 47c51fa

Model card: macOS 27.2 re-measurement, NagaYu links, hf download, Space not provided

Browse files
Files changed (1) hide show
  1. README.md +43 -30
README.md CHANGED
@@ -24,7 +24,7 @@ converted for on-device image captioning and VQA on Apple silicon.
24
  > crash, so nothing tells you they are wrong. Use `.all` or `.cpuAndGPU`, and compile the
25
  > package before loading it. Details below.
26
 
27
- Captions an image in **~1.3 s end-to-end**, entirely on device. The companion Swift CLI
28
  that drives these packages has **no Python and no third-party runtime dependencies**, and
29
  its preprocessing is bit-exact with the HuggingFace reference β€” a libjpeg-compatible JPEG
30
  decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified
@@ -52,8 +52,10 @@ Correctness needs the GPU in the mix. The two GPU-less settings **do not crash**
52
  silently return wrong output, which is the harder failure to notice.
53
 
54
  Separately, loading the `.mlpackage` **from Python** with `compute_units=ALL`
55
- (`ct.models.MLModel(...)`) still crashes with SIGSEGV on this environment. Compile to
56
- `.mlmodelc` first, or stay on `.cpuAndGPU` in Python.
 
 
57
 
58
  Because "it loaded" is not evidence of correctness, the companion CLI defaults to
59
  `--compute-units auto`: it probes each candidate in a **subprocess** (these failures are
@@ -63,9 +65,9 @@ matches `.cpuAndGPU`. The result is cached per OS build, costing about 0.7 s onc
63
  ## Companion repos
64
 
65
  - **Verify with the dataset β€” [NagaYu/smolvlm-coreml-golden](https://huggingface.co/datasets/NagaYu/smolvlm-coreml-golden)** (~43 MB). It holds the golden token ids, pixel tensors, raw PIL decode and test image that `smolvlm --check --root .` compares against on every run. Download it next to these packages if you want the bit-exactness claims on this card proven on your machine rather than taken on faith.
66
- - **Compare against the Space β€” [YutaN13/smolvlm-coreml-oracle](https://huggingface.co/spaces/YutaN13/smolvlm-coreml-oracle)**. It runs the original SmolVLM-256M in **PyTorch on CPU**; Hugging Face Spaces are Linux, so Core ML cannot execute there and this Space is deliberately *not* a demo of these packages. It is the OS-independent reference oracle β€” it emits the reference artifacts you diff your on-device Core ML run against, and its output does not move when your Mac's OS moves.
67
 
68
- > **Arrived here after an Apple OS update?** Read [docs/os-upgrade.md](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/os-upgrade.md) first β€” these packages still load (built with `minimum_deployment_target = iOS18`, specificationVersion 9 = iOS18/macOS15), but the goldens, the timings, and the compute-unit verdicts are all stamped to macOS 26.5 / coremltools 9.0, and that guide is the checklist for re-proving them.
69
 
70
  ## Files
71
 
@@ -91,18 +93,18 @@ The Swift CLI, the conversion scripts, and the bit-exactness verification all li
91
  companion repository: **[NagaYu/smolvlm-coreml](https://github.com/NagaYu/smolvlm-coreml)**.
92
 
93
  The CLI resolves both the `.mlpackage` files **and** `exports/` from a single `--root`
94
- directory, so download the packages **into the cloned repo**:
 
95
 
96
  ```bash
97
  git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml
98
- pip install torch transformers coremltools pillow
99
- python export_artifacts.py # tokenizer, embeddings, golden data
100
 
101
- # download only the minimal fp16 set, into the repo root so --root . finds it
102
- huggingface-cli download NagaYu/SmolVLM-256M-Instruct-CoreML --local-dir . \
103
  --include "smolvlm_vision.mlpackage/*" \
104
- "smolvlm_prefill_kv.mlpackage/*" \
105
- "smolvlm_decode_hybrid.mlpackage/*"
 
106
 
107
  cd SmolVLMSwift && swift build -c release && cd ..
108
  ./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
@@ -112,25 +114,31 @@ cd SmolVLMSwift && swift build -c release && cd ..
112
  ```
113
 
114
  Add `--include "*_1024*"` for the hi-res split path, or `--include "*_int8*"` for the
115
- INT8 variants. You can also regenerate every package yourself with `python convert_all.py`
116
- instead of downloading them.
 
117
 
118
  ## Measured performance
119
 
120
- Apple silicon Mac, macOS 26.6.2, fp16, warm process. The CLI default
121
- (`--compute-units auto`) selects `.all` here; `.cpuAndGPU` is shown as the conservative
122
- fallback:
 
123
 
124
- | Stage | `.all` (auto-selected) | `.cpuAndGPU` | `--split`, 13 tiles (P=1024, `.cpuAndGPU`) |
125
  |---|---|---|---|
126
- | Model load (warm) | 0.3 s | 0.3 s | 0.3 s |
127
- | Vision | ~87 ms (1 tile) | ~167 ms | ~1.3 s (13 tiles) |
128
- | Prefill + first logits | ~200 ms (81 tok) | ~180 ms | ~300 ms (876 tok) |
129
- | Decode | ~43 tok/s | ~40 tok/s | ~34 tok/s |
130
-
131
- Decode rate varies by a few tok/s between sessions (40–43 and 34–37 observed across runs on
132
- the same machine). The first run in a fresh process also pays a cold-start penalty on vision
133
- and prefill; the numbers above are warm.
 
 
 
 
134
 
135
  ## Why the architecture looks like this
136
 
@@ -152,7 +160,10 @@ The full catalog of seven reproducible crashes, with the reasoning and repro ste
152
 
153
  ## Verified parity
154
 
155
- The companion CLI's `--check` mode re-proves all of this on every run:
 
 
 
156
 
157
  | Component | Match level |
158
  |---|---|
@@ -167,9 +178,11 @@ The non-split caption also matches HuggingFace `transformers` greedy output toke
167
 
168
  ## Limitations β€” please read
169
 
170
- - **Environment-stamped.** Everything above was measured on macOS 26.5 / coremltools 9.0 /
171
- Apple silicon. A later OS may fix the crashes described here β€” or quietly change the
172
- numerics. Both have been observed.
 
 
173
  - **Greedy output drifts across OS sessions.** With bit-identical inputs and artifacts,
174
  greedy decode produced different captions across OS sessions days apart, while remaining
175
  fully deterministic within a session. Likely last-ulp changes in GPU kernels between
 
24
  > crash, so nothing tells you they are wrong. Use `.all` or `.cpuAndGPU`, and compile the
25
  > package before loading it. Details below.
26
 
27
+ Captions an image in **~2 s of model time on macOS 27.2** (~1.3 s on macOS 26.6.2), entirely on device. The companion Swift CLI
28
  that drives these packages has **no Python and no third-party runtime dependencies**, and
29
  its preprocessing is bit-exact with the HuggingFace reference β€” a libjpeg-compatible JPEG
30
  decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified
 
52
  silently return wrong output, which is the harder failure to notice.
53
 
54
  Separately, loading the `.mlpackage` **from Python** with `compute_units=ALL`
55
+ (`ct.models.MLModel(...)`) still crashes with SIGSEGV on this environment β€” re-confirmed on
56
+ macOS 27.2 with the real `smolvlm_decode_hybrid.mlpackage`, where it dies during the load. Compile to
57
+ `.mlmodelc` first, or stay on `.cpuAndGPU` in Python. On macOS 27.2 the CLI's `auto` probe still
58
+ selects `.all`; `.cpuAndNeuralEngine` and `.cpuOnly` were not re-tested there.
59
 
60
  Because "it loaded" is not evidence of correctness, the companion CLI defaults to
61
  `--compute-units auto`: it probes each candidate in a **subprocess** (these failures are
 
65
  ## Companion repos
66
 
67
  - **Verify with the dataset β€” [NagaYu/smolvlm-coreml-golden](https://huggingface.co/datasets/NagaYu/smolvlm-coreml-golden)** (~43 MB). It holds the golden token ids, pixel tensors, raw PIL decode and test image that `smolvlm --check --root .` compares against on every run. Download it next to these packages if you want the bit-exactness claims on this card proven on your machine rather than taken on faith.
68
+ - **Compare against the Space β€” `smolvlm-coreml-oracle` (currently not provided).** The Space is not published at the moment, so there is no link; its source is in [`space/`](https://github.com/NagaYu/smolvlm-coreml/tree/main/space) of the companion repo. It runs the original SmolVLM-256M in **PyTorch on CPU**; Hugging Face Spaces are Linux, so Core ML cannot execute there and this Space is deliberately *not* a demo of these packages. It is the OS-independent reference oracle β€” it emits the reference artifacts you diff your on-device Core ML run against, and its output does not move when your Mac's OS moves.
69
 
70
+ > **Arrived here after an Apple OS update?** Read [docs/os-upgrade.md](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/os-upgrade.md) first β€” these packages still load (built with `minimum_deployment_target = iOS18`, specificationVersion 9 = iOS18/macOS15), but the goldens and the compute-unit verdicts are stamped to macOS 26.5 / coremltools 9.0 (the timings below were re-measured on macOS 27.2), and that guide is the checklist for re-proving them.
71
 
72
  ## Files
73
 
 
93
  companion repository: **[NagaYu/smolvlm-coreml](https://github.com/NagaYu/smolvlm-coreml)**.
94
 
95
  The CLI resolves both the `.mlpackage` files **and** `exports/` from a single `--root`
96
+ directory, so download the packages and runtime assets **into the cloned repo**. This repo
97
+ mirrors the clone's layout, so `--local-dir .` puts every file where `--root .` looks for it:
98
 
99
  ```bash
100
  git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml
 
 
101
 
102
+ # download only the minimal fp16 set + runtime assets, into the repo root
103
+ hf download NagaYu/SmolVLM-256M-Instruct-CoreML --local-dir . \
104
  --include "smolvlm_vision.mlpackage/*" \
105
+ --include "smolvlm_prefill_kv.mlpackage/*" \
106
+ --include "smolvlm_decode_hybrid.mlpackage/*" \
107
+ --include "exports/embeddings.bin" --include "exports/tok/*" --include "exports/specials.json"
108
 
109
  cd SmolVLMSwift && swift build -c release && cd ..
110
  ./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
 
114
  ```
115
 
116
  Add `--include "*_1024*"` for the hi-res split path, or `--include "*_int8*"` for the
117
+ INT8 variants. The non-split goldens for `--check` are committed in the GitHub repo; the
118
+ split-path pixel golden is in the dataset repo. You can also regenerate every package
119
+ yourself with `python convert_all.py && python export_artifacts.py` instead of downloading them.
120
 
121
  ## Measured performance
122
 
123
+ Apple M2 MacBook Pro, macOS 27.2 (build 26B123), fp16, warm process, median of 10 runs after
124
+ 2 warmups. The CLI default (`--compute-units auto`) still selects `.all` here; `.cpuAndGPU`
125
+ is shown as the conservative fallback. The `--split` column was not re-measured on 27.2 and
126
+ is the earlier macOS 26.6.2 measurement:
127
 
128
+ | Stage | `.all` (auto-selected) | `.cpuAndGPU` | `--split`, 13 tiles (P=1024, `.cpuAndGPU`) β€” 26.6.2 |
129
  |---|---|---|---|
130
+ | Model load (warm) | 0.25 s | 0.2 s | 0.3 s |
131
+ | Vision | ~90 ms (1 tile) | ~168 ms | ~1.3 s (13 tiles) |
132
+ | Prefill + first logits | ~950 ms (81 tok) | ~960 ms | ~300 ms (876 tok) |
133
+ | Decode | ~41 tok/s | ~41 tok/s | ~34 tok/s |
134
+ | 40-token caption, model time | ~2.0 s | ~2.1 s | β€” |
135
+
136
+ On macOS 26.6.2 the same packages measured vision ~87 ms / ~167 ms, prefill + first logits
137
+ ~200 ms / ~180 ms and decode ~43 / ~40 tok/s (`.all` / `.cpuAndGPU`), for a ~1.3 s caption.
138
+ The 27.2 prefill slowdown reproduces with both the original compile and a fresh 27.2 compile,
139
+ so it comes from the OS runtime, not the packages. Decode rate varies by a few tok/s between
140
+ sessions. The first run in a fresh process also pays a cold-start penalty on vision and
141
+ prefill; the numbers above are warm.
142
 
143
  ## Why the architecture looks like this
144
 
 
160
 
161
  ## Verified parity
162
 
163
+ The companion CLI's `--check` mode re-proves the tokenizer, JPEG decode and resize rows on
164
+ every run (re-confirmed on macOS 27.2 / Swift 6.4: 81/81, 876/876, 0/255, maxdiff 0.000000).
165
+ It does not run Core ML; the normalization, prefill and caption rows were measured against
166
+ the Python pipeline separately.
167
 
168
  | Component | Match level |
169
  |---|---|
 
178
 
179
  ## Limitations β€” please read
180
 
181
+ - **Environment-stamped.** The packages were built and the goldens exported on macOS 26.5 /
182
+ coremltools 9.0 / Apple silicon; the timings, the `--check` results and the Python `ALL`
183
+ crash were re-measured on macOS 27.2 (Apple M2, Swift 6.4) without rebuilding anything. A
184
+ later OS may fix the crashes described here β€” or quietly change the numerics or the
185
+ speed. All three have been observed.
186
  - **Greedy output drifts across OS sessions.** With bit-identical inputs and artifacts,
187
  greedy decode produced different captions across OS sessions days apart, while remaining
188
  fully deterministic within a session. Likely last-ulp changes in GPU kernels between