Model card: macOS 27.2 re-measurement, NagaYu links, hf download, Space not provided
Browse files
README.md
CHANGED
|
@@ -24,7 +24,7 @@ converted for on-device image captioning and VQA on Apple silicon.
|
|
| 24 |
> crash, so nothing tells you they are wrong. Use `.all` or `.cpuAndGPU`, and compile the
|
| 25 |
> package before loading it. Details below.
|
| 26 |
|
| 27 |
-
Captions an image in **~1.3 s
|
| 28 |
that drives these packages has **no Python and no third-party runtime dependencies**, and
|
| 29 |
its preprocessing is bit-exact with the HuggingFace reference β a libjpeg-compatible JPEG
|
| 30 |
decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified
|
|
@@ -52,8 +52,10 @@ Correctness needs the GPU in the mix. The two GPU-less settings **do not crash**
|
|
| 52 |
silently return wrong output, which is the harder failure to notice.
|
| 53 |
|
| 54 |
Separately, loading the `.mlpackage` **from Python** with `compute_units=ALL`
|
| 55 |
-
(`ct.models.MLModel(...)`) still crashes with SIGSEGV on this environment
|
| 56 |
-
|
|
|
|
|
|
|
| 57 |
|
| 58 |
Because "it loaded" is not evidence of correctness, the companion CLI defaults to
|
| 59 |
`--compute-units auto`: it probes each candidate in a **subprocess** (these failures are
|
|
@@ -63,9 +65,9 @@ matches `.cpuAndGPU`. The result is cached per OS build, costing about 0.7 s onc
|
|
| 63 |
## Companion repos
|
| 64 |
|
| 65 |
- **Verify with the dataset β [NagaYu/smolvlm-coreml-golden](https://huggingface.co/datasets/NagaYu/smolvlm-coreml-golden)** (~43 MB). It holds the golden token ids, pixel tensors, raw PIL decode and test image that `smolvlm --check --root .` compares against on every run. Download it next to these packages if you want the bit-exactness claims on this card proven on your machine rather than taken on faith.
|
| 66 |
-
- **Compare against the Space β
|
| 67 |
|
| 68 |
-
> **Arrived here after an Apple OS update?** Read [docs/os-upgrade.md](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/os-upgrade.md) first β these packages still load (built with `minimum_deployment_target = iOS18`, specificationVersion 9 = iOS18/macOS15), but the goldens
|
| 69 |
|
| 70 |
## Files
|
| 71 |
|
|
@@ -91,18 +93,18 @@ The Swift CLI, the conversion scripts, and the bit-exactness verification all li
|
|
| 91 |
companion repository: **[NagaYu/smolvlm-coreml](https://github.com/NagaYu/smolvlm-coreml)**.
|
| 92 |
|
| 93 |
The CLI resolves both the `.mlpackage` files **and** `exports/` from a single `--root`
|
| 94 |
-
directory, so download the packages **into the cloned repo**
|
|
|
|
| 95 |
|
| 96 |
```bash
|
| 97 |
git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml
|
| 98 |
-
pip install torch transformers coremltools pillow
|
| 99 |
-
python export_artifacts.py # tokenizer, embeddings, golden data
|
| 100 |
|
| 101 |
-
# download only the minimal fp16 set, into the repo root
|
| 102 |
-
|
| 103 |
--include "smolvlm_vision.mlpackage/*" \
|
| 104 |
-
|
| 105 |
-
|
|
|
|
| 106 |
|
| 107 |
cd SmolVLMSwift && swift build -c release && cd ..
|
| 108 |
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
|
|
@@ -112,25 +114,31 @@ cd SmolVLMSwift && swift build -c release && cd ..
|
|
| 112 |
```
|
| 113 |
|
| 114 |
Add `--include "*_1024*"` for the hi-res split path, or `--include "*_int8*"` for the
|
| 115 |
-
INT8 variants.
|
| 116 |
-
|
|
|
|
| 117 |
|
| 118 |
## Measured performance
|
| 119 |
|
| 120 |
-
Apple
|
| 121 |
-
(`--compute-units auto`) selects `.all` here; `.cpuAndGPU`
|
| 122 |
-
fallback
|
|
|
|
| 123 |
|
| 124 |
-
| Stage | `.all` (auto-selected) | `.cpuAndGPU` | `--split`, 13 tiles (P=1024, `.cpuAndGPU`) |
|
| 125 |
|---|---|---|---|
|
| 126 |
-
| Model load (warm) | 0.
|
| 127 |
-
| Vision | ~
|
| 128 |
-
| Prefill + first logits | ~
|
| 129 |
-
| Decode | ~
|
| 130 |
-
|
| 131 |
-
|
| 132 |
-
|
| 133 |
-
and
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
|
| 135 |
## Why the architecture looks like this
|
| 136 |
|
|
@@ -152,7 +160,10 @@ The full catalog of seven reproducible crashes, with the reasoning and repro ste
|
|
| 152 |
|
| 153 |
## Verified parity
|
| 154 |
|
| 155 |
-
The companion CLI's `--check` mode re-proves
|
|
|
|
|
|
|
|
|
|
| 156 |
|
| 157 |
| Component | Match level |
|
| 158 |
|---|---|
|
|
@@ -167,9 +178,11 @@ The non-split caption also matches HuggingFace `transformers` greedy output toke
|
|
| 167 |
|
| 168 |
## Limitations β please read
|
| 169 |
|
| 170 |
-
- **Environment-stamped.**
|
| 171 |
-
|
| 172 |
-
|
|
|
|
|
|
|
| 173 |
- **Greedy output drifts across OS sessions.** With bit-identical inputs and artifacts,
|
| 174 |
greedy decode produced different captions across OS sessions days apart, while remaining
|
| 175 |
fully deterministic within a session. Likely last-ulp changes in GPU kernels between
|
|
|
|
| 24 |
> crash, so nothing tells you they are wrong. Use `.all` or `.cpuAndGPU`, and compile the
|
| 25 |
> package before loading it. Details below.
|
| 26 |
|
| 27 |
+
Captions an image in **~2 s of model time on macOS 27.2** (~1.3 s on macOS 26.6.2), entirely on device. The companion Swift CLI
|
| 28 |
that drives these packages has **no Python and no third-party runtime dependencies**, and
|
| 29 |
its preprocessing is bit-exact with the HuggingFace reference β a libjpeg-compatible JPEG
|
| 30 |
decoder and a Pillow-exact LANCZOS resampler, both reimplemented in Swift and re-verified
|
|
|
|
| 52 |
silently return wrong output, which is the harder failure to notice.
|
| 53 |
|
| 54 |
Separately, loading the `.mlpackage` **from Python** with `compute_units=ALL`
|
| 55 |
+
(`ct.models.MLModel(...)`) still crashes with SIGSEGV on this environment β re-confirmed on
|
| 56 |
+
macOS 27.2 with the real `smolvlm_decode_hybrid.mlpackage`, where it dies during the load. Compile to
|
| 57 |
+
`.mlmodelc` first, or stay on `.cpuAndGPU` in Python. On macOS 27.2 the CLI's `auto` probe still
|
| 58 |
+
selects `.all`; `.cpuAndNeuralEngine` and `.cpuOnly` were not re-tested there.
|
| 59 |
|
| 60 |
Because "it loaded" is not evidence of correctness, the companion CLI defaults to
|
| 61 |
`--compute-units auto`: it probes each candidate in a **subprocess** (these failures are
|
|
|
|
| 65 |
## Companion repos
|
| 66 |
|
| 67 |
- **Verify with the dataset β [NagaYu/smolvlm-coreml-golden](https://huggingface.co/datasets/NagaYu/smolvlm-coreml-golden)** (~43 MB). It holds the golden token ids, pixel tensors, raw PIL decode and test image that `smolvlm --check --root .` compares against on every run. Download it next to these packages if you want the bit-exactness claims on this card proven on your machine rather than taken on faith.
|
| 68 |
+
- **Compare against the Space β `smolvlm-coreml-oracle` (currently not provided).** The Space is not published at the moment, so there is no link; its source is in [`space/`](https://github.com/NagaYu/smolvlm-coreml/tree/main/space) of the companion repo. It runs the original SmolVLM-256M in **PyTorch on CPU**; Hugging Face Spaces are Linux, so Core ML cannot execute there and this Space is deliberately *not* a demo of these packages. It is the OS-independent reference oracle β it emits the reference artifacts you diff your on-device Core ML run against, and its output does not move when your Mac's OS moves.
|
| 69 |
|
| 70 |
+
> **Arrived here after an Apple OS update?** Read [docs/os-upgrade.md](https://github.com/NagaYu/smolvlm-coreml/blob/main/docs/os-upgrade.md) first β these packages still load (built with `minimum_deployment_target = iOS18`, specificationVersion 9 = iOS18/macOS15), but the goldens and the compute-unit verdicts are stamped to macOS 26.5 / coremltools 9.0 (the timings below were re-measured on macOS 27.2), and that guide is the checklist for re-proving them.
|
| 71 |
|
| 72 |
## Files
|
| 73 |
|
|
|
|
| 93 |
companion repository: **[NagaYu/smolvlm-coreml](https://github.com/NagaYu/smolvlm-coreml)**.
|
| 94 |
|
| 95 |
The CLI resolves both the `.mlpackage` files **and** `exports/` from a single `--root`
|
| 96 |
+
directory, so download the packages and runtime assets **into the cloned repo**. This repo
|
| 97 |
+
mirrors the clone's layout, so `--local-dir .` puts every file where `--root .` looks for it:
|
| 98 |
|
| 99 |
```bash
|
| 100 |
git clone https://github.com/NagaYu/smolvlm-coreml && cd smolvlm-coreml
|
|
|
|
|
|
|
| 101 |
|
| 102 |
+
# download only the minimal fp16 set + runtime assets, into the repo root
|
| 103 |
+
hf download NagaYu/SmolVLM-256M-Instruct-CoreML --local-dir . \
|
| 104 |
--include "smolvlm_vision.mlpackage/*" \
|
| 105 |
+
--include "smolvlm_prefill_kv.mlpackage/*" \
|
| 106 |
+
--include "smolvlm_decode_hybrid.mlpackage/*" \
|
| 107 |
+
--include "exports/embeddings.bin" --include "exports/tok/*" --include "exports/specials.json"
|
| 108 |
|
| 109 |
cd SmolVLMSwift && swift build -c release && cd ..
|
| 110 |
./SmolVLMSwift/.build/release/smolvlm --image test_image.jpg --root .
|
|
|
|
| 114 |
```
|
| 115 |
|
| 116 |
Add `--include "*_1024*"` for the hi-res split path, or `--include "*_int8*"` for the
|
| 117 |
+
INT8 variants. The non-split goldens for `--check` are committed in the GitHub repo; the
|
| 118 |
+
split-path pixel golden is in the dataset repo. You can also regenerate every package
|
| 119 |
+
yourself with `python convert_all.py && python export_artifacts.py` instead of downloading them.
|
| 120 |
|
| 121 |
## Measured performance
|
| 122 |
|
| 123 |
+
Apple M2 MacBook Pro, macOS 27.2 (build 26B123), fp16, warm process, median of 10 runs after
|
| 124 |
+
2 warmups. The CLI default (`--compute-units auto`) still selects `.all` here; `.cpuAndGPU`
|
| 125 |
+
is shown as the conservative fallback. The `--split` column was not re-measured on 27.2 and
|
| 126 |
+
is the earlier macOS 26.6.2 measurement:
|
| 127 |
|
| 128 |
+
| Stage | `.all` (auto-selected) | `.cpuAndGPU` | `--split`, 13 tiles (P=1024, `.cpuAndGPU`) β 26.6.2 |
|
| 129 |
|---|---|---|---|
|
| 130 |
+
| Model load (warm) | 0.25 s | 0.2 s | 0.3 s |
|
| 131 |
+
| Vision | ~90 ms (1 tile) | ~168 ms | ~1.3 s (13 tiles) |
|
| 132 |
+
| Prefill + first logits | ~950 ms (81 tok) | ~960 ms | ~300 ms (876 tok) |
|
| 133 |
+
| Decode | ~41 tok/s | ~41 tok/s | ~34 tok/s |
|
| 134 |
+
| 40-token caption, model time | ~2.0 s | ~2.1 s | β |
|
| 135 |
+
|
| 136 |
+
On macOS 26.6.2 the same packages measured vision ~87 ms / ~167 ms, prefill + first logits
|
| 137 |
+
~200 ms / ~180 ms and decode ~43 / ~40 tok/s (`.all` / `.cpuAndGPU`), for a ~1.3 s caption.
|
| 138 |
+
The 27.2 prefill slowdown reproduces with both the original compile and a fresh 27.2 compile,
|
| 139 |
+
so it comes from the OS runtime, not the packages. Decode rate varies by a few tok/s between
|
| 140 |
+
sessions. The first run in a fresh process also pays a cold-start penalty on vision and
|
| 141 |
+
prefill; the numbers above are warm.
|
| 142 |
|
| 143 |
## Why the architecture looks like this
|
| 144 |
|
|
|
|
| 160 |
|
| 161 |
## Verified parity
|
| 162 |
|
| 163 |
+
The companion CLI's `--check` mode re-proves the tokenizer, JPEG decode and resize rows on
|
| 164 |
+
every run (re-confirmed on macOS 27.2 / Swift 6.4: 81/81, 876/876, 0/255, maxdiff 0.000000).
|
| 165 |
+
It does not run Core ML; the normalization, prefill and caption rows were measured against
|
| 166 |
+
the Python pipeline separately.
|
| 167 |
|
| 168 |
| Component | Match level |
|
| 169 |
|---|---|
|
|
|
|
| 178 |
|
| 179 |
## Limitations β please read
|
| 180 |
|
| 181 |
+
- **Environment-stamped.** The packages were built and the goldens exported on macOS 26.5 /
|
| 182 |
+
coremltools 9.0 / Apple silicon; the timings, the `--check` results and the Python `ALL`
|
| 183 |
+
crash were re-measured on macOS 27.2 (Apple M2, Swift 6.4) without rebuilding anything. A
|
| 184 |
+
later OS may fix the crashes described here β or quietly change the numerics or the
|
| 185 |
+
speed. All three have been observed.
|
| 186 |
- **Greedy output drifts across OS sessions.** With bit-identical inputs and artifacts,
|
| 187 |
greedy decode produced different captions across OS sessions days apart, while remaining
|
| 188 |
fully deterministic within a session. Likely last-ulp changes in GPU kernels between
|