Drop MLX: 5.13x/2.25x slower than Core ML fp16 with no size win

0cbd6b4 verified 2 months ago

1.97 kB

# MLX is not shipped for this model

The MLX artifacts for this model were removed deliberately. The exporter in
`export-scripts` is kept so the decision can be re-tested against a future
ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.

## Why

Measured on device (iPhone 16, ExecuTorch 1.4.1, 3 warmup runs + 15 timed runs,
median, identical input shapes allocated from each model's own schema):

| model | Core ML fp16 | MLX fp16 | MLX penalty |
|---|---|---|---|
| rfdetr-nano detector | **8.4 ms** | 43.1 ms | **5.13x slower** |
| rfdetr-nano segmentation | **31.4 ms** | 70.5 ms | **2.25x slower** |

Both backends delegate fully and return finite output; MLX is simply slower.

**There is no size argument either.** The MLX and Core ML builds are within 1 MB
of each other (detector 55.7 vs 54.7 MB, segmentation 62.0 vs 62.4 MB), so
unlike CLIP and distiluse, where MLX int8 ships precisely because it is half the
size of the Core ML build, there is nothing here to trade the latency against.

## Why RF-DETR is not a special case

MLX runs on the Metal GPU; Core ML lowers to the ANE, which is purpose-built for
this kind of network. MLX was measured behind Core ML on every convolutional
vision model tested, by 2.25x to 19.6x. Its real advantage is transformer
workloads, where weight-only quantization shrinks the artifact substantially:
CLIP vision is 97% `nn.Linear` and goes 351 MB to 98 MB, and distiluse goes
541 MB to 140 MB. RF-DETR is a hybrid conv/transformer model and lands on the
wrong side of that line.

## A correction to the earlier figure

An earlier note put the RF-DETR MLX penalty at about 33% (Core ML 51 ms vs MLX
68 ms). That comparison is superseded and should not be quoted: it was measured
on an iPhone SE 3rd gen, and against the Core ML **int8** build that has since
been replaced by fp16, which is roughly 4x faster. The table above remeasures
both backends at the same precision on the same device.

	# MLX is not shipped for this model

	The MLX artifacts for this model were removed deliberately. The exporter in
	`export-scripts` is kept so the decision can be re-tested against a future
	ExecuTorch/MLX release, but no `.pte` or `config.json` is published here.

	## Why

	Measured on device (iPhone 16, ExecuTorch 1.4.1, 3 warmup runs + 15 timed runs,
	median, identical input shapes allocated from each model's own schema):

	\| model \| Core ML fp16 \| MLX fp16 \| MLX penalty \|
	\|---\|---\|---\|---\|
	\| rfdetr-nano detector \| 8.4 ms \| 43.1 ms \| 5.13x slower \|
	\| rfdetr-nano segmentation \| 31.4 ms \| 70.5 ms \| 2.25x slower \|

	Both backends delegate fully and return finite output; MLX is simply slower.

	There is no size argument either. The MLX and Core ML builds are within 1 MB
	of each other (detector 55.7 vs 54.7 MB, segmentation 62.0 vs 62.4 MB), so
	unlike CLIP and distiluse, where MLX int8 ships precisely because it is half the
	size of the Core ML build, there is nothing here to trade the latency against.

	## Why RF-DETR is not a special case

	MLX runs on the Metal GPU; Core ML lowers to the ANE, which is purpose-built for
	this kind of network. MLX was measured behind Core ML on every convolutional
	vision model tested, by 2.25x to 19.6x. Its real advantage is transformer
	workloads, where weight-only quantization shrinks the artifact substantially:
	CLIP vision is 97% `nn.Linear` and goes 351 MB to 98 MB, and distiluse goes
	541 MB to 140 MB. RF-DETR is a hybrid conv/transformer model and lands on the
	wrong side of that line.

	## A correction to the earlier figure

	An earlier note put the RF-DETR MLX penalty at about 33% (Core ML 51 ms vs MLX
	68 ms). That comparison is superseded and should not be quoted: it was measured
	on an iPhone SE 3rd gen, and against the Core ML int8 build that has since
	been replaced by fp16, which is roughly 4x faster. The table above remeasures
	both backends at the same precision on the same device.