Neural Engine evaluation

Decision

Do not select the Apple Neural Engine for the current GTCRN Core AI assets. The graph does not successfully specialize and load on ANE, so it cannot meet the project's gate of equal-or-better latency with stable, full-model residency. Use explicit CPU specialization as the conservative default; GPU specialization is also numerically validated.

This is an implementation result, not a claim that GTCRN can never run on ANE. An ANE version would require re-authoring and revalidating the recurrent graph, not merely changing a compute-unit preference.

Gate

ANE is accepted only if all of the following are true:

The static float16 asset specializes and loads without warnings or a process abort.
The complete recurrent graph stays on ANE. Partial placement is insufficient for this 48.2K-parameter model because CPU/ANE transfers can dominate its already tiny compute cost.
Consecutive-frame output parity remains at least 40 dB PSNR against PyTorch.
Warm steady-state p50 and p95 inference latency are no worse than the fastest already-valid CPU/GPU specialization on the target device.

Environment and method

Apple M4 Max, 64 GB
macOS 27.0 (26A5406e)
Python 3.12.13
coreai-core==1.0.0b2
coreai-torch==0.4.1
DNS3 float16 streaming asset
64 warm-up frames followed by 512 timed consecutive frames
recurrent outputs fed directly into the next call
16 ms audio frame period
timing covers the Core AI function call and state outputs, but excludes model load, STFT, iSTFT, and audio I/O

The benchmark is reproducible with:

uv run gtcrn-coreai-benchmark \
  exports/gtcrn_dns3_float16_streaming.aimodel \
  --compute-unit cpu

Run accelerator experiments in separate processes. An explicit ANE attempt is:

uv run gtcrn-coreai-benchmark \
  exports/gtcrn_dns3_float16_streaming.aimodel \
  --compute-unit neural-engine

Be aware that the tested beta Core AI/macOS runtime aborts the Python process on this failure instead of returning a catchable Python exception.

Results

The successful CPU baseline measured:

Compute unit	Mean	p50	p95	p99	Maximum	Mean / 16 ms budget
CPU	0.232 ms	0.231 ms	0.240 ms	0.248 ms	0.499 ms	1.45%
ANE	No result	No result	No result	No result	No result	Load failed

The CPU result is a representative local run, not a cross-device guarantee. Existing parity measurements separately show that both CPU and GPU produce more than 56 dB minimum PSNR across the bundled checkpoints.

ANE/default specialization attempted to compile decomposed PReLU and GRU regions, emitted unsupported-pattern diagnostics, then failed to load the ANE program with Core AI error Code 53 (Program load failed - no memory). The runtime subsequently aborted. Therefore an ANE performance comparison is not defined: no valid ANE inference completed.

Enabling the OS runtime's private debug-residency metadata also proved unstable on this beta stack. It exposed GPU residency for sampled operations, but a later debug specialization trapped in CoreAIRuntime and left subsequent model-load attempts failing until the OS service is reset. The repository benchmark intentionally does not enable that debug mode.

What would be required for ANE

The main blockers are the decomposed recurrent and activation patterns. A real ANE port would likely replace the exported GRU/PReLU regions with an ANE-oriented, static-shape formulation (typically convolution/matrix primitives and explicit recurrence), then remap the original weights and repeat full-stream parity and audio-quality validation.

That rewrite is not selected here because:

changing only the compute-unit preference does not produce a loadable model;
partial placement would add transfer overhead to a CPU path already far below the real-time budget; and
a recurrent rewrite is a new model-authoring effort with meaningful numerical and audio-quality risk, not a safe packaging optimization.

Revisit this decision when the Core AI runtime/compiler changes or when a fully ANE-oriented recurrent implementation can be tested on every deployment device.