wegotomars/OmniVoice-CoreAI-ANE

OmniVoice on Apple Neural Engine

Core AI conversion of k2-fsa/OmniVoice for multilingual speech synthesis, voice cloning, and voice design, using tinkerbox-ane.

The FP16 Qwen3 diffusion transformer and audio projection run fully on the ANE. The 24 kHz acoustic codec decoder also runs fully on the ANE. Text processing, embeddings, scheduling, sampling, codec lookup/projection and reference encoding run on the CPU while your GPU stays idle and cool 😎❄️.

Requires macOS 27, Apple Silicon M4 or newer, and Python 3.12 or 3.13. M1 and M2 are not supported: their ANE compiler rejects the long-context attention. M3 is not supported because the codec exceeds its ANE tensor-width limit.

Usage

Install tinkerbox-ane[speech]>=2.2.0. The downloader fetches only this Mac's bundle: its checkpoint folder carries the tokenizer, configuration, embeddings, and audio codec the CPU uses.

Generate speech and save it as WAV:

import soundfile as sf
from tinkerbox_ane import omnivoice
from tinkerbox_ane.bundle import download_bundle

bundle = download_bundle("omnivoice", "wegotomars/OmniVoice-CoreAI-ANE", "v2.2.0")
with omnivoice.load(bundle) as model:
    audio = model.generate("Hello! Welcome to Tinkerbox.", language="en", seed=42)[0]
    sf.write("speech.wav", audio, model.sampling_rate)

Clone a voice from a recording and its transcript, reusing the downloaded bundle:

with omnivoice.load(bundle) as model:
    voice = model.create_voice_clone_prompt(
        "reference.m4a", ref_text="The words spoken in the reference recording."
    )
    audio = model.generate("A new sentence in the same voice.", voice_clone_prompt=voice, seed=42)[0]
    sf.write("cloned.wav", audio, model.sampling_rate)

For voice design, pass an instruction such as instruct="female, low pitch, British accent" to generate. Voice prompts can be saved with voice.save("voice.pt") and reused with model.load_voice_clone_prompt("voice.pt").

Text, reference, and generated audio share the original model's full 40,960-position context per diffusion call. Up to 4,096 positions run as one fused ANE call, longer sequences use blockwise attention. Long-form speech uses upstream chunking above its default 30-second threshold. OmniVoice refines every audio frame together until the last diffusion step, so it does not stream partial audio.

Supported architectures

Architecture	SoC name
h16g	Apple M4
h16s	Apple M4 Pro
h16c	Apple M4 Max
h17g	Apple M5
h17s	Apple M5 Pro
h17c	Apple M5 Max
h17d	Apple M5 Ultra
h18g	Apple M6

Benchmarks

These measurements predate the ANE codec decoder update and describe the previous CPU-codec bundles.

Measured October 26, 2026 on Apple M5 Pro (h17s, 24 GiB), macOS 27.0.1, warm. Real-time multiplier is audio duration divided by generation time (higher is faster); RTF is its inverse (lower is faster).

Request	Audio	Generation time	Real-time multiplier	RTF	ANE energy
Automatic voice	4.4 s	1.11 s	4.0×	0.25	4.2 J
Saved clone	2.9 s	1.03 s	2.8×	0.35	3.8 J
Passage, default chunking	18.5 s	5.74 s	3.2×	0.31	21.2 J

Memory	Peak
Wired, including ANE weights	1.35 GiB
Python process	1.49 GiB

Accuracy

The updated ANE-codec bundle passed public speech regressions on M5 Pro (h17s), macOS 27.0.1, on October 29, 2026: automatic voice, design, cloning, ordered batches, seed replay, request isolation, transcription and offline bundled-checkpoint loading. Other advertised chips are cross-compiled with full ANE placement verification; on-device correctness and performance are established only on M5 Pro.

Compared with the original omnivoice==0.2.1 in CPU FP16 on identical inputs:

Check	Result
Full-transformer logits (96–3,000 positions), cosine	≥ 0.999985
40,960-key attention kernel, cosine	≥ 0.999998
60-second passage, mean transcription error	0.187 (original 0.193)

Original checkpoint: k2-fsa/OmniVoice@c5fdb5ccb189668d56333f77ba2629f4cd7535f4. Weights and converted assets retain CC-BY-NC-4.0. Upstream code is Apache-2.0. The bundled Higgs audio codec retains the Boson Higgs Audio 2 Community License, which incorporates the Meta Llama 3 Community License. See NOTICE.

Built with Higgs Materials licensed from Boson AI USA, Inc., Copyright Boson AI USA, Inc., All Rights Reserved and Meta Llama 3 licensed under the Meta Llama 3 Community License, Copyright Meta Platforms, Inc., All Rights Reserved.