coreai-community/Nemotron-3.5-ASR-Streaming-CoreAI

Core AI is Apple's on-device ML runtime in iOS 27 / macOS 27 and the successor to Core ML: PyTorch models are exported with Apple's coreai-torch (LLMs: coreai.llm.export) into .aimodel bundles that run on the GPU or the Neural Engine, e.g. Qwen3-8B 4-bit decodes at 94 tok/s on an M4 Max GPU, MLX 90 under the same protocol (apple-silicon-llm-bench, macOS 27 beta 26A5353q, 2026-06-11).

Mirror of mlboydaisuke/Nemotron-3.5-ASR-Streaming-CoreAI — the canonical repo (CoreAI Model Zoo). Updates land there first.

This model has no row on DeviceMark, the on-device LLM leaderboard.

Nemotron 3.5 ASR Streaming 0.6B — Core AI

nvidia/nemotron-3.5-asr-streaming-0.6b (OpenMDW-1.1 — commercial use OK, 600M) converted to Apple Core AI .aimodel — the first STREAMING ASR in the zoo: live microphone transcription in 320 ms chunks, on-device, any audio length (no 30 s bucket).

• 40 locales in ONE checkpoint — the language is a one-hot graph input (ja-JP, en-US, zh-CN, … or auto for built-in language ID), switchable per session at run time.
• Punctuation + capitalization built in.
• Cache-aware streaming encoder: the FastConformer's attention KV sliding window (56 frames) and causal-conv left context are explicit graph I/O, so each 320 ms chunk is one static-shape inference — no re-encoding, constant latency forever.
• Pure-RNNT decode (LSTM predictor + joint) driven by a tiny host greedy loop.

Graphs (per 320 ms chunk at lookahead 3)

mel chunk (25 frames first, then 32)   [host: preemphasis→STFT→slaney mel→log, NO normalization]
 1. stream_pre_first / stream_pre : mel + 3 conv caches → embeds[1,4,1024] + caches   (fp16, 9 MB)
 2. stream_conformer_a : x + neg_mask[1,1,4,60]
                         + k/v_cache[12,8,56,128] + conv_cache[12,1024,8]
                         → x + updated caches                     (layers 0-11, fp16, 605 MB)
 3. stream_conformer_b : x + one_hot[1,128] + neg_mask + caches
                         → enc_proj[1,4,640] + updated caches     (layers 12-23 + prompt fusion
                                                                   + projector, fp16, 615 MB)
 host greedy RNN-T over the 4 new frames:
 4. predict : token[1,1] i32 · h,c[2,1,640] → dec_out[1,640] · h',c'                   (fp32, 61 MB)
 5. joint   : dec_out + enc_frame[1,640] → token_logits[1,13088]                       (fp32, 34 MB)
    blank(13087) advances a frame; a token emits + steps the predictor; 10/frame cap

The conformer ships in TWO halves: a single 24-layer AOT bundle (2.4 GB resources.bin) fails to load on-device (instant POSIX-2 from the loader — bisected: identical topology at 1 and 12 layers loads fine), so each half stays ~1.1 GB compiled, for one extra ~1 ms call per chunk. Platform subtrees: macos/ and ios/ hold the same six JIT .aimodel graphs and the tokenizer; every iPhone generation specializes them on its first load. ios-h18p/ holds the two halves compiled ahead of time for the iPhone 17 Pro (h18p), that phone only, with the four small JIT graphs and the tokenizer beside them. The halves moved there from ios/ in revision 73c45366 (2026-09-26). ModelID.nemotronASRStreaming picks the right one automatically.

The JIT conformer halves in ios/ have not been run on an iPhone yet. The iPhone 18 Pro specialized other JIT graphs of up to 1.6 GB on its own. It refuses an h18p bundle with incompatibleCompiledAssetArchitecture (knowledge/jit-distribution.md).

Gated token-exact end-to-end vs the HF streaming reference (99/99 on LibriSpeech, chunked use_cache=True oracle == offline), and again token-exact through Swift CoreAIKit (KitNemotronModel, packet-size-independent mel frontend). blank 13087 · vocab 13088 · max 10 symbols/frame · 16 kHz.

Use (CoreAIKit)

let nemotron = try await KitNemotronModel(model: .nemotronASRStreaming)

// LIVE: feed mic packets as they arrive; the transcript grows while you speak.
let session = try nemotron.makeSession(language: "en-US")   // or "ja-JP", … or "auto"
for await packet in micPackets {                            // 16 kHz mono Float, any packet size
    let partial = try await session.feed(samples: packet)
}
let result = try await session.finish()

// OFFLINE: any-length clip through the same streaming pipeline.
let result = try await nemotron.transcribe(samples: pcm16kMono, language: "en-US")

Try it in the zoo's coreai-audio app (Transcribe tab → "Nemotron Streaming 0.6B" → Live).

Speed

	per 320 ms chunk (warm)	real-time factor
M4 Max (GPU)	~26 ms	0.08 (12× real-time)
iPhone 17 Pro (GPU, AOT)	~53 ms	0.167 (6.0× real-time)

Load on the iPhone 17 Pro: ~52 s the first time after install (one-time GPU specialization of the two AOT halves), ~4 s cached thereafter.

Streaming latency = the model's lookahead (320 ms at the shipped lookahead=3) + chunk compute. The checkpoint also supports lookahead 0/6/13 (80 ms – 1.12 s); those variants re-export with a parameter change in the conversion scripts.

Convert yourself

conversion/nemotron_asr/ — streaming oracle (gen_oracle_streaming.py, HF chunked use_cache=True), cache-explicit re-author + export (export_encoder_streaming.py), token-exact gates (gate_e2e_streaming.py, gate_mel_swift_streaming.py).

License

OpenMDW-1.1 (see LICENSE) — the upstream NVIDIA model's license; this conversion redistributes the weights unchanged in a different serialization.

---

More models in this format: Core AI Model Zoo — 75 models, each with the recipe that produced it.

Want a different model on-device? Open a request — free, open weights only; the export and its measured numbers get published publicly.