harikanth/indic-conformer-600m-multilingual-coreml

IndicConformer 600M multilingual for Core ML

On-device speech recognition for 22 Indian languages on Apple silicon. This is AI4Bharat's IndicConformer 600M multilingual model, converted to Core ML so it runs on the Apple Neural Engine. It gives words with timings and needs no internet: the audio never leaves the device.

Converted and tested by ezRecord, a macOS screen and camera recorder. It's how ezRecord transcribes and captions Telugu, Hindi, Tamil and the other scheduled languages.

All credit for the model goes to AI4Bharat (IIT Madras): IndicConformerASR. It's MIT-licensed, and so is this conversion.

What's in it

File	What
IndicConformerCTC-pal8.mlpackage	Encoder plus the shared CTC head, 8-bit palettized weights (572 MB). Input: a log-mel [1, 80, 1536] (15.36 s at 10 ms hop) and its length. Output: raw logits [1, 192, 5633] (80 ms per frame).
IndicConformerCTC-pal8.mlmodelc	The same model, compiled for macOS.
vocab.json	Tokens for every language (SentencePiece pieces, ▁ starts a word).
language_columns.json	Which 257 of the 5,633 output columns belong to each language (256 tokens plus blank).
preprocessor.json, preprocessor.bin	The mel frontend's constants: window, mel filterbank, pre-emphasis and normalisation, matching NeMo.
assets.json	Languages, column layout and blank index.
manifest.json	File list with sizes and SHA-256, used by ezRecord's resumable downloader.

One download covers all 22 languages: the CTC head is shared, and the language is chosen by masking the output to that language's 257 columns. The same trick gives language detection almost for free: decode the first window under every language's mask and keep the most confident one.

Languages: Assamese, Bengali, Bodo, Dogri, Gujarati, Hindi, Kannada, Kashmiri, Konkani, Maithili, Malayalam, Manipuri, Marathi, Nepali, Odia, Punjabi, Sanskrit, Santali, Sindhi, Tamil, Telugu, Urdu.

How well it works

Measured on an Apple M4 Mac (Neural Engine):

• Speed: about 400× real time end to end. An hour of audio takes about 9 s. Peak memory is about 0.75 GB. The first load takes 15–20 s while the Neural Engine compiles the model; later loads take about 0.1 s.
• Matches the original: on public FLEURS clips, this build is within 0.7–1.3 points of word error rate of the same windows run in PyTorch, and within 0.5 points against the reference transcripts, for Hindi, Tamil and Telugu.
• Real speech: on a native speaker's own 40 s Telugu recording, made in a normal room, it got every word right (0.0 % character error rate). The remaining word differences were only spacing conventions ("ఈ రోజుల్లో" vs "ఈరోజుల్లో").
• Language detection: it picked the right language on 11 of 12 FLEURS clips and on the Telugu recording. The miss was a 5 s Hindi clip that scored Urdu by a hair.

Things to know

• No punctuation: the model writes words only. Break lines on pauses, or add a punctuation pass afterwards (e.g. AI4Bharat's Cadence).
• Long audio: run it in 15.36 s windows, featurise each window's own audio, and join neighbouring windows at a word both of them emit. Cutting at a fixed frame duplicates syllables, because CTC places tokens a few frames later when it has less audio ahead.
• CTC word ends come out slightly early. Refine them with forced alignment if you need tight karaoke timing.
• Accuracy varies by language. It's strongest on widely spoken languages and weaker on low-resource ones such as Bodo and Dogri (see AI4Bharat's published results).
• iPhone: not measured yet. Some iPhone Neural Engines reject palettized weights; an fp16 build is planned for them.

Using it

Use the open-source Swift package IndicSpeech (Apache-2.0). It has the mel frontend, windowing, CTC decoding with word timings and language detection; the conversion and benchmark scripts are in its scripts/ folder.

.package(url: "https://github.com/harikanthl/IndicSpeech.git", from: "1.0.0")

import IndicSpeech
let engine = try await IndicConformerEngine(modelDirectory: modelURL)
let result = try await engine.transcribe(contentsOf: audioURL, language: .telugu)

Licence and credits

• Model: AI4Bharat IndicConformer, MIT. © AI4Bharat.
• This Core ML conversion: MIT.
• The conversion scripts (IndicSpeech/scripts) were adapted from FluidInference's möbius Parakeet scripts (Apache-2.0). The mel frontend constants follow AI4Bharat's NeMo preprocessor.