Instructions to use acul3/zapformer-id-en-streaming with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LiteRT
How to use acul3/zapformer-id-en-streaming with LiteRT:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
Zapformer-48 streaming ASR, Indonesian + English: epoch 48 (final)
A 167M-parameter streaming transducer for Indonesian, with English as a second language, built to run on device. This revision ships the same checkpoint in three formats:
- ONNX for sherpa-onnx (patched; patch included).
- LiteRT (
.tflite) for Android/iOS LiteRT runtimes. - PyTorch weights for re-export.
The architecture is Zapformer, Dan Povey's successor to Zipformer (icefall PR #2082).
Status: this is the final checkpoint of the v2.0 run: epoch 48 of 48. The learning rate has decayed to about 2% of peak.
main= epoch 48 + thelm/folder.
Quick start
Access. This repo is private. Log in once with an account that has access: hf auth login, or set HF_TOKEN.
Pick a runtime:
| You want | Use | Download | Needs |
|---|---|---|---|
| Transcribe files or live audio on a desktop or server, with LM / hotwords / two-pass | sherpa-onnx (patched, built from source) | ~380 MB | C++ toolchain + cmake; build β 5 min |
| The simplest check that the model works, or a reference loop for a mobile app | LiteRT (.tflite, pure Python) |
~170 MB | pip only |
| Re-export, fine-tune or debug | PyTorch (model_code/) |
~640 MB | pip only |
Tested with Python 3.12 on macOS (Apple Silicon) and Linux x86-64. The whole repo is 3.6 GB; download only what you need:
pip install -U huggingface_hub # provides the `hf` command
R=acul3/zapformer-id-en-streaming
# common files (tokens + example scripts), always needed
hf download $R --local-dir zapformer --include "tokens.txt" --include "bpe.*" --include "examples/*.py"
# sherpa-onnx: 640 ms streaming (~170 MB) + the sherpa patches
hf download $R --local-dir zapformer --include "streaming-640ms/*int8.onnx" --include "streaming-640ms/decoder-*" \
--include "sherpa-onnx/*"
# + offline model for --two-pass (~180 MB): --include "offline/*int8.onnx" --include "offline/decoder-*"
# + language model for --lm (~30 MB): --include "lm/*int8.onnx" --include "lm/*.fst"
# + other latencies: --include "streaming-320ms/*" --include "streaming-1280ms/*"
# LiteRT (~170 MB)
hf download $R --local-dir zapformer --include "litert/*int8.tflite" --include "litert/init_states.npz" \
--include "litert/meta.json"
# PyTorch (~640 MB)
hf download $R --local-dir zapformer --include "pretrained-epoch-48-avg-2.pt" --include "model_code/*"
Audio. The model takes 16 kHz mono. examples/decode_file.py resamples WAV/FLAC itself. decode_file_litert.py
and model_code/infer.py need a 16 kHz file:
ffmpeg -i input.m4a -ac 1 -ar 16000 audio.wav
Run (from the zapformer/ folder):
# LiteRT: no build needed
pip install ai-edge-litert kaldi-native-fbank soundfile numpy
python examples/decode_file_litert.py audio.wav
# sherpa-onnx: build it first (see "How to run: sherpa-onnx" below)
python examples/decode_file.py audio.wav # streaming, one line per utterance
python examples/decode_file.py audio.wav --two-pass --lm # best accuracy for a final transcript
# PyTorch
pip install torch numpy soundfile kaldi-native-fbank
python model_code/infer.py --weights pretrained-epoch-48-avg-2.pt --tokens tokens.txt --blank-penalty 0.5 audio.wav
Output is cased and punctuated text. decode_file.py prints [mm:ss.ss] text per utterance.
Before using it on code-switched audio, read Known issues.
Results
Real streaming decoding at 640 ms (chunk by chunk, with state carried between chunks), int8, blank penalty 0.5. WER in %; lower is better.
| Benchmark (normalizer) | This model, greedy | This model + LM (beam 4, lm/) |
Qwen3-ASR-1.7B (offline, ~2B) | Nemotron-0.6B P13 (our previous best, 1,040 ms) |
|---|---|---|---|---|
| conversational-v2: Indonesian conversational speech, 357 human-reviewed clips (lowercase, punctuation stripped) | 18.66 | 18.06 | 33.45 | 29.80 |
| GigaSpeechBench-IDN: 21 h, human labels (official IDN normalizer) | 21.44 | 20.31 | 22.29 | 29.67 |
| GigaSpeech 2 id test (num2words) | 18.07 | 17.38 | 17.03 | 23.38 |
| FLEURS id test (lowercase, punctuation stripped / numbers spelled out) | 12.02 / 9.76 | 10.46 / 8.29 | 5.65 / 5.12 | 13.65 |
| FLEURS en test | 19.22 | 17.72 | β | 15.23 |
Columns:
- Greedy is the default and cheapest setting.
- + LM adds a 25 MB LSTM language model and a 4.9 MB LODR bigram with beam search (
lm/, ~1.4Γ compute, ~+100 MB memory). LM settings were tuned on dev sets only; seelm/README.md.
Decoding: both columns were decoded with the patched sherpa-onnx (Linux build), at batch 8.
Test sets: conversational-v2 is our internal set of 357 human-reviewed Indonesian conversational clips (not public). GSB-IDN, GigaSpeech 2 and FLEURS are public.
Comparisons:
- Qwen3-ASR-1.7B was scored by us on the same clips with the same normalizer (GS2: the full 17,271-clip test). The exception is GSB, where we use the paper's figure.
- Nemotron P13 is NVIDIA nemotron-3.5-asr-streaming-0.6b fine-tuned on our Indonesian data. It is 3.6Γ larger and uses 1,040 ms of lookahead.
Blank penalty subtracts a constant from the blank logit at every frame (
blank_penalty=in sherpa-onnx). It costs nothing at runtime. See "Decoding settings" below for why 0.5 is the default.
GigaSpeechBench-IDN in context (arXiv 2606.28884, Table 2):
| System | IDN WER |
|---|---|
| FunASR-Realtime | 14.87 |
| Qwen3.5-Omni-Plus | 18.05 |
| Chirp 3 | 19.98 |
| Qwen3-ASR-Flash | 20.45 |
| This model + LM (640 ms streaming, 167M + 30 MB LM, int8, on device) | 20.31 |
| This model, greedy (640 ms streaming, 167M, int8, on device) | 21.44 |
| Qwen3-ASR-1.7B | 22.29 |
| ElevenLabs Scribe v2 | 22.91 |
| Gemini 3.0 Flash | 24.18 |
| Azure | 25.50 |
| Whisper large-v3 | 27.40 |
| FunASR-MLT-Nano | 27.68 |
| GPT-4o Transcribe / Meta OmniASR-3B | 37.95 / 37.91 |
GSB-IDN has 122 videos. By our video-level bootstrap, a single system's score is uncertain by about Β±2.2 points (95%). The systems between 20 and 23 are therefore effectively tied. Paired differences on the same model, such as two checkpoints or two decoding settings, are good to about Β±0.2.
Learning curve (640 ms, blank penalty 0):
| ep 5 | ep 10 | ep 15 | ep 20 | ep 25 | ep 29 | ep 35 | ep 40 | ep 48 | |
|---|---|---|---|---|---|---|---|---|---|
| conversational-v2 | 29.22 | 26.86 | 24.83 | 23.87 | 21.91 | 21.51 | 21.05 | 19.09 | 19.17 |
| GSB-IDN | 28.28 | 25.85 | 25.18 | 24.18 | 23.51 | 22.88 | 22.80 | 22.52 | 22.75 |
| GS2 | 22.01 | 20.50 | 20.07 | 19.67 | 19.30 | 19.04 | 19.28 | 18.94 | 18.80 |
| FLEURS-id | 13.84 | 13.45 | 13.56 | 12.90 | 13.10 | 13.29 | 12.86 | 12.46 | 12.13 |
| FLEURS-en | 35.77 | 31.03 | 28.45 | 26.82 | 25.11 | 23.47 | 22.37 | 20.80 | 19.59 |
Epoch 48 was decoded with our Python streaming loop (same search as sherpa-onnx, batch 1); earlier epochs with sherpa-onnx. Small differences from the Results table (blank penalty 0.5, batch 8) come from that.
Decoding settings
Blank penalty 0.5 (recommended default). As the learning rate decays, the model's wrong-word errors keep falling, but it outputs nothing on more very short clips:
| GSB empty outputs (of 22,978 clips) | ep 25 | ep 29 | ep 35 | ep 40 | ep 48 |
|---|---|---|---|---|---|
| no output at all | 1,659 | 1,807 | 2,267 | 2,449 | 2,860 |
A small blank penalty corrects this. Sweep on epoch 35 (GSB quarter subset, which is harder than the full set; full conversational-v2):
| blank penalty | GSB subset (sub / del / ins) | conversational-v2 (sub / del / ins) |
|---|---|---|
| 0 | 25.89 (11.2 / 13.6 / 1.0) | 21.03 (11.7 / 7.1 / 2.3) |
| 0.5 | 24.68 (12.0 / 11.4 / 1.3) | 20.49 (12.2 / 5.2 / 3.0) |
| 1.0 | 24.15 (12.7 / 9.9 / 1.6) | 20.80 (13.1 / 3.8 / 3.9) |
| 1.5 | 24.02 (13.2 / 8.7 / 2.1) | 21.85 (13.7 / 2.9 / 5.3) |
- Default: use 0.5 for mixed audio.
- Short-clip audio: 1.0 helps a little more on short clips (full GSB 21.01) but costs about 0.3 on conversational-v2. The 1.0 result was picked by looking at GSB, so we don't headline it.
Punctuation boost. The model outputs cased, punctuated text. In streaming it emits a full stop only once the next sentence starts, so the last sentence of an utterance is often left open.
- ONNX: the
*.punct.*joiners have a small logit boost baked in:.+2,?+2,,+1. - LiteRT: apply the same boost in the search loop;
examples/decode_file_litert.pydoes this. - Either way: add
.to any utterance that ends without a mark at an endpoint.
On conversational-v2 at epoch 15, the boost raised comma F1 from 24 to 49 and sentence-end F1 from 40 to 74, with WER unchanged. The offline export closes sentences by itself.
Latency modes. One checkpoint supports several modes. These numbers are from epoch 10; the relative gaps have held:
| 320 ms | 640 ms | 1,280 ms | offline (full context) | |
|---|---|---|---|---|
| conversational-v2 | 27.79 | 26.86 | 26.24 | 23.34 |
| GSB-IDN | 26.96 | 25.85 | 25.95 | 23.96 |
| GS2 | 21.05 | 20.50 | 20.70 | 19.58 |
| FLEURS-id | 13.87 | 13.45 | 13.13 | 12.10 |
| FLEURS-en | 33.01 | 31.03 | 29.73 | 27.89 |
- Live captions: 640 ms is the best choice.
- Final transcript: use offline mode, re-decoding each finished utterance ("two-pass", see below).
Files
| Path | What it is |
|---|---|
streaming-640ms/ |
Main ONNX model. encoder-β¦int8.onnx, decoder-β¦.onnx, joiner-β¦int8.onnx, joiner-β¦punct.int8.onnx, plus fp32 encoder and joiners (for other quantization schemes) |
streaming-320ms/, streaming-1280ms/ |
The same checkpoint exported for 320 ms and 1,280 ms chunks (int8; plain and punct joiners) |
offline/ |
Full-context ONNX export for a second pass or file decoding (int8 + fp32) |
litert/ |
LiteRT 640 ms streaming model: encoder/decoder/joiner.int8.tflite (recommended) and .fp32.tflite, init_states.npz, meta.json, plus the conversion scripts |
pretrained-epoch-48-avg-2.pt |
PyTorch weights ({"model": state_dict}), the averaged model over epochs 46β48, for icefall export*.py |
model_code/ |
Pinned PyTorch model code (icefall PR #2082 at 0ee2a6d5, the version this model was trained with) + zap_model.py loader + infer.py streaming transcription. Runs without icefall/k2/lhotse. Newer upstream Zapformer code has a different architecture and cannot load these weights |
lm/ |
Language model for rare words: lstm-lm-2x1024.int8.onnx (25 MB) + lodr-2gram-p300k.fst (4.9 MB), fp32 LM, and its README (results, settings, how it was built) |
tokens.txt, bpe.model, bpe.vocab |
SentencePiece unigram, 4,096 tokens, cased and punctuated, byte fallback, digits split. bpe.vocab is what sherpa needs to tokenize hotwords |
sherpa-onnx/ |
The sherpa-onnx patches this model needs, plus the ONNX export scripts |
examples/decode_file.py |
Transcribe a file with sherpa-onnx: streaming pass, optional --two-pass, --lm, --hotwords "A/B" or --hotwords-file, --latency 320/640/1280, --beam, --threads, --no-punct-boost. Resamples WAV/FLAC to 16 kHz |
examples/decode_file_litert.py |
Transcribe a file with the LiteRT model in pure Python; this is the reference loop for a mobile runtime |
examples/poc/ |
Internal web demo (live mic, latency switch, two-pass). Not runnable from this repo: it is hard-coded to the author's machine layout. Use decode_file.py instead |
How to run: sherpa-onnx (ONNX)
1. Build sherpa-onnx with the Zapformer patch. Stock sherpa-onnx can't load this model: Zapformer has 9 streaming states per layer (326 in total), and its causal norms start from learned values.
Needs git, cmake (β₯ 3.13) and a C++17 compiler (Xcode command-line tools on macOS, build-essential on Linux). The
build downloads a prebuilt ONNX Runtime and takes about 5 minutes. Clone sherpa-onnx next to the model folder, not
inside it (the model repo already has a sherpa-onnx/ folder holding the patches):
ZAP=$PWD/zapformer # the folder you downloaded the model into
git clone https://github.com/k2-fsa/sherpa-onnx sherpa-onnx-src && cd sherpa-onnx-src
git checkout 040afe360a38e25daaa325ce8889abf93ea02609
git apply $ZAP/sherpa-onnx/sherpa-onnx-zapformer.patch # model type "zapformer"
git apply $ZAP/sherpa-onnx/sherpa-onnx-ort-1.30.patch # ONNX Runtime 1.30: ~150 MB less memory
python -m venv .venv && . .venv/bin/activate # any venv; Python 3.12 tested
pip install . soundfile numpy scipy # Python package, used by examples/decode_file.py
cd $ZAP
For Android/iOS, run ./build-android-arm64-v8a.sh or ./build-ios.sh in sherpa-onnx-src instead of pip install .
(details in sherpa-onnx/README.md).
2. Streaming (live captions):
import sherpa_onnx
d, s = "streaming-640ms", "epoch-48-avg-2-chunk-32-left-128"
rec = sherpa_onnx.OnlineRecognizer.from_transducer(
tokens="tokens.txt", encoder=f"{d}/encoder-{s}.int8.onnx", decoder=f"{d}/decoder-{s}.onnx",
joiner=f"{d}/joiner-{s}.punct.int8.onnx", num_threads=1, decoding_method="greedy_search", blank_penalty=0.5,
enable_endpoint_detection=True, rule2_min_trailing_silence=1.2, rule3_min_utterance_length=20)
stream = rec.create_stream()
# loop: stream.accept_waveform(16000, pcm_float32); while rec.is_ready(stream): rec.decode_stream(stream)
# rec.get_result(stream) -> partial text
# rec.is_endpoint(stream) -> utterance done: take the text, then start a NEW stream (not rec.reset(stream)),
# feeding it the last ~1 s of audio first (see Known issues; examples/decode_file.py does this)
- Audio: 16 kHz mono float32, samples in [β1, 1].
- New stream per utterance:
rec.reset(stream)clears the text but keeps the encoder state. Keeping that state across utterances makes the model drop English that follows Indonesian, so create a fresh stream at each endpoint. - Tail padding: append about 1 s of silence before
input_finished()when decoding a file or push-to-talk. Otherwise the last chunk is never decoded.
3. Two-pass / offline. Use sherpa_onnx.OfflineRecognizer.from_transducer(...) with the offline/ files, which
needs no patch. Two quirks of this export:
- Length limit: inputs must be at most about 30 s, so split longer audio.
- Allowed lengths: it only accepts fbank lengths with
frames % 8 == 5. Pad up to 70 ms of silence, asexamples/decode_file.pydoes. Without the padding ONNX Runtime fails with an error likeWhere node β¦ broadcast 513 by 516. - Cutting utterances: cut each one using the streaming decoder's timing (
start_time+ last token timestamp + ~0.7 s), not the moment the endpoint fired.
python examples/decode_file.py audio.wav --latency 640 --two-pass
decode_file.py keeps the streaming text for a segment when the offline pass returns far fewer words (the offline
model can drop leading English, see Known issues). Segment boundaries can repeat a word.
4. Rare words: LM and hotwords (both need decoding_method="modified_beam_search", max_active_paths=4).
- LM (
lm/): general language knowledge from a large text corpus. It fixes common words and text-known names, and improves every test set (table above). Uselm_scale=0.3together withlodr_fst=lm/lodr-2gram-p300k.fst, lodr_scale=-0.25. Without LODR the decoder drops words. - Hotwords: boost phrases you know in advance, such as speaker names, client names or a product glossary. Pass
them per stream with
rec.create_stream(hotwords="Yohannis Lema/RKP/Bhinneka")or globally withhotwords_file=, plusmodeling_unit="bpe", bpe_vocab="bpe.vocab",hotwords_score=2.0.- Case-sensitive: lowercase-only lists lose ~8 points of recall on names. Add case variants (as given + lowercase + Title + UPPER). In our test that beat even perfectly cased lists: conversational rare-word recall 60% vs 58% (vs 38% without hotwords).
- Our test: each utterance got its rare words plus 100 distractors. Rare-word recall rose from 38β45% to 58β67% with no harm to other words.
- With the LM: LM + hotwords (score 2.0) is the best combination. On FLEURS-id dev, WER drops 9.9 β 8.0 (LM alone 8.8, hotwords alone 9.2). On conversational speech, hotwords carry the gain.
- Score: 2.0 is the tuned default over many utterances. A single stubborn word may need more (in one test,
"Bhinneka" needed 8 with hotwords alone but was fixed at 2.0 together with
--lm). Higher scores add false insertions, so check on your own audio. - They complement each other: the LM can't know session-specific names, and hotwords can't fix general language. Both can run together.
python examples/decode_file.py audio.wav --lm --hotwords "Yohannis Lema/Bhinneka/CCUS"
How to run: LiteRT (.tflite)
litert/ holds the 640 ms streaming model, converted from PyTorch with litert-torch 0.9.4 and quantized with
ai-edge-quantizer (dynamic int8: int8 weights, activations quantized on the fly). This is the same scheme as the
ONNX int8 export. Accuracy matches ONNX int8 within eval noise (Β±0.25), and the encoder is about 25% faster per chunk on Mac CPU.
| Epoch 48 (same Python loop, batch 1) | LiteRT int8 | ONNX int8 |
|---|---|---|
| encoder size | 155.7 MiB | 155.8 MiB |
| conversational-v2, blank penalty 0 | 18.95 | 19.17 |
| FLEURS-id (norm / n2w), blank penalty 0 | 11.55 / 9.61 | 12.13 / 9.96 |
| encoder time per 640 ms chunk (Mac M5, 1 thread; measured on epoch 35, same model size) | 16.4 ms | 21.8 ms |
| peak memory of the benchmark process (Python) | 424 MB | 351 MB |
pip install ai-edge-litert kaldi-native-fbank soundfile numpy
python examples/decode_file_litert.py audio.wav # int8, blank penalty 0.5, punctuation boost
This decodes the whole file as one stream with no endpointing, so it shows the code-switching issue below on
mixed audio. Use it for single utterances or as the reference loop; use decode_file.py for long recordings.
Interface (what an Android/iOS runtime must implement):
- Encoder (
serving_default): 327 inputs, 327 outputs.- Inputs:
args_0is fbank[1, 71, 80]float32;args_1_<i>is state i. - Outputs:
output_0isencoder_out [1, 16, 576];output_<i+1>is the new state i. - States: 36 layers Γ 9 (
cached_key,cached_value,cached_conv,cached_norm_stats,cached_norm_len,cached_attn_wm_sum,cached_attn_wm_num_frames,cached_conv_wm_sum,cached_conv_wm_num_frames), thenembed_cache [1, 128, 6, 19]andprocessed_lens [1]. 73 states are int64. - Shapes and dtypes are in
meta.json. - Start every stream from
init_states.npz, not zeros. The causal-norm states have learned initial values.
- Inputs:
- Features: kaldi fbank, 80 bins, dither 0,
snip_edges=false,high_freq=-400, samples in [β1, 1].- Feed 71 frames and advance by 64.
- At end of stream, pad ~1 s of silence.
- Decoder:
args_0is the context[1, 2]int64. Start with[-1, 0]; β1 is masked inside the model. Output is[1, 576]. - Joiner:
args_0is encoder frame[1, 576],args_1is decoder out[1, 576]. Output is logits[1, 4096]. - Search: greedy, at most one symbol per frame, blank = 0.
- Subtract the blank penalty (0.5) from
logit[0]. - Optionally add the punctuation boost (token 260
.+2, 275?+2, 259,+1).
- Subtract the blank penalty (0.5) from
- Tokens:
βmeans a space;<0xNN>are byte-fallback pieces (decode as UTF-8). - Not provided yet:
- A Kotlin/C++ runtime. sherpa-onnx can't load
.tflite, so use the LiteRTCompiledModelAPI plus kaldi-native-fbank plus the loop above. - GPU/NPU delegates. These need int32 states, state packing and full static int8.
- A LiteRT offline (two-pass) encoder.
- A Kotlin/C++ runtime. sherpa-onnx can't load
- Re-export: the conversion scripts are in
litert/scripts/, with their own README.
How to run: PyTorch
model_code/ has the exact model code this checkpoint was trained with, frozen. Upstream Zapformer changed the
architecture after 2026-08: the central stack has no conv and no second feed-forward, and it uses SiLU. Newer upstream
code cannot load these weights.
pip install torch numpy soundfile kaldi-native-fbank
python model_code/infer.py --weights pretrained-epoch-48-avg-2.pt --tokens tokens.txt --blank-penalty 0.5 audio.wav
Pass --blank-penalty 0.5 explicitly (the script's default is 0). Like the LiteRT example, it decodes the whole file as
one stream, with no endpointing and no punctuation boost.
infer.py runs real 640 ms streaming with greedy search, the same algorithm as sherpa-onnx. On FLEURS-id, 17/20 outputs
were identical to the int8 ONNX model; the rest differed by a comma or one word (fp32 vs int8). For your own code, use
from zap_model import load_model.
Speed and memory
Apple M5, single stream, int8, ONNX Runtime 1.30 (the model size is the same at every epoch):
| value | |
|---|---|
| RTF, 640 ms, 1 thread, greedy | 0.037β0.039 (about 24 ms of compute per 640 ms chunk) |
| RTF, 640 ms, 1 thread, beam 4 + LM + LODR (300k) | 0.052 (33 ms per chunk), peak memory +100 MB |
| RTF, 320 ms / 1,280 ms (2 threads) | 0.055 / 0.027 |
| Memory while streaming (Python process) | ~370 MB |
| Time to first partial result | 0.8 s at 640 ms |
One thread is the best setting. Phone numbers are still to be measured; expect roughly 2β6Γ the Mac's RTF, which is still well within real time.
Throughput vs battery. A single live stream is light: about 4% of one core. Batch-decoding hours of audio keeps every core busy, so plan for mains power.
Model
| Architecture | Zapformer (icefall PR #2082, danpovey/zapformer3127 @ 0ee2a6d5), causal, --base-dim 48 |
| Parameters | 166.9M total (encoder 156.5M; stateless decoder with a 2-token context; joiner) |
| Frontend | 80-dim fbank (lhotse = kaldi-native-fbank; sherpa defaults) |
| Training losses | pruned RNN-T + CTC auxiliary (scale 0.2) |
| Streaming training | random chunk sizes {16, 32, 64, full} Γ left context {64, 128, 256, full}, at 50 Hz after the embed layer |
| Optimizer / schedule | BatchedRubik (Muon-style), base LR 0.02, InterpCosine over 48 virtual epochs |
| Hardware / precision | 2 Γ A100-80GB, bf16, 1,000 s of audio per GPU per batch |
| Augmentation | synthetic room reverb (p = 0.2), Opus/MP3 recompression (p = 0.1), on-GPU alternating SpecAugment |
| Checkpoint | epoch 48 (final), --avg 2 --use-averaged-model 1 (averaged model between epochs 46 and 48) |
Training data
Indonesian and English speech, mostly Indonesian. The data sources are not disclosed. All test sets reported above were excluded from training.
Known issues
- English after Indonesian in the same stream is dropped. Once the encoder has processed Indonesian speech, English
that follows in the same stream comes out empty or nearly empty. Silence in between does not help (tested 0β5 s);
English β Indonesian and Indonesian β Indonesian are fine. It happens in all three runtimes and comes from the
encoder's streaming state (its causal norms keep running statistics over the whole stream).
- Workaround (verified): start a new stream at every endpoint, re-feeding the last ~1 s of audio.
examples/decode_file.pydoes this by default (--keep-stateturns it off). English then decodes normally after Indonesian utterances. - Still affected: English inside the same utterance as preceding Indonesian (no pause long enough to trigger an
endpoint), and the single-stream examples (
decode_file_litert.py,model_code/infer.py). - Offline model: in full-context mode, English at the start of a segment that is mostly Indonesian can be dropped.
- Found on short test clips (conversational Indonesian + read English, and synthetic voices); not yet measured on real code-switched recordings.
- Workaround (verified): start a new stream at every endpoint, re-feeding the last ~1 s of audio.
Limitations
- Spurious "Hai". On read speech the model sometimes starts an utterance with "Hai" (18 times in the 687 FLEURS-id clips; never on conversational-v2). This is a label artifact in the training data. It will be cleaned for the next model.
- Rare words and names. The prediction network only sees the last 2 tokens. Foreign names, technical terms and Indonesian look-alike words are the main real errors. This is why FLEURS-id is far behind LLM-based ASR (Qwen3-ASR: 5.7). The bundled LM recovers part of this (FLEURS-id 12.0 β 10.5). Hotwords fix names you know in advance. Targeted rare-vocabulary training data is the next fix.
- Very short clips. Clips under 1.5 s are often left empty, and this grew in the last epochs (GSB: 2,860 empty clips at epoch 48 with blank penalty 0). Use blank penalty 0.5. Most live speech is longer, so the effect on real use is smaller than on GSB.
- English is secondary (FLEURS-en 19.6). More English data is planned for the next, larger model.
- Long silences inside a sentence (over 1.2 s) end an utterance with the default endpoint rules.
- Quantization is dynamic int8 only, in both ONNX and LiteRT. There's no static int8 for NPUs yet.
Other runtimes (not in this repo)
An ExecuTorch (.pte) export of this checkpoint for react-native-executorch (greedy, 640 ms, int8, fbank inside the
model) exists but is not published here; ask the maintainer.
Intended use
Internal research and product evaluation (on-device Indonesian speech recognition). Private repository.
- Downloads last month
- -