zipformer2-66m-streaming-sk

Streaming Slovak automatic speech recognition: a 66 M-parameter Zipformer2 transducer, fine-tuned from icefall's English streaming Zipformer on 3300+ hours of Slovak speech. Decodes in 1.28 s chunks as audio arrives.

Usage (sherpa-onnx)

uv pip install sherpa-onnx soundfile numpy huggingface_hub
uv run hf download neurlang/zipformer2-66m-streaming-sk --include "sherpa-onnx/*" --local-dir .

import numpy as np, sherpa_onnx, soundfile as sf
model_dir = "sherpa-onnx"
recognizer = sherpa_onnx.OnlineRecognizer.from_transducer(
    tokens=f"{model_dir}/tokens.txt",
    encoder=f"{model_dir}/encoder.int8.onnx",
    decoder=f"{model_dir}/decoder.int8.onnx",
    joiner=f"{model_dir}/joiner.int8.onnx",
    model_type="zipformer2",
    num_threads=4,
)
audio, sample_rate = sf.read("audio.wav", dtype="float32", always_2d=True)
stream = recognizer.create_stream()
stream.accept_waveform(sample_rate, audio.mean(axis=1))
stream.accept_waveform(sample_rate, np.zeros(int(1.5 * sample_rate), dtype=np.float32))  # flush the last chunk
stream.input_finished()
while recognizer.is_ready(stream):
    recognizer.decode_stream(stream)
print(recognizer.get_result(stream))

Results

WER (%, lower is better) on Slovak test sets with the leaderboard normalization (no punct errors, no casing errors).

Model	SLAIH ↓	commonvoice	fleurs	lili	tedxsk	trepal
KInIT Whisper Large v3 Turbo SK	5.60%	-	-	-	-	-
KInIT Whisper Large v3 SK	5.67%	-	-	-	-	-
🌊 zipformer2-66m-streaming-sk (this model)	⚠️ 6.166% + Telco + Noise impact	5.8%	12.6%	11.8%	11.8%	4.5%
KInIT Whisper Medium SK	6.28%	-	-	-	-	-
KInIT Canary 1B v2 SK	6.36%	-	-	-	-	-
Microsoft MAI-Transcribe-2	6,54 %	-	-	-	-	-
Google Chirp 3	8,14 %	-	-	-	-	-
ElevenLabs Scribe v2	9,15 %	-	-	-	-	-

⚠️ Unofficial estimate: Hours-weighted WER on our own dev sets, clean audio, lowercase and no punctuation impact, no noise impact, telephone domain not evaluated yet.

Limitations

• English is weak (spelled letter by letter).
• Numbers come out as words or digits inconsistently.

Fine-tuning & training notes

Fine-tune this model

Load model.pt["model"] into icefall's Zipformer2 AsrModel (config above) and continue with icefall's finetune.py or any transducer training loop.

• Audio 16 kHz mono; normalize text like training: replace ';' by ','.
• ScaledAdam base_lr 0.0045, Eden lr_batches 7500 / lr_epochs 2, ~600 s of audio per step.

How it was trained (new language on an English streaming Zipformer)

• Extend the tokenizer, don't replace it. A 1,024-piece Slovak SentencePiece model, trained on ~2800 h of SloPal transcripts, appended after the English 500 pieces (1,493 total). New embedding and output rows start at the mean of the old rows (+1% noise).
• 10× LR on decoder + joiner, so the new rows separate quickly.
• Let the LR decay. icefall's finetune.py defaults (lr_batches 100000 / lr_epochs 100) keep the LR flat, which stalled a new language at ~20% WER; lr_batches 7500 / lr_epochs 2 fixed it.
• Data: See below
• Finish: weight EMA, export to ONNX, re-verify model on eval sets.

License

CC-BY-4.0. Base model from k2-fsa/icefall (Apache-2.0).

Trained on open audio from:

• NaiveNeuron/SloPalSpeech
• streams-sro/trepal
• Common Voice Scripted Speech 27.0 - Slovak
• google/fleurs - fixxed muted audio issue
• TEDxSK and JumpSK Lectures
• General Read Speech (Lili) 1.0
• Greenie Audiobooks, scraped 2026-11-11

Credits

• Thanks to thewh1teagle for original idea, code, and patience
• Also thanks to Daniel Povey and the team at k2 icefall
• Thanks to my company for allowing me to pursue this in my spare time.