zipformer-rnnt-v3

Vietnamese Zipformer2 transducer (RNNT) fine-tuned for Actable meeting audio, starting from actableai/zipformer-rnnt-v2 and trained on actableai/vi-meeting-soniox-wer16 with Soniox transcripts as golden labels.

Base	actableai/zipformer-rnnt-v2 (policy-domain Zipformer, BPE-3000)
Fine-tune data	Meeting segments with WER(Soniox, baseline Qwen) ≤ 16%
Labels	Soniox text only (mixed-case; not uppercased)
Params	~70.7 M (transducer; no CTC head in this fine-tune)
Sample rate	16 kHz, 80-dim log-Mel fbank (on-the-fly)

---

Performance

Meeting dev (Soniox references)

Eval set: session_soniox_dev.jsonl → 5,453 utterances, ~13.8 h
Decoding: greedy search, icefall greedy_search_batch
Text norm for WER: lowercase, strip punctuation / bracket tags, collapse whitespace
Checkpoint: best-valid-loss.pt using model_avg weights

Model	Meeting-dev WER	Errors / words
v2 baseline (actableai/zipformer-rnnt-v2)	31.07%	56,369 / 181,450
v3 (this model)	20.71%	37,580 / 181,450
Δ absolute	−10.36	—
Δ relative	−33.3%	—

Measured 2026-07-27 with eval_zipformer_meeting_wer16.py (full meeting dev).

Validation RNNT loss (during fine-tune)

Loss on meeting dev CutSet at the start of each epoch (icefall validation):

Epoch	Valid loss	Simple	Pruned
1	0.4787	0.4529	0.2577
2	0.3846	0.3782	0.1805
3	0.3605	0.3684	0.1725
4	0.3478	0.3633	0.1662
5	0.3433	0.3604	0.1631
6	0.3416	0.3589	0.1622
7	0.3398	0.3578	0.1609
8	0.3385	0.3565	0.1603
9	0.3369	0.3558	0.1590
10 (best)	0.3362	0.3552	0.1586

• Best valid loss: 0.3362 @ epoch 10
• Best train loss: 0.2452 @ epoch 10
• Wall time: ~1 h 38 m (single NVIDIA A100 40GB)

---

Training details

Data

Split	Segments	Hours	Source
train	22,693	~68.1	actableai/vi-meeting-soniox-wer16 (= local session_soniox_train_filtered.jsonl)
dev	5,453	~13.8	session_soniox_dev.jsonl (Soniox labels)

• Filter: keep train segments where WER(Soniox ref, baseline Qwen hyp) ≤ 16%
• Labels: Soniox field text only — never Qwen hypotheses
• Audio: 16 kHz mono session WAVs (full_session.wav + start/end offsets)
• Mix / replay: none (use_mux=0) — pure meeting wer16
• Utt duration filter: 0.3–30 s

Optimization

Knob	Value
Recipe	icefall Zipformer2 finetune.py
Init	load encoder,encoder_embed,decoder,joiner,simple_am_proj,simple_lm_proj from v2
CTC	disabled (use_ctc=0) — joint CTC FT was unstable (CTC bias grad explosion under fp16)
Optimizer	ScaledAdam + Eden LR schedule
base_lr	5e-4
Epochs	10
Batching	max_duration=300 s / batch, dynamic bucketing
Precision	fp16
Features	on-the-fly 80-dim Kaldi fbank
SpecAugment	on (time-warp factor 80)
MUSAN	off
Speed perturb	off
Seed	42
Hardware	1× A100-SXM4-40GB
Global steps	~8,674

Architecture (unchanged from v2)

Param	Value
num_encoder_layers	2,2,3,4,3,2
encoder_dim	192,256,384,512,384,256
downsampling_factor	1,2,4,8,4,2
feedforward_dim	512,768,1024,1536,1024,768
num_heads	4,4,4,8,4,4
cnn_module_kernel	31,31,15,15,15,31
encoder_unmasked_dim	192,192,256,256,256,192
decoder_dim / joiner_dim	512 / 512
Vocab	BPE-3000 (mixed-case Vietnamese; same as v2)

---

Files

File	Description
best-valid-loss.pt	Best icefall checkpoint (model + model_avg + train metadata)
bpe.model	SentencePiece BPE-3000 (shared with v2)
tokens.txt	Token id map for sherpa / icefall
config.json	Architecture + training metadata
train_zipformer_meeting_wer16.sh	Launch script used for this run
prep_lhotse_meeting_wer16.py	Lhotse CutSet prep (Soniox labels)
eval_zipformer_meeting_wer16.py	Greedy WER eval helper

---

Usage (icefall)

import torch, sys
sys.path.insert(0, "/path/to/icefall")
sys.path.insert(0, "/path/to/zipformer_work")  # finetune.py / model defs

from finetune import get_model, get_params, add_model_arguments
from beam_search import greedy_search_batch
import argparse, sentencepiece as spm

parser = argparse.ArgumentParser()
add_model_arguments(parser)
params = get_params()
params.update(vars(parser.parse_args([])))
params.encoder_dim = "192,256,384,512,384,256"
params.num_encoder_layers = "2,2,3,4,3,2"
params.downsampling_factor = "1,2,4,8,4,2"
params.feedforward_dim = "512,768,1024,1536,1024,768"
params.num_heads = "4,4,4,8,4,4"
params.cnn_module_kernel = "31,31,15,15,15,31"
params.encoder_unmasked_dim = "192,192,256,256,256,192"
params.decoder_dim = 512
params.joiner_dim = 512
params.causal = False
params.vocab_size = 3000
params.blank_id = 0
params.context_size = 2
params.use_transducer = True
params.use_ctc = False

model = get_model(params)
ckpt = torch.load("best-valid-loss.pt", map_location="cpu")
state = ckpt.get("model_avg") or ckpt["model"]
model.load_state_dict(state, strict=False)
model.eval().cuda()

sp = spm.SentencePieceProcessor()
sp.load("bpe.model")

# feature: (1, T, 80) float32 fbank; feature_lens: (1,) int
with torch.no_grad():
    encoder_out, encoder_out_lens = model.forward_encoder(feature, feature_lens)
    token_ids = greedy_search_batch(model, encoder_out, encoder_out_lens)
text = sp.decode(token_ids[0])
print(text)

---

Notes / limitations

• Optimized for Vietnamese meeting / conversational audio; not re-evaluated on open-domain policy or YouTube suites in this card.
• Greedy decoding only in the reported WER; beam search may improve further.
• Transcripts are Soniox pseudo-labels (filtered), not human gold.
• CTC head from v2 was not fine-tuned / not present in the shipped transducer-only graph.

Changelog

Date	Event
2026-07-27	Fine-tune v2 → meeting wer16, 10 epochs; valid loss 0.479 → 0.336
2026-07-27	Meeting-dev WER 31.07% (v2) → 20.71% (v3); push actableai/zipformer-rnnt-v3