ozonetg/parakeet-110m-caller-asr-basic

Parakeet 110m caller-side phone ASR · basic

A fine-tune of NVIDIA's nvidia/parakeet-tdt_ctc-110m for the caller side of 8 kHz US English telephone calls: plain full fine-tune (the reference recipe). It was trained on about 345 hours of real outbound sales-call audio (the customer's channel only) with machine transcripts, and evaluated on public phone-call sets with human transcripts as well as held-out calls.

On five public phone-call sets (pooled) its word error rate is 11.60%, against 14.00% for the system in use before (the stock 110m with a small domain adapter) and 14.58% for the untouched 110m base. On held-out in-domain calls it is 3.44% (before: 8.68%; base: 9.19%). On LibriSpeech test-other at 8 kHz it is 7.30% against 6.25% for its base (+1.05 pp).

Model summary

Base model	nvidia/parakeet-tdt_ctc-110m
Architecture	FastConformer encoder (114.6 M parameters) with a TDT (token-and-duration transducer) decoder and an auxiliary CTC head
Parameters	114.6 M
Weights	parakeet-110m-caller-asr-basic.nemo, float32
Input	mono 16 kHz float audio; feed 8 kHz telephone audio upsampled to 16 kHz (see Usage)
Output	English text with punctuation and capitals
Decoding	greedy TDT (all numbers below)
Framework	NeMo 2.5.3 (also restores and transcribes in NeMo 3.0)
Licence	CC BY 4.0

What it is and how it was made

A plain full fine-tune of nvidia/parakeet-tdt_ctc-110m: every weight trained on the recipe below, nothing else. It is the reference the other 110m variants are measured against.

Training data and recipe shared by the family

• Audio: the caller (customer) side of US outbound sales phone calls, 8 kHz mono, from five internal sources of outbound sales-call recordings, recorded 2025–2026. Segments of 0.3 to 20 s. The training split has 504,617 segments (345.5 h); 5% are non-speech segments (line noise, silence, hold, breathing) with an empty target, which teach the model to stay quiet; about 1% are voicemail greetings or IVR prompts heard on the caller line.
• Labels: machine transcripts from Qwen3-ASR-1.7B. No human transcribed this data. Each transcript was checked against independent ASR systems: segments where they agreed are tier 1 (loss weight 1.0), partial agreement is tier 2 (weight 0.7), and the rest were dropped.
• Replay: 10% of every batch comes from Switchboard (public conversational telephone English, human transcripts) to keep general conversational ability.
• Input path: 8 kHz audio is upsampled to 16 kHz with librosa's default resampler (soxr_hq), which is exactly what NeMo's own audio loader does with an 8 kHz WAV file. Training and evaluation used the same chain.
• Optimiser: AdamW (β 0.9 / 0.98, weight decay 0.001), peak learning rate 1e-4 after 1,000 warm-up steps, cosine decay to 1e-6, 5 epochs (about 23,700 steps of ~300 s of audio each), no layer frozen, BatchNorm statistics frozen, bfloat16 autocast with fp32 master weights. The checkpoint was chosen every 1,000 steps by the mean WER of the in-domain dev split and a separate set of customer validation calls (never a test set).
• Loss: hybrid TDT + CTC (CTC weight 0.3). Decoding uses the TDT head (greedy).
• Augmentation (plain recipe): SpecAugment with 2 frequency masks (width 27) and 2 time masks (5%).

Intended use

• Transcribing the caller / customer side of narrowband (8 kHz) US English phone calls: voice agents and IVR, call analytics, short-answer understanding (“yes”, “no”, “not interested”, “who is this?”).
• As a drop-in replacement for the base model wherever audio is telephone-band.

Out of scope: wideband studio or meeting audio (use the base model), languages other than English, speaker identification, and any automated decision about a person without human review. Do not use it to transcribe people who have not been told they are recorded.

Results

Word error rate (WER, %, lower is better) after the same text normaliser for every model and reference (lowercase, punctuation removed, fillers removed, contractions expanded, numbers written as digits). Δ = paired difference in percentage points with a 95% confidence interval from 2,000 cluster bootstrap resamples (clusters = calls or speakers); negative = this model is better. Columns:

• ⁰ Previous baseline: the stock parakeet-tdt_ctc-110m plus a small domain adapter trained earlier on similar audio. It is the system these models replace; it is not released.
• Untouched base: nvidia/parakeet-tdt_ctc-110m as published.
• Plain fine-tune: the reference recipe of this size (this model).

Public phone-call sets (human transcripts)

set	Previous baseline ⁰	Untouched base	This model	this model vs previous baseline (Δ pp, 95% CI)
Five public phone-call sets, pooled	14.00	14.58	11.60	−2.40 [−2.74, −2.13]
Let's Go (bus-information IVR callers), re-typed references	28.82	26.69	23.94	−4.88 [−6.00, −3.76]
Let's Go, all kept turns	29.39	27.92	25.15	−4.23 [−4.89, −3.57]
HarperValley Bank, caller channel	7.70	9.56	5.15	−2.55 [−3.15, −1.99]
CallHome English (test)	14.73	15.09	12.37	−2.36 [−2.83, −2.03]
CallFriend English (dev)	21.19	20.63	17.47	−3.72 [−5.76, −2.30]
AppTek call-center dialogues, US customers (8 kHz μ-law)	10.56	11.83	8.93	−1.63 [−1.83, −1.44]
Switchboard, 3,000-utterance subset ¹	10.44	10.20	7.54	−2.90 [−3.31, −2.52]

Pooled = Let's Go (re-typed), HarperValley, CallHome, CallFriend and AppTek US scored as one set (Switchboard is not in the pool).

¹ The Switchboard training replay may share speakers with this subset, so Switchboard numbers of the fine-tunes are slightly optimistic; without replay the plain 110m still beat the previous baseline there.

In-domain sets

set	Previous baseline ⁰	Untouched base	This model	this model vs previous baseline (Δ pp, 95% CI)
Held-out in-domain test calls (11,807 segments) ²	8.68	9.19	3.44	−5.25 [−5.52, −4.97]
Held-out in-domain test calls, every speech segment (20,078)	15.54	16.76	8.07	−7.47 [−7.82, −7.14]
In-domain slice A	9.54	10.03	4.23	−5.32 [−5.94, −4.78]
In-domain slice B	9.03	9.60	3.14	−5.89 [−6.39, −5.37]
In-domain slice C	8.73	8.75	3.04	−5.70 [−6.53, −4.90]
In-domain slice D	7.58	8.24	3.14	−4.44 [−4.91, −3.99]
In-domain dev split (used for checkpoint selection)	15.48	16.27	8.03	−7.46 [−8.16, −6.81]
30-call customer set with LLM-corrected references ³	45.46	47.46	40.74	−4.71 [−5.92, −3.77]

² Held-out calls (by call, never seen in training) where the teacher transcript and independent ASR systems agreed. The references are machine transcripts from the same teacher the models learned from, so these numbers overstate the real gain; the public sets above are the honest measure. The slices A–D split the same test set by recording source. ³ 30 real customer calls whose references were produced by an LLM correcting machine transcripts, not by humans: the absolute WER includes reference errors and hard audio, so read the differences, not the level.

General English: LibriSpeech vs the untouched base

set	Untouched base	This model	this model vs the untouched base (Δ pp, 95% CI)
LibriSpeech test-clean, 8 kHz	2.74	3.27	+0.52 [+0.39, +0.67]
LibriSpeech test-other, 8 kHz	6.25	7.30	+1.05 [+0.76, +1.36]
LibriSpeech test-clean, 16 kHz	2.36	2.78	+0.42 [+0.28, +0.56]
LibriSpeech test-other, 16 kHz	5.26	6.03	+0.77 [+0.53, +1.03]

Guard verdict (8 kHz): worse than base — “worse than base” means the 95% interval of the difference lies above zero; “non-inferior” means within max(seed noise, 3% relative) of the base on both sets. The 8 kHz rows are the fair test for a telephone model (the audio goes through the same 8 kHz → 16 kHz path); the 16 kHz rows are LibriSpeech as published.

Behaviour

measure	Previous baseline ⁰	Untouched base	This model
Non-speech segments left empty (of 1,056 held-out), %	98.96	97.63	98.20
Outputs on those non-speech segments (count)	11	25	19
…of which “yes / yeah / okay” (count)	8	6	4
Empty output on real speech, %	2.24	1.85	0.32
WER on 1–3-word answers (in-domain), %	14.97	16.49	4.44
Contractions written joined (“don't”, not “don t”), %	0.1	86.0	94.5
Number-bearing turns written with digits, %	0.0	0.0	0.0
Outputs with punctuation or capitals, %	0.2	94.6	100.0

Non-speech = the 1,056 held-out segments with no speech (line noise, silence, hold, breathing). The fine-tunes spell most numbers as words; they write punctuation and capitals, so strip punctuation before any keyword or phrase matching.

yes/no polarity kept, %	Previous baseline ⁰	Untouched base	This model	this model vs previous baseline
Held-out in-domain test calls	97.69	95.18	99.61	+1.93 [+1.36, +2.55]
30-call customer set	68.61	63.50	69.34	+0.73 [−2.30, +4.13]
Let's Go, re-typed	90.26	88.70	93.74	+3.48 [+1.52, +5.52]
HarperValley Bank	95.85	94.54	97.31	+1.46 [+0.54, +2.64]
CallHome	90.84	88.29	90.81	−0.03 [−0.83, +0.78]
CallFriend	91.52	88.10	92.34	+0.82 [−0.39, +2.05]
AppTek US	97.09	93.26	97.40	+0.31 [−0.96, +1.62]
Switchboard subset	93.44	93.18	97.38	+3.94 [+2.10, +6.04]

Yes/no polarity kept = the share of reference answers of 1–10 words that lead with a yes or a no, whose output leads with the same polarity.

Optional: confidence gate against short outputs on non-speech

Like most transducers, the model sometimes writes a short “Yeah.” or “Okay.” on noise. A simple gate removes most of these: treat an output of at most 2 words whose mean word confidence is below τ as silence. Word confidence = the minimum token confidence in the word (NeMo entropy confidence, Tsallis α = 1/3, exponential normalisation, blank steps excluded). τ was chosen on dev data only, as the largest value that keeps keyword-intent recall at ≥ 99% and costs at most +0.10 pp WER; the table is the held-out test.

τ for this model: 4.56e-05 (mean word confidence; the scale is model-specific, so never reuse another model's τ).

held-out in-domain test	without gate	with gate
non-speech segments left empty (of 1,056)	1037	1052
“yes / yeah / okay” outputs on non-speech	4	0
WER, every speech segment	8.07%	8.19%
yes/no answers whose polarity is kept	98.06%	97.67%
real speech segments turned into silence	0%	1.84%

import copy, numpy as np
from omegaconf import open_dict

cfg = copy.deepcopy(model.cfg.decoding)
with open_dict(cfg):
    cfg.confidence_cfg = {"preserve_frame_confidence": True, "preserve_token_confidence": True,
                          "preserve_word_confidence": False, "exclude_blank": True, "aggregation": "min",
                          "method_cfg": {"name": "entropy", "entropy_type": "tsallis", "alpha": 1 / 3,
                                         "entropy_norm": "exp"}}
    if cfg.get("model_type") == "tdt":
        cfg.preserve_alignments = True          # the 110m TDT needs alignments to map confidence to tokens
model.change_decoding_strategy(cfg)

def gated_text(hyp, tau):
    """'' when the output has <= 2 words and their mean word confidence is below tau (word = min over its tokens)"""
    ids = [int(i) for i in hyp.y_sequence]
    words = []
    for piece, c in zip(model.tokenizer.ids_to_tokens(ids), [float(x) for x in hyp.token_confidence]):
        if piece.startswith("\u2581") or not words:
            words.append(c)
        else:
            words[-1] = min(words[-1], c)
    if 0 < len(hyp.text.split()) <= 2 and words and np.mean(words) < tau:
        return ""
    return hyp.text

hyp = model.transcribe([audio16], batch_size=1, return_hypotheses=True)[0]
print(gated_text(hyp, 4.562e-05))

Latency

measurement	1–3 s clip, median	7–9 s clip, median	peak GPU memory
previous baseline (110m + adapter), RTX 6000 Ada	27.5 ms	44.0 ms	0.50 GB
this model, RTX 6000 Ada	22.9 ms	38.7 ms	0.48 GB

Batch 1, fp32, greedy TDT, CUDA graphs off, transcribe() on an 8 kHz WAV; 200 clips of 1–3 s and 100 of 7–9 s after warm-up. Same size and speed as the stock 110m (slightly faster than the previous baseline, which adds an adapter). On an RTX 4000 SFF Ada (20 GB) the same architecture measured about 23 ms (1–3 s) and 45 ms (~8 s) at the median. Latency class: small and fast; real-time batch-1 decoding on any recent NVIDIA GPU, and usable on CPU.

Privacy and leak check

The training audio is real customer speech. Every model was checked for memorised training content before release. The test inputs contained no customer speech: 83 synthetic clips with no speech at all (digital silence of 0.5–10 s; white, pink and brown noise from −70 to −20 dBFS; 300–3,400 Hz line noise; mains hum; DTMF digits; ring-back, busy, reorder, dial and SIT tones; clicks) and 200 public LibriSpeech clips. Every output was scanned for digit runs, phone numbers, ZIP codes, SSN-like, member-ID-like and date-of-birth patterns, e-mail addresses, street addresses and capitalised first + last name pairs. It was also scanned for any sequence of four or more words that occurs only one to three times in the training transcripts.

• Greedy decoding (as deployed): no text at all on any of the 83 no-speech clips. On LibriSpeech, every pattern or rare-phrase hit was the book text being read, or a misheard word that never occurs in the training text.
• Beam search, beam 8, every hypothesis kept (863 hypotheses): only fragments of one or two words; nothing was flagged.
• Stress test, decoder forced to talk: on the no-speech clips the blank symbol was suppressed at four strengths (332 decodes, outputs of up to 20 words). This exposes the decoder's internal language prior. It produced generic conversational phrases and repetition loops. 6 distinct word sequences of 4–10 words from these outputs occur one to three times verbatim in the training transcripts. All of them are made of common everyday words, and none contains a name-like or rare word.

No audio, transcript or call metadata is published with the model. A leak check lowers the risk of memorised content but cannot prove it is absent; report anything suspicious through the repository's community tab.

Usage

pip install "nemo_toolkit[asr]" librosa huggingface_hub   # tested with NeMo 2.5.3 and 3.0

import librosa
from huggingface_hub import hf_hub_download
from nemo.collections.asr.models import ASRModel

path = hf_hub_download("ozonetg/parakeet-110m-caller-asr-basic", "parakeet-110m-caller-asr-basic.nemo")
model = ASRModel.restore_from(path, map_location="cuda").eval()   # or map_location="cpu"

# 8 kHz telephone audio, mono. Upsample to 16 kHz with librosa's default resampler (soxr_hq):
# that is exactly the evaluated input path (NeMo 2.5.x does the same itself if you pass an 8 kHz WAV path).
audio, _ = librosa.load("call_8k.wav", sr=8000, mono=True)
audio16 = librosa.resample(audio, orig_sr=8000, target_sr=16000)
out = model.transcribe([audio16], batch_size=1)
print(out[0].text if hasattr(out[0], "text") else out[0])

The model is hybrid TDT + CTC; it decodes with the TDT head by default, which is what every number on this card uses. model.change_decoding_strategy(decoder_type="ctc") switches to the CTC head (about 0.3–0.6 pp worse pooled public WER for these models).

Release check: this exact file was restored fresh with NeMo 2.5.3 on CPU and transcribed 40 public LibriSpeech 8 kHz clips (WER 4.96% clean / 4.64% other on that sample); the text matched the evaluation decodes (all 40 identical).

Limitations

• Machine-labelled training data. No human transcribed the training audio; the model learned the teacher's conventions and some of its errors. In-domain test references come from the same teacher (see ²).
• A small general-English cost. See the LibriSpeech table: fine-tuning on phone calls costs a little accuracy on read wideband speech.
• Short outputs on non-speech. On noise or silence it occasionally writes “Yeah.” / “Okay.” (see Behaviour). Use a voice-activity detector, the confidence gate above, or both, where a false “yes” matters.
• Domain. Outbound sales calls in US English, customer side. Agent-side audio, other call types and non-US accents were not the target; the public sets give an idea of the spread.
• Text format. Punctuation and capitals in the output; numbers mostly as words. Normalise text before matching phrases.
• bfloat16 / fp32. Evaluated in fp32.

The family

model	size	what it is	public pooled WER	in-domain WER	LibriSpeech-other 8 kHz vs base	“yes” on non-speech
★ parakeet-110m-caller-asr	110m	best plain retrain (SpecAugment off, weight EMA, ~30 h extra caller audio), blended 0.8 / 0.2 with the base	11.42	3.35	+0.30 pp	3
parakeet-110m-caller-asr-extra-audio	110m	trained on the 0.6B fine-tune's transcripts plus 170 h of extra caller audio, blended 0.7 / 0.3	11.56	3.45	+0.26 pp	1
parakeet-110m-caller-asr-averaged	110m	greedy weight average of three 110m fine-tunes, blended 0.7 / 0.3	11.47	3.48	+0.19 pp	8
parakeet-110m-caller-asr-basic (this model)	110m	plain full fine-tune (the reference recipe)	11.60	3.44	+1.05 pp	4
parakeet-110m-caller-asr-nopunct	110m	plain fine-tune on lowercase text without punctuation	11.59	3.33	+1.06 pp	6
★ parakeet-0.6b-caller-asr	0.6B	fine-tune with layer-wise learning-rate decay 0.9, blended 0.8 / 0.2 with the base	9.89	1.95	+0.05 pp	4
parakeet-0.6b-caller-asr-averaged	0.6B	average of two seeds of the plain 0.6B fine-tune, blended 0.7 / 0.3	9.87	1.85	+0.16 pp	11
parakeet-0.6b-caller-asr-basic	0.6B	plain full fine-tune of the 0.6B (the reference recipe)	9.97	1.96	+0.62 pp	7
parakeet-110m-caller-asr-unmixed	110m	the retrain behind the 110m pick (SpecAugment off, weight EMA, ~30 h extra caller audio), not blended	11.42	3.29	+0.78 pp	1
parakeet-110m-caller-asr-adapter	110m	the stock 110m kept frozen, with a small adapter trained in every encoder layer	11.91	3.80	+0.88 pp	3
parakeet-0.6b-caller-asr-unmixed	0.6B	the layer-wise learning-rate-decay fine-tune behind the 0.6B pick, not blended	9.84	1.98	+0.15 pp	4
parakeet-0.6b-caller-asr-averaged-unmixed	0.6B	average of two seeds of the plain 0.6B fine-tune, not blended	9.84	1.85	+0.44 pp	7
parakeet-0.6b-caller-asr-extra-audio	0.6B	plain 0.6B recipe retrained on its own transcripts of 170 h of extra caller audio (noisy student)	10.06	2.10	+0.62 pp	0
previous baseline (not released): stock 110m + a small domain adapter	110m	–	14.00	8.68	+1.45 pp	8

★ = recommended pick of each size. Full comparison, method and per-set tables: the report Space ozonetg/caller-asr-report.

How to choose: the 110m models need the smallest GPU, run in NeMo 2.5 or 3.0, and are about as fast as the stock 110m; the 0.6B models are more accurate (about 1.5 pp lower pooled public WER) but need NeMo 3.x, 1.5–2× the latency and about 0.7 GB more GPU memory. Within a size, the ★ blend trades a little in-domain accuracy for the best general-English retention.

Licence and attribution

This model is a derivative of nvidia/parakeet-tdt_ctc-110m by NVIDIA, released under the Creative Commons Attribution 4.0 International licence (CC BY 4.0). It is released under the same licence. Changes from the original: every weight was fine-tuned on telephone audio as described above; the architecture, tokenizer and configuration are unchanged. NVIDIA does not endorse this model.

The training data is not released.