acoustic-abstention

Learning When to Stay Silent: Acoustic Abstention Encoding for Automatic Speech Recognition

Aditya Pujari and Ajita Rattani, University of North Texas

Released checkpoints for acoustic-abstention. Acoustic Abstention Encoding (AAE) fine-tunes a pretrained ASR model's audio encoder while its text decoder stays frozen: an abstention loss makes the decoder return an empty transcript for non-speech, and a feature-preservation loss keeps the encoder's features of speech-containing audio close to the original encoder's. Training needs no speech transcripts. The LoRA adapters, in the last half of the encoder blocks and, for the audio-LLMs, in the audio projector, are merged into the weights: a checkpoint has the original architecture and parameter count, and decodes with the same procedure as the original model. Every number below is measured on complete evaluation sets.

Hallucination rate (HR, %: non-speech segments that receive any text) on the complete non-speech sets, original model → with AAE (Whisper large-v3: 9,250 steps; Hojo-ASR-V1: the run with higher rates):

model	UrbanSound8K (8,732)	WHAM! noise (8,000)	filtered FSD50K (6,414)
Whisper large-v3	100.00 → 0.34	100.00 → 0.00	100.00 → 0.02
Qwen3-ASR-1.7B	2.68 → 0.11	9.20 → 0.03	0.59 → 0.12
ARK-ASR-3B	99.86 → 0.11	100.00 → 0.00	99.81 → 0.03
Hojo-ASR-V1	99.99 → 0.18	100.00 → 0.00	99.98 → 0.14

Speech error rate on the complete test sets (WER; CER for the Mandarin sets), original model → with AAE, and same: the share of the model's 15,680 normalized transcripts identical to the original model's:

model	LibriSpeech clean	other	tail	5 dB	FLEURS Mandarin	AISHELL-1	same
Whisper large-v3	2.15 → 2.15	4.02 → 4.07	3.44 → 3.57	5.17 → 5.62	7.60 → 7.72	8.10 → 8.10	97.0
Qwen3-ASR-1.7B	1.71 → 1.71	3.47 → 3.53	2.61 → 2.63	4.02 → 3.86	7.19 → 7.44	1.53 → 1.50	97.7
ARK-ASR-3B	1.43 → 1.43	2.84 → 2.84	2.01 → 2.07	3.96 → 3.98	8.76 → 8.75	1.83 → 1.83	99.0
Hojo-ASR-V1	1.79 → 1.80	4.01 → 4.07	3.04 → 3.06	5.71 → 5.78	15.22 → 15.44	1.71 → 1.71	96.6

tail: a LibriSpeech test utterance, 1 s of silence, then 3 s of noise; 5 dB: noise added at 5 dB SNR.

Ablation study on the complete sets, every row at 8,500 steps (Whisper large-v3: aae-8500; Hojo-ASR-V1: the other run, aae-run1): mean HR over the three non-speech sets (%) / same (%), and in parentheses the number of the six speech sets whose error rate exceeds the original model's by more than the tolerance fixed in advance (+0.3 WER test-clean, +0.5 test-other, +1.0 on each noisy set, +1.0 CER on each Mandarin set); bold: every speech set within its tolerance.

variant	Whisper large-v3	Qwen3-ASR-1.7B	ARK-ASR-3B	Hojo-ASR-V1
AAE	0.35 / 96.6	0.09 / 97.7	0.05 / 99.0	0.06 / 96.6
λ = 0: no preservation loss	0.00 / 0.0 (6)	0.00 / 0.0 (6)	0.00 / 1.5 (6)	0.00 / 0.0 (6)
CE on m: transcript cross-entropy instead	0.18 / 70.2 (2)	0.04 / 81.3 (3)	0.12 / 85.6 (2)	0.02 / 78.3 (2)
indep. noise: another segment in m	0.03 / 92.5	0.08 / 97.8	0.39 / 99.2	0.04 / 96.6
dec. adapters: CE-trained, in the decoder	1.09 / 76.4	0.35 / 85.0 (2)	0.05 / 87.3	0.06 / 81.1 (3)

Checkpoints

Twenty-four checkpoints: the released model for each ASR model, the four variants of the ablation study for each, the two full-recipe rows of the ablation that differ from the released model (the other Hojo-ASR-V1 run, aae-run1, and Whisper large-v3 at 8,500 steps, aae-8500), and the decoder-adaptation control: Whisper large-v3's decoder adapted at AAE's parameter count (LoRA of rank 18 in the decoder's self-attention) with a KL term at two weights (decoder-kl, lambda = 50, and decoder-kl-1, lambda = 1). Together they cover every evaluation behind the paper and the repository's results.

folder	what it is	adapter parameters
whisper-large-v3/aae/	the full recipe	5.90M
whisper-large-v3/aae-8500/	Whisper large-v3 only: the aae run at 8,500 steps, the length of every variant (the full-recipe row of the ablation study); aae continues the same run to 9,250 steps	5.90M
whisper-large-v3/no-preservation/	preservation loss removed (lambda = 0)	5.90M
whisper-large-v3/transcript-ce/	preservation loss replaced by cross-entropy on m toward the original model's transcript of s	5.90M
whisper-large-v3/independent-noise/	the noise in m drawn independently of n	5.90M
whisper-large-v3/decoder-adapter/	rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m	2.95M
whisper-large-v3/decoder-kl/	Whisper large-v3 only: rank-18 LoRA on the decoder self-attention, as many parameters as AAE's, with the KL divergence to the original model's next-token distributions on m (lambda = 50)	5.90M
whisper-large-v3/decoder-kl-1/	Whisper large-v3 only: the same decoder edit with the KL weight lambda = 1	5.90M
qwen3-asr-1.7b/aae/	the full recipe	3.62M
qwen3-asr-1.7b/no-preservation/	preservation loss removed (lambda = 0)	3.62M
qwen3-asr-1.7b/transcript-ce/	preservation loss replaced by cross-entropy on m toward the original model's transcript of s	3.62M
qwen3-asr-1.7b/independent-noise/	the noise in m drawn independently of n	3.62M
qwen3-asr-1.7b/decoder-adapter/	rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m	3.61M
ark-asr-3b/aae/	the full recipe	6.14M
ark-asr-3b/no-preservation/	preservation loss removed (lambda = 0)	6.14M
ark-asr-3b/transcript-ce/	preservation loss replaced by cross-entropy on m toward the original model's transcript of s	6.14M
ark-asr-3b/independent-noise/	the noise in m drawn independently of n	6.14M
ark-asr-3b/decoder-adapter/	rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m	4.15M
hojo-asr-v1/aae/	the full recipe	7.29M
hojo-asr-v1/aae-run1/	Hojo-ASR-V1 only: the first run of the full recipe (the full-recipe row of the ablation study)	7.29M
hojo-asr-v1/no-preservation/	preservation loss removed (lambda = 0)	7.29M
hojo-asr-v1/transcript-ce/	preservation loss replaced by cross-entropy on m toward the original model's transcript of s	7.29M
hojo-asr-v1/independent-noise/	the noise in m drawn independently of n	7.29M
hojo-asr-v1/decoder-adapter/	rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m	6.64M

Each folder holds model.safetensors, the merged float16 weights of every tensor the adapter changed (the exact tensors that were evaluated), and training.json, the training arguments, loss curve and held-out curve. hojo-asr-v1/aae is the second of two runs of the recipe; the first, aae-run1, is the full-recipe row of the ablation study because its sampling matches the variants. whisper-large-v3/aae trained for 9,250 steps; whisper-large-v3/aae-8500 is the same run at 8,500 steps, the length of every variant, and is that model's full-recipe row.

Use

git clone https://github.com/pujariaditya/acoustic-abstention && cd acoustic-abstention
./setup.sh                                   # two environments + the four original ASR models, pinned
envs/main/bin/python scripts/download_weights.py     # these checkpoints, sha256-verified

Applying a checkpoint copies its tensors into the original model:

from aae import models, weights
ad = models.load("whisper-large-v3")                           # original model, pinned revision
weights.apply(ad.model, weights.resolve("whisper-large-v3/aae"))
print(ad.transcribe([waveform_16k]))                           # [''] for non-speech

The Whisper checkpoints use the parameter names of openai-whisper's large-v3.pt (sha256 e5b1a55b...), not the transformers layout. The other base models are pinned to Qwen/Qwen3-ASR-1.7B-hf @ bcd2b5b7, Edge0/ARK-ASR-3B @ 1e28271b and HojoAI/Hojo-ASR-V1 @ a22c3818. ./scripts/reproduce.sh re-runs every evaluation behind the paper and checks each number against the paper's value.

Intended use and limitations

Research artifact for reproducing the reported results. The recipe preserves each original model's speech behavior rather than improving it. When every utterance of LibriSpeech test-other and FLEURS Mandarin test is mixed with training-pool noise at -5 dB SNR, 2.4% to 6.0% receive an empty output with AAE, against at most 1.0% for the original models; at 5 and 15 dB, at most 0.15% do. The noisy speech test sets use noise from the training pools. One training run per model and variant (two runs of the full recipe for Hojo-ASR-V1, both released).

Licenses

Each checkpoint is a derivative of its base model and is distributed under that model's license: Whisper large-v3 (MIT), Qwen3-ASR-1.7B, ARK-ASR-3B and Hojo-ASR-V1 (Apache-2.0); LICENSE.md has the folder-by-folder terms and the full texts. The checkpoints were trained on audio from WHAM! noise, FSD50K, MUSAN, ESC-50, DEMAND, LibriSpeech, FLEURS and AISHELL-1, each under its own terms; some are non-commercial (ESC-50 is CC BY-NC 3.0, and FSD50K clips carry per-clip Creative Commons licenses, some of them CC BY-NC). The code is MIT.