---
license: other
license_name: mit-and-apache-2.0
license_link: LICENSE.md
library_name: pytorch
pipeline_tag: automatic-speech-recognition
language:
  - en
  - zh
base_model:
  - openai/whisper-large-v3
  - Qwen/Qwen3-ASR-1.7B-hf
  - Edge0/ARK-ASR-3B
  - HojoAI/Hojo-ASR-V1
tags:
  - speech-recognition
  - hallucination
  - non-speech
  - whisper
  - audio-llm
  - lora
  - robustness
metrics:
  - wer
  - cer
---

# acoustic-abstention

**Learning When to Stay Silent: Acoustic Abstention Encoding for Automatic Speech Recognition**

Aditya Pujari and Ajita Rattani, University of North Texas

Released checkpoints for [acoustic-abstention](https://github.com/pujariaditya/acoustic-abstention). Acoustic
Abstention Encoding (AAE) fine-tunes a pretrained ASR model's audio encoder while its text decoder stays **frozen**: an
abstention loss makes the decoder return an empty transcript for non-speech, and a feature-preservation loss keeps the
encoder's features of speech-containing audio close to the original encoder's. Training needs no speech transcripts.
The LoRA adapters, in the last half of the encoder blocks and, for the audio-LLMs, in the audio projector, are merged
into the weights: a checkpoint has the original architecture and parameter count, and decodes with the same procedure
as the original model. Every number below is measured on complete evaluation sets.

Hallucination rate (HR, %: non-speech segments that receive any text) on the complete non-speech sets, original model →
with AAE (Whisper large-v3: 9,250 steps; Hojo-ASR-V1: the run with higher rates):

| model | UrbanSound8K (8,732) | WHAM! noise (8,000) | filtered FSD50K (6,414) |
|---|---|---|---|
| Whisper large-v3 | 100.00 → **0.34** | 100.00 → **0.00** | 100.00 → **0.02** |
| Qwen3-ASR-1.7B | 2.68 → **0.11** | 9.20 → **0.03** | 0.59 → **0.12** |
| ARK-ASR-3B | 99.86 → **0.11** | 100.00 → **0.00** | 99.81 → **0.03** |
| Hojo-ASR-V1 | 99.99 → **0.18** | 100.00 → **0.00** | 99.98 → **0.14** |

Speech error rate on the complete test sets (WER; CER for the Mandarin sets), original model → with AAE, and *same*:
the share of the model's 15,680 normalized transcripts identical to the original model's:

| model | LibriSpeech clean | other | tail | 5 dB | FLEURS Mandarin | AISHELL-1 | same |
|---|---|---|---|---|---|---|---|
| Whisper large-v3 | 2.15 → 2.15 | 4.02 → 4.07 | 3.44 → 3.57 | 5.17 → 5.62 | 7.60 → 7.72 | 8.10 → 8.10 | 97.0 |
| Qwen3-ASR-1.7B | 1.71 → 1.71 | 3.47 → 3.53 | 2.61 → 2.63 | 4.02 → 3.86 | 7.19 → 7.44 | 1.53 → 1.50 | 97.7 |
| ARK-ASR-3B | 1.43 → 1.43 | 2.84 → 2.84 | 2.01 → 2.07 | 3.96 → 3.98 | 8.76 → 8.75 | 1.83 → 1.83 | 99.0 |
| Hojo-ASR-V1 | 1.79 → 1.80 | 4.01 → 4.07 | 3.04 → 3.06 | 5.71 → 5.78 | 15.22 → 15.44 | 1.71 → 1.71 | 96.6 |

*tail*: a LibriSpeech test utterance, 1 s of silence, then 3 s of noise; *5 dB*: noise added at 5 dB SNR.

Ablation study on the complete sets, every row at 8,500 steps (Whisper large-v3: `aae-8500`; Hojo-ASR-V1: the other
run, `aae-run1`): mean HR over the three non-speech sets (%) / *same* (%), and in parentheses the number of the six
speech sets whose error rate exceeds the original model's by more than the tolerance fixed in advance (+0.3 WER
test-clean, +0.5 test-other, +1.0 on each noisy set, +1.0 CER on each Mandarin set); bold: every speech set within its
tolerance.

| variant | Whisper large-v3 | Qwen3-ASR-1.7B | ARK-ASR-3B | Hojo-ASR-V1 |
|---|---|---|---|---|
| AAE | **0.35 / 96.6** | **0.09 / 97.7** | **0.05 / 99.0** | **0.06 / 96.6** |
| λ = 0: no preservation loss | 0.00 / 0.0 (6) | 0.00 / 0.0 (6) | 0.00 / 1.5 (6) | 0.00 / 0.0 (6) |
| CE on m: transcript cross-entropy instead | 0.18 / 70.2 (2) | 0.04 / 81.3 (3) | 0.12 / 85.6 (2) | 0.02 / 78.3 (2) |
| indep. noise: another segment in m | **0.03 / 92.5** | **0.08 / 97.8** | **0.39 / 99.2** | **0.04 / 96.6** |
| dec. adapters: CE-trained, in the decoder | **1.09 / 76.4** | 0.35 / 85.0 (2) | **0.05 / 87.3** | 0.06 / 81.1 (3) |

## Checkpoints

Twenty-four checkpoints: the released model for each ASR model, the four variants of the ablation study for each,
the two full-recipe rows of the ablation that differ from the released model (the other Hojo-ASR-V1 run, `aae-run1`,
and Whisper large-v3 at 8,500 steps, `aae-8500`), and the decoder-adaptation control: Whisper large-v3's decoder adapted
at AAE's parameter count (LoRA of rank 18 in the decoder's self-attention) with a KL term at two weights (`decoder-kl`,
lambda = 50, and `decoder-kl-1`, lambda = 1). Together they cover every evaluation behind
the paper and the repository's results.

| folder | what it is | adapter parameters |
|---|---|---|
| `whisper-large-v3/aae/` | the full recipe | 5.90M |
| `whisper-large-v3/aae-8500/` | Whisper large-v3 only: the aae run at 8,500 steps, the length of every variant (the full-recipe row of the ablation study); aae continues the same run to 9,250 steps | 5.90M |
| `whisper-large-v3/no-preservation/` | preservation loss removed (lambda = 0) | 5.90M |
| `whisper-large-v3/transcript-ce/` | preservation loss replaced by cross-entropy on m toward the original model's transcript of s | 5.90M |
| `whisper-large-v3/independent-noise/` | the noise in m drawn independently of n | 5.90M |
| `whisper-large-v3/decoder-adapter/` | rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m | 2.95M |
| `whisper-large-v3/decoder-kl/` | Whisper large-v3 only: rank-18 LoRA on the decoder self-attention, as many parameters as AAE's, with the KL divergence to the original model's next-token distributions on m (lambda = 50) | 5.90M |
| `whisper-large-v3/decoder-kl-1/` | Whisper large-v3 only: the same decoder edit with the KL weight lambda = 1 | 5.90M |
| `qwen3-asr-1.7b/aae/` | the full recipe | 3.62M |
| `qwen3-asr-1.7b/no-preservation/` | preservation loss removed (lambda = 0) | 3.62M |
| `qwen3-asr-1.7b/transcript-ce/` | preservation loss replaced by cross-entropy on m toward the original model's transcript of s | 3.62M |
| `qwen3-asr-1.7b/independent-noise/` | the noise in m drawn independently of n | 3.62M |
| `qwen3-asr-1.7b/decoder-adapter/` | rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m | 3.61M |
| `ark-asr-3b/aae/` | the full recipe | 6.14M |
| `ark-asr-3b/no-preservation/` | preservation loss removed (lambda = 0) | 6.14M |
| `ark-asr-3b/transcript-ce/` | preservation loss replaced by cross-entropy on m toward the original model's transcript of s | 6.14M |
| `ark-asr-3b/independent-noise/` | the noise in m drawn independently of n | 6.14M |
| `ark-asr-3b/decoder-adapter/` | rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m | 4.15M |
| `hojo-asr-v1/aae/` | the full recipe | 7.29M |
| `hojo-asr-v1/aae-run1/` | Hojo-ASR-V1 only: the first run of the full recipe (the full-recipe row of the ablation study) | 7.29M |
| `hojo-asr-v1/no-preservation/` | preservation loss removed (lambda = 0) | 7.29M |
| `hojo-asr-v1/transcript-ce/` | preservation loss replaced by cross-entropy on m toward the original model's transcript of s | 7.29M |
| `hojo-asr-v1/independent-noise/` | the noise in m drawn independently of n | 7.29M |
| `hojo-asr-v1/decoder-adapter/` | rank-9 LoRA on the decoder self-attention instead of the encoder, with transcript CE on m | 6.64M |

Each folder holds `model.safetensors`, the merged float16 weights of every tensor the adapter changed (the exact
tensors that were evaluated), and `training.json`, the training arguments, loss curve and held-out curve.
`hojo-asr-v1/aae` is the second of two runs of the recipe; the first, `aae-run1`, is the full-recipe row of the
ablation study because its sampling matches the variants. `whisper-large-v3/aae` trained for 9,250 steps;
`whisper-large-v3/aae-8500` is the same run at 8,500 steps, the length of every variant, and is that model's
full-recipe row.

## Use

```bash
git clone https://github.com/pujariaditya/acoustic-abstention && cd acoustic-abstention
./setup.sh                                   # two environments + the four original ASR models, pinned
envs/main/bin/python scripts/download_weights.py     # these checkpoints, sha256-verified
```

Applying a checkpoint copies its tensors into the original model:

```python
from aae import models, weights
ad = models.load("whisper-large-v3")                           # original model, pinned revision
weights.apply(ad.model, weights.resolve("whisper-large-v3/aae"))
print(ad.transcribe([waveform_16k]))                           # [''] for non-speech
```

The Whisper checkpoints use the parameter names of openai-whisper's `large-v3.pt` (sha256 e5b1a55b...), not the
transformers layout. The other base models are pinned to Qwen/Qwen3-ASR-1.7B-hf @ bcd2b5b7, Edge0/ARK-ASR-3B @
1e28271b and HojoAI/Hojo-ASR-V1 @ a22c3818. `./scripts/reproduce.sh` re-runs every evaluation behind the paper and
checks each number against the paper's value.

## Intended use and limitations

Research artifact for reproducing the reported results. The recipe preserves each original model's speech behavior
rather than improving it. When every utterance of LibriSpeech test-other and FLEURS Mandarin test is mixed with
training-pool noise at -5 dB SNR, 2.4% to 6.0% receive an
empty output with AAE, against at most 1.0% for the original models; at 5 and 15 dB,
at most 0.15% do. The noisy speech test sets use noise from the training pools. One
training run per model and variant (two runs of the full recipe for Hojo-ASR-V1, both released).

## Licenses

Each checkpoint is a derivative of its base model and is distributed under that model's license: Whisper large-v3
(MIT), Qwen3-ASR-1.7B, ARK-ASR-3B and Hojo-ASR-V1 (Apache-2.0); `LICENSE.md` has the folder-by-folder terms and the
full texts. The checkpoints were trained on audio from WHAM! noise, FSD50K, MUSAN, ESC-50, DEMAND, LibriSpeech, FLEURS
and AISHELL-1, each under its own terms; some are non-commercial (ESC-50 is CC BY-NC 3.0, and FSD50K clips carry
per-clip Creative Commons licenses, some of them CC BY-NC). The code is MIT.