erickvs/whisper-tiny.en-hand-edited

Whisper tiny.en, hand-edited to stop writing words on silence and everyday sounds

An unofficial, hand-edited version of openai/whisper-tiny.en. It wasn't retrained: 6,152 of its 37.8 million weights were changed by hand, in two rounds.

When nobody is speaking, the original writes words anyway:

• over silence or noise, it's almost always "you";
• over real sounds like a crying baby, laughter or a cat, it writes things like "AHHHH…", "Thank you." or "Ha ha ha";
• under the pipeline's beam search, it often writes looping sentences.

This version writes a lone "." instead. Its transcripts of real speech are as accurate as before.

Held-out audio with nobody speaking	The original	Round 1	This version (round 2)
Generated noise: silence, hiss, hum, tones and more	89.7%	0.0%	0.0%
Real recordings of everyday sounds (ESC-50)	96.5%	21.0%	3.75%
Sounds nobody tested during editing: music, alerts and new noise types	97.3%	2.9%	0.7%

Each cell is the share of clips with any words, with greedy decoding. 5-beam search tells the same story (see Results).

Round 1, the first release, is still available as revision round-1. It was perfect on generated noise but missed many real animal and human sounds.

One pre-registered check was missed, narrowly.

• The check was speech with loud sounds playing underneath, under 5-beam search. The upper end of the WER interval came out at +0.27 points, against a limit of +0.25 set before any edit existed.
• The cause is one 11-word sentence under a vacuum cleaner at 0 dB. The original garbles it, and this version loops on it.
• Round 1 loops on it identically and misses the same check by the same amount. So on speech this version is no worse than round 1.
• It was published on the owner's call, with this stated (see Limits).

Made with WSP Lab, coming soon: join the waitlist at wsp-lab.ai

WSP Lab is a tool that lets AI agents edit a model's weights directly, quickly and verifiably. These edits used its approach, with a purpose-built harness.

Watch how the edit works

What Whisper does, then from the whole model down to the one unit that was rewritten: what the numbers were, why they were changed, what that should do, what it did, and the model writing one word piece at a time. 5:59, narrated, with subtitles. If the player doesn't load, the file is videos/explainer.mp4.

How long it took

Round 2 (this version), on 7 November 2026:

Step	Wall-clock time	Done by
Plan, test harness and a short investigation of the model	23 min	Claude Opus 5.5
The edit, from brief to a tested edit	94 min	Claude Fable 5.1, one of three editors working in parallel
Waiting for the slowest of the three editors	20 min
Choosing between the edits on the selection split	47 min	Claude Opus 5.5
Scoring them on the final split	45 min	Claude Opus 5.5
Independent audit	9 min	Claude Opus 5.5
Deciding to publish despite the narrow miss	24 min	the model's owner, after being asked
Release	5 min to this upload	Claude Opus 5.5

• Total for round 2: 4 hours 21 minutes from the owner's request (18:50:22) to the upload (23:11:36), on one laptop (an Apple M5 Max), with no training. It was 4 hours 25 minutes to the download check below, which finished at 23:15:22.
  • Scoring took about as long as editing, because every edit was checked on 14 speech sets and 4 non-speech sets.
  • The three editors and the scoring jobs shared one GPU.
• Round 1 took 2 hours 19 minutes, from the start of its run to its own download check.
• Download check: downloaded with no login into an empty cache, it wrote no words on 400 fresh noise clips with either decoding. On 400 real recordings, its greedy transcripts matched the experiment's exactly, as did those of 100 utterances.
• Budget: the edit used 30 of its 60 test runs. Most of its 94 minutes went to the agent's reasoning and to sharing the GPU.
• Timing source: every time above comes from the experiment's own logs and file timestamps.

Results

All numbers are on held-out data that the editors never saw:

• Real recordings: ESC-50's fold 5, 400 five-second recordings of 50 kinds of everyday sound;
• Generated noise: seeded 10-second clips of 8 kinds, 4,000 with greedy decoding and 1,000 with beams;
• Speech: 820 LibriSpeech test-clean utterances (items 1,800 to 2,619, sorted by id), scored with Whisper's English text normaliser.

A clip counts as having words if its transcript contains any letter or digit.

Decoding	Model	Real recordings with words	Generated noise with words	LibriSpeech WER
Greedy, which model.generate() uses	This version	3.75%	0.0%	6.369%
	Round 1	21.0%	0.0%	6.375%
	The original	96.5%	89.7%	6.375%
5-beam search, which pipeline() uses by default	This version	3.75%	0.0%	5.506%
	Round 1	20.25%	0.0%	5.500%
	The original	86.25%	23.0%	5.531%

• Real recordings: against round 1, −17.25 points with greedy decoding (−21.0 to −13.75) and −16.5 with beams (−20.25 to −13.0).

  • 15 of the 400 recordings still get words: 5 cats, 3 sneezes, 3 crying babies, 2 car horns, a frog and a crow.
  • Every recording of rain, wind, fire, water, engines, machines, laughter and coughing gets ".".

• Clean speech, paired against the original on the same utterances, with 95% bootstrap intervals:

  • greedy: −0.006 points (−0.042 to +0.025), with 9 of 820 transcripts changed;
  • 5-beam search: −0.025 points (−0.059 to 0.000).

• Harder speech: six more versions of the 820 utterances:

  • quieted by 26 dB;
  • after 8 seconds of silence;
  • in white noise at 10 dB SNR;
  • with an ESC-50 recording underneath at 10, 5 and 0 dB SNR.

  Pooled over all six, against the original:

  • greedy: +0.026 points (−0.021 to +0.078);
  • 5-beam search: +0.094 points (−0.008 to +0.270). That's the narrow miss.

  Under both decodings, 2 of the 820 quiet utterances came out as ".". They were "Quick, quick." and "Indeed.", the same two as round 1.

Sounds nobody tested during editing. There were 592 clips:

• 200 instrumental music loops;
• the 14 macOS alert sounds, each at 3 positions in silence;
• 350 clips of 7 new noise types: pink, brown, chirps, square waves, band-passed, modulated and speech-shaped.

Decoding	The original	Round 1	This version	Music	Alert sounds	New noise types
Greedy	97.3%	2.9%	0.7%	0.5%	7.1%	0%
5-beam search	65.9%	1.0%	0.7%	0.5%	7.1%	0%

The last three columns are this version's rate on each kind of sound.

Measured on round 1 only. This version contains round 1's edit unchanged. Its new parts act only when the model is about to caption a sound, at the first two tokens. These tests weren't re-run for it:

• Timestamp mode (return_timestamps=True): words on non-speech fell from 49.1% to 0%, and WER changed by −0.037 points.

• Long recordings. There were 273 recordings of about 2.5 minutes, each with 3 LibriSpeech utterances and 5 to 60 seconds of generated noise or silence around them, plus 50 recordings of 90 seconds of non-speech only.

  Decoding	Non-speech recordings with words	Inserted words per 100 spoken	WER
  pipeline(chunk_length_s=30)	34% → 4%	29.5 → 1.0	40.2% → 11.2%
  pipeline(return_timestamps=True)	100% → 6%	853 → 106	859% → 112%
  generate(return_timestamps=True), greedy	94% → 6%	12.8 → 12.2	19.4% → 18.7%

  Each cell is the original, then round 1. It missed no more of the spoken words in any of the three. For long audio, use chunk_length_s=30. Sequential decoding through the pipeline loops badly with either model.

Compared with a rule in the decoder

The original already computes a no-speech signal: its probability of "no captions" (token 50361) at the first decoding position. OpenAI's own transcribe() reads it to skip silent segments. So the original was also scored with two rules in the decoder, on the same held-out data:

Model	Real recordings with words, greedy	5-beam search	Generated noise, greedy	5-beam search	Unseen sounds, greedy	5-beam search
This version	3.75%	3.75%	0.0%	0.0%	0.7%	0.7%
The original, skipping when that probability is above 0.5	20.75%	19.75%	0.1%	0.0%	2.5%	2.4%
The original with OpenAI's default rule: above 0.6, with an average log-probability below −1.0	42.75%	62.75%	13.6%	15.7%	14.9%	35.6%
The original	96.5%	86.25%	89.7%	23.0%	97.3%	65.9%

• On real recordings, this version now beats the plain threshold by far.
  • On many animal and human sounds the original's no-speech probability is no higher than on some real speech: up to 0.42 on development speech after silence or under loud sounds.
  • So a threshold low enough to catch those sounds would also skip speech.
  • Round 2's edit reads other signals inside the model, which the threshold doesn't see.
• On generated noise the threshold does about as well, and it outputs an empty string rather than ".". In round 1's checks it skipped no real speech.
• OpenAI's default rule does much worse. It also needs the transcript to look unconfident, and these hallucinations often aren't.
• transformers 5.22.2's built-in version, generate(no_speech_threshold=0.6, logprob_threshold=-1.0), fails with greedy decoding.
• This version needs no code at all: it works in pipeline() and generate() as they are.

What was changed

Round 1 changed 3,845 numbers, all still in place. They carry the original's own "no captions" prediction from the first position to the first word, in three steps:

1. Detect. Two unused units in decoder layer 2's MLP (267 and 718) form a soft step. It reads the "no captions" direction at the first position and writes a marker when it fires.
2. Carry. One value channel (dimension 145) of attention head 2 in decoder layer 3 carries the marker forward.
3. Act. Two unused units in layer 3's MLP (1426 and 621) raise " ." and end-of-text when the marker arrives.

That works on generated noise. But on real animal and human sounds the "no captions" prediction is unsure, so round 1 misses many of them.

Round 2 added 2,307 numbers. They read units the model already has that react to such sounds:

1. First token. Units 10 and 1100 of decoder layer 3's MLP fire at the first-word position on sounds like crying, barking or laughter, and never as high on speech.
   • Unit 10's own output lowers " (", the opener of captions like "(laughing)". Whisper's decoding config suppresses those anyway.
   • A spare unit (1354) reads the two units' sum. Above a threshold, it makes " ." the most likely first token.
2. Second token. Units 1241 and 1248 of layer 2 fire after that first " ." on such sounds, but at no point in speech. Two spare units (1112 and 1335) read their sum and set round 1's marker, so round 1's last step ends the transcript.

Every direction comes from the model's own weights, and every number was chosen by hand. An ablation on the held-out recordings confirms the first step does the work: without it, the rate goes back to round 1's 21%.

A trap in beam search, found independently by two of the editors:

• transformers' beam search scores tokens by a log-softmax over the whole vocabulary before masking the suppressed tokens, and doesn't renormalise.
• So an edit that boosts a blocked token, such as end-of-text at the first position, quietly penalises every beam, and long confident repetitions win.
• Round 2's first-token push avoids blocked tokens.

edit.safetensors holds just the 9 changed tensors. model.safetensors is the complete model.

Usage

import numpy as np
from transformers import pipeline

asr = pipeline("automatic-speech-recognition", model="erickvs/whisper-tiny.en-hand-edited")

print(asr("speech.wav")["text"])  # transcribes as usual
print(asr("long_recording.wav", chunk_length_s=30)["text"])  # for audio over 30 seconds

noise = (0.05 * np.random.default_rng(0).standard_normal(16000 * 10)).astype(np.float32)  # 10 s of white noise
print(asr({"raw": noise, "sampling_rate": 16000})["text"])  # " ."; the original writes " I'm going to do it. I'm going to do it. ..."

round_1 = pipeline("automatic-speech-recognition", model="erickvs/whisper-tiny.en-hand-edited", revision="round-1")

To apply just the edit to the original weights:

from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
from transformers import WhisperForConditionalGeneration

model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en")
edit = load_file(hf_hub_download("erickvs/whisper-tiny.en-hand-edited", "edit.safetensors"))
model.load_state_dict(edit, strict=False)  # replaces 9 tensors and leaves the rest untouched

It was tested with transformers 5.22.2 and torch 2.15.0, on Apple's MPS backend, in float32.

Limits

• It writes "." rather than nothing. Whisper's generation config blocks end-of-text as the first token, so the shortest possible transcript is one token. Strip a lone "." if you need an empty string.
• Real sounds. It still writes words on 3.75% of held-out recordings, mostly cats, sneezes and crying babies.
  • Its editors tuned it on other recordings of the same 50 kinds of sound, ESC-50's folds 1 to 3. Kinds of sound outside ESC-50 may do worse.
  • Crowds, rooms and music with singing haven't been tested.
• Loops under loud noise with beam search: the narrow miss.
  • Under 5-beam search, with a loud sound underneath at 0 dB, one of 820 test utterances fell into a repeating loop ("…who are you, who are you…"). The original only garbled it.
  • Round 1 does exactly the same. With greedy decoding, both versions repeat a short phrase four times on it.
• Rare false alarms. 2 of 820 quiet one- or two-word replies came out as ".", as in round 1. The first-position decision has a small margin on short, quiet or very noisy speech.
• Speech coverage: read English from LibriSpeech only, clean, quiet, after silence, or with white noise or real recordings underneath. Other accents, conversational speech, speech with laughter in it, children's voices and singing haven't been tested.
• One base model: the numbers were chosen for whisper-tiny.en, and they won't carry over to other Whisper sizes.
• Comparison: Calm-Whisper (2025) cut such hallucinations by more than 80% by fine-tuning. These edits weren't compared with it on the same data.

How it was made

The model comes from an experiment in editing neural networks by hand, with no training.

• The rules: every editor was an AI agent (Claude Fable 5.1, by Anthropic), working independently under the same fixed rules. They banned gradients, optimizers, fitting weights to data, automated search and weights from other models.
• The splits: editors saw only development data. A separate selection split chose between the finished edits, and a final split, which no editor could run, gave one final score.
• Round 1: three editors worked on generated noise, with 40 test runs each.
• Round 2: three new editors had real recordings (ESC-50 folds 1 to 3) in their development data, and 60 test runs each.
  • The bar, set before any edit existed, had to hold under greedy decoding and again under 5-beam search:
    • halve round 1's rate on held-out real recordings;
    • keep generated noise at or below 1%;
    • keep clean and harder speech no worse than the original's.
  • Selection picked editor 4's edit, the only one that kept speech safe on the selection split. The other two each cut one held-out sentence short.
  • On the final split it met every condition except the narrow miss above.

Both releases departed from the pre-registration, and both departures are declared:

• Round 1: this repository published a different editor's edit than the rule picked, because the pick broke under beam search.
• Round 2: this version was published despite the narrow miss, on the owner's call, because round 1 misses the same check the same way.

An independent audit recomputed every headline number from the raw transcripts. It also checked that each edit file holds only what its author described, and tested the explanations with ablations on held-out data. The full write-up, including the other four edits and everything that went wrong, is in the experiment notes.

Support the experiments

These experiments run on one laptop, but the AI agents that make and audit the edits cost real money: the six editors behind this release, over two rounds, used about 2.1 million tokens. If you'd like more models fixed by hand like this one, published with their full results and limits, you can support the experiments on Patreon. It pays for the tokens behind the next ones.

Credits and license

Derived from openai/whisper-tiny.en by OpenAI (Robust Speech Recognition via Large-Scale Weak Supervision), and released under the same Apache License 2.0. The license text is in LICENSE, and NOTICE lists the modifications.

• LibriSpeech (CC BY 4.0) was used for evaluation.
• ESC-50 (K. J. Piczak, 2015, CC BY-NC 3.0) was used for evaluation. Round 2's editors also used its folds 1 to 3 as development data, looking at the model's activations on them to choose thresholds by hand.
• No ESC-50 audio is included here.

This model isn't affiliated with or endorsed by OpenAI.