BuzzASR

/

danish

Like 1

Follow

lemn-lab 10

Hi Lars,

Thanks for bringing this up! it's a genuinely interesting piece of analysis, and we haven't specifically studied what the encoder does with the padded region for short utterances, so this is new to us.
So from a viewpoint, Whisper's encoder is also fully bidirectional, which is part of why it's awkward for real-time/low-latency use in general. Technically, there's no structural reason the padding positions should stay "empty" imho, they're free to accumulate whatever the attention heads find useful from the real frames, and a narrow domain like short voice commands should plausibly compress that into a low-rank subspace. The layer-8 numbers you're reporting (cosine ~0.997, RMSE ~0.042, consistent across all 20 utterances) are a nice, tight result for that setting.
I also think your experiment setting right now with predicting 48 coefficients from a 500-position encoder state is a much smaller and more tractable problem than reconstructing the tail directly, therefore, the direction you're heading in next (layers 9–32 + decoder on the spliced state) seems like the right next check.

A few things we'd be curious about before this generalizes beyond the voice-control setting:

1. Does the same 6-spatial/8-temporal basis hold on longer, more varied utterances (e.g. our FLEURS/Common Voice eval sets), or is the low-rank structure specific to the short, constrained-vocabulary commands you tested on?
2. Is the basis language-specific, or does it transfer across the other BuzzASR language models, given they all share Whisper's architecture and only differ in fine-tuning?
3. How does the predicted (non-oracle) version hold up once you have it same 14/15 or 15/15 exact-match rate, or does it degrade?

I'd be glad to take a look at your scripts and methodology if you're up for sharing them happy to help verify or extend this on our end if it holds up on a broader eval set.

Thanks again for digging into this and writing it up so clearly.

Best,
Shivam

Hi Shivam,
Thanks — this is very helpful, and particularly interesting to hear that the padded-region behaviour hasn't been specifically studied by the team.
I've moved the experiment a little further since writing the original post.
Using a genuine 1000-mel / 500-position input through layer 8, then splicing in the 6-spatial / 8-temporal oracle-derived tail reconstruction before layers 9–32, I get 19/20 exact token matches against the full encoder, with 20/20 EOS.
I also tested the already-selected spatial basis/components and temporal basis/rank on 15 new recordings/sentences that were not used when selecting them. The per-utterance amplitudes were still oracle-derived in this experiment, so this was a test of the representation rather than a deployable predictor. On those recordings, the temporal rank-8 reconstruction agreed with the sparse-6 reconstruction on 15/15, and both matched the full model on 14/15.
I then tried removing the utterance-specific amplitudes entirely and using a single fixed mean set of amplitudes derived from the original 20 examples. That produced 18/20 exact matches on the original set, but only 10/15 on the new recordings (with EOS preserved in all 15). That suggests that a surprisingly large part of the padded-tail structure can be represented by a fixed component, but some utterance-specific correction still matters.
I also tried a simple PCA/ridge approach to predict all 48 amplitudes from the genuine short-context layer-8 state. With only 20 development examples it did not generalize in leave-one-out testing — the best normalized MSE was about 1.12 relative to the target-mean baseline — so I don't think that result justifies a predictor yet.
I'm now testing whether the problem can instead be decomposed into a fixed mean tail plus a much smaller dynamic residual. The idea is to determine at decoder level how many utterance-specific degrees of freedom are actually necessary before trying to learn them from the short-context state.
Your suggestions about FLEURS/Common Voice and cross-language transfer make a lot of sense. I deliberately started with the constrained voice-command domain because it gives a clean experimental setting, but broader evaluation would be the obvious next step if the non-oracle version holds up.
I'm absolutely happy to share the scripts, intermediate results and methodology. It would be great if you or the team wanted to reproduce or extend this on a larger evaluation set.
Best,
Lars Folmann

One small addition, just to answer your three questions explicitly:

1. FLEURS / Common Voice / more varied speech: Not tested yet. The additional 15 recordings I've used are independent of the original 20 used for basis selection, but they are still short, voice-control-style utterances. So I would currently only claim the low-rank result for this constrained short-command setting. Testing FLEURS/Common Voice would be the next proper generalization test.
2. Cross-language transfer: Not tested yet either. Everything so far has been done with BuzzASR Danish. It would be particularly interesting to distinguish between the same basis transferring across languages and merely the same low dimensionality appearing with a different basis.
3. Predicted/non-oracle version: I have started testing this. A simple PCA/ridge predictor of all 48 amplitudes from the genuine short-context layer-8 state did not generalize in leave-one-out testing on the 20-example development set (best normalized MSE ≈ 1.12 relative to the target-mean baseline), so I don't yet have a successful non-oracle predictor.
   Interestingly, a single fixed mean derived from the original 20 examples still gives 18/20 exact matches there, but only 10/15 on the new recordings. That is what led me to the current experiment: treating the tail as a large fixed component plus a much smaller utterance-specific residual, and measuring at decoder level how many dynamic degrees of freedom are actually required.
   So the short version is: (1) not yet, (2) not yet, (3) promising representation, but no successful learned predictor yet.
   Lars

Hi Shivam,
You asked three very useful questions after our original Danish experiments:

1. Does the same 6-spatial / 8-temporal structure hold for longer or more varied utterances, e.g. FLEURS/Common Voice?
2. Is the basis language-specific, or does it transfer across other BuzzASR language models?
3. How does the non-oracle prediction hold up — can it retain the 14/15 or 15/15 results?
   We took those questions seriously, particularly #2 and #3.
   To make the follow-up practical, we rented an NVIDIA RTX 5090, ported our Hugging Face FP32 experiment path to CUDA, created a separate English BuzzASR branch, and ran the English equivalent of our experimental sequence through V1–V23.
   The English dataset mirrors the Danish command-style experiment: ORIGINAL20 for development/training and a frozen HOLDOUT15 for evaluation. We used a fixed Piper en_US-lessac-medium voice.
   One important limitation up front: the Danish material is human speech and the English material is synthetic speech. We therefore treat this as cross-language/model/speaker replication, not a controlled language-only comparison.
   Here is where we are now on your three questions.
4. Does the same structure hold for longer/more varied utterances?
   We cannot answer that one yet.
   We have not yet run FLEURS, Common Voice, or an equivalent corpus of longer and more varied speech.
   What we can say is that the structural observation survived the move from our Danish setup to a separate English BuzzASR model and English dataset.
   On English, the speech-dependent padding-tail delta was again strongly low-dimensional. At encoder layer 8, the global spatial basis captured approximately:

• K4: 95.3%
• K8: 96.7%
• K16: 97.5%
• K32: 98.3%
• K64: 99.0%
  More importantly, the architecture we had arrived at from the Danish work — layer 8, six spatial components and temporal rank 8 — also worked on the English frozen holdout.
  In English V8, TEMP-R8 reproduced SPARSE6 on 15/15 holdout utterances.
  So our answer to #1 is:
  The compact structure replicates beyond the original Danish experiment, but we have not yet established that it holds for long/unrestricted speech. FLEURS/Common Voice remains an important next test.

2. Is the basis language-specific?
   We can now answer part of this much more confidently:
   The phenomenon itself does not appear to be uniquely Danish.
   We independently reproduced the low-dimensional padding-tail structure using the English BuzzASR model, and the same 6-spatial / rank-8-temporal architecture remained effective.
   However, there is an important distinction between:
   A) the same kind of low-dimensional structure existing in both models, and
   B) the actual numerical basis learned from Danish transferring directly to English.
   We have demonstrated A, but we have not yet tested B.
   So we do not want to overstate this as language-independent basis transfer.
   The clean next experiment is now obvious:
   frozen Danish basis → English model/data
   and
   frozen English basis → Danish model/data.
   Our current answer to #2 is therefore:
   The structural phenomenon and compact architecture replicate across Danish and English BuzzASR models, but direct transfer of the learned numerical basis vectors remains untested.
3. How does non-oracle prediction hold up?
   This is where the experiments became particularly interesting.
   Non-oracle prediction is still the bottleneck. We cannot yet reliably retain the oracle-level 14/15–15/15 behaviour.
   The oracle experiments continue to tell us that the representation itself can be extremely compact.
   The difficult part is predicting the particular correction that matters to the decoder from the short-context representation.
   On English, a global residual basis trained only on ORIGINAL20 reached:
   14/15 exact-control at rank 128
   on the frozen HOLDOUT15.
   Interestingly, that basis represented only about 69.9% of the training residual energy. Indsat tekst
   We then tried to target the decoder more directly.
   In V22 we ranked coordinates according to decoder-derived importance learned from ORIGINAL20.
   Increasing the budget all the way from K64 to K8192 did not solve the remaining holdouts. Performance plateaued at:
   13/15 exact-control. Indsat tekst
   Then in V23 we tested a coherent decoder-gradient subspace rather than individual coordinates.
   At rank 128, the frozen global subspace captured:
   99.138% of the ORIGINAL20 decoder-gradient energy.
   Yet HOLDOUT15 performance was still:
   13/15 exact-control.
   In fact, R1, R2, R4, R8, R16, R32, R64 and R128 all ended at 13/15 with the same two remaining failures. Indsat tekst
   That result changed our interpretation of the problem.
   It increasingly looks as though variance explained, representation energy, and even global first-order decoder-gradient energy are not sufficient proxies for decoder-critical information.
   A relatively low-energy component can matter disproportionately to the decoder, while a global approximation containing almost all measured gradient energy can still miss the information required for an unseen utterance.
   Our working hypothesis is now that the missing correction is utterance-conditioned and/or nonlinear, rather than something that can be represented by one fixed global subspace.
   So our answer to #3 is:
   No, not reliably yet. The oracle representation is compact, but predicting the decoder-critical correction remains unsolved. The English results strengthen the evidence that simply increasing global rank or captured energy is not the answer.
   One important note about “exact-control”
   We should clarify what 13/15 or 14/15 means here.
   Our primary diagnostic is deliberately strict: exact-control means reproducing the exact decoder output of the normal full 3000-mel path.
   It is a representation-equivalence test, not simply an ASR accuracy metric.
   Some “failures” are therefore trivial transcription differences.
   For example, one English holdout compares:
   Play radio on over.
   with
   Play radio on over
   and therefore fails exact-control. Indsat tekst
   Another persistent case is essentially:
   ...stop the music.
   versus
   ...stop the music.! Indsat tekst
   There are also cases where the full 3000-mel control itself produces a linguistically worse transcription than the shortened/reconstructed path.
   For that reason we keep representation equivalence and ASR quality separate.
   Why the RTX 5090 mattered
   The V1–V23 sequence had grown large enough that iteration on our local CPU setup was becoming impractical.
   We therefore rented RTX 5090 compute and ported the experimental HF FP32 path to CUDA, while retaining the original source tree separately.
   That made it possible to run the complete English follow-up rather than selecting only a few experiments because of compute time.
   For perspective, V22 completed in about 54 seconds and V23 in about 55 seconds on the RTX 5090. Indsat tekst Indsat tekst
   The English V1–V23 series has now completed successfully. Indsat tekst
   Reproducibility
   If you or anyone else working on BuzzASR would find it useful, we are happy to prepare the complete test material for independent reproduction.
   We can package:

• Danish and English audio test sets
• ORIGINAL20 / frozen HOLDOUT15 splits
• manifests and ground truth
• exact BuzzASR model revisions
• SHA256 hashes
• V1–V23 experiment scripts
• CUDA-port versions
• learned basis/artifact files
• result JSON files and logs
• a README explaining the experiment sequence
  We would actually welcome independent reproduction — or someone finding a flaw in our interpretation.
  At this point your original three questions have also given us three fairly obvious directions for the next round:
  A. FLEURS/Common Voice or another longer, more varied corpus
  B. direct frozen DA-basis → EN and EN-basis → DA transfer
  C. an utterance-conditioned predictor instead of another larger global basis
  If one of these would be more useful to the BuzzASR project, please let us know and we can prioritize it.
  Thanks again for the questions. They pushed this quite a bit further than our original Danish experiment.
  — Lars

Hi Lars,

Thanks for the thorough follow-up, and for going the extra distance with the English replication. The result that the same layer-8 / 6-spatial / rank-8-temporal structure appears on a separate model is the most convincing part so far.

A few thoughts. With 15 holdout utterances, 13/15 vs 14/15 is a one-utterance difference, so I'd be careful drawing conclusions about rank and energy from it. Some of the failures you describe are also punctuation-only. A continuous comparison would help: for example the full model's log-likelihood of its own transcript under the reconstructed path, or normalized WER between the two outputs.

One hypothesis I had been thinking is that maybe the padded positions may be acting as register-like scratch space, which would explain why the content is low-rank and utterance-dependent. It would also suggest a training-side alternative, fine-tuning with randomly shortened encoder input, which we could try maybe from our end.

On your next steps, I'd prioritize A (FLEURS/Common Voice) and C (a utterance-conditioned predictor). The predictor is where scale matters most, with 20 examples, failure tells us little, but we can generate thousands of (short-context state, oracle tail) pairs across several BuzzASR languages. I'd be glad to run that if you send the scripts and the packaged reproduction material.

Thanks again,
Shivam

Hi Shivam,
Thanks — and agreed on both methodological points.
With only 15 holdout utterances, we shouldn't read too much into 13/15 versus 14/15, particularly when some of the exact-control failures are punctuation-only. Our exact-control metric has been useful as a very strict representation-equivalence test, but it is clearly too binary to carry the evaluation on its own.
For the next round, we'll add continuous comparisons as you suggest. The full model's log-likelihood of its own transcript under the reconstructed path sounds particularly useful, alongside normalized WER/CER between the full and reconstructed outputs.
Your “register-like scratch space” hypothesis is also very close to what the experiments have been making us suspect.
The padded positions do not seem passive. Speech-dependent information appears to propagate into them, yet much of that resulting state is highly structured and low-rank. At the same time, our later experiments suggest that a small amount of apparently low-energy information can still be disproportionately important to the decoder.
Fine-tuning with randomly shortened encoder input is therefore a very interesting alternative: rather than trying to reconstruct the missing state afterwards, teach the model not to depend on having that scratch space in the first place.
A little context we probably should have given earlier
This did not start as an ASR research project.
I'm Lars, in Denmark, and the original project is my personal Home Assistant system.
The practical goal is simple: I want fast, fully local Danish voice control for Home Assistant — lighting, Music Assistant, temperature queries, locks and similar everyday commands — with response time approaching the experience of Alexa or Google, while keeping speech recognition local.
That is also why our ORIGINAL20/HOLDOUT15 datasets look unusually command-oriented. They weren't intended as general speech benchmarks. They represent the actual workload we're trying to make fast.
Commands such as:

• “Turn on the light in the living room”
• “What is the temperature outside?”
• “Play some music”
• “Is the door locked?”
  are essentially the real application.
  We initially weren't trying to discover anything about Whisper/BuzzASR internals. We were simply trying to understand why a very short voice command still required so much inference time.
  That led us to the 3000-mel input, then to experiments with shortening it, then to the observation that simply removing the padded portion changes decoder behaviour — and eventually into the padding-tail experiments we're discussing now.
  So the research rabbit hole grew out of a very practical latency problem.
  And a clarification about “we”
  There isn't a research lab, university or company behind these experiments.
  When I've written “we”, that means me working together with ChatGPT as a technical/research assistant.
  I've been doing the practical work, running the experiments and maintaining the Home Assistant/BuzzASR setup. ChatGPT has been helping extensively with experiment design, scripting, analysis, interpreting results and keeping track of what eventually became the V1–V23 experiment series.
  I think it's worth making that explicit, particularly now that we're talking about sending you material for independent reproduction.
  The rented RTX 5090 was also simply something we decided to do when the CPU experiments became too slow to iterate on sensibly. We ported the Hugging Face FP32 experimental path to CUDA and used that to complete the English replication through V23.
  So this is very much an enthusiast project that unexpectedly turned into a fairly substantial investigation.
  On your suggested next steps
  We agree with your priorities.
  A — FLEURS/Common Voice is the right way to test whether the phenomenon survives outside our small command-style workload.
  Our English replication gives us more confidence that the layer-8 / 6-spatial / rank-8-temporal observation wasn't just an accident of the original Danish model/data, but it still doesn't tell us what happens with longer and substantially more varied speech.
  And C — an utterance-conditioned predictor is probably where scale matters most.
  Our 20 development utterances were enough to expose structure and generate hypotheses, but they clearly aren't enough to properly answer the prediction/generalization question.
  Your proposal to generate thousands of (short-context state, oracle tail) pairs across several BuzzASR languages would move that question into a completely different regime.
  We would be very interested to see the result.
  It might tell us whether the difficulty we've been seeing is because the mapping is genuinely difficult/nonlinear/utterance-conditioned, or simply because we've been trying to learn it from an absurdly small sample.
  Reproduction package
  Thanks for offering to run this from your end.
  We're preparing a clean reproduction package rather than sending the entire experimental machine image.
  The plan is to include:
• Danish ORIGINAL20 and HOLDOUT15 audio
• English ORIGINAL20 and HOLDOUT15 audio
• manifests and ground truth
• exact BuzzASR model revisions
• SHA256 hashes
• relevant V1–V23 experiment scripts
• CUDA versions used for the RTX 5090 runs
• spatial/temporal/residual basis artifacts where required
• result JSON/log material
• a README describing the experiment sequence and how to reproduce the important results
  We won't package the multi-GB model copies themselves; the README will point to the exact pinned Hugging Face revisions so they can be fetched reproducibly.
  We'll put the material in a Hugging Face repository so it is versioned and accessible to everyone rather than sending around an opaque ZIP archive.
  And before packaging it, we'll clean the experiment tree so you're getting the actual reproduction material rather than months of accumulated test debris. :-)
  One thing I particularly like about your suggestion
  If you can generate thousands of examples across several BuzzASR languages, it would also give us a much cleaner separation between three questions that have become somewhat entangled in our small experiments:
  Is the scratch-space phenomenon general?
  Is its compact representation general?
  Can the required state actually be predicted from the shortened input?
  Our Danish→English replication gives us some evidence for the first two.
  The third is still very much open.
  And I agree that our current 13/15 versus 14/15 results aren't sufficient evidence to settle it.
  Thanks again for taking this seriously — and especially for offering compute and experimentation from your side.
  What started here as “why does turning on a light locally take this long?” has gone quite a bit further than I expected.
  We'll get the reproduction repository ready for you.
  — Lars

Hi Shivam,
I have now published the independent reproduction package for the BuzzASR padding-tail experiments (V1–V23).
It contains the Danish recordings, experiment scripts, preserved results, reproduction artefacts, manifests and documentation. The historical English Piper WAV files are not redistributed, but their prompts, provenance, metadata and SHA-256 hashes are retained so comparable input can be regenerated.
The repository is available here:
BuzzASR Padding-Tail Reproduction
The datasets are deliberately small, so I present this as independent reproduction/investigation material rather than as a broad claim about BuzzASR or ASR generally.
Feedback and independent replication are very welcome.
Best regards,
Lars

Repository link:
https://huggingface.co/LFCDK/buzzasr-padding-tail-reproduction

Hi Lars,

Thank you for putting this together, and for being so open about the project's origins and your use of ChatGPT. I like how the project grew out of a practical question.

I've read through the repository. It's carefully organized, and I appreciate that the negative results, the limits of the holdout set and the punctuation-only failures are documented so plainly.

I'll keep you posted on how things develop from our side. We have some experiments running in parallel and a deadline coming up, but I'll discuss it with my team and get back to you soon.

Best,
Shivam