rumblex: word timestamps, speakers and subtitles for Rumble-2-Flash (open source)

#6
by edwardgl - opened

Hi all,

I built an open-source pipeline around Rumble-2-Flash for long recordings: VAD chunking, batched transcription, word-level timestamps from the model's own CTC head, optional speaker labels (pyannote), and SRT/VTT/JSON output. Code: https://github.com/EdwardGlockner/rumblex

The CTC head made this easy. Its classes map 1:1 onto the decoder vocabulary, and the encoder states that generate() already computes are exactly its input. So the decoder's own tokens can be force-aligned with one extra linear layer and a Viterbi pass, and no second model is needed. In my runs that took 0.3–0.9 s per 20–60 minute file, against 7–17 s for WhisperX's wav2vec2 alignment step.

Measured against WhisperX large-v3 on one RTX 4090, same audio, same session:

Rumble-2-Flash (rumblex) WhisperX large-v3
AMI meetings, WER (500 utt.) 10.9% 17.2%
Earnings-22, WER (500 utt.) 9.3% 10.6%
LibriSpeech test-clean, WER (200 utt.) 2.2% 1.6%
FLEURS Dutch, WER (364 utt.) 9.2% 7.0%
speed 329–467× real time 79–104×
word start / end error vs MFA (LibriSpeech) 58 / 57 ms 75 / 34 ms

A few things I ran into, in case they're useful to others:

  • Batching: padded batching changes the encoder output unless clips are trimmed to whole mel hops, padded with a little silence, and the masked feature frames are zeroed. With that, batch 8 and batch 1 give identical output in float32.
  • Word ends: the CTC head is very peaky (about one frame per token at 13 frames/s), so word ends need a rule. "End = next word's start, cut back at the first silence" brought end error from 75 to 57 ms.
  • FLEURS Dutch: I measure 9.2–9.6% in every setup I tried (transformers bf16 and fp32, vLLM, with and without VAD), using Whisper's basic normalizer. The FLEURS Dutch references mix digits and spelled-out numbers, which costs points whichever digits setting is used. I haven't reproduced the leaderboard's scoring, so I don't think my number and the 6.05% on the model card are comparable. Corrections welcome.

Thanks for releasing the CTC head alongside the model. It is what makes this kind of tool cheap to run.

Love seeing this!

On FLEURS Dutch: Whisper's basic normalizer is english only, I assume that’s the issue. You should be able to reproduce the reported number with HF’s text normalisation used for the leaderboard.

You can find the exact code used for those numbers at https://github.com/huggingface/open_asr_leaderboard

Sign up or log in to comment