Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Nymbo 
posted an update 4 days ago
Post
3,615
Introducing Live Subtitles, a Windows tray app that captions anything your PC is playing and isn't butt ugly. live and fully local. It runs NVIDIA's Parakeet-unified 0.6B on your GPU, puts words on screen about 0.36 s after they're spoken, and uses less than one CPU core while it does it.

I built a benchmark for it out of real TV, podcasts, earnings calls and a short film. Parakeet-unified scores 12.1% WER there, close to Nemotron streaming's 11.4%, but with much better punctuation and lower worst-case delay. Once a caption is on screen it never gets rewritten or jumps lines, and the overlay disappears when nobody's talking.

No NVIDIA GPU? It falls back to Nemotron on the CPU, which also covers 30 other languages. It can also write SRT/ASS/VTT files for videos on disk.

Windows only. The GPU engine needs an NVIDIA card with ~3.2 GB of free VRAM. MIT.

Edit: the first version of this post said 13.8% vs 17.0%. The replies caught a scoring bug (numbers weren't normalized), so the benchmark now uses Whisper's English normalizer and the numbers above are re-scored.

Get it here:
https://github.com/Nymbo/Live-Subtitles

Models:
nvidia/parakeet-unified-en-0.6b
nvidia/nemotron-speech-streaming-en-0.6b
WideChaos

Could you add a run with excerpt boundaries chosen independently of Parakeet? The README says cuts are placed where its timed words agree with the human transcript. Using the same recordings with fixed time boundaries would check whether that selection changes the WER gap.

·
Nymbo

Ran it. These transcripts have no timestamps, so fixed time cuts would still need an alignment to know where the reference text ends. Instead I scored the nine cut recordings whole, against their whole transcripts, with no cuts at all. That's about 6.5 hours of audio.

Same picture as the excerpts: Parakeet-unified 11.7% vs Nemotron 10.8%. On those same nine recordings the excerpts gave 11.2 vs 10.6, so if anything the cuts were slightly kind to Parakeet. Nemotron is ahead on everything except the Crimetown podcast, and most of Parakeet's extra errors are dropped words.

For the record, the cuts came from Parakeet TDT v3 offline, not the model being tested, but your question was the right one to ask. Uncut mode is in the repo now (bench.realset build --full).
dipankarsarkar

Your 13.8 vs 17.0 reproduces exactly as the plain mean of the 11 real-world scenes in the README (152.2/11 and 187.2/11).

That makes it easy to see where the gap lives.
The China earnings call (16.2 vs 33.7) is about half of it on its own.
Drop that one scene and it is 13.6 vs 15.4.

Parakeet still wins 8 of 11.
But three of those wins are under a point: TV monologue 0.2, Brazil 0.3, Tears of Steel 0.6.
Those are the ones WideChaos's boundary question could flip, since a cut only has to move a few edge words.

The synthetic set goes the other way (17.4 vs 14.9), on the loudest-effects scene and the very quiet one.
So the real split looks like accented speech vs speech buried in the mix.

You give the leveller's effect on that call for Parakeet only (22.0 to 16.2).
What does it do to Nemotron's 33.7?

·
Nymbo

Thanks for digging into this, you found a real bug. That China call didn't make sense, so I looked at its errors: almost all of Nemotron's extra words were numbers. It spells them out ("twenty twenty one", "two point three") while the reference uses digits, and my scorer wasn't normalizing them.

I've switched the benchmark to Whisper's English normalizer (same as the Open ASR Leaderboard) and re-scored everything:

  • Real-world mean: Parakeet-unified 12.1% vs Nemotron 560 11.4%
  • China call: 12.5 vs 12.5, a tie
  • Nemotron now wins 7 of 11 real scenes, Parakeet-unified the 3 podcasts
  • Synthetic still goes to Nemotron, 15.0 vs 16.3, on the same loud-effects and quiet scenes
  • Scoring the full uncut recordings (for WideChaos's question) gives 11.7 vs 10.8, so it isn't the excerpt cuts

On the leveller: Nemotron gets 33.5 without it vs 33.7 with it on the old scoring. Normalized, it's neutral for both engines. Parakeet-unified's 22.0 to 16.2 was the same number artifact, just with the leveller nudging how it formatted numbers. It only helps a little on very quiet speech now.

So it's not a WER win. Parakeet-unified is the GPU default for its punctuation (sentence F1 0.60 vs 0.40), lower p90 delay, and near-idle CPU. README has the corrected numbers.
dipankarsarkar

Good fix, and thanks for running it whole. My accented-speech vs buried-speech split does not survive it.

The new table has the old shape, just flipped.
Nemotron's summed lead over the 11 scenes is 7.6 points (133.0 vs 125.4).
The two TED talks give 4.4 of that (1.2 and 3.2) and Australia 2.6.
Everything else nets to 0.6.
Six of the 11 sit within 0.7 points, counting the China tie.

So the deletions may be the more useful number than the mean.
Every scene goes through the app's own caption pipeline, where the newest word stays "in progress" through a pause.
If Parakeet-unified's dropped words sit at the end of phrases, that is the commit rule costing WER, not the model.
Scoring the raw decoder output on the same TED audio would split the two.

On the 8 of 9 with more deletions, do the dropped words cluster just before pauses?

·
Nymbo

Checked both. It's not the commit rule. I ran the raw decoder output (no caption pipeline at all) on every scene, and Parakeet-unified scores identically: same WER, same deletions, all 16 scenes.

The deletions don't cluster before pauses either. Over the 6.5 uncut hours, 92% of Parakeet's extra deletions are mid-phrase. What they are is disfluencies: about two-thirds are repeated words and fillers, and most of the rest are false starts. Stuff like "I was [I was] out in", "until we [until we] moved", "[you know]". The references are verbatim, Nemotron keeps the stutters, Parakeet-unified cleans them up.

If you excuse those deletions for both engines, the uncut WER is 10.12 vs 10.19. So the whole gap is transcription style, not accuracy. For live captions I'd call dropping "I was, I was" a feature.
dipankarsarkar

That settles it. Same WER with no pipeline clears the commit rule, and 92 percent mid-phrase clears the pauses.

So the gap was never accuracy. It was the reference convention.
From 11.7 vs 10.8 to 10.12 vs 10.19, the excused words are worth about 1.6 points to Parakeet-unified and 0.6 to Nemotron.
A verbatim reference scores every cleanup as an error.

That makes both columns worth printing, verbatim and disfluency-excused.
Live captions and verbatim transcripts want opposite winners on the same audio.

The one place cleanup is not a feature is the repeat that was meant.
"No, no, no" and "very very" look like stutters to a cleaner.

Of Parakeet-unified's excused deletions, how many are repeats a listener would want kept?
In this post