Benchmark + GPU-agnostic usage (AMD/Intel/NVIDIA/Apple) via whisper.cpp
#2
by phonology024 - opened
Thank you for Pathumma Whisper! I benchmarked Thai models on Google FLEURS (th_th, 50 utterances, Whisper's
BasicTextNormalizer rule, CER):
| Model | CER |
|---|---|
| openai/whisper-large-v3-turbo | 15.9% |
| openai/whisper-large-v3 | 11.5% |
| Typhoon Whisper | 11.9% |
| Thonburian Whisper | 9.1% |
| Pathumma-whisper-th-large-v3 | 8.9% |
Pathumma had the lowest CER, so I made it the default Thai model for the --accurate mode of
babelscribe (MIT). It converts the checkpoint to ggml on the user's
machine (weights are never redistributed) and runs it with whisper.cpp, so it works on AMD / Intel GPUs via Vulkan as well
as NVIDIA and Apple. Since the fine-tune doesn't predict timestamps reliably, babelscribe takes the text from Pathumma
and the timing from whisper-large-v3-turbo, aligned character by character and cut only at Thai word boundaries:
pip install babelscribe
babelscribe video.mp4 -l th --accurate # -> video.srt
Benchmark script: bench/fleurs.py in the repo. Happy to share per-utterance outputs if useful.