← Back to the results

About GAB

GAB (Gurbani Autopilot Bench) measures how well Voice Autopilot in SikhiToTheMax (STTM) Desktop keeps the right Gurbani on screen during kirtan and paath. All results, by category and by recording, are on the results page; this page only explains the method.

How it works

Each recording is replayed through Voice Autopilot exactly as it would hear it in a Gurdwara Sahib. Every second, what was on screen is compared with an answer key of what was being sung or read. Where the answer key only names the Shabad, the Shabad is judged; where it names each pangti, the exact pangti is judged. Answer keys made by machine alignment are marked provisional until a person has checked them.

Outside check: Karanbir Singh's benchmark

Karanbir Singh's public Live Captioning for Gurbani Kirtan benchmark v1 has 4 kirtan recordings with person-checked pangti keys, each also started one third and two thirds of the way in (12 cases). Scored with his own scorer (which allows 1 second of slack at each pangti change and counts a blank screen in pauses as right):

SystemHis score
Voice Autopilot (9.0.5, 10.0.4 and 10.0.5 give the same result)90.5%
His own system (kirtan-captioning 0.2.0)57.9%
Reference: right pangti, but 5 s late85.5%

His score differs from the "right pangti" percentage on the results page, which gives no slack. The speech model Voice Autopilot uses is also Karanbir Singh's; whether these 4 recordings were in its training data is not yet known.

What is not shown

Only recordings that are public on YouTube appear here. Private Gurdwara Sahib recordings and the held-out test set are never published, and no audio is hosted.