GAB / about.html
Arash2348's picture
About page: method and outside check only (no numbers that could contradict the results page); README: Voice Autopilot name, all categories
1c560ed verified
Raw History Blame Contribute Delete
3.66 kB
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>About GAB</title>
<style>
:root { --bg: #f6f5f1; --card: #ffffff; --ink: #1c1d1f; --muted: #686b70; --line: #e2e1dc; --accent: #b5561c; }
@media (prefers-color-scheme: dark) { :root:not([data-theme="light"]) { --bg: #111315; --card: #1a1d20; --ink: #e8e9ea; --muted: #a2a6ab; --line: #2f3337; --accent: #e08a50; color-scheme: dark; } }
:root[data-theme="dark"] { --bg: #111315; --card: #1a1d20; --ink: #e8e9ea; --muted: #a2a6ab; --line: #2f3337; --accent: #e08a50; color-scheme: dark; }
body { margin: 0; background: var(--bg); color: var(--ink); font: 15px/1.6 -apple-system, BlinkMacSystemFont, "Segoe UI", Helvetica, Arial, sans-serif; }
.wrap { max-width: 760px; margin: 0 auto; padding: 24px 16px 56px; }
a { color: var(--accent); }
h1 { font-size: 22px; margin: 0 0 8px; } h2 { font-size: 16px; margin: 22px 0 6px; }
.box { background: var(--card); border: 1px solid var(--line); border-radius: 10px; padding: 4px 14px; overflow-x: auto; }
table { border-collapse: collapse; width: 100%; font-size: 14px; } th, td { text-align: left; padding: 6px 8px; border-bottom: 1px solid var(--line); }
th { color: var(--muted); font-weight: 500; } tr:last-child td { border-bottom: 0; } td.n { text-align: right; font-variant-numeric: tabular-nums; }
.muted { color: var(--muted); font-size: 14px; }
</style>
</head>
<body>
<div class="wrap">
<p><a href="./index.html">← Back to the results</a></p>
<h1>About GAB</h1>
<p>GAB (Gurbani Autopilot Bench) measures how well Voice Autopilot in SikhiToTheMax (STTM) Desktop keeps the right Gurbani on screen during kirtan and paath. All results, by category and by recording, are on the <a href="./index.html">results page</a>; this page only explains the method.</p>
<h2>How it works</h2>
<p>Each recording is replayed through Voice Autopilot exactly as it would hear it in a Gurdwara Sahib. Every second, what was on screen is compared with an answer key of what was being sung or read. Where the answer key only names the Shabad, the Shabad is judged; where it names each pangti, the exact pangti is judged. Answer keys made by machine alignment are marked provisional until a person has checked them.</p>
<h2>Outside check: Karanbir Singh's benchmark</h2>
<p>Karanbir Singh's public <a href="https://github.com/karanbirsingh/live-gurbani-captioning-benchmark-v1">Live Captioning for Gurbani Kirtan benchmark v1</a> has 4 kirtan recordings with person-checked pangti keys, each also started one third and two thirds of the way in (12 cases). Scored with <b>his own scorer</b> (which allows 1 second of slack at each pangti change and counts a blank screen in pauses as right):</p>
<div class="box"><table>
<tr><th>System</th><th class="n">His score</th></tr>
<tr><td>Voice Autopilot (9.0.5, 10.0.4 and 10.0.5 give the same result)</td><td class="n"><b>90.5%</b></td></tr>
<tr><td>His own system (kirtan-captioning 0.2.0)</td><td class="n">57.9%</td></tr>
<tr><td>Reference: right pangti, but 5 s late</td><td class="n">85.5%</td></tr>
</table></div>
<p class="muted">His score differs from the "right pangti" percentage on the results page, which gives no slack. The speech model Voice Autopilot uses is also Karanbir Singh's; whether these 4 recordings were in its training data is not yet known.</p>
<h2>What is not shown</h2>
<p class="muted">Only recordings that are public on YouTube appear here. Private Gurdwara Sahib recordings and the held-out test set are never published, and no audio is hosted.</p>
</div>
</body>
</html>