benchmark/README.md

Benchmark outputs and scorer

Every number on the model card can be checked from this folder. We decoded every system ourselves, and here are the per-clip outputs and the scorer we used.

What is here

path	contents
hypotheses/<system>_<ms>ms_<set>.jsonl	one line per clip: {"id", "pred_text"}, the system's output
hypotheses/<system>_560ms_<set>_split.jsonl	the joined long recordings (tv_long, gs2_long) decoded in pieces cut at pauses, as serve/ does (chinda_realtime.transcribe_long), both models through the same code
ids/<set>.jsonl	one line per clip: {"id", "cluster"}, the clips of each test set and the group the bootstrap resamples
make_refs.py	rebuilds the reference transcripts from the public datasets, keyed by the same ids
score.py	character error rate, and the paired difference of two systems with a 95% bootstrap interval
chinda_norm.py, num_norm.py	the normalisation score.py applies to both sides

Systems: openthai2.0-asr-streaming (this model, th-TH prompt), openthai2.0-asr-streaming_th-mix (th-mix prompt), typhoon-asr-streaming-0.6b (Typhoon ASR Streaming 0.6B as released) and typhoon-asr-streaming-0.6b_4gram (the same with its published 4-gram language model, weight 0.5).

We do not redistribute the reference transcripts, because the datasets carry their own terms (GigaSpeech2 is gated). make_refs.py downloads only the transcript files and writes them in the format score.py reads.

Test sets and ids

set	source	id	cluster
fleurs	google/fleurs, th_th test	audio file name	the clip
tvspeech	typhoon-ai/TVSpeech, test	row index	YouTube video
gs2	speechcolab/gigaspeech2, Thai test	segment id (0-0)	recording
isan	typhoon-ai/thai-dialect-isan-dataset, test	dataset id	speaker
lotus_con123, lotus_lav123, lotus_jbl, lotus_bt3m, lotus_bt10m	nectec/LOTUSDIS, test: the same 5,516 turns on five devices (conference microphone, lapel microphones, speakerphone and the two far channels)	chunk file name	meeting
tv_long	TVSpeech, each video's clips joined in row order with 1 s of silence	video id	the file
gs2_long	GigaSpeech2 test, each recording's segments joined in segment order with 1 s of silence; recording 26 left out because its transcripts do not match its audio	recording id	the file

Score

pip install huggingface_hub pyarrow pythainlp==5.3.5 editdistance
python make_refs.py --out refs/             # GigaSpeech2: accept its gate on Hugging Face first
python score.py refs/tvspeech.jsonl hypotheses/openthai2.0-asr-streaming_560ms_tvspeech.jsonl \
                hypotheses/typhoon-asr-streaming-0.6b_560ms_tvspeech.jsonl

With one hypothesis file, score.py prints its character error rate. With two, it also prints the difference A − B with a 95% interval from 1,000 bootstrap resamples of whole clusters (seed 1). An interval that includes zero is a tie.

make_refs.py cleans the FLEURS and GigaSpeech2 transcripts the way our benchmark references were cleaned when we built them: ASCII punctuation except ? and ! removed (decimal points too), curly quotes, dashes and … removed, lower case, and PyThaiNLP's normalize. The other sets are used as published.

Normalisation in score.py, the same for reference and output: Thai digits become Arabic digits, curly quotes become straight ones, ำ is decomposed, spoken Thai numbers become digits (on whole words found by PyThaiNLP's newmm), and then every space is removed. The error rate is the edit distance over all characters of a set, divided by the number of reference characters.

To score the joined long recordings with your own system, build the audio the same way: the clips of one video or recording, in order, with exactly one second of digital silence between them, 16 kHz mono.