Model card; benchmark outputs and scorer; serve/: long recordings cut at pauses, WS /v1/live

0a9a98f verified 4 days ago

3.96 kB

# Benchmark outputs and scorer

Every number on the model card can be checked from this folder. We decoded every system ourselves, and here are the per-clip outputs and the scorer we used.

## What is here

| path | contents |
|---|---|
| `hypotheses/<system>_<ms>ms_<set>.jsonl` | one line per clip: `{"id", "pred_text"}`, the system's output |
| `hypotheses/<system>_560ms_<set>_split.jsonl` | the joined long recordings (`tv_long`, `gs2_long`) decoded in pieces cut at pauses, as `serve/` does (`chinda_realtime.transcribe_long`), both models through the same code |
| `ids/<set>.jsonl` | one line per clip: `{"id", "cluster"}`, the clips of each test set and the group the bootstrap resamples |
| `make_refs.py` | rebuilds the reference transcripts from the public datasets, keyed by the same ids |
| `score.py` | character error rate, and the paired difference of two systems with a 95% bootstrap interval |
| `chinda_norm.py`, `num_norm.py` | the normalisation `score.py` applies to both sides |

Systems: `openthai2.0-asr-streaming` (this model, `th-TH` prompt), `openthai2.0-asr-streaming_th-mix` (`th-mix` prompt), `typhoon-asr-streaming-0.6b` (Typhoon ASR Streaming 0.6B as released) and `typhoon-asr-streaming-0.6b_4gram` (the same with its published 4-gram language model, weight 0.5).

We do not redistribute the reference transcripts, because the datasets carry their own terms (GigaSpeech2 is gated). `make_refs.py` downloads only the transcript files and writes them in the format `score.py` reads.

## Test sets and ids

| set | source | id | cluster |
|---|---|---|---|
| `fleurs` | `google/fleurs`, `th_th` test | audio file name | the clip |
| `tvspeech` | `typhoon-ai/TVSpeech`, test | row index | YouTube video |
| `gs2` | `speechcolab/gigaspeech2`, Thai test | segment id (`0-0`) | recording |
| `isan` | `typhoon-ai/thai-dialect-isan-dataset`, test | dataset id | speaker |
| `lotus_con123`, `lotus_lav123`, `lotus_jbl`, `lotus_bt3m`, `lotus_bt10m` | `nectec/LOTUSDIS`, test: the same 5,516 turns on five devices (conference microphone, lapel microphones, speakerphone and the two far channels) | chunk file name | meeting |
| `tv_long` | TVSpeech, each video's clips joined in row order with 1 s of silence | video id | the file |
| `gs2_long` | GigaSpeech2 test, each recording's segments joined in segment order with 1 s of silence; recording 26 left out because its transcripts do not match its audio | recording id | the file |

## Score

```bash
pip install huggingface_hub pyarrow pythainlp==5.3.5 editdistance
python make_refs.py --out refs/             # GigaSpeech2: accept its gate on Hugging Face first
python score.py refs/tvspeech.jsonl hypotheses/openthai2.0-asr-streaming_560ms_tvspeech.jsonl \
                hypotheses/typhoon-asr-streaming-0.6b_560ms_tvspeech.jsonl
```

With one hypothesis file, `score.py` prints its character error rate. With two, it also prints the difference A − B with a 95% interval from 1,000 bootstrap resamples of whole clusters (seed 1). An interval that includes zero is a tie.

`make_refs.py` cleans the FLEURS and GigaSpeech2 transcripts the way our benchmark references were cleaned when we built them: ASCII punctuation except `?` and `!` removed (decimal points too), curly quotes, dashes and `…` removed, lower case, and PyThaiNLP's `normalize`. The other sets are used as published.

Normalisation in `score.py`, the same for reference and output: Thai digits become Arabic digits, curly quotes become straight ones, `ำ` is decomposed, spoken Thai numbers become digits (on whole words found by PyThaiNLP's `newmm`), and then every space is removed. The error rate is the edit distance over all characters of a set, divided by the number of reference characters.

To score the joined long recordings with your own system, build the audio the same way: the clips of one video or recording, in order, with exactly one second of digital silence between them, 16 kHz mono.

	# Benchmark outputs and scorer

	Every number on the model card can be checked from this folder. We decoded every system ourselves, and here are the per-clip outputs and the scorer we used.

	## What is here

	\| path \| contents \|
	\|---\|---\|
	\| `hypotheses/<system>_<ms>ms_<set>.jsonl` \| one line per clip: `{"id", "pred_text"}`, the system's output \|
	\| `hypotheses/<system>_560ms_<set>_split.jsonl` \| the joined long recordings (`tv_long`, `gs2_long`) decoded in pieces cut at pauses, as `serve/` does (`chinda_realtime.transcribe_long`), both models through the same code \|
	\| `ids/<set>.jsonl` \| one line per clip: `{"id", "cluster"}`, the clips of each test set and the group the bootstrap resamples \|
	\| `make_refs.py` \| rebuilds the reference transcripts from the public datasets, keyed by the same ids \|
	\| `score.py` \| character error rate, and the paired difference of two systems with a 95% bootstrap interval \|
	\| `chinda_norm.py`, `num_norm.py` \| the normalisation `score.py` applies to both sides \|

	Systems: `openthai2.0-asr-streaming` (this model, `th-TH` prompt), `openthai2.0-asr-streaming_th-mix` (`th-mix` prompt), `typhoon-asr-streaming-0.6b` (Typhoon ASR Streaming 0.6B as released) and `typhoon-asr-streaming-0.6b_4gram` (the same with its published 4-gram language model, weight 0.5).

	We do not redistribute the reference transcripts, because the datasets carry their own terms (GigaSpeech2 is gated). `make_refs.py` downloads only the transcript files and writes them in the format `score.py` reads.

	## Test sets and ids

	\| set \| source \| id \| cluster \|
	\|---\|---\|---\|---\|
	\| `fleurs` \| `google/fleurs`, `th_th` test \| audio file name \| the clip \|
	\| `tvspeech` \| `typhoon-ai/TVSpeech`, test \| row index \| YouTube video \|
	\| `gs2` \| `speechcolab/gigaspeech2`, Thai test \| segment id (`0-0`) \| recording \|
	\| `isan` \| `typhoon-ai/thai-dialect-isan-dataset`, test \| dataset id \| speaker \|
	\| `lotus_con123`, `lotus_lav123`, `lotus_jbl`, `lotus_bt3m`, `lotus_bt10m` \| `nectec/LOTUSDIS`, test: the same 5,516 turns on five devices (conference microphone, lapel microphones, speakerphone and the two far channels) \| chunk file name \| meeting \|
	\| `tv_long` \| TVSpeech, each video's clips joined in row order with 1 s of silence \| video id \| the file \|
	\| `gs2_long` \| GigaSpeech2 test, each recording's segments joined in segment order with 1 s of silence; recording 26 left out because its transcripts do not match its audio \| recording id \| the file \|

	## Score

	```bash
	pip install huggingface_hub pyarrow pythainlp==5.3.5 editdistance
	python make_refs.py --out refs/ # GigaSpeech2: accept its gate on Hugging Face first
	python score.py refs/tvspeech.jsonl hypotheses/openthai2.0-asr-streaming_560ms_tvspeech.jsonl \
	hypotheses/typhoon-asr-streaming-0.6b_560ms_tvspeech.jsonl
	```

	With one hypothesis file, `score.py` prints its character error rate. With two, it also prints the difference A − B with a 95% interval from 1,000 bootstrap resamples of whole clusters (seed 1). An interval that includes zero is a tie.

	`make_refs.py` cleans the FLEURS and GigaSpeech2 transcripts the way our benchmark references were cleaned when we built them: ASCII punctuation except `?` and `!` removed (decimal points too), curly quotes, dashes and `…` removed, lower case, and PyThaiNLP's `normalize`. The other sets are used as published.

	Normalisation in `score.py`, the same for reference and output: Thai digits become Arabic digits, curly quotes become straight ones, `ำ` is decomposed, spoken Thai numbers become digits (on whole words found by PyThaiNLP's `newmm`), and then every space is removed. The error rate is the edit distance over all characters of a set, divided by the number of reference characters.

	To score the joined long recordings with your own system, build the audio the same way: the clips of one video or recording, in order, with exactly one second of digital silence between them, 16 kHz mono.