Automatic Speech Recognition
NeMo
Japanese
speech
audio
japanese
TDT
FastConformer
Conformer
KDA
NeMo
sabizushi commited on
Commit
efd7b81
·
verified ·
1 Parent(s): 8474e1e

Model card: custom vocabulary, speaker labels and updated limitations

Browse files

Adds the custom vocabulary and the speaker-label example (with NVIDIA's Nemotron-3-Diarization) to How to use, names GPT-Transcribe as on par in the results, updates the limitations to match, adds a link to the demo Space, and follows the inference code's changes (Python 3.11+, example file names).

Files changed (1) hide show
  1. README.md +49 -13
README.md CHANGED
@@ -22,6 +22,7 @@ tags:
22
  # Haas-ja-v1.0
23
 
24
  [![Code: GitHub](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/third-intelligence/Haas-ASR)
 
25
  [![Blog](https://img.shields.io/badge/Blog-Third%20Intelligence-green)](TODO-blog-url)
26
  [![Benchmark: asr-ja-bench](https://img.shields.io/badge/Benchmark-asr--ja--bench-orange)](https://huggingface.co/datasets/third-intelligence/asr-ja-bench)
27
  [![Metrics: jacer](https://img.shields.io/badge/Metrics-jacer-purple)](https://github.com/third-intelligence/jacer)
@@ -67,7 +68,7 @@ The full architecture configuration is in [`config.yaml`](config.yaml).
67
 
68
  ## How to use
69
 
70
- Requirements: Linux with an NVIDIA GPU, Python 3.10+ and NeMo 2.7. The prediction
71
  network runs on kernels from
72
  [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which
73
  needs Triton and CUDA, so CPU and macOS are not supported.
@@ -82,7 +83,7 @@ dependency versions we tested:
82
  git clone https://github.com/third-intelligence/Haas-ASR.git
83
  cd Haas-ASR
84
  uv sync
85
- uv run python examples/quickstart.py --model third-intelligence/Haas-ja-v1.0 audio.wav
86
  ```
87
 
88
  **Use it in your own project**
@@ -99,7 +100,7 @@ Then transcribe:
99
  import haas_asr
100
 
101
  model = haas_asr.load_model("third-intelligence/Haas-ja-v1.0")
102
- audio = haas_asr.load_audio("audio.wav")
103
  output = model.transcribe([audio])
104
  print(output[0].text)
105
  ```
@@ -112,10 +113,42 @@ For recordings longer than 30 seconds, use `transcribe_longform`. It splits the
112
  at quiet points (detected by volume) and transcribes the pieces in batches:
113
 
114
  ```python
115
- audio = haas_asr.load_audio("lecture.flac")
116
  text = haas_asr.transcribe_longform(model, audio, batch_size=64)
117
  ```
118
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
  ## Training data
120
 
121
  The model was trained on about 46,000 hours of audio from public corpora
@@ -173,8 +206,8 @@ In each cell, the top value is CER strict and the bottom values are CER flex / k
173
  (%, lower is better). Models are sorted by weighted CER strict.
174
 
175
  Haas-ja-v1.0 reaches a weighted CER strict of 6.12%, the best of the open models
176
- evaluated. It is competitive with commercial APIs, though MAI-Transcribe 2, Scribe v2 and
177
- Gemini 3.1 Pro remain more accurate.
178
 
179
  ![CER of each model on the five datasets and RTFx](assets/cer_table.png)
180
 
@@ -210,13 +243,14 @@ running a service. API costs are what OpenRouter billed during the evaluation.
210
  utterance at once. It can still produce live captions by re-transcribing the current
211
  phrase as audio arrives (see the demo in the repository), but low-latency streaming
212
  may need further tuning.
213
- - **Non-verbal information.** The model transcribes what is said, not who says it or how:
214
- speaker identity, emotion and other non-verbal information are not recognized.
215
- - **Custom vocabulary.** There is no built-in way to supply your own domain terms or
216
- proper nouns.
217
  - **Output format.** The output style cannot be controlled, for example whether fillers
218
  are included. The model tends to omit fillers, and its choices of kanji, kana and
219
- numerals follow its training data.
 
220
 
221
  ## Acknowledgements
222
 
@@ -228,8 +262,10 @@ It builds on [NVIDIA NeMo](https://github.com/NVIDIA/NeMo) and the
228
  prediction network uses Kimi Delta Attention from
229
  [Kimi Linear](https://arxiv.org/abs/2510.26692), run on the kernels of
230
  [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), and it was
231
- trained with [Lhotse](https://github.com/lhotse-speech/lhotse). We thank the authors
232
- of these projects for making them openly available.
 
 
233
 
234
  We also thank the creators of [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech),
235
  [Common Voice](https://commonvoice.mozilla.org) and
 
22
  # Haas-ja-v1.0
23
 
24
  [![Code: GitHub](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/third-intelligence/Haas-ASR)
25
+ [![Demo: Hugging Face Space](https://img.shields.io/badge/Demo-Hugging%20Face%20Space-yellow?logo=huggingface)](https://huggingface.co/spaces/third-intelligence/Haas-ja-v1.0)
26
  [![Blog](https://img.shields.io/badge/Blog-Third%20Intelligence-green)](TODO-blog-url)
27
  [![Benchmark: asr-ja-bench](https://img.shields.io/badge/Benchmark-asr--ja--bench-orange)](https://huggingface.co/datasets/third-intelligence/asr-ja-bench)
28
  [![Metrics: jacer](https://img.shields.io/badge/Metrics-jacer-purple)](https://github.com/third-intelligence/jacer)
 
68
 
69
  ## How to use
70
 
71
+ Requirements: Linux with an NVIDIA GPU, Python 3.11+ and NeMo 2.7. The prediction
72
  network runs on kernels from
73
  [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), which
74
  needs Triton and CUDA, so CPU and macOS are not supported.
 
83
  git clone https://github.com/third-intelligence/Haas-ASR.git
84
  cd Haas-ASR
85
  uv sync
86
+ uv run python examples/quickstart.py --model third-intelligence/Haas-ja-v1.0 example.wav
87
  ```
88
 
89
  **Use it in your own project**
 
100
  import haas_asr
101
 
102
  model = haas_asr.load_model("third-intelligence/Haas-ja-v1.0")
103
+ audio = haas_asr.load_audio("example.wav")
104
  output = model.transcribe([audio])
105
  print(output[0].text)
106
  ```
 
113
  at quiet points (detected by volume) and transcribes the pieces in batches:
114
 
115
  ```python
116
+ audio = haas_asr.load_audio("example.flac")
117
  text = haas_asr.transcribe_longform(model, audio, batch_size=64)
118
  ```
119
 
120
+ **Custom vocabulary**
121
+
122
+ Names and terms the model rarely heard in training can be listed, so that it
123
+ recognizes them more often and writes them as you want:
124
+
125
+ ```python
126
+ vocabulary = haas_asr.Vocabulary(
127
+ {"Haas": ["ハース"], "Third Intelligence": ["サードインテリジェンス"]}
128
+ )
129
+ texts = haas_asr.transcribe(model, [audio], vocabulary=vocabulary) # short clips
130
+ text = haas_asr.transcribe_longform(model, audio, vocabulary=vocabulary)
131
+ ```
132
+
133
+ Each word maps its written form to its readings, the ways Haas would write it (usually
134
+ in kana; several, or none if Haas already writes it as listed). The readings are
135
+ boosted while decoding with NeMo's
136
+ [GPU phrase boosting](https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/asr/asr_customization/word_boosting.html),
137
+ and replaced by the written form wherever found. Boosting can also change text where
138
+ no listed word was said, so audio without speech is not boosted, and each audio is
139
+ also decoded without the words: a change is kept only where it includes a whole listed
140
+ word. Both decodes share one encoder pass, so a vocabulary costs about 1.3 times the
141
+ time. See the
142
+ [repository](https://github.com/third-intelligence/Haas-ASR#custom-vocabulary) for
143
+ details.
144
+
145
+ **Speaker labels**
146
+
147
+ Combined with NVIDIA's speaker diarization model
148
+ [Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization), Haas
149
+ produces transcripts with speaker labels. The repository has a
150
+ [runnable example](https://github.com/third-intelligence/Haas-ASR#speaker-labels-with-nemotron-3-diarization).
151
+
152
  ## Training data
153
 
154
  The model was trained on about 46,000 hours of audio from public corpora
 
206
  (%, lower is better). Models are sorted by weighted CER strict.
207
 
208
  Haas-ja-v1.0 reaches a weighted CER strict of 6.12%, the best of the open models
209
+ evaluated. It is competitive with commercial APIs: on par with GPT-Transcribe (6.07%),
210
+ while MAI-Transcribe 2, Scribe v2 and Gemini 3.1 Pro remain more accurate.
211
 
212
  ![CER of each model on the five datasets and RTFx](assets/cer_table.png)
213
 
 
243
  utterance at once. It can still produce live captions by re-transcribing the current
244
  phrase as audio arrives (see the demo in the repository), but low-latency streaming
245
  may need further tuning.
246
+ - **Non-verbal information.** On its own, the model transcribes what is said, not who
247
+ says it or how: speaker identity, emotion and other non-verbal information are not
248
+ recognized. Speaker labels are possible by combining it with a diarization model (see
249
+ How to use).
250
  - **Output format.** The output style cannot be controlled, for example whether fillers
251
  are included. The model tends to omit fillers, and its choices of kanji, kana and
252
+ numerals follow its training data. The custom vocabulary can fix the spelling of
253
+ individual words, but not the style as a whole.
254
 
255
  ## Acknowledgements
256
 
 
262
  prediction network uses Kimi Delta Attention from
263
  [Kimi Linear](https://arxiv.org/abs/2510.26692), run on the kernels of
264
  [flash-linear-attention](https://github.com/fla-org/flash-linear-attention), and it was
265
+ trained with [Lhotse](https://github.com/lhotse-speech/lhotse). The speaker-label
266
+ example uses NVIDIA's
267
+ [Nemotron-3-Diarization](https://huggingface.co/nvidia/Nemotron-3-Diarization)
268
+ (OpenMDW-1.1). We thank the authors of these projects for making them openly available.
269
 
270
  We also thank the creators of [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech),
271
  [Common Voice](https://commonvoice.mozilla.org) and