Kodama-1

The Japanese speech recognition model used by Yamabiko. It is efwkjn/whisper-ja-760M with a fine-tuned decoder, in CTranslate2 format for faster-whisper.

Decode settings: no prompt (initial_prompt=None, condition_on_previous_text=False), and drop segments whose no_speech_prob is above 0.95.

Evaluation

Measured inside Yamabiko on 20 full anime episodes (about 8 hours) that weren't used for training. The episodes are scored against their Japanese subtitles. Subtitles leave out some of what's said, so a model that writes down more of the speech can lose a little on the error rate.

These numbers come from Yamabiko's whole pipeline, so they can't be compared with error rates published for other models. The base model ran with Yamabiko's older settings, so not all of the difference comes from the model.

	faster-whisper-ja-760M	Kodama-1	Difference (95% CI)
Text where the subtitles show nothing	51.7 per hour	37.0 per hour	−28% (−22.3 to −8.1 per hour)
Phantom ちゅ/じゅ sounds	18.6 per hour	3.1 per hour	−83% (−21.7 to −9.6 per hour)
Interjections caught	51.9%	60.4%	+8.6 points (+2.4 to +15.4)
Key words caught	63.1%	69.8%	+6.7 points (+4.3 to +9.1)
Subtitle lines caught	87.3%	88.2%	+0.9 points (+0.3 to +1.5)
Character error rate (kana readings)	15.7%	15.9%	+0.3 points (−0.6 to +1.2)

Licence and attributions

CC BY-SA 4.0.

• Base model: whisper-ja-760M by efwkjn, Apache License 2.0 (copy in LICENSE-APACHE-2.0). Modified: decoder fine-tuned, converted to CTranslate2.
• Training data:
  • ReazonSpeech v2 (used under Article 30-4 of the Japanese Copyright Act).
  • OOPPEENN visual-novel dataset.
  • JVNV (Xin et al.) and JNV (Xin et al.), both CC BY-SA 4.0.
  • MUSAN (Snyder et al.), CC BY 4.0.