Cite the arXiv paper (2609.27289); add paper/code links; fill author list

22839f9 verified about 1 month ago

8.92 kB

---
license: apache-2.0
language:
- ja
base_model: Qwen/Qwen3-ASR-1.7B
pipeline_tag: automatic-speech-recognition
library_name: transformers
tags:
- automatic-speech-recognition
- japanese
- furigana
- ruby
- ctc
- qwen3-asr
---
# Ruby-ASR-1.7B: Japanese ASR with inline furigana + a CTC mora-reading head

📄 Paper: [arXiv:2609.27289](https://arxiv.org/abs/2609.27289) ·
💻 Code: [hshi-speech/Ruby-ASR-1.7B](https://github.com/hshi-speech/Ruby-ASR-1.7B)

Ruby-ASR-1.7B is a pair of Japanese fine-tunes of
[Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), shipped as
two variants in this repo:

| model | subfolder | style |
|---|---|---|
| **Ruby-ASR-sub** | `subtitle/` | subtitle-style transcription |
| **Ruby-ASR-ver** | `verbatim/` | verbatim-style transcription |

Use `subtitle` when prioritizing concise, readable orthographic transcription;
use `verbatim` when prioritizing script-aware and lexical-reading accuracy
(see [Evaluation](#evaluation)).

Both variants share the same architecture, mora vocab, and layout, and each
adds **two complementary outputs**:

1. **Ruby (furigana) transcription** — the standard Qwen3-ASR seq2seq decoder
   emits the transcript with inline bracket furigana:
   `七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、`
   Every annotated word carries its reading, so downstream text display
   (HTML `<ruby>`), reading-aware subtitling, and pronunciation-controlled TTS
   all get the reading for free.
2. **Mora reading (CTC)** — an auxiliary branch on the audio encoder
   (2-layer Transformer adapter + linear head, trained jointly with CTC loss)
   that outputs the **actual pronunciation** as kana morae directly from the
   encoder, without running the LLM decoder:
   `ナナネンカンデロスニモトモダチデキタシ`

Each variant's sharded main weights are a **stock Qwen3-ASR checkpoint** (the
CTC branch is stored separately under `ctc/`), so Mode 2 works with unmodified
vLLM and the official [`qwen-asr`](https://github.com/QwenLM/Qwen3-ASR)
toolkit. Mode 1 is provided by the
[`ruby_asr`](https://github.com/hshi-speech/Ruby-ASR-1.7B) package.

## Repo layout

```
subtitle/  verbatim/                    (same layout in both)
├── model-0000x-of-00003.safetensors    sharded Qwen3-ASR weights (CTC stripped) — Mode 2
├── model.safetensors.index.json
├── config.json, generation_config.json
├── tokenizer.json, tokenizer_config.json, vocab.json, merges.txt,
│   added_tokens.json, special_tokens_map.json, chat_template.json,
│   preprocessor_config.json
└── ctc/
    ├── model.safetensors               the CTC branch (ctc_adapter.* + ctc_head.*) — Mode 1
    ├── config.json                     adapter hyper-params (transformer, 2 layers, nhead 8), blank id
    └── mora_vocab.json                 {mora: id}; id 0 = <blank>, ids 1-26 = a-z, then morae
```

## Mode 1 — mora reading via encoder + CTC

No LLM decoding: audio → encoder frames → CTC adapter → head → greedy collapse.
Fast, and reflects the *actual pronunciation* (e.g. numbers and kanji resolved
to what was said). Loads straight from this repo:

```python
# pip install git+https://github.com/hshi-speech/Ruby-ASR-1.7B.git
from ruby_asr import MoraCTCRecognizer
rec = MoraCTCRecognizer.from_pretrained("hshispeech/Ruby-ASR-1.7B",
                                        subfolder="subtitle")   # or "verbatim"
print(rec.transcribe("audio.wav")[0])
# ナナネンカンデロスニモトモダチデキタシ
```

Output alphabet: katakana morae (`キョ`, `ッ`, `ン`, `ー` are single tokens)
plus `a`–`z` for embedded English. Long vowels are unified to `ー`
(`トウキョウ` → `トーキョー`), matching the pyopenjtalk `g2p(kana=True)`
convention the training targets were built with.

## Mode 2 — ruby transcription (transformers)

Download the variant you need, then load it like stock Qwen3-ASR:

```python
# pip install qwen-asr
import torch
from huggingface_hub import snapshot_download
from qwen_asr import Qwen3ASRModel
root = snapshot_download("hshispeech/Ruby-ASR-1.7B", allow_patterns=["subtitle/*"])
model = Qwen3ASRModel.from_pretrained(f"{root}/subtitle",
                                      dtype=torch.bfloat16, device_map="cuda")
print(model.transcribe(audio="audio.wav")[0].text)
# 七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、
```

Post-process the bracket output (surface / reading / HTML `<ruby>`) with the
`ruby_asr` helpers:

```python
from ruby_asr import to_surface, to_reading, to_html, brackets_to_parens
text = "七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、"
to_surface(text)   # 七年間でロスにも友達できたし、
to_reading(text)   # ななねんかんでロスにもともだちできたし、
to_html(brackets_to_parens(text))  # <ruby><rb>七</rb><rt>なな</rt></ruby>…
```

## Mode 2 — ruby transcription (vLLM)

```bash
pip install "qwen-asr[vllm]"     # or your own vLLM >= 0.14 install
hf download hshispeech/Ruby-ASR-1.7B --include "subtitle/*" --local-dir Ruby-ASR-1.7B
vllm serve Ruby-ASR-1.7B/subtitle --served-model-name ruby-asr --max-model-len 8192
```

```python
import base64
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
audio_b64 = base64.b64encode(open("audio.wav", "rb").read()).decode()
resp = client.chat.completions.create(
    model="ruby-asr",
    messages=[
        {"role": "system", "content": ""},
        {"role": "user", "content": [
            {"type": "input_audio",
             "input_audio": {"data": audio_b64, "format": "wav"}}]},
    ],
    temperature=0.0, max_tokens=512)
print(resp.choices[0].message.content)
# language Japanese<asr_text>七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、
```

> **Use `/v1/chat/completions`, not `/v1/audio/transcriptions`.** The furigana
> is the model's learned behaviour under the standard Qwen3-ASR chat prompt
> (empty system + audio). The transcriptions endpoint builds a different
> internal prompt and returns plain text without brackets.

## Training objective

Fine-tuned from Qwen3-ASR-1.7B with a joint loss `0.3 · CTC + 0.7 · CE`: the
standard cross-entropy over the seq2seq ruby transcript plus a CTC loss over
mora targets from the auxiliary encoder branch. The seq2seq path never sees
the CTC adapter, so ruby decoding is bit-identical to a checkpoint without
the branch.

## Evaluation

Character-count-weighted averages (W.Avg.) over five Japanese benchmarks —
B5K, CSJ, JSUT-Book, Common Voice 8, and TEDx — weighted by the number of
reference characters per test set. All values are percentages; lower is
better.

| Model           |  Raw CER |   SA-CER | Kana CER |
| --------------- | -------: | -------: | -------: |
| ReazonSpeech-k2 |    10.35 |     8.51 |     5.64 |
| Qwen3-ASR-1.7B  |    10.78 |     8.59 |     5.74 |
| Kana-Whisper    |      N/A |      N/A |     4.66 |
| Ruby-ASR-ver    |     9.10 | **6.59** | **3.75** |
| Ruby-ASR-sub    | **8.51** |     6.64 |     4.01 |

On this weighted aggregate, **Ruby-ASR-sub** achieves the best Raw CER
(8.51%, a 21.1% relative reduction over Qwen3-ASR), and **Ruby-ASR-ver** the
best SA-CER (6.59%) and Kana CER (3.75%) — relative reductions of 23.3% and
34.7% over Qwen3-ASR. Compared with Kana-Whisper, Ruby-ASR-ver reduces
weighted Kana CER from 4.66% to 3.75%, a 19.5% relative reduction.

Metrics:

- **Raw CER** — normalized orthographic transcription, with the original
  script preserved.
- **SA-CER** — script-aware: kana regions are scored by reading, kanji
  regions by orthography.
- **Kana CER** — lexical-reading transcription.
- Ruby-ASR and Kana-Whisper are scored on their direct reading outputs; the
  orthographic baselines are converted through the shared G2P pipeline.

## Limitations

- Japanese-focused fine-tune; other Qwen3-ASR languages were not part of
  training and quality on them is untested.
- Mode 1 output is pronunciation kana only — no kanji, punctuation, or word
  boundaries.
- Furigana brackets are learned behaviour, not constrained decoding: rare
  formatting slips (e.g. an empty `[]` on numbers) are possible; the parser in
  the code repo handles them.

## Citation

```bibtex
@misc{shi2026rubyasrevidencepreservingsupervisionjoint,
  title         = {Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition},
  author        = {Hao Shi and Yun Liu and Xuehao Yang and Jun Liu and Chuanbo Hua and Xuanjun Chen and Lianbo Liu and Shiao Zhu and Zixiong Su},
  year          = {2026},
  eprint        = {2609.27289},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2609.27289},
}
```

## License

Apache-2.0, same as the Qwen3-ASR base model.

	---
	license: apache-2.0
	language:
	- ja
	base_model: Qwen/Qwen3-ASR-1.7B
	pipeline_tag: automatic-speech-recognition
	library_name: transformers
	tags:
	- automatic-speech-recognition
	- japanese
	- furigana
	- ruby
	- ctc
	- qwen3-asr
	---

	# Ruby-ASR-1.7B: Japanese ASR with inline furigana + a CTC mora-reading head

	📄 Paper: [arXiv:2609.27289](https://arxiv.org/abs/2609.27289) ·
	💻 Code: [hshi-speech/Ruby-ASR-1.7B](https://github.com/hshi-speech/Ruby-ASR-1.7B)

	Ruby-ASR-1.7B is a pair of Japanese fine-tunes of
	[Qwen/Qwen3-ASR-1.7B](https://huggingface.co/Qwen/Qwen3-ASR-1.7B), shipped as
	two variants in this repo:

	\| model \| subfolder \| style \|
	\|---\|---\|---\|
	\| Ruby-ASR-sub \| `subtitle/` \| subtitle-style transcription \|
	\| Ruby-ASR-ver \| `verbatim/` \| verbatim-style transcription \|

	Use `subtitle` when prioritizing concise, readable orthographic transcription;
	use `verbatim` when prioritizing script-aware and lexical-reading accuracy
	(see [Evaluation](#evaluation)).

	Both variants share the same architecture, mora vocab, and layout, and each
	adds two complementary outputs:

	1. Ruby (furigana) transcription — the standard Qwen3-ASR seq2seq decoder
	emits the transcript with inline bracket furigana:
	`七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、`
	Every annotated word carries its reading, so downstream text display
	(HTML `<ruby>`), reading-aware subtitling, and pronunciation-controlled TTS
	all get the reading for free.
	2. Mora reading (CTC) — an auxiliary branch on the audio encoder
	(2-layer Transformer adapter + linear head, trained jointly with CTC loss)
	that outputs the actual pronunciation as kana morae directly from the
	encoder, without running the LLM decoder:
	`ナナネンカンデロスニモトモダチデキタシ`

	Each variant's sharded main weights are a stock Qwen3-ASR checkpoint (the
	CTC branch is stored separately under `ctc/`), so Mode 2 works with unmodified
	vLLM and the official [`qwen-asr`](https://github.com/QwenLM/Qwen3-ASR)
	toolkit. Mode 1 is provided by the
	[`ruby_asr`](https://github.com/hshi-speech/Ruby-ASR-1.7B) package.

	## Repo layout

	```
	subtitle/ verbatim/ (same layout in both)
	├── model-0000x-of-00003.safetensors sharded Qwen3-ASR weights (CTC stripped) — Mode 2
	├── model.safetensors.index.json
	├── config.json, generation_config.json
	├── tokenizer.json, tokenizer_config.json, vocab.json, merges.txt,
	│ added_tokens.json, special_tokens_map.json, chat_template.json,
	│ preprocessor_config.json
	└── ctc/
	├── model.safetensors the CTC branch (ctc_adapter.* + ctc_head.*) — Mode 1
	├── config.json adapter hyper-params (transformer, 2 layers, nhead 8), blank id
	└── mora_vocab.json {mora: id}; id 0 = <blank>, ids 1-26 = a-z, then morae
	```

	## Mode 1 — mora reading via encoder + CTC

	No LLM decoding: audio → encoder frames → CTC adapter → head → greedy collapse.
	Fast, and reflects the actual pronunciation (e.g. numbers and kanji resolved
	to what was said). Loads straight from this repo:

	```python
	# pip install git+https://github.com/hshi-speech/Ruby-ASR-1.7B.git
	from ruby_asr import MoraCTCRecognizer

	rec = MoraCTCRecognizer.from_pretrained("hshispeech/Ruby-ASR-1.7B",
	subfolder="subtitle") # or "verbatim"
	print(rec.transcribe("audio.wav")[0])
	# ナナネンカンデロスニモトモダチデキタシ
	```

	Output alphabet: katakana morae (`キョ`, `ッ`, `ン`, `ー` are single tokens)
	plus `a`–`z` for embedded English. Long vowels are unified to `ー`
	(`トウキョウ` → `トーキョー`), matching the pyopenjtalk `g2p(kana=True)`
	convention the training targets were built with.

	## Mode 2 — ruby transcription (transformers)

	Download the variant you need, then load it like stock Qwen3-ASR:

	```python
	# pip install qwen-asr
	import torch
	from huggingface_hub import snapshot_download
	from qwen_asr import Qwen3ASRModel

	root = snapshot_download("hshispeech/Ruby-ASR-1.7B", allow_patterns=["subtitle/*"])
	model = Qwen3ASRModel.from_pretrained(f"{root}/subtitle",
	dtype=torch.bfloat16, device_map="cuda")
	print(model.transcribe(audio="audio.wav")[0].text)
	# 七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、
	```

	Post-process the bracket output (surface / reading / HTML `<ruby>`) with the
	`ruby_asr` helpers:

	```python
	from ruby_asr import to_surface, to_reading, to_html, brackets_to_parens
	text = "七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、"
	to_surface(text) # 七年間でロスにも友達できたし、
	to_reading(text) # ななねんかんでロスにもともだちできたし、
	to_html(brackets_to_parens(text)) # <ruby><rb>七</rb><rt>なな</rt></ruby>…
	```

	## Mode 2 — ruby transcription (vLLM)

	```bash
	pip install "qwen-asr[vllm]" # or your own vLLM >= 0.14 install
	hf download hshispeech/Ruby-ASR-1.7B --include "subtitle/*" --local-dir Ruby-ASR-1.7B
	vllm serve Ruby-ASR-1.7B/subtitle --served-model-name ruby-asr --max-model-len 8192
	```

	```python
	import base64
	from openai import OpenAI

	client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
	audio_b64 = base64.b64encode(open("audio.wav", "rb").read()).decode()
	resp = client.chat.completions.create(
	model="ruby-asr",
	messages=[
	{"role": "system", "content": ""},
	{"role": "user", "content": [
	{"type": "input_audio",
	"input_audio": {"data": audio_b64, "format": "wav"}}]},
	],
	temperature=0.0, max_tokens=512)
	print(resp.choices[0].message.content)
	# language Japanese<asr_text>七[なな]年間[ねんかん]でロスにも友達[ともだち]できたし、
	```

	> Use `/v1/chat/completions`, not `/v1/audio/transcriptions`. The furigana
	> is the model's learned behaviour under the standard Qwen3-ASR chat prompt
	> (empty system + audio). The transcriptions endpoint builds a different
	> internal prompt and returns plain text without brackets.

	## Training objective

	Fine-tuned from Qwen3-ASR-1.7B with a joint loss `0.3 · CTC + 0.7 · CE`: the
	standard cross-entropy over the seq2seq ruby transcript plus a CTC loss over
	mora targets from the auxiliary encoder branch. The seq2seq path never sees
	the CTC adapter, so ruby decoding is bit-identical to a checkpoint without
	the branch.

	## Evaluation

	Character-count-weighted averages (W.Avg.) over five Japanese benchmarks —
	B5K, CSJ, JSUT-Book, Common Voice 8, and TEDx — weighted by the number of
	reference characters per test set. All values are percentages; lower is
	better.

	\| Model \| Raw CER \| SA-CER \| Kana CER \|
	\| --------------- \| -------: \| -------: \| -------: \|
	\| ReazonSpeech-k2 \| 10.35 \| 8.51 \| 5.64 \|
	\| Qwen3-ASR-1.7B \| 10.78 \| 8.59 \| 5.74 \|
	\| Kana-Whisper \| N/A \| N/A \| 4.66 \|
	\| Ruby-ASR-ver \| 9.10 \| 6.59 \| 3.75 \|
	\| Ruby-ASR-sub \| 8.51 \| 6.64 \| 4.01 \|

	On this weighted aggregate, Ruby-ASR-sub achieves the best Raw CER
	(8.51%, a 21.1% relative reduction over Qwen3-ASR), and Ruby-ASR-ver the
	best SA-CER (6.59%) and Kana CER (3.75%) — relative reductions of 23.3% and
	34.7% over Qwen3-ASR. Compared with Kana-Whisper, Ruby-ASR-ver reduces
	weighted Kana CER from 4.66% to 3.75%, a 19.5% relative reduction.

	Metrics:

	- Raw CER — normalized orthographic transcription, with the original
	script preserved.
	- SA-CER — script-aware: kana regions are scored by reading, kanji
	regions by orthography.
	- Kana CER — lexical-reading transcription.
	- Ruby-ASR and Kana-Whisper are scored on their direct reading outputs; the
	orthographic baselines are converted through the shared G2P pipeline.

	## Limitations

	- Japanese-focused fine-tune; other Qwen3-ASR languages were not part of
	training and quality on them is untested.
	- Mode 1 output is pronunciation kana only — no kanji, punctuation, or word
	boundaries.
	- Furigana brackets are learned behaviour, not constrained decoding: rare
	formatting slips (e.g. an empty `[]` on numbers) are possible; the parser in
	the code repo handles them.

	## Citation

	```bibtex
	@misc{shi2026rubyasrevidencepreservingsupervisionjoint,
	title = {Ruby-ASR: Evidence-Preserving Supervision for Joint Orthographic and Lexical-Reading Recognition},
	author = {Hao Shi and Yun Liu and Xuehao Yang and Jun Liu and Chuanbo Hua and Xuanjun Chen and Lianbo Liu and Shiao Zhu and Zixiong Su},
	year = {2026},
	eprint = {2609.27289},
	archivePrefix = {arXiv},
	primaryClass = {cs.CL},
	url = {https://arxiv.org/abs/2609.27289},
	}
	```

	## License

	Apache-2.0, same as the Qwen3-ASR base model.