X-MDD / README.md
walston's picture
Update CLI usage
52c0277 verified
|
Raw History Blame Contribute Delete
3.54 kB
---
license: apache-2.0
base_model: Qwen/Qwen3-ASR-1.7B
language:
- zh
- en
- ar
tags:
- mispronunciation-detection
- pronunciation-assessment
- speech
- qwen3-asr
- in-context-learning
- grpo
pipeline_tag: automatic-speech-recognition
---
# X-MDD (Qwen3-ASR-1.7B, canonical speech + phones, ICL + GRPO)
Main checkpoint of **X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages
with Canonical References** ([arXiv:2610.23079](https://arxiv.org/abs/2610.23079)).
Given the **canonical speech** (TTS-synthesized, voice-cloned from the learner), the **canonical phones**
and the **learner speech**, plus optional in-context demonstrations, the model generates the phone sequence
the learner actually produced. Aligning it to the canonical phones gives detection and diagnosis.
- Backbone: Qwen3-ASR-1.7B (AuT + projector + LLM decoder); all parameters fine-tuned except the AuT Conv2D front end
- Training: SFT on Mandarin + English (7k steps, 0–4 demos), then GRPO (300 steps, 1–2 demos, reward = 0.5·F1 + 0.5·(1−PER)); this is GRPO step 275
- Canonical speech TTS: Fun-CosyVoice3-0.5B (zh/en), SILMA TTS (ar)
| Test set | Demos | PER ↓ | F1 ↑ | DER ↓ |
|---|---|---|---|---|
| Mandarin | 4 | 17.66 | 79.00 | 31.32 |
| English (L2-ARCTIC) | 4 | 12.86 | 58.88 | 19.47 |
| Arabic, QuranMB (unseen) | 128 | 7.75 | 38.07 | 40.33 |
## Files
| File | Content |
|---|---|
| `model.safetensors` | Weights (bf16; keys `asr.*` = Qwen3-ASR; `ctc_head.*` = an unused CTC head, ignored at inference) |
| `config.yaml` | Inference config (prompt layout, language prefixes, model settings) |
| `processor/` | Qwen3-ASR processor and tokenizer, including the added prompt boundary tokens |
| `phone_vocabulary.txt` | The 242-phone training vocabulary (zh tonal phones + CMU 39 phones), in index order |
| `arabic_phone_symbols.txt` | Phone symbols used for constrained decoding on Arabic (QuranMB) |
## Usage
Install the [X-MDD code](https://github.com/oshindow/whisper_pinyin) (`xmdd` package), then:
```bash
hf download walston/X-MDD --local-dir pretrained/X-MDD
python -m xmdd.infer --model-dir pretrained/X-MDD --language en \
--manifest my_en_test.jsonl --support-manifest my_en_test.jsonl \
--constrain-phones my_en_phones.txt --shots 4 --out-dir results/en
```
```python
from xmdd import XMDD, load_manifest, select_demos
model = XMDD.from_pretrained("pretrained/X-MDD")
queries = load_manifest("my_en_test.jsonl", "en", model.cfg)
print(model.predict(queries[0], select_demos(queries[0], queries, shots=4)))
```
For a language other than zh/en/ar also pass `--language-name`. The base `Qwen/Qwen3-ASR-1.7B-hf` is downloaded for
the architecture and then overwritten by these weights.
Input manifest rows (JSONL) look like:
```json
{"id": "...", "speaker_id": "...", "l2_wav": "learner.wav", "canonical_wav": "canonical_tts.wav",
"canonical_phones": ["F", "AO", "R"], "actual_phones": ["F", "AO", "R"]}
```
`actual_phones` is only needed for demonstrations and for scoring. Demonstrations are picked from the same
speaker first. For an unseen language, pass raw phones, the language prefix (`ar` is already in
`config.yaml`) and optionally constrain decoding to the target phone set.
## Citation
```bibtex
@article{wang2026xmdd,
title = {X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References},
author = {Wang, Xintong and Zhao, Junchuan and Wang, Ye},
journal = {arXiv preprint arXiv:2610.23079},
year = {2026}
}
```