--- license: apache-2.0 base_model: Qwen/Qwen3-ASR-1.7B language: - zh - en - ar tags: - mispronunciation-detection - pronunciation-assessment - speech - qwen3-asr - in-context-learning - grpo pipeline_tag: automatic-speech-recognition --- # X-MDD (Qwen3-ASR-1.7B, canonical speech + phones, ICL + GRPO) Main checkpoint of **X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References** ([arXiv:2610.23079](https://arxiv.org/abs/2610.23079)). Given the **canonical speech** (TTS-synthesized, voice-cloned from the learner), the **canonical phones** and the **learner speech**, plus optional in-context demonstrations, the model generates the phone sequence the learner actually produced. Aligning it to the canonical phones gives detection and diagnosis. - Backbone: Qwen3-ASR-1.7B (AuT + projector + LLM decoder); all parameters fine-tuned except the AuT Conv2D front end - Training: SFT on Mandarin + English (7k steps, 0–4 demos), then GRPO (300 steps, 1–2 demos, reward = 0.5·F1 + 0.5·(1−PER)); this is GRPO step 275 - Canonical speech TTS: Fun-CosyVoice3-0.5B (zh/en), SILMA TTS (ar) | Test set | Demos | PER ↓ | F1 ↑ | DER ↓ | |---|---|---|---|---| | Mandarin | 4 | 17.66 | 79.00 | 31.32 | | English (L2-ARCTIC) | 4 | 12.86 | 58.88 | 19.47 | | Arabic, QuranMB (unseen) | 128 | 7.75 | 38.07 | 40.33 | ## Files | File | Content | |---|---| | `model.safetensors` | Weights (bf16; keys `asr.*` = Qwen3-ASR; `ctc_head.*` = an unused CTC head, ignored at inference) | | `config.yaml` | Inference config (prompt layout, language prefixes, model settings) | | `processor/` | Qwen3-ASR processor and tokenizer, including the added prompt boundary tokens | | `phone_vocabulary.txt` | The 242-phone training vocabulary (zh tonal phones + CMU 39 phones), in index order | | `arabic_phone_symbols.txt` | Phone symbols used for constrained decoding on Arabic (QuranMB) | ## Usage Install the [X-MDD code](https://github.com/oshindow/whisper_pinyin) (`xmdd` package), then: ```bash hf download walston/X-MDD --local-dir pretrained/X-MDD python -m xmdd.infer --model-dir pretrained/X-MDD --language en \ --manifest my_en_test.jsonl --support-manifest my_en_test.jsonl \ --constrain-phones my_en_phones.txt --shots 4 --out-dir results/en ``` ```python from xmdd import XMDD, load_manifest, select_demos model = XMDD.from_pretrained("pretrained/X-MDD") queries = load_manifest("my_en_test.jsonl", "en", model.cfg) print(model.predict(queries[0], select_demos(queries[0], queries, shots=4))) ``` For a language other than zh/en/ar also pass `--language-name`. The base `Qwen/Qwen3-ASR-1.7B-hf` is downloaded for the architecture and then overwritten by these weights. Input manifest rows (JSONL) look like: ```json {"id": "...", "speaker_id": "...", "l2_wav": "learner.wav", "canonical_wav": "canonical_tts.wav", "canonical_phones": ["F", "AO", "R"], "actual_phones": ["F", "AO", "R"]} ``` `actual_phones` is only needed for demonstrations and for scoring. Demonstrations are picked from the same speaker first. For an unseen language, pass raw phones, the language prefix (`ar` is already in `config.yaml`) and optionally constrain decoding to the target phone set. ## Citation ```bibtex @article{wang2026xmdd, title = {X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References}, author = {Wang, Xintong and Zhao, Junchuan and Wang, Ye}, journal = {arXiv preprint arXiv:2610.23079}, year = {2026} } ```