|
Download README.md from walston/X-MDD: direct link, hf CLI and curl.
- Browser
- Download file 3.54 kB
-
https://huggingface.co/walston/X-MDD/resolve/main/README.md
- Command line
-
hf download hf://walston/X-MDD/README.md
-
curl -L -o README.md https://huggingface.co/walston/X-MDD/resolve/main/README.md
3.54 kB
| license: apache-2.0 | |
| base_model: Qwen/Qwen3-ASR-1.7B | |
| language: | |
| - zh | |
| - en | |
| - ar | |
| tags: | |
| - mispronunciation-detection | |
| - pronunciation-assessment | |
| - speech | |
| - qwen3-asr | |
| - in-context-learning | |
| - grpo | |
| pipeline_tag: automatic-speech-recognition | |
| # X-MDD (Qwen3-ASR-1.7B, canonical speech + phones, ICL + GRPO) | |
| Main checkpoint of **X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages | |
| with Canonical References** ([arXiv:2610.23079](https://arxiv.org/abs/2610.23079)). | |
| Given the **canonical speech** (TTS-synthesized, voice-cloned from the learner), the **canonical phones** | |
| and the **learner speech**, plus optional in-context demonstrations, the model generates the phone sequence | |
| the learner actually produced. Aligning it to the canonical phones gives detection and diagnosis. | |
| - Backbone: Qwen3-ASR-1.7B (AuT + projector + LLM decoder); all parameters fine-tuned except the AuT Conv2D front end | |
| - Training: SFT on Mandarin + English (7k steps, 0–4 demos), then GRPO (300 steps, 1–2 demos, reward = 0.5·F1 + 0.5·(1−PER)); this is GRPO step 275 | |
| - Canonical speech TTS: Fun-CosyVoice3-0.5B (zh/en), SILMA TTS (ar) | |
| | Test set | Demos | PER ↓ | F1 ↑ | DER ↓ | | |
| |---|---|---|---|---| | |
| | Mandarin | 4 | 17.66 | 79.00 | 31.32 | | |
| | English (L2-ARCTIC) | 4 | 12.86 | 58.88 | 19.47 | | |
| | Arabic, QuranMB (unseen) | 128 | 7.75 | 38.07 | 40.33 | | |
| ## Files | |
| | File | Content | | |
| |---|---| | |
| | `model.safetensors` | Weights (bf16; keys `asr.*` = Qwen3-ASR; `ctc_head.*` = an unused CTC head, ignored at inference) | | |
| | `config.yaml` | Inference config (prompt layout, language prefixes, model settings) | | |
| | `processor/` | Qwen3-ASR processor and tokenizer, including the added prompt boundary tokens | | |
| | `phone_vocabulary.txt` | The 242-phone training vocabulary (zh tonal phones + CMU 39 phones), in index order | | |
| | `arabic_phone_symbols.txt` | Phone symbols used for constrained decoding on Arabic (QuranMB) | | |
| ## Usage | |
| Install the [X-MDD code](https://github.com/oshindow/whisper_pinyin) (`xmdd` package), then: | |
| ```bash | |
| hf download walston/X-MDD --local-dir pretrained/X-MDD | |
| python -m xmdd.infer --model-dir pretrained/X-MDD --language en \ | |
| --manifest my_en_test.jsonl --support-manifest my_en_test.jsonl \ | |
| --constrain-phones my_en_phones.txt --shots 4 --out-dir results/en | |
| ``` | |
| ```python | |
| from xmdd import XMDD, load_manifest, select_demos | |
| model = XMDD.from_pretrained("pretrained/X-MDD") | |
| queries = load_manifest("my_en_test.jsonl", "en", model.cfg) | |
| print(model.predict(queries[0], select_demos(queries[0], queries, shots=4))) | |
| ``` | |
| For a language other than zh/en/ar also pass `--language-name`. The base `Qwen/Qwen3-ASR-1.7B-hf` is downloaded for | |
| the architecture and then overwritten by these weights. | |
| Input manifest rows (JSONL) look like: | |
| ```json | |
| {"id": "...", "speaker_id": "...", "l2_wav": "learner.wav", "canonical_wav": "canonical_tts.wav", | |
| "canonical_phones": ["F", "AO", "R"], "actual_phones": ["F", "AO", "R"]} | |
| ``` | |
| `actual_phones` is only needed for demonstrations and for scoring. Demonstrations are picked from the same | |
| speaker first. For an unseen language, pass raw phones, the language prefix (`ar` is already in | |
| `config.yaml`) and optionally constrain decoding to the target phone set. | |
| ## Citation | |
| ```bibtex | |
| @article{wang2026xmdd, | |
| title = {X-MDD: LLM-Based Mispronunciation Detection and Diagnosis for Unseen Languages with Canonical References}, | |
| author = {Wang, Xintong and Zhao, Junchuan and Wang, Ye}, | |
| journal = {arXiv preprint arXiv:2610.23079}, | |
| year = {2026} | |
| } | |
| ``` | |