| --- |
| base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct |
| library_name: peft |
| license: apache-2.0 |
| pipeline_tag: audio-text-to-text |
| tags: |
| - qwen3_omni_moe |
| - audio |
| - audio-language-model |
| - audio-text-alignment |
| - rubric-based-evaluation |
| - judge-model |
| - lora |
| --- |
| |
| # FineAudio-Judge |
|
|
| > A Fine-Grained Evaluation Framework for Complex Text-to-Audio Generation |
|
|
| [Code](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge) | |
| [Benchmark](https://huggingface.co/datasets/Zixchen/FineAudio-Judge-Benchmark) |
|
|
| FineAudio-Judge is a fine-grained evaluation framework for assessing |
| whether audio matches text descriptions or generation instructions |
| for complex audio scenes. |
|
|
| Given an audio–text pair, the framework first structurally decomposes |
| the text into events, attributes, and inter-event relations, then |
| converts them into atomic binary rubric questions. The audio judge |
| answers each question with yes or no based on the audio, assessing |
| whether the audio matches the corresponding aspect of the text |
| description. The proportion of yes answers forms the overall |
| audio–text consistency score. |
|
|
| To train the audio judge, we construct a multi-route corpus covering |
| real, composed, and generated audio. Rubric-guided prompt/audio |
| perturbations and cross-sample mismatches provide fine-grained |
| supervision across varying levels of audio–text consistency. |
|
|
| This repository releases the resulting LoRA adapter, obtained by |
| supervised fine-tuning of |
| [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct). |
| The adapter needs to be loaded together with the base model. |
| Rubric decomposition uses a separately configured language-model |
| backend in the |
| [code repository](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge), |
| while the released adapter performs audio-based rubric judgments. |
|
|
| ## Files |
|
|
| - `adapter_config.json`: PEFT LoRA configuration. |
| - `adapter_model.safetensors`: FineAudio-Judge LoRA adapter weights. |
|
|
| ## Training |
|
|
| - **Base**: [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct) |
| - **Corpus**: 12,508 constructed audio–prompt pairs, with 278,199 rubric-level question–answer examples used for training |
| - **Stage**: LoRA supervised fine-tuning with a 1-epoch training schedule |
| - 8 × NVIDIA A800 80GB GPUs |
| - per-device batch size 1, gradient accumulation 8 |
| - learning rate 5e-5 |
| - LoRA rank 64, alpha 64, dropout 0.05 |
|
|
| ## Usage |
|
|
| The evaluation scripts are provided in the |
| [FineAudio-Judge code repository](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge). |
| They use ms-swift's vLLM inference engine to load |
| Qwen3-Omni-30B-A3B-Instruct together with the released LoRA adapter. |
|
|
| Clone the repository and enter the FineAudio-Judge directory: |
|
|
| ```bash |
| git clone https://github.com/QwenAudio/FunResearch.git |
| cd FunResearch/FineAudio-Judge |
| ``` |
|
|
| Follow the setup instructions in the code repository's README to install |
| the project and configure the inference environment. Run the evaluation |
| commands below from the `FunResearch/FineAudio-Judge/` directory. |
|
|
| Replace `/path/to/benchmark` with the downloaded benchmark repository |
| root and `/path/to/adapter` with the directory containing the released |
| adapter files. |
|
|
| ### Rubric-QA evaluation |
|
|
| Each audio–rubric question pair is evaluated independently. |
|
|
| ```bash |
| python -m judge.cli.reproduce_rubric_qa \ |
| --dataset-root /path/to/benchmark \ |
| --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ |
| --adapter /path/to/adapter \ |
| --output outputs/rubric_qa.jsonl |
| ``` |
|
|
| ### Human-preference evaluation |
|
|
| Rubrics are evaluated jointly for each candidate audio, and the proportion |
| of yes predictions forms its prompt–audio consistency score. |
|
|
| ```bash |
| python -m judge.cli.reproduce_preference \ |
| --dataset-root /path/to/benchmark \ |
| --model Qwen/Qwen3-Omni-30B-A3B-Instruct \ |
| --adapter /path/to/adapter \ |
| --output outputs/preference.jsonl |
| ``` |
|
|
| ## License |
|
|
| The FineAudio-Judge LoRA adapter is released under the Apache License 2.0. |
| See [LICENSE](LICENSE) for details. |
|
|
| The adapter is fine-tuned from |
| [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct), |
| which is also released under the |
| [Apache License 2.0](https://github.com/QwenLM/Qwen3-Omni/blob/main/LICENSE). |
| The base model weights are not included in this repository. |
|
|
| Base model copyright: Copyright 2025 Alibaba Cloud. |
|
|
