File size: 4,378 Bytes
ccb63f9 b5acfa3 ccb63f9 b5acfa3 8f55826 ccb63f9 b5acfa3 8f55826 b5acfa3 8f55826 b5acfa3 8f55826 b5acfa3 389462f b5acfa3 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 | ---
base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct
library_name: peft
license: apache-2.0
pipeline_tag: audio-text-to-text
tags:
- qwen3_omni_moe
- audio
- audio-language-model
- audio-text-alignment
- rubric-based-evaluation
- judge-model
- lora
---
# FineAudio-Judge
> A Fine-Grained Evaluation Framework for Complex Text-to-Audio Generation
[Code](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge) |
[Benchmark](https://huggingface.co/datasets/Zixchen/FineAudio-Judge-Benchmark)
FineAudio-Judge is a fine-grained evaluation framework for assessing
whether audio matches text descriptions or generation instructions
for complex audio scenes.
Given an audio–text pair, the framework first structurally decomposes
the text into events, attributes, and inter-event relations, then
converts them into atomic binary rubric questions. The audio judge
answers each question with yes or no based on the audio, assessing
whether the audio matches the corresponding aspect of the text
description. The proportion of yes answers forms the overall
audio–text consistency score.
To train the audio judge, we construct a multi-route corpus covering
real, composed, and generated audio. Rubric-guided prompt/audio
perturbations and cross-sample mismatches provide fine-grained
supervision across varying levels of audio–text consistency.
This repository releases the resulting LoRA adapter, obtained by
supervised fine-tuning of
[Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct).
The adapter needs to be loaded together with the base model.
Rubric decomposition uses a separately configured language-model
backend in the
[code repository](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge),
while the released adapter performs audio-based rubric judgments.
## Files
- `adapter_config.json`: PEFT LoRA configuration.
- `adapter_model.safetensors`: FineAudio-Judge LoRA adapter weights.
## Training
- **Base**: [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct)
- **Corpus**: 12,508 constructed audio–prompt pairs, with 278,199 rubric-level question–answer examples used for training
- **Stage**: LoRA supervised fine-tuning with a 1-epoch training schedule
- 8 × NVIDIA A800 80GB GPUs
- per-device batch size 1, gradient accumulation 8
- learning rate 5e-5
- LoRA rank 64, alpha 64, dropout 0.05
## Usage
The evaluation scripts are provided in the
[FineAudio-Judge code repository](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge).
They use ms-swift's vLLM inference engine to load
Qwen3-Omni-30B-A3B-Instruct together with the released LoRA adapter.
Clone the repository and enter the FineAudio-Judge directory:
```bash
git clone https://github.com/QwenAudio/FunResearch.git
cd FunResearch/FineAudio-Judge
```
Follow the setup instructions in the code repository's README to install
the project and configure the inference environment. Run the evaluation
commands below from the `FunResearch/FineAudio-Judge/` directory.
Replace `/path/to/benchmark` with the downloaded benchmark repository
root and `/path/to/adapter` with the directory containing the released
adapter files.
### Rubric-QA evaluation
Each audio–rubric question pair is evaluated independently.
```bash
python -m judge.cli.reproduce_rubric_qa \
--dataset-root /path/to/benchmark \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
--adapter /path/to/adapter \
--output outputs/rubric_qa.jsonl
```
### Human-preference evaluation
Rubrics are evaluated jointly for each candidate audio, and the proportion
of yes predictions forms its prompt–audio consistency score.
```bash
python -m judge.cli.reproduce_preference \
--dataset-root /path/to/benchmark \
--model Qwen/Qwen3-Omni-30B-A3B-Instruct \
--adapter /path/to/adapter \
--output outputs/preference.jsonl
```
## License
The FineAudio-Judge LoRA adapter is released under the Apache License 2.0.
See [LICENSE](LICENSE) for details.
The adapter is fine-tuned from
[Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct),
which is also released under the
[Apache License 2.0](https://github.com/QwenLM/Qwen3-Omni/blob/main/LICENSE).
The base model weights are not included in this repository.
Base model copyright: Copyright 2025 Alibaba Cloud.
|