---
base_model: Qwen/Qwen3-Omni-30B-A3B-Instruct
library_name: peft
license: apache-2.0
pipeline_tag: audio-text-to-text
tags:
  - qwen3_omni_moe
  - audio
  - audio-language-model
  - audio-text-alignment
  - rubric-based-evaluation
  - judge-model
  - lora
---

# FineAudio-Judge

> A Fine-Grained Evaluation Framework for Complex Text-to-Audio Generation

[Code](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge) |
[Benchmark](https://huggingface.co/datasets/Zixchen/FineAudio-Judge-Benchmark)

FineAudio-Judge is a fine-grained evaluation framework for assessing
whether audio matches text descriptions or generation instructions
for complex audio scenes.

Given an audio–text pair, the framework first structurally decomposes
the text into events, attributes, and inter-event relations, then
converts them into atomic binary rubric questions. The audio judge
answers each question with yes or no based on the audio, assessing
whether the audio matches the corresponding aspect of the text
description. The proportion of yes answers forms the overall
audio–text consistency score.

To train the audio judge, we construct a multi-route corpus covering
real, composed, and generated audio. Rubric-guided prompt/audio
perturbations and cross-sample mismatches provide fine-grained
supervision across varying levels of audio–text consistency.

This repository releases the resulting LoRA adapter, obtained by
supervised fine-tuning of
[Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct).
The adapter needs to be loaded together with the base model.
Rubric decomposition uses a separately configured language-model
backend in the
[code repository](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge),
while the released adapter performs audio-based rubric judgments.

## Files

- `adapter_config.json`: PEFT LoRA configuration.
- `adapter_model.safetensors`: FineAudio-Judge LoRA adapter weights.

## Training

- **Base**: [Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct)
- **Corpus**: 12,508 constructed audio–prompt pairs, with 278,199 rubric-level question–answer examples used for training
- **Stage**: LoRA supervised fine-tuning with a 1-epoch training schedule
  - 8 × NVIDIA A800 80GB GPUs
  - per-device batch size 1, gradient accumulation 8
  - learning rate 5e-5
  - LoRA rank 64, alpha 64, dropout 0.05

## Usage

The evaluation scripts are provided in the
[FineAudio-Judge code repository](https://github.com/QwenAudio/FunResearch/tree/main/FineAudio-Judge).
They use ms-swift's vLLM inference engine to load
Qwen3-Omni-30B-A3B-Instruct together with the released LoRA adapter.

Clone the repository and enter the FineAudio-Judge directory:

```bash
git clone https://github.com/QwenAudio/FunResearch.git
cd FunResearch/FineAudio-Judge
```

Follow the setup instructions in the code repository's README to install
the project and configure the inference environment. Run the evaluation
commands below from the `FunResearch/FineAudio-Judge/` directory.

Replace `/path/to/benchmark` with the downloaded benchmark repository
root and `/path/to/adapter` with the directory containing the released
adapter files.

### Rubric-QA evaluation

Each audio–rubric question pair is evaluated independently.

```bash
python -m judge.cli.reproduce_rubric_qa \
  --dataset-root /path/to/benchmark \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --adapter /path/to/adapter \
  --output outputs/rubric_qa.jsonl
```

### Human-preference evaluation

Rubrics are evaluated jointly for each candidate audio, and the proportion
of yes predictions forms its prompt–audio consistency score.

```bash
python -m judge.cli.reproduce_preference \
  --dataset-root /path/to/benchmark \
  --model Qwen/Qwen3-Omni-30B-A3B-Instruct \
  --adapter /path/to/adapter \
  --output outputs/preference.jsonl
```

## License

The FineAudio-Judge LoRA adapter is released under the Apache License 2.0.
See [LICENSE](LICENSE) for details.

The adapter is fine-tuned from
[Qwen3-Omni-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Omni-30B-A3B-Instruct),
which is also released under the
[Apache License 2.0](https://github.com/QwenLM/Qwen3-Omni/blob/main/LICENSE).
The base model weights are not included in this repository.

Base model copyright: Copyright 2025 Alibaba Cloud.