Mirror of iflytek/AudioFly from ModelScope. All credit to the original authors. Not an official release from them. License unchanged (apache-2.0).

Model Card for AudioFly

Model Introduction

AudioFly is an audio generation model. It synthesizes sound effects based on textual descriptions. The model can produce high-quality audio at a sampling rate of 44.1 kHz. The generated audio shows strong alignment with the prompt text.

AudioFly adopts the Latent Diffusion Model architecture. The model has 1 billion parameters and is trained on a large and diverse corpus. The training data include open-source datasets, such as AudioSet, AudioCaps, and TUT, as well as proprietary internal data. The model performs well in both single-event and multi-event scenarios. In these cases, the generated audio accurately reflects the described content. AudioFly achieves superior performance compared to previous audio generation models on the AudioCaps dataset.

Evaluation Results

The experimental results are reported on the AudioCaps dataset. For the baseline models, we reused the evaluation results from STABLE AUDIO OPEN. We followed the same evaluation methodology to ensure consistency. The evaluation results are shown as follows.

Model	FD ↓	KL ↓	CLAP ↑
AudioLDM2-48kHz	101.11	2.04	0.37
AudioGen-medium	186.53	1.42	0.45
Stable Audio 1.0	103.66	2.89	0.24
Stable Audio 2.0	110.62	2.70	0.23
Stable Audio Open	78.24	2.14	0.29
AudioFly	40.1	1.35	0.45

Usage

Requirements

We recommend setting up the runtime environment using the provided configuration file by running:

pip install -r requirements.txt 
# make sure to set the PYTHONPATH to include the AudioFly project root
export PYTHONPATH=/path/to/AudioFly:$PYTHONPATH
cd /path/to/AudioFly

Quickstart

import yaml
import torch
from ldm.utils.util import instantiate_from_config


configs = yaml.load(open('./config/config.yaml', "r"), Loader=yaml.FullLoader)
model = instantiate_from_config(configs["model"])

checkpoint = torch.load('./models/ldm/model.ckpt')
model.load_state_dict(checkpoint, strict=False)
model.eval()
model = model.cuda()
text = 'Fierce winds howl through the valley' 
name = 'result'
savedir = './result'
model.generate_sample(
        textlist=[text],
        name=name,
        cfg=3.5,# Guidance scale (controls how strongly generation follows the text prompt）; not recommended to change
        ddim_steps=200,  # Number of denoising steps in the diffusion process; not recommended to change
        outputdir=f"{savedir}")

License

AudioFly is licensed under Apache 2.0.