Tradi-fusion v2

Tradi-fusion v2 is a fine-tuned version of Riffusion that generates mel spectrograms of Irish traditional music. Riffusion is a Stable Diffusion model that draws music as images of spectrograms, and you can turn a generated spectrogram back into about five seconds of audio. I trained the model for my master's thesis at KTH Royal Institute of Technology in 2024.

How to use

The quickest way to get a spectrogram image is the standard Diffusers pipeline.

from diffusers import StableDiffusionPipeline
import torch

pipe = StableDiffusionPipeline.from_pretrained(
    "hdparmar/tradfusion-v2", torch_dtype=torch.float16
).to("cuda")

image = pipe("An Irish traditional tune", num_inference_steps=50).images[0]
image.save("spectrogram.png")

To get audio, use the pipeline in the GitHub repository. It converts the spectrogram back to sound, and it can also interpolate between two prompts or two seeds to make longer clips.

from inference import TradifusionPipeline  # run from src/tradifusion_pipeline

pipeline = TradifusionPipeline.load_checkpoint("hdparmar/tradfusion-v2", device="cuda")
pipeline.enable_attention_slicing()  # fits a 4 GB GPU

audio = pipeline.txt2audio_tradfusion(
    "An Irish traditional tune",
    "An Irish traditional tune with acoustic fiddle lead",
    num_steps=2,
)

Training

Setting Value
Starting checkpoint riffusion-model-v1.ckpt
Training data 9,604 mel spectrograms, 512 x 512 pixels, from more than 1,700 recordings
Caption "An Irish Traditional Tune" for every image
Trained parts U-Net and VAE; the CLIP text encoder stayed frozen
Learning rate 1.0e-4
Batch size 4
Length of training about 122,000 steps (202 epochs), about 4 days on two NVIDIA A6000 GPUs

Each spectrogram covers 5,119 ms of mono audio at 44.1 kHz. The mel scale has 512 bands from 60 Hz to 8,500 Hz, and the short-time Fourier transform uses a Hann window of 100 ms with a step of 10 ms. The training followed Lambda Labs' guide to fine-tuning Stable Diffusion, and the full configuration is in config/spectrogram.yaml in the GitHub repository.

Files

The folders unet, vae, text_encoder, tokenizer, scheduler, feature_extractor and safety_checker hold the model in Diffusers format, and you should load the model from them. The file checkpoint_epoch202.ckpt (14.6 GB) is the original training checkpoint. It is a pickle file, so load it only if you trust the source.

Limitations

  • Each image is about five seconds long, and the model has no sense of the structure of a tune, such as the repeated parts of a reel or a jig. Longer pieces are frames placed side by side, and long interpolations can fall into repeating patterns.
  • The audio sounds metallic, because the conversion from spectrogram to audio has to estimate the phase.
  • Every training image had the same caption, so words such as "banjo" or "harp" have little effect on the output.
  • I evaluated the model by listening to examples and studying failure cases, and I did not run a formal listening test or measure quality with a metric.

License

The model is a fine-tuned version of Riffusion, which uses the CreativeML Open RAIL-M license, so the same license and its use restrictions apply to Tradi-fusion v2.

Citation

@mastersthesis{parmar2024tradifusion,
  author  = {Parmar, Harshdeepsinh},
  title   = {Tradi-fusion Refined: Evaluating and Fine-tuning
             the Riffusion Model for Irish Traditional Music},
  school  = {KTH Royal Institute of Technology},
  address = {Stockholm, Sweden},
  year    = {2024}
}

Please also cite Riffusion.

@article{Forsgren_Martiros_2022,
  author = {Forsgren, Seth* and Martiros, Hayk*},
  title  = {{Riffusion - Stable diffusion for real-time music generation}},
  url    = {https://riffusion.com/about},
  year   = {2022}
}

Acknowledgements

The thesis was supervised by Assoc. Prof. Bob L. T. Sturm and examined by Assoc. Prof. André Holzapfel at KTH. The recordings came from a collection provided by Bob L. T. Sturm.

Downloads last month
68
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for hdparmar/tradfusion-v2

Finetuned
(9)
this model

Dataset used to train hdparmar/tradfusion-v2