Instructions to use hdparmar/tradfusion-v2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use hdparmar/tradfusion-v2 with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("hdparmar/tradfusion-v2", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- Draw Things
- DiffusionBee
Tradi-fusion v2
Tradi-fusion v2 is a fine-tuned version of Riffusion that generates mel spectrograms of Irish traditional music. Riffusion is a Stable Diffusion model that draws music as images of spectrograms, and you can turn a generated spectrogram back into about five seconds of audio. I trained the model for my master's thesis at KTH Royal Institute of Technology in 2024.
- Code and inference pipeline: github.com/hdparmar/Tradifusion
- Listening examples: hdparmar.github.io/Tradifusion
- Training data: hdparmar/irish-traditional-tunes
- Thesis: Tradi-fusion Refined (DiVA)
How to use
The quickest way to get a spectrogram image is the standard Diffusers pipeline.
from diffusers import StableDiffusionPipeline
import torch
pipe = StableDiffusionPipeline.from_pretrained(
"hdparmar/tradfusion-v2", torch_dtype=torch.float16
).to("cuda")
image = pipe("An Irish traditional tune", num_inference_steps=50).images[0]
image.save("spectrogram.png")
To get audio, use the pipeline in the GitHub repository. It converts the spectrogram back to sound, and it can also interpolate between two prompts or two seeds to make longer clips.
from inference import TradifusionPipeline # run from src/tradifusion_pipeline
pipeline = TradifusionPipeline.load_checkpoint("hdparmar/tradfusion-v2", device="cuda")
pipeline.enable_attention_slicing() # fits a 4 GB GPU
audio = pipeline.txt2audio_tradfusion(
"An Irish traditional tune",
"An Irish traditional tune with acoustic fiddle lead",
num_steps=2,
)
Training
| Setting | Value |
|---|---|
| Starting checkpoint | riffusion-model-v1.ckpt |
| Training data | 9,604 mel spectrograms, 512 x 512 pixels, from more than 1,700 recordings |
| Caption | "An Irish Traditional Tune" for every image |
| Trained parts | U-Net and VAE; the CLIP text encoder stayed frozen |
| Learning rate | 1.0e-4 |
| Batch size | 4 |
| Length of training | about 122,000 steps (202 epochs), about 4 days on two NVIDIA A6000 GPUs |
Each spectrogram covers 5,119 ms of mono audio at 44.1 kHz. The mel scale has 512 bands from 60 Hz to 8,500 Hz, and the short-time Fourier transform uses a Hann window of 100 ms with a step of 10 ms. The training followed Lambda Labs' guide to fine-tuning Stable Diffusion, and the full configuration is in config/spectrogram.yaml in the GitHub repository.
Files
The folders unet, vae, text_encoder, tokenizer, scheduler, feature_extractor and safety_checker hold the model in Diffusers format, and you should load the model from them. The file checkpoint_epoch202.ckpt (14.6 GB) is the original training checkpoint. It is a pickle file, so load it only if you trust the source.
Limitations
- Each image is about five seconds long, and the model has no sense of the structure of a tune, such as the repeated parts of a reel or a jig. Longer pieces are frames placed side by side, and long interpolations can fall into repeating patterns.
- The audio sounds metallic, because the conversion from spectrogram to audio has to estimate the phase.
- Every training image had the same caption, so words such as "banjo" or "harp" have little effect on the output.
- I evaluated the model by listening to examples and studying failure cases, and I did not run a formal listening test or measure quality with a metric.
License
The model is a fine-tuned version of Riffusion, which uses the CreativeML Open RAIL-M license, so the same license and its use restrictions apply to Tradi-fusion v2.
Citation
@mastersthesis{parmar2024tradifusion,
author = {Parmar, Harshdeepsinh},
title = {Tradi-fusion Refined: Evaluating and Fine-tuning
the Riffusion Model for Irish Traditional Music},
school = {KTH Royal Institute of Technology},
address = {Stockholm, Sweden},
year = {2024}
}
Please also cite Riffusion.
@article{Forsgren_Martiros_2022,
author = {Forsgren, Seth* and Martiros, Hayk*},
title = {{Riffusion - Stable diffusion for real-time music generation}},
url = {https://riffusion.com/about},
year = {2022}
}
Acknowledgements
The thesis was supervised by Assoc. Prof. Bob L. T. Sturm and examined by Assoc. Prof. André Holzapfel at KTH. The recordings came from a collection provided by Bob L. T. Sturm.
- Downloads last month
- 68
Model tree for hdparmar/tradfusion-v2
Base model
riffusion/riffusion-model-v1