--- language: - en tags: - audio - audio-editing - flow-matching - steersound - steer library_name: pytorch pipeline_tag: audio-to-audio --- [![Project Page](https://img.shields.io/badge/Project-Page-2563EB?style=flat-square&logo=googlechrome&logoColor=white)](https://gewu-lab.github.io/SteerSound/) [![GitHub Code](https://img.shields.io/badge/GitHub-Code-181717?style=flat-square&logo=github&logoColor=white)](https://github.com/GeWu-Lab/SteerSound) [![Hugging Face Weights](https://img.shields.io/badge/Hugging_Face-Weights-FFD21E?style=flat-square&logo=huggingface&logoColor=FFD21E)](https://huggingface.co/GeWu-Lab/SteerSound) [![Hugging Face Dataset](https://img.shields.io/badge/Hugging_Face-Dataset-FF9D00?style=flat-square&logo=huggingface&logoColor=FFD21E)](https://huggingface.co/datasets/GeWu-Lab/SteerSound) # SteerSound **Paper:** Steer Unified Audio Editing Toward Reliable Source Selection SteerSound is an instruction-guided audio editing model supporting sound generation, addition, removal, replacement, pitch modification and volume modification. This repository provides the final SteerSound weights and the components needed for inference. ## Model components The audio VAE and DiT editor are trained by the SteerSound team. The text encoder/tokenizer and vocoder use pretrained weights from the sources below. | Component | Source | Files in this repository | |---|---|---| | Audio VAE (encoder and decoder) | Trained by us | `vae/` | | DiT audio editor | Trained by us | `dit/` | | T5 text encoder and tokenizer | [google-t5/t5-base](https://huggingface.co/google-t5/t5-base), using its encoder | `text_encoder/` | | HiFi-GAN vocoder | [cvssp/audioldm2 — vocoder](https://huggingface.co/cvssp/audioldm2/tree/main/vocoder) | `vocoder/` | ## Quick start Use Linux, Python 3.10 or 3.11, and an NVIDIA GPU. Install a CUDA-enabled PyTorch build and the inference package: ```bash git clone https://github.com/GeWu-Lab/SteerSound.git cd SteerSound python -m venv .venv source .venv/bin/activate pip install --upgrade pip pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126 pip install -r requirements.txt pip install -e . --no-deps ``` ```python import soundfile as sf from steersound import SteerSoundPipeline pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda") # Edit an existing recording. audio = pipe("Remove the sound of a dog barking.", audio="input.wav") sf.write("edited.wav", audio, pipe.sample_rate, subtype="PCM_16") # Generate audio without an input recording. audio = pipe("Generate the sound of rain.") sf.write("rain.wav", audio, pipe.sample_rate, subtype="PCM_16") ``` Command-line inference: ```bash python inference/infer.py \ --model GeWu-Lab/SteerSound \ --instruction "Remove the sound of a dog barking." \ --input input.wav --output edited.wav ``` ## Instruction examples | Task | Instruction | |---|---| | Generation | `Generate the sound of rain.` | | Addition | `Add the sound of a dog barking.` | | Removal | `Remove the sound of a dog barking.` | | Replacement | `Replace the sound of a dog barking by the sound of a cat meowing.` | | Volume modification | `Increase the volume of the sound of a dog barking by 3.0 dB.` | | Pitch modification | `Reduce the pitch of the sound of a dog barking by 2.0 semitones.` | Supply a plain English instruction. The inference code identifies the task and change amount and adds model-control tokens internally. Editing tasks require input audio; generation does not. For pitch and volume, use `Increase` or `Reduce` with an explicit magnitude. Volume changes support ±5 dB; pitch changes support ±4 semitones in increments of 0.5. A temporal instruction can specify a region, for example: `Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB.` Temporal regions are interpreted by the model from the instruction. Unsupported or ambiguous instructions raise an error. ## Audio format and inference settings Input files are downmixed to mono, resampled to 16 kHz, and truncated or right-padded to 10 seconds. Output is a 10-second mono waveform at 16 kHz. The default runtime uses BF16, PyTorch SDPA and 50 Euler steps, with guidance scale 8.5 for Addition and 3.5 for the other tasks. Use `--seed` with the CLI or `seed=...` with the Python API for repeatable sampling. ## Model size and GPU memory | Component | Parameters | |---|---:| | DiT editor | 0.673B | | Audio VAE | 0.509B | | T5 encoder | 0.110B | | HiFi-GAN vocoder | 0.055B | | Total | **1.348B** | **GPU memory for inference: approximately 8 GB** (BF16, batch size 1, 10-second audio). ## Fine-tuning **Pretraining and fine-tuning with standard FM:** follow [Source audio and preparation](https://github.com/GeWu-Lab/SteerSound/blob/main/pretrain_and_standard_FM_finetune/dataset/README.md) to prepare the paper's source audio or your own audio–caption data. This shared guide covers source dataset links, formats, processing steps, and expected outputs. Editing examples are constructed online. Use the [standard training guide](https://github.com/GeWu-Lab/SteerSound/tree/main/pretrain_and_standard_FM_finetune) to pretrain or fine-tune, then evaluate on Complex val/test. **STEER post-training:** start from the released [SteerSound checkpoint](https://huggingface.co/GeWu-Lab/SteerSound) and choose one of these settings: 1. **Complex audio editing data:** prepare the paper's source audio or your own audio–caption data with the same [Source audio and preparation](https://github.com/GeWu-Lab/SteerSound/blob/main/pretrain_and_standard_FM_finetune/dataset/README.md) guide. Apply [Emphasis](https://github.com/GeWu-Lab/SteerSound/tree/main/STEER_post_training/emphasis/complex) or [Recovery](https://github.com/GeWu-Lab/SteerSound/tree/main/STEER_post_training/recovery/complex) with online editing-example construction; validate and evaluate on `complex_val` / `complex_test`. 2. **Selective audio editing data:** download the released `selective_train`, `selective_val`, and `selective_test` from the [🤗 dataset](https://huggingface.co/datasets/GeWu-Lab/SteerSound). Follow [Emphasis / Selective](https://github.com/GeWu-Lab/SteerSound/tree/main/STEER_post_training/emphasis/selective) or [Recovery / Selective](https://github.com/GeWu-Lab/SteerSound/tree/main/STEER_post_training/recovery/selective) to post-train on the supplied editing pairs and evaluate on the corresponding splits. ## Model files The DiT and VAE weights are stored in BF16. The T5 encoder and vocoder files are stored in FP32 and cast to the runtime dtype when loaded. Architecture and audio settings are in `config.json`.