Present final SteerSound weights and credit pretrained components
Browse files- README.md +58 -22
- config.json +0 -6
- manifest.json +4 -9
README.md
CHANGED
|
@@ -13,9 +13,24 @@ pipeline_tag: audio-to-audio
|
|
| 13 |
|
| 14 |
# SteerSound
|
| 15 |
|
| 16 |
-
|
| 17 |
|
| 18 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
|
| 20 |
```bash
|
| 21 |
git clone https://github.com/GeWu-Lab/SteerSound.git
|
|
@@ -26,33 +41,56 @@ pip install --upgrade pip
|
|
| 26 |
pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
|
| 27 |
pip install -r requirements.txt
|
| 28 |
pip install -e . --no-deps
|
| 29 |
-
python inference/infer.py --model GeWu-Lab/SteerSound --instruction "Remove the sound of a dog barking." --input input.wav --output edited.wav
|
| 30 |
```
|
| 31 |
|
|
|
|
|
|
|
| 32 |
```python
|
| 33 |
import soundfile as sf
|
| 34 |
from steersound import SteerSoundPipeline
|
|
|
|
| 35 |
pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")
|
| 36 |
-
|
| 37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 38 |
```
|
| 39 |
|
| 40 |
-
|
| 41 |
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
|
| 51 |
-
|
| 52 |
|
| 53 |
-
|
| 54 |
|
| 55 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 56 |
|
| 57 |
| Component | Parameters | BF16 weight storage (GiB) |
|
| 58 |
|---|---:|---:|
|
|
@@ -62,12 +100,10 @@ For one 10-second mono waveform, batch size 1, BF16 and all four components resi
|
|
| 62 |
| HiFi-GAN vocoder | 55,264,897 | 0.103 |
|
| 63 |
| Total | 1,347,580,003 | 2.510 |
|
| 64 |
|
| 65 |
-
|
| 66 |
-
|
| 67 |
-
Training memory is substantially higher: all 673M DiT parameters are trained; VAE, text encoder and vocoder stay frozen. Adam states, gradients, activations and communication buffers are additional. The training entry points use ZeRO-2 without CPU offload. Match the desired global batch through GPU count, microbatch and accumulation; no training-memory minimum has been measured.
|
| 68 |
|
| 69 |
## Files and data
|
| 70 |
|
| 71 |
-
|
| 72 |
|
| 73 |
-
|
|
|
|
| 13 |
|
| 14 |
# SteerSound
|
| 15 |
|
| 16 |
+
SteerSound is an instruction-guided audio editing model supporting sound generation, addition, removal, replacement, pitch modification and volume modification. This repository provides the final SteerSound weights and the components needed for inference.
|
| 17 |
|
| 18 |
+
[Code and training instructions](https://github.com/GeWu-Lab/SteerSound) · [Dataset](https://huggingface.co/datasets/GeWu-Lab/SteerSound)
|
| 19 |
+
|
| 20 |
+
## Model components
|
| 21 |
+
|
| 22 |
+
The audio VAE and DiT editor are trained by the SteerSound team. The text encoder/tokenizer and vocoder use pretrained weights from the sources below.
|
| 23 |
+
|
| 24 |
+
| Component | Source | Files in this repository |
|
| 25 |
+
|---|---|---|
|
| 26 |
+
| Audio VAE (encoder and decoder) | Trained by us | `vae/` |
|
| 27 |
+
| DiT audio editor | Trained by us | `dit/` |
|
| 28 |
+
| T5 text encoder and tokenizer | [google-t5/t5-base](https://huggingface.co/google-t5/t5-base), using its encoder | `text_encoder/` |
|
| 29 |
+
| HiFi-GAN vocoder | [cvssp/audioldm2 — vocoder](https://huggingface.co/cvssp/audioldm2/tree/main/vocoder) | `vocoder/` |
|
| 30 |
+
|
| 31 |
+
## Quick start
|
| 32 |
+
|
| 33 |
+
Use Linux, Python 3.10 or 3.11, and an NVIDIA GPU. Install a CUDA-enabled PyTorch build and the inference package:
|
| 34 |
|
| 35 |
```bash
|
| 36 |
git clone https://github.com/GeWu-Lab/SteerSound.git
|
|
|
|
| 41 |
pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
|
| 42 |
pip install -r requirements.txt
|
| 43 |
pip install -e . --no-deps
|
|
|
|
| 44 |
```
|
| 45 |
|
| 46 |
+
For private model access, run `huggingface-cli login` with an account that has access. Access to the GitHub repository is also required while the code is private.
|
| 47 |
+
|
| 48 |
```python
|
| 49 |
import soundfile as sf
|
| 50 |
from steersound import SteerSoundPipeline
|
| 51 |
+
|
| 52 |
pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")
|
| 53 |
+
|
| 54 |
+
# Edit an existing recording.
|
| 55 |
+
audio = pipe("Remove the sound of a dog barking.", audio="input.wav")
|
| 56 |
+
sf.write("edited.wav", audio, pipe.sample_rate, subtype="PCM_16")
|
| 57 |
+
|
| 58 |
+
# Generate audio without an input recording.
|
| 59 |
+
audio = pipe("Generate the sound of rain.")
|
| 60 |
+
sf.write("rain.wav", audio, pipe.sample_rate, subtype="PCM_16")
|
| 61 |
```
|
| 62 |
|
| 63 |
+
Command-line inference:
|
| 64 |
|
| 65 |
+
```bash
|
| 66 |
+
python inference/infer.py \
|
| 67 |
+
--model GeWu-Lab/SteerSound \
|
| 68 |
+
--instruction "Remove the sound of a dog barking." \
|
| 69 |
+
--input input.wav --output edited.wav
|
| 70 |
+
```
|
| 71 |
+
|
| 72 |
+
## Instruction examples
|
| 73 |
+
|
| 74 |
+
| Task | Instruction |
|
| 75 |
+
|---|---|
|
| 76 |
+
| Generation | `Generate the sound of rain.` |
|
| 77 |
+
| Addition | `Add the sound of a dog barking.` |
|
| 78 |
+
| Removal | `Remove the sound of a dog barking.` |
|
| 79 |
+
| Replacement | `Replace the sound of a dog barking by the sound of a cat meowing.` |
|
| 80 |
+
| Volume modification | `Increase the volume of the sound of a dog barking by 3.0 dB.` |
|
| 81 |
+
| Pitch modification | `Reduce the pitch of the sound of a dog barking by 2.0 semitones.` |
|
| 82 |
|
| 83 |
+
Supply a plain English instruction. The inference code identifies the task and change amount and adds model-control tokens internally. Editing tasks require input audio; generation does not.
|
| 84 |
|
| 85 |
+
For pitch and volume, use `Increase` or `Reduce` with an explicit magnitude. Volume changes support ±5 dB; pitch changes support ±4 semitones in increments of 0.5. A temporal instruction can specify a region, for example: `Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB.` Temporal regions are interpreted by the model from the instruction. Unsupported or ambiguous instructions raise an error.
|
| 86 |
|
| 87 |
+
## Audio format and inference settings
|
| 88 |
+
|
| 89 |
+
Input files are downmixed to mono, resampled to 16 kHz, and truncated or right-padded to 10 seconds. Output is a 10-second mono waveform at 16 kHz. The default runtime uses BF16, PyTorch SDPA and 50 Euler steps, with guidance scale 8.5 for Addition and 3.5 for the other tasks. Use `--seed` with the CLI or `seed=...` with the Python API for repeatable sampling.
|
| 90 |
+
|
| 91 |
+
## Estimated GPU memory
|
| 92 |
+
|
| 93 |
+
For one 10-second sample at batch size 1 with all components on the GPU:
|
| 94 |
|
| 95 |
| Component | Parameters | BF16 weight storage (GiB) |
|
| 96 |
|---|---:|---:|
|
|
|
|
| 100 |
| HiFi-GAN vocoder | 55,264,897 | 0.103 |
|
| 101 |
| Total | 1,347,580,003 | 2.510 |
|
| 102 |
|
| 103 |
+
Estimated total inference memory is **4–8 GiB VRAM**; **8 GiB or more is recommended**. This is an estimate, not a CUDA measurement. The total includes weight storage, activations, workspaces and runtime overhead, and can vary with hardware and library versions. Inference uses no CPU offload. See [MEMORY.md](https://github.com/GeWu-Lab/SteerSound/blob/main/MEMORY.md) for details.
|
|
|
|
|
|
|
| 104 |
|
| 105 |
## Files and data
|
| 106 |
|
| 107 |
+
The DiT and VAE weights are stored in BF16. The T5 encoder and vocoder files are stored in FP32 and cast to the runtime dtype when loaded. Architecture and audio settings are in `config.json`.
|
| 108 |
|
| 109 |
+
The [SteerSound dataset](https://huggingface.co/datasets/GeWu-Lab/SteerSound) contains plain instructions and input audio. Only the training split provides paired target audio; all validation/test splits, including Complex, provide input audio only. Evaluation protocols and training code are documented in the [code repository](https://github.com/GeWu-Lab/SteerSound).
|
config.json
CHANGED
|
@@ -47,12 +47,6 @@
|
|
| 47 |
"num_inference_steps": 50,
|
| 48 |
"max_text_length": 128,
|
| 49 |
"default_dtype": "bfloat16",
|
| 50 |
-
"checkpoint_selection": {
|
| 51 |
-
"dataset": "complex",
|
| 52 |
-
"split": "val",
|
| 53 |
-
"recovery_k": 3,
|
| 54 |
-
"epoch": 0.5
|
| 55 |
-
},
|
| 56 |
"task_ids": {
|
| 57 |
"add": 0,
|
| 58 |
"remove": 1,
|
|
|
|
| 47 |
"num_inference_steps": 50,
|
| 48 |
"max_text_length": 128,
|
| 49 |
"default_dtype": "bfloat16",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
"task_ids": {
|
| 51 |
"add": 0,
|
| 52 |
"remove": 1,
|
manifest.json
CHANGED
|
@@ -1,18 +1,13 @@
|
|
| 1 |
{
|
| 2 |
"format_version": 1,
|
| 3 |
-
"selection": {
|
| 4 |
-
"split": "val",
|
| 5 |
-
"k": 3,
|
| 6 |
-
"epoch": 0.5
|
| 7 |
-
},
|
| 8 |
"files": {
|
| 9 |
"config.json": {
|
| 10 |
-
"bytes":
|
| 11 |
-
"sha256": "
|
| 12 |
},
|
| 13 |
"README.md": {
|
| 14 |
-
"bytes":
|
| 15 |
-
"sha256": "
|
| 16 |
},
|
| 17 |
"dit/model.safetensors": {
|
| 18 |
"bytes": 1346719328,
|
|
|
|
| 1 |
{
|
| 2 |
"format_version": 1,
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
"files": {
|
| 4 |
"config.json": {
|
| 5 |
+
"bytes": 1232,
|
| 6 |
+
"sha256": "1851bd54305514445591afe73dec20f7142d93892ae24d0fc62dc2855f4d0009"
|
| 7 |
},
|
| 8 |
"README.md": {
|
| 9 |
+
"bytes": 5283,
|
| 10 |
+
"sha256": "1a0488a3dfabd191e0d46d6a591f8c6302a61b10a0e1d59f5b47c4715758052f"
|
| 11 |
},
|
| 12 |
"dit/model.safetensors": {
|
| 13 |
"bytes": 1346719328,
|