Document SteerSound inference and release contents
Browse files
README.md
ADDED
|
@@ -0,0 +1,73 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
language:
|
| 3 |
+
- en
|
| 4 |
+
tags:
|
| 5 |
+
- audio
|
| 6 |
+
- audio-editing
|
| 7 |
+
- flow-matching
|
| 8 |
+
- steersound
|
| 9 |
+
- steer
|
| 10 |
+
library_name: pytorch
|
| 11 |
+
pipeline_tag: audio-to-audio
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# SteerSound
|
| 15 |
+
|
| 16 |
+
Final STEER-Recovery checkpoint for the base audio editor on Complex audio editing. Selected using validation only at recovery depth k=3, epoch 0.5. Checkpoint selection used the highest six-task validation score; test data did not select this checkpoint. Frozen audio VAE, T5 encoder/tokenizer and HiFi-GAN vocoder are included for inference. This repository uses the custom SteerSound architecture; it is not directly loadable with Transformers AutoModel.
|
| 17 |
+
|
| 18 |
+
## Install and run
|
| 19 |
+
|
| 20 |
+
```bash
|
| 21 |
+
git clone https://github.com/GeWu-Lab/SteerSound.git
|
| 22 |
+
cd SteerSound
|
| 23 |
+
python -m venv .venv
|
| 24 |
+
source .venv/bin/activate
|
| 25 |
+
pip install --upgrade pip
|
| 26 |
+
pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
|
| 27 |
+
pip install -r requirements.txt
|
| 28 |
+
pip install -e . --no-deps
|
| 29 |
+
python inference/infer.py --model GeWu-Lab/SteerSound --instruction "Remove the sound of a dog barking." --input input.wav --output edited.wav
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
```python
|
| 33 |
+
import soundfile as sf
|
| 34 |
+
from steersound import SteerSoundPipeline
|
| 35 |
+
pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")
|
| 36 |
+
wave = pipe("Generate the sound of rain.")
|
| 37 |
+
sf.write("rain.wav", wave, pipe.sample_rate, subtype="PCM_16")
|
| 38 |
+
```
|
| 39 |
+
|
| 40 |
+
Provide plain English instructions. Task type and pitch/volume amount are parsed internally; model-control tokens are not user inputs. Editing requires input audio; generation does not. Audio is converted to mono 16 kHz and truncated/right-padded to 10 seconds. Output is 16 kHz mono, 10 seconds. Inference uses BF16, native SDPA, 50 Euler steps and same-reference classifier-free guidance (Add 8.5; all other tasks 3.5). The empty CFG branch remains empty text. Seeded VAE sampling and diffusion noise are keyed by sample ID for evaluation.
|
| 41 |
+
|
| 42 |
+
| Task | Plain instruction | Numeric conditioning |
|
| 43 |
+
|---|---|---|
|
| 44 |
+
| Generation | `Generate the sound of rain.` | None |
|
| 45 |
+
| Addition | `Add the sound of a dog barking.` | None |
|
| 46 |
+
| Removal | `Remove the sound of a dog barking.` | None |
|
| 47 |
+
| Replacement | `Replace the sound of a dog barking by the sound of a cat meowing.` | None |
|
| 48 |
+
| Volume | `Increase the volume of the sound of a dog barking by 3.0 dB.` | +3 dB |
|
| 49 |
+
| Pitch | `Reduce the pitch of the sound of a dog barking by 2.0 semitones.` | −2 semitones |
|
| 50 |
+
|
| 51 |
+
Use `Increase` or `Reduce` and an explicit nonnegative magnitude for pitch/volume. Volume supports −5 to +5 dB; pitch supports −4 to +4 semitones in steps of 0.5. Native dataset instructions omit units; units in user input are converted internally to the native one-decimal, unit-free text template. For temporal editing, place the region before the amount, e.g. `Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB.` Use one operation per instruction. This deterministic parser supports the documented English templates; it does not use another language model or guess missing amounts. Unsupported or ambiguous instructions raise an error. Time intervals are conveyed through text to the editor, rather than hard masks applied to the output.
|
| 52 |
+
|
| 53 |
+
## Estimated inference memory
|
| 54 |
+
|
| 55 |
+
For one 10-second mono waveform, batch size 1, BF16 and all four components resident on the GPU:
|
| 56 |
+
|
| 57 |
+
| Component | Parameters | BF16 weight storage (GiB) |
|
| 58 |
+
|---|---:|---:|
|
| 59 |
+
| DiT editor | 673,349,696 | 1.254 |
|
| 60 |
+
| Audio VAE | 509,336,866 | 0.949 |
|
| 61 |
+
| T5 encoder | 109,628,544 | 0.204 |
|
| 62 |
+
| HiFi-GAN vocoder | 55,264,897 | 0.103 |
|
| 63 |
+
| Total | 1,347,580,003 | 2.510 |
|
| 64 |
+
|
| 65 |
+
These are calculated tensor sizes, not GPU measurements. A planning estimate is **about 4–8 GiB VRAM**, including activations, convolution workspaces, CUDA context and allocator overhead; **8 GiB or more is recommended**. This is not a verified minimum or a guarantee. FP32 weights alone occupy 5.020 GiB. Text is at most 128 tokens; the audio latent is 16×125×8 and has 252 packed tokens. Euler/CFG uses 50 steps and conditional/null forwards run sequentially, so activations from 50 steps are not all retained. Inference has no optimizer or training gradients and uses no CPU offload. Batch size, dtype and GPU/library versions can change the peak. Metric models consume additional memory and are excluded from this estimate.
|
| 66 |
+
|
| 67 |
+
Training memory is substantially higher: all 673M DiT parameters are trained; VAE, text encoder and vocoder stay frozen. Adam states, gradients, activations and communication buffers are additional. The training entry points use ZeRO-2 without CPU offload. Match the desired global batch through GPU count, microbatch and accumulation; no training-memory minimum has been measured.
|
| 68 |
+
|
| 69 |
+
## Files and data
|
| 70 |
+
|
| 71 |
+
`dit/model.safetensors`, `vae/model.safetensors`, `text_encoder/`, `vocoder/` and `config.json` contain the inference components. Optimizer state and private training paths are excluded. The DiT and VAE are stored in BF16; text/vocoder checkpoints are stored in FP32 and cast to BF16 by the default runtime. [Datasets](https://huggingface.co/datasets/GeWu-Lab/SteerSound) use plain instructions and preserve original audio bytes and numeric metadata. [Training and evaluation code](https://github.com/GeWu-Lab/SteerSound) documents STEER-Emphasis, STEER-Recovery and the metric protocols.
|
| 72 |
+
|
| 73 |
+
CPU strict loading and component checks are used during release preparation. This host has no NVIDIA GPU; CUDA execution, inference speed and peak VRAM have not been measured. The installation target above is a GPU environment specification, not a report of a CUDA benchmark.
|