GeWu-Lab commited on
Commit
634aadc
·
verified ·
1 Parent(s): 835cf81

Present final SteerSound weights and credit pretrained components

Browse files
Files changed (3) hide show
  1. README.md +58 -22
  2. config.json +0 -6
  3. manifest.json +4 -9
README.md CHANGED
@@ -13,9 +13,24 @@ pipeline_tag: audio-to-audio
13
 
14
  # SteerSound
15
 
16
- Final STEER-Recovery checkpoint for the base audio editor on Complex audio editing. Selected using validation only at recovery depth k=3, epoch 0.5. Checkpoint selection used the highest six-task validation score; test data did not select this checkpoint. Frozen audio VAE, T5 encoder/tokenizer and HiFi-GAN vocoder are included for inference. This repository uses the custom SteerSound architecture; it is not directly loadable with Transformers AutoModel.
17
 
18
- ## Install and run
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
19
 
20
  ```bash
21
  git clone https://github.com/GeWu-Lab/SteerSound.git
@@ -26,33 +41,56 @@ pip install --upgrade pip
26
  pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
27
  pip install -r requirements.txt
28
  pip install -e . --no-deps
29
- python inference/infer.py --model GeWu-Lab/SteerSound --instruction "Remove the sound of a dog barking." --input input.wav --output edited.wav
30
  ```
31
 
 
 
32
  ```python
33
  import soundfile as sf
34
  from steersound import SteerSoundPipeline
 
35
  pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")
36
- wave = pipe("Generate the sound of rain.")
37
- sf.write("rain.wav", wave, pipe.sample_rate, subtype="PCM_16")
 
 
 
 
 
 
38
  ```
39
 
40
- Provide plain English instructions. Task type and pitch/volume amount are parsed internally; model-control tokens are not user inputs. Editing requires input audio; generation does not. Audio is converted to mono 16 kHz and truncated/right-padded to 10 seconds. Output is 16 kHz mono, 10 seconds. Inference uses BF16, native SDPA, 50 Euler steps and same-reference classifier-free guidance (Add 8.5; all other tasks 3.5). The empty CFG branch remains empty text. Seeded VAE sampling and diffusion noise are keyed by sample ID for evaluation.
41
 
42
- | Task | Plain instruction | Numeric conditioning |
43
- |---|---|---|
44
- | Generation | `Generate the sound of rain.` | None |
45
- | Addition | `Add the sound of a dog barking.` | None |
46
- | Removal | `Remove the sound of a dog barking.` | None |
47
- | Replacement | `Replace the sound of a dog barking by the sound of a cat meowing.` | None |
48
- | Volume | `Increase the volume of the sound of a dog barking by 3.0 dB.` | +3 dB |
49
- | Pitch | `Reduce the pitch of the sound of a dog barking by 2.0 semitones.` | −2 semitones |
 
 
 
 
 
 
 
 
 
50
 
51
- Use `Increase` or `Reduce` and an explicit nonnegative magnitude for pitch/volume. Volume supports −5 to +5 dB; pitch supports −4 to +4 semitones in steps of 0.5. Native dataset instructions omit units; units in user input are converted internally to the native one-decimal, unit-free text template. For temporal editing, place the region before the amount, e.g. `Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB.` Use one operation per instruction. This deterministic parser supports the documented English templates; it does not use another language model or guess missing amounts. Unsupported or ambiguous instructions raise an error. Time intervals are conveyed through text to the editor, rather than hard masks applied to the output.
52
 
53
- ## Estimated inference memory
54
 
55
- For one 10-second mono waveform, batch size 1, BF16 and all four components resident on the GPU:
 
 
 
 
 
 
56
 
57
  | Component | Parameters | BF16 weight storage (GiB) |
58
  |---|---:|---:|
@@ -62,12 +100,10 @@ For one 10-second mono waveform, batch size 1, BF16 and all four components resi
62
  | HiFi-GAN vocoder | 55,264,897 | 0.103 |
63
  | Total | 1,347,580,003 | 2.510 |
64
 
65
- These are calculated tensor sizes, not GPU measurements. A planning estimate is **about 4–8 GiB VRAM**, including activations, convolution workspaces, CUDA context and allocator overhead; **8 GiB or more is recommended**. This is not a verified minimum or a guarantee. FP32 weights alone occupy 5.020 GiB. Text is at most 128 tokens; the audio latent is 16×125×8 and has 252 packed tokens. Euler/CFG uses 50 steps and conditional/null forwards run sequentially, so activations from 50 steps are not all retained. Inference has no optimizer or training gradients and uses no CPU offload. Batch size, dtype and GPU/library versions can change the peak. Metric models consume additional memory and are excluded from this estimate.
66
-
67
- Training memory is substantially higher: all 673M DiT parameters are trained; VAE, text encoder and vocoder stay frozen. Adam states, gradients, activations and communication buffers are additional. The training entry points use ZeRO-2 without CPU offload. Match the desired global batch through GPU count, microbatch and accumulation; no training-memory minimum has been measured.
68
 
69
  ## Files and data
70
 
71
- `dit/model.safetensors`, `vae/model.safetensors`, `text_encoder/`, `vocoder/` and `config.json` contain the inference components. Optimizer state and private training paths are excluded. The DiT and VAE are stored in BF16; text/vocoder checkpoints are stored in FP32 and cast to BF16 by the default runtime. [Datasets](https://huggingface.co/datasets/GeWu-Lab/SteerSound) use plain instructions. Only the training split provides paired target audio; all validation/test splits, including Complex, contain input audio only. Remaining input WAV bytes and numeric metadata are preserved. [Training and evaluation code](https://github.com/GeWu-Lab/SteerSound) documents STEER-Emphasis, STEER-Recovery and the metric protocols.
72
 
73
- CPU strict loading and component checks are used during release preparation. This host has no NVIDIA GPU; CUDA execution, inference speed and peak VRAM have not been measured. The installation target above is a GPU environment specification, not a report of a CUDA benchmark.
 
13
 
14
  # SteerSound
15
 
16
+ SteerSound is an instruction-guided audio editing model supporting sound generation, addition, removal, replacement, pitch modification and volume modification. This repository provides the final SteerSound weights and the components needed for inference.
17
 
18
+ [Code and training instructions](https://github.com/GeWu-Lab/SteerSound) · [Dataset](https://huggingface.co/datasets/GeWu-Lab/SteerSound)
19
+
20
+ ## Model components
21
+
22
+ The audio VAE and DiT editor are trained by the SteerSound team. The text encoder/tokenizer and vocoder use pretrained weights from the sources below.
23
+
24
+ | Component | Source | Files in this repository |
25
+ |---|---|---|
26
+ | Audio VAE (encoder and decoder) | Trained by us | `vae/` |
27
+ | DiT audio editor | Trained by us | `dit/` |
28
+ | T5 text encoder and tokenizer | [google-t5/t5-base](https://huggingface.co/google-t5/t5-base), using its encoder | `text_encoder/` |
29
+ | HiFi-GAN vocoder | [cvssp/audioldm2 — vocoder](https://huggingface.co/cvssp/audioldm2/tree/main/vocoder) | `vocoder/` |
30
+
31
+ ## Quick start
32
+
33
+ Use Linux, Python 3.10 or 3.11, and an NVIDIA GPU. Install a CUDA-enabled PyTorch build and the inference package:
34
 
35
  ```bash
36
  git clone https://github.com/GeWu-Lab/SteerSound.git
 
41
  pip install torch==2.7.1 torchaudio==2.7.1 torchvision==0.22.1 --index-url https://download.pytorch.org/whl/cu126
42
  pip install -r requirements.txt
43
  pip install -e . --no-deps
 
44
  ```
45
 
46
+ For private model access, run `huggingface-cli login` with an account that has access. Access to the GitHub repository is also required while the code is private.
47
+
48
  ```python
49
  import soundfile as sf
50
  from steersound import SteerSoundPipeline
51
+
52
  pipe = SteerSoundPipeline.from_pretrained("GeWu-Lab/SteerSound", device="cuda")
53
+
54
+ # Edit an existing recording.
55
+ audio = pipe("Remove the sound of a dog barking.", audio="input.wav")
56
+ sf.write("edited.wav", audio, pipe.sample_rate, subtype="PCM_16")
57
+
58
+ # Generate audio without an input recording.
59
+ audio = pipe("Generate the sound of rain.")
60
+ sf.write("rain.wav", audio, pipe.sample_rate, subtype="PCM_16")
61
  ```
62
 
63
+ Command-line inference:
64
 
65
+ ```bash
66
+ python inference/infer.py \
67
+ --model GeWu-Lab/SteerSound \
68
+ --instruction "Remove the sound of a dog barking." \
69
+ --input input.wav --output edited.wav
70
+ ```
71
+
72
+ ## Instruction examples
73
+
74
+ | Task | Instruction |
75
+ |---|---|
76
+ | Generation | `Generate the sound of rain.` |
77
+ | Addition | `Add the sound of a dog barking.` |
78
+ | Removal | `Remove the sound of a dog barking.` |
79
+ | Replacement | `Replace the sound of a dog barking by the sound of a cat meowing.` |
80
+ | Volume modification | `Increase the volume of the sound of a dog barking by 3.0 dB.` |
81
+ | Pitch modification | `Reduce the pitch of the sound of a dog barking by 2.0 semitones.` |
82
 
83
+ Supply a plain English instruction. The inference code identifies the task and change amount and adds model-control tokens internally. Editing tasks require input audio; generation does not.
84
 
85
+ For pitch and volume, use `Increase` or `Reduce` with an explicit magnitude. Volume changes support ±5 dB; pitch changes support ±4 semitones in increments of 0.5. A temporal instruction can specify a region, for example: `Increase the volume of the sound of a dog barking from 2.0 to 4.0 seconds by 3.0 dB.` Temporal regions are interpreted by the model from the instruction. Unsupported or ambiguous instructions raise an error.
86
 
87
+ ## Audio format and inference settings
88
+
89
+ Input files are downmixed to mono, resampled to 16 kHz, and truncated or right-padded to 10 seconds. Output is a 10-second mono waveform at 16 kHz. The default runtime uses BF16, PyTorch SDPA and 50 Euler steps, with guidance scale 8.5 for Addition and 3.5 for the other tasks. Use `--seed` with the CLI or `seed=...` with the Python API for repeatable sampling.
90
+
91
+ ## Estimated GPU memory
92
+
93
+ For one 10-second sample at batch size 1 with all components on the GPU:
94
 
95
  | Component | Parameters | BF16 weight storage (GiB) |
96
  |---|---:|---:|
 
100
  | HiFi-GAN vocoder | 55,264,897 | 0.103 |
101
  | Total | 1,347,580,003 | 2.510 |
102
 
103
+ Estimated total inference memory is **4–8 GiB VRAM**; **8 GiB or more is recommended**. This is an estimate, not a CUDA measurement. The total includes weight storage, activations, workspaces and runtime overhead, and can vary with hardware and library versions. Inference uses no CPU offload. See [MEMORY.md](https://github.com/GeWu-Lab/SteerSound/blob/main/MEMORY.md) for details.
 
 
104
 
105
  ## Files and data
106
 
107
+ The DiT and VAE weights are stored in BF16. The T5 encoder and vocoder files are stored in FP32 and cast to the runtime dtype when loaded. Architecture and audio settings are in `config.json`.
108
 
109
+ The [SteerSound dataset](https://huggingface.co/datasets/GeWu-Lab/SteerSound) contains plain instructions and input audio. Only the training split provides paired target audio; all validation/test splits, including Complex, provide input audio only. Evaluation protocols and training code are documented in the [code repository](https://github.com/GeWu-Lab/SteerSound).
config.json CHANGED
@@ -47,12 +47,6 @@
47
  "num_inference_steps": 50,
48
  "max_text_length": 128,
49
  "default_dtype": "bfloat16",
50
- "checkpoint_selection": {
51
- "dataset": "complex",
52
- "split": "val",
53
- "recovery_k": 3,
54
- "epoch": 0.5
55
- },
56
  "task_ids": {
57
  "add": 0,
58
  "remove": 1,
 
47
  "num_inference_steps": 50,
48
  "max_text_length": 128,
49
  "default_dtype": "bfloat16",
 
 
 
 
 
 
50
  "task_ids": {
51
  "add": 0,
52
  "remove": 1,
manifest.json CHANGED
@@ -1,18 +1,13 @@
1
  {
2
  "format_version": 1,
3
- "selection": {
4
- "split": "val",
5
- "k": 3,
6
- "epoch": 0.5
7
- },
8
  "files": {
9
  "config.json": {
10
- "bytes": 1349,
11
- "sha256": "f2051f654d6ef8cd43d0a879ee2474c57eba574d6710baf0f831ef417b5d1360"
12
  },
13
  "README.md": {
14
- "bytes": 5725,
15
- "sha256": "dcac99e34b7da6ba221965ce7cdfb0027de2c93d30271a94d4421c1b202185bb"
16
  },
17
  "dit/model.safetensors": {
18
  "bytes": 1346719328,
 
1
  {
2
  "format_version": 1,
 
 
 
 
 
3
  "files": {
4
  "config.json": {
5
+ "bytes": 1232,
6
+ "sha256": "1851bd54305514445591afe73dec20f7142d93892ae24d0fc62dc2855f4d0009"
7
  },
8
  "README.md": {
9
+ "bytes": 5283,
10
+ "sha256": "1a0488a3dfabd191e0d46d6a591f8c6302a61b10a0e1d59f5b47c4715758052f"
11
  },
12
  "dit/model.safetensors": {
13
  "bytes": 1346719328,