|
Download README.md from audiojs/restore: direct link, hf CLI and curl.
- Browser
- Download file 7.97 kB
-
https://huggingface.co/audiojs/restore/resolve/main/README.md
- Command line
-
hf download hf://audiojs/restore/README.md
-
curl -L -o README.md https://huggingface.co/audiojs/restore/resolve/main/README.md
7.97 kB
| license: openrail | |
| license_link: LICENSE | |
| library_name: onnx | |
| pipeline_tag: audio-to-audio | |
| language: | |
| - en | |
| - fr | |
| - ru | |
| tags: | |
| - audio | |
| - speech-enhancement | |
| - speech-restoration | |
| - bandwidth-extension | |
| - dereverberation | |
| - onnx | |
| # restore, ONNX (float16 weights) | |
| A voice restored to a studio take: speech or singing, English, French or Russian, recorded in a room, in noise, on a | |
| phone or through a codec, back as 48 kHz full band. One network, one pass, in the STFT. Trained by audiojs for | |
| [@audio/neural-restore](https://github.com/audiojs/neural/tree/main/packages/neural-restore), which computes the STFT, | |
| the chunks and the level in JavaScript and runs this file through onnxruntime (Node, or a browser's onnxruntime-web on | |
| WebGPU or wasm); [`audio`](https://github.com/audiojs/audio)'s `restore()` runs it. | |
| | File | Size | SHA-256 | | |
| |---|---|---| | |
| | `restore.fp16.onnx` | 35.9 MB (35,929,213 bytes) | `46eb6ebb05cc50bb6c3b1af87e3c4542bf30cedbcc356e2c68646d3bbf9a65f1` | | |
| Float16 weights, float32 compute: every weight tensor of 1024 values or more stored as float16 and Cast to float32 in | |
| the graph (`scripts/export.py`). The float32 export it is made from is 71,671,387 bytes. | |
| ## Model | |
| Vocos's ConvNeXt stack over STFT frames (dim 512, 8 blocks) with two self-attention layers for a room's tail, reading | |
| each frame's log magnitude and phase and writing a change of log magnitude (unbounded: a band can be taken down 60 dB | |
| or raised from nothing) and a rotation of the input's phase (the identity at init: what is there keeps its phase | |
| unless the net turns it); 17.9M parameters, the STFT outside the graph. | |
| | | name | shape | | | |
| |---|---|---|---| | |
| | input | `spec` | [1, 2, 1025, T] float32 | STFT of one channel at 48 kHz: n_fft 2048, hop 512, periodic Hann, centered with reflect padding, unnormalized; re, im | | |
| | output | `spec_out` | [1, 2, 1025, T] float32 | the restored STFT, the same layout | | |
| T is dynamic; opset 17. The network hears its input at β20 dBFS RMS: @audio/neural-restore brings the whole input to | |
| that level, runs it in 10 s runs overlapping by 1 s, raised-cosine crossfaded, and undoes the gain, as | |
| `scripts/infer.py` does; other rates are resampled to 48 kHz and back. | |
| ```js | |
| import restore from '@audio/neural-restore' | |
| let y = await restore(samples, { sampleRate: 44100 }) // mono Float32Array; the same rate and length back | |
| ``` | |
| ## Training | |
| Trained from scratch: 300,000 steps on an Apple M4 Max GPU (PyTorch 2.14 on MPS, batch 8 Γ 4 s), 2026-10-15 to | |
| 2026-10-22. These are step 290,000's weights: best SI-SDR and mel distance on the validation draw; 270kβ300k are within | |
| each other's noise on every measure. A regressor first (multi-resolution STFT, log-mel, a compressed complex L1 on the | |
| model's grid for the phase, WavLM Base+ features: FINALLY's recipe), then adversarially against HiFi-GAN's | |
| multi-period and UnivNet's multi-resolution discriminators with feature matching. Every loss sees target and estimate | |
| through a low-pass at the target's own measured bandwidth, so 24 kHz corpora train beside full-band ones. The inputs | |
| are made on the fly from clean voices: a room (a measured response or a synthetic one), noise, a cheap microphone, a | |
| compressor, clipping, a band limit, MP3 / AAC / Opus, 8β16-bit words. The code is `scripts/` in @audio/neural-restore. | |
| | Set | Hours | Rate | Licence | Role | | |
| |---|---|---|---|---| | |
| | VCTK 0.92, mic1, speakers p232 and p257 held out | 44 | 48 kHz | CC BY 4.0 | English, full band | | |
| | LibriTTS-R train-clean-100 | 54 | 24 kHz | CC BY 4.0 | English, to 12 kHz | | |
| | SIWIS | 10 | 44.1 kHz | CC BY 4.0 | French, one professional voice | | |
| | Dialogs (langswap) | 17.4 | 44.1 kHz | OpenRAIL (commercial use allowed; its use restrictions flow down to derived models) | Russian, three actors, studio; its manifest is separate so a licence-clean variant is one flag away | | |
| | Supertonic 3 (Supertone, OpenRAIL-M weights), ten voices, public-domain text | ru 12, fr 8, en 4 | 44.1 kHz | the synthesised audio is ours | the only full-band Russian beyond Dialogs; DNSMOS OVRL 3.43 / 3.40 / 3.39 (ru / fr / en) against 3.39 for VCTK's clean test set and 3.20 for Dialogs | | |
| | Pocket TTS (Kyutai, CC BY 4.0), eight voices | fr 6, en 3 | 24 kHz | ours | more voices, to 12 kHz | | |
| | VocalSet (the local subset) | 1.8 | 44.1 kHz | CC BY 4.0 | singing, full band | | |
| | MIT IR Survey, the even-numbered rooms | 135 rooms | 32 kHz | unstated: training only, never redistributed | rooms; the odd-numbered ones are `audio`'s bench rooms and stay out | | |
| | MUSAN noise | 6 | 16 kHz | CC BY 4.0 | noise | | |
| Texts read by the synthetic voices: Project Gutenberg (Austen, Dickens, Melville, Doyle; Hugo, Flaubert, Dumas, Proust; | |
| four Russian titles) and Dialogs' own transcripts. WavLM Base+ (Microsoft, MIT) is a feature loss, not data. A | |
| measured room or noise recorded at 16 or 32 kHz is continued to 24 kHz with synthetic content of the same decay or | |
| colour, so a measured room never low-passes the voice by its own rate. Validation pairs come from VoiceBank's clean | |
| test set (p232, p257), never trained on. | |
| ## Measured | |
| 24 validation pairs: VoiceBank's test speakers through the chain (rooms, noise, microphones, codecs, band limits), | |
| each output at the reference's level, scored through the pair's band limit. The baselines are `audio`'s own ops on the | |
| same inputs (DeepFilterNet3 through `deepfilter()`; `auto('speech')`). | |
| | System | LSD β | SI-SDR β | PESQ β | DNSMOS SIG | BAK | OVRL | band, Hz | | |
| |---|---|---|---|---|---|---|---| | |
| | input | 3.30 | β5.2 | 1.77 | 2.50 | 2.41 | 1.97 | 16 947 | | |
| | `deepfilter()` | 2.85 | β2.1 | 2.13 | 2.93 | 3.75 | 2.59 | 17 104 | | |
| | `deepfilter(0)` | 2.71 | β2.1 | 2.07 | 2.91 | 3.90 | 2.63 | 16 366 | | |
| | `auto('speech')` | 2.91 | β4.0 | 1.88 | 2.91 | 3.29 | 2.45 | 16 265 | | |
| | these weights (step 290k) | 0.79 | +6.0 | 2.40 | 3.21 | 3.91 | 2.89 | 24 000 | | |
| | reference | 0 | | 4.64 | 3.29 | 3.89 | 2.95 | 24 000 | | |
| SI-SDR against a dry reference is a hard measure for this chain (rooms and codec lags put the input itself at β5 dB); | |
| the listen decides. | |
| This file against PyTorch: the float32 export is 108β121 dB from PyTorch on five real takes, one of them 16 s in | |
| chunks; this file 68β74 dB, the same scores to 0.1 dB SI-SDR. @audio/neural-restore's JavaScript pipeline with this | |
| file is 73 dB from `scripts/infer.py` on its fixture in Node, 70 dB on WebGPU and wasm in a browser; 10.5 s restored in | |
| 0.36 s on the M4 Max's WebGPU, 1.3 s on wasm. | |
| ## Limits | |
| - Measured on the 24 English pairs above only. French, Russian, singing, real recordings, lost or invented words | |
| (WER) and Adobe Podcast Enhance are not measured. | |
| - Above an input's band limit the network writes content the input does not hold: plausible, not the original. | |
| - Full-band Russian comes from three Dialogs actors and synthetic voices, French from one professional voice (SIWIS) | |
| and synthetic voices, singing from 1.8 h of VocalSet. | |
| - One channel a run, 48 kHz in and out of the graph. | |
| ## Licence and attribution | |
| OpenRAIL. These weights are trained on Dialogs, whose licence names "any models or other works trained on or derived | |
| from it together with their outputs" as Derivatives and requires that a Derivative be distributed with that licence, | |
| its Section 2 use restrictions included, and its recipients bound to them. [LICENSE](LICENSE) is that licence verbatim | |
| ([langswap/dialogs-ru-emotional-conversations](https://huggingface.co/datasets/langswap/dialogs-ru-emotional-conversations) | |
| at `a74ff44`): whoever uses these weights, or audio they make, agrees to its Section 2. | |
| The MIT IR Survey states no licence: its responses trained the model and are not redistributed, here or elsewhere. | |
| The CC BY 4.0 sets: VCTK (CSTR, University of Edinburgh), LibriTTS-R (Koizumi et al.), SIWIS (Honnet et al.), VocalSet | |
| (Wilkins et al.), MUSAN (Snyder, Chen, Povey), and Pocket TTS's weights (Kyutai). Dialogs by Ilya Shigabeev and Ilya | |
| Latyshev (Langswap); Supertonic 3 by Supertone. | |