ORB detail model β€” research only

Code Β· Planner (Apache-2.0) Β· Project page

Non-commercial. These weights were trained on ESC-50 (Piczak, 2015), which is CC BY-NC 3.0, so this model is released under CC BY-NC-4.0. It is not part of the Apache-2.0 ORB engine and must not be used commercially until it is retrained on permissively licensed audio.

Detail v6: a 9.06M-parameter (9,064,387) complex-STFT U-Net that predicts a high-frequency residual on top of an ORB DSP render.

DSP render ──► this model ──► residual (above 2 kHz) ──► refined WAV

What it does, honestly

It is a subtle texture stage, not a restoration model. It passes ORB's no-noise gate (scripts/eval_detail_artifacts.py): it adds no tonal lines, no energy above 10 kHz and almost nothing in quiet frames. On the ESC-50 degradation harness it reaches only a +0.84% MR-STFT improvement, against a 10% restoration target, which it does not meet.

no-noise gate (400 eval clips) v6 limit
tonal prominence 2.20 dB < 6.0
periodicity, output / residual 0.006 / 0.252 ≀ 0.2 / ≀ 0.3
eval-pairs prominence, max / median 3.71 / 0.86 dB < 6.0
energy ratio > 10 kHz / > 12 kHz 1.000Γ— / 1.000Γ— ≀ 1.25
quiet-frame rise +0.063 dB ≀ 0.5
MR-STFT improvement +0.84% β‰₯ 0
onset F1 (identity 0.749) 0.749 β‰₯ identity βˆ’ 0.02
gate PASS

The previous weights (v5) reached 1.4% MR-STFT but failed this gate: their ConvTranspose2d upsampling left checkerboard tonal stripes around 10–13 kHz plus a ~4 kHz line. v6 is more conservative and cleaner.

The DSP path is the shippable engine. This model is included for research and for the texture option.

Architecture

  • Complex-STFT U-Net, base 64, 4 levels, GroupNorm, SiLU, FiLM conditioning, bilinear resize-conv upsampling.
  • Input: (2, 513, T) real/imag STFT; no noise channel.
  • Output: a smoothed mask residual (kernel 9Γ—3) restricted to bins above 2 kHz, tapered off between 8 and 9.5 kHz, with an energy gate and a quiet-frame gate (30 dB, 10 dB ramp).
  • Losses: band-energy 2.0, reconstruction 0.05, envelope 0.2, overshoot 1.0, idempotence 5.0; no adversarial loss.
  • Trained on 256-frame crops; inference uses 256-frame windows with 50% overlap and a Hann crossfade.

Usage

pip install "orb-audio[train,hub] @ git+https://github.com/stdevMac/orb-audio"
from huggingface_hub import hf_hub_download
from orb.api import load_detail, render_spec, save_wav

path = hf_hub_download("marcosmaceo/orb-detail", "detail.safetensors")
hf_hub_download("marcosmaceo/orb-detail", "config.json")  # read from the same folder
detail = load_detail(path)

spec = {
    "duration": 6.0, "texture": "rain", "mood": "calm",
    "events": [{"role": "impact", "t": 1.0, "dur": 0.9, "gain_db": -6.0, "pan": 0.0, "seed": 7}],
}
audio = render_spec(spec, detail=detail, strength=1.0)
save_wav(audio, "cue.wav")

strength scales the residual from 0 (DSP only) to 1 (full).

Training data

ESC-50 folds 1–4 (1599 pairs), degraded to a 40-band mel envelope plus a 2–6 kHz lowpass. Evaluation on fold 5 (400 clips). ESC-50 is not redistributed here. Batch 8; these weights are step 1250 of a 1500-step run on one NVIDIA H100 80GB (about 2 minutes).

Limitations

  • Does not meet its own restoration gate (10% MR-STFT).
  • Trained on a synthetic degradation, not on real recording pairs.
  • Non-commercial only.

Citation

If you use this model, please also cite ESC-50:

@inproceedings{piczak2015esc,
  title     = {ESC: Dataset for Environmental Sound Classification},
  author    = {Piczak, Karol J.},
  booktitle = {Proceedings of the 23rd ACM International Conference on Multimedia},
  pages     = {1015--1018},
  year      = {2015}
}
Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support