ORB planner

Code · Live demo (Space) · Project page · Training data

A 5.10M-parameter decoder-only transformer that turns a one-line caption, or plain words like "rain on a tin roof with distant thunder" (since v8), into a compact, typed Spec. It does not generate audio by itself: the Spec is compiled by a deterministic DSP renderer in the ORB repository.

caption ──► this model ──► Spec ──► DSP render ──► 48 kHz WAV

Results

Metric Value
Held-out grammar captions (500) 500/500 (100%), every field 100%
Held-out grammar captions (2,000) 2000/2000 (100%)
Plain-text prompts, our set (209) 58.4% fully right (v7 + keyword rules: 45.0%)
Plain-text prompts, blind set (40) 70.0% fully right (v7 + keyword rules: 77.5%)
Decode failures 0
Parameters 5,096,192
Context 1024
Vocab 395 (byte-level + structural; 47 roles, 13 textures, NOBED)

A plain-text prompt passes only if roles, extra roles, texture, mood, intensity and duration are all right. The blind set was written by someone else; the gap to our own set shows that the free-text skill generalises less than our set suggests.

Usage

pip install "orb-audio[train,hub] @ git+https://github.com/stdevMac/orb-audio"
from huggingface_hub import hf_hub_download
from orb.api import load_planner, generate, save_wav

path = hf_hub_download("marcosmaceo/orb-planner", "planner.safetensors")
hf_hub_download("marcosmaceo/orb-planner", "config.json")  # read from the same folder
tok, planner = load_planner(path)
out = generate(
    "6.00-second bright cue, wind bed, with whoosh @1.00 0.90s -6dB L2 #2a",
    planner=(tok, planner),
)
save_wav(out["audio"], "whoosh.wav")
print(out["spec"])

Or without this package, using the config and safetensors directly:

import json, torch
from safetensors.torch import load_file
from orb.model.planner import Planner

cfg = json.load(open("config.json"))
model = Planner(cfg["vocab_size"], **cfg["model"])
model.load_state_dict(load_file("planner.safetensors"))
model.eval()

Decoding is constrained by a grammar mask, so every output is a valid Spec (t + dur <= duration, categorical ranges, at most 8 events).

The Spec

{
  "duration": 6.0,
  "texture": "wind",
  "mood": "bright",
  "events": [
    {"role": "whoosh", "t": 1.0, "dur": 0.9, "gain_db": -6.0, "pan": -0.5, "seed": 42}
  ]
}
  • 47 roles: the 10 core ones (whoosh, impact, ui_pop, riser, swell, click, bell, sub_drop, glitch, noise_sweep) plus 37 physically modelled natural, foley and everyday sounds (thunder, door_slam, footsteps, bird_chirp, wave_crash, glass_break, phone_ring, siren, …)
  • 13 textures: room_cold, wind, city_hum, vinyl, drone, rain, fire, sci_fi, ocean, stream, forest, night, traffic, or none
  • 10 moods: calm, warm, playful, hopeful, bright, tense, dark, eerie, neutral, intense

For agents

The Spec is plain JSON, so an agent can read, diff and edit a sound like any other config. The repository ships an MCP server (orb-mcp, install with the mcp extra) that exposes planning and rendering as tools; see docs/agents.md.

Training

Trained from scratch on synthetic data only: v8 is warm-started from v7 and trained for 40k steps (batch 32, AdamW, cosine schedule, one H100) on 150k procedurally generated grammar (caption, Spec) pairs from orb.grammar.sample_spec plus 150k synthetic plain-text (prompt, Spec) pairs from orb.freetext. No pretrained weights, no external audio, no human-labelled data. The published orb-planner-data dataset holds exactly these 300k training rows, plus the held-out sets.

Intended use

Turning structured captions into an editable sound description for the ORB renderer. Grammar captions are read exactly. Plain words are read on a best-effort basis, and only into the roles and textures above.

Limitations

  • For a noun it has no sound for, the planner may invent the nearest role. Counts ("three knocks") and long multi-clause prompts are weak. Write a grammar caption or pass a Spec dict when exactness matters.
  • v1 is SFX and texture only; music is not modelled.

License

Apache-2.0. See the repository for the renderer, examples and the rest of the engine.

Downloads last month
-
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Dataset used to train marcosmaceo/orb-planner

Space using marcosmaceo/orb-planner 1

Evaluation results

  • Held-out exact Spec match (%) on ORB scores (500 held-out captions)
    self-reported
    100.000