| # HANDOFF — stable-jam resume guide |
|
|
| > This file exists so a fresh session can `cd D:/CODE/stable-jam` and pick up |
| > exactly where the previous one left off — no loss of context. Read this first, |
| > then the linked docs. |
|
|
| **Product:** Jam Buddy (consolidated here as **stable-jam**). "You start playing, |
| it joins in." A call-and-response AI music companion using Stable Audio 3, |
| built for the Stability AI Challenge at Music Hackspace Montreal (Aug 22–23 2026). |
|
|
| --- |
|
|
| ## What this repo is (and is not) |
|
|
| `stable-jam/` is the **source + docs + tests** for the Jam Buddy product. It does |
| NOT contain the heavy vendored model repos — those stay in the previous |
| location and are referenced by env vars (below). Don't expect to `git clone` and |
| get a GPU model; the SA3 weights are a separate download. |
|
|
| **Copied from** `D:/CODE/unstable-drums/` (the previous working dir, still intact |
| if you need to recover anything). This dir was created to give the hackathon a |
| clean, product-accurate home. |
|
|
| ## Product state (what actually works, verified this session) |
|
|
| - **Web app** (`apps/web`): hardware-sampler rack UI. Instrument pads (bass/lead/ |
| rhythm/synth/drums), Genre/Mood/Tempo knobs, a "Your take" section with BOTH a |
| MIDI file input and an audio file input, JOIN IN + PLAY BOTH buttons, status |
| panel (aria-live) showing "Buddy tempo: N BPM", and an audio player. Every |
| response is saved to `generations/` at the repo root. |
| - **SA3 pipeline** (`tools/jam_buddy.py`): the CLI that does the generation. |
| Two input modes: |
| - `--midi take.mid` — detects tempo (exact) + response length from the take, |
| generates a complementary melodic part at that tempo. **Ignores the groove** |
| (SA3 can't parse MIDI notes). |
| - `--wav take.wav` — TRUE audio-to-audio via `init_audio` + `init_noise_level` |
| (default 0.4). The buddy actually HEARS the groove and responds rhythmically. |
| - `--bpm N` / `--model small-music|small-sfx` / `--genre` / `--duration`. |
| - Drums force `small-sfx` (clean isolated hits); everything else `small-music`. |
| - **API route** (`apps/web/app/api/jambuddy/route.ts`): POST /api/jambuddy with |
| `{knobs, bpm?, midi?, audio?, duration?}`, shells to the python, returns the |
| WAV + `X-Jam-Buddy-BPM` header. Writes output to `<repo>/generations/`. |
| - **Prompt builder** (`apps/web/lib/jambuddy/prompt.ts`): pure function building |
| the SA3 prompt from the knobs per the official SA3 prompting guide |
| (TrackType: Instrument, instrument, genre, mood, BPM, studio recording). |
| Negative prompt steers away from a full mix. `MODEL_FOR_INSTRUMENT` maps |
| drums→small-sfx. |
| - **Player** (`apps/web/lib/jambuddy/player.ts`): browser Web Audio. `playTogether` |
| renders the MIDI take as oscillators and plays it with the buddy WAV on the |
| same clock. **Note: this was "banked" — the user reported PLAY BOTH only |
| played the generated part, not the MIDI synth. It's parked, not debugged.** |
| |
| ## Verified this session (proof) |
|
|
| - Web: `pnpm run test` -> 31/31 pass (prompt, player, engine, conversation); |
| `tsc --noEmit` clean; page serves the rack. |
| - CLI: `jam_buddy.py --midi <file>` -> detected 158 BPM, 24.3s response matching |
| the take. Audio-to-audio on a 5s take -> ~10s wall, valid WAV (peak 0.43, |
| rms 0.08). A ~378s take times out on CPU — keep audio takes short. |
| - The generations dir fills with timestamped WAVs. |
|
|
| ## Key environment / paths |
|
|
| SA3 (the engine) now lives IN this repo at `stable-audio-3/`. The venv's |
| editable-install `.pth` was repointed to this location (was `D:/CODE/unstable-drums/...`). |
| The `/api/jambuddy` route auto-resolves it via `join(repoRoot, "stable-audio-3", ...)`. |
|
|
| | Var | Value | |
| |---|---| |
| | `JAM_BUDDY_ROOT` | `D:/CODE/stable-jam` (repo root; route walks up to find `tools/jam_buddy.py`) | |
| | `JAM_BUDDY_PYTHON` | `D:/CODE/stable-jam/stable-audio-3/.venv/Scripts/python.exe` (the SA3 venv) | |
| | SA3 weights | cached in `stable-audio-3/.venv` + HF cache on G:/AI/models/huggingface | |
| | `HF_TOKEN` | in old repo's `.env` (needed for gated SA3 model access) — copy if regenerating weights | |
|
|
| The SA3 venv was extended with `mido` and `librosa` |
| (`uv pip install --python .../stable-audio-3/.venv/Scripts/python.exe mido librosa`). |
| A fresh session must use that venv or recreate it. |
|
|
| ## How to run (fresh session) |
|
|
| ```bash |
| cd D:/CODE/stable-jam |
| # 1. install web deps (node_modules was NOT copied) |
| cd apps/web && pnpm install && cd ../.. |
| |
| # 2. run tests + typecheck |
| cd apps/web && npx vitest run && npx tsc --noEmit |
| |
| # 3. dev server |
| cd apps/web && pnpm dev # -> http://localhost:3000 |
| |
| # 4. python pipeline (SA3 venv is in THIS repo) |
| ./stable-audio-3/.venv/Scripts/python.exe tools/jam_buddy.py --midi take.mid --instrument bass --out out.wav |
| ``` |
|
|
| ## Docs (all in `docs/`) |
|
|
| - `01-vision.md` — Jam Buddy vision (rewritten from the old PatternTalk framing). |
| - `02-architecture.md` — note: may still describe the old service layout; the |
| real arch is: Next.js web + `tools/jam_buddy.py` + SA3 venv. |
| - `05-accessibility.md` — the a11y strategy (still relevant; the product is |
| screenreader-compatible by design). |
| - `06-stable-audio-integration.md` — SA3 setup, audio-to-audio, Vega notes. |
| - `HARNESS.md` — agent harness commands + test contract (may still say |
| "PatternTalk" in places; the real product is Jam Buddy). |
|
|
| ## What's deliberately NOT in this repo (and why) |
|
|
| - `text2midi/` (2.8G vendored model) — not part of the Jam Buddy product; it |
| stays in the previous working dir if needed. |
| - The SA3 weights / `.venv` are inside `stable-audio-3/` here but **gitignored** |
| (large, regenerable, license-gated). See `.gitignore`. |
| - `node_modules/`, `.next/` — regenerable, not committed. |
| - `generations/*.wav` — gitignored (regenerable output). |
| - Source song/GP files + scratch WAVs in the old `tools/` — not product source. |
|
|
| ## Pending / next steps (from the last session) |
|
|
| 1. **PLAY BOTH** — unbank / fix: the MIDI synth part wasn't audible. Likely the |
| object URL or the oscillator gain. That's the "co-play" wow moment. |
| 2. **Metadata sidecar** — add a `<file>.json` next to each `generations/*.wav` |
| with prompt / negative / model / BPM / duration / source. Answers "what did it |
| go off on." |
| 3. **Re-render the docs** `02-architecture.md`, `03-data-model.md`, |
| `04-ux-voice-first.md`, `07-reaper-integration.md` to match Jam Buddy (they |
| still describe the old PatternTalk drum-machine in places). |
| 4. **LoRA / underfit** (style trainer) is a real path but needs a GPU — not shipped. |
| 5. **Demo script** — the judge's-eye review (session bg_161745) recommended a |
| 3-min demo script; see `docs/01-vision.md` "What success looks like". |
| |
| ## The judge's review (session `bg_161745` — hackathon judge design review) |
|
|
| Key finding: **docs and product were telling different stories** (PatternTalk vs |
| Jam Buddy). That's why this repo was consolidated to `stable-jam`. Scoring |
| rubric + recs live in that session. The #1 fix was "kill the PatternTalk framing" |
| — largely done in README + 01-vision; the deeper docs still need the same pass. |
|
|
| ## The SA3 tempo/time-signature reality (don't re-learn it) |
|
|
| Stable Audio 3 has NO tempo or time-signature conditioning channel. "BPM" is a |
| weak semantic hint in the prompt. So "make SA3 follow a time signature" is not |
| possible with the stock model. The answer is MIDI-first (timing from MIDI/audio |
| detection) + SA3 as the timbre generator. Full reasoning is in the skill ref |
| `sa3-no-tempo-control.md`. The LoRA style path is separate. |
|
|
