docs/PROGRESS.md

Progress log

Last updated: 2026-05-12

Pass 1: project review, timing, and Gradio replacement

Completed:

1. Inspected the original project structure and active Gradio entrypoints.
2. Moved previous Gradio interfaces into legacy/.
3. Created pipeline_runner.py as the timed orchestration layer.
4. Created app.py as a FastAPI backend.
5. Created a custom no-build browser frontend under web/.
6. Added stage timing to each extraction run.
7. Added synthetic benchmarking via scripts/benchmark_subprocesses.py.
8. Added initial docs for project review, timing/realtime, API, UI, and remaining work.

Outcome:

The application became usable without Gradio and produced per-run manifests/artifacts.

Pass 2: feature ledger and continued development

Completed in this pass:

1. Added first-class docs for features, tasks, and progress.
2. Added GET /api/jobs for active/completed run listing.
3. Added run-history UI panel that indexes .runs/*/output/manifest.json.
4. Added disk caching for decoded full mix and Demucs stem outputs.
5. Extended cache clearing to remove both memory and disk cache.
6. Added clustering_mode pipeline parameter.
7. Added online_preview clustering using prototype assignment.
8. Added frontend controls for clustering mode and disk cache.
9. Fixed duplicate sample writes in sample_extractor.build_archive.
10. Updated README and docs to reflect the new state.

Outcome:

The project now has a clearer product surface: final-quality batch extraction, faster online-style preview clustering, persistent run history, and explicit docs tracking what is done versus still missing.

Current assessment

The application is not “fully complete” as an editing workstation, but it is substantially implemented as an extraction and review workstation. The remaining gaps are concentrated around mutating corrections/editing, run comparison, and frontend engineering hardening.

Next recommended pass

Implement the editing loop:

1. Add edit state for deleted/restored hits and shifted onsets.
2. Add label change, cluster reassignment, merge, and split.
3. Re-export without rerunning Demucs/onset detection when only grouping changes.
4. Save edit decisions into the manifest.
5. Add side-by-side run comparison for parameter tuning.

Validation performed in this pass

• Compiled active Python files with python3 -m py_compile app.py pipeline_runner.py sample_extractor.py scripts/*.py.
• Ran FastAPI smoke job through scripts/test_api_job.py.
• Ran an online-preview API smoke job with synthetic audio.
• Verified GET /api/jobs history output and POST /api/cache/clear behavior.
• Verified SSE completion and review-hit artifact serving.
• Refreshed batch and online benchmark JSON files:
  • docs/benchmark-subprocesses.json
  • docs/benchmark-online-preview.json

Pass 3: hit review and streaming progress

Completed in this pass:

1. Added GET /api/jobs/{job_id}/events as a server-sent-events progress stream.
2. Updated the frontend to consume SSE via EventSource, with the existing polling loop retained as fallback.
3. Added per-hit review artifact export under review/hits/.
4. Added a top-level hits array to each run manifest with onset, duration, classification, cluster label, representative flag, and file path.
5. Added API serialization for hit playback/download URLs.
6. Added selected-hit and selected-sample audio players.
7. Made waveform onset markers clickable by selecting the nearest detected hit.
8. Added hit table and sample-table audition controls.
9. Hardened artifact file serving by using resolved path containment via Path.relative_to().
10. Refreshed batch and online benchmark JSON files after the review-hit export change.

Outcome:

The app now supports a real review loop for inspecting what the onset detector and clustering produced. Users can audition individual detected slices, representative samples, stem audio, and reconstruction audio from one screen. Progress updates are lower-latency and less wasteful via SSE while still remaining robust in browsers that need polling fallback.

Pass 4: interactive supervised UX foundation

Completed in this pass:

1. Added the supplied interactive UX document set under docs/interactive-ux/.
2. Read and aligned the UX documents with the project as currently implemented.
3. Added supervised_state.py for persistent semantic state beside each completed run manifest.
4. Added supervision_state.json generation after each successful extraction.
5. Added state schema for hits, clusters, constraints, events, suggestions, confidence, review queue, and undo snapshots.
6. Added supervised editing endpoints:
   • GET /api/jobs/{job_id}/state
   • POST /api/jobs/{job_id}/hits/{hit_id}/move
   • POST /api/jobs/{job_id}/hits/{hit_id}/pull-out
   • POST /api/jobs/{job_id}/hits/{hit_id}/suppress
   • POST /api/jobs/{job_id}/hits/{hit_id}/review
   • POST /api/jobs/{job_id}/clusters/{cluster_id}/lock
   • GET /api/jobs/{job_id}/suggestions
   • POST /api/jobs/{job_id}/suggestions/{suggestion_id}/accept
   • POST /api/jobs/{job_id}/suggestions/{suggestion_id}/reject
   • GET /api/jobs/{job_id}/explain/cluster/{cluster_id}
   • POST /api/jobs/{job_id}/undo
7. Added an interactive supervision UI panel with:
   • state summary,
   • selected-hit actions,
   • target cluster picker,
   • outlier-first review queue,
   • cluster board,
   • suggestion inbox,
   • constraint/event log,
   • cluster explanation drawer.
8. Added scripts/test_interactive_supervision.py to verify the supervised API loop.

Outcome:

The app is now an extraction and supervised-review workstation at the semantic-state level. User corrections are persisted as constraints/events and can be inspected, suggested from, and undone. The next required step is edited-state export so these decisions affect downloadable artifacts.

Current assessment after Pass 4

The project now satisfies the first interactive UX milestone for replayable supervised state:

analyze audio
→ inspect hits/clusters
→ move/pull/suppress/favorite/lock
→ persist constraints/events
→ update confidence and review queue
→ generate/accept/reject suggestions
→ explain clusters
→ undo semantic edits
→ reload completed run with decisions intact

It now satisfies the first full semantic-edit loop because edited semantic state can be rendered into separate supervised sample WAVs, MIDI, reconstruction, and ZIP output.

Next recommended pass after Pass 4

1. Add cluster merge/relabel/split workflows.
2. Add feature-vector cache for true local reclustering.
3. Add edited-vs-original run comparison.
4. Add browser-level UI tests and migrate frontend to TypeScript/Vite after UX stabilizes.

Pass 5: supervised export, force-onset, restore, and suggestion diffs

Completed in this pass:

1. Added supervised_export.py to render supervision_state.json into edited artifacts under supervised/.
2. Added POST /api/jobs/{job_id}/export for edited sample-pack export.
3. Added POST /api/jobs/{job_id}/hits/force-onset to create user-forced hit slices from stem.wav.
4. Added add-onset waveform mode in the frontend.
5. Added POST /api/jobs/{job_id}/hits/{hit_id}/restore and a restore button for suppressed hits.
6. Added exact suggestion diff previews through suggestion.diff and a UI Diff action.
7. Updated supervised export to exclude suppressed hits and honor favorite/pinned representatives.
8. Added scripts/test_supervised_export_and_force_onset.py.
9. Added docs/SUPERVISED_EXPORT_AND_FORCE_ONSET.md and updated feature/API/task/progress docs.

Outcome:

The project now closes the main semantic-edit loop:

analyze audio
→ inspect hits/clusters
→ move/pull/suppress/restore/favorite/lock/force-onset
→ inspect suggestions and diffs
→ export edited WAV/MIDI/reconstruction/ZIP artifacts

The original batch artifacts remain immutable. Edited outputs are written separately under supervised/.

Validation performed in this pass:

• python3 -m py_compile app.py pipeline_runner.py sample_extractor.py supervised_state.py supervised_export.py scripts/*.py
• node --check web/app.js
• python3 scripts/test_supervised_export_and_force_onset.py
• python3 scripts/test_sse_and_review_hits.py
• python3 scripts/test_interactive_supervision.py
• python3 scripts/test_api_job.py

Next recommended pass after Pass 5:

1. Add cluster merge/relabel/split workflows.
2. Add cached feature-vector local reclustering around edited hits.
3. Add edited-vs-original run comparison.
4. Add browser-level UI tests and migrate the frontend to TypeScript/Vite once the UX stops shifting.

Pass 6: visual simplification toward the supplied reference UI

Completed in this pass:

1. Reworked web/index.html around a simpler first screen: top file/action bar, large waveform workspace, compact right-side extraction controls, sample cards, and lower utility panels.
2. Replaced the previous dark dashboard styling with a light, minimal, card-based visual system closer to the supplied screenshot.
3. Moved advanced extraction controls into a collapsible panel so the primary workflow exposes only stem, sensitivity, cluster count, extraction, export, and fast-mode decisions.
4. Added representative sample cards with waveform thumbnails decoded from the sample WAV URLs in the browser.
5. Updated waveform rendering to use grey filled waveforms plus colored lollipop onset markers similar to the reference image.
6. Preserved all existing IDs/API wiring so supervised editing, SSE progress, run history, force-onset, and edited export continue to work.

Validation performed in this pass:

• python3 -m py_compile app.py pipeline_runner.py sample_extractor.py supervised_state.py supervised_export.py scripts/*.py
• node --check web/app.js
• HTML parser sanity check for web/index.html
• python3 scripts/test_api_job.py

Outcome:

The app now looks and behaves more like a focused sample-extraction tool instead of a generic control dashboard while keeping the advanced review/supervision functionality available below the main workflow.

Pass 6: minimal waveform-first UI

Completed in this pass:

1. Reworked the frontend visual language from a dense workstation dashboard into a light waveform-first interface.
2. Added the top file identity bar and primary purple extraction action.
3. Promoted the waveform and extracted sample cards as the default workflow.
4. Moved advanced controls and power-user tools into collapsible areas.

Outcome:

The app matched the general direction of the supplied reference image, but still had visible complexity: separate audio players, a waveform header, fast-mode buttons in the right card, and multiple utility panels visible in the main page flow.

Pass 7: reference-alignment hardening

Completed in this pass:

1. Removed the visible waveform header and made the waveform card visually quiet.
2. Added a single custom transport row with play, elapsed/total time, and seek line.
3. Hid native source/stem/reconstruction/hit/sample audio elements from the default layout while preserving playback behavior.
4. Moved Online preview mode and Fast full-mix mode into the collapsed Advanced panel.
5. Collapsed pipeline, run history, supervision, and detailed tables into one Review & edit workbench below the sample cards.
6. Simplified sample cards to thumbnail waveform, play affordance, and label-first presentation.
7. Updated docs to distinguish visual-reference work from remaining interaction-depth work.

Outcome:

The default UI is now aligned with the supplied image: file/action bar, large waveform card, compact right controls, one transport row, and sample cards. Advanced extraction and semantic editing remain available but no longer dominate the first screen.

Pass 8: fixed no-scroll workstation layout and upload repair

Completed in this pass:

1. Reworked web/index.html into a fixed workstation shell: top bar, left sidebar, center workspace, right sidebar, and bottom dock.
2. Locked the document viewport with overflow: hidden and moved long content into panel-local scroll containers.
3. Moved pipeline logs/history/selection context into the left sidebar.
4. Moved extraction, export, and advanced settings into the right sidebar.
5. Moved semantic review/edit tools and raw tables into the bottom dock.
6. Added an explicit top-bar Upload audio affordance.
7. Added whole-window drag/drop handling so dropping a file anywhere on the app selects it instead of opening it in the browser.
8. Added a full-screen drag overlay to make the drop target obvious.
9. Preserved existing frontend ids and backend APIs so extraction, SSE, supervision, force-onset, and edited export remain wired.

Outcome:

The UI no longer behaves like a scrollable webpage. It now behaves like a compact desktop-style sample extraction workstation with simple expandable tool panels around a central waveform/sample workspace.

Pass 9: full-context reproduced audio and clearer parameters

Completed in this pass:

1. Added explicit source/context/reconstruction layers to the pipeline export:
   • source.wav
   • stem.wav
   • context_bed.wav
   • target_reconstruction.wav
   • reconstruction.wav
2. Changed reconstruction.wav to a full-context reproduced mix: non-target context bed plus sample-triggered target reconstruction.
3. Kept target_reconstruction.wav as the isolated sample-only target layer for debugging and focused listening.
4. Matched the target reconstruction RMS to the target stem before mixing it back into context.
5. Updated sample-pack.zip to include both the full-context reproduced mix and target-stem reconstruction.
6. Updated supervised edited export so edited packs follow the same audio-layer model under supervised/.
7. Added Source / Stem / Reproduced preview modes to the single transport row; completed jobs default to Reproduced.
8. Reworked the right sidebar into Common controls vs Advanced parameters.
9. Grouped advanced parameters by pipeline stage: stem separation, hit detection, grouping, export/cache.
10. Added docs/REPRODUCED_AUDIO_AND_PARAMETERS.md.

Outcome:

The app is easier to understand for normal use: the main right-side controls are now only the few controls users are likely to touch repeatedly, while lower-level DSP/model controls stay available but grouped by stage. Reproduced audio is now useful for musical judgment because separated-stem runs are previewed inside the rest of the mix rather than as an isolated sample-triggered stem only.

Pass 10: clean default UI

Completed in this pass:

1. Collapsed secondary panels by default: source hints, selection context, pipeline, run history, exports, and review/edit no longer open on initial load.
2. Reduced the bottom dock to a compact tab bar unless a bottom tool is explicitly opened.
3. Shortened the top bar and reduced logo/upload/backend/action button visual weight.
4. Shrunk sidebars, panel padding, gaps, transport height, and sample-card dimensions to give the waveform and samples more room.
5. Hid common-control helper paragraphs and long field hints from the default extraction flow.
6. Preserved advanced controls, logs, history, supervision, tables, and edited export behind existing expandable panels.
7. Added docs/CLEAN_DEFAULT_UI.md to document the clean-default layout contract.

Outcome:

The UI remains a fixed workstation with sidebars and a bottom dock, but the first screen is now much less cluttered. The visible default workflow is upload, extract, listen, inspect waveform, and audition sample cards; debug/review/power-user tools are available only when opened.

Pass 11: immediate waveform and real progress flow

Completed in this pass:

1. Added browser-side waveform rendering immediately after file selection/drop using Web Audio decoding.
2. Added a persistent uploaded-source overview so the waveform is visible before backend extraction starts.
3. Added a backend progress object to job payloads and SSE job events.
4. Extended StageTiming with progress, work_done, and work_total.
5. Added chunk-accurate Demucs stem progress reporting through a progress pool passed to apply_model.
6. Changed the waveform renderer to show active extraction progress as an accent-colored completed region on the left and a neutral remaining region on the right.
7. Added a Start here left-sidebar flow with Load → Controls → Extract → Review/export states.
8. Updated the stage list to include exact per-stage progress bars where progress is available.
9. Added docs/IMMEDIATE_WAVEFORM_AND_REAL_PROGRESS.md.

Outcome:

The app now starts with a clear sequence instead of a blank workstation. Uploading or dropping a file immediately produces a waveform, and extraction progress is visualized directly on that waveform using backend-reported progress only. The UI does not fake ETA or time-based progress; non-instrumented stages step only at real boundaries.

2026-05-12 — API validation and visible error pass

Completed:

• Fixed a real /api/jobs regression where the UI submitted subdivision as a string and the backend rejected it with 400.
• Added backend parameter coercion before validation for integer, float, and boolean pipeline params.
• Updated frontend collectParams() to use /api/config.defaults as the type source for controls, including <select> elements.
• Added a global error banner so failed API requests and pipeline errors are visible without browser devtools.
• Removed a duplicate historyList id in the HTML.
• Added docs/API_ERRORS_AND_PARAMETER_VALIDATION.md and scripts/test_param_validation_and_api_errors.py.

Validation:

python3 scripts/test_param_validation_and_api_errors.py
node --check web/app.js
python3 -m py_compile app.py pipeline_runner.py sample_extractor.py supervised_state.py supervised_export.py scripts/*.py

Pass: automatic simple card flow

Completed in this pass:

1. Changed the default interaction model from manual configuration to automatic processing after drop/upload.
2. Added PipelineParams.auto_tune and a new Automatic parameter tuning pipeline stage.
3. Added progressive sample events from the export stage and serialized active-job partial_samples.
4. Redesigned sample cards into type-grouped columns with draw/dismiss/trim/extend actions.
5. Added waveform zoom and pan while preserving click-to-select/force-onset semantics.
6. Replaced expensive representative scoring internals with a lightweight deterministic scorer.
7. Added fallback MIDI writing when pretty_midi is missing.
8. Hardened BPM detection against sparse/empty beat events.

Validation performed:

• Python compile checks for app, pipeline, DSP, supervision, export, and scripts.
• node --check web/app.js.
• DOM id/reference consistency check.
• scripts/test_param_validation_and_api_errors.py.
• scripts/test_progress_contract.py.
• scripts/test_api_job.py.
• scripts/test_sse_and_review_hits.py.
• scripts/test_supervised_export_and_force_onset.py.
• scripts/test_interactive_supervision.py.

2026-05-12 Reference image UI alignment

• Reworked the frontend shell to match the provided Sample Extractor reference image.
• Added centered file picker/current filename, right-aligned export actions, persistent right settings panel, waveform-first canvas, grouped sample columns, and compact bottom selection bar.
• Kept automatic drop-to-process behavior and progressive sample-card rendering.
• Moved secondary pipeline/history/supervision/tables into a compact tools drawer.

Pass 14: selected cards and Spleeter backend

Completed in this pass:

1. Added spleeter as the default separation backend, with selectable spleeter:2stems, spleeter:4stems, and spleeter:5stems profiles.
2. Kept demucs as a quality/fallback backend and none as the full-mix preview backend.
3. Added optional requirements-spleeter.txt instead of forcing TensorFlow/Spleeter into the base install.
4. Added per-card checkbox state with manual select-all/clear behavior.
5. Added selected-only backend export via POST /api/jobs/{job_id}/export-selected.
6. Made draw another persist the chosen representative in semantic state.
7. Made trim/extend rewrite playable preview audio immediately under overrides/hits/.
8. Added scripts/test_selected_export_card_actions.py.

Outcome:

The default app now behaves more like a card review tool: drop audio, let Spleeter/fallback separation run, review grouped cards, select/dismiss/draw/trim, and export only the selected pack.

Upload/runtime fallback update (2026-05-12)

• Added a visible top-bar Choose audio affordance in addition to whole-app drag/drop.
• Fixed the default hidden state of the error banner so placeholder errors are not shown on page load.
• API errors now surface request path/status/detail in the visible banner and pipeline logs.
• /api/config now includes runtime diagnostics for optional separation backends.
• If Spleeter is unavailable, the simple UI keeps the app usable by switching to full-mix mode; backend fallback also uses full-mix rather than silently launching Demucs.