Deploy dense piano fitting in both Space input modes

README.md

@@ -16,13 +16,13 @@ models:
 
 An interactive demonstration of **fixed-timbre inverse synthesis**: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then listen to the source and play the resulting MIDI in an interactive piano roll. Download the actual MIDI from the same result.
 
-English and Chinese examples are available instantly through the two example buttons. They contain about eight seconds of speech each and use the full refinement algorithm. New jobs default to a smaller search budget for responsive CPU experimentation. The technique approximates spectral structure; it does not guarantee intelligible words or perform conventional melody transcription.
+The two instant-play examples now use the dense algorithm: **The Tyger** in English and **Quiet Night Thoughts** in Chinese. Both input modes use the same dense fitter, with 600 optimization iterations by default. The research backtest reduced pooled Chinese CER from 100% to 68.3% and English WER from 116.7% to 50% on five poetry recordings. Recognition remains imperfect, and these results do not establish general human intelligibility. See the [ASR report](https://github.com/k-l-lambda/speaking-midi/blob/5caf4e2/experiments/DENSE_ASR.md).
 
 ## MIDI player
 
 The demo centers on a piano roll inspired by the interaction design of `klstudio/app/views/midi-player.vue`: horizontal note blocks, active-note highlighting, a playback cursor, click-to-seek, and automatic scrolling at higher zoom. Controls provide play/pause, restart, a seek slider, playback speed, and zoom; left/right arrows on the roll seek by one second. Loading another result stops the previous playback.
 
-The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; research diagnostics remain in the generated files.
+The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; fitting settings and numerical results remain in each job’s `report.json`.
 
 ## Poetry voice examples
 
@@ -35,31 +35,28 @@ Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-dom
 
 Inputs are limited to 25 MB, 300 text characters, and a fitted prefix of at most 12 seconds. Longer recordings are read only through the selected prefix. New jobs share one compute queue, with at most eight queued requests. Temporary outputs are ephemeral, are cleaned after 24 hours when another job starts, and disappear when the Space restarts. Do not treat this public demonstration as private storage.
 
-## Algorithm
+## Dense fitting algorithm
 
-The renderer is FluidSynth with bank 0, program 0 of the bundled MuseScore General SoundFont, fixed gain 0.8, and disabled chorus/reverb. Each candidate MIDI event has a pitch, velocity, onset and gate duration. We seek events whose **rendered sound** resembles the input, rather than merely extracting its fundamental frequency.
+The renderer is FluidSynth with bank 0, program 0 of the bundled Grand Piano SoundFont, fixed gain 0.8, and disabled chorus/reverb. The output contains only piano MIDI notes: no source-speech layer, pitch bend, or output EQ.
 
-1. Convert the input to mono at 24 kHz and normalize RMS to 0.035. Analyze magnitudes using a 2048-sample Hann STFT with 240-sample (10 ms) hops and project them onto 96 triangular frequency bands.
-2. Render a dictionary of 657 templates: pitches 36–108, velocities 45/75/105, and gate durations 60/120/240 ms. Each template includes a short release tail. Cache the dictionary by renderer configuration and SoundFont hash.
-3. Weight each band inversely to the square root of its RMS magnitude, with a floor to avoid amplifying near-silence. Greedily select a template and onset that most reduce weighted squared residual error. For residual $R$ and a shifted atom $A$, the improvement is
+1. Convert input to 24 kHz mono and normalize its RMS to 0.035. Analyze squared STFT magnitudes with FFT 1024 / hop 240 and 128 Slaney-normalized mel bands spanning 40–11,000 Hz.
+2. Render temporal templates for all 88 piano keys (MIDI 21–108), at velocity 48 and a 20 ms gate. A 400 ms template window retains attack, decay and release. Cache templates and per-pitch velocity calibration by SoundFont hash and configuration.
+3. Give every key an onset candidate every 20 ms. Jointly optimize nonnegative strengths with projected Adam rather than greedily selecting a small note list. The default is 600 iterations, with a cosine learning-rate schedule from 0.04 to 0.001.
+4. Map strengths to integer velocities using measured energy ratios at 13 velocity levels. Drop strengths below the velocity-8 threshold and events extending beyond the clip. The selected research clips contain 1,218–3,140 notes; the iteration setting is not a note-count limit.
+5. Save MIDI, reload it and render through FluidSynth. The interactive player displays these exported events and plays their corresponding piano WAV, including a 0.5-second release tail.
 
-$$
-\Delta = 2\langle R,A\rangle - \lVert A\rVert_2^2.
-$$
-
-Subtract the selected atom and forbid overlapping holds of the same MIDI pitch. Stop on nonpositive improvement or the iteration budget. The budget counts search attempts, so the accepted note count may be smaller.
-
-4. Export and decode the MIDI, then render it through the actual synthesizer. Test global velocity offsets against a log-band objective. With **Full per-note refinement**, also try each note's velocity ±10, onset ±10 ms, gate duration ±20 ms, and deletion, accepting only measured improvements:
+For target mel power $M$, temporal templates $D$, and nonnegative note strengths $A$, fitting minimizes:
 
 $$
-L = \frac{1}{BT}\sum_{b,t}\left|\log(1+F(y)_{b,t})-\log(1+F(S(M))_{b,t})\right|.
+L=\operatorname{mean}\left[\log(1+\widehat M/10^{-3})-\log(1+M/10^{-3})\right]^2,
+\qquad \widehat M=D*A,\quad A\geq0.
 $$
 
-Here $F$ is band-magnitude analysis, $y$ is the normalized target, $M$ is MIDI, and $S$ is the fixed piano renderer. The quick setting runs dictionary search and global velocity refinement; it skips the per-note sweep. The baseline rounds strong spectral peaks to piano pitches and separately calibrates global velocity.
+The convolution sums shifted template powers. It approximates phase interactions, velocity-dependent timbre and repeated-note behavior, so validation uses actual re-rendered MIDI. No transcript or ASR gradient enters the optimizer. ASR selected a configuration on two pilot clips before testing three further poetry clips; repeated renders, temperature-zero decoding and a complete CPU path confirmed the corpus-level improvement. Some conventional spectral metrics worsen, and there is no blind human listening result.
 
-Saved diagnostic spectrograms use the same color scale for all four panels. Recorded metrics compare the input-length interval; audio includes a 0.5-second release tail. Log-band MAE is the optimization target, while spectral convergence is reported separately and need not improve. Neither metric measures speech intelligibility. Magnitude addition is only an approximation because real notes interact in phase, which motivates the renderer refinement step.
+`Fitting iterations` ranges from 200 to 1,200; 600 is the tested setting. More iterations cost more time and do not guarantee clearer speech. The old note-budget and full-refinement controls have been removed from the demo.
 
-The fitting core is vendored from [Speaking MIDI](https://github.com/k-l-lambda/speaking-midi/tree/932f8eb), with a portable SoundFont path and an optional quick-refinement flag. See that project for full experiments, derivations, and reproducible demo provenance.
+The dense fitter is vendored without algorithm changes from [Speaking MIDI commit 5caf4e2](https://github.com/k-l-lambda/speaking-midi/tree/5caf4e2). The shared rendering core retains the Space’s portable SoundFont path. The instant examples in `examples/dense` are byte-identical MIDI/WAV copies of the selected research outputs, with normalized source audio for comparison and a hash manifest. Archived sparse examples remain in `examples/en` and `examples/zh` for provenance.
 
 ## CPU and ZeroGPU execution
 
@@ -67,7 +64,7 @@ The hosted [Speaking MIDI Space](https://huggingface.co/spaces/k-l-lambda/speaki
 
 The requirements use `torch==2.10.0` and matching torchaudio. Do not add `+cpu` or the CPU-only wheel index to the hosted requirements: ZeroGPU rejects that version specification and needs GPU-capable PyTorch.
 
-Local execution defaults to `INFERENCE_DEVICE=cpu`. TTS uses float32 with SDPA in a separate process, releasing model memory before MIDI fitting. No GPU is needed for this path.
+Local execution defaults to `INFERENCE_DEVICE=cpu`. TTS uses float32 with SDPA in a separate process, releasing model memory before MIDI fitting. Dense fitting explicitly uses CPU in every app mode, including ZeroGPU. Both template creation and optimization work without a GPU.
 
 ```bash
 # Python 3.10; install libfluidsynth3, libsndfile1 and sox using your OS package manager.

VALIDATION.md

@@ -24,3 +24,12 @@ These timings are the app's reported job durations, excluding client upload/down
 ## Interactive MIDI player (2026-10-02)
 
 The demo now shows source audio and an interactive piano roll instead of diagnostic tabs. Seven backend tests passed, including real CPU fitting and the ZeroGPU dispatch test. Browser checks passed for playback and pause, click and keyboard seeking, zoom and scrolling, replacing a playing example, removal of diagnostic tabs, and mobile layout. No JavaScript errors were observed in those checks. The player decodes note events from the downloaded MIDI and synchronizes them to that MIDI's Grand Piano audio render.
+
+## Dense fitting deployment (2026-10-02)
+
+- Vendored `speaking_midi/dense.py` is byte-identical to research commit `5caf4e2`; the portable bundled-SoundFont renderer is retained.
+- Ten automated tests passed, including an actual CPU audio job, mocked ZeroGPU speech dispatch followed by real CPU dense fitting, single-note dense recovery, input validation, MIDI round-trip and packaged example loading.
+- A local Gradio API job fitted the complete 8.64-second Quiet Night recording at 600 iterations, producing 2,758 notes in 14.1 seconds. Small floating-point/renderer differences can change the discrete notes; these are workstation timings, not hosted benchmarks.
+- Chrome/Playwright verified the 3,140-note English example and 2,759-note Chinese example, playback/pause, replacement while loaded, updated fitting controls, and a 390-pixel mobile layout. No JavaScript errors or horizontal overflow were observed.
+- The instant-play MIDI and WAV assets are exact copies of the dense research examples. Their hashes and research revision are in `examples/dense/manifest.json`. Both input modes now invoke the same dense CPU worker; the old note-budget/refinement controls are removed.
+- Earlier hosted measurements above describe the previous sparse implementation and must not be treated as dense-performance measurements.

app.py

@@ -42,7 +42,7 @@ def worker(command, log, timeout=1800):
         except subprocess.TimeoutExpired:
             process.kill()
             process.wait()
-            raise gr.Error('This job exceeded 30 minutes. Try a shorter clip or a smaller note budget.') from None
+            raise gr.Error('This job exceeded 30 minutes. Try a shorter clip or fewer fitting iterations.') from None
         except BaseException:
             process.kill()
             process.wait()
@@ -68,13 +68,13 @@ def prepare_audio(path, out, seconds):
 
 
 def result(directory, status):
-    return (status, str(directory/'target.wav'), player_payload(directory),
-            str(directory/'refined.mid'))
+    return (status, str(directory/'target.wav'), player_payload(directory, stem="dense"),
+            str(directory/'dense.mid'))
 
 
-def run(mode, audio, text, language, description, seconds, budget, refine, progress=gr.Progress()):
-    if mode not in {'Audio → MIDI', 'Text → speech → MIDI'} or not .5 <= float(seconds) <= 12 or not 20 <= int(budget) <= 240:
-        raise gr.Error('Choose a valid mode, duration (0.5–12 s), and budget (20–240).')
+def run(mode, audio, text, language, description, seconds, steps, progress=gr.Progress()):
+    if mode not in {'Audio → MIDI', 'Text → speech → MIDI'} or not .5 <= float(seconds) <= 12 or not 200 <= int(steps) <= 1200:
+        raise gr.Error('Choose a valid mode, duration (0.5–12 s), and iteration count (200–1200).')
     if language not in {'Auto', 'English', 'Chinese', 'Japanese', 'Korean', 'French', 'German', 'Spanish', 'Portuguese', 'Russian'}:
         raise gr.Error('Choose a supported language.')
     for old in RUNS.glob('job-*'):
@@ -86,7 +86,7 @@ def run(mode, audio, text, language, description, seconds, budget, refine, progr
     if mode == 'Text → speech → MIDI':
         if not text.strip() or len(text) > 300 or not description.strip() or len(description) > 600:
             raise gr.Error('Enter 1–300 text characters and a voice description of 1–600 characters.')
-        yield ('1/4 · Generating speech. '+('Waiting for a shared GPU.' if DEVICE == 'zerogpu' else 'CPU VoiceDesign can take several minutes on first use.'), *empty)
+        yield ('Generating speech. '+('Waiting for a shared GPU.' if DEVICE == 'zerogpu' else 'CPU VoiceDesign can take several minutes on first use.'), *empty)
         progress(.05, desc='Loading VoiceDesign and generating speech')
         prompts = directory/'prompts.json'
         prompts.write_text(json.dumps([dict(name='speech', text=text.strip(), language=language, instruct=description.strip())]))
@@ -101,21 +101,20 @@ def run(mode, audio, text, language, description, seconds, budget, refine, progr
             worker(command, directory/'tts.log')
         audio = str(directory/'speech.wav')
     duration = prepare_audio(audio, directory/'input.wav', seconds)
-    yield (f'2/4 · Analyzing the first {duration:.2f} s and building piano templates.',
+    yield (f'Fitting the first {duration:.2f} s with dense piano notes.',
            str(directory/'input.wav'), '', None)
-    progress(.3, desc='Fitting piano note templates; then checking the real synthesizer')
-    command = ['-m', 'speaking_midi', str(directory/'input.wav'), '--out', str(directory),
-               '--soundfont', str(ROOT/'assets/gm.sf3'), '--max-notes', str(int(budget))]
-    if not refine:
-        command.append('--skip-local-refinement')
+    progress(.3, desc='Fitting dense piano notes and rendering the MIDI')
+    command = ['-m', 'speaking_midi.dense', str(directory/'input.wav'), '--out', str(directory),
+               '--soundfont', str(ROOT/'assets/gm.sf3'), '--steps', str(int(steps)), '--device', 'cpu']
     worker(command, directory/'fit.log')
     progress(.95, desc='Preparing the MIDI player')
-    yield result(directory, f'4/4 · Complete in {time.monotonic()-started:.1f} s on {DEVICE}. '
-                 + ('Full renderer refinement.' if refine else 'Quick fit with global velocity refinement.'))
+    report = json.loads((directory/'report.json').read_text())
+    yield result(directory, f"Complete in {time.monotonic()-started:.1f} s · {report['notes']:,} piano notes.")
 
 
 def example(language):
-    return result(ROOT/'examples'/language, 'Precomputed VoiceDesign example · full renderer refinement · generated on GPU.')
+    name = {'en': 'the-tyger', 'zh': 'quiet-night'}[language]
+    return result(ROOT/'examples/dense'/name, 'Precomputed poetry voice · dense piano fitting.')
 
 
 def build_ui():
@@ -123,8 +122,8 @@ def build_ui():
         gr.Markdown('# Speaking MIDI\nTurn a voice into piano notes. Listen to your voice on piano, follow the notes, and download the MIDI.\n'
                     'This is experimental sound approximation: words may become difficult to understand.')
         with gr.Row():
-            english = gr.Button('Listen to English example')
-            chinese = gr.Button('Listen to Chinese example')
+            english = gr.Button('Listen to The Tyger · English')
+            chinese = gr.Button('Listen to Quiet Night · Chinese')
         with gr.Row():
             with gr.Column(scale=1):
                 mode = gr.Radio(['Audio → MIDI', 'Text → speech → MIDI'], value='Audio → MIDI', label='Start with')
@@ -135,10 +134,10 @@ def build_ui():
                     description = gr.Textbox(value=VOICE, label='Voice description', lines=3, max_length=600)
                 with gr.Accordion('Fitting options', open=False):
                     seconds = gr.Slider(.5, 12, value=8, step=.5, label='Fit the first N seconds')
-                    budget = gr.Slider(20, 240, value=80, step=20, label='Search iteration budget')
-                    refine = gr.Checkbox(value=False, label='Full per-note refinement (slower)')
+                    steps = gr.Slider(200, 1200, value=600, step=200, label='Fitting iterations',
+                                      info='600 is the tested setting. More iterations take longer and may not improve clarity.')
                 submit = gr.Button('Generate piano MIDI', variant='primary')
-                gr.Markdown('CPU inference is supported. Speech generation and full refinement can take several minutes. '
+                gr.Markdown('CPU inference is supported. Speech generation can take several minutes; piano fitting runs on CPU. '
                             'Try the ready-made examples for instant playback. Clips are processed up to the selected duration.')
             with gr.Column(scale=2):
                 status = gr.Markdown('Choose an example or create a new fit.')
@@ -155,14 +154,14 @@ def build_ui():
         )
         selected_text = gr.Textbox(label='Selected text', interactive=False, lines=2)
         text.change(lambda value: value, text, selected_text, queue=False)
-        gr.Markdown('**How it works:** synthesize 657 piano templates → match weighted spectral bands → '
-                    'choose discrete notes → re-render MIDI → reduce measured spectral error. '
+        gr.Markdown('**How it works:** render piano templates → jointly fit dense 20 ms notes → '
+                    'convert strengths to MIDI velocities → render the MIDI with the same piano sound. '
                     '[Algorithm and experiments](https://github.com/k-l-lambda/speaking-midi) · '
                     '[Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign)')
         outputs = [status, source, player, midi]
         mode.change(lambda value: (gr.update(visible=value=='Audio → MIDI'), gr.update(visible=value!='Audio → MIDI')),
                     mode, [audio, speech_controls], queue=False)
-        submit.click(run, [mode,audio,text,language,description,seconds,budget,refine], outputs,
+        submit.click(run, [mode,audio,text,language,description,seconds,steps], outputs,
                      concurrency_limit=1, concurrency_id='compute', api_name='generate')
         english.click(lambda: example('en'), outputs=outputs, api_name='english_example')
         chinese.click(lambda: example('zh'), outputs=outputs, api_name='chinese_example')

examples/dense/README.md

@@ -0,0 +1,5 @@
+# Dense piano examples
+
+The instant-play English and Chinese examples use The Tyger and Quiet Night Thoughts, respectively. `dense.mid` and `dense.wav` are copied byte-for-byte from the selected outputs of [research commit 5caf4e2](https://github.com/k-l-lambda/speaking-midi/tree/5caf4e2/examples/poetry/dense). `target.wav` is the corresponding source normalized to 24 kHz mono and RMS 0.035 for fitting. Source voice provenance is in `../poetry/manifest.json`.
+
+The MIDI was reloaded and rendered with FluidSynth and the bundled Grand Piano preset. No source speech is mixed into the piano audio. Each output includes 0.5 seconds of release. File hashes are recorded in `manifest.json`; the original fitting reports are retained per sample.

examples/dense/manifest.json

@@ -0,0 +1,23 @@
+{
+  "research_commit": "5caf4e2",
+  "samples": [
+    {
+      "name": "quiet-night",
+      "files": {
+        "report.json": "38bf70c1a63eb1751437db2ba67a4ee31724fa5097480ba9b0a54fa5af4b7377",
+        "dense.wav": "4cc520461e30b844776b5a38445fd66beed5b9ffc85877e59d72a83be0099d42",
+        "dense.mid": "5408c4eba06808db8f15d5f1fc59340313bc8b1dde0855f2d4dd13e9c844374d",
+        "target.wav": "850ffb95218374a0934cc2af2e0e9b8eff62f88a777c8f9503943ee47ad28baa"
+      }
+    },
+    {
+      "name": "the-tyger",
+      "files": {
+        "report.json": "1a24a559190165b63b744817db6916c1a2c40a7c05e588ae68398142f7e6abf8",
+        "dense.wav": "de80935e2f6dc060b88b5bb756cfce4eb58fcb16f64a466b038ea492b2c79281",
+        "dense.mid": "9300cdbce9a1fd07655fa48845ba225a8f9b8ce9b83e07ef8bd471962fe62b13",
+        "target.wav": "b102db0c980e4b31962656c7487e4edcb730870e08c98b97e77e0fb44bf77fa2"
+      }
+    }
+  ]
+}

examples/dense/quiet-night/dense.mid

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5408c4eba06808db8f15d5f1fc59340313bc8b1dde0855f2d4dd13e9c844374d
+size 17250

examples/dense/quiet-night/dense.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4cc520461e30b844776b5a38445fd66beed5b9ffc85877e59d72a83be0099d42
+size 877520

examples/dense/quiet-night/report.json

@@ -0,0 +1,149 @@
+{
+  "config": {
+    "power": 2,
+    "gate": 0.02,
+    "velocity": 48,
+    "sr": 24000,
+    "nfft": 1024,
+    "hop": 240,
+    "pitches": [
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35,
+      36,
+      37,
+      38,
+      39,
+      40,
+      41,
+      42,
+      43,
+      44,
+      45,
+      46,
+      47,
+      48,
+      49,
+      50,
+      51,
+      52,
+      53,
+      54,
+      55,
+      56,
+      57,
+      58,
+      59,
+      60,
+      61,
+      62,
+      63,
+      64,
+      65,
+      66,
+      67,
+      68,
+      69,
+      70,
+      71,
+      72,
+      73,
+      74,
+      75,
+      76,
+      77,
+      78,
+      79,
+      80,
+      81,
+      82,
+      83,
+      84,
+      85,
+      86,
+      87,
+      88,
+      89,
+      90,
+      91,
+      92,
+      93,
+      94,
+      95,
+      96,
+      97,
+      98,
+      99,
+      100,
+      101,
+      102,
+      103,
+      104,
+      105,
+      106,
+      107,
+      108
+    ],
+    "soundfont_sha256": "5b85b6c2c61d10b2b91cddd41efcce7b25cd31c8271d511c73afafbef20b6fa3",
+    "revision": 1
+  },
+  "optimization": {
+    "steps": 600,
+    "grid_frames": 2,
+    "floor": 0.001,
+    "device": "cuda",
+    "lr_schedule": "cosine .04 to .001"
+  },
+  "metrics": {
+    "spectral_convergence": 0.8104307055473328,
+    "log_band_mae": 0.4648427665233612,
+    "waveform_rmse": 0.03722667321562767,
+    "peak": 0.10009765625,
+    "mr_log_mae": 0.07809968789418538,
+    "envelope_correlation": 0.8604701076958263
+  },
+  "notes": 2759,
+  "loss_history": [
+    [
+      0,
+      4.463901519775391
+    ],
+    [
+      100,
+      0.7258548736572266
+    ],
+    [
+      200,
+      0.7223665118217468
+    ],
+    [
+      300,
+      0.7200113534927368
+    ],
+    [
+      400,
+      0.7193714380264282
+    ],
+    [
+      500,
+      0.7192818522453308
+    ],
+    [
+      599,
+      0.7192714214324951
+    ]
+  ],
+  "wall_seconds": 4.883578613866121
+}

examples/dense/quiet-night/target.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:850ffb95218374a0934cc2af2e0e9b8eff62f88a777c8f9503943ee47ad28baa
+size 829520

examples/dense/the-tyger/dense.mid

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9300cdbce9a1fd07655fa48845ba225a8f9b8ce9b83e07ef8bd471962fe62b13
+size 19577

examples/dense/the-tyger/dense.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:de80935e2f6dc060b88b5bb756cfce4eb58fcb16f64a466b038ea492b2c79281
+size 839120

examples/dense/the-tyger/report.json

@@ -0,0 +1,149 @@
+{
+  "config": {
+    "power": 2,
+    "gate": 0.02,
+    "velocity": 48,
+    "sr": 24000,
+    "nfft": 1024,
+    "hop": 240,
+    "pitches": [
+      21,
+      22,
+      23,
+      24,
+      25,
+      26,
+      27,
+      28,
+      29,
+      30,
+      31,
+      32,
+      33,
+      34,
+      35,
+      36,
+      37,
+      38,
+      39,
+      40,
+      41,
+      42,
+      43,
+      44,
+      45,
+      46,
+      47,
+      48,
+      49,
+      50,
+      51,
+      52,
+      53,
+      54,
+      55,
+      56,
+      57,
+      58,
+      59,
+      60,
+      61,
+      62,
+      63,
+      64,
+      65,
+      66,
+      67,
+      68,
+      69,
+      70,
+      71,
+      72,
+      73,
+      74,
+      75,
+      76,
+      77,
+      78,
+      79,
+      80,
+      81,
+      82,
+      83,
+      84,
+      85,
+      86,
+      87,
+      88,
+      89,
+      90,
+      91,
+      92,
+      93,
+      94,
+      95,
+      96,
+      97,
+      98,
+      99,
+      100,
+      101,
+      102,
+      103,
+      104,
+      105,
+      106,
+      107,
+      108
+    ],
+    "soundfont_sha256": "5b85b6c2c61d10b2b91cddd41efcce7b25cd31c8271d511c73afafbef20b6fa3",
+    "revision": 1
+  },
+  "optimization": {
+    "steps": 600,
+    "grid_frames": 2,
+    "floor": 0.001,
+    "device": "cuda",
+    "lr_schedule": "cosine .04 to .001"
+  },
+  "metrics": {
+    "spectral_convergence": 0.7890517115592957,
+    "log_band_mae": 0.4172348976135254,
+    "waveform_rmse": 0.03725587949156761,
+    "peak": 0.088348388671875,
+    "mr_log_mae": 0.07024333005150159,
+    "envelope_correlation": 0.860843754328336
+  },
+  "notes": 3140,
+  "loss_history": [
+    [
+      0,
+      4.080715656280518
+    ],
+    [
+      100,
+      0.6443037390708923
+    ],
+    [
+      200,
+      0.6393406987190247
+    ],
+    [
+      300,
+      0.6365151405334473
+    ],
+    [
+      400,
+      0.6353092193603516
+    ],
+    [
+      500,
+      0.6349682807922363
+    ],
+    [
+      599,
+      0.634937584400177
+    ]
+  ],
+  "wall_seconds": 3.3393354159779847
+}

examples/dense/the-tyger/target.wav

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b102db0c980e4b31962656c7487e4edcb730870e08c98b97e77e0fb44bf77fa2
+size 791120

player.py

@@ -8,12 +8,12 @@ import soundfile as sf
 UI = Path(__file__).parent / 'ui'
 
 
-def player_payload(directory):
+def player_payload(directory, stem="refined"):
     directory = Path(directory)
     clock = 0.0
     active = {}
     notes = []
-    for message in mido.MidiFile(directory / 'refined.mid'):
+    for message in mido.MidiFile(directory / f'{stem}.mid'):
         clock += message.time
         if message.type == 'note_on' and message.velocity:
             active[(message.channel, message.note)] = (clock, message.velocity)
@@ -22,7 +22,7 @@ def player_payload(directory):
             notes.append(dict(pitch=message.note, start=start, duration=clock-start, velocity=velocity))
     if active:
         raise ValueError('MIDI contains unterminated notes')
-    wav = directory / 'refined.wav'
+    wav = directory / f'{stem}.wav'
     return json.dumps(dict(notes=notes, duration=sf.info(wav).duration,
                            audio='data:audio/wav;base64,' + base64.b64encode(wav.read_bytes()).decode()))
 

requirements.txt

@@ -10,3 +10,4 @@ pyfluidsynth==1.4.0
 mido==1.3.3
 matplotlib==3.10.8
 spaces
+librosa==0.11.0

speaking_midi/dense.py

@@ -0,0 +1,140 @@
+"""Dense convolutional piano fitting without text or an ASR model in the optimizer."""
+import argparse
+import hashlib
+import json
+import time
+from pathlib import Path
+import librosa
+import numpy as np
+import soundfile as sf
+import torch
+import torch.nn.functional as F
+from .core import Piano, Note, SR, SFONT, load_audio, render_saved, metrics
+
+
+class DenseFeatures:
+    def __init__(self, device, nfft=1024, power=2):
+        self.device, self.nfft, self.power = device, nfft, power
+        self.bank = torch.tensor(librosa.filters.mel(sr=SR, n_fft=nfft, n_mels=128,
+                                                    fmin=40, fmax=11000), device=device)
+        self.window = torch.hann_window(nfft, device=device)
+
+    def __call__(self, y):
+        x = torch.as_tensor(y, device=self.device, dtype=torch.float32)
+        spectrum = torch.stft(x, self.nfft, 240, window=self.window,
+                              pad_mode='constant', return_complex=True).abs()
+        return self.bank @ spectrum.pow(self.power)
+
+
+def decode_activations(activation, calibration, pitches, velocities, gate, grid_frames, seconds):
+    """Map per-pitch nonnegative energy coefficients to calibrated MIDI velocity."""
+    notes = []
+    for pi, pitch in enumerate(pitches):
+        curve = np.maximum.accumulate(calibration[pi])
+        for frame, amount in enumerate(activation[pi]):
+            onset = frame * grid_frames * .01
+            if onset + gate > seconds or amount < curve[2]:
+                continue
+            velocity = int(np.clip(np.rint(np.interp(amount, curve, velocities)), 1, 127))
+            notes.append(Note(pitch, velocity, onset, gate))
+    return notes
+
+
+def fit_dense(y, piano, *, cache='outputs/cache', device=None, nfft=1024,
+              power=2, gate=.02, grid_frames=2, velocity=48, floor=.001, steps=600):
+    """Fit dense note strengths and quantize them; callers must render saved MIDI.
+
+    No reference text, recognizer, pitch bend, source audio layer, or output EQ is
+    used. Templates approximate additive mel energy, not the exact synthesizer.
+    """
+    if (steps < 1 or grid_frames < 1 or gate <= 0 or gate > grid_frames*.01+1e-9
+            or not np.isfinite(floor) or floor <= 0 or power not in [1,2]
+            or not 1 <= velocity <= 127 or nfft < 2 or len(y) == 0
+            or not np.isfinite(y).all()):
+        raise ValueError('Invalid dense fitting configuration or audio')
+    started = time.monotonic()
+    device = device or ('cuda' if torch.cuda.is_available() else 'cpu')
+    feature = DenseFeatures(device, nfft, power)
+    pre, length = 4, 40
+    pitches = list(range(21,109))
+    velocities = [1,4,8,12,16,24,32,48,64,80,96,112,127]
+    config = dict(power=power, gate=gate, velocity=velocity, sr=SR, nfft=nfft, hop=240,
+                  pitches=pitches, soundfont_sha256=hashlib.sha256(Path(piano.path).read_bytes()).hexdigest(), revision=1)
+    key = hashlib.sha256(json.dumps(config,sort_keys=True).encode()).hexdigest()[:16]
+    path = Path(cache)/f'dense-{key}.npz'
+    path.parent.mkdir(parents=True,exist_ok=True)
+    cache_hit = path.exists()
+    if cache_hit:
+        with np.load(path) as data:
+            kernels, calibration = data['kernels'], data['calibration']
+    else:
+        kernels, calibration = [], []
+        for pitch in pitches:
+            base = piano.render([Note(pitch,velocity,pre*.01,gate)],length*.01)
+            kernels.append(feature(base).cpu().numpy()[:,:length])
+            levels = []
+            for value in velocities:
+                wave = piano.render([Note(pitch,value,pre*.01,gate)],length*.01)
+                levels.append(float(np.sum(wave*wave)))
+            calibration.append((np.array(levels)/max(float(np.sum(base*base)),1e-12))**(power/2))
+        kernels, calibration = np.array(kernels), np.array(calibration)
+        np.savez_compressed(path,kernels=kernels,calibration=calibration)
+    dictionary_seconds = time.monotonic()-started
+    kernel = torch.tensor(kernels,device=device)
+    target = feature(y)[None]
+    frames = target.shape[-1]
+    count = (frames+grid_frames-1)//grid_frames
+    activation = torch.full((1,len(pitches),count),.01,device=device,requires_grad=True)
+    optimizer = torch.optim.Adam([activation],lr=.04)
+    scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optimizer,steps,eta_min=.001)
+    target_log = torch.log1p(target/floor)
+    history = []
+    for iteration in range(steps):
+        optimizer.zero_grad()
+        estimate = F.conv_transpose1d(activation,kernel,stride=grid_frames)[:,:,pre:pre+frames]
+        loss = (torch.log1p(estimate.clamp_min(0)/floor)-target_log).square().mean()
+        loss.backward()
+        optimizer.step()
+        scheduler.step()
+        with torch.no_grad():
+            activation.clamp_(min=0,max=500)
+        if iteration%100 == 0 or iteration == steps-1:
+            history.append([iteration,float(loss.detach())])
+            print('dense',iteration,float(loss.detach()),flush=True)
+    values = activation.detach().cpu().numpy()[0]
+    notes = decode_activations(values,calibration,pitches,velocities,gate,grid_frames,len(y)/SR)
+    report = dict(config=config, optimization=dict(steps=steps,grid_frames=grid_frames,floor=floor,
+                  device=device,lr_schedule='cosine .04 to .001'), loss_history=history,
+                  dictionary=dict(cache_hit=cache_hit,seconds=dictionary_seconds,
+                                  sha256=hashlib.sha256(path.read_bytes()).hexdigest()),
+                  fitting_seconds=time.monotonic()-started)
+    return notes, report, values
+
+
+def main():
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument('input')
+    ap.add_argument('--out',type=Path,default=Path('outputs/dense-demo'))
+    ap.add_argument('--soundfont',default=SFONT)
+    ap.add_argument('--steps',type=int,default=600)
+    ap.add_argument('--cache',type=Path,default=Path('outputs/cache'))
+    ap.add_argument('--device',choices=['auto','cpu','cuda'],default='auto')
+    args = ap.parse_args()
+    if args.steps < 1:ap.error('steps must be positive')
+    args.out.mkdir(parents=True,exist_ok=True)
+    started = time.monotonic()
+    y = load_audio(args.input)
+    sf.write(args.out/'target.wav',y,SR,subtype='FLOAT')
+    piano = Piano(args.soundfont)
+    try:
+        notes, report, activation = fit_dense(y,piano,steps=args.steps,cache=args.cache,
+                                             device=None if args.device=='auto' else args.device)
+        audio = render_saved(piano,notes,args.out/'dense',len(y)/SR+.5)
+        np.savez_compressed(args.out/'activation.npz',activation=activation)
+        report.update(metrics=metrics(y,audio),notes=len(notes),wall_seconds=time.monotonic()-started)
+        (args.out/'report.json').write_text(json.dumps(report,indent=2))
+        print(json.dumps(report['metrics'],indent=2))
+    finally:piano.close()
+
+
+if __name__ == '__main__':main()

tests/test_app.py

@@ -17,26 +17,26 @@ class AppTests(unittest.TestCase):
             sf.write(src,np.zeros(16000),16000)
             with self.assertRaises(Exception): prepare_audio(src,out,1)
     def test_cpu_audio_pipeline(self):
-        events=list(run('Audio → MIDI','examples/en/source.wav','','English','',.5,20,False))
+        events=list(run('Audio → MIDI','examples/en/source.wav','','English','',.5,200))
         self.assertEqual(len(events),2)
         self.assertIn('Complete',events[-1][0])
         for path in (events[-1][1], events[-1][3]): self.assertTrue(Path(path).exists())
         self.assertGreater(len(json.loads(events[-1][2])['notes']), 0)
-        report=json.loads((Path(events[-1][3]).parent/'metrics.json').read_text())
-        self.assertEqual(report['pursuit']['device'],'cpu')
-        self.assertGreater(report['results']['refined']['notes'],0)
+        report=json.loads((Path(events[-1][3]).parent/'report.json').read_text())
+        self.assertEqual(report['optimization']['device'],'cpu')
+        self.assertGreater(report['notes'],0)
     def test_zerogpu_speech_dispatch_keeps_fitting_on_cpu(self):
         wave = .05 * np.sin(2*np.pi*220*np.arange(12000)/24000)
         with patch.object(app, 'DEVICE', 'zerogpu'), patch.object(
                 app, 'generate_zero_speech', return_value=(wave, 24000), create=True) as generate:
             events = list(run('Text → speech → MIDI', None, 'Hello.', 'English',
-                              app.VOICE, .5, 20, False))
+                              app.VOICE, .5, 200))
         generate.assert_called_once_with('Hello.', 'English', app.VOICE)
         output = Path(events[-1][3]).parent
         self.assertTrue((output/'speech.wav').exists())
         self.assertFalse((output/'tts.log').exists())
-        report = json.loads((output/'metrics.json').read_text())
-        self.assertEqual(report['pursuit']['device'], 'cpu')
+        report = json.loads((output/'report.json').read_text())
+        self.assertEqual(report['optimization']['device'], 'cpu')
 
     def test_cached_examples(self):
         for language in ['en','zh']:

tests/test_dense.py

@@ -0,0 +1,39 @@
+import tempfile
+import unittest
+from pathlib import Path
+import numpy as np
+from speaking_midi.core import Piano, Note, SR, write_midi, read_midi, metrics
+from speaking_midi.dense import decode_activations, fit_dense
+
+
+class DenseFitting(unittest.TestCase):
+    def test_calibration_and_nonoverlapping_midi_grid(self):
+        coefficients=np.array([[0.,.3,.6,.15],[.9,0.,0.,0.]])
+        calibration=np.array([[0.,.1,.2,.4,1.],[0.,.2,.4,.8,2.]])
+        notes=decode_activations(coefficients,calibration,[60,64],[1,4,8,16,32],.02,2,.08)
+        self.assertEqual([(n.pitch,n.velocity) for n in notes],[(60,12),(60,21),(64,17)])
+        with tempfile.TemporaryDirectory() as d:
+            path=Path(d)/'control.mid';write_midi(notes,path)
+            decoded=read_midi(path)
+            self.assertEqual(len(decoded),3)
+            for n in decoded:
+                self.assertGreaterEqual(n.start,0)
+                self.assertLessEqual(n.start+n.duration,.08+1e-9)
+
+    def test_single_piano_note_recovery_on_cpu(self):
+        piano=Piano()
+        try:
+            y=piano.render([Note(60,64,.06,.02)],.3)
+            notes,report,_=fit_dense(y,piano,device='cpu',steps=150)
+            self.assertEqual(report['optimization']['device'],'cpu')
+            self.assertLess(report['loss_history'][-1][1],report['loss_history'][0][1])
+            self.assertTrue(any(n.pitch==60 and abs(n.start-.06)<=.02 for n in notes))
+            rendered=piano.render(notes,.8)
+            self.assertLess(metrics(y,rendered)['log_band_mae'],metrics(y,np.zeros_like(y))['log_band_mae'])
+        finally:piano.close()
+
+    def test_reject_overlapping_gate(self):
+        with self.assertRaises(ValueError):
+            fit_dense(np.ones(SR,dtype=np.float32),None,gate=.04,grid_frames=2)
+
+if __name__=='__main__':unittest.main()