Spaces:
Running on Zero
Running on Zero
Commit ·
59e4843
1
Parent(s): 7a3d495
Replace diagnostic panels with an interactive MIDI piano roll
Browse files- README.md +8 -2
- VALIDATION.md +4 -0
- app.py +9 -34
- deploy.py +1 -1
- player.py +33 -0
- tests/test_app.py +6 -4
- ui/player.css +4 -0
- ui/player.html +13 -0
- ui/player.js +105 -0
README.md
CHANGED
|
@@ -14,10 +14,16 @@ models:
|
|
| 14 |
|
| 15 |
# Speaking MIDI
|
| 16 |
|
| 17 |
-
An interactive demonstration of **fixed-timbre inverse synthesis**: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then
|
| 18 |
|
| 19 |
English and Chinese examples are available instantly through the two example buttons. They contain about eight seconds of speech each and use the full refinement algorithm. New jobs default to a smaller search budget for responsive CPU experimentation. The technique approximates spectral structure; it does not guarantee intelligible words or perform conventional melody transcription.
|
| 20 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 21 |
## Poetry voice examples
|
| 22 |
|
| 23 |
Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s **Quiet Night Thoughts**, Meng Haoran’s **Spring Dawn**, Wang Zhihuan’s **On the Stork Tower**, the opening couplet of Shakespeare’s **Sonnet 18**, and the first stanza of Blake’s **The Tyger**. Select an example to load and listen to its recording, then press **Generate piano MIDI**. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in [the poetry examples](examples/poetry/README.md).
|
|
@@ -51,7 +57,7 @@ $$
|
|
| 51 |
|
| 52 |
Here $F$ is band-magnitude analysis, $y$ is the normalized target, $M$ is MIDI, and $S$ is the fixed piano renderer. The quick setting runs dictionary search and global velocity refinement; it skips the per-note sweep. The baseline rounds strong spectral peaks to piano pitches and separately calibrates global velocity.
|
| 53 |
|
| 54 |
-
|
| 55 |
|
| 56 |
The fitting core is vendored from [Speaking MIDI](https://github.com/k-l-lambda/speaking-midi/tree/932f8eb), with a portable SoundFont path and an optional quick-refinement flag. See that project for full experiments, derivations, and reproducible demo provenance.
|
| 57 |
|
|
|
|
| 14 |
|
| 15 |
# Speaking MIDI
|
| 16 |
|
| 17 |
+
An interactive demonstration of **fixed-timbre inverse synthesis**: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then listen to the source and play the resulting MIDI in an interactive piano roll. Download the actual MIDI from the same result.
|
| 18 |
|
| 19 |
English and Chinese examples are available instantly through the two example buttons. They contain about eight seconds of speech each and use the full refinement algorithm. New jobs default to a smaller search budget for responsive CPU experimentation. The technique approximates spectral structure; it does not guarantee intelligible words or perform conventional melody transcription.
|
| 20 |
|
| 21 |
+
## MIDI player
|
| 22 |
+
|
| 23 |
+
The demo centers on a piano roll inspired by the interaction design of `klstudio/app/views/midi-player.vue`: horizontal note blocks, active-note highlighting, a playback cursor, click-to-seek, and automatic scrolling at higher zoom. Controls provide play/pause, restart, a seek slider, playback speed, and zoom; left/right arrows on the roll seek by one second. Loading another result stops the previous playback.
|
| 24 |
+
|
| 25 |
+
The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; research diagnostics remain in the generated files.
|
| 26 |
+
|
| 27 |
## Poetry voice examples
|
| 28 |
|
| 29 |
Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s **Quiet Night Thoughts**, Meng Haoran’s **Spring Dawn**, Wang Zhihuan’s **On the Stork Tower**, the opening couplet of Shakespeare’s **Sonnet 18**, and the first stanza of Blake’s **The Tyger**. Select an example to load and listen to its recording, then press **Generate piano MIDI**. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in [the poetry examples](examples/poetry/README.md).
|
|
|
|
| 57 |
|
| 58 |
Here $F$ is band-magnitude analysis, $y$ is the normalized target, $M$ is MIDI, and $S$ is the fixed piano renderer. The quick setting runs dictionary search and global velocity refinement; it skips the per-note sweep. The baseline rounds strong spectral peaks to piano pitches and separately calibrates global velocity.
|
| 59 |
|
| 60 |
+
Saved diagnostic spectrograms use the same color scale for all four panels. Recorded metrics compare the input-length interval; audio includes a 0.5-second release tail. Log-band MAE is the optimization target, while spectral convergence is reported separately and need not improve. Neither metric measures speech intelligibility. Magnitude addition is only an approximation because real notes interact in phase, which motivates the renderer refinement step.
|
| 61 |
|
| 62 |
The fitting core is vendored from [Speaking MIDI](https://github.com/k-l-lambda/speaking-midi/tree/932f8eb), with a portable SoundFont path and an optional quick-refinement flag. See that project for full experiments, derivations, and reproducible demo provenance.
|
| 63 |
|
VALIDATION.md
CHANGED
|
@@ -20,3 +20,7 @@ Actual hosted API checks passed on 2026-10-01:
|
|
| 20 |
- Text → speech → MIDI: `Hello, piano.`, English, the requested female voice description, followed by fitting a 0.5-second prefix with 20 search iterations; completed in 8.6 seconds. Speech used the decorated ZeroGPU function; MIDI fitting used CPU.
|
| 21 |
|
| 22 |
These timings are the app's reported job durations, excluding client upload/download and queue transit. They validate short jobs, not the maximum 12-second/full-refinement configuration. The hosted PyTorch configuration error is resolved. No subscription or dedicated paid hardware was requested.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
- Text → speech → MIDI: `Hello, piano.`, English, the requested female voice description, followed by fitting a 0.5-second prefix with 20 search iterations; completed in 8.6 seconds. Speech used the decorated ZeroGPU function; MIDI fitting used CPU.
|
| 21 |
|
| 22 |
These timings are the app's reported job durations, excluding client upload/download and queue transit. They validate short jobs, not the maximum 12-second/full-refinement configuration. The hosted PyTorch configuration error is resolved. No subscription or dedicated paid hardware was requested.
|
| 23 |
+
|
| 24 |
+
## Interactive MIDI player (2026-10-02)
|
| 25 |
+
|
| 26 |
+
The demo now shows source audio and an interactive piano roll instead of diagnostic tabs. Seven backend tests passed, including real CPU fitting and the ZeroGPU dispatch test. Browser checks passed for playback and pause, click and keyboard seeking, zoom and scrolling, replacing a playing example, removal of diagnostic tabs, and mobile layout. No JavaScript errors were observed in those checks. The player decodes note events from the downloaded MIDI and synchronizes them to that MIDI's Grand Piano audio render.
|
app.py
CHANGED
|
@@ -13,6 +13,7 @@ os.environ.setdefault('MPLBACKEND', 'Agg')
|
|
| 13 |
import gradio as gr
|
| 14 |
import numpy as np
|
| 15 |
import soundfile as sf
|
|
|
|
| 16 |
|
| 17 |
ROOT = Path(__file__).resolve().parent
|
| 18 |
VOICE = '一位温柔自信的年轻女性,声音清晰,语气亲切,表达轻快而富有感染力。'
|
|
@@ -66,27 +67,9 @@ def prepare_audio(path, out, seconds):
|
|
| 66 |
return len(audio) / rate
|
| 67 |
|
| 68 |
|
| 69 |
-
def piano_roll(directory):
|
| 70 |
-
import matplotlib.pyplot as plt
|
| 71 |
-
from matplotlib.patches import Rectangle
|
| 72 |
-
notes = json.loads((directory / 'refined.json').read_text())
|
| 73 |
-
fig, ax = plt.subplots(figsize=(11, 3))
|
| 74 |
-
for note in notes:
|
| 75 |
-
ax.add_patch(Rectangle((note['start'], note['pitch']-.4), note['duration'], .8,
|
| 76 |
-
color=plt.cm.viridis(note['velocity']/127)))
|
| 77 |
-
duration = sf.info(directory / 'target.wav').duration
|
| 78 |
-
ax.set(xlim=(0, duration), ylim=(35,109), xlabel='Seconds', ylabel='MIDI pitch',
|
| 79 |
-
title=f'{len(notes)} piano notes · color indicates velocity')
|
| 80 |
-
ax.grid(alpha=.15); fig.tight_layout()
|
| 81 |
-
path = directory / 'piano-roll.png'; fig.savefig(path, dpi=130); plt.close(fig)
|
| 82 |
-
return str(path)
|
| 83 |
-
|
| 84 |
-
|
| 85 |
def result(directory, status):
|
| 86 |
-
|
| 87 |
-
|
| 88 |
-
str(directory/'refined.wav'), str(directory/'spectrogram.png'),
|
| 89 |
-
(str(directory/'piano-roll.png') if (directory/'piano-roll.png').exists() else piano_roll(directory)), str(directory/'refined.mid'), metrics['results'])
|
| 90 |
|
| 91 |
|
| 92 |
def run(mode, audio, text, language, description, seconds, budget, refine, progress=gr.Progress()):
|
|
@@ -98,7 +81,7 @@ def run(mode, audio, text, language, description, seconds, budget, refine, progr
|
|
| 98 |
if old.is_dir() and time.time() - old.stat().st_mtime > 86400:
|
| 99 |
shutil.rmtree(old)
|
| 100 |
directory = Path(tempfile.mkdtemp(prefix='job-', dir=RUNS))
|
| 101 |
-
empty = (None,
|
| 102 |
started = time.monotonic()
|
| 103 |
if mode == 'Text → speech → MIDI':
|
| 104 |
if not text.strip() or len(text) > 300 or not description.strip() or len(description) > 600:
|
|
@@ -119,14 +102,14 @@ def run(mode, audio, text, language, description, seconds, budget, refine, progr
|
|
| 119 |
audio = str(directory/'speech.wav')
|
| 120 |
duration = prepare_audio(audio, directory/'input.wav', seconds)
|
| 121 |
yield (f'2/4 · Analyzing the first {duration:.2f} s and building piano templates.',
|
| 122 |
-
str(directory/'input.wav'),
|
| 123 |
progress(.3, desc='Fitting piano note templates; then checking the real synthesizer')
|
| 124 |
command = ['-m', 'speaking_midi', str(directory/'input.wav'), '--out', str(directory),
|
| 125 |
'--soundfont', str(ROOT/'assets/gm.sf3'), '--max-notes', str(int(budget))]
|
| 126 |
if not refine:
|
| 127 |
command.append('--skip-local-refinement')
|
| 128 |
worker(command, directory/'fit.log')
|
| 129 |
-
progress(.95, desc='
|
| 130 |
yield result(directory, f'4/4 · Complete in {time.monotonic()-started:.1f} s on {DEVICE}. '
|
| 131 |
+ ('Full renderer refinement.' if refine else 'Quick fit with global velocity refinement.'))
|
| 132 |
|
|
@@ -137,7 +120,7 @@ def example(language):
|
|
| 137 |
|
| 138 |
def build_ui():
|
| 139 |
with gr.Blocks(title='Speaking MIDI') as demo:
|
| 140 |
-
gr.Markdown('# Speaking MIDI\nTurn a voice into piano notes.
|
| 141 |
'This is experimental sound approximation: words may become difficult to understand.')
|
| 142 |
with gr.Row():
|
| 143 |
english = gr.Button('Listen to English example')
|
|
@@ -160,8 +143,7 @@ def build_ui():
|
|
| 160 |
with gr.Column(scale=2):
|
| 161 |
status = gr.Markdown('Choose an example or create a new fit.')
|
| 162 |
source = gr.Audio(label='1 · Source used for fitting', interactive=False)
|
| 163 |
-
|
| 164 |
-
piano = gr.Audio(label='3 · Optimized piano MIDI rendered with Grand Piano', interactive=False)
|
| 165 |
midi = gr.File(label='Download MIDI', interactive=False)
|
| 166 |
poetry = json.loads((ROOT/'examples/poetry/prompts.json').read_text())
|
| 167 |
gr.Examples(
|
|
@@ -173,18 +155,11 @@ def build_ui():
|
|
| 173 |
)
|
| 174 |
selected_text = gr.Textbox(label='Selected text', interactive=False, lines=2)
|
| 175 |
text.change(lambda value: value, text, selected_text, queue=False)
|
| 176 |
-
with gr.Tabs():
|
| 177 |
-
with gr.Tab('Spectrogram comparison'):
|
| 178 |
-
spectrum = gr.Image(label='Shared scale · source / baseline / dictionary / optimized', interactive=False)
|
| 179 |
-
with gr.Tab('MIDI piano roll'):
|
| 180 |
-
roll = gr.Image(label='Note timing, pitch and velocity', interactive=False)
|
| 181 |
-
with gr.Tab('Measurements'):
|
| 182 |
-
measurements = gr.JSON(label='Lower log-band MAE is better; this does not measure intelligibility')
|
| 183 |
gr.Markdown('**How it works:** synthesize 657 piano templates → match weighted spectral bands → '
|
| 184 |
'choose discrete notes → re-render MIDI → reduce measured spectral error. '
|
| 185 |
'[Algorithm and experiments](https://github.com/k-l-lambda/speaking-midi) · '
|
| 186 |
'[Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign)')
|
| 187 |
-
outputs = [status, source,
|
| 188 |
mode.change(lambda value: (gr.update(visible=value=='Audio → MIDI'), gr.update(visible=value!='Audio → MIDI')),
|
| 189 |
mode, [audio, speech_controls], queue=False)
|
| 190 |
submit.click(run, [mode,audio,text,language,description,seconds,budget,refine], outputs,
|
|
|
|
| 13 |
import gradio as gr
|
| 14 |
import numpy as np
|
| 15 |
import soundfile as sf
|
| 16 |
+
from player import player_component, player_payload
|
| 17 |
|
| 18 |
ROOT = Path(__file__).resolve().parent
|
| 19 |
VOICE = '一位温柔自信的年轻女性,声音清晰,语气亲切,表达轻快而富有感染力。'
|
|
|
|
| 67 |
return len(audio) / rate
|
| 68 |
|
| 69 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 70 |
def result(directory, status):
|
| 71 |
+
return (status, str(directory/'target.wav'), player_payload(directory),
|
| 72 |
+
str(directory/'refined.mid'))
|
|
|
|
|
|
|
| 73 |
|
| 74 |
|
| 75 |
def run(mode, audio, text, language, description, seconds, budget, refine, progress=gr.Progress()):
|
|
|
|
| 81 |
if old.is_dir() and time.time() - old.stat().st_mtime > 86400:
|
| 82 |
shutil.rmtree(old)
|
| 83 |
directory = Path(tempfile.mkdtemp(prefix='job-', dir=RUNS))
|
| 84 |
+
empty = (None, '', None)
|
| 85 |
started = time.monotonic()
|
| 86 |
if mode == 'Text → speech → MIDI':
|
| 87 |
if not text.strip() or len(text) > 300 or not description.strip() or len(description) > 600:
|
|
|
|
| 102 |
audio = str(directory/'speech.wav')
|
| 103 |
duration = prepare_audio(audio, directory/'input.wav', seconds)
|
| 104 |
yield (f'2/4 · Analyzing the first {duration:.2f} s and building piano templates.',
|
| 105 |
+
str(directory/'input.wav'), '', None)
|
| 106 |
progress(.3, desc='Fitting piano note templates; then checking the real synthesizer')
|
| 107 |
command = ['-m', 'speaking_midi', str(directory/'input.wav'), '--out', str(directory),
|
| 108 |
'--soundfont', str(ROOT/'assets/gm.sf3'), '--max-notes', str(int(budget))]
|
| 109 |
if not refine:
|
| 110 |
command.append('--skip-local-refinement')
|
| 111 |
worker(command, directory/'fit.log')
|
| 112 |
+
progress(.95, desc='Preparing the MIDI player')
|
| 113 |
yield result(directory, f'4/4 · Complete in {time.monotonic()-started:.1f} s on {DEVICE}. '
|
| 114 |
+ ('Full renderer refinement.' if refine else 'Quick fit with global velocity refinement.'))
|
| 115 |
|
|
|
|
| 120 |
|
| 121 |
def build_ui():
|
| 122 |
with gr.Blocks(title='Speaking MIDI') as demo:
|
| 123 |
+
gr.Markdown('# Speaking MIDI\nTurn a voice into piano notes. Listen to your voice on piano, follow the notes, and download the MIDI.\n'
|
| 124 |
'This is experimental sound approximation: words may become difficult to understand.')
|
| 125 |
with gr.Row():
|
| 126 |
english = gr.Button('Listen to English example')
|
|
|
|
| 143 |
with gr.Column(scale=2):
|
| 144 |
status = gr.Markdown('Choose an example or create a new fit.')
|
| 145 |
source = gr.Audio(label='1 · Source used for fitting', interactive=False)
|
| 146 |
+
player = player_component(gr)
|
|
|
|
| 147 |
midi = gr.File(label='Download MIDI', interactive=False)
|
| 148 |
poetry = json.loads((ROOT/'examples/poetry/prompts.json').read_text())
|
| 149 |
gr.Examples(
|
|
|
|
| 155 |
)
|
| 156 |
selected_text = gr.Textbox(label='Selected text', interactive=False, lines=2)
|
| 157 |
text.change(lambda value: value, text, selected_text, queue=False)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 158 |
gr.Markdown('**How it works:** synthesize 657 piano templates → match weighted spectral bands → '
|
| 159 |
'choose discrete notes → re-render MIDI → reduce measured spectral error. '
|
| 160 |
'[Algorithm and experiments](https://github.com/k-l-lambda/speaking-midi) · '
|
| 161 |
'[Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign)')
|
| 162 |
+
outputs = [status, source, player, midi]
|
| 163 |
mode.change(lambda value: (gr.update(visible=value=='Audio → MIDI'), gr.update(visible=value!='Audio → MIDI')),
|
| 164 |
mode, [audio, speech_controls], queue=False)
|
| 165 |
submit.click(run, [mode,audio,text,language,description,seconds,budget,refine], outputs,
|
deploy.py
CHANGED
|
@@ -8,7 +8,7 @@ if __name__ == '__main__':
|
|
| 8 |
api.create_repo(repo, repo_type='space', space_sdk='gradio', space_hardware='zero-a10g', exist_ok=True)
|
| 9 |
api.add_space_variable(repo, 'INFERENCE_DEVICE', 'zerogpu')
|
| 10 |
commit = api.upload_folder(repo_id=repo, repo_type='space', folder_path=Path(__file__).parent,
|
| 11 |
-
allow_patterns=['app.py', 'zero_tts.py', 'deploy.py', 'README.md', 'VALIDATION.md', 'requirements.txt', 'packages.txt',
|
| 12 |
'.gitignore', 'speaking_midi/*.py', 'scripts/*.py', 'tests/*.py',
|
| 13 |
'assets/*', 'examples/**'],
|
| 14 |
commit_message='Build CPU-capable Speaking MIDI audio and VoiceDesign demo')
|
|
|
|
| 8 |
api.create_repo(repo, repo_type='space', space_sdk='gradio', space_hardware='zero-a10g', exist_ok=True)
|
| 9 |
api.add_space_variable(repo, 'INFERENCE_DEVICE', 'zerogpu')
|
| 10 |
commit = api.upload_folder(repo_id=repo, repo_type='space', folder_path=Path(__file__).parent,
|
| 11 |
+
allow_patterns=['app.py', 'player.py', 'ui/*', 'zero_tts.py', 'deploy.py', 'README.md', 'VALIDATION.md', 'requirements.txt', 'packages.txt',
|
| 12 |
'.gitignore', 'speaking_midi/*.py', 'scripts/*.py', 'tests/*.py',
|
| 13 |
'assets/*', 'examples/**'],
|
| 14 |
commit_message='Build CPU-capable Speaking MIDI audio and VoiceDesign demo')
|
player.py
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Build a browser player payload from the exported MIDI and its exact piano render."""
|
| 2 |
+
import base64
|
| 3 |
+
import json
|
| 4 |
+
from pathlib import Path
|
| 5 |
+
import mido
|
| 6 |
+
import soundfile as sf
|
| 7 |
+
|
| 8 |
+
UI = Path(__file__).parent / 'ui'
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
def player_payload(directory):
|
| 12 |
+
directory = Path(directory)
|
| 13 |
+
clock = 0.0
|
| 14 |
+
active = {}
|
| 15 |
+
notes = []
|
| 16 |
+
for message in mido.MidiFile(directory / 'refined.mid'):
|
| 17 |
+
clock += message.time
|
| 18 |
+
if message.type == 'note_on' and message.velocity:
|
| 19 |
+
active[(message.channel, message.note)] = (clock, message.velocity)
|
| 20 |
+
elif message.type == 'note_off' or (message.type == 'note_on' and not message.velocity):
|
| 21 |
+
start, velocity = active.pop((message.channel, message.note))
|
| 22 |
+
notes.append(dict(pitch=message.note, start=start, duration=clock-start, velocity=velocity))
|
| 23 |
+
if active:
|
| 24 |
+
raise ValueError('MIDI contains unterminated notes')
|
| 25 |
+
wav = directory / 'refined.wav'
|
| 26 |
+
return json.dumps(dict(notes=notes, duration=sf.info(wav).duration,
|
| 27 |
+
audio='data:audio/wav;base64,' + base64.b64encode(wav.read_bytes()).decode()))
|
| 28 |
+
|
| 29 |
+
|
| 30 |
+
def player_component(gr):
|
| 31 |
+
return gr.HTML(value='', html_template=(UI/'player.html').read_text(),
|
| 32 |
+
css_template=(UI/'player.css').read_text(),
|
| 33 |
+
js_on_load=(UI/'player.js').read_text(), elem_id='midi-player')
|
tests/test_app.py
CHANGED
|
@@ -20,8 +20,9 @@ class AppTests(unittest.TestCase):
|
|
| 20 |
events=list(run('Audio → MIDI','examples/en/source.wav','','English','',.5,20,False))
|
| 21 |
self.assertEqual(len(events),2)
|
| 22 |
self.assertIn('Complete',events[-1][0])
|
| 23 |
-
for path in events[-1][1
|
| 24 |
-
|
|
|
|
| 25 |
self.assertEqual(report['pursuit']['device'],'cpu')
|
| 26 |
self.assertGreater(report['results']['refined']['notes'],0)
|
| 27 |
def test_zerogpu_speech_dispatch_keeps_fitting_on_cpu(self):
|
|
@@ -31,7 +32,7 @@ class AppTests(unittest.TestCase):
|
|
| 31 |
events = list(run('Text → speech → MIDI', None, 'Hello.', 'English',
|
| 32 |
app.VOICE, .5, 20, False))
|
| 33 |
generate.assert_called_once_with('Hello.', 'English', app.VOICE)
|
| 34 |
-
output = Path(events[-1][
|
| 35 |
self.assertTrue((output/'speech.wav').exists())
|
| 36 |
self.assertFalse((output/'tts.log').exists())
|
| 37 |
report = json.loads((output/'metrics.json').read_text())
|
|
@@ -40,6 +41,7 @@ class AppTests(unittest.TestCase):
|
|
| 40 |
def test_cached_examples(self):
|
| 41 |
for language in ['en','zh']:
|
| 42 |
output=example(language)
|
| 43 |
-
for path in output[1
|
|
|
|
| 44 |
|
| 45 |
if __name__=='__main__': unittest.main()
|
|
|
|
| 20 |
events=list(run('Audio → MIDI','examples/en/source.wav','','English','',.5,20,False))
|
| 21 |
self.assertEqual(len(events),2)
|
| 22 |
self.assertIn('Complete',events[-1][0])
|
| 23 |
+
for path in (events[-1][1], events[-1][3]): self.assertTrue(Path(path).exists())
|
| 24 |
+
self.assertGreater(len(json.loads(events[-1][2])['notes']), 0)
|
| 25 |
+
report=json.loads((Path(events[-1][3]).parent/'metrics.json').read_text())
|
| 26 |
self.assertEqual(report['pursuit']['device'],'cpu')
|
| 27 |
self.assertGreater(report['results']['refined']['notes'],0)
|
| 28 |
def test_zerogpu_speech_dispatch_keeps_fitting_on_cpu(self):
|
|
|
|
| 32 |
events = list(run('Text → speech → MIDI', None, 'Hello.', 'English',
|
| 33 |
app.VOICE, .5, 20, False))
|
| 34 |
generate.assert_called_once_with('Hello.', 'English', app.VOICE)
|
| 35 |
+
output = Path(events[-1][3]).parent
|
| 36 |
self.assertTrue((output/'speech.wav').exists())
|
| 37 |
self.assertFalse((output/'tts.log').exists())
|
| 38 |
report = json.loads((output/'metrics.json').read_text())
|
|
|
|
| 41 |
def test_cached_examples(self):
|
| 42 |
for language in ['en','zh']:
|
| 43 |
output=example(language)
|
| 44 |
+
for path in (output[1], output[3]): self.assertTrue(Path(path).exists())
|
| 45 |
+
self.assertTrue(json.loads(output[2])['audio'].startswith('data:audio/wav;base64,'))
|
| 46 |
|
| 47 |
if __name__=='__main__': unittest.main()
|
ui/player.css
ADDED
|
@@ -0,0 +1,4 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
.player{font:14px system-ui,sans-serif;color:#dce7f7;background:#101b2d;border:1px solid #253651;border-radius:14px;overflow:hidden;min-width:0}
|
| 2 |
+
.player header{display:flex;justify-content:space-between;gap:12px;align-items:center;padding:16px 18px 10px}.player strong{font-size:17px}.subtitle{color:#91a5c2;margin-left:12px;font-size:12px}.note-count{color:#91a5c2;font-size:12px}
|
| 3 |
+
.transport{display:flex;align-items:center;gap:10px;flex-wrap:wrap;padding:4px 18px}.player button,.player select{border:1px solid #3b506c;border-radius:7px;background:#24364f;color:#eef5ff;padding:7px 11px;cursor:pointer;font:inherit}.player button:disabled{opacity:.4;cursor:default}.player .play{background:#6ce2b6;color:#102d28;border-color:transparent;min-width:82px;font-weight:650}.clock{font-variant-numeric:tabular-nums;font-size:12px;min-width:90px}.speed-label{margin-left:auto;font-size:12px;color:#b3c5dd}.player select{padding:5px;font-size:12px}.seek{display:block;width:calc(100% - 36px);margin:10px 18px;accent-color:#6ce2b6}.roll-wrap{position:relative;height:300px}.roll{display:block;width:100%;height:300px;touch-action:pan-y;cursor:crosshair}.empty{position:absolute;top:42%;left:60px;right:12px;text-align:center;color:#a5b7ce;pointer-events:none}.player footer{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:10px 18px;color:#91a5c2;font-size:11px}.player footer label{display:flex;align-items:center;gap:8px}.zoom{width:85px;accent-color:#6ce2b6}.error{margin:0;padding:0 18px 10px;color:#ffb3a9;font-size:12px}.error:empty{display:none}.player :focus-visible{outline:2px solid #84bfff;outline-offset:2px}@media(max-width:480px){.player header,.transport{padding-left:10px;padding-right:10px}.subtitle{display:block;margin:3px 0 0}.player footer{flex-wrap:wrap}.transport{gap:7px}.speed-label{margin-left:0}}
|
| 4 |
+
.player strong{color:#e7f0ff!important}.player .clock{color:#dce7f7!important}.player footer span,.player footer label{color:#a9bed8!important}.player .empty{color:#a5b7ce!important}
|
ui/player.html
ADDED
|
@@ -0,0 +1,13 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
<section class="player" aria-label="MIDI piano roll player">
|
| 2 |
+
<header><div><strong>MIDI piano roll</strong><span class="subtitle">Grand Piano</span></div><span class="note-count">No MIDI loaded</span></header>
|
| 3 |
+
<div class="transport">
|
| 4 |
+
<button class="play" disabled aria-label="Play piano MIDI">▶ Play</button>
|
| 5 |
+
<button class="restart" disabled aria-label="Restart playback">↺</button>
|
| 6 |
+
<output class="clock" aria-label="Playback time">0:00 / 0:00</output>
|
| 7 |
+
<label class="speed-label">Speed <select class="speed" aria-label="Playback speed"><option value="0.5">0.5×</option><option value="0.75">0.75×</option><option value="1" selected>1×</option><option value="1.25">1.25×</option><option value="1.5">1.5×</option></select></label>
|
| 8 |
+
</div>
|
| 9 |
+
<input class="seek" type="range" min="0" max="1" value="0" step="0.01" disabled aria-label="Playback position">
|
| 10 |
+
<div class="roll-wrap"><canvas class="roll" tabindex="0" role="img" aria-label="MIDI notes by time and pitch. Click to seek; use left and right arrows to move one second."></canvas><p class="empty">Choose an example or generate piano MIDI to play.</p></div>
|
| 11 |
+
<footer><span>Click the roll to seek · lit notes are playing</span><label>Zoom <input class="zoom" type="range" min="1" max="4" step="0.25" value="1" aria-label="Piano roll zoom"></label></footer>
|
| 12 |
+
<p class="error" role="status"></p>
|
| 13 |
+
</section>
|
ui/player.js
ADDED
|
@@ -0,0 +1,105 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
// Interaction reference: klstudio/app/views/midi-player.vue.
|
| 2 |
+
// The renderer's WAV is the playback clock; notes come from the exported MIDI.
|
| 3 |
+
let dispose = () => {};
|
| 4 |
+
function mount() {
|
| 5 |
+
dispose();
|
| 6 |
+
const root = element.querySelector('.player');
|
| 7 |
+
if (!root) return;
|
| 8 |
+
const $ = (s) => root.querySelector(s);
|
| 9 |
+
const canvas = $('.roll'), ctx = canvas.getContext('2d');
|
| 10 |
+
const play = $('.play'), seek = $('.seek'), clock = $('.clock');
|
| 11 |
+
const audio = new Audio();
|
| 12 |
+
const listeners = new AbortController();
|
| 13 |
+
let frame = 0, stopped = false, data = null, offset = 0, zoom = 1;
|
| 14 |
+
let width = 600, height = 300, low = 36, high = 84;
|
| 15 |
+
try { data = props.value ? JSON.parse(props.value) : null; } catch (_) {}
|
| 16 |
+
const duration = data ? data.duration : 8;
|
| 17 |
+
const notes = data ? data.notes : [];
|
| 18 |
+
const left = 46, top = 23, bottom = 10;
|
| 19 |
+
const clamp = (n, a, b) => Math.min(b, Math.max(a, n));
|
| 20 |
+
const format = (n) => `${Math.floor(n / 60)}:${Math.floor(n % 60).toString().padStart(2, '0')}`;
|
| 21 |
+
const on = (target, type, fn) => target.addEventListener(type, fn, {signal: listeners.signal});
|
| 22 |
+
const span = () => Math.max(duration / zoom, .1);
|
| 23 |
+
function follow() {
|
| 24 |
+
const time = audio.currentTime;
|
| 25 |
+
if (time < offset + span() * .15 || time > offset + span() * .7)
|
| 26 |
+
offset = clamp(time - span() * .4, 0, Math.max(0, duration - span()));
|
| 27 |
+
}
|
| 28 |
+
function draw() {
|
| 29 |
+
if (stopped) return;
|
| 30 |
+
const time = audio.currentTime;
|
| 31 |
+
const scale = (width-left)/span(), row = (height-top-bottom)/(high-low+1);
|
| 32 |
+
ctx.clearRect(0,0,width,height);
|
| 33 |
+
ctx.fillStyle='#101b2d';ctx.fillRect(0,0,width,height);
|
| 34 |
+
const playing = new Set(notes.filter(n=>!audio.paused && n.start<=time && n.start+n.duration>time).map(n=>n.pitch));
|
| 35 |
+
for (let pitch=low;pitch<=high;pitch++) {
|
| 36 |
+
const y=top+(high-pitch)*row, black=[1,3,6,8,10].includes(pitch%12);
|
| 37 |
+
ctx.fillStyle=black?'#0c1626':'#142136';ctx.fillRect(left,y,width-left,row);
|
| 38 |
+
ctx.fillStyle=playing.has(pitch)?'#6ce2b6':black?'#23334a':'#e1e8f2';ctx.fillRect(0,y,left-2,Math.max(.5,row-1));
|
| 39 |
+
if (pitch%12===0) {
|
| 40 |
+
ctx.fillStyle=playing.has(pitch)?'#102d28':'#4f627a';ctx.font='10px system-ui';ctx.fillText(`C${Math.floor(pitch/12)-1}`,7,y+Math.max(8,row*.8));
|
| 41 |
+
ctx.strokeStyle='#33425b';ctx.beginPath();ctx.moveTo(left,y+row);ctx.lineTo(width,y+row);ctx.stroke();
|
| 42 |
+
}
|
| 43 |
+
}
|
| 44 |
+
const step=span()>6?1:.5;
|
| 45 |
+
for(let t=Math.ceil(offset/step)*step;t<=offset+span();t+=step) {
|
| 46 |
+
const x=left+(t-offset)*scale;
|
| 47 |
+
ctx.strokeStyle='#26364d';ctx.beginPath();ctx.moveTo(x,top);ctx.lineTo(x,height-bottom);ctx.stroke();
|
| 48 |
+
ctx.fillStyle='#9bb0ce';ctx.font='10px system-ui';ctx.fillText(`${t.toFixed(step===1?0:1)}s`,x+3,14);
|
| 49 |
+
}
|
| 50 |
+
ctx.save();ctx.beginPath();ctx.rect(left,top,width-left,height-top);ctx.clip();
|
| 51 |
+
for(const n of notes) {
|
| 52 |
+
const x=left+(n.start-offset)*scale,y=top+(high-n.pitch)*row;
|
| 53 |
+
const active=!audio.paused && n.start<=time && n.start+n.duration>time;
|
| 54 |
+
ctx.fillStyle=active?'#d2ffe6':`hsl(${200-n.velocity*.35} 70% ${42+n.velocity*.15}%)`;
|
| 55 |
+
ctx.fillRect(x,y+Math.min(1,row*.1),Math.max(2,n.duration*scale),Math.max(1,row-1));
|
| 56 |
+
if(active){ctx.strokeStyle='#6ce2b6';ctx.strokeRect(x,y,Math.max(2,n.duration*scale),row);}
|
| 57 |
+
}
|
| 58 |
+
const cursor=left+(time-offset)*scale;
|
| 59 |
+
ctx.strokeStyle='#f8bc68';ctx.lineWidth=2;ctx.beginPath();ctx.moveTo(cursor,top);ctx.lineTo(cursor,height);ctx.stroke();ctx.lineWidth=1;ctx.restore();
|
| 60 |
+
canvas.dataset.time=String(time);canvas.dataset.offset=String(offset);canvas.dataset.active=String(playing.size);
|
| 61 |
+
clock.textContent=`${format(time)} / ${format(duration)}`;
|
| 62 |
+
seek.value=String(time);
|
| 63 |
+
}
|
| 64 |
+
function animate() {
|
| 65 |
+
if(stopped || !element.isConnected) { dispose();return; }
|
| 66 |
+
follow();draw();
|
| 67 |
+
if(!audio.paused) frame=requestAnimationFrame(animate);
|
| 68 |
+
}
|
| 69 |
+
function seekTo(time) {if(!data)return;audio.currentTime=clamp(time,0,duration);follow();draw();}
|
| 70 |
+
function resize() {
|
| 71 |
+
width=Math.max(180,canvas.clientWidth);const dpr=window.devicePixelRatio||1;
|
| 72 |
+
canvas.width=width*dpr;canvas.height=height*dpr;ctx.setTransform(dpr,0,0,dpr,0,0);draw();
|
| 73 |
+
}
|
| 74 |
+
play.textContent='▶ Play';play.setAttribute('aria-label','Play piano MIDI');
|
| 75 |
+
play.disabled=true;$('.restart').disabled=true;seek.disabled=true;seek.value='0';
|
| 76 |
+
$('.empty').hidden=false;$('.note-count').textContent='No MIDI loaded';
|
| 77 |
+
$('.speed').value='1';$('.zoom').value='1';$('.error').textContent='';
|
| 78 |
+
if(data) {
|
| 79 |
+
if(notes.length){low=Math.max(0,Math.min(...notes.map(n=>n.pitch))-2);high=Math.min(127,Math.max(...notes.map(n=>n.pitch))+2);}
|
| 80 |
+
audio.src=data.audio;audio.preload='auto';seek.max=String(duration);
|
| 81 |
+
$('.empty').hidden=true;play.disabled=false;$('.restart').disabled=false;seek.disabled=false;
|
| 82 |
+
$('.note-count').textContent=`${notes.length} notes`;
|
| 83 |
+
}
|
| 84 |
+
on(play,'click',async()=>{
|
| 85 |
+
if(!data)return;
|
| 86 |
+
if(audio.paused) {
|
| 87 |
+
if(audio.ended || audio.currentTime>=duration)seekTo(0);
|
| 88 |
+
try{await audio.play();$('.error').textContent='';}catch(_){$('.error').textContent='Playback could not start. Please press Play again.';}
|
| 89 |
+
} else audio.pause();
|
| 90 |
+
});
|
| 91 |
+
on(audio,'play',()=>{play.textContent='Ⅱ Pause';play.setAttribute('aria-label','Pause piano MIDI');cancelAnimationFrame(frame);animate();});
|
| 92 |
+
on(audio,'pause',()=>{play.textContent='▶ Play';play.setAttribute('aria-label','Play piano MIDI');cancelAnimationFrame(frame);draw();});
|
| 93 |
+
on(audio,'ended',()=>{cancelAnimationFrame(frame);draw();});
|
| 94 |
+
on(audio,'error',()=>{$('.error').textContent='Unable to load the piano audio.';});
|
| 95 |
+
on($('.restart'),'click',()=>seekTo(0));
|
| 96 |
+
on(seek,'input',()=>seekTo(Number(seek.value)));
|
| 97 |
+
on($('.speed'),'change',(e)=>{audio.playbackRate=Number(e.target.value);});
|
| 98 |
+
on($('.zoom'),'input',(e)=>{zoom=Number(e.target.value);offset=clamp(audio.currentTime-span()*.4,0,Math.max(0,duration-span()));draw();});
|
| 99 |
+
on(canvas,'click',(e)=>{const x=e.clientX-canvas.getBoundingClientRect().left;if(x>=left)seekTo(offset+(x-left)/(width-left)*span());});
|
| 100 |
+
on(canvas,'keydown',(e)=>{if(['ArrowLeft','ArrowRight'].includes(e.key)){e.preventDefault();seekTo(audio.currentTime+(e.key==='ArrowRight'?1:-1));}});
|
| 101 |
+
const observer=new ResizeObserver(resize);observer.observe(canvas);resize();
|
| 102 |
+
dispose=()=>{if(stopped)return;stopped=true;listeners.abort();cancelAnimationFrame(frame);observer.disconnect();audio.pause();audio.removeAttribute('src');audio.load();};
|
| 103 |
+
}
|
| 104 |
+
watch('value', mount);
|
| 105 |
+
mount();
|