k-l-lambda commited on
Commit
59e4843
·
1 Parent(s): 7a3d495

Replace diagnostic panels with an interactive MIDI piano roll

Browse files
Files changed (9) hide show
  1. README.md +8 -2
  2. VALIDATION.md +4 -0
  3. app.py +9 -34
  4. deploy.py +1 -1
  5. player.py +33 -0
  6. tests/test_app.py +6 -4
  7. ui/player.css +4 -0
  8. ui/player.html +13 -0
  9. ui/player.js +105 -0
README.md CHANGED
@@ -14,10 +14,16 @@ models:
14
 
15
  # Speaking MIDI
16
 
17
- An interactive demonstration of **fixed-timbre inverse synthesis**: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then compare the source, a spectral-peak baseline, and optimized piano playback. Download the actual MIDI and inspect aligned spectrograms and a velocity-colored piano roll.
18
 
19
  English and Chinese examples are available instantly through the two example buttons. They contain about eight seconds of speech each and use the full refinement algorithm. New jobs default to a smaller search budget for responsive CPU experimentation. The technique approximates spectral structure; it does not guarantee intelligible words or perform conventional melody transcription.
20
 
 
 
 
 
 
 
21
  ## Poetry voice examples
22
 
23
  Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s **Quiet Night Thoughts**, Meng Haoran’s **Spring Dawn**, Wang Zhihuan’s **On the Stork Tower**, the opening couplet of Shakespeare’s **Sonnet 18**, and the first stanza of Blake’s **The Tyger**. Select an example to load and listen to its recording, then press **Generate piano MIDI**. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in [the poetry examples](examples/poetry/README.md).
@@ -51,7 +57,7 @@ $$
51
 
52
  Here $F$ is band-magnitude analysis, $y$ is the normalized target, $M$ is MIDI, and $S$ is the fixed piano renderer. The quick setting runs dictionary search and global velocity refinement; it skips the per-note sweep. The baseline rounds strong spectral peaks to piano pitches and separately calibrates global velocity.
53
 
54
- All four spectrogram panels use the same color scale. Metrics compare the input-length interval; audio includes a 0.5-second release tail. Log-band MAE is the optimization target, while spectral convergence is reported separately and need not improve. Neither metric measures speech intelligibility. Magnitude addition is only an approximation because real notes interact in phase, which motivates the renderer refinement step.
55
 
56
  The fitting core is vendored from [Speaking MIDI](https://github.com/k-l-lambda/speaking-midi/tree/932f8eb), with a portable SoundFont path and an optional quick-refinement flag. See that project for full experiments, derivations, and reproducible demo provenance.
57
 
 
14
 
15
  # Speaking MIDI
16
 
17
+ An interactive demonstration of **fixed-timbre inverse synthesis**: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then listen to the source and play the resulting MIDI in an interactive piano roll. Download the actual MIDI from the same result.
18
 
19
  English and Chinese examples are available instantly through the two example buttons. They contain about eight seconds of speech each and use the full refinement algorithm. New jobs default to a smaller search budget for responsive CPU experimentation. The technique approximates spectral structure; it does not guarantee intelligible words or perform conventional melody transcription.
20
 
21
+ ## MIDI player
22
+
23
+ The demo centers on a piano roll inspired by the interaction design of `klstudio/app/views/midi-player.vue`: horizontal note blocks, active-note highlighting, a playback cursor, click-to-seek, and automatic scrolling at higher zoom. Controls provide play/pause, restart, a seek slider, playback speed, and zoom; left/right arrows on the roll seek by one second. Loading another result stops the previous playback.
24
+
25
+ The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; research diagnostics remain in the generated files.
26
+
27
  ## Poetry voice examples
28
 
29
  Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s **Quiet Night Thoughts**, Meng Haoran’s **Spring Dawn**, Wang Zhihuan’s **On the Stork Tower**, the opening couplet of Shakespeare’s **Sonnet 18**, and the first stanza of Blake’s **The Tyger**. Select an example to load and listen to its recording, then press **Generate piano MIDI**. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in [the poetry examples](examples/poetry/README.md).
 
57
 
58
  Here $F$ is band-magnitude analysis, $y$ is the normalized target, $M$ is MIDI, and $S$ is the fixed piano renderer. The quick setting runs dictionary search and global velocity refinement; it skips the per-note sweep. The baseline rounds strong spectral peaks to piano pitches and separately calibrates global velocity.
59
 
60
+ Saved diagnostic spectrograms use the same color scale for all four panels. Recorded metrics compare the input-length interval; audio includes a 0.5-second release tail. Log-band MAE is the optimization target, while spectral convergence is reported separately and need not improve. Neither metric measures speech intelligibility. Magnitude addition is only an approximation because real notes interact in phase, which motivates the renderer refinement step.
61
 
62
  The fitting core is vendored from [Speaking MIDI](https://github.com/k-l-lambda/speaking-midi/tree/932f8eb), with a portable SoundFont path and an optional quick-refinement flag. See that project for full experiments, derivations, and reproducible demo provenance.
63
 
VALIDATION.md CHANGED
@@ -20,3 +20,7 @@ Actual hosted API checks passed on 2026-10-01:
20
  - Text → speech → MIDI: `Hello, piano.`, English, the requested female voice description, followed by fitting a 0.5-second prefix with 20 search iterations; completed in 8.6 seconds. Speech used the decorated ZeroGPU function; MIDI fitting used CPU.
21
 
22
  These timings are the app's reported job durations, excluding client upload/download and queue transit. They validate short jobs, not the maximum 12-second/full-refinement configuration. The hosted PyTorch configuration error is resolved. No subscription or dedicated paid hardware was requested.
 
 
 
 
 
20
  - Text → speech → MIDI: `Hello, piano.`, English, the requested female voice description, followed by fitting a 0.5-second prefix with 20 search iterations; completed in 8.6 seconds. Speech used the decorated ZeroGPU function; MIDI fitting used CPU.
21
 
22
  These timings are the app's reported job durations, excluding client upload/download and queue transit. They validate short jobs, not the maximum 12-second/full-refinement configuration. The hosted PyTorch configuration error is resolved. No subscription or dedicated paid hardware was requested.
23
+
24
+ ## Interactive MIDI player (2026-10-02)
25
+
26
+ The demo now shows source audio and an interactive piano roll instead of diagnostic tabs. Seven backend tests passed, including real CPU fitting and the ZeroGPU dispatch test. Browser checks passed for playback and pause, click and keyboard seeking, zoom and scrolling, replacing a playing example, removal of diagnostic tabs, and mobile layout. No JavaScript errors were observed in those checks. The player decodes note events from the downloaded MIDI and synchronizes them to that MIDI's Grand Piano audio render.
app.py CHANGED
@@ -13,6 +13,7 @@ os.environ.setdefault('MPLBACKEND', 'Agg')
13
  import gradio as gr
14
  import numpy as np
15
  import soundfile as sf
 
16
 
17
  ROOT = Path(__file__).resolve().parent
18
  VOICE = '一位温柔自信的年轻女性,声音清晰,语气亲切,表达轻快而富有感染力。'
@@ -66,27 +67,9 @@ def prepare_audio(path, out, seconds):
66
  return len(audio) / rate
67
 
68
 
69
- def piano_roll(directory):
70
- import matplotlib.pyplot as plt
71
- from matplotlib.patches import Rectangle
72
- notes = json.loads((directory / 'refined.json').read_text())
73
- fig, ax = plt.subplots(figsize=(11, 3))
74
- for note in notes:
75
- ax.add_patch(Rectangle((note['start'], note['pitch']-.4), note['duration'], .8,
76
- color=plt.cm.viridis(note['velocity']/127)))
77
- duration = sf.info(directory / 'target.wav').duration
78
- ax.set(xlim=(0, duration), ylim=(35,109), xlabel='Seconds', ylabel='MIDI pitch',
79
- title=f'{len(notes)} piano notes · color indicates velocity')
80
- ax.grid(alpha=.15); fig.tight_layout()
81
- path = directory / 'piano-roll.png'; fig.savefig(path, dpi=130); plt.close(fig)
82
- return str(path)
83
-
84
-
85
  def result(directory, status):
86
- metrics = json.loads((directory / 'metrics.json').read_text())
87
- return (status, str(directory/'target.wav'), str(directory/'baseline.wav'),
88
- str(directory/'refined.wav'), str(directory/'spectrogram.png'),
89
- (str(directory/'piano-roll.png') if (directory/'piano-roll.png').exists() else piano_roll(directory)), str(directory/'refined.mid'), metrics['results'])
90
 
91
 
92
  def run(mode, audio, text, language, description, seconds, budget, refine, progress=gr.Progress()):
@@ -98,7 +81,7 @@ def run(mode, audio, text, language, description, seconds, budget, refine, progr
98
  if old.is_dir() and time.time() - old.stat().st_mtime > 86400:
99
  shutil.rmtree(old)
100
  directory = Path(tempfile.mkdtemp(prefix='job-', dir=RUNS))
101
- empty = (None,) * 7
102
  started = time.monotonic()
103
  if mode == 'Text → speech → MIDI':
104
  if not text.strip() or len(text) > 300 or not description.strip() or len(description) > 600:
@@ -119,14 +102,14 @@ def run(mode, audio, text, language, description, seconds, budget, refine, progr
119
  audio = str(directory/'speech.wav')
120
  duration = prepare_audio(audio, directory/'input.wav', seconds)
121
  yield (f'2/4 · Analyzing the first {duration:.2f} s and building piano templates.',
122
- str(directory/'input.wav'), None, None, None, None, None, None)
123
  progress(.3, desc='Fitting piano note templates; then checking the real synthesizer')
124
  command = ['-m', 'speaking_midi', str(directory/'input.wav'), '--out', str(directory),
125
  '--soundfont', str(ROOT/'assets/gm.sf3'), '--max-notes', str(int(budget))]
126
  if not refine:
127
  command.append('--skip-local-refinement')
128
  worker(command, directory/'fit.log')
129
- progress(.95, desc='Drawing spectrograms and MIDI piano roll')
130
  yield result(directory, f'4/4 · Complete in {time.monotonic()-started:.1f} s on {DEVICE}. '
131
  + ('Full renderer refinement.' if refine else 'Quick fit with global velocity refinement.'))
132
 
@@ -137,7 +120,7 @@ def example(language):
137
 
138
  def build_ui():
139
  with gr.Blocks(title='Speaking MIDI') as demo:
140
- gr.Markdown('# Speaking MIDI\nTurn a voice into piano notes. Compare the sound, inspect the spectrum, and download the MIDI.\n'
141
  'This is experimental sound approximation: words may become difficult to understand.')
142
  with gr.Row():
143
  english = gr.Button('Listen to English example')
@@ -160,8 +143,7 @@ def build_ui():
160
  with gr.Column(scale=2):
161
  status = gr.Markdown('Choose an example or create a new fit.')
162
  source = gr.Audio(label='1 · Source used for fitting', interactive=False)
163
- baseline = gr.Audio(label='2 · Simple spectral-peak baseline', interactive=False)
164
- piano = gr.Audio(label='3 · Optimized piano MIDI rendered with Grand Piano', interactive=False)
165
  midi = gr.File(label='Download MIDI', interactive=False)
166
  poetry = json.loads((ROOT/'examples/poetry/prompts.json').read_text())
167
  gr.Examples(
@@ -173,18 +155,11 @@ def build_ui():
173
  )
174
  selected_text = gr.Textbox(label='Selected text', interactive=False, lines=2)
175
  text.change(lambda value: value, text, selected_text, queue=False)
176
- with gr.Tabs():
177
- with gr.Tab('Spectrogram comparison'):
178
- spectrum = gr.Image(label='Shared scale · source / baseline / dictionary / optimized', interactive=False)
179
- with gr.Tab('MIDI piano roll'):
180
- roll = gr.Image(label='Note timing, pitch and velocity', interactive=False)
181
- with gr.Tab('Measurements'):
182
- measurements = gr.JSON(label='Lower log-band MAE is better; this does not measure intelligibility')
183
  gr.Markdown('**How it works:** synthesize 657 piano templates → match weighted spectral bands → '
184
  'choose discrete notes → re-render MIDI → reduce measured spectral error. '
185
  '[Algorithm and experiments](https://github.com/k-l-lambda/speaking-midi) · '
186
  '[Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign)')
187
- outputs = [status, source, baseline, piano, spectrum, roll, midi, measurements]
188
  mode.change(lambda value: (gr.update(visible=value=='Audio → MIDI'), gr.update(visible=value!='Audio → MIDI')),
189
  mode, [audio, speech_controls], queue=False)
190
  submit.click(run, [mode,audio,text,language,description,seconds,budget,refine], outputs,
 
13
  import gradio as gr
14
  import numpy as np
15
  import soundfile as sf
16
+ from player import player_component, player_payload
17
 
18
  ROOT = Path(__file__).resolve().parent
19
  VOICE = '一位温柔自信的年轻女性,声音清晰,语气亲切,表达轻快而富有感染力。'
 
67
  return len(audio) / rate
68
 
69
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
70
  def result(directory, status):
71
+ return (status, str(directory/'target.wav'), player_payload(directory),
72
+ str(directory/'refined.mid'))
 
 
73
 
74
 
75
  def run(mode, audio, text, language, description, seconds, budget, refine, progress=gr.Progress()):
 
81
  if old.is_dir() and time.time() - old.stat().st_mtime > 86400:
82
  shutil.rmtree(old)
83
  directory = Path(tempfile.mkdtemp(prefix='job-', dir=RUNS))
84
+ empty = (None, '', None)
85
  started = time.monotonic()
86
  if mode == 'Text → speech → MIDI':
87
  if not text.strip() or len(text) > 300 or not description.strip() or len(description) > 600:
 
102
  audio = str(directory/'speech.wav')
103
  duration = prepare_audio(audio, directory/'input.wav', seconds)
104
  yield (f'2/4 · Analyzing the first {duration:.2f} s and building piano templates.',
105
+ str(directory/'input.wav'), '', None)
106
  progress(.3, desc='Fitting piano note templates; then checking the real synthesizer')
107
  command = ['-m', 'speaking_midi', str(directory/'input.wav'), '--out', str(directory),
108
  '--soundfont', str(ROOT/'assets/gm.sf3'), '--max-notes', str(int(budget))]
109
  if not refine:
110
  command.append('--skip-local-refinement')
111
  worker(command, directory/'fit.log')
112
+ progress(.95, desc='Preparing the MIDI player')
113
  yield result(directory, f'4/4 · Complete in {time.monotonic()-started:.1f} s on {DEVICE}. '
114
  + ('Full renderer refinement.' if refine else 'Quick fit with global velocity refinement.'))
115
 
 
120
 
121
  def build_ui():
122
  with gr.Blocks(title='Speaking MIDI') as demo:
123
+ gr.Markdown('# Speaking MIDI\nTurn a voice into piano notes. Listen to your voice on piano, follow the notes, and download the MIDI.\n'
124
  'This is experimental sound approximation: words may become difficult to understand.')
125
  with gr.Row():
126
  english = gr.Button('Listen to English example')
 
143
  with gr.Column(scale=2):
144
  status = gr.Markdown('Choose an example or create a new fit.')
145
  source = gr.Audio(label='1 · Source used for fitting', interactive=False)
146
+ player = player_component(gr)
 
147
  midi = gr.File(label='Download MIDI', interactive=False)
148
  poetry = json.loads((ROOT/'examples/poetry/prompts.json').read_text())
149
  gr.Examples(
 
155
  )
156
  selected_text = gr.Textbox(label='Selected text', interactive=False, lines=2)
157
  text.change(lambda value: value, text, selected_text, queue=False)
 
 
 
 
 
 
 
158
  gr.Markdown('**How it works:** synthesize 657 piano templates → match weighted spectral bands → '
159
  'choose discrete notes → re-render MIDI → reduce measured spectral error. '
160
  '[Algorithm and experiments](https://github.com/k-l-lambda/speaking-midi) · '
161
  '[Qwen3-TTS](https://huggingface.co/Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign)')
162
+ outputs = [status, source, player, midi]
163
  mode.change(lambda value: (gr.update(visible=value=='Audio → MIDI'), gr.update(visible=value!='Audio → MIDI')),
164
  mode, [audio, speech_controls], queue=False)
165
  submit.click(run, [mode,audio,text,language,description,seconds,budget,refine], outputs,
deploy.py CHANGED
@@ -8,7 +8,7 @@ if __name__ == '__main__':
8
  api.create_repo(repo, repo_type='space', space_sdk='gradio', space_hardware='zero-a10g', exist_ok=True)
9
  api.add_space_variable(repo, 'INFERENCE_DEVICE', 'zerogpu')
10
  commit = api.upload_folder(repo_id=repo, repo_type='space', folder_path=Path(__file__).parent,
11
- allow_patterns=['app.py', 'zero_tts.py', 'deploy.py', 'README.md', 'VALIDATION.md', 'requirements.txt', 'packages.txt',
12
  '.gitignore', 'speaking_midi/*.py', 'scripts/*.py', 'tests/*.py',
13
  'assets/*', 'examples/**'],
14
  commit_message='Build CPU-capable Speaking MIDI audio and VoiceDesign demo')
 
8
  api.create_repo(repo, repo_type='space', space_sdk='gradio', space_hardware='zero-a10g', exist_ok=True)
9
  api.add_space_variable(repo, 'INFERENCE_DEVICE', 'zerogpu')
10
  commit = api.upload_folder(repo_id=repo, repo_type='space', folder_path=Path(__file__).parent,
11
+ allow_patterns=['app.py', 'player.py', 'ui/*', 'zero_tts.py', 'deploy.py', 'README.md', 'VALIDATION.md', 'requirements.txt', 'packages.txt',
12
  '.gitignore', 'speaking_midi/*.py', 'scripts/*.py', 'tests/*.py',
13
  'assets/*', 'examples/**'],
14
  commit_message='Build CPU-capable Speaking MIDI audio and VoiceDesign demo')
player.py ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Build a browser player payload from the exported MIDI and its exact piano render."""
2
+ import base64
3
+ import json
4
+ from pathlib import Path
5
+ import mido
6
+ import soundfile as sf
7
+
8
+ UI = Path(__file__).parent / 'ui'
9
+
10
+
11
+ def player_payload(directory):
12
+ directory = Path(directory)
13
+ clock = 0.0
14
+ active = {}
15
+ notes = []
16
+ for message in mido.MidiFile(directory / 'refined.mid'):
17
+ clock += message.time
18
+ if message.type == 'note_on' and message.velocity:
19
+ active[(message.channel, message.note)] = (clock, message.velocity)
20
+ elif message.type == 'note_off' or (message.type == 'note_on' and not message.velocity):
21
+ start, velocity = active.pop((message.channel, message.note))
22
+ notes.append(dict(pitch=message.note, start=start, duration=clock-start, velocity=velocity))
23
+ if active:
24
+ raise ValueError('MIDI contains unterminated notes')
25
+ wav = directory / 'refined.wav'
26
+ return json.dumps(dict(notes=notes, duration=sf.info(wav).duration,
27
+ audio='data:audio/wav;base64,' + base64.b64encode(wav.read_bytes()).decode()))
28
+
29
+
30
+ def player_component(gr):
31
+ return gr.HTML(value='', html_template=(UI/'player.html').read_text(),
32
+ css_template=(UI/'player.css').read_text(),
33
+ js_on_load=(UI/'player.js').read_text(), elem_id='midi-player')
tests/test_app.py CHANGED
@@ -20,8 +20,9 @@ class AppTests(unittest.TestCase):
20
  events=list(run('Audio → MIDI','examples/en/source.wav','','English','',.5,20,False))
21
  self.assertEqual(len(events),2)
22
  self.assertIn('Complete',events[-1][0])
23
- for path in events[-1][1:7]: self.assertTrue(Path(path).exists())
24
- report=json.loads((Path(events[-1][6]).parent/'metrics.json').read_text())
 
25
  self.assertEqual(report['pursuit']['device'],'cpu')
26
  self.assertGreater(report['results']['refined']['notes'],0)
27
  def test_zerogpu_speech_dispatch_keeps_fitting_on_cpu(self):
@@ -31,7 +32,7 @@ class AppTests(unittest.TestCase):
31
  events = list(run('Text → speech → MIDI', None, 'Hello.', 'English',
32
  app.VOICE, .5, 20, False))
33
  generate.assert_called_once_with('Hello.', 'English', app.VOICE)
34
- output = Path(events[-1][6]).parent
35
  self.assertTrue((output/'speech.wav').exists())
36
  self.assertFalse((output/'tts.log').exists())
37
  report = json.loads((output/'metrics.json').read_text())
@@ -40,6 +41,7 @@ class AppTests(unittest.TestCase):
40
  def test_cached_examples(self):
41
  for language in ['en','zh']:
42
  output=example(language)
43
- for path in output[1:7]: self.assertTrue(Path(path).exists())
 
44
 
45
  if __name__=='__main__': unittest.main()
 
20
  events=list(run('Audio → MIDI','examples/en/source.wav','','English','',.5,20,False))
21
  self.assertEqual(len(events),2)
22
  self.assertIn('Complete',events[-1][0])
23
+ for path in (events[-1][1], events[-1][3]): self.assertTrue(Path(path).exists())
24
+ self.assertGreater(len(json.loads(events[-1][2])['notes']), 0)
25
+ report=json.loads((Path(events[-1][3]).parent/'metrics.json').read_text())
26
  self.assertEqual(report['pursuit']['device'],'cpu')
27
  self.assertGreater(report['results']['refined']['notes'],0)
28
  def test_zerogpu_speech_dispatch_keeps_fitting_on_cpu(self):
 
32
  events = list(run('Text → speech → MIDI', None, 'Hello.', 'English',
33
  app.VOICE, .5, 20, False))
34
  generate.assert_called_once_with('Hello.', 'English', app.VOICE)
35
+ output = Path(events[-1][3]).parent
36
  self.assertTrue((output/'speech.wav').exists())
37
  self.assertFalse((output/'tts.log').exists())
38
  report = json.loads((output/'metrics.json').read_text())
 
41
  def test_cached_examples(self):
42
  for language in ['en','zh']:
43
  output=example(language)
44
+ for path in (output[1], output[3]): self.assertTrue(Path(path).exists())
45
+ self.assertTrue(json.loads(output[2])['audio'].startswith('data:audio/wav;base64,'))
46
 
47
  if __name__=='__main__': unittest.main()
ui/player.css ADDED
@@ -0,0 +1,4 @@
 
 
 
 
 
1
+ .player{font:14px system-ui,sans-serif;color:#dce7f7;background:#101b2d;border:1px solid #253651;border-radius:14px;overflow:hidden;min-width:0}
2
+ .player header{display:flex;justify-content:space-between;gap:12px;align-items:center;padding:16px 18px 10px}.player strong{font-size:17px}.subtitle{color:#91a5c2;margin-left:12px;font-size:12px}.note-count{color:#91a5c2;font-size:12px}
3
+ .transport{display:flex;align-items:center;gap:10px;flex-wrap:wrap;padding:4px 18px}.player button,.player select{border:1px solid #3b506c;border-radius:7px;background:#24364f;color:#eef5ff;padding:7px 11px;cursor:pointer;font:inherit}.player button:disabled{opacity:.4;cursor:default}.player .play{background:#6ce2b6;color:#102d28;border-color:transparent;min-width:82px;font-weight:650}.clock{font-variant-numeric:tabular-nums;font-size:12px;min-width:90px}.speed-label{margin-left:auto;font-size:12px;color:#b3c5dd}.player select{padding:5px;font-size:12px}.seek{display:block;width:calc(100% - 36px);margin:10px 18px;accent-color:#6ce2b6}.roll-wrap{position:relative;height:300px}.roll{display:block;width:100%;height:300px;touch-action:pan-y;cursor:crosshair}.empty{position:absolute;top:42%;left:60px;right:12px;text-align:center;color:#a5b7ce;pointer-events:none}.player footer{display:flex;align-items:center;justify-content:space-between;gap:12px;padding:10px 18px;color:#91a5c2;font-size:11px}.player footer label{display:flex;align-items:center;gap:8px}.zoom{width:85px;accent-color:#6ce2b6}.error{margin:0;padding:0 18px 10px;color:#ffb3a9;font-size:12px}.error:empty{display:none}.player :focus-visible{outline:2px solid #84bfff;outline-offset:2px}@media(max-width:480px){.player header,.transport{padding-left:10px;padding-right:10px}.subtitle{display:block;margin:3px 0 0}.player footer{flex-wrap:wrap}.transport{gap:7px}.speed-label{margin-left:0}}
4
+ .player strong{color:#e7f0ff!important}.player .clock{color:#dce7f7!important}.player footer span,.player footer label{color:#a9bed8!important}.player .empty{color:#a5b7ce!important}
ui/player.html ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ <section class="player" aria-label="MIDI piano roll player">
2
+ <header><div><strong>MIDI piano roll</strong><span class="subtitle">Grand Piano</span></div><span class="note-count">No MIDI loaded</span></header>
3
+ <div class="transport">
4
+ <button class="play" disabled aria-label="Play piano MIDI">▶ Play</button>
5
+ <button class="restart" disabled aria-label="Restart playback">↺</button>
6
+ <output class="clock" aria-label="Playback time">0:00 / 0:00</output>
7
+ <label class="speed-label">Speed <select class="speed" aria-label="Playback speed"><option value="0.5">0.5×</option><option value="0.75">0.75×</option><option value="1" selected>1×</option><option value="1.25">1.25×</option><option value="1.5">1.5×</option></select></label>
8
+ </div>
9
+ <input class="seek" type="range" min="0" max="1" value="0" step="0.01" disabled aria-label="Playback position">
10
+ <div class="roll-wrap"><canvas class="roll" tabindex="0" role="img" aria-label="MIDI notes by time and pitch. Click to seek; use left and right arrows to move one second."></canvas><p class="empty">Choose an example or generate piano MIDI to play.</p></div>
11
+ <footer><span>Click the roll to seek · lit notes are playing</span><label>Zoom <input class="zoom" type="range" min="1" max="4" step="0.25" value="1" aria-label="Piano roll zoom"></label></footer>
12
+ <p class="error" role="status"></p>
13
+ </section>
ui/player.js ADDED
@@ -0,0 +1,105 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ // Interaction reference: klstudio/app/views/midi-player.vue.
2
+ // The renderer's WAV is the playback clock; notes come from the exported MIDI.
3
+ let dispose = () => {};
4
+ function mount() {
5
+ dispose();
6
+ const root = element.querySelector('.player');
7
+ if (!root) return;
8
+ const $ = (s) => root.querySelector(s);
9
+ const canvas = $('.roll'), ctx = canvas.getContext('2d');
10
+ const play = $('.play'), seek = $('.seek'), clock = $('.clock');
11
+ const audio = new Audio();
12
+ const listeners = new AbortController();
13
+ let frame = 0, stopped = false, data = null, offset = 0, zoom = 1;
14
+ let width = 600, height = 300, low = 36, high = 84;
15
+ try { data = props.value ? JSON.parse(props.value) : null; } catch (_) {}
16
+ const duration = data ? data.duration : 8;
17
+ const notes = data ? data.notes : [];
18
+ const left = 46, top = 23, bottom = 10;
19
+ const clamp = (n, a, b) => Math.min(b, Math.max(a, n));
20
+ const format = (n) => `${Math.floor(n / 60)}:${Math.floor(n % 60).toString().padStart(2, '0')}`;
21
+ const on = (target, type, fn) => target.addEventListener(type, fn, {signal: listeners.signal});
22
+ const span = () => Math.max(duration / zoom, .1);
23
+ function follow() {
24
+ const time = audio.currentTime;
25
+ if (time < offset + span() * .15 || time > offset + span() * .7)
26
+ offset = clamp(time - span() * .4, 0, Math.max(0, duration - span()));
27
+ }
28
+ function draw() {
29
+ if (stopped) return;
30
+ const time = audio.currentTime;
31
+ const scale = (width-left)/span(), row = (height-top-bottom)/(high-low+1);
32
+ ctx.clearRect(0,0,width,height);
33
+ ctx.fillStyle='#101b2d';ctx.fillRect(0,0,width,height);
34
+ const playing = new Set(notes.filter(n=>!audio.paused && n.start<=time && n.start+n.duration>time).map(n=>n.pitch));
35
+ for (let pitch=low;pitch<=high;pitch++) {
36
+ const y=top+(high-pitch)*row, black=[1,3,6,8,10].includes(pitch%12);
37
+ ctx.fillStyle=black?'#0c1626':'#142136';ctx.fillRect(left,y,width-left,row);
38
+ ctx.fillStyle=playing.has(pitch)?'#6ce2b6':black?'#23334a':'#e1e8f2';ctx.fillRect(0,y,left-2,Math.max(.5,row-1));
39
+ if (pitch%12===0) {
40
+ ctx.fillStyle=playing.has(pitch)?'#102d28':'#4f627a';ctx.font='10px system-ui';ctx.fillText(`C${Math.floor(pitch/12)-1}`,7,y+Math.max(8,row*.8));
41
+ ctx.strokeStyle='#33425b';ctx.beginPath();ctx.moveTo(left,y+row);ctx.lineTo(width,y+row);ctx.stroke();
42
+ }
43
+ }
44
+ const step=span()>6?1:.5;
45
+ for(let t=Math.ceil(offset/step)*step;t<=offset+span();t+=step) {
46
+ const x=left+(t-offset)*scale;
47
+ ctx.strokeStyle='#26364d';ctx.beginPath();ctx.moveTo(x,top);ctx.lineTo(x,height-bottom);ctx.stroke();
48
+ ctx.fillStyle='#9bb0ce';ctx.font='10px system-ui';ctx.fillText(`${t.toFixed(step===1?0:1)}s`,x+3,14);
49
+ }
50
+ ctx.save();ctx.beginPath();ctx.rect(left,top,width-left,height-top);ctx.clip();
51
+ for(const n of notes) {
52
+ const x=left+(n.start-offset)*scale,y=top+(high-n.pitch)*row;
53
+ const active=!audio.paused && n.start<=time && n.start+n.duration>time;
54
+ ctx.fillStyle=active?'#d2ffe6':`hsl(${200-n.velocity*.35} 70% ${42+n.velocity*.15}%)`;
55
+ ctx.fillRect(x,y+Math.min(1,row*.1),Math.max(2,n.duration*scale),Math.max(1,row-1));
56
+ if(active){ctx.strokeStyle='#6ce2b6';ctx.strokeRect(x,y,Math.max(2,n.duration*scale),row);}
57
+ }
58
+ const cursor=left+(time-offset)*scale;
59
+ ctx.strokeStyle='#f8bc68';ctx.lineWidth=2;ctx.beginPath();ctx.moveTo(cursor,top);ctx.lineTo(cursor,height);ctx.stroke();ctx.lineWidth=1;ctx.restore();
60
+ canvas.dataset.time=String(time);canvas.dataset.offset=String(offset);canvas.dataset.active=String(playing.size);
61
+ clock.textContent=`${format(time)} / ${format(duration)}`;
62
+ seek.value=String(time);
63
+ }
64
+ function animate() {
65
+ if(stopped || !element.isConnected) { dispose();return; }
66
+ follow();draw();
67
+ if(!audio.paused) frame=requestAnimationFrame(animate);
68
+ }
69
+ function seekTo(time) {if(!data)return;audio.currentTime=clamp(time,0,duration);follow();draw();}
70
+ function resize() {
71
+ width=Math.max(180,canvas.clientWidth);const dpr=window.devicePixelRatio||1;
72
+ canvas.width=width*dpr;canvas.height=height*dpr;ctx.setTransform(dpr,0,0,dpr,0,0);draw();
73
+ }
74
+ play.textContent='▶ Play';play.setAttribute('aria-label','Play piano MIDI');
75
+ play.disabled=true;$('.restart').disabled=true;seek.disabled=true;seek.value='0';
76
+ $('.empty').hidden=false;$('.note-count').textContent='No MIDI loaded';
77
+ $('.speed').value='1';$('.zoom').value='1';$('.error').textContent='';
78
+ if(data) {
79
+ if(notes.length){low=Math.max(0,Math.min(...notes.map(n=>n.pitch))-2);high=Math.min(127,Math.max(...notes.map(n=>n.pitch))+2);}
80
+ audio.src=data.audio;audio.preload='auto';seek.max=String(duration);
81
+ $('.empty').hidden=true;play.disabled=false;$('.restart').disabled=false;seek.disabled=false;
82
+ $('.note-count').textContent=`${notes.length} notes`;
83
+ }
84
+ on(play,'click',async()=>{
85
+ if(!data)return;
86
+ if(audio.paused) {
87
+ if(audio.ended || audio.currentTime>=duration)seekTo(0);
88
+ try{await audio.play();$('.error').textContent='';}catch(_){$('.error').textContent='Playback could not start. Please press Play again.';}
89
+ } else audio.pause();
90
+ });
91
+ on(audio,'play',()=>{play.textContent='Ⅱ Pause';play.setAttribute('aria-label','Pause piano MIDI');cancelAnimationFrame(frame);animate();});
92
+ on(audio,'pause',()=>{play.textContent='▶ Play';play.setAttribute('aria-label','Play piano MIDI');cancelAnimationFrame(frame);draw();});
93
+ on(audio,'ended',()=>{cancelAnimationFrame(frame);draw();});
94
+ on(audio,'error',()=>{$('.error').textContent='Unable to load the piano audio.';});
95
+ on($('.restart'),'click',()=>seekTo(0));
96
+ on(seek,'input',()=>seekTo(Number(seek.value)));
97
+ on($('.speed'),'change',(e)=>{audio.playbackRate=Number(e.target.value);});
98
+ on($('.zoom'),'input',(e)=>{zoom=Number(e.target.value);offset=clamp(audio.currentTime-span()*.4,0,Math.max(0,duration-span()));draw();});
99
+ on(canvas,'click',(e)=>{const x=e.clientX-canvas.getBoundingClientRect().left;if(x>=left)seekTo(offset+(x-left)/(width-left)*span());});
100
+ on(canvas,'keydown',(e)=>{if(['ArrowLeft','ArrowRight'].includes(e.key)){e.preventDefault();seekTo(audio.currentTime+(e.key==='ArrowRight'?1:-1));}});
101
+ const observer=new ResizeObserver(resize);observer.observe(canvas);resize();
102
+ dispose=()=>{if(stopped)return;stopped=true;listeners.abort();cancelAnimationFrame(frame);observer.disconnect();audio.pause();audio.removeAttribute('src');audio.load();};
103
+ }
104
+ watch('value', mount);
105
+ mount();