Spaces:

k-l-lambda
/

speaking-midi

Sleeping

App Files Files Community

speaking-midi / README.md

k-l-lambda

Default audio fitting duration to 30 seconds

04f9110 about 1 month ago

preview code

Raw History Blame Contribute Delete

12.1 kB

---
title: Speaking MIDI
emoji: 🎹
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.17.3
python_version: '3.10'
app_file: app.py
pinned: false
models:
  - Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
---
# Speaking MIDI

An interactive demonstration of **fixed-timbre inverse synthesis**: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then listen to the source and play the resulting MIDI in an interactive piano roll. Download the actual MIDI from the same result.

The two instant-play examples now use the dense algorithm: **The Tyger** in English and **Quiet Night Thoughts** in Chinese. Both input modes offer Speech (dense) and Instrumental (sparse) fitting, with 600 initial optimization iterations by default. Speech is the default. The research backtest reduced pooled Chinese CER from 100% to 68.3% and English WER from 116.7% to 50% on five poetry recordings. Recognition remains imperfect, and these results do not establish general human intelligibility. See the [ASR report](https://github.com/k-l-lambda/speaking-midi/blob/5caf4e2/experiments/DENSE_ASR.md).

## MIDI player

The demo centers on a piano roll inspired by the interaction design of `klstudio/app/views/midi-player.vue`: horizontal note blocks, active-note highlighting, a playback cursor, click-to-seek, and automatic scrolling at higher zoom. Controls provide play/pause, restart, a seek slider, playback speed, and zoom; left/right arrows on the roll seek by one second. Loading another result stops the previous playback.

The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; fitting settings and numerical results remain in each job’s `report.json`.

## Poetry voice examples

Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s **Quiet Night Thoughts**, Meng Haoran’s **Spring Dawn**, Wang Zhihuan’s **On the Stork Tower**, the opening couplet of Shakespeare’s **Sonnet 18**, and the first stanza of Blake’s **The Tyger**. Select an example to load and listen to its recording, then press **Generate piano MIDI**. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in [the poetry examples](examples/poetry/README.md).

## Two input modes

- **Audio → MIDI:** upload or record sound, select the prefix duration, and fit piano notes. This path never loads a TTS model.
- **Text → speech → MIDI:** generate speech with Qwen3-TTS 1.7B VoiceDesign, then run exactly the same fitting pipeline. The default voice description is `一位温柔自信的年轻女性，声音清晰，语气亲切，表达轻快而富有感染力。` (a gentle, confident young woman with clear speech and a warm, lively delivery).

Inputs are limited to 25 MB, 300 text characters, and a fitted prefix of at most 30 seconds. Longer recordings are read only through the selected prefix. New jobs share one compute queue, with at most eight queued requests. Temporary outputs are ephemeral, are cleaned after 24 hours when another job starts, and disappear when the Space restarts. Do not treat this public demonstration as private storage.

## Dense fitting algorithm

The renderer is FluidSynth with bank 0, program 0 of the bundled Grand Piano SoundFont, fixed gain 0.8, and disabled chorus/reverb. The output contains only piano MIDI notes: no source-speech layer, pitch bend, or output EQ.

1. Convert input to 24 kHz mono and normalize its RMS to 0.035. Analyze squared STFT magnitudes with FFT 1024 / hop 240 and 128 Slaney-normalized mel bands spanning 40–11,000 Hz.
2. Render temporal templates for all 88 piano keys (MIDI 21–108), at velocity 48 and a 20 ms gate. A 400 ms template window retains attack, decay and release. Cache templates and per-pitch velocity calibration by SoundFont hash and configuration.
3. Give every key an onset candidate every 20 ms. Jointly optimize nonnegative strengths with projected Adam rather than greedily selecting a small note list. The default is 600 iterations, with a cosine learning-rate schedule from 0.04 to 0.001.
4. Map strengths to integer velocities using measured energy ratios at 13 velocity levels. Drop strengths below the velocity-8 threshold and events extending beyond the clip. The selected research clips contain 1,218–3,140 notes; the iteration setting is not a note-count limit.
5. Save MIDI, reload it and render through FluidSynth. The interactive player displays these exported events and plays their corresponding piano WAV, including a 0.5-second release tail.

For target mel power $M$, temporal templates $D$, and nonnegative note strengths $A$, fitting minimizes:

$$
L=\operatorname{mean}\left[\log(1+\widehat M/10^{-3})-\log(1+M/10^{-3})\right]^2,
\qquad \widehat M=D*A,\quad A\geq0.
$$
The convolution sums shifted template powers. It approximates phase interactions, velocity-dependent timbre and repeated-note behavior, so validation uses actual re-rendered MIDI. No transcript or ASR gradient enters the optimizer. ASR selected a configuration on two pilot clips before testing three further poetry clips; repeated renders, temperature-zero decoding and a complete CPU path confirmed the corpus-level improvement. Some conventional spectral metrics worsen, and there is no blind human listening result.
`Fitting iterations` ranges from 200 to 1,200; 600 is the tested setting. More iterations cost more time and do not guarantee clearer speech. The old note-budget and full-refinement controls have been removed from the demo.
The dense fitter is vendored without algorithm changes from [Speaking MIDI commit 5caf4e2](https://github.com/k-l-lambda/speaking-midi/tree/5caf4e2). The shared rendering core retains the Space’s portable SoundFont path. The instant examples in `examples/dense` are byte-identical MIDI/WAV copies of the selected research outputs, with normalized source audio for comparison and a hash manifest. Archived sparse examples remain in `examples/en` and `examples/zh` for provenance.
## Instrumental fitting option
Select **Instrumental (sparse)** for music that benefits from fewer attacks and
longer held notes. **Speech (dense)** keeps the existing 20 ms speech-oriented
representation. Fitting style is independent of the input mode, so either is
available after uploading audio or generating speech. The duration slider accepts
0.5–30 seconds and defaults to 30; short recordings use their actual length. Instant poetry playback
remains precomputed dense output, and selecting a poetry input resets the fitting
style to Speech.
The instrumental fitter is copied from research commit
[292f5e0](https://github.com/k-l-lambda/speaking-midi/tree/292f5e0).
It uses 80/240/640 ms gate templates on a 40 ms onset grid, a 2048-point FFT,
128 mel bands, and an activation-sparsity penalty of 0.03. The objective combines
squared log-mel-power error with the sum of nonnegative activations divided by
analysis frame count. After the selected initial iteration count, competing
same-key notes are removed and 200 additional iterations refit strengths on the
remaining support. Energy calibration converts strengths to MIDI velocities;
notes below velocity 12 are omitted.
Long inputs are fitted in 12-second sections with 1.2 seconds of context on each
side. Only central onsets are retained, and same-key overlaps across boundaries
are shortened. Export preserves the source timeline, including trailing silence.
The Space adapter reloads the saved MIDI, renders it with the bundled Grand Piano,
and passes that exact audio and MIDI to the existing player. Both dictionary
construction and fitting run on CPU. This option does not perform vocal separation.
It substantially reduced note counts in local music experiments, but improved
listening quality is not established. See the
[instrumental report](https://github.com/k-l-lambda/speaking-midi/blob/292f5e0/experiments/INSTRUMENTAL_FITTING.md).
## CPU and ZeroGPU execution
The hosted [Speaking MIDI Space](https://huggingface.co/spaces/k-l-lambda/speaking-midi) uses `INFERENCE_DEVICE=zerogpu`. VoiceDesign loads at startup with CUDA placement and bfloat16, following the ZeroGPU model-loading contract. A `@spaces.GPU(duration=120)` function requests a shared GPU only for speech generation. MIDI fitting runs in a separate CPU worker and does not consume GPU quota. ZeroGPU requests can queue or fail when the visitor's GPU quota is exhausted; the precomputed speech examples can still be fitted without generating another voice.
The requirements use `torch==2.10.0` and matching torchaudio. Do not add `+cpu` or the CPU-only wheel index to the hosted requirements: ZeroGPU rejects that version specification and needs GPU-capable PyTorch.
Local execution defaults to `INFERENCE_DEVICE=cpu`. TTS uses float32 with SDPA in a separate process, releasing model memory before MIDI fitting. Both fitting styles explicitly use CPU in every app mode, including ZeroGPU. Both template creation and optimization work without a GPU.
```bash
# Python 3.10; install libfluidsynth3, libsndfile1 and sox using your OS package manager.
python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt gradio==6.17.3
INFERENCE_DEVICE=cpu python app.py
```
Optional variables: `QWEN_MODEL_PATH` selects a local VoiceDesign snapshot; `PORT` defaults to 7860; `OMP_NUM_THREADS` defaults to 2. `INFERENCE_DEVICE=cuda` enables a regular local GPU, and `auto` detects CUDA availability. Keep `zerogpu` for a configured ZeroGPU runtime. For a smaller CPU-only local install, install PyTorch and torchaudio from the CPU wheel index separately; those wheels must not be used on ZeroGPU.
Each CPU worker has a 30-minute timeout. Hosted GPU time is limited to 120 seconds per speech call. The Space uses shared ZeroGPU hardware, without dedicated paid hardware. The local directory remains `hf-speaking-midi`.
## Relationship to the Qwen Space
The [official Qwen3-TTS Space](https://huggingface.co/spaces/Qwen/Qwen3-TTS/blob/6ae5dfa/app.py) was inspected as the backend reference. This app follows its `Qwen3TTSModel.from_pretrained(..., attn_implementation='sdpa')` and `generate_voice_design(text, language, instruct)` flow. Our backend is written separately, supports both startup-loaded ZeroGPU inference and on-demand float32 CPU inference, and limits generation to 512 new tokens. We deliberately use VoiceDesign 1.7B because the 0.6B CustomVoice model does not honor natural-language voice instructions.
## Assets and attribution
The bundled `assets/gm.sf3` is MuseScore General v0.2, with SHA-256 `5b85b6c2c61d10b2b91cddd41efcce7b25cd31c8271d511c73afafbef20b6fa3`. See [its license](assets/MuseScore_General_License.md) for authors, MIT terms, and sample credits. Model weights remain in the upstream Qwen repository and are downloaded at runtime; they are not included here. Qwen3-TTS model license: Apache-2.0. The generated demo assets and their model revision, settings, and hashes are recorded in [examples/manifest.json](examples/manifest.json).
## Deployment
Authenticate using `hf auth login` or set `HF_TOKEN`, then run `python deploy.py`. The script selects ZeroGPU hardware and creates `<authenticated-user>/speaking-midi` and commits the application and assets to that Space. It never upgrades an account or requests paid hardware. See [validation results](VALIDATION.md) for what has actually been tested.

	---
	title: Speaking MIDI
	emoji: 🎹
	colorFrom: indigo
	colorTo: purple
	sdk: gradio
	sdk_version: 6.17.3
	python_version: '3.10'
	app_file: app.py
	pinned: false
	models:
	- Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign
	---

	# Speaking MIDI

	An interactive demonstration of fixed-timbre inverse synthesis: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then listen to the source and play the resulting MIDI in an interactive piano roll. Download the actual MIDI from the same result.

	The two instant-play examples now use the dense algorithm: The Tyger in English and Quiet Night Thoughts in Chinese. Both input modes offer Speech (dense) and Instrumental (sparse) fitting, with 600 initial optimization iterations by default. Speech is the default. The research backtest reduced pooled Chinese CER from 100% to 68.3% and English WER from 116.7% to 50% on five poetry recordings. Recognition remains imperfect, and these results do not establish general human intelligibility. See the [ASR report](https://github.com/k-l-lambda/speaking-midi/blob/5caf4e2/experiments/DENSE_ASR.md).

	## MIDI player

	The demo centers on a piano roll inspired by the interaction design of `klstudio/app/views/midi-player.vue`: horizontal note blocks, active-note highlighting, a playback cursor, click-to-seek, and automatic scrolling at higher zoom. Controls provide play/pause, restart, a seek slider, playback speed, and zoom; left/right arrows on the roll seek by one second. Loading another result stops the previous playback.

	The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; fitting settings and numerical results remain in each job’s `report.json`.

	## Poetry voice examples

	Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s Quiet Night Thoughts, Meng Haoran’s Spring Dawn, Wang Zhihuan’s On the Stork Tower, the opening couplet of Shakespeare’s Sonnet 18, and the first stanza of Blake’s The Tyger. Select an example to load and listen to its recording, then press Generate piano MIDI. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in [the poetry examples](examples/poetry/README.md).

	## Two input modes

	- Audio → MIDI: upload or record sound, select the prefix duration, and fit piano notes. This path never loads a TTS model.
	- Text → speech → MIDI: generate speech with Qwen3-TTS 1.7B VoiceDesign, then run exactly the same fitting pipeline. The default voice description is `一位温柔自信的年轻女性，声音清晰，语气亲切，表达轻快而富有感染力。` (a gentle, confident young woman with clear speech and a warm, lively delivery).

	Inputs are limited to 25 MB, 300 text characters, and a fitted prefix of at most 30 seconds. Longer recordings are read only through the selected prefix. New jobs share one compute queue, with at most eight queued requests. Temporary outputs are ephemeral, are cleaned after 24 hours when another job starts, and disappear when the Space restarts. Do not treat this public demonstration as private storage.

	## Dense fitting algorithm

	The renderer is FluidSynth with bank 0, program 0 of the bundled Grand Piano SoundFont, fixed gain 0.8, and disabled chorus/reverb. The output contains only piano MIDI notes: no source-speech layer, pitch bend, or output EQ.

	1. Convert input to 24 kHz mono and normalize its RMS to 0.035. Analyze squared STFT magnitudes with FFT 1024 / hop 240 and 128 Slaney-normalized mel bands spanning 40–11,000 Hz.
	2. Render temporal templates for all 88 piano keys (MIDI 21–108), at velocity 48 and a 20 ms gate. A 400 ms template window retains attack, decay and release. Cache templates and per-pitch velocity calibration by SoundFont hash and configuration.
	3. Give every key an onset candidate every 20 ms. Jointly optimize nonnegative strengths with projected Adam rather than greedily selecting a small note list. The default is 600 iterations, with a cosine learning-rate schedule from 0.04 to 0.001.
	4. Map strengths to integer velocities using measured energy ratios at 13 velocity levels. Drop strengths below the velocity-8 threshold and events extending beyond the clip. The selected research clips contain 1,218–3,140 notes; the iteration setting is not a note-count limit.
	5. Save MIDI, reload it and render through FluidSynth. The interactive player displays these exported events and plays their corresponding piano WAV, including a 0.5-second release tail.

	For target mel power $M$, temporal templates $D$, and nonnegative note strengths $A$, fitting minimizes:

	$$
	L=\operatorname{mean}\left[\log(1+\widehat M/10^{-3})-\log(1+M/10^{-3})\right]^2,
	\qquad \widehat M=D*A,\quad A\geq0.
	$$

	The convolution sums shifted template powers. It approximates phase interactions, velocity-dependent timbre and repeated-note behavior, so validation uses actual re-rendered MIDI. No transcript or ASR gradient enters the optimizer. ASR selected a configuration on two pilot clips before testing three further poetry clips; repeated renders, temperature-zero decoding and a complete CPU path confirmed the corpus-level improvement. Some conventional spectral metrics worsen, and there is no blind human listening result.

	`Fitting iterations` ranges from 200 to 1,200; 600 is the tested setting. More iterations cost more time and do not guarantee clearer speech. The old note-budget and full-refinement controls have been removed from the demo.

	The dense fitter is vendored without algorithm changes from [Speaking MIDI commit 5caf4e2](https://github.com/k-l-lambda/speaking-midi/tree/5caf4e2). The shared rendering core retains the Space’s portable SoundFont path. The instant examples in `examples/dense` are byte-identical MIDI/WAV copies of the selected research outputs, with normalized source audio for comparison and a hash manifest. Archived sparse examples remain in `examples/en` and `examples/zh` for provenance.

	## Instrumental fitting option

	Select Instrumental (sparse) for music that benefits from fewer attacks and
	longer held notes. Speech (dense) keeps the existing 20 ms speech-oriented
	representation. Fitting style is independent of the input mode, so either is
	available after uploading audio or generating speech. The duration slider accepts
	0.5–30 seconds and defaults to 30; short recordings use their actual length. Instant poetry playback
	remains precomputed dense output, and selecting a poetry input resets the fitting
	style to Speech.

	The instrumental fitter is copied from research commit
	[292f5e0](https://github.com/k-l-lambda/speaking-midi/tree/292f5e0).
	It uses 80/240/640 ms gate templates on a 40 ms onset grid, a 2048-point FFT,
	128 mel bands, and an activation-sparsity penalty of 0.03. The objective combines
	squared log-mel-power error with the sum of nonnegative activations divided by
	analysis frame count. After the selected initial iteration count, competing
	same-key notes are removed and 200 additional iterations refit strengths on the
	remaining support. Energy calibration converts strengths to MIDI velocities;
	notes below velocity 12 are omitted.

	Long inputs are fitted in 12-second sections with 1.2 seconds of context on each
	side. Only central onsets are retained, and same-key overlaps across boundaries
	are shortened. Export preserves the source timeline, including trailing silence.
	The Space adapter reloads the saved MIDI, renders it with the bundled Grand Piano,
	and passes that exact audio and MIDI to the existing player. Both dictionary
	construction and fitting run on CPU. This option does not perform vocal separation.
	It substantially reduced note counts in local music experiments, but improved
	listening quality is not established. See the
	[instrumental report](https://github.com/k-l-lambda/speaking-midi/blob/292f5e0/experiments/INSTRUMENTAL_FITTING.md).

	## CPU and ZeroGPU execution

	The hosted [Speaking MIDI Space](https://huggingface.co/spaces/k-l-lambda/speaking-midi) uses `INFERENCE_DEVICE=zerogpu`. VoiceDesign loads at startup with CUDA placement and bfloat16, following the ZeroGPU model-loading contract. A `@spaces.GPU(duration=120)` function requests a shared GPU only for speech generation. MIDI fitting runs in a separate CPU worker and does not consume GPU quota. ZeroGPU requests can queue or fail when the visitor's GPU quota is exhausted; the precomputed speech examples can still be fitted without generating another voice.

	The requirements use `torch==2.10.0` and matching torchaudio. Do not add `+cpu` or the CPU-only wheel index to the hosted requirements: ZeroGPU rejects that version specification and needs GPU-capable PyTorch.

	Local execution defaults to `INFERENCE_DEVICE=cpu`. TTS uses float32 with SDPA in a separate process, releasing model memory before MIDI fitting. Both fitting styles explicitly use CPU in every app mode, including ZeroGPU. Both template creation and optimization work without a GPU.

	```bash
	# Python 3.10; install libfluidsynth3, libsndfile1 and sox using your OS package manager.
	python -m venv .venv
	. .venv/bin/activate
	pip install -r requirements.txt gradio==6.17.3
	INFERENCE_DEVICE=cpu python app.py
	```

	Optional variables: `QWEN_MODEL_PATH` selects a local VoiceDesign snapshot; `PORT` defaults to 7860; `OMP_NUM_THREADS` defaults to 2. `INFERENCE_DEVICE=cuda` enables a regular local GPU, and `auto` detects CUDA availability. Keep `zerogpu` for a configured ZeroGPU runtime. For a smaller CPU-only local install, install PyTorch and torchaudio from the CPU wheel index separately; those wheels must not be used on ZeroGPU.

	Each CPU worker has a 30-minute timeout. Hosted GPU time is limited to 120 seconds per speech call. The Space uses shared ZeroGPU hardware, without dedicated paid hardware. The local directory remains `hf-speaking-midi`.

	## Relationship to the Qwen Space

	The [official Qwen3-TTS Space](https://huggingface.co/spaces/Qwen/Qwen3-TTS/blob/6ae5dfa/app.py) was inspected as the backend reference. This app follows its `Qwen3TTSModel.from_pretrained(..., attn_implementation='sdpa')` and `generate_voice_design(text, language, instruct)` flow. Our backend is written separately, supports both startup-loaded ZeroGPU inference and on-demand float32 CPU inference, and limits generation to 512 new tokens. We deliberately use VoiceDesign 1.7B because the 0.6B CustomVoice model does not honor natural-language voice instructions.

	## Assets and attribution

	The bundled `assets/gm.sf3` is MuseScore General v0.2, with SHA-256 `5b85b6c2c61d10b2b91cddd41efcce7b25cd31c8271d511c73afafbef20b6fa3`. See [its license](assets/MuseScore_General_License.md) for authors, MIT terms, and sample credits. Model weights remain in the upstream Qwen repository and are downloaded at runtime; they are not included here. Qwen3-TTS model license: Apache-2.0. The generated demo assets and their model revision, settings, and hashes are recorded in [examples/manifest.json](examples/manifest.json).

	## Deployment

	Authenticate using `hf auth login` or set `HF_TOKEN`, then run `python deploy.py`. The script selects ZeroGPU hardware and creates `<authenticated-user>/speaking-midi` and commits the application and assets to that Space. It never upgrades an account or requests paid hardware. See [validation results](VALIDATION.md) for what has actually been tested.