README.md

metadata

title: Speaking MIDI
emoji: 🎹
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 6.17.3
python_version: '3.10'
app_file: app.py
pinned: false
models:
  - Qwen/Qwen3-TTS-12Hz-1.7B-VoiceDesign

Speaking MIDI

An interactive demonstration of fixed-timbre inverse synthesis: approximate speech or other audio using only Grand Piano MIDI notes. Upload/record audio, or generate a voice from text with Qwen3-TTS, then listen to the source and play the resulting MIDI in an interactive piano roll. Download the actual MIDI from the same result.

The two instant-play examples now use the dense algorithm: The Tyger in English and Quiet Night Thoughts in Chinese. Both input modes offer Speech (dense) and Instrumental (sparse) fitting, with 600 initial optimization iterations by default. Speech is the default. The research backtest reduced pooled Chinese CER from 100% to 68.3% and English WER from 116.7% to 50% on five poetry recordings. Recognition remains imperfect, and these results do not establish general human intelligibility. See the ASR report.

MIDI player

The demo centers on a piano roll inspired by the interaction design of klstudio/app/views/midi-player.vue: horizontal note blocks, active-note highlighting, a playback cursor, click-to-seek, and automatic scrolling at higher zoom. Controls provide play/pause, restart, a seek slider, playback speed, and zoom; left/right arrows on the roll seek by one second. Loading another result stops the previous playback.

The displayed notes are decoded directly from the exported MIDI. Playback uses the corresponding FluidSynth Grand Piano render, with the audio clock driving the cursor and highlights; it does not substitute a different browser synthesizer. The render includes release tails, while note blocks show MIDI key-hold durations. The browser receives a self-contained audio-and-note payload and needs no external JavaScript or sound library. Spectrogram, static roll, baseline player, and metric panels have been removed from the demo interface; fitting settings and numerical results remain in each job’s report.json.

Poetry voice examples

Five ready-made Qwen VoiceDesign recordings cover Chinese and English public-domain poetry: Li Bai’s Quiet Night Thoughts, Meng Haoran’s Spring Dawn, Wang Zhihuan’s On the Stork Tower, the opening couplet of Shakespeare’s Sonnet 18, and the first stanza of Blake’s The Tyger. Select an example to load and listen to its recording, then press Generate piano MIDI. Selection switches to audio mode and sets the fitting window to 12 seconds. These are speech inputs; their MIDI is generated on demand. Text, authorship, durations, and generation provenance are available in the poetry examples.

Two input modes

• Audio → MIDI: upload or record sound, select the prefix duration, and fit piano notes. This path never loads a TTS model.
• Text → speech → MIDI: generate speech with Qwen3-TTS 1.7B VoiceDesign, then run exactly the same fitting pipeline. The default voice description is 一位温柔自信的年轻女性，声音清晰，语气亲切，表达轻快而富有感染力。 (a gentle, confident young woman with clear speech and a warm, lively delivery).

Inputs are limited to 25 MB, 300 text characters, and a fitted prefix of at most 30 seconds. Longer recordings are read only through the selected prefix. New jobs share one compute queue, with at most eight queued requests. Temporary outputs are ephemeral, are cleaned after 24 hours when another job starts, and disappear when the Space restarts. Do not treat this public demonstration as private storage.

Dense fitting algorithm

The renderer is FluidSynth with bank 0, program 0 of the bundled Grand Piano SoundFont, fixed gain 0.8, and disabled chorus/reverb. The output contains only piano MIDI notes: no source-speech layer, pitch bend, or output EQ.

1. Convert input to 24 kHz mono and normalize its RMS to 0.035. Analyze squared STFT magnitudes with FFT 1024 / hop 240 and 128 Slaney-normalized mel bands spanning 40–11,000 Hz.
2. Render temporal templates for all 88 piano keys (MIDI 21–108), at velocity 48 and a 20 ms gate. A 400 ms template window retains attack, decay and release. Cache templates and per-pitch velocity calibration by SoundFont hash and configuration.
3. Give every key an onset candidate every 20 ms. Jointly optimize nonnegative strengths with projected Adam rather than greedily selecting a small note list. The default is 600 iterations, with a cosine learning-rate schedule from 0.04 to 0.001.
4. Map strengths to integer velocities using measured energy ratios at 13 velocity levels. Drop strengths below the velocity-8 threshold and events extending beyond the clip. The selected research clips contain 1,218–3,140 notes; the iteration setting is not a note-count limit.
5. Save MIDI, reload it and render through FluidSynth. The interactive player displays these exported events and plays their corresponding piano WAV, including a 0.5-second release tail.

For target mel power $M$, temporal templates $D$, and nonnegative note strengths $A$, fitting minimizes:

$$L=\mathrm{mean}{[\log (1+\widehat{M}\mathrm{/}{10}^{-3})-\log (1+M\mathrm{/}{10}^{-3})]}^2,\widehat{M}=D\ast A,A\ge 0.L=\backslash operatorname\{mean\}\backslash left[\backslash log(1+\backslash widehat\; M/10^\{-3\})-\backslash log(1+M/10^\{-3\})\backslash right]^2,\; \backslash qquad\; \backslash widehat\; M=D*A,\backslash quad\; A\backslash geq0.$$

The convolution sums shifted template powers. It approximates phase interactions, velocity-dependent timbre and repeated-note behavior, so validation uses actual re-rendered MIDI. No transcript or ASR gradient enters the optimizer. ASR selected a configuration on two pilot clips before testing three further poetry clips; repeated renders, temperature-zero decoding and a complete CPU path confirmed the corpus-level improvement. Some conventional spectral metrics worsen, and there is no blind human listening result.

Fitting iterations ranges from 200 to 1,200; 600 is the tested setting. More iterations cost more time and do not guarantee clearer speech. The old note-budget and full-refinement controls have been removed from the demo.

The dense fitter is vendored without algorithm changes from Speaking MIDI commit 5caf4e2. The shared rendering core retains the Space’s portable SoundFont path. The instant examples in examples/dense are byte-identical MIDI/WAV copies of the selected research outputs, with normalized source audio for comparison and a hash manifest. Archived sparse examples remain in examples/en and examples/zh for provenance.

Instrumental fitting option

Select Instrumental (sparse) for music that benefits from fewer attacks and longer held notes. Speech (dense) keeps the existing 20 ms speech-oriented representation. Fitting style is independent of the input mode, so either is available after uploading audio or generating speech. The duration slider accepts 0.5–30 seconds and defaults to 30; short recordings use their actual length. Instant poetry playback remains precomputed dense output, and selecting a poetry input resets the fitting style to Speech.

The instrumental fitter is copied from research commit 292f5e0. It uses 80/240/640 ms gate templates on a 40 ms onset grid, a 2048-point FFT, 128 mel bands, and an activation-sparsity penalty of 0.03. The objective combines squared log-mel-power error with the sum of nonnegative activations divided by analysis frame count. After the selected initial iteration count, competing same-key notes are removed and 200 additional iterations refit strengths on the remaining support. Energy calibration converts strengths to MIDI velocities; notes below velocity 12 are omitted.

Long inputs are fitted in 12-second sections with 1.2 seconds of context on each side. Only central onsets are retained, and same-key overlaps across boundaries are shortened. Export preserves the source timeline, including trailing silence. The Space adapter reloads the saved MIDI, renders it with the bundled Grand Piano, and passes that exact audio and MIDI to the existing player. Both dictionary construction and fitting run on CPU. This option does not perform vocal separation. It substantially reduced note counts in local music experiments, but improved listening quality is not established. See the instrumental report.

CPU and ZeroGPU execution

The hosted Speaking MIDI Space uses INFERENCE_DEVICE=zerogpu. VoiceDesign loads at startup with CUDA placement and bfloat16, following the ZeroGPU model-loading contract. A @spaces.GPU(duration=120) function requests a shared GPU only for speech generation. MIDI fitting runs in a separate CPU worker and does not consume GPU quota. ZeroGPU requests can queue or fail when the visitor's GPU quota is exhausted; the precomputed speech examples can still be fitted without generating another voice.

The requirements use torch==2.10.0 and matching torchaudio. Do not add +cpu or the CPU-only wheel index to the hosted requirements: ZeroGPU rejects that version specification and needs GPU-capable PyTorch.

Local execution defaults to INFERENCE_DEVICE=cpu. TTS uses float32 with SDPA in a separate process, releasing model memory before MIDI fitting. Both fitting styles explicitly use CPU in every app mode, including ZeroGPU. Both template creation and optimization work without a GPU.

# Python 3.10; install libfluidsynth3, libsndfile1 and sox using your OS package manager.
python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt gradio==6.17.3
INFERENCE_DEVICE=cpu python app.py

Optional variables: QWEN_MODEL_PATH selects a local VoiceDesign snapshot; PORT defaults to 7860; OMP_NUM_THREADS defaults to 2. INFERENCE_DEVICE=cuda enables a regular local GPU, and auto detects CUDA availability. Keep zerogpu for a configured ZeroGPU runtime. For a smaller CPU-only local install, install PyTorch and torchaudio from the CPU wheel index separately; those wheels must not be used on ZeroGPU.

Each CPU worker has a 30-minute timeout. Hosted GPU time is limited to 120 seconds per speech call. The Space uses shared ZeroGPU hardware, without dedicated paid hardware. The local directory remains hf-speaking-midi.

Relationship to the Qwen Space

The official Qwen3-TTS Space was inspected as the backend reference. This app follows its Qwen3TTSModel.from_pretrained(..., attn_implementation='sdpa') and generate_voice_design(text, language, instruct) flow. Our backend is written separately, supports both startup-loaded ZeroGPU inference and on-demand float32 CPU inference, and limits generation to 512 new tokens. We deliberately use VoiceDesign 1.7B because the 0.6B CustomVoice model does not honor natural-language voice instructions.

Assets and attribution

The bundled assets/gm.sf3 is MuseScore General v0.2, with SHA-256 5b85b6c2c61d10b2b91cddd41efcce7b25cd31c8271d511c73afafbef20b6fa3. See its license for authors, MIT terms, and sample credits. Model weights remain in the upstream Qwen repository and are downloaded at runtime; they are not included here. Qwen3-TTS model license: Apache-2.0. The generated demo assets and their model revision, settings, and hashes are recorded in examples/manifest.json.

Deployment

Authenticate using hf auth login or set HF_TOKEN, then run python deploy.py. The script selects ZeroGPU hardware and creates <authenticated-user>/speaking-midi and commits the application and assets to that Space. It never upgrades an account or requests paid hardware. See validation results for what has actually been tested.