File size: 6,135 Bytes
bc3f652
 
 
 
 
 
 
 
 
 
7a3d495
 
 
 
 
 
 
 
 
 
 
 
59e4843
 
 
 
f97fef1
 
 
 
 
 
 
 
 
f5a62b9
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
# Validation

Checked on 2026-10-01 with Python 3.10, Gradio 6.17.3, Qwen-TTS 0.1.1, PyTorch 2.10.0 and FluidSynth 2.2.5.

- Six automated tests passed: input validation/prefix reading, actual CPU audio fitting, both packaged examples, MIDI timing/audio round-trip, known-note dictionary recovery, and cached objective equivalence.
- Real CPU VoiceDesign generation with float32, SDPA and the requested voice description produced a 1.12-second `Hello, piano.` clip in 8.60 seconds of generation time, excluding loading. Snapshot: `5ecdb67327fd37bb2e042aab12ff7391903235d3`.
- Local Gradio API exercised both modes. A 0.5-second uploaded prefix with 20 search iterations completed in 7.2 seconds; text → speech → MIDI completed in 29.1 seconds, including model loading and fitting. Both ran on CPU. These are local workstation measurements, not hosted CPU-basic benchmarks.
- Chrome/Playwright checked example loading, input mode switching and 390-pixel mobile layout. No JavaScript errors or horizontal page overflow were observed. The desktop screenshot was visually inspected.
- Existing English and Chinese eight-second demos were generated on GPU and are labeled as precomputed. Their generation and fit provenance remain in `examples/manifest.json`.

## ZeroGPU deployment

The initial CPU-basic creation attempt returned HTTP 402, but ZeroGPU creation succeeded. The Space is now `k-l-lambda/speaking-midi`; the local folder remains `hf-speaking-midi`. The requirements use supported `torch==2.10.0` and `torchaudio==2.10.0` instead of CPU-only wheel pins. The container built successfully and HF reported `RUNNING` on `zero-a10g`.

The six original tests passed after the change. An additional local test mocks speech generation and uses the real MIDI fitter, verifying that ZeroGPU speech bypasses the CPU TTS subprocess while fitting stays on CPU.

Actual hosted API checks passed on 2026-10-01:

- Audio → MIDI: the first 0.5 seconds of the Quiet Night Thoughts recording, 20 search iterations, quick refinement; completed in 9.8 seconds.
- Text → speech → MIDI: `Hello, piano.`, English, the requested female voice description, followed by fitting a 0.5-second prefix with 20 search iterations; completed in 8.6 seconds. Speech used the decorated ZeroGPU function; MIDI fitting used CPU.

These timings are the app's reported job durations, excluding client upload/download and queue transit. They validate short jobs, not the maximum 12-second/full-refinement configuration. The hosted PyTorch configuration error is resolved. No subscription or dedicated paid hardware was requested.

## Interactive MIDI player (2026-10-02)

The demo now shows source audio and an interactive piano roll instead of diagnostic tabs. Seven backend tests passed, including real CPU fitting and the ZeroGPU dispatch test. Browser checks passed for playback and pause, click and keyboard seeking, zoom and scrolling, replacing a playing example, removal of diagnostic tabs, and mobile layout. No JavaScript errors were observed in those checks. The player decodes note events from the downloaded MIDI and synchronizes them to that MIDI's Grand Piano audio render.

## Dense fitting deployment (2026-10-02)

- Vendored `speaking_midi/dense.py` is byte-identical to research commit `5caf4e2`; the portable bundled-SoundFont renderer is retained.
- Ten automated tests passed, including an actual CPU audio job, mocked ZeroGPU speech dispatch followed by real CPU dense fitting, single-note dense recovery, input validation, MIDI round-trip and packaged example loading.
- A local Gradio API job fitted the complete 8.64-second Quiet Night recording at 600 iterations, producing 2,758 notes in 14.1 seconds. Small floating-point/renderer differences can change the discrete notes; these are workstation timings, not hosted benchmarks.
- Chrome/Playwright verified the 3,140-note English example and 2,759-note Chinese example, playback/pause, replacement while loaded, updated fitting controls, and a 390-pixel mobile layout. No JavaScript errors or horizontal overflow were observed.
- The instant-play MIDI and WAV assets are exact copies of the dense research examples. Their hashes and research revision are in `examples/dense/manifest.json`. Both input modes now invoke the same dense CPU worker; the old note-budget/refinement controls are removed.
- Earlier hosted measurements above describe the previous sparse implementation and must not be treated as dense-performance measurements.

## Instrumental option and 30-second clips — 2026-10-03

- Vendored `speaking_midi/instrumental.py` and its tests from research commit
  `292f5e0`. The separate `instrumental_demo` adapter writes the normalized target,
  preserves MIDI duration, reloads MIDI for actual Grand Piano rendering, and
  supplies the existing piano-roll payload. No vocal-separation code was imported.
- Added an independent Speech (dense) / Instrumental (sparse) fitting-style control
  to both input modes. Both run on CPU. Expanded UI and request validation from
  12 to 30 seconds; poetry selection still uses 12 seconds and resets Speech mode.
- All 14 tests passed locally (112.35 s). A real 31-second synthetic input was
  truncated to 30 seconds and fitted on CPU in three contextual sections; the
  resulting player audio is 30.5 seconds including release. Invalid style and
  durations above 30 seconds are rejected. Existing dense fitting, precomputed
  examples and mocked ZeroGPU TTS dispatch followed by real CPU fitting passed.
- A Chrome/Playwright check generated Instrumental output from Quiet Night through
  the Gradio UI: 674 notes, reported completion in 31.7 seconds. MIDI download,
  30-second slider maximum, desktop and 390 px mobile layout passed, with no
  JavaScript errors or horizontal document overflow. Used system Google Chrome
  because the Playwright package's expected bundled browser revision was absent.
- Local evidence is ignored under `outputs/instrumental-tests.log`,
  `instrumental-browser.log`, and the desktop/mobile screenshots. Timings are
  observations from this workstation, not hosted throughput guarantees.