sujitvasanth/piper-mnn-cpugpu-inputoutput-streaming

Piper MNN — CPU/GPU, streaming input and output

A Piper voice (en_US-data_7024-medium, 22050 Hz) split into two fixed-shape MNN models so it can run entirely on CPU or GPU (MNN OpenCL) with no graph rebuilding, and accept text word by word (for example straight from an LLM) while audio streams out.

Files

File	What it is	SHA-256
data.encoder.fixed64x160.static.mnn	Encoder: 64 phoneme IDs → 160 latent frames + per-ID durations	31a85913846c7c4cefee0db9bf46c917be9558c34aff99edcb72da032ab75584
data.decoder.fixed64.mnn	Decoder: 64 latent frames → 16384 samples (256 per frame)	390d1c8037a82dfa48e34c2b0dd9e128627cfe10c4823e9e4778a9056de7d0af
data.onnx.json	Piper voice config (phoneme ID map, sample rate, default scales)
onnx/	The same two models in ONNX
scripts/	Export scripts, Python reference implementation, C++ streaming engine

Encoder — data.encoder.fixed64x160.static.mnn

Inputs:

• input int32 [1, 64] — phoneme IDs, zero-padded
• input_lengths int32 [1] — number of real IDs
• scales float [3] — noise_scale, length_scale, noise_w (voice default 0.667, 1.0, 0.8)
• noise_dp float [1, 2, 64] — standard normal noise for the duration predictor
• noise_z float [1, 192, 160] — standard normal noise for the latent prior

Outputs:

• z float [1, 192, 160] — latent frames (frames past frames are masked to zero)
• durations float [1, 1, 64] — frames per input ID
• frames float [1] — real frame count (speech longer than 160 frames is cut off)

Changes from the stock Piper export:

• fixed input/output sizes (total frame count is a constant, so no data-dependent shapes)
• the two random-noise nodes are inputs, so overlapping windows can reuse the same noise
• the duration predictor's spline flows no longer use NonZero/GatherND/ScatterND with data-dependent sizes; they run densely with masks. Output is bit-identical to the original.

Decoder — data.decoder.fixed64.mnn

Inputs: z float [1, 192, 64] and masks valid_mask_1/8/64/256 float [1, 1, 64×scale] (1 where the frame is real, 0 for padding). Output: output float [1, 1, 1, 16384].

Use 12 frames of context on each side and keep the middle 40 frames (40 × 256 samples). Assembled chunks match whole-utterance decoding to about 7×10⁻⁷ relative error.

Streaming text in

1. Phonemize the text so far (eSpeak NG, as Piper does) and turn each new word into IDs: a leading space ID (or ^ for the first word), then each phoneme followed by _. End of text adds $.
2. Window = 2 words of left context + new words, padded to 64 IDs.
3. Run the encoder; use durations to find frame offsets.
4. Commit audio up to the last word boundary that has 1 whole word of look-ahead after it (and keeps 12 frames of right context for the decoder).
5. Decode the committed frames in 64-frame chunks; play them.
6. Next window starts 2 words before the commit point.

Tested settings (Android, GPU): commit 2–3 words per window (3 sounds more coherent), no pre-buffer, 2-frame (≈23 ms) crossfade between windows. On a desktop CPU each window is about 17 ms encoder + 17 ms decoder.

See scripts/word_stream_piper.cpp for the C++ engine and scripts/stream_text_windows.py for the Python reference.

Source

Derived from the Piper voice en_US-data_7024-medium (see data.onnx.json). Check that voice's original licence before redistributing.