Text-to-Speech
Safetensors
English
steertts
steer-tts
tts
speech-synthesis
english
controllable-tts
steerable-tts
prosody-control
expressive-tts
style-control
inline-markup
word-emphasis
zero-shot-voice-cloning
voice-cloning
interpretable
non-autoregressive
fastpitch
mamba
mamba-3
state-space-model
hybrid-attention
flow-matching
vocos
small-model
quantized
int8
on-device
streaming
non-commercial
Eval Results (legacy)
| language: | |
| - en | |
| license: cc-by-nc-4.0 | |
| library_name: steertts | |
| pipeline_tag: text-to-speech | |
| tags: | |
| - text-to-speech | |
| - tts | |
| - speech-synthesis | |
| - english | |
| - controllable-tts | |
| - steerable-tts | |
| - prosody-control | |
| - expressive-tts | |
| - style-control | |
| - inline-markup | |
| - word-emphasis | |
| - zero-shot-voice-cloning | |
| - voice-cloning | |
| - interpretable | |
| - non-autoregressive | |
| - fastpitch | |
| - mamba | |
| - mamba-3 | |
| - state-space-model | |
| - hybrid-attention | |
| - flow-matching | |
| - vocos | |
| - small-model | |
| - quantized | |
| - int8 | |
| - on-device | |
| - streaming | |
| - non-commercial | |
| model-index: | |
| - name: steer-tts | |
| results: | |
| - task: | |
| type: text-to-speech | |
| name: Text-to-Speech | |
| dataset: | |
| name: Seed-TTS-eval (English) | |
| type: bezzam/seed_tts_eval | |
| split: test | |
| metrics: | |
| - type: wer | |
| value: 1.0 | |
| name: WER % (voice_1361, the v1.0.0 default, Qwen3-ASR-1.7B) | |
| verified: false | |
| - type: wer | |
| value: 1.01 | |
| name: WER % (zero-shot cloning, prompted flow) | |
| verified: false | |
| - type: speaker_similarity | |
| value: 37.4 | |
| name: SIM x100 (zero-shot cloning, WavLM-large) | |
| verified: false | |
| source: | |
| url: https://huggingface.co/spaces/hf-audio/open_tts_leaderboard | |
| name: Open TTS Leaderboard protocol (local run, not an official submission) | |
| - task: | |
| type: text-to-speech | |
| name: Text-to-Speech | |
| dataset: | |
| name: CV3-Eval (English, zero-shot) | |
| type: bezzam/cv3_eval | |
| split: test | |
| metrics: | |
| - type: wer | |
| value: 2.63 | |
| name: WER % (voice_1361, the v1.0.0 default, Qwen3-ASR-1.7B) | |
| verified: false | |
| - type: wer | |
| value: 2.73 | |
| name: WER % (zero-shot cloning, prompted flow) | |
| verified: false | |
| - type: speaker_similarity | |
| value: 28.7 | |
| name: SIM x100 (zero-shot cloning, WavLM-large) | |
| verified: false | |
| source: | |
| url: https://huggingface.co/spaces/hf-audio/open_tts_leaderboard | |
| name: Open TTS Leaderboard protocol (local run, not an official submission) | |
| > Mirror of [WhissleAI/steer-tts](https://huggingface.co/WhissleAI/steer-tts). All credit to the original authors. Not an official release from them. License unchanged: code MIT, weights and voice data CC BY-NC 4.0 (non-commercial only); see `LICENSE.md`. | |
| # SteerTTS v1 | |
| A small English text-to-speech model that you steer with numbers. Each request can set any subset of **12 acoustic | |
| controls**, for example speaking rate, loudness, median pitch, pitch range, final rise or fall, pauses, spectral tilt | |
| and breathiness. Each control is a z-score relative to the chosen voice's own neutral speech. You can also pick a | |
| **style tag**, give a **free-text direction** (which a small adapter turns into a request you can inspect), or use | |
| **inline markup**. The model can **clone** a voice from a short reference clip. | |
| | | | | |
| |---|---| | |
| | Acoustic model | **9.59M** parameters (encoder/decoder: attention + bidirectional Mamba-3, AdaLN conditioning) | | |
| | + Vocos vocoder | 13.53M (`charactr/vocos-mel-24khz`, frozen), **23.1M** total | | |
| | Flow-matching refiner (optional, on by default) | 17.64M β 40.8M total | | |
| | Prompted flow (cloning only) | 24.67M, used in place of the refiner when `ref_wav` is given β 47.8M total | | |
| | Frozen external encoders | WavLM-base-plus-sv 101M (cloning only), all-MiniLM-L6-v2 22.7M (free-text prompts only) | | |
| | Output | 24 kHz mono, fp32, CPU or CUDA | | |
| | Voices | 2,825 built-in voices under anonymised names (`voice_0001` β¦), default `voice_1688` (v1.0.0: `voice_1361`) | | |
| > **Licence: weights CC BY-NC 4.0 (non-commercial use only); package code MIT.** The weights were trained partly on | |
| > non-commercial data (see [Training data](#training-data)). | |
| ## Listen | |
| All samples are raw model output (acoustic model + flow refiner + Vocos, seed 0), rendered with the code in this repo. | |
| **New default voice.** v1.0.1 switched the default from `voice_1361` to `voice_1688`, which sounds more natural and less rushed. | |
| | Text | `voice_1361` (v1.0.0 default) | `voice_1688` (v1.0.1 default) | | |
| |---|---|---| | |
| | "Hi, thanks for calling. I can help you reschedule your appointment today." | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_intro_voice_1361.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_intro_voice_1688.mp3"></audio> | | |
| | "The weather was cold, but the view from the top was worth every single step." | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_story_voice_1361.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_story_voice_1688.mp3"></audio> | | |
| **Steering `voice_1688`.** | |
| | Request | Audio | | |
| |---|---| | |
| | "Are you coming with us tonight?" (no controls) | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_plain.mp3"></audio> | | |
| | Same text, `ctrl={"f0_end": 2, "f0_iqr": 1.5}`: rising, livelier | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_question_rise.mp3"></audio> | | |
| | "Please keep this between the two of us." `ctrl={"rate": -1.5, "loud": -1.5}` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_slow_soft.mp3"></audio> | | |
| | Same text, `tag="whisper"` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_whisper_tag.mp3"></audio> | | |
| | "I really thought it would work this time." `tag="sadness"`, `guidance=1.5` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_sad_tag.mp3"></audio> | | |
| | "I can't believe we actually won!" `prompt="excited, a bit faster"` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_prompt.mp3"></audio> | | |
| | Markup: `Well, *that* was fast. [pause 400ms] <slow>Let me think about it.</slow> <whisper>Don't tell anyone.</whisper>` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_markup.mp3"></audio> | | |
| Found a bug or a voice that sounds wrong? Open an issue at [github.com/WhissleAI/steer-tts](https://github.com/WhissleAI/steer-tts/issues) or start a discussion in the Community tab. | |
| ## Quick start | |
| ```bash | |
| git clone https://huggingface.co/WhissleAI/steer-tts && cd steer-tts # or huggingface_hub.snapshot_download("WhissleAI/steer-tts") | |
| pip install ".[all]" # core: torch, vocos, g2p_en β¦; [all] adds transformers (prompts, cloning), parselmouth (measuring), sounddevice (playback) | |
| # or, code only (weights download on first use): pip install "steertts[all] @ git+https://github.com/WhissleAI/steer-tts" | |
| steertts say "Hello there, nice to meet you." -o hello.wav | |
| steertts say "I can't believe it worked." --tag happiness --ctrl rate=1,f0_iqr=1.5 -o happy.wav | |
| steertts say "Please keep this quiet." --prompt "whispering, slow and secretive" -o secret.wav # prints the resolved request | |
| steertts stream "Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow>" -o markup.wav | |
| steertts play "Hi there. <whisper>This plays as it is rendered.</whisper>" # streams to your speakers | |
| steertts say "Now in your voice." --ref-wav consenting_speaker.wav -o clone.wav | |
| steertts voices --gender female | head | |
| steertts --precision int8 say "Same voice, a quarter of the download." -o int8.wav # int8 weights (see Quantized int8) | |
| ``` | |
| The CLI looks for the model in the current directory. From anywhere else, pass it before the command | |
| (`steertts --model WhissleAI/steer-tts say "Hi." -o hi.wav`) or set `STEERTTS_MODEL`. | |
| ```python | |
| from steertts import SteerTTS | |
| tts = SteerTTS.from_pretrained("WhissleAI/steer-tts") # HF repo id or a local dir; device: cuda if available, else cpu | |
| wav = tts("The meeting moved to Thursday.") # float32 numpy, 24 kHz, default voice voice_1688 | |
| wav = tts("Are you serious?", voice="voice_0042", ctrl={"f0_end": 2.0, "f0_iqr": 1.5}) # rising, lively | |
| wav = tts("It is over.", tag="sadness", ctrl={"loud": -1}, guidance=1.5) | |
| req = tts.resolve({"text": "See you soon.", "prompt": "excited, a bit faster"}) | |
| print(req) # {'text': ..., 'voice': 'voice_1688', 'tag': 'excited', 'ctrl': {'rate': 0.29}, 'prompt_mapping': {...}, ...} | |
| wav = tts.synthesize(req) # a resolved request renders exactly as the prompt would | |
| from steertts.player import Player # plays chunks as they arrive (sounddevice, else ffplay / aplay) | |
| player = Player(tts.sample_rate) | |
| for chunk in tts.stream("First sentence. Second sentence."): # sentence-level streaming | |
| player.play(chunk.audio) | |
| player.close() # waits until playback finishes | |
| tts.save(wav, "out.wav") | |
| tts8 = SteerTTS.from_pretrained("WhissleAI/steer-tts", precision="int8") # downloads only int8/ (52 MB, not 215 MB) | |
| ``` | |
| `inference.py` renders one example of each feature. `python -m steertts.cli --help` lists the CLI. | |
| ## The request | |
| Every synthesis is one JSON object. It is documented field by field in [`request.schema.json`](request.schema.json) | |
| (JSON Schema 2020-12; `steertts schema` prints it). Only `text` is required. | |
| | Field | Meaning | | |
| |---|---| | |
| | `text` | English text. Numbers, units, currency, ordinals, ranges and unknown acronyms are spelled out. | | |
| | `voice` / `ref_wav` | A built-in voice, **or** a reference clip to clone (first 8 s, at least 0.5 s). | | |
| | `tag` | One of 51 style tags (`none`, `whisper`, `sadness`, `anger`, `fast`, `high_pitch`, β¦). | | |
| | `ctrl` | `{control: z}` over the 12 controls below, each in [-5, 5]. | | |
| | `prompt` | Free-text direction. `resolve()` maps it to `tag` + `ctrl`; an explicit `tag` or `ctrl` dim wins. | | |
| | `guidance` | Steering scale s (default 1). See [Guidance](#how-well-it-steers). | | |
| | `pace` | Global tempo divisor (>1 is faster). It does not change the style. | | |
| | `word_shift` | `{word_index: {pitch: st, energy: dB, dur: Γ}}`, per-word prosody edits. Not with `markup` or `stream()` (they raise an error); use inline markup there. | | |
| | `markup` | Parse inline markup in `text` (see below). `stream()` always parses it. | | |
| | `flow`, `seed` | Use the flow refiner (default true) and set the sampler seed. Renders are deterministic per device. | | |
| **Controls and the mask.** The 12 controls are utterance-level acoustic measurements: | |
| - `rate`: articulation rate | |
| - `loud`: speech level | |
| - `f0_med`: median pitch | |
| - `f0_iqr`: pitch range | |
| - `f0_end`: final pitch slope | |
| - `pause`: pause share | |
| - `tilt`: spectral tilt | |
| - `voiced`: voiced share | |
| - `flat`: spectral flatness | |
| - `centroid`: spectral centroid | |
| - `cpps`: cepstral peak prominence | |
| - `h1h2`: H1βH2 | |
| Each value is a z-score against the chosen voice's own neutral speech, in units of the pooled within-speaker SD | |
| (`control_stats.json`). The model also receives an availability mask: | |
| - A dim named in `ctrl` is requested. | |
| - A dim left out is unspecified, and the model chooses it from voice, tag and text. | |
| - `{"rate": 0}` asks for this voice's neutral rate. Omitting `rate` lets a `fast` tag or the text decide. | |
| `steertts measure out.wav --text "β¦" --voice voice_1688` reads the controls back with the same measurement code | |
| that labelled the training data. | |
| **Markup** (`markup: true` or `stream()`): | |
| | Markup | Effect | | |
| |---|---| | |
| | `*word*` or `<emph>β¦</emph>` | Emphasis: +2.5 st, +4 dB, Γ1.2 duration | | |
| | `<slow> <fast> <high> <low> <loud> <soft>` | Per-word changes; the sentence is still rendered in one pass | | |
| | `[pause]`, `[pause 300ms]`, `[pause 1.5s]` | Exact silence | | |
| | `<whisper>β¦</whisper>` (any tag), `<style prompt="β¦">β¦</style>` | The span is rendered as its own segment | | |
| Text is rendered sentence by sentence, so the first sentence can play while the rest renders. | |
| ## Architecture | |
| FastPitch-style explicit-variance TTS: | |
| 1. **Front end.** g2p_en ARPAbet phonemes with word indices. | |
| 2. **Encoder.** 4 blocks. | |
| 3. **Variance predictors.** Per-phone duration, pitch and energy. | |
| 4. **Length regulator.** | |
| 5. **Decoder.** 4 blocks. | |
| 6. **Output.** A 100-band log-mel, rendered by Vocos. | |
| Each encoder/decoder block is a **hybrid**: multi-head self-attention, then a **bidirectional Mamba-3** branch (two | |
| Mamba-3 layers run forward and backward and summed; complex-valued state, trapezoidal discretisation; 3.70M | |
| parameters in total), then a conv FFN. | |
| Every LayerNorm in every block and predictor is an **adaptive LayerNorm**. Its scale and shift are predicted from | |
| g = MLP([speaker vector, tag embedding, ctrl β mask, mask]). The speaker slot is a linear projection of a frozen | |
| 512-d WavLM-base-plus-sv x-vector. Built-in voices use their stored mean vector; cloning uses the vector of the | |
| reference clip. | |
| **Guidance** renders the request and the uninstructed request, then extrapolates durations, pitch, energy and mel: | |
| out = outβ + sΒ·(out β outβ). | |
| Optional decoders: | |
| - **Flow refiner.** A conditional flow-matching ConvNeXt network (OT-CFM, 10 midpoint steps, temperature 0.4). It | |
| samples mel detail given the acoustic model's mel and g, so every control still comes from the acoustic model. | |
| - **Prompted flow.** The same network plus attention, given the reference clip's own mel as context | |
| (Voicebox-style infilling). Used for cloning. | |
| The `steertts` package here is self-contained. It is the inference subset of the training code, operation for | |
| operation, and its outputs are bit-identical to the training repository's synthesis on the same device (see | |
| [Release verification](#release-verification)). | |
| ## How well it steers | |
| The numbers below are **held-out**: 16 LibriTTS-R test-clean speakers who share no speaker, chapter or recording | |
| with the training data (8 female, 8 male by a median-F0 rule; the corpus has no gender labels on disk), each | |
| conditioned on the speaker vector of one recording, Γ 8 texts. They are for the acoustic model at guidance 1 **without the flow refiner**. | |
| Steering through the refiner was not measured in the paper; the release check below confirms only the direction. | |
| Columns: | |
| - **Hit**: share of renders that move more than 0.5 SD the requested way. | |
| - **Ξz**: achieved change for a request of Β±2. | |
| - **Ο**: Spearman correlation between request and readback (by our extractor / by Praat). | |
| | Tier | Control | Hit at β2 / +2 | Ξz at β2 / +2 | Ο (ours / Praat) | | |
| |---|---|---|---|---| | |
| | **Reliable** | loud | 1.00 / 1.00 | β1.57 / +1.81 | .97 / .97 | | |
| | | f0_med | .98 / 1.00 | β1.15 / +1.18 | .95 / .95 | | |
| | | h1h2 | 1.00 / 1.00 | β1.59 / +1.49 | .96 / .96 | | |
| | | tilt | 1.00 / .99 | β1.39 / +0.97 | .96 / .87 | | |
| | | centroid | .78 / .94 | β0.79 / +1.20 | .92 / .92 | | |
| | | f0_iqr | .66 / .91 | β0.63 / +1.09 | .88 / .88 | | |
| | **Usable** | rate | 1.00 / 1.00 | β1.61 / +1.30 | .96 / .66 | | |
| | | f0_end | .96 / .99 | β0.85 / +1.31 | .94 / .83 | | |
| | | flat | .62 / .41 | β0.64 / +0.51 | .85 / .68 | | |
| | **Weak** | pause | .38 / .64 | β0.37 / +0.71 | .76 / .70 | | |
| | | voiced | .39 / .12 | β0.44 / +0.26 | .79 / .60 | | |
| | | cpps | .56 / .35 | β0.45 / +0.30 | .58 / .46 | | |
| **Larger re-measurement** (same model, protocol and extractor; 62 unseen voices: 31 LibriTTS-R test-clean + 31 GLOBE | |
| test, a corpus the model never trained on; Γ 16 sentences, 12 of them newly written; 992 renders per direction per | |
| control; 95% CI from a crossed voice Γ sentence bootstrap): | |
| | Control | Hit at β2 / +2 | Mean hit [95% CI] | Ξz at β2 / +2 | | |
| |---|---|---|---| | |
| | loud | 1.00 / 1.00 | 1.00 [1.00, 1.00] | β1.54 / +1.87 | | |
| | rate | 1.00 / 1.00 | 1.00 [1.00, 1.00] | β1.52 / +1.27 | | |
| | h1h2 | 1.00 / 1.00 | 1.00 [1.00, 1.00] | β1.63 / +1.53 | | |
| | tilt | 1.00 / .98 | .99 [.98, 1.00] | β1.42 / +0.93 | | |
| | f0_med | 1.00 / .99 | .99 [.98, 1.00] | β1.21 / +1.31 | | |
| | centroid | .90 / .96 | .93 [.83, 1.00] | β0.93 / +1.37 | | |
| | f0_end | .85 / .93 | .89 [.81, .95] | β0.81 / +1.11 | | |
| | f0_iqr | .59 / .88 | .73 [.63, .83] | β0.58 / +0.95 | | |
| LibriTTS-R and GLOBE voices are statistically indistinguishable. `f0_end` depends on the sentence more than the other | |
| controls do. | |
| - **Tiers.** Reliable: Ο β₯ 0.85 under both extractors. Usable: under one. Weak: under neither. | |
| - **Gain.** The mean gain (achieved / requested) is about 0.5. `rate` and `f0_med` at +1 hit only .48 / .42. | |
| - **Cross-talk.** Mean leakage into the other 11 dims is 0.15β0.30 SD. The off-diagonal / diagonal ratio is 0.10. | |
| - **CPPS** is not a working control: a request moves pitch and level about as much as CPPS itself. | |
| - **Intelligibility.** WER (Whisper-large-v3) is 0.94 % uninstructed and 0.84β1.01 % across all requests. | |
| **Guidance** (pooled over loud, rate, f0_med, f0_iqr and tilt; Β±2 requests; n = 1,280): | |
| | s | Steer ratio (Ξz / requested) | Hit | Leak | WER | | |
| |---|---|---|---|---| | |
| | 1 | 0.65 | .95 | .20 | 0.9 % | | |
| | 2 | 1.35 | .99 | .34 | 1.0 % | | |
| | 3 | 1.97 | .98 | .48 | 1.0 % | | |
| - s β 1.5 roughly delivers the requested size, at the cost of more leakage. | |
| - Guidance on tags is riskier: `whisper` at s = 3 raises WER to 16 %. | |
| **Markup** (8 held-out voices Γ 6 sentences Γ 2 words): | |
| - Emphasis achieved +2.58 st and +3.92 dB, against +2.5 st and +4 dB requested. | |
| - `<high>` / `<low>` gave +3.18 / β2.96 st for Β±3. | |
| - `<loud>` / `<soft>` gave +4.84 / β5.54 dB for Β±6. The decoder attenuates level by 10β20 %. | |
| - `<slow>` / `<fast>` and pauses are exact by construction. | |
| - Neighbouring words move by at most 0.19 st and 0.16 dB. | |
| **Free-text prompts** (161 hand-written prompts): | |
| - Which-dims precision / recall: .85 / .66. Sign accuracy: .88. | |
| - Tag top-1 / top-3: .56 / .69. | |
| - Composition (.23) and negation (.22) mostly fail. "Not fast" can come out fast. | |
| - Absolute-register prompts ("a deep male voice") are not supported. Pick a voice instead (`voices.json`). | |
| - **Always look at the resolved request.** | |
| ## Quality | |
| Results under the Open TTS Leaderboard protocol (run locally on an RTX 5070, not an official submission; | |
| Qwen3-ASR-1.7B WER, WavLM-large SIM): | |
| | Setting | Seed-TTS en | CV3-Eval en | Avg | | |
| |---|---|---|---| | |
| | `voice_1361` (v1.0.0 default), acoustic model + Vocos, WER | 1.00 % | 2.63 % | **1.81 %** | | |
| | Cloning (prompted flow), WER / SIM Γ100 | 1.01 % / 37.4 | 2.73 % / 28.7 | 1.87 % / 33.1 | | |
| - The flow refiner was scored with an earlier acoustic checkpoint: 1.04 % / 2.70 % WER (no change). | |
| - **Cloning similarity is low.** The board's lowest cloning SIM is about 37, and its median is about 65. Do not | |
| expect a convincing likeness. | |
| ## Quantized int8 | |
| `precision="int8"` (CLI `--precision int8`) loads the weight files in `int8/` and downloads only those: 52 MB instead | |
| of 215 MB. Vocos, WavLM-base-plus-sv and all-MiniLM-L6-v2 still come from their own repos and are rounded to int8 the | |
| same way when they load. The whole install shrinks from 765 MB to 193 MB of weights, and RAM use does not change. | |
| - **Scheme.** Weight-only, round-to-nearest, symmetric. Tensors with β₯ 2 dimensions and β₯ 1,024 values are stored as | |
| int8 with one fp16 scale per output row; everything else as fp16. Weights are dequantized at load and compute stays | |
| fp32, so int8 is not faster than fp32. `steertts/quant.py` has the code. | |
| - **Parity.** The package's int8 renders are bit-identical to the evaluated int8 model on plain, control, prompt, markup | |
| and cloning requests. | |
| Same protocol and machine as [Quality](#quality), default voice `voice_1361`, refiner on: | |
| | | fp32 | int8 | | |
| |---|---|---| | |
| | Seed-TTS en WER | 0.94 % | 1.03 % | | |
| | CV3-Eval en WER | 2.62 % | 2.66 % | | |
| | Cloning WER, Seed / CV3 | 0.99 % / 2.70 % | 1.06 % / 2.70 % | | |
| | Cloning SIM Γ100, Seed / CV3 | 37.4 / 28.7 | 37.5 / 28.3 | | |
| | UTMOS22, Seed / CV3 | 4.27 / 4.24 | 4.27 / 4.25 | | |
| | UTMOS22 cloning, Seed / CV3 | 4.15 / 4.08 | 4.13 / 4.07 | | |
| Steering readback (8 voices Γ 6 texts, request β2 / +2, achieved Ξz): | |
| | control | fp32 | int8 | | |
| |---|---|---| | |
| | loud | β1.73 / +1.92 | β1.71 / +1.91 | | |
| | f0_med | β1.02 / +1.39 | β1.01 / +1.50 | | |
| | rate | β1.55 / +1.29 | β1.54 / +1.33 | | |
| | h1h2 | β1.07 / +1.21 | β1.05 / +1.14 | | |
| | tilt | β1.28 / +0.86 | β1.31 / +0.92 | | |
| | f0_iqr | β0.58 / +0.80 | β0.67 / +0.75 | | |
| - **Free-text prompts.** Of the 161 hand-written prompts, 97 % map to the same tag and 96 % to the same set of | |
| controls as fp32. | |
| - **Not released: int4.** Uniform int4 (7Γ smaller) loses about 0.5 UTMOS and 5β7 cloning SIM points, and pitch-range | |
| steering falls to Β±0.3. Keeping the conditioning path in int8 recovers the steering but not the audio quality. | |
| ## Speed | |
| Measured on an RTX 5070 and on a shared, heavily loaded 8-thread CPU, so treat CPU figures as lower bounds: | |
| - **GPU.** Batch-1 RTFx 61β81 for the acoustic model + Vocos (leaderboard runs). | |
| - **CPU, 8 threads.** RTFx 3.6β7.9 for the acoustic model + Vocos. The flow refiner adds 12Γ the compute and | |
| drops RTFx from 6.7 to 0.8 (0.4 with 1 thread). Use `flow=False` / `--no-flow` for real-time CPU use. | |
| - **Cloning on CPU.** RTFx 0.2 with the prompted flow, including the WavLM speaker encoder. | |
| - **Streaming on CPU.** First audio after about 1 s. | |
| - See [Release verification](#release-verification) for the numbers measured on this exact package. | |
| ## Limitations | |
| - **English only.** The front end is ARPAbet. Single speaker per request, and the controls are utterance-level. | |
| - **Emotion tags are style switches, not emotions.** An independent emotion classifier (emotion2vec+ large) | |
| recognises only `sadness` (0.16 top-1 vs 0.01 uninstructed). Every other emotion tag scores at most 0.02. The same | |
| classifier recognises 7 of 10 of these emotions on real EARS recordings. 27 of 49 tags are acoustically within 0.5 | |
| of `neutral`. Voice-quality tags such as `whisper` act strongly (voiced share β9.9 SD). | |
| - **Event tokens only reserve time.** `[laugh]`, `[cough]`, `[dog]`, β¦ add about 0.12 s of near-silence. An audio | |
| tagger detects 0 of 1,728 renders. The project's demo mixed recorded sounds in, but that mixer is not part of this | |
| release. | |
| - **The acoustic-environment input is untrained.** The weights contain an environment projection, but the final | |
| training stage never saw a non-zero environment vector, so this package does not expose it. | |
| - **Cloning similarity is low** (see above), and cloning speed on CPU is slow with the prompted flow. | |
| - **The controls are acoustic measurements, not percepts.** `voiced`, `flat` and `cpps` barely move. `rate` and | |
| `tilt` depend on how they are measured. z = 0 is the voice's typical render, not its real recordings. | |
| - **Test-set-informed choices.** The prompted-flow checkpoint was chosen on subsets of the Seed-TTS / CV3 test | |
| sets, and the text-normalisation rules came from CV3 error analysis. The leaderboard rows use `voice_1361`, | |
| the pre-registered v1.0.0 default: a sweep picked a lower-WER voice on the test sets, but it was not used. | |
| v1.0.1 switched the package default to `voice_1688` after users found `voice_1361` fast and synthetic-sounding. | |
| It was picked by UTMOS22 over 300 random voices on our own sentences, not on the test sets. On 16 sentences it | |
| matches `voice_1361` on WER (1.1 %), scores higher on UTMOS22 (4.43 vs 4.24) and speaks a little slower (3.4 vs 3.6 | |
| words/s). Pass `voice="voice_1361"` for the v1.0.0 sound. | |
| - **No human evaluation.** Every number above is from automatic metrics, with one training seed. | |
| ## Intended use and misuse | |
| **Intended uses:** | |
| - Research on controllable / interpretable speech synthesis. | |
| - Accessibility and prototyping, where explicit, inspectable control matters more than peak naturalness. | |
| - On-device experiments with a ~23M-parameter TTS. | |
| **Voice cloning requires consent.** Clone only a voice whose owner has agreed, and do not use this model to: | |
| - impersonate real people; | |
| - produce deceptive or fraudulent audio; | |
| - bypass voice authentication; | |
| - create non-consensual content. | |
| Disclose synthetic speech to listeners. There is no watermark yet. | |
| **Built-in voices are anonymised.** Built-in voices are training speakers under opaque names. Speaker IDs from the | |
| source corpora are not included. Speakers from a celebrity-interview corpus (VoxCeleb, via NonverbalTTS) were | |
| excluded from the voice list. | |
| ## Training data | |
| 410,566 utterances (659 h) from 4,919 speakers: | |
| | Source | Hours | Licence | | |
| |---|---|---| | |
| | LibriTTS-R (clean-100, clean-360, other-500) | 516 | CC BY 4.0 | | |
| | EARS (read + freeform) | 78 | CC BY-NC 4.0 | | |
| | Expresso | 11 | CC BY-NC 4.0 | | |
| | NonverbalTTS (VoxCeleb and Expresso sources) | 12.5 | Source terms apply | | |
| | Synthetic speech from a commercial TTS API | 43 | The provider's terms of service, incl. a competing-model clause | | |
| Control labels are measured from the audio. Style tags come from the corpora's own style/emotion labels. | |
| **Licence: CC BY-NC 4.0. Non-commercial use only.** | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `model.safetensors` | Acoustic model (fp32) | | |
| | `int8/*.safetensors` | The same five weight files in int8 (see [Quantized int8](#quantized-int8)) | | |
| | `flow.safetensors`, `pflow.safetensors` | Flow refiner and prompted flow | | |
| | `prompt_adapter.safetensors` | Free-text β request adapter (0.5M) | | |
| | `samples/*.mp3` | The audio clips in [Listen](#listen) | | |
| | `speaker_vectors.safetensors` | 512-d vector per built-in voice, in `config.json:voices_order` | | |
| | `voices.json` | Measured descriptions (see below) | | |
| | `control_stats.json` | Pooled SDs, tag list, and each voice's neutral control means (for `measure_z`) | | |
| | `config.json` | Architecture and file map | | |
| | `request.schema.json` | The request schema | | |
| | `steertts/` | The package (incl. `steertts/mamba3` with its own invariant tests); also on [GitHub](https://github.com/WhissleAI/steer-tts) | | |
| | `tests/` | pytest suite | | |
| | `leaderboard/` | Open TTS Leaderboard backend | | |
| | `TEST_LOG.md` | Release verification log | | |
| | `LICENSE.md` | Code (MIT) and weights (CC BY-NC 4.0) | | |
| `voices.json` descriptions are measured, not copied from corpus metadata: | |
| - **Gender.** A probe on the speaker vector; 99.3 % 5-fold accuracy on 534 self-labelled speakers. | |
| - **Pitch register.** Median F0 within that gender. | |
| - **Pace.** Articulation rate. | |
| ## Release verification | |
| The package was verified in a fresh CPU environment; `TEST_LOG.md` has the full log. | |
| - **Tests.** The pytest suite passes (16 tests at v1.0.0; 17 at v1.0.1, re-run with the new default voice; 21 at | |
| v1.1.0, with the int8 tests). | |
| - **Parity.** Seven renders are bit-identical to the training code's synthesis on the same device: acoustic | |
| only, controls, tag + guidance, per-word shifts, the flow refiner, and cloning through the prompted flow. | |
| - **Speed.** On this package, CPU with 4 threads on a loaded machine: RTFx 4.9 without the refiner, 0.7 with it. | |
| First streamed audio arrives after 0.5 s. | |
| ## Changelog | |
| - **v1.1.0.** int8 weights in `int8/`, `precision="int8"` and `--precision int8` (4Γ smaller download, same quality; see | |
| [Quantized int8](#quantized-int8)). New `steertts/quant.py` and `tests/test_int8.py`. fp32 behaviour is unchanged, | |
| except that fp32 downloads now skip `int8/`. | |
| - **v1.0.2.** `steertts play "β¦"` and `--play` stream audio to the speakers as each sentence is rendered; | |
| `steertts.player.Player` does the same from Python. New `[play]` extra (sounddevice), also in `[all]`. | |
| - **v1.0.1.** Default voice `voice_1688` instead of `voice_1361`, which users found fast and synthetic-sounding (see | |
| [Listen](#listen) and Limitations). `stream()` now raises an error for `word_shift`, which it used to drop silently. | |
| The CLI and `inference.py` download this repo when no local model is given. Leaderboard scripts point at this repo | |
| and stay pinned to `voice_1361`. Code is mirrored on GitHub for issues. | |
| - **v1.0.0.** First release. | |
| ## Citation | |
| ```bibtex | |
| @misc{steertts2026, | |
| title = {SteerTTS: A Tiny, Measurably Steerable Speech Synthesizer}, | |
| author = {{Whissle AI}}, | |
| year = {2026}, | |
| howpublished = {\url{https://huggingface.co/WhissleAI/steer-tts}} | |
| } | |
| ``` | |