Text-to-Speech
Safetensors
English
steertts
steer-tts
tts
speech-synthesis
english
controllable-tts
steerable-tts
prosody-control
expressive-tts
style-control
inline-markup
word-emphasis
zero-shot-voice-cloning
voice-cloning
interpretable
non-autoregressive
fastpitch
hybrid-attention
flow-matching
vocos
small-model
quantized
int8
on-device
c
embedded
streaming
non-commercial
Eval Results (legacy)
steertts 1.3.0: sound events, text front end. Code, weights and voices CC BY-NC 4.0 + Whissle terms (non-commercial).
7558665 | language: | |
| - en | |
| license: cc-by-nc-4.0 | |
| extra_gated_heading: "Accept the SteerTTS licence (non-commercial use only)" | |
| extra_gated_prompt: "SteerTTS weights and voices are licensed by Whissle Inc. under CC BY-NC 4.0 and the terms in LICENSE.md, for non-commercial use only (the package code is under the same licence). Commercial use (in or with a product or service, paid or hosted services, business use, or commercial use of generated audio) requires a separate written licence from Whissle Inc.: contact ksingla@whissle.ai. Clone only voices whose owners have consented. Whissle Inc. keeps all rights not expressly granted; no patent or trademark rights are granted." | |
| extra_gated_fields: | |
| Name: text | |
| Affiliation (company, university or none): text | |
| Country: country | |
| Intended use: | |
| type: select | |
| options: | |
| - Academic research | |
| - Education | |
| - Accessibility / personal use | |
| - Personal / hobby project | |
| - Interested in a commercial licence (Whissle will contact you) | |
| - Other non-commercial use | |
| I agree to use the SteerTTS weights and voices only for non-commercial purposes under CC BY-NC 4 and the LICENSE file: checkbox | |
| I will only clone voices whose owners have consented, and will not use SteerTTS to impersonate or deceive: checkbox | |
| I understand that commercial use requires a separate written licence from Whissle Inc: checkbox | |
| extra_gated_button_content: "Accept licence" | |
| library_name: steertts | |
| pipeline_tag: text-to-speech | |
| tags: | |
| - text-to-speech | |
| - tts | |
| - speech-synthesis | |
| - english | |
| - controllable-tts | |
| - steerable-tts | |
| - prosody-control | |
| - expressive-tts | |
| - style-control | |
| - inline-markup | |
| - word-emphasis | |
| - zero-shot-voice-cloning | |
| - voice-cloning | |
| - interpretable | |
| - non-autoregressive | |
| - fastpitch | |
| - hybrid-attention | |
| - flow-matching | |
| - vocos | |
| - small-model | |
| - quantized | |
| - int8 | |
| - on-device | |
| - c | |
| - embedded | |
| - streaming | |
| - non-commercial | |
| model-index: | |
| - name: steer-tts | |
| results: | |
| - task: | |
| type: text-to-speech | |
| name: Text-to-Speech | |
| dataset: | |
| type: bezzam/seed_tts_eval | |
| name: Seed-TTS-eval (English) | |
| split: test | |
| metrics: | |
| - type: wer | |
| value: 1.00 | |
| name: WER % (voice_1361, the v1.0.0 default, Qwen3-ASR-1.7B) | |
| verified: false | |
| - type: wer | |
| value: 1.01 | |
| name: WER % (zero-shot cloning, prompted flow) | |
| verified: false | |
| - type: speaker_similarity | |
| value: 37.4 | |
| name: SIM x100 (zero-shot cloning, WavLM-large) | |
| verified: false | |
| source: | |
| name: Open TTS Leaderboard protocol (local run, not an official submission) | |
| url: https://huggingface.co/spaces/hf-audio/open_tts_leaderboard | |
| - task: | |
| type: text-to-speech | |
| name: Text-to-Speech | |
| dataset: | |
| type: bezzam/cv3_eval | |
| name: CV3-Eval (English, zero-shot) | |
| split: test | |
| metrics: | |
| - type: wer | |
| value: 2.63 | |
| name: WER % (voice_1361, the v1.0.0 default, Qwen3-ASR-1.7B) | |
| verified: false | |
| - type: wer | |
| value: 2.73 | |
| name: WER % (zero-shot cloning, prompted flow) | |
| verified: false | |
| - type: speaker_similarity | |
| value: 28.7 | |
| name: SIM x100 (zero-shot cloning, WavLM-large) | |
| verified: false | |
| source: | |
| name: Open TTS Leaderboard protocol (local run, not an official submission) | |
| url: https://huggingface.co/spaces/hf-audio/open_tts_leaderboard | |
| # SteerTTS v1 | |
| A small English text-to-speech model that you steer with numbers. Each request can set any subset of **12 acoustic | |
| controls**, for example speaking rate, loudness, median pitch, pitch range, final rise or fall, pauses, spectral tilt | |
| and breathiness. Each control is a z-score relative to the chosen voice's own neutral speech. You can also pick a | |
| **style tag**, give a **free-text direction** (which a small adapter turns into a request you can inspect), or use | |
| **inline markup**. The model can **clone** a voice from a short reference clip. | |
| | | | | |
| |---|---| | |
| | Acoustic model | **9.59M** parameters (encoder/decoder: attention + a bidirectional sequence-mixing branch, AdaLN conditioning) | | |
| | + Vocos vocoder | 13.53M (`charactr/vocos-mel-24khz`, frozen), **23.1M** total | | |
| | Flow-matching refiner (optional, on by default) | 17.64M β 40.8M total | | |
| | Prompted flow (cloning only) | 24.67M, used in place of the refiner when `ref_wav` is given β 47.8M total | | |
| | Frozen external encoders | WavLM-base-plus-sv 101M (cloning only), all-MiniLM-L6-v2 22.7M (free-text prompts only) | | |
| | Output | 24 kHz mono, fp32, CPU or CUDA | | |
| | Voices | 2,825 built-in voices under anonymised names (`voice_0001` β¦), default `voice_1688` (v1.0.0: `voice_1361`) | | |
| > **Licence: weights, voices and package code CC BY-NC 4.0 + Whissle terms (non-commercial use only).** | |
| ## Listen | |
| All samples are raw model output (acoustic model + flow refiner + Vocos, seed 0), rendered with the code in this repo. | |
| **New default voice.** v1.0.1 switched the default from `voice_1361` to `voice_1688`, which sounds more natural and less rushed. | |
| | Text | `voice_1361` (v1.0.0 default) | `voice_1688` (v1.0.1 default) | | |
| |---|---|---| | |
| | "Hi, thanks for calling. I can help you reschedule your appointment today." | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_intro_voice_1361.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_intro_voice_1688.mp3"></audio> | | |
| | "The weather was cold, but the view from the top was worth every single step." | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_story_voice_1361.mp3"></audio> | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/cmp_story_voice_1688.mp3"></audio> | | |
| **Steering `voice_1688`.** | |
| | Request | Audio | | |
| |---|---| | |
| | "Are you coming with us tonight?" (no controls) | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_plain.mp3"></audio> | | |
| | Same text, `ctrl={"f0_end": 2, "f0_iqr": 1.5}`: rising, livelier | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_question_rise.mp3"></audio> | | |
| | "Please keep this between the two of us." `ctrl={"rate": -1.5, "loud": -1.5}` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_slow_soft.mp3"></audio> | | |
| | Same text, `tag="whisper"` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_whisper_tag.mp3"></audio> | | |
| | "I really thought it would work this time." `tag="sadness"`, `guidance=1.5` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_sad_tag.mp3"></audio> | | |
| | "I can't believe we actually won!" `prompt="excited, a bit faster"` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_prompt.mp3"></audio> | | |
| | Markup: `Well, *that* was fast. [pause 400ms] <slow>Let me think about it.</slow> <whisper>Don't tell anyone.</whisper>` | <audio controls preload="none" src="https://huggingface.co/WhissleAI/steer-tts/resolve/main/samples/steer_markup.mp3"></audio> | | |
| Found a bug or a voice that sounds wrong? Open an issue at [github.com/WhissleAI/steer-tts](https://github.com/WhissleAI/steer-tts/issues) or start a discussion in the Community tab. | |
| ## Quick start | |
| ```bash | |
| hf download WhissleAI/steer-tts --local-dir steer-tts --exclude "native/*" && cd steer-tts # native/ is for the C library | |
| pip install ".[all]" # core: torch, vocos, g2p_en β¦; [all] adds transformers (prompts, cloning), parselmouth (measuring), sounddevice (playback) | |
| # or, code only (weights download on first use): pip install "steertts[all] @ git+https://github.com/WhissleAI/steer-tts" | |
| steertts say "Hello there, nice to meet you." -o hello.wav | |
| steertts say "I can't believe it worked." --tag happiness --ctrl rate=1,f0_iqr=1.5 -o happy.wav | |
| steertts say "Please keep this quiet." --prompt "whispering, slow and secretive" -o secret.wav # prints the resolved request | |
| steertts stream "Well, *that* was fast. [pause 400ms] <slow>Let me think.</slow>" -o markup.wav | |
| steertts play "Hi there. <whisper>This plays as it is rendered.</whisper>" # streams to your speakers | |
| steertts say "Now in your voice." --ref-wav consenting_speaker.wav -o clone.wav | |
| steertts voices --gender female | head | |
| steertts --precision int8 say "Same voice, a quarter of the download." -o int8.wav # int8 weights (see Quantized int8) | |
| ``` | |
| The CLI looks for the model in the current directory. From anywhere else, pass it before the command | |
| (`steertts --model WhissleAI/steer-tts say "Hi." -o hi.wav`) or set `STEERTTS_MODEL`. | |
| ```python | |
| from steertts import SteerTTS | |
| tts = SteerTTS.from_pretrained("WhissleAI/steer-tts") # HF repo id or a local dir; device: cuda if available, else cpu | |
| wav = tts("The meeting moved to Thursday.") # float32 numpy, 24 kHz, default voice voice_1688 | |
| wav = tts("Are you serious?", voice="voice_0042", ctrl={"f0_end": 2.0, "f0_iqr": 1.5}) # rising, lively | |
| wav = tts("It is over.", tag="sadness", ctrl={"loud": -1}, guidance=1.5) | |
| req = tts.resolve({"text": "See you soon.", "prompt": "excited, a bit faster"}) | |
| print(req) # {'text': ..., 'voice': 'voice_1688', 'tag': 'excited', 'ctrl': {'rate': 0.29}, 'prompt_mapping': {...}, ...} | |
| wav = tts.synthesize(req) # a resolved request renders exactly as the prompt would | |
| from steertts.player import Player # plays chunks as they arrive (sounddevice, else ffplay / aplay) | |
| player = Player(tts.sample_rate) | |
| for chunk in tts.stream("First sentence. Second sentence."): # sentence-level streaming | |
| player.play(chunk.audio) | |
| player.close() # waits until playback finishes | |
| tts.save(wav, "out.wav") | |
| tts8 = SteerTTS.from_pretrained("WhissleAI/steer-tts", precision="int8") # downloads only int8/ (52 MB, not 215 MB) | |
| ``` | |
| `inference.py` renders one example of each feature. `python -m steertts.cli --help` lists the CLI. | |
| ### Without Python: the C library | |
| The [stand-alone C library](https://github.com/WhissleAI/steer-tts/blob/main/docs/c.md) (C11, no dependencies: no | |
| Python, PyTorch or ONNX Runtime) runs this model with every feature: flow refiner, free-text prompts and voice cloning, | |
| fp32 or int8. It reads the same weight files plus `native/` (Vocos, the MiniLM prompt encoder and the WavLM speaker | |
| encoder converted to safetensors, and the text front-end assets): | |
| ```bash | |
| git clone https://github.com/WhissleAI/steer-tts && cd steer-tts/c && make # or cmake -B build && cmake --build build | |
| tools/get_models.sh v1 int8 # curl only; HF_TOKEN=<read token> after accepting the licence; CLONE=1 adds WavLM | |
| build/steertts --model models/steer-tts --precision int8 "Are you serious?" --ctrl f0_end=2,f0_iqr=1.5 -o q.wav | |
| build/steertts --model models/steer-tts --precision int8 "See you soon." --prompt "excited, a bit faster" -o p.wav | |
| ``` | |
| Same audio as the Python package: on 218 Seed-TTS sentences the C renders have identical lengths, ~88 dB median | |
| waveform SNR to the Python renders and the same WER; renders are bit-identical across thread counts. One CPU thread: | |
| RTF 0.88 with the flow refiner (Python + PyTorch: 1.06), peak RAM 173 MB fp32 / 65 MB int8 (Python: 930 MB). | |
| ### Runtimes and files | |
| Each runtime downloads only what it reads: | |
| | Runtime | Reads | | |
| |---|---| | |
| | Python, fp32 (`SteerTTS.from_pretrained`) | top-level `*.json` + `*.safetensors`; Vocos, MiniLM and WavLM from their own repos | | |
| | Python, int8 (`precision="int8"`) | top-level `*.json` + `int8/` | | |
| | C library, fp32 / int8 (`--precision`) | top-level `*.json` + the fp32 files or `int8/`, and `native/` (`native/int8/` for int8) | | |
| ## The request | |
| Every synthesis is one JSON object. It is documented field by field in [`request.schema.json`](request.schema.json) | |
| (JSON Schema 2020-12; `steertts schema` prints it). Only `text` is required. | |
| | Field | Meaning | | |
| |---|---| | |
| | `text` | English text. Numbers, units, currency, ordinals, ranges and unknown acronyms are spelled out. | | |
| | `voice` / `ref_wav` | A built-in voice, **or** a reference clip to clone (first 8 s, at least 0.5 s). | | |
| | `tag` | One of 51 style tags (`none`, `whisper`, `sadness`, `anger`, `fast`, `high_pitch`, β¦). | | |
| | `ctrl` | `{control: z}` over the 12 controls below, each in [-5, 5]. | | |
| | `prompt` | Free-text direction. `resolve()` maps it to `tag` + `ctrl`; an explicit `tag` or `ctrl` dim wins. | | |
| | `guidance` | Steering scale s (default 1). See [Guidance](#how-well-it-steers). | | |
| | `pace` | Global tempo divisor (>1 is faster). It does not change the style. | | |
| | `word_shift` | `{word_index: {pitch: st, energy: dB, dur: Γ}}`, per-word prosody edits. Not with `markup` or `stream()` (they raise an error); use inline markup there. | | |
| | `markup` | Parse inline markup in `text` (see below). `stream()` always parses it. | | |
| | `flow`, `seed` | Use the flow refiner (default true) and set the sampler seed. Renders are deterministic per device. | | |
| **Controls and the mask.** The 12 controls are utterance-level acoustic measurements: | |
| - `rate`: articulation rate | |
| - `loud`: speech level | |
| - `f0_med`: median pitch | |
| - `f0_iqr`: pitch range | |
| - `f0_end`: final pitch slope | |
| - `pause`: pause share | |
| - `tilt`: spectral tilt | |
| - `voiced`: voiced share | |
| - `flat`: spectral flatness | |
| - `centroid`: spectral centroid | |
| - `cpps`: cepstral peak prominence | |
| - `h1h2`: H1βH2 | |
| Each value is a z-score against the chosen voice's own neutral speech, in units of the pooled within-speaker SD | |
| (`control_stats.json`). The model also receives an availability mask: | |
| - A dim named in `ctrl` is requested. | |
| - A dim left out is unspecified, and the model chooses it from voice, tag and text. | |
| - `{"rate": 0}` asks for this voice's neutral rate. Omitting `rate` lets a `fast` tag or the text decide. | |
| `steertts measure out.wav --text "β¦" --voice voice_1688` reads the controls back from | |
| the audio. | |
| **Markup** (`markup: true` or `stream()`): | |
| | Markup | Effect | | |
| |---|---| | |
| | `*word*` or `<emph>β¦</emph>` | Emphasis: +2.5 st, +4 dB, Γ1.2 duration | | |
| | `<slow> <fast> <high> <low> <loud> <soft>` | Per-word changes; the sentence is still rendered in one pass | | |
| | `[pause]`, `[pause 300ms]`, `[pause 1.5s]` | Exact silence | | |
| | `<whisper>β¦</whisper>` (any tag), `<style prompt="β¦">β¦</style>` | The span is rendered as its own segment | | |
| | `[laugh]` `[cough]` `[sigh]` `[sneeze]` `[throat]` | A recorded sound of the voice's gender and nearest pitch, re-pitched to the voice and set 2 dB under its speech (see [Sound events](#sound-events)) | | |
| | `[dog]` `[cat]` `[knock]` `[door]` `[bell]` `[horn]` `[siren]` `[applause]` `[thunder]` `[typing]` | A recorded environmental sound, 3 dB under the speech | | |
| Text is rendered sentence by sentence, so the first sentence can play while the rest renders. | |
| ### Sound events | |
| The acoustic model cannot draw a laugh or a cough: those are bursty, noisy sounds, and it renders them as a smeared, | |
| voiced hum (an AudioSet tagger heard 0 of 18 tagged renders). Since | |
| v1.3.0 the event tokens are rendered from `events.safetensors`, a bank of real recordings stored as Vocos log-mels | |
| that goes through the same vocoder as the speech: | |
| - **In-voice sounds** (`laugh`, `cough`, `sigh`, `sneeze`, `throat`): a clip of the voice's gender, the one of | |
| the two nearest in pitch picked by the seed, shifted to the voice's median F0 (at most Β±4 semitones) and set 2 dB | |
| below the level of the first rendered sentence, with 80 ms of silence on each side. It is a different person's | |
| laugh matched to the voice, not the voice itself laughing. | |
| - **Environmental sounds**: a clip of that class, 3 dB below the speech. | |
| - The event is its own chunk, so text with an event token is rendered segment by segment even without | |
| `markup: true` (`word_shift` requests keep the single-pass path and the old behaviour). | |
| - A token the bank does not hold (`[phone]`, `[footsteps]`, `[clock]`, `[glass]`, `[cry]` β¦) keeps the old | |
| behaviour: about 0.12 s of near-silence. Those classes failed the bank's quality check. | |
| **Measured** (AudioSet tagger `MIT/ast-finetuned-audioset-10-10-0.4593`, the target class's best 1 s window): on six | |
| sentences with one tag each, in three voices, the tagged sound is detected (p > 0.1) in **18 of 18** renders, mean | |
| p = 0.58 (laugh 0.41, cough 0.71, sigh 0.79, sneeze 0.46, throat clearing 0.68). The model rendering the same tags | |
| itself: 0 of 18, mean p = 0.003. | |
| The bank is licensed separately (CC BY-SA 4.0, from VocalSound and CC0 FSD50K clips): see `NOTICE`. | |
| ## Architecture | |
| FastPitch-style explicit-variance TTS: | |
| 1. **Front end.** g2p_en ARPAbet phonemes with word indices. | |
| 2. **Encoder.** 4 blocks. | |
| 3. **Variance predictors.** Per-phone duration, pitch and energy. | |
| 4. **Length regulator.** | |
| 5. **Decoder.** 4 blocks. | |
| 6. **Output.** A 100-band log-mel, rendered by Vocos. | |
| Each encoder/decoder block is a **hybrid**: multi-head self-attention, then a **bidirectional sequence-mixing** branch | |
| (run forward and backward and summed; 3.70M parameters in total), then a conv FFN. | |
| Every LayerNorm in every block and predictor is an **adaptive LayerNorm**. Its scale and shift are predicted from | |
| g = MLP([speaker vector, tag embedding, ctrl β mask, mask]). The speaker slot is a linear projection of a frozen | |
| 512-d WavLM-base-plus-sv x-vector. Built-in voices use their stored mean vector; cloning uses the vector of the | |
| reference clip. | |
| **Guidance** renders the request and the uninstructed request, then extrapolates durations, pitch, energy and mel: | |
| out = outβ + sΒ·(out β outβ). | |
| Optional decoders: | |
| - **Flow refiner.** A conditional flow-matching ConvNeXt network (OT-CFM, 10 midpoint steps, temperature 0.4). It | |
| samples mel detail given the acoustic model's mel and g, so every control still comes from the acoustic model. | |
| - **Prompted flow.** The same network plus attention, given the reference clip's own mel as context | |
| (Voicebox-style infilling). Used for cloning. | |
| The `steertts` package here is self-contained (see [Release verification](#release-verification)). | |
| ## How well it steers | |
| The numbers below are for **unseen voices**: 16 LibriTTS-R test-clean speakers (8 female, 8 male by a median-F0 | |
| rule), each conditioned on the speaker vector of one recording, Γ 8 texts. They are for the acoustic model at guidance 1 **without the flow refiner**. | |
| Steering through the refiner was not measured in the paper; the release check below confirms only the direction. | |
| Columns: | |
| - **Hit**: share of renders that move more than 0.5 SD the requested way. | |
| - **Ξz**: achieved change for a request of Β±2. | |
| - **Ο**: Spearman correlation between request and readback (by our extractor / by Praat). | |
| | Tier | Control | Hit at β2 / +2 | Ξz at β2 / +2 | Ο (ours / Praat) | | |
| |---|---|---|---|---| | |
| | **Reliable** | loud | 1.00 / 1.00 | β1.57 / +1.81 | .97 / .97 | | |
| | | f0_med | .98 / 1.00 | β1.15 / +1.18 | .95 / .95 | | |
| | | h1h2 | 1.00 / 1.00 | β1.59 / +1.49 | .96 / .96 | | |
| | | tilt | 1.00 / .99 | β1.39 / +0.97 | .96 / .87 | | |
| | | centroid | .78 / .94 | β0.79 / +1.20 | .92 / .92 | | |
| | | f0_iqr | .66 / .91 | β0.63 / +1.09 | .88 / .88 | | |
| | **Usable** | rate | 1.00 / 1.00 | β1.61 / +1.30 | .96 / .66 | | |
| | | f0_end | .96 / .99 | β0.85 / +1.31 | .94 / .83 | | |
| | | flat | .62 / .41 | β0.64 / +0.51 | .85 / .68 | | |
| | **Weak** | pause | .38 / .64 | β0.37 / +0.71 | .76 / .70 | | |
| | | voiced | .39 / .12 | β0.44 / +0.26 | .79 / .60 | | |
| | | cpps | .56 / .35 | β0.45 / +0.30 | .58 / .46 | | |
| **Larger re-measurement** (same model, protocol and extractor; 62 unseen voices: 31 LibriTTS-R test-clean + 31 GLOBE | |
| test; Γ 16 sentences, 12 of them newly written; 992 renders per direction per | |
| control; 95% CI from a crossed voice Γ sentence bootstrap): | |
| | Control | Hit at β2 / +2 | Mean hit [95% CI] | Ξz at β2 / +2 | | |
| |---|---|---|---| | |
| | loud | 1.00 / 1.00 | 1.00 [1.00, 1.00] | β1.54 / +1.87 | | |
| | rate | 1.00 / 1.00 | 1.00 [1.00, 1.00] | β1.52 / +1.27 | | |
| | h1h2 | 1.00 / 1.00 | 1.00 [1.00, 1.00] | β1.63 / +1.53 | | |
| | tilt | 1.00 / .98 | .99 [.98, 1.00] | β1.42 / +0.93 | | |
| | f0_med | 1.00 / .99 | .99 [.98, 1.00] | β1.21 / +1.31 | | |
| | centroid | .90 / .96 | .93 [.83, 1.00] | β0.93 / +1.37 | | |
| | f0_end | .85 / .93 | .89 [.81, .95] | β0.81 / +1.11 | | |
| | f0_iqr | .59 / .88 | .73 [.63, .83] | β0.58 / +0.95 | | |
| LibriTTS-R and GLOBE voices are statistically indistinguishable. `f0_end` depends on the sentence more than the other | |
| controls do. | |
| - **Tiers.** Reliable: Ο β₯ 0.85 under both extractors. Usable: under one. Weak: under neither. | |
| - **Gain.** The mean gain (achieved / requested) is about 0.5. `rate` and `f0_med` at +1 hit only .48 / .42. | |
| - **Cross-talk.** Mean leakage into the other 11 dims is 0.15β0.30 SD. The off-diagonal / diagonal ratio is 0.10. | |
| - **CPPS** is not a working control: a request moves pitch and level about as much as CPPS itself. | |
| - **Intelligibility.** WER (Whisper-large-v3) is 0.94 % uninstructed and 0.84β1.01 % across all requests. | |
| **Guidance** (pooled over loud, rate, f0_med, f0_iqr and tilt; Β±2 requests; n = 1,280): | |
| | s | Steer ratio (Ξz / requested) | Hit | Leak | WER | | |
| |---|---|---|---|---| | |
| | 1 | 0.65 | .95 | .20 | 0.9 % | | |
| | 2 | 1.35 | .99 | .34 | 1.0 % | | |
| | 3 | 1.97 | .98 | .48 | 1.0 % | | |
| - s β 1.5 roughly delivers the requested size, at the cost of more leakage. | |
| - Guidance on tags is riskier: `whisper` at s = 3 raises WER to 16 %. | |
| **Markup** (8 unseen voices Γ 6 sentences Γ 2 words): | |
| - Emphasis achieved +2.58 st and +3.92 dB, against +2.5 st and +4 dB requested. | |
| - `<high>` / `<low>` gave +3.18 / β2.96 st for Β±3. | |
| - `<loud>` / `<soft>` gave +4.84 / β5.54 dB for Β±6. The decoder attenuates level by 10β20 %. | |
| - `<slow>` / `<fast>` and pauses are exact by construction. | |
| - Neighbouring words move by at most 0.19 st and 0.16 dB. | |
| **Free-text prompts** (161 hand-written prompts): | |
| - Which-dims precision / recall: .85 / .66. Sign accuracy: .88. | |
| - Tag top-1 / top-3: .56 / .69. | |
| - Composition (.23) and negation (.22) mostly fail. "Not fast" can come out fast. | |
| - Absolute-register prompts ("a deep male voice") are not supported. Pick a voice instead (`voices.json`). | |
| - **Always look at the resolved request.** | |
| ## Quality | |
| Results under the Open TTS Leaderboard protocol (run locally on an RTX 5070, not an official submission; | |
| Qwen3-ASR-1.7B WER, WavLM-large SIM): | |
| | Setting | Seed-TTS en | CV3-Eval en | Avg | | |
| |---|---|---|---| | |
| | `voice_1361` (v1.0.0 default), acoustic model + Vocos, WER | 1.00 % | 2.63 % | **1.81 %** | | |
| | Cloning (prompted flow), WER / SIM Γ100 | 1.01 % / 37.4 | 2.73 % / 28.7 | 1.87 % / 33.1 | | |
| - With the flow refiner (scored with an earlier build of the acoustic model): 1.04 % / 2.70 % WER. | |
| - **Cloning similarity is low.** The board's lowest cloning SIM is about 37, and its median is about 65. Do not | |
| expect a convincing likeness. | |
| ## Quantized int8 | |
| `precision="int8"` (CLI `--precision int8`) loads the weight files in `int8/` and downloads only those: 52 MB instead | |
| of 215 MB. Vocos, WavLM-base-plus-sv and all-MiniLM-L6-v2 still come from their own repos and are rounded to int8 the | |
| same way when they load. The whole install shrinks from 765 MB to 193 MB of weights, and RAM use does not change. | |
| - **Scheme.** Weight-only, round-to-nearest, symmetric. Tensors with β₯ 2 dimensions and β₯ 1,024 values are stored as | |
| int8 with one fp16 scale per output row; everything else as fp16. Weights are dequantized at load and compute stays | |
| fp32, so int8 is not faster than fp32. `steertts/quant.py` has the code. | |
| - **Parity.** The package's int8 renders are bit-identical to the evaluated int8 model on plain, control, prompt, markup | |
| and cloning requests. | |
| Same protocol and machine as [Quality](#quality), default voice `voice_1361`, refiner on: | |
| | | fp32 | int8 | | |
| |---|---|---| | |
| | Seed-TTS en WER | 0.94 % | 1.03 % | | |
| | CV3-Eval en WER | 2.62 % | 2.66 % | | |
| | Cloning WER, Seed / CV3 | 0.99 % / 2.70 % | 1.06 % / 2.70 % | | |
| | Cloning SIM Γ100, Seed / CV3 | 37.4 / 28.7 | 37.5 / 28.3 | | |
| | UTMOS22, Seed / CV3 | 4.27 / 4.24 | 4.27 / 4.25 | | |
| | UTMOS22 cloning, Seed / CV3 | 4.15 / 4.08 | 4.13 / 4.07 | | |
| Steering readback (8 voices Γ 6 texts, request β2 / +2, achieved Ξz): | |
| | control | fp32 | int8 | | |
| |---|---|---| | |
| | loud | β1.73 / +1.92 | β1.71 / +1.91 | | |
| | f0_med | β1.02 / +1.39 | β1.01 / +1.50 | | |
| | rate | β1.55 / +1.29 | β1.54 / +1.33 | | |
| | h1h2 | β1.07 / +1.21 | β1.05 / +1.14 | | |
| | tilt | β1.28 / +0.86 | β1.31 / +0.92 | | |
| | f0_iqr | β0.58 / +0.80 | β0.67 / +0.75 | | |
| - **Free-text prompts.** Of the 161 hand-written prompts, 97 % map to the same tag and 96 % to the same set of | |
| controls as fp32. | |
| - **Not released: int4.** Uniform int4 (7Γ smaller) loses about 0.5 UTMOS and 5β7 cloning SIM points, and pitch-range | |
| steering falls to Β±0.3. Keeping the conditioning path in int8 recovers the steering but not the audio quality. | |
| ## Speed | |
| Measured on an RTX 5070 and on a shared, heavily loaded 8-thread CPU, so treat CPU figures as lower bounds: | |
| - **GPU.** Batch-1 RTFx 61β81 for the acoustic model + Vocos (leaderboard runs). | |
| - **CPU, 8 threads.** RTFx 3.6β7.9 for the acoustic model + Vocos. The flow refiner adds 12Γ the compute and | |
| drops RTFx from 6.7 to 0.8 (0.4 with 1 thread). Use `flow=False` / `--no-flow` for real-time CPU use. | |
| - **Cloning on CPU.** RTFx 0.2 with the prompted flow, including the WavLM speaker encoder. | |
| - **Streaming on CPU.** First audio after about 1 s. | |
| - See [Release verification](#release-verification) for the numbers measured on this exact package. | |
| ## Limitations | |
| - **English only.** The front end is ARPAbet. Single speaker per request, and the controls are utterance-level. | |
| - **Emotion tags are style switches, not emotions.** An independent emotion classifier (emotion2vec+ large) | |
| recognises only `sadness` (0.16 top-1 vs 0.01 uninstructed). Every other emotion tag scores at most 0.02. The same | |
| classifier recognises 7 of 10 of these emotions on real acted-emotion recordings. 27 of 49 tags are acoustically within 0.5 | |
| of `neutral`. Voice-quality tags such as `whisper` act strongly (voiced share β9.9 SD). | |
| - **Sound events are recordings, not the voice.** Since v1.3.0, `[laugh]` `[cough]` `[sigh]` `[sneeze]` `[throat]` | |
| and ten environmental tokens play a recorded clip matched to the voice's gender, pitch and level (see | |
| [Sound events](#sound-events)). The model itself cannot render them, the clip is a separate chunk (a laugh does not | |
| run through a word), and other tokens still only reserve time. | |
| - **No acoustic-environment control.** The weights contain an environment projection that does not work, so this | |
| package does not expose it. | |
| - **Cloning similarity is low** (see above), and cloning speed on CPU is slow with the prompted flow. | |
| - **The controls are acoustic measurements, not percepts.** `voiced`, `flat` and `cpps` barely move. `rate` and | |
| `tilt` depend on how they are measured. z = 0 is the voice's typical render, not its real recordings. | |
| - **Test-set-informed choices.** The prompted-flow version was chosen on subsets of the Seed-TTS / CV3 test | |
| sets, and the text-normalisation rules came from CV3 error analysis. The leaderboard rows use `voice_1361`, | |
| the pre-registered v1.0.0 default: a sweep picked a lower-WER voice on the test sets, but it was not used. | |
| v1.0.1 switched the package default to `voice_1688` after users found `voice_1361` fast and synthetic-sounding. | |
| It was picked by UTMOS22 over 300 random voices on our own sentences, not on the test sets. On 16 sentences it | |
| matches `voice_1361` on WER (1.1 %), scores higher on UTMOS22 (4.43 vs 4.24) and speaks a little slower (3.4 vs 3.6 | |
| words/s). Pass `voice="voice_1361"` for the v1.0.0 sound. | |
| - **No human evaluation.** Every number above is from automatic metrics, on a single model. | |
| ## Intended use and misuse | |
| **Intended uses:** | |
| - Research on controllable / interpretable speech synthesis. | |
| - Accessibility and prototyping, where explicit, inspectable control matters more than peak naturalness. | |
| - On-device experiments with a ~23M-parameter TTS. | |
| **Voice cloning requires consent.** Clone only a voice whose owner has agreed, and do not use this model to: | |
| - impersonate real people; | |
| - produce deceptive or fraudulent audio; | |
| - bypass voice authentication; | |
| - create non-consensual content. | |
| Disclose synthetic speech to listeners. There is no watermark yet. | |
| **Built-in voices are anonymised.** Built-in voices come from real recorded speakers and carry opaque names; no | |
| source speaker identifiers are included. | |
| ## Licence | |
| Weights, voices and package code: CC BY-NC 4.0 + Whissle terms, non-commercial use only. See `LICENSE.md`. The | |
| recorded event bank `events.safetensors` is licensed separately under CC BY-SA 4.0 (see `NOTICE`). | |
| Commercial use of the code, weights and voices (including commercial use of generated audio) needs a separate written | |
| licence from Whissle Inc.: contact ksingla@whissle.ai. Whissle Inc. owns SteerTTS and keeps all rights not expressly | |
| granted; no patent or trademark rights are granted, and "SteerTTS" and "Whissle" are its trademarks. Provided as is, | |
| without warranty. See `LICENSE.md` (Whissle terms). | |
| ## Files | |
| | File | Contents | | |
| |---|---| | |
| | `model.safetensors` | Acoustic model (fp32) | | |
| | `int8/*.safetensors` | The same five weight files in int8 (see [Quantized int8](#quantized-int8)) | | |
| | `native/` | For the [C library](https://github.com/WhissleAI/steer-tts/blob/main/docs/c.md) only: Vocos (`vocos.safetensors`), MiniLM (`minilm.safetensors`, `minilm_vocab.txt`) and WavLM-base-plus-sv (`wavlm_sv.safetensors`) converted to safetensors, int8 copies in `native/int8/`, and the text front end (`front/`). Each keeps its own licence: Vocos MIT, MiniLM Apache-2.0, **WavLM CC BY-SA 3.0** (see `native/NOTICE`) | | |
| | `flow.safetensors`, `pflow.safetensors` | Flow refiner and prompted flow | | |
| | `prompt_adapter.safetensors` | Free-text β request adapter (0.5M) | | |
| | `samples/*.mp3` | The audio clips in [Listen](#listen) | | |
| | `speaker_vectors.safetensors` | 512-d vector per built-in voice, in `config.json:voices_order` | | |
| | `voices.json` | Measured descriptions (see below) | | |
| | `control_stats.json` | Pooled SDs, tag list, and each voice's neutral control means (for `measure_z`) | | |
| | `config.json` | Architecture and file map | | |
| | `request.schema.json` | The request schema | | |
| | `steertts/` | The package ; also on [GitHub](https://github.com/WhissleAI/steer-tts) | | |
| | `tests/` | pytest suite | | |
| | `leaderboard/` | Open TTS Leaderboard backend | | |
| | `TEST_LOG.md` | Release verification log | | |
| | `LICENSE.md` | Code, weights and voices: CC BY-NC 4.0 + Whissle terms | | |
| | `events.safetensors` | Recorded sound events for `[laugh]` `[cough]` `[dog]` β¦ (1.5 MB; CC BY-SA 4.0, see `NOTICE`) | | |
| | `NOTICE` | Third-party notices for `events.safetensors` | | |
| `voices.json` descriptions are measured from the audio: | |
| - **Gender.** A probe on the speaker vector; 99.3 % 5-fold accuracy on 534 self-labelled speakers. | |
| - **Pitch register.** Median F0 within that gender. | |
| - **Pace.** Articulation rate. | |
| ## Release verification | |
| The package was verified in a fresh CPU environment; `TEST_LOG.md` has the full log. | |
| - **Tests.** The pytest suite passes (16 tests at v1.0.0; 17 at v1.0.1, re-run with the new default voice; 21 at | |
| v1.1.0, with the int8 tests; 23 at v1.2.0). | |
| - **Parity.** Seven renders are bit-identical to the reference implementation on the same device: acoustic | |
| only, controls, tag + guidance, per-word shifts, the flow refiner, and cloning through the prompted flow. | |
| - **Speed.** On this package, CPU with 4 threads on a loaded machine: RTFx 4.9 without the refiner, 0.7 with it. | |
| First streamed audio arrives after 0.5 s. | |
| ## Changelog | |
| - **v1.3.0.** Sound events: `[laugh]` `[cough]` `[sigh]` `[sneeze]` `[throat]` and ten environmental tokens are | |
| rendered from the new recorded event bank `events.safetensors` (see [Sound events](#sound-events)), in Python and | |
| in the C library. The text front end reads more real-world text (the same rules in the Python, C and C++ front ends): 21-99 are | |
| spelled without hyphens (`seventy two`, now a dictionary pronunciation), phone numbers digit by digit, clock times | |
| and am/pm, titles and common abbreviations (`Dr.`, `e.g.`, `etc.`), `$4.2M`, versions, roman numerals after | |
| `Chapter` / `Part` β¦, simple fractions, and a lone `A` in an acronym is `ay`; a sentence no longer splits after | |
| `Dr.` or an initial. Weights are unchanged; text without these cases renders bit-identically to 1.2.0. New `tests/test_events.py`. | |
| - **v1.2.0.** Code clean-up: the sequence-mixing layer is one inference-only module (`steertts/seqmix.py`) with | |
| renamed parameters, and the weight files were re-saved with the new names. Audio is bit-identical to 1.1.1; update the | |
| package and the weights together (`pip install -U`; older packages do not load these files). | |
| - **v1.1.1.** `native/`: the files the new stand-alone [C library](https://github.com/WhissleAI/steer-tts/blob/main/docs/c.md) | |
| needs (converted Vocos, MiniLM and WavLM, fp32 and int8, plus the text front-end assets). The Python package now | |
| downloads an explicit list of its own files, so it never fetches `native/`; update with `pip install -U` | |
| (1.1.0 downloads `native/` too, once). Weights and Python behaviour are unchanged. | |
| - **v1.1.0.** int8 weights in `int8/`, `precision="int8"` and `--precision int8` (4Γ smaller download, same quality; see | |
| [Quantized int8](#quantized-int8)). New `steertts/quant.py` and `tests/test_int8.py`. fp32 behaviour is unchanged, | |
| except that fp32 downloads now skip `int8/`. | |
| - **v1.0.2.** `steertts play "β¦"` and `--play` stream audio to the speakers as each sentence is rendered; | |
| `steertts.player.Player` does the same from Python. New `[play]` extra (sounddevice), also in `[all]`. | |
| - **v1.0.1.** Default voice `voice_1688` instead of `voice_1361`, which users found fast and synthetic-sounding (see | |
| [Listen](#listen) and Limitations). `stream()` now raises an error for `word_shift`, which it used to drop silently. | |
| The CLI and `inference.py` download this repo when no local model is given. Leaderboard scripts point at this repo | |
| and stay pinned to `voice_1361`. Code is mirrored on GitHub for issues. | |
| - **v1.0.0.** First release. | |
| ## Citation | |
| ```bibtex | |
| @misc{steertts2026, | |
| title = {SteerTTS: A Tiny, Measurably Steerable Speech Synthesizer}, | |
| author = {{Whissle Inc.}}, | |
| year = {2026}, | |
| howpublished = {\url{https://huggingface.co/WhissleAI/steer-tts}} | |
| } | |
| ``` | |