File size: 10,901 Bytes
b9fa092 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 | ---
language:
- he
license: other
license_name: private-research-only
license_link: https://huggingface.co/useless-ai/pitput/blob/main/README.md#license
pipeline_tag: text-to-speech
library_name: pytorch
tags:
- text-to-speech
- tts
- hebrew
- fastspeech2
- ddsp
---
# pitput
**pitput** (ืคึดึผืึฐืคึผืึผื) is a Hebrew take on [tiny-voca](https://github.com/n4ze3m/tiny-voca): a tiny TTS model
built to run on a CPU. The whole model, vocoder included, is **only 18 MB** (4.5M parameters). It outputs 24 kHz
mono and runs faster than real time on a laptop CPU. It reads plain Hebrew without niqqud, in one voice with
**8 delivery styles** and **8 expression tags** (`<laugh>`, `<sigh>`, `<short pause>` โฆ).
It is a personal learning project, trained from scratch on synthetic speech. This is **try 1** (step 104,000,
November 18, 2026). **Private research use only; see [License](#license).**
## Samples
The 10 listening texts that were synthesized after every epoch, here from the final weights.
| Style | Text | Audio |
|---|---|---|
| default | ืืคื ื ืชืื ื ืืืฉืื ืืืจืืืืช ืืกืืืืกืืืงื, ืืืืืืกืืืช ืืฉืจืื ืืืื ืืฉื ื ืืืืจืื ื ืืืืื ืืฉืืื ืื ืืืฃ ืืืฉ. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/01-default.wav"></audio> |
| default, addressee f | ืื ืฉืืืื? ืฉืืขืชื ืฉืื ืืจืืฉืช ืืื ืืฉืืืข ืฉืขืืจ. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/02-default.wav"></audio> |
| default | ืืจืืืช ืืืืคื ืืืฆืืช ืืฉืืข ืืขืฉืจืื ืืืืืขื ืืฉืืื ื ืืืฆื, `<chuckle>` ืื ืืื ืืืื ืชืืืข. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/03-default.wav"></audio> |
| excited | `<laugh>` ืื ื ืื ืืืืื ื, ืงืืืื ื ืืช ืืืืจื! | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/04-excited.wav"></audio> |
| sad | `<sigh>` ืืืืืชื ืื ืื ืืขืจื, ืืืื ืคืฉืื ืื ืืืืข. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/05-sad.wav"></audio> |
| angry, addressee m | ืืื ืคืขืืื ืืืจืชื ืื ืืกืืืจ ืืช ืืืืช?! | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/06-angry.wav"></audio> |
| fearful | ืฉืืขืช ืืช ืื? `<gasp>` ืืฉ ืืืฉืื ืืืืฅ. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/07-fearful.wav"></audio> |
| surprised | ืจืืข, `<short pause>` ืืื ืืืืช ืืชืคืืจ? ืืืจื ืขืฉืจืื ืฉื ื? | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/08-surprised.wav"></audio> |
| calm | `<breath>` ืงืื ื ืฉืืื ืขืืืงื, `<long pause>` ืืชื ื ืืืืฃ ืืืืจืืข ืืื ืืื. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/09-calm.wav"></audio> |
| whisper | ืืื ื ืืืจ ืืฉืงื, ืืืื ืืืจ ืืฉื ืื. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/10-whisper.wav"></audio> |
## Usage
```bash
huggingface-cli login # the repository is private
git clone https://huggingface.co/useless-ai/pitput
cd pitput
pip install -r requirements.txt # PyTorch, Phonikud, onnxruntime, soundfile, huggingface_hub
python -m pitput.infer --style excited --text "<laugh> ืื ื ืื ืืืืื ื, ืงืืืื ื ืืช ืืืืจื!" --output out.wav
python -m pitput.infer --style default --addressee f --text "ืื ืฉืืืื? ืฉืืขืชื ืฉืื ืืจืืฉืช ืืื." --output hello.wav
```
On first use the Phonikud model (~300 MB) is downloaded to the Hugging Face cache. Options:
| Option | Meaning |
|---|---|
| `--style` | `default`, `excited`, `sad`, `angry`, `fearful`, `surprised`, `calm`, `whisper` |
| `--addressee m` / `f` | Gender of the person spoken to (ืื, ืืืชื, ืืืจืช โฆ). Without it Phonikud reads the masculine form |
| `--text-file` | Read the text from a UTF-8 file instead of `--text` |
| `--speed` | 1.0 = as trained; 0.9 is 10% slower |
| `--pitch-shift` | Added to log F0 (0.058 โ one semitone) |
| `--pitch-range` | Scales the melody around each sentence's mean pitch (1.0 = as predicted) |
| `--device` | `cpu` (default) or `cuda` |
From Python:
```python
import soundfile as sf
from pitput.infer import CHECKPOINT, load_model, speak_text
model = load_model(CHECKPOINT) # inference.pt next to the package
wave, utterances = speak_text(model, "ืฉืืื, <sigh> ืื ืฉืืืื?", "calm", addressee="f")
sf.write("out.wav", wave.numpy(), model.config.sample_rate)
```
### What the input must look like
- **Plain Hebrew in spoken form.** There is no text normalizer: write numbers, dates, times, acronyms and
foreign words out in Hebrew words (`ืืฉืืข ืืขืฉืจืื`, not `7:20`). Characters it can't read are skipped.
- **No niqqud needed.** [Phonikud](https://github.com/thewh1teagle/phonikud) vocalizes each sentence in context
and adds stress; the model reads its phonemes.
- **Tags** go where the sound should be, in English inside the Hebrew: `<laugh>`, `<chuckle>`, `<sigh>`,
`<breath>`, `<gasp>`, `<throat-clearing>`, `<short pause>`, `<long pause>`.
- **Punctuation** shapes the delivery (`, . ? ! โฆ`). Long texts are read sentence by sentence.
## Model
A fork of [fiato](https://huggingface.co/cfahlgren1/fiato), itself a fork of
[tiny-voca](https://github.com/n4ze3m/tiny-voca): a FastSpeech2-style model with a built-in
differentiable source-filter vocoder, so there is no separate neural vocoder.
- **Input:** 47 tokens: 29 Phonikud phonemes, 5 punctuation marks, word boundary, utterance edges and the 8
expression tags. Stress is a parallel feature, not a token.
- **Text encoder:** 4 transformer layers, 192 channels, with the style embedding added to every token.
- **Duration predictor:** frames per token, plus a "pause present" switch for punctuation, word boundaries and tags.
- **Frame level:** the style embedding is added again; 2 layers predict pitch (log F0), voicing and energy, and 5
layers predict the acoustic features.
- **Vocoder:** harmonic pulse train (from the predicted pitch) plus noise, each shaped by a complex cepstrum per frame.
- **Size:** 4,534,027 parameters at inference (18 MB checkpoint). Training also uses 2.7M parameters that are not
in the checkpoint (discriminators and an auxiliary mel head).
## Training data
The dataset is **synthetic only**, with no recorded human speech: 65,722 clips (139.3 h) of one synthetic voice,
of which 63,745 training clips (136.4 h) passed forced alignment. 160 held-out sentences (20 per style) were kept
for validation and 160 for test.
- **Text:** 65,728 Hebrew sentences from news (archive, HeSum, RSS), HeDC4 web text and stories, Hebrew Wikipedia,
ivrit.ai transcripts and LLM-written scenes (dialogue for the expressive styles).
- **Spoken form:** an LLM rewrote numbers, dates, money, acronyms and Latin script as Hebrew words, and every answer
was checked by code.
- **Addressee gender:** an LLM decided the gender of ambiguous second-person words (ืื, ืืืชื, ืืืจืช) in context.
The audio and the training phonemes follow the same decision.
- **Styles:** an LLM picked a style for each dialogue sentence and inserted the tags; calm and whisper were assigned
to plain statements. Each style was rendered with one fixed delivery instruction; default is a bright, friendly
delivery.
| Style | Clips synthesized | Rejected at alignment |
|---|---|---|
| default | 52,193 | 1% |
| calm | 4,900 | 1% |
| whisper | 4,900 | 1% |
| fearful | 1,104 | 42% |
| angry | 1,027 | 12% |
| sad | 685 | 19% |
| excited | 470 | 18% |
| surprised | 449 | 14% |
4,582 clips contain tags (at most 2 per clip); `<laugh>` appears in only about 90 of them.
- **Labels:** Phonikud phonemes with the addressee decisions applied, phonemized word by word with the tags at
their positions. Durations come from forced alignment with
[facebook/wav2vec2-lv-60-espeak-cv-ft](https://huggingface.co/facebook/wav2vec2-lv-60-espeak-cv-ft), which is used
only for labeling and is not part of the model. A tag owns the sound between the silences around it.
## Limitations
- **Laughs and other non-speech tags are weak.** The model gives each tag its time, but fills it with a quiet,
mostly unvoiced sound: a laugh comes out about 15 dB quieter than in the recordings and is heard as a soft breath.
Chuckle, throat-clearing and gasp suffer the same way. Pauses and breaths work better.
- **Intonation can sound a little robotic.** Pitch range is close to the recordings for most styles, but fearful
and surprised have only 70โ75% of it. The model speaks 3โ17% faster than the recordings, with shorter pauses.
- **Fearful is the weakest style:** it has the fewest usable training clips and the most pronunciation errors.
- **Pronunciation follows Phonikud,** including its mistakes (e.g. ืืืื read as "vaสu").
- **Second person defaults to masculine** unless `--addressee f` is given; there is no per-word decision at
inference.
- One voice only. Sentences are synthesized independently, with no prosody carried between them.
## License
- **Code** (`pitput/`): Apache License 2.0. It is a fork of fiato by cfahlgren1 and tiny-voca by Muhammed Nazeem,
both Apache-2.0; see `NOTICE`, `LICENSE` and `LICENSE.tiny-voca`.
- **Weights and audio samples: private research use only, not licensed for redistribution or any commercial use.**
Keep this repository private.
- **Phonikud** ([thewh1teagle/phonikud-onnx](https://huggingface.co/thewh1teagle/phonikud-onnx), MIT) is downloaded
at first use and is not included here.
## Files
| File | Contents |
|---|---|
| `inference.pt` | Weights, model configuration and front-end fingerprint (loaded with `weights_only=True`) |
| `pitput/` | Inference code: `infer.py` (command line and `speak_text`), `model.py`, `text.py` (phoneme tokens), `g2p.py` (Phonikud and addressee gender), `labels.py`, `config.py` |
| `samples/` | The audio above |
| `config.json`, `recipe.json` | Model and training settings of this run |
| `requirements.txt` | Python dependencies |
| `NOTICE`, `LICENSE`, `LICENSE.tiny-voca` | Attribution and license of the code |
## Acknowledgements
[tiny-voca](https://github.com/n4ze3m/tiny-voca) for the architecture and training recipe,
[fiato](https://huggingface.co/cfahlgren1/fiato) for expression tags on it, [Phonikud](https://github.com/thewh1teagle/phonikud)
for Hebrew vocalization and phonemes, and the Hebrew text sources: HeDC4, HeSum, Hebrew Wikipedia and ivrit.ai.
|