pitput / README.md
useless-ai's picture
pitput try 1 (step 104,000)
b9fa092 verified
|
Raw History Blame Contribute Delete
10.9 kB
---
language:
- he
license: other
license_name: private-research-only
license_link: https://huggingface.co/useless-ai/pitput/blob/main/README.md#license
pipeline_tag: text-to-speech
library_name: pytorch
tags:
- text-to-speech
- tts
- hebrew
- fastspeech2
- ddsp
---
# pitput
**pitput** (ืคึดึผื˜ึฐืคึผื•ึผื˜) is a Hebrew take on [tiny-voca](https://github.com/n4ze3m/tiny-voca): a tiny TTS model
built to run on a CPU. The whole model, vocoder included, is **only 18 MB** (4.5M parameters). It outputs 24 kHz
mono and runs faster than real time on a laptop CPU. It reads plain Hebrew without niqqud, in one voice with
**8 delivery styles** and **8 expression tags** (`<laugh>`, `<sigh>`, `<short pause>` โ€ฆ).
It is a personal learning project, trained from scratch on synthetic speech. This is **try 1** (step 104,000,
November 18, 2026). **Private research use only; see [License](#license).**
## Samples
The 10 listening texts that were synthesized after every epoch, here from the final weights.
| Style | Text | Audio |
|---|---|---|
| default | ืœืคื™ ื ืชื•ื ื™ ื”ืœืฉื›ื” ื”ืžืจื›ื–ื™ืช ืœืกื˜ื˜ื™ืกื˜ื™ืงื”, ืื•ื›ืœื•ืกื™ื™ืช ื™ืฉืจืืœ ื’ื“ืœื” ื‘ืฉื ื” ื”ืื—ืจื•ื ื” ื‘ื›ืžืื” ื•ืฉืžื•ื ื™ื ืืœืฃ ืื™ืฉ. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/01-default.wav"></audio> |
| default, addressee f | ืžื” ืฉืœื•ืžืš? ืฉืžืขืชื™ ืฉืœื ื”ืจื’ืฉืช ื˜ื•ื‘ ื‘ืฉื‘ื•ืข ืฉืขื‘ืจ. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/02-default.wav"></audio> |
| default | ื”ืจื›ื‘ืช ืœื—ื™ืคื” ื™ื•ืฆืืช ื‘ืฉื‘ืข ื•ืขืฉืจื™ื ื•ืžื’ื™ืขื” ื‘ืฉืžื•ื ื” ื•ื—ืฆื™, `<chuckle>` ืื ื”ื™ื ื‘ื›ืœืœ ืชื’ื™ืข. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/03-default.wav"></audio> |
| excited | `<laugh>` ืื ื™ ืœื ืžืืžื™ื ื”, ืงื™ื‘ืœื ื• ืืช ื”ื“ื™ืจื”! | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/04-excited.wav"></audio> |
| sad | `<sigh>` ื—ื™ื›ื™ืชื™ ืœื• ื›ืœ ื”ืขืจื‘, ื•ื”ื•ื ืคืฉื•ื˜ ืœื ื”ื’ื™ืข. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/05-sad.wav"></audio> |
| angry, addressee m | ื›ืžื” ืคืขืžื™ื ืืžืจืชื™ ืœืš ืœืกื’ื•ืจ ืืช ื”ื“ืœืช?! | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/06-angry.wav"></audio> |
| fearful | ืฉืžืขืช ืืช ื–ื”? `<gasp>` ื™ืฉ ืžื™ืฉื”ื• ื‘ื—ื•ืฅ. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/07-fearful.wav"></audio> |
| surprised | ืจื’ืข, `<short pause>` ื”ื•ื ื‘ืืžืช ื”ืชืคื˜ืจ? ืื—ืจื™ ืขืฉืจื™ื ืฉื ื”? | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/08-surprised.wav"></audio> |
| calm | `<breath>` ืงื—ื• ื ืฉื™ืžื” ืขืžื•ืงื”, `<long pause>` ื•ืชื ื• ืœื’ื•ืฃ ืœื”ื™ืจื’ืข ืœืื˜ ืœืื˜. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/09-calm.wav"></audio> |
| whisper | ื‘ื•ื ื ื“ื‘ืจ ื‘ืฉืงื˜, ื›ื•ืœื ื›ื‘ืจ ื™ืฉื ื™ื. | <audio controls src="https://huggingface.co/useless-ai/pitput/resolve/main/samples/10-whisper.wav"></audio> |
## Usage
```bash
huggingface-cli login # the repository is private
git clone https://huggingface.co/useless-ai/pitput
cd pitput
pip install -r requirements.txt # PyTorch, Phonikud, onnxruntime, soundfile, huggingface_hub
python -m pitput.infer --style excited --text "<laugh> ืื ื™ ืœื ืžืืžื™ื ื”, ืงื™ื‘ืœื ื• ืืช ื”ื“ื™ืจื”!" --output out.wav
python -m pitput.infer --style default --addressee f --text "ืžื” ืฉืœื•ืžืš? ืฉืžืขืชื™ ืฉืœื ื”ืจื’ืฉืช ื˜ื•ื‘." --output hello.wav
```
On first use the Phonikud model (~300 MB) is downloaded to the Hugging Face cache. Options:
| Option | Meaning |
|---|---|
| `--style` | `default`, `excited`, `sad`, `angry`, `fearful`, `surprised`, `calm`, `whisper` |
| `--addressee m` / `f` | Gender of the person spoken to (ืœืš, ืื•ืชืš, ืืžืจืช โ€ฆ). Without it Phonikud reads the masculine form |
| `--text-file` | Read the text from a UTF-8 file instead of `--text` |
| `--speed` | 1.0 = as trained; 0.9 is 10% slower |
| `--pitch-shift` | Added to log F0 (0.058 โ‰ˆ one semitone) |
| `--pitch-range` | Scales the melody around each sentence's mean pitch (1.0 = as predicted) |
| `--device` | `cpu` (default) or `cuda` |
From Python:
```python
import soundfile as sf
from pitput.infer import CHECKPOINT, load_model, speak_text
model = load_model(CHECKPOINT) # inference.pt next to the package
wave, utterances = speak_text(model, "ืฉืœื•ื, <sigh> ืžื” ืฉืœื•ืžืš?", "calm", addressee="f")
sf.write("out.wav", wave.numpy(), model.config.sample_rate)
```
### What the input must look like
- **Plain Hebrew in spoken form.** There is no text normalizer: write numbers, dates, times, acronyms and
foreign words out in Hebrew words (`ื‘ืฉื‘ืข ื•ืขืฉืจื™ื`, not `7:20`). Characters it can't read are skipped.
- **No niqqud needed.** [Phonikud](https://github.com/thewh1teagle/phonikud) vocalizes each sentence in context
and adds stress; the model reads its phonemes.
- **Tags** go where the sound should be, in English inside the Hebrew: `<laugh>`, `<chuckle>`, `<sigh>`,
`<breath>`, `<gasp>`, `<throat-clearing>`, `<short pause>`, `<long pause>`.
- **Punctuation** shapes the delivery (`, . ? ! โ€ฆ`). Long texts are read sentence by sentence.
## Model
A fork of [fiato](https://huggingface.co/cfahlgren1/fiato), itself a fork of
[tiny-voca](https://github.com/n4ze3m/tiny-voca): a FastSpeech2-style model with a built-in
differentiable source-filter vocoder, so there is no separate neural vocoder.
- **Input:** 47 tokens: 29 Phonikud phonemes, 5 punctuation marks, word boundary, utterance edges and the 8
expression tags. Stress is a parallel feature, not a token.
- **Text encoder:** 4 transformer layers, 192 channels, with the style embedding added to every token.
- **Duration predictor:** frames per token, plus a "pause present" switch for punctuation, word boundaries and tags.
- **Frame level:** the style embedding is added again; 2 layers predict pitch (log F0), voicing and energy, and 5
layers predict the acoustic features.
- **Vocoder:** harmonic pulse train (from the predicted pitch) plus noise, each shaped by a complex cepstrum per frame.
- **Size:** 4,534,027 parameters at inference (18 MB checkpoint). Training also uses 2.7M parameters that are not
in the checkpoint (discriminators and an auxiliary mel head).
## Training data
The dataset is **synthetic only**, with no recorded human speech: 65,722 clips (139.3 h) of one synthetic voice,
of which 63,745 training clips (136.4 h) passed forced alignment. 160 held-out sentences (20 per style) were kept
for validation and 160 for test.
- **Text:** 65,728 Hebrew sentences from news (archive, HeSum, RSS), HeDC4 web text and stories, Hebrew Wikipedia,
ivrit.ai transcripts and LLM-written scenes (dialogue for the expressive styles).
- **Spoken form:** an LLM rewrote numbers, dates, money, acronyms and Latin script as Hebrew words, and every answer
was checked by code.
- **Addressee gender:** an LLM decided the gender of ambiguous second-person words (ืœืš, ืื•ืชืš, ืืžืจืช) in context.
The audio and the training phonemes follow the same decision.
- **Styles:** an LLM picked a style for each dialogue sentence and inserted the tags; calm and whisper were assigned
to plain statements. Each style was rendered with one fixed delivery instruction; default is a bright, friendly
delivery.
| Style | Clips synthesized | Rejected at alignment |
|---|---|---|
| default | 52,193 | 1% |
| calm | 4,900 | 1% |
| whisper | 4,900 | 1% |
| fearful | 1,104 | 42% |
| angry | 1,027 | 12% |
| sad | 685 | 19% |
| excited | 470 | 18% |
| surprised | 449 | 14% |
4,582 clips contain tags (at most 2 per clip); `<laugh>` appears in only about 90 of them.
- **Labels:** Phonikud phonemes with the addressee decisions applied, phonemized word by word with the tags at
their positions. Durations come from forced alignment with
[facebook/wav2vec2-lv-60-espeak-cv-ft](https://huggingface.co/facebook/wav2vec2-lv-60-espeak-cv-ft), which is used
only for labeling and is not part of the model. A tag owns the sound between the silences around it.
## Limitations
- **Laughs and other non-speech tags are weak.** The model gives each tag its time, but fills it with a quiet,
mostly unvoiced sound: a laugh comes out about 15 dB quieter than in the recordings and is heard as a soft breath.
Chuckle, throat-clearing and gasp suffer the same way. Pauses and breaths work better.
- **Intonation can sound a little robotic.** Pitch range is close to the recordings for most styles, but fearful
and surprised have only 70โ€“75% of it. The model speaks 3โ€“17% faster than the recordings, with shorter pauses.
- **Fearful is the weakest style:** it has the fewest usable training clips and the most pronunciation errors.
- **Pronunciation follows Phonikud,** including its mistakes (e.g. ื•ื•ืื• read as "vaส”u").
- **Second person defaults to masculine** unless `--addressee f` is given; there is no per-word decision at
inference.
- One voice only. Sentences are synthesized independently, with no prosody carried between them.
## License
- **Code** (`pitput/`): Apache License 2.0. It is a fork of fiato by cfahlgren1 and tiny-voca by Muhammed Nazeem,
both Apache-2.0; see `NOTICE`, `LICENSE` and `LICENSE.tiny-voca`.
- **Weights and audio samples: private research use only, not licensed for redistribution or any commercial use.**
Keep this repository private.
- **Phonikud** ([thewh1teagle/phonikud-onnx](https://huggingface.co/thewh1teagle/phonikud-onnx), MIT) is downloaded
at first use and is not included here.
## Files
| File | Contents |
|---|---|
| `inference.pt` | Weights, model configuration and front-end fingerprint (loaded with `weights_only=True`) |
| `pitput/` | Inference code: `infer.py` (command line and `speak_text`), `model.py`, `text.py` (phoneme tokens), `g2p.py` (Phonikud and addressee gender), `labels.py`, `config.py` |
| `samples/` | The audio above |
| `config.json`, `recipe.json` | Model and training settings of this run |
| `requirements.txt` | Python dependencies |
| `NOTICE`, `LICENSE`, `LICENSE.tiny-voca` | Attribution and license of the code |
## Acknowledgements
[tiny-voca](https://github.com/n4ze3m/tiny-voca) for the architecture and training recipe,
[fiato](https://huggingface.co/cfahlgren1/fiato) for expression tags on it, [Phonikud](https://github.com/thewh1teagle/phonikud)
for Hebrew vocalization and phonemes, and the Hebrew text sources: HeDC4, HeSum, Hebrew Wikipedia and ivrit.ai.