pitput / README.md
useless-ai's picture
pitput try 1 (step 104,000)
b9fa092 verified
|
Raw History Blame Contribute Delete
10.9 kB
metadata
language:
  - he
license: other
license_name: private-research-only
license_link: https://huggingface.co/useless-ai/pitput/blob/main/README.md#license
pipeline_tag: text-to-speech
library_name: pytorch
tags:
  - text-to-speech
  - tts
  - hebrew
  - fastspeech2
  - ddsp

pitput

pitput (ืคึดึผื˜ึฐืคึผื•ึผื˜) is a Hebrew take on tiny-voca: a tiny TTS model built to run on a CPU. The whole model, vocoder included, is only 18 MB (4.5M parameters). It outputs 24 kHz mono and runs faster than real time on a laptop CPU. It reads plain Hebrew without niqqud, in one voice with 8 delivery styles and 8 expression tags (<laugh>, <sigh>, <short pause> โ€ฆ).

It is a personal learning project, trained from scratch on synthetic speech. This is try 1 (step 104,000, November 18, 2026). Private research use only; see License.

Samples

The 10 listening texts that were synthesized after every epoch, here from the final weights.

Style Text Audio
default ืœืคื™ ื ืชื•ื ื™ ื”ืœืฉื›ื” ื”ืžืจื›ื–ื™ืช ืœืกื˜ื˜ื™ืกื˜ื™ืงื”, ืื•ื›ืœื•ืกื™ื™ืช ื™ืฉืจืืœ ื’ื“ืœื” ื‘ืฉื ื” ื”ืื—ืจื•ื ื” ื‘ื›ืžืื” ื•ืฉืžื•ื ื™ื ืืœืฃ ืื™ืฉ.
default, addressee f ืžื” ืฉืœื•ืžืš? ืฉืžืขืชื™ ืฉืœื ื”ืจื’ืฉืช ื˜ื•ื‘ ื‘ืฉื‘ื•ืข ืฉืขื‘ืจ.
default ื”ืจื›ื‘ืช ืœื—ื™ืคื” ื™ื•ืฆืืช ื‘ืฉื‘ืข ื•ืขืฉืจื™ื ื•ืžื’ื™ืขื” ื‘ืฉืžื•ื ื” ื•ื—ืฆื™, <chuckle> ืื ื”ื™ื ื‘ื›ืœืœ ืชื’ื™ืข.
excited <laugh> ืื ื™ ืœื ืžืืžื™ื ื”, ืงื™ื‘ืœื ื• ืืช ื”ื“ื™ืจื”!
sad <sigh> ื—ื™ื›ื™ืชื™ ืœื• ื›ืœ ื”ืขืจื‘, ื•ื”ื•ื ืคืฉื•ื˜ ืœื ื”ื’ื™ืข.
angry, addressee m ื›ืžื” ืคืขืžื™ื ืืžืจืชื™ ืœืš ืœืกื’ื•ืจ ืืช ื”ื“ืœืช?!
fearful ืฉืžืขืช ืืช ื–ื”? <gasp> ื™ืฉ ืžื™ืฉื”ื• ื‘ื—ื•ืฅ.
surprised ืจื’ืข, <short pause> ื”ื•ื ื‘ืืžืช ื”ืชืคื˜ืจ? ืื—ืจื™ ืขืฉืจื™ื ืฉื ื”?
calm <breath> ืงื—ื• ื ืฉื™ืžื” ืขืžื•ืงื”, <long pause> ื•ืชื ื• ืœื’ื•ืฃ ืœื”ื™ืจื’ืข ืœืื˜ ืœืื˜.
whisper ื‘ื•ื ื ื“ื‘ืจ ื‘ืฉืงื˜, ื›ื•ืœื ื›ื‘ืจ ื™ืฉื ื™ื.

Usage

huggingface-cli login                  # the repository is private
git clone https://huggingface.co/useless-ai/pitput
cd pitput
pip install -r requirements.txt        # PyTorch, Phonikud, onnxruntime, soundfile, huggingface_hub

python -m pitput.infer --style excited --text "<laugh> ืื ื™ ืœื ืžืืžื™ื ื”, ืงื™ื‘ืœื ื• ืืช ื”ื“ื™ืจื”!" --output out.wav
python -m pitput.infer --style default --addressee f --text "ืžื” ืฉืœื•ืžืš? ืฉืžืขืชื™ ืฉืœื ื”ืจื’ืฉืช ื˜ื•ื‘." --output hello.wav

On first use the Phonikud model (~300 MB) is downloaded to the Hugging Face cache. Options:

Option Meaning
--style default, excited, sad, angry, fearful, surprised, calm, whisper
--addressee m / f Gender of the person spoken to (ืœืš, ืื•ืชืš, ืืžืจืช โ€ฆ). Without it Phonikud reads the masculine form
--text-file Read the text from a UTF-8 file instead of --text
--speed 1.0 = as trained; 0.9 is 10% slower
--pitch-shift Added to log F0 (0.058 โ‰ˆ one semitone)
--pitch-range Scales the melody around each sentence's mean pitch (1.0 = as predicted)
--device cpu (default) or cuda

From Python:

import soundfile as sf
from pitput.infer import CHECKPOINT, load_model, speak_text

model = load_model(CHECKPOINT)  # inference.pt next to the package
wave, utterances = speak_text(model, "ืฉืœื•ื, <sigh> ืžื” ืฉืœื•ืžืš?", "calm", addressee="f")
sf.write("out.wav", wave.numpy(), model.config.sample_rate)

What the input must look like

  • Plain Hebrew in spoken form. There is no text normalizer: write numbers, dates, times, acronyms and foreign words out in Hebrew words (ื‘ืฉื‘ืข ื•ืขืฉืจื™ื, not 7:20). Characters it can't read are skipped.
  • No niqqud needed. Phonikud vocalizes each sentence in context and adds stress; the model reads its phonemes.
  • Tags go where the sound should be, in English inside the Hebrew: <laugh>, <chuckle>, <sigh>, <breath>, <gasp>, <throat-clearing>, <short pause>, <long pause>.
  • Punctuation shapes the delivery (, . ? ! โ€ฆ). Long texts are read sentence by sentence.

Model

A fork of fiato, itself a fork of tiny-voca: a FastSpeech2-style model with a built-in differentiable source-filter vocoder, so there is no separate neural vocoder.

  • Input: 47 tokens: 29 Phonikud phonemes, 5 punctuation marks, word boundary, utterance edges and the 8 expression tags. Stress is a parallel feature, not a token.
  • Text encoder: 4 transformer layers, 192 channels, with the style embedding added to every token.
  • Duration predictor: frames per token, plus a "pause present" switch for punctuation, word boundaries and tags.
  • Frame level: the style embedding is added again; 2 layers predict pitch (log F0), voicing and energy, and 5 layers predict the acoustic features.
  • Vocoder: harmonic pulse train (from the predicted pitch) plus noise, each shaped by a complex cepstrum per frame.
  • Size: 4,534,027 parameters at inference (18 MB checkpoint). Training also uses 2.7M parameters that are not in the checkpoint (discriminators and an auxiliary mel head).

Training data

The dataset is synthetic only, with no recorded human speech: 65,722 clips (139.3 h) of one synthetic voice, of which 63,745 training clips (136.4 h) passed forced alignment. 160 held-out sentences (20 per style) were kept for validation and 160 for test.

  • Text: 65,728 Hebrew sentences from news (archive, HeSum, RSS), HeDC4 web text and stories, Hebrew Wikipedia, ivrit.ai transcripts and LLM-written scenes (dialogue for the expressive styles).

  • Spoken form: an LLM rewrote numbers, dates, money, acronyms and Latin script as Hebrew words, and every answer was checked by code.

  • Addressee gender: an LLM decided the gender of ambiguous second-person words (ืœืš, ืื•ืชืš, ืืžืจืช) in context. The audio and the training phonemes follow the same decision.

  • Styles: an LLM picked a style for each dialogue sentence and inserted the tags; calm and whisper were assigned to plain statements. Each style was rendered with one fixed delivery instruction; default is a bright, friendly delivery.

    Style Clips synthesized Rejected at alignment
    default 52,193 1%
    calm 4,900 1%
    whisper 4,900 1%
    fearful 1,104 42%
    angry 1,027 12%
    sad 685 19%
    excited 470 18%
    surprised 449 14%

    4,582 clips contain tags (at most 2 per clip); <laugh> appears in only about 90 of them.

  • Labels: Phonikud phonemes with the addressee decisions applied, phonemized word by word with the tags at their positions. Durations come from forced alignment with facebook/wav2vec2-lv-60-espeak-cv-ft, which is used only for labeling and is not part of the model. A tag owns the sound between the silences around it.

Limitations

  • Laughs and other non-speech tags are weak. The model gives each tag its time, but fills it with a quiet, mostly unvoiced sound: a laugh comes out about 15 dB quieter than in the recordings and is heard as a soft breath. Chuckle, throat-clearing and gasp suffer the same way. Pauses and breaths work better.
  • Intonation can sound a little robotic. Pitch range is close to the recordings for most styles, but fearful and surprised have only 70โ€“75% of it. The model speaks 3โ€“17% faster than the recordings, with shorter pauses.
  • Fearful is the weakest style: it has the fewest usable training clips and the most pronunciation errors.
  • Pronunciation follows Phonikud, including its mistakes (e.g. ื•ื•ืื• read as "vaส”u").
  • Second person defaults to masculine unless --addressee f is given; there is no per-word decision at inference.
  • One voice only. Sentences are synthesized independently, with no prosody carried between them.

License

  • Code (pitput/): Apache License 2.0. It is a fork of fiato by cfahlgren1 and tiny-voca by Muhammed Nazeem, both Apache-2.0; see NOTICE, LICENSE and LICENSE.tiny-voca.
  • Weights and audio samples: private research use only, not licensed for redistribution or any commercial use. Keep this repository private.
  • Phonikud (thewh1teagle/phonikud-onnx, MIT) is downloaded at first use and is not included here.

Files

File Contents
inference.pt Weights, model configuration and front-end fingerprint (loaded with weights_only=True)
pitput/ Inference code: infer.py (command line and speak_text), model.py, text.py (phoneme tokens), g2p.py (Phonikud and addressee gender), labels.py, config.py
samples/ The audio above
config.json, recipe.json Model and training settings of this run
requirements.txt Python dependencies
NOTICE, LICENSE, LICENSE.tiny-voca Attribution and license of the code

Acknowledgements

tiny-voca for the architecture and training recipe, fiato for expression tags on it, Phonikud for Hebrew vocalization and phonemes, and the Hebrew text sources: HeDC4, HeSum, Hebrew Wikipedia and ivrit.ai.