Make your own Hugin voice

hugin-finetune adapts Hugin to a new voice from about 500 recorded sentences. Pronunciation, word stress, rhythm and the English reading stay Hugin's; only the voice changes. Three commands do the whole job:

pip install -e ".[finetune,prominence]"     # torchaudio, transformers (+ Stanza, as for normal use)
hugin-finetune prepare my_recordings/ my_voice_work/      # alignment + training data (GPU or CPU)
hugin-finetune train   my_voice_work/ --steps 4000       # adaptation (NVIDIA GPU)
hugin-finetune export  my_voice_work/ my_voice.pt --name "My voice"
python -m hugin "Hej, det här är min egen röst." -o hej.wav --voice my_voice.pt

(python -m hugin.finetune ... works the same as hugin-finetune ....)

Rights and consent: read this first

A voice file reproduces a real person's voice. Only make voices from:

your own voice, or
the voice of someone who has given you explicit, informed permission, preferably in writing, for this specific use (training a synthetic voice, and where it will be used).

Do not make voices of public figures, celebrities, politicians, colleagues or anyone else from recordings you happen to have, and do not use a voice to impersonate someone, mislead listeners or get around consent. Treat the recordings and the finished .pt file as personal data: keep them private, and do not publish them unless the speaker agrees. Generated speech should not be presented as a real recording of the person. The B1 voice that ships with Hugin is synthetic; your own voice files are your responsibility.

What you need

Recordings: about 500 sentences, 35–45 minutes of speech, one speaker (see below). 100–200 sentences already give a recognisable voice, but rougher.
An NVIDIA GPU for train: the default settings use about 7 GB of GPU memory, so a 12 GB card is comfortable; on an 8 GB card use --frames 12000. Training on a CPU is not supported (it would take days). prepare and synthesis also run on a CPU.
Disk: about 1.2 GB for the alignment model (downloaded once), a few MB of training data and about 350 MB of checkpoints.
Internet on the first prepare: it downloads KBLab/wav2vec2-large-voxrex-swedish (Swedish speech recogniser, licence CC0-1.0) through transformers into the Hugging Face cache. Later runs work offline (HF_HUB_OFFLINE=1).

Recording

The model learns exactly what it hears, including the room, the microphone and the reading style.

Room: quiet, with soft furnishings (a wardrobe full of clothes, a sofa, curtains). No echo, no fans, no traffic, no music. A small room with carpets is better than a big bare one.
Microphone: the same microphone, position and distance (20–30 cm, slightly off-axis or with a pop filter) for every session. A decent USB microphone or headset is fine; a phone in a quiet room works too. Avoid laptop microphones.
Levels: peaks around -6 dBFS, never clipping. Record WAV or FLAC at 24 kHz or more (44.1/48 kHz is fine). No noise reduction, compression, EQ or other effects.
Reading: one sentence per file, 2–15 seconds (at most 20 s), with a short silence at the start and the end. Read in the style you want to hear back: calm narration gives a calm voice; mixed emotions give a more variable voice. Keep your energy and distance steady across sessions; take breaks, drink water.
Text: varied everyday sentences: statements, questions, exclamations, short and long sentences, numbers, names. If you want your voice to read English well, include 30–50 English sentences.
Mistakes: if you misread a sentence, record it again (or correct the transcript so it says exactly what you said).

Transcripts

Each clip needs the exact words that were said. Either put a metadata.csv in the folder:

# file|text   (lines starting with # are skipped; the extension is optional; subfolders are fine)
0001.wav|Hej, det här är den första meningen.
0002.wav|Kan du ringa mig i morgon efter lunch?
happy/0003.wav|Vilken fantastisk dag!

or put a text file next to every audio file: 0001.wav + 0001.txt. Audio may be WAV, FLAC, MP3 or OGG, mono or stereo, any sample rate.

The transcript goes through Hugin's own text front-end, exactly as at synthesis. Numbers, times and abbreviations are therefore expanded the way Hugin reads them (18:30 -> "arton trettio", kl. -> "klockan"). If you said something else, write it out as words. Check a transcript with python -m hugin --phonemes "Kl. 18:30 ...", which prints the normalised text. Inline pronunciations ({...}) are not supported in transcripts.

Step 1: prepare

hugin-finetune prepare my_recordings/ my_voice_work/ [--device cuda] [--user-lex my_lex.json] [--limit 20]

For every clip:

Audio: read, mixed to mono, resampled to 24 kHz, 80 Hz high-pass filter (as in Hugin's own training data), and exactly 200 ms of silence before the first sound and 250 ms after the last (longer silences are cut, shorter ones padded).
Units: the transcript becomes Hugin's phoneme units with the same code that synthesis uses: normalisation, automatic English detection and English units, word prominence when Stanza is installed, and your user dictionary if you pass --user-lex. Use the same settings when you later synthesise.
Alignment: the KBLab wav2vec2 model gives letter probabilities every 20 ms, and torchaudio's CTC forced alignment places each word of the normalised text in the audio. Word edges are then refined with the signal energy: a quiet stretch of at least 80 ms between two words becomes a pause, and the silence before the first and after the last word becomes the start and end silence. Inside each word the time is shared out over its phonemes in proportion to Hugin's own predicted durations, and pauses go to the space after a comma or full stop, where Hugin itself puts them.
Mel: the 80-band mel spectrogram that Hugin produces and its HiFT vocoder reads.

Clips that cannot be aligned reliably are rejected and listed: a word that cannot be found in the audio (usually a transcript that does not match), low overall alignment confidence (noise, another language, wrong file), a speaking time far from what the text needs, too long (> 20 s, set with --max-sec), too short, silent or unreadable files. Everything is written to my_voice_work/prepare_report.json (kept clips with their confidence, rejected clips with the reason). Fix the transcripts of rejected clips, or simply leave them out: losing a few percent does not matter.

Time: about 20 seconds for 250 clips on an RTX 5090, about a minute per 100 clips on a 12-core CPU (after the models are loaded).

Step 2: train

hugin-finetune train my_voice_work/ --steps 4000

This is frozen-language adaptation, the recipe used for Hugin's own adapted voices. It starts from the B1 model in weights/. A new voice tag (tag 2) starts as a copy of B1's tag 1. The language side (phoneme embeddings, encoder, duration predictor, encoder voice tags; 5.7 M parameters) is frozen, so pronunciation, stress and timing stay exactly Hugin's. The decoder, postnet and decoder voice tag (9.1 M) learn the new voice. The loss is L1 on the mel spectrogram plus a light GAN loss (a mel patch discriminator, which starts after 1000 steps). The settings are learning rate 1e-4 and batches of 20,000 mel frames; weights are fp32, with bf16 autocast where the GPU supports it. Ten clips are held out for validation.

Every 1000 steps it saves a checkpoint (the latest with optimiser state for resuming, older ones model-only for export --step), the validation loss and three test sentences in my_voice_work/train/ (samples/step004000_0.wav, ...); listen to them. loss.json has the loss curve. If training is interrupted, run the same command again: it continues from the last checkpoint. --steps 6000 on a finished run trains on from step 4000.

Time: about 7 minutes for 4000 steps on an RTX 5090 (18 min of audio). Expect roughly 20–40 minutes on a 12 GB card such as an RTX 3060 or 4070 (an estimate; the speed hardly depends on the amount of audio).

Useful options: --frames 10000 (less GPU memory), --adv 0 (no GAN loss), --save-every 500, --val 5, --fp32, --device cuda:1.

Step 3: export and use

hugin-finetune export my_voice_work/ my_voice.pt --name "My voice"         # latest checkpoint
hugin-finetune export my_voice_work/ my_voice_2k.pt --step 2000             # an earlier one

The voice file (about 30 MB) holds the adapted model and says which vocoder to use: the HiFT vocoder in weights/hift-s3gopen.fp16.pt, since the fast B1 Vocos vocoder only knows the B1 voice. Use it like the built-in voice:

python -m hugin "Hej, det här är min egen röst." -o hej.wav --voice my_voice.pt
python -m hugin.server --voice my_voice.pt          # then request model=my_voice

from hugin import Hugin
tts = Hugin.from_pretrained(voice="my_voice.pt")
audio, sr = tts.tts("Hej, det här är min egen röst.")

HiFT is heavier than Vocos: an own voice runs about 4–5× faster than real time on a CPU and much faster on a GPU.

What to expect

From a test with 250 sentences (18 minutes, read with five different emotions) by one speaker, 4000 steps:

prepare kept 248 of 250 clips. The 2 rejected clips had transcripts that did not match the recording (a word that was not said). Alignment agreed with Whisper's word timestamps to a median of 40 ms.
Validation mel loss went from 0.55 (B1 decoder) to 0.31 after 1000 steps, then about 0.32 once the GAN loss started (the GAN loss trades a little L1 for crisper sound). Training loss went from 0.13 to 0.11.
Speaker similarity (CAMPPlus speaker embeddings, against 10 held-out real clips): own voice 0.79, about the same as real clips against each other (0.79). The B1 voice scored 0.24.
Intelligibility (Whisper letter error): Swedish 2.3 % (B1: 0.2 %), English 0.0 %, mixed 1.9 % (same as B1).

More and cleaner recordings in one consistent style give a cleaner voice.

Troubleshooting

Problem	What to do
Many clips rejected with "word ... not found in the audio"	The transcripts do not match what was said. Compare with the audio, or check numbers and abbreviations with `python -m hugin --phonemes "..."`.
"low alignment confidence"	Noisy or reverberant audio, another language, or the wrong file for the transcript.
"too long"	Split long recordings into sentences (at most 20 s each).
"word count mismatch"	Unusual characters or markup in the transcript; write it as plain text.
`needs an NVIDIA GPU`	`train` needs CUDA. Check `python -c "import torch; print(torch.cuda.is_available())"`, or rent a GPU for an hour.
CUDA out of memory	`--frames 10000` (or lower).
The voice sounds rough, buzzy or muffled	More data, a quieter room, steady microphone distance. Compare the samples from step 1000 to 4000 and export the best one with `--step`.
Pronunciation differs from what you expect	Pronunciation comes from Hugin, not from your recordings. Use a user dictionary (`--user-lex`) both in prepare and at synthesis.
prepare and synthesis used different Stanza settings	Prepare with the same prominence setting you synthesise with (both with Stanza, or both with `--no-prominence`).
First prepare fails offline	The alignment model is downloaded on first use; run once with internet, or copy the Hugging Face cache.