Ronin48LLC/generic-science-professor-chatterbox-lora

Generic Science Professor

A LoRA adapter for Chatterbox (Resemble AI's open text-to-speech model) that gives the T3 text-to-speech-token stage the cadence, pacing and phrasing of a mid-century American physics professor: a plain-spoken, slightly amused lecture-hall delivery that reads technical prose without sounding like a newsreader.

It was built to narrate graduate textbooks (AI, machine learning, statistics) as long-form audiobooks for one listener. It is good at exactly that: paragraphs of expository English with equations spelled out in words.

This is v2. v2 fixes v1's clipped sentence endings: v1 often cut off the last sound of a sentence ("actions" came out "action", "a reference" came out "a ref"). The cause was the training data, not the base model; see Version history. v1 is still available at revision v1.

Sample

samples/sample.wav is a 35-second v2 render of a short original monologue ("Hello, Solutions Architect, are you tired..."), written as one comma-spliced run-on sentence so it is read as two long generate() calls rather than many short ones. Settings are those in the usage example: exaggeration 0.5, cfg_weight 0.3, temperature 0.8, repetition penalty 1.4, loudness-normalized (EBU R128, I = -16 LUFS). Each call was re-seeded until Whisper small.en read it back at 0.95 or better and the last 30 ms of speech had decayed to 5% of speech level or less (first call: 1 seed, second: 9). Take selection only: no audio was spliced or faded by hand.

Hear the difference: v1 vs v2 sentence endings

The same five sentences from the preface of an AI textbook, read by v1 and by v2 with the same prompt clip, the same random seed per sentence and the same settings. Plain generate(): no text tricks, no retries, no post-processing. Listen to the last word of each sentence.

v1 (clips the final sound):

v2 (finishes the word):

What Whisper small.en hears at the end of each sentence:

sentence ends with	v1	v2
"...and perform actions."	"perform action"	"perform actions"
"...the instructor and students."	"instructor and student"	"instructor and students"
"...or as a reference."	"as a ref"	"as a reference"
"...and an extensive bibliography."	"an extensive bibriography"	"an extensive bibliography"
"...some of the topics."	"of the top"	"of the topics"

Model details

Model type	PEFT LoRA adapter on the T3 stage of Chatterbox (Llama-style text-to-speech-token transformer). S3Gen and the voice encoder are untouched.
Base model	ResembleAI/chatterbox (chatterbox-tts 0.1.2 weights, English tokenizer, 704 tokens)
Adapter	r = 32, alpha = 64, dropout 0.05, on q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, spkr_enc. text_emb and text_head are fully trained and saved with the adapter (modules_to_save).
Size	96 MB (adapter_model.safetensors)
Language	English
License	MIT, matching the base model
Developed by	Ronin48 LLC
Training toolkit	gokhaneraslan/chatterbox-finetuning at commit fac31c4, with local patches described below

How to use

The adapter loads onto the stock Chatterbox T3 module with PEFT. Chatterbox still needs an audio prompt: the adapter carries the speaker's rhythm and phrasing inside T3, but timbre comes from the prompt clip that S3Gen conditions on. A prompt clip of 5 to 15 seconds from the same speaker gives the most faithful result; any clean prompt still gets the delivery.

import torchaudio
from chatterbox.tts import ChatterboxTTS
from peft import PeftModel

model = ChatterboxTTS.from_pretrained(device="cuda")
model.t3 = PeftModel.from_pretrained(
    model.t3, "Ronin48LLC/generic-science-professor-chatterbox-lora"
    # , revision="v1"   # the original release, if you need it
)
model.t3 = model.t3.merge_and_unload().eval()

text = (
    "The field of artificial intelligence is concerned with not just "
    "understanding but also building intelligent entities."
)
wav = model.generate(
    text,
    audio_prompt_path="prompt.wav",
    exaggeration=0.5,
    cfg_weight=0.3,
    temperature=0.8,
    repetition_penalty=1.4,
)
torchaudio.save("out.wav", wav, model.sr)

Settings that worked for narration: exaggeration 0.5, cfg_weight 0.3 (slower, more deliberate), temperature 0.8, repetition_penalty 1.4, text chunked at sentence boundaries to at most 380 characters, and every chunk transcribed back with Whisper small.en and re-rendered if the read-back drifts. Spell numbers, symbols and equations out as words before synthesis; the training text was prepared that way.

You can also merge the adapter into the T3 weights with the toolkit's merge_lora.py and drop the result in as t3_cfg.safetensors next to the stock ve, s3gen, conds and tokenizer files. Plain Chatterbox then loads it with no PEFT dependency.

Training data

About 135 minutes of speech from a single speaker, cut from archival interview and lecture recordings. The recordings are not included here and are not redistributed.

v2 uses the same speech as v1 but re-cuts and re-labels every clip. Nothing that v1 accepted was dropped.

1. Source audio resampled to 24 kHz mono.
2. faster-whisper large-v3 with word timestamps, utterances of 2 to 17 seconds.
3. Speaker filter: Chatterbox's own VoiceEncoder cosine similarity against a few reference clips of the speaker, floor 0.83 or median minus 3 MAD, transcript probability >= 0.55. This drops interviewers, audience and noisy passages.
4. Honest clip edges (new in v2). Whisper's word end times run early, so v1's clips were cut at the timestamp and most ended mid-sound. v2 finds where speech actually starts and stops from a 10 ms loudness envelope against the local noise floor. Where the speaker runs straight on into the next utterance, the two are merged until a real pause (69 merges). Where he never pauses, the cut goes at the quietest point between words (176 clips).
5. One format for every clip (new in v2). 10 ms fades, then exactly 150 ms of digital silence before and after the speech, and RMS-normalised to the same speech level (peak at most 0.89).
6. Audio and text agree (new in v2). Every clip is transcribed again with large-v3. If the transcript's last word is not the last word heard, the end is moved out to the next pause and re-checked (80 clips). If that still does not match, the label is replaced with what was actually said (117 clips). 602 clips matched as cut.
7. Numbers spelled out, terminal punctuation added unless the clip trails off mid-thought, written to LJSpeech metadata.csv (id|text|text).

Result: 799 clips, 135.0 minutes. 720 train / 79 validation (fixed seed, 10 % held out). After cleaning, no clip ends with less than 30 ms of silence; in v1, 87 % did.

Training procedure

• LoRA r = 32, alpha = 64, learning rate 3e-5, 20 epochs, batch 8 x grad-accum 4 (effective 32), fp16 mixed precision, dataloader_num_workers = 0. Same recipe as v1; only the data changed.
• The upstream toolkit has no validation. train.py was patched to hold out a fixed-seed 10 % split, evaluate every 40 steps, stop early (patience 3) and keep the checkpoint with the lowest validation loss (load_best_model_at_end).
• fp16 rather than bf16 because the training card (Tesla PG500-216, compute 7.0) has no bf16.
• Wall clock: 30 minutes on one 32 GB Tesla.

Validation loss (T3 token cross-entropy, lower is better)

epoch	0	1.8	3.6	5.3	7.1	8.9	10.7	12.4	14.2	16.0	17.8	19.6
eval loss	7.54	6.03	5.79	5.71	5.66	5.64	5.62	5.62	5.61	5.61	5.62	5.61

The best checkpoint, at epoch 16.0 (step 360, eval loss 5.6115), is what ships here. Epoch 0 is zero-shot Chatterbox on the same held-out clips. Validation loss flattens after epoch 10 and never turns upward, and training loss stays about 0.3 below it, so the adapter is not memorising its clips. v2's validation set differs from v1's, so the two tables are not directly comparable.

Sentence endings

Five sentences whose last word v1 reliably clipped, three seeds each, same prompt clip, Whisper read-back of the last word:

model	plain text	text ending in "..."
stock Chatterbox, no adapter	15 / 15	15 / 15
v1 adapter	3 / 15	7 / 15
v2 adapter	14 / 15	14 / 15

What did not work

Earlier runs on a 50-minute subset at r = 128, alpha = 256, lr 1e-4 memorised within five epochs: validation loss bottomed at epoch 5 and then climbed past the untrained baseline, and the samples sounded robotic and eerie. A small adapter at a low learning rate converged cleanly, and adding data (2.7x more data) beat adding epochs everywhere it was measured. On 49 held-out clips from material neither adapter had seen, the 50-minute r = 32 adapter scored 5.99 and the full-data v1 adapter scored 5.74.

Filtering out the awkward clips instead of fixing them was also tried for v2: it kept 435 clips (63 minutes) and was abandoned before training. Reformatting every clip kept all 135 minutes.

Intended use

• Narrating long technical prose: textbooks, papers, lecture notes, documentation.
• Personal audiobooks and study material.
• A starting point for further Chatterbox LoRA work; the recipe above is the useful part.

Out of scope

• Impersonating any real person, living or dead, or presenting output as a genuine recording of anyone.
• Anything that needs the speaker's actual timbre without a consenting prompt clip.
• Short conversational lines, dialogue, singing or emotional range. The adapter was trained on expository lecture speech and tends to flatten anything else.
• Languages other than English.

Limitations

• Prosody is learned; timbre is not. Without a matching prompt clip you get the delivery on someone else's voice.
• Hallucinated or dropped words happen on long chunks. Keep chunks under about 380 characters and verify with ASR read-back if the text matters.
• Numbers, symbols and abbreviations must be pre-expanded to words.
• The validation set is small (86 clips), so the loss numbers above are a guide, not a benchmark.

Version history

• v2 (2026-09-30, revision v2, current main): same recipe, re-cut and re-labelled data as described above. Fixes clipped sentence endings.
• v1 (2026-09-30, revision v1): the original release. 868 clips cut at Whisper word timestamps with 80 ms of padding; 87 % of clips ended with under 30 ms of silence and 57 % were still loud at the cut, so the adapter learned to end sentences abruptly. Validation loss 7.72 (zero-shot) to 5.62 at epoch 16.3 on its own 86-clip split, 47 minutes of training.

Framework versions

• PEFT 0.17.1
• chatterbox-tts 0.1.2
• torch 2.6 (cu124)
• transformers as pinned by chatterbox-tts 0.1.2

Citation

@misc{generic-science-professor-2026,
  title  = {Generic Science Professor: a Chatterbox LoRA for technical narration},
  author = {Ronin48 LLC},
  year   = {2026},
  url    = {https://huggingface.co/Ronin48LLC/generic-science-professor-chatterbox-lora}
}

Base model: Resemble AI, Chatterbox, https://github.com/resemble-ai/chatterbox Training toolkit: https://github.com/gokhaneraslan/chatterbox-finetuning