laion/Humaneness-Voice-Large-1.1-Alpha

Humaneness Voice Large 1.1 Alpha

Completed full fine-tune. Current download: step 3,055 / 3,055. The verified complete weight file is selected by default.json, which updates automatically at training completion. Uploaded older snapshots remain individually labeled under checkpoints/. This is a full-parameter continuation, not a LoRA, and not a new pretraining from scratch.

Public English listening demo: matched old/new checkpoints, seven-emotion and 40-emotion cases, natural/organic variants, casual conversation and assistant replies, plus Gaston and ElevenLabs voice examples with/without references. Exact prompts, MP3 audio and metrics are public. Pending cells are labeled; no quality improvement is asserted before evaluation completes.

What it does

The model synthesizes speech from literal text and optional English directions for emotion, performance, pacing and voice. Use a clean, distinct 5–15-second reference for identity, or a known name such as Alice, Bob, Gaston or a trained Eleven-prefixed alias (see voices.json). Names are separate control labels, not words to pronounce. English/German and the added French/Dutch/Spanish material were trained; language reliability is experimental and needs testing.

Architecture, in plain language

A semantic transformer advances one 80-ms audio frame at a time with earlier text/audio frames in its context. Its hidden state conditions a local autoregressive Talker that predicts 12 residual codec indices per frame, each from a 1,024-entry codebook. A separate MOSS Audio Tokenizer v2 decodes them. 12.5 frames/s × 12 codes/frame = 150 indices/s, not 12 tokens per second. This is neither the flat-token experimental architecture nor the four-frame memory Talker.

Large inherits the SFT3/SFT4 MOSS local-transformer architecture. The 36-layer semantic transformer and one-layer local Talker both use 2,560-dimensional states. It has 4,022,468,096 semantic parameters and 107,522,048 local Talker/audio parameters: 4,129,990,144 total, counting tied heads once. Historical informal 4.55B/4.6B names are not the measured count of this checkpoint.

Codec weights are an upstream dependency, not bundled. Included tokenizer/configuration assets and implementation reconstruct the full model without separately downloading and then overwriting an initial semantic backbone.

Training

Large starts at SFT4 step 288, the final two-epoch Gemini continuation of the SFT3-based three-epoch acting model. Its earlier SFT3 and exact SFT4 starting weights are separate listening comparators.

Both models receive the same 390,914 presentations / 3,284.943 presentation-hours. Eligible unique training targets: original Gemini DE/EN 36,754 once; five-language extension 170,020 twice; Gaston 388 five times; ElevenLabs 2,030 six times. Original TAR payloads are read with a SQLite offset map, not duplicated per presentation. The mix is stratified over 100 run sections and shuffled so oversampled small sources appear throughout the beginning, middle and end. Gaston always has an explicit French Accent cue in GENERAL.

Approximately half the presentations use a different utterance by the same provider speaker, with different normalized transcript/audio hash, same split and speech-rich 5–15-second crop. The rest use stable learned pseudonyms. Three prompt views rotate: GENERAL + inline SCRIPT; GENERAL + plain SCRIPT; inline SCRIPT without GENERAL. Ten percent drop caption/control cues while retaining spoken text and identity; Gaston retains its accent cue. Validation targets never receive optimizer updates.

AdamW updates all active semantic/Talker parameters with BF16 compute and FP32 weights. 64 GH200 GPUs, microbatch one and accumulation 2, global batch 128. Peak LR: 8e-06 semantic / 2.4e-05 Talker. 5% linear warmup, then one cosine decay to 10% of peak covers the entire mix. Allocation changes resume the same optimizer/RNG/schedule, not a new warmup. Latest 14-case validation loss: 4.164423295429775; this diagnostic is not a representative speech-quality test.

Inference

Download the WHOLE repository (hf download laion/Humaneness-Voice-Large-1.1-Alpha --local-dir model), install requirements.txt, and use a CUDA GPU:

python model/infer.py --model-dir model --language en --alias Alice   --text 'We finally made it home.'   --prompt 'GENERAL: Warm, relieved and naturally conversational.
SCRIPT:
(relieved, softly exhaling) [3.2 seconds duration] We finally made it home.'   --frames 75 --seed 777 --output example.wav

Replace --alias Alice with --reference-wav clean-speaker.wav for cloning. Gaston: --alias Gaston plus French Accent in GENERAL. Other names are listed in voices.json; Eleven aliases retain their prefix to avoid collisions with Gemini names. --language supports en/de/fr/nl/es. --codec-path LOCAL_SNAPSHOT supports offline inference. The cap is 15 seconds, never the old 37-frame/2.96-second crop. Audio is cropped BEFORE codec encoding; shorter-than-five-second inputs warn.

The release generator uses this same public loader. It strips Accelerate's base. prefix, rejects unexpected missing keys and restores the shared LM heads by re-tying weights. Do not call AutoModel.from_pretrained directly on these checkpoint files. GPU canaries verify the actual Small/Large load, generation and long-reference paths; see stats/public_inference_canary.json when available. The tested stack is GH200, PyTorch 2.9 and Transformers 5.14.1; other stacks require portability testing.

Prompting and reproduction

Only literal speech goes in --text. GENERAL/SCRIPT directions go in --prompt; round parentheses hold sentence delivery and vocal bursts, square brackets hold durations and pauses. --alias goes into the Reference(s) speaker field, never the spoken transcript. Use inline [3.7 seconds duration] next to each sentence for timing prompts; this release does not claim perfect duration realization.

TRAINING.md gives exact data selection, lineage, source repetitions, resumption, code layout and evaluation limits. training/ contains the actual compiler, packed reader, full trainer, causal attention memory fix and Slurm scripts. stats/ retains readiness proofs, contracts and loss/validation histories. JUPITER constants need adapting elsewhere. Full optimizer/RNG states are retained locally; public snapshots contain complete model weights, not the very large optimizer state. evaluation/ contains the exact frozen listening manifest and pipeline code. Both repositories link the same public demo.

Research use and ethics

Primary recommended uses are research and dataset augmentation. Obtain speaker consent, disclose synthetic speech and avoid deceptive impersonation. Test identity, linguistic performance, safety and artifacts before production. LAION does not endorse deceptive uses or guarantee deployment fitness. These recommendations are not additional restrictions on CC BY 4.0. Model weights/newly authored release materials: CC BY 4.0; upstream software/model/codec and personality rights remain applicable.

Optional RL v1 rank-128 adapter (Large)

A separate experimental RL v1 adapter is available for this final Alpha checkpoint. It improves predicted Content Enjoyment and reference-conditioned WER in the matched listening comparison, but does not clearly improve voice cloning and reduces VoiceCLAP instruction similarity in several conditions. It is optional: the base model default above is unchanged. Use the adapter repository's pinned-base inference loader, not PEFT or an arbitrary merge. Listen to the matched controls and RL recordings; the adapter card includes full metrics, training code and limitations. This is the original CE/VoiceCLAP/identity reward, not the newer emotion/talking-style reward proposal.