laion/Humaneness-Voice-Small-Acting

Humaneness Voice Small · Acting fine-tune (three-epoch checkpoint)

This is the complete Small-model checkpoint after three full-parameter fine-tuning epochs on the English/German acting-challenges tuning split. It starts from the reference-conditioned Humaneness Voice Small full fine-tune at step 26,774 in the base repository. It is not a LoRA or the original S1–S10 base checkpoint. Its final training step is 1,725. The checkpoint is released as a historical, immutable comparison point; a further two-epoch continuation may be added separately after evaluation. Do not interpret this repository as proof that learned voice names already work reliably.

The model has a Qwen3-0.6B-family semantic backbone, a compact MOSS-style local Talker, and uses MOSS Audio Tokenizer v2 for 12 RVQ codes per 80-ms audio frame. The checkpoint contains about 708.8 million trainable parameters. Codec weights are an upstream dependency and are not included here.

What was trained

The pinned dataset revision is 2f914a5081f6c0d6eb348b509ea50cf1d56aa7e3. After excluding one anomalous sample with empty ASR, the training split contains 36,754 clips / 300.27 unique audio hours. Every clip is presented three times (110,262 presentations, 900.81 presentation-hours): once with GENERAL and inline SCRIPT cues, once with GENERAL and plain SCRIPT, and once with inline SCRIPT but no GENERAL. The three views are shuffled across epochs rather than batched by format. Exactly half the presentations use a different utterance by the same speaker, cropped to 12 seconds, as an audio reference; the rest use one of 30 stable pseudonyms such as Alice and Bob. A balanced 10% control-dropout arm removes the GENERAL text and inline cues while preserving literal speech and its assigned speaker condition. The held-out challenge groups never become training targets.

The Text field of the MOSS user message contains only the spoken transcript. Acting directions, durations, pauses and vocal bursts go in the Instruction field. The alias occupies - Reference(s):\nSpeaker: NAME, not the spoken text. When audio-conditioned, a distinct same-speaker reference occupies the model's actual audio-reference channel. Do not replace it with text. For voice cloning, prefer a clean 5–15-second reference; the training crop here is 12 seconds. The older 2.96-second cap was a historical loader/inference limit, not a codec requirement.

Training used full parameters with 32 GH200 GPUs (8 JUPITER nodes), global batch 64, AdamW, BF16 compute with FP32 parameters, 3% linear warmup and one cosine decay to 10% of peak. Peak learning rates were 2.5e-5 for the semantic backbone and 5e-5 for Talker/bridge. All three epoch-boundary checkpoints retained optimizer/RNG state locally; this repository publishes the final inference weights and the exact training code, not the much larger optimizer files. The full training contract and packed-data path conventions are under training/.

Checkpoint and use

checkpoints/step-00001725/model.safetensors contains all trained model weights. It was produced by Accelerate from a wrapper called base; tensor names therefore have a base. prefix. The tested local generator strips that prefix before loading and re-ties the intentionally omitted shared LM heads. Do not pass this file directly to AutoModel.from_pretrained(). The standalone infer.py in this repository demonstrates loading, alias/reference conditioning, codec decoding and a WAV output. Example:

python infer.py --checkpoint checkpoints/step-00001725/model.safetensors \
  --text 'We finally made it home.' \
  --prompt 'GENERAL: relieved and warm, natural close-mic speech.\nSCRIPT:\n(relieved, exhaling) [2.7 seconds duration] We finally made it home.' \
  --alias Alice --frames 65 --output example.wav

Use --reference-wav speaker.wav instead of --alias Alice to condition on a real speaker recording. Run on a CUDA GPU, install the Python dependencies listed in requirements.txt, and accept the upstream codec's own license/terms. The code requires trust_remote_code=True for the included MOSS processor/model modules. It was tested on JUPITER GH200 with Python 3.13, PyTorch 2.9 and BF16; other platforms may require version adjustments.

Reproducibility and caveats

training/ includes the manifest/prompt compiler, packed TAR reader, full-FT code and Slurm template used for this run. The scripts contain JUPITER-specific absolute paths and a hash-checked source revision. To retrain on another system, stage the same licensed TAR data, replace the path constants, recompile the prompt cache and run a multi-GPU canary before the full job. Neither per-sample audio files nor optimizer-state blobs are published here.

The dataset and the original Small model explain provenance and prompting in more detail. A local 70-prompt A/B listening grid compared this checkpoint with the concurrent Large-model fine-tune; it is a listening aid, not a blinded or objective quality benchmark. Formal reference-vs-alias speaker-consistency evaluation is ongoing. Identity prompts, especially name-only prompts, may fail or sound unstable. This is a research release; disclose synthetic speech, avoid impersonation and obtain consent for any real speaker references.

© LAION contributors. Model weights and newly authored release materials: CC BY 4.0. Attribute the base model and dataset; upstream codec and Qwen components have their own terms.