Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# VibeVoice Final Samples β the whole story in audio
|
| 2 |
+
|
| 3 |
+
Joint-finetuned VibeVoice-1.5B (LLM-LoRA + full diffusion head, warm-started chain
|
| 4 |
+
fullv3 -> emo3 -> emo4). All samples generated with our emotion-steering stack.
|
| 5 |
+
|
| 6 |
+
## Listening order
|
| 7 |
+
1. **01_dialogue_baseline_stock** β stock VibeVoice (the read-only baseline)
|
| 8 |
+
2. **02_dialogue_our_model_fullv3** β OUR model, ear-certified intelligible dialogue
|
| 9 |
+
(WER 0.36 vs stock 0.03; 5 prior head-only attempts produced babble β this is the joint-training proof)
|
| 10 |
+
3. **03_dialogue_polished_joint2** β the polished twin (more training: a wash statistically, different samples)
|
| 11 |
+
4. **04_emotion_reference_steering_cremad** β R1: swap the reference clip's EMOTION (same actors,
|
| 12 |
+
same dialogue): ANGRY_ANGRY / SAD_SAD / **ANGRY_SAD_mixed** (the per-speaker money clip).
|
| 13 |
+
Note: CREMA-D actors carry non-US accents β the accent transfer is itself evidence of total reference conditioning.
|
| 14 |
+
5. **05_emotion_us_accent_palette** β R1 US-accent (RAVDESS): same sentence, same voice, 8 emotions
|
| 15 |
+
(measured F0 tracks emotion: ANGRY 328 Hz ... SAD lowest)
|
| 16 |
+
6. **06_emotion_3speaker_drama** β 3-speaker scene: angry / fearful / surprised in one conversation
|
| 17 |
+
7. **07_emotion_per_turn_switch** β same actor goes ANGRY -> relieved mid-dialogue (dual-slot authoring)
|
| 18 |
+
8. **08_ship_config_emo4_20ddpm** β THE SHIP CONFIG: emo4 (clean-data scale-up) at 20 DDPM steps,
|
| 19 |
+
<emotion> tags + matching references. tag2_tags_matching_refs_BEST.wav = the final product demo.
|
| 20 |
+
9. **09_quality_ab** β the quality diagnosis: the 'webcam-mic static' was REFERENCE-carried
|
| 21 |
+
(CREMA-D refs: noise floor 0.006 in generation vs 0.0001-0.0004 with clean refs; 20 DDPM steps = 4x high-band)
|
| 22 |
+
|
| 23 |
+
## Ship recipe
|
| 24 |
+
emo4 adapter (vibevoice-lora-joint/emo4) | 20 DDPM steps | script tags <angry>/<sad>/... |
|
| 25 |
+
per-speaker emotional reference clips (clean recordings) | CFG 1.3
|
| 26 |
+
|
| 27 |
+
## Metrics
|
| 28 |
+
- Intelligibility: our model WER 0.32-0.41 band vs stock 0.03 (single-run; run-to-run +-0.1)
|
| 29 |
+
- Speaker identity: SIM 0.92-0.93 vs stock 0.94
|
| 30 |
+
- Emotion: F0 shifts 300->184 Hz (reference steering); tag channel -87 Hz with refs held constant
|
| 31 |
+
- Speed: architecture-bound ~1.4-1.5x slower than stock (launch-bound; measured ceiling)
|