marcosremar2 commited on
Commit
2bb66fc
·
verified ·
1 Parent(s): 8f69ad7

iSTFT 16,7M pt-BR com emoções (professor VoxCPM2)

Browse files
.gitattributes CHANGED
@@ -33,3 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
 
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ samples/f3_istft.wav filter=lfs diff=lfs merge=lfs -text
37
+ samples/f3_istft_emo_animado.wav filter=lfs diff=lfs merge=lfs -text
38
+ samples/f3_istft_emo_raiva.wav filter=lfs diff=lfs merge=lfs -text
39
+ samples/f3_istft_emo_suave.wav filter=lfs diff=lfs merge=lfs -text
40
+ samples/f3_istft_emo_surpresa.wav filter=lfs diff=lfs merge=lfs -text
41
+ samples/f3_istft_emo_triste.wav filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,56 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ language:
4
+ - pt
5
+ base_model: hexgrad/Kokoro-82M
6
+ pipeline_tag: text-to-speech
7
+ tags:
8
+ - kokoro
9
+ - tts
10
+ - emotion
11
+ - synthetic-voice
12
+ ---
13
+
14
+ # Kokoro 16,7M iSTFT pt-BR com emoções (Rasalgethi)
15
+
16
+ Modelo derivado do [Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) (Apache 2.0): mesma arquitetura e mesmos fonemas, com o BERT do Kokoro, gerador iSTFT e 16,7M parâmetros. A voz é sintética.
17
+
18
+ ## O que tem de novo
19
+
20
+ - Controle de emoção: `triste`, `animado`, `suave`, `raiva`, `surpresa`, além do neutro.
21
+ - Cada emoção é um vetor aprendido (`f3_istft.labels.pt`) somado ao estilo de cada frase, mais um banco de estilos por emoção (`f3_istft.bank.pt`).
22
+ - O ganho da emoção (`EMO_GAIN`) aumenta ou diminui a intensidade.
23
+
24
+ ## Arquivos
25
+
26
+ | arquivo | o que é |
27
+ |---|---|
28
+ | `f3_istft.pth` | pesos do modelo (state_dict do treino) |
29
+ | `f3_istft.kmodel.pt` | os mesmos pesos no formato do `KModel` do Kokoro |
30
+ | `config_v3_istft.json` | configuração do modelo |
31
+ | `f3_istft.labels.pt` | `{emo_<emoção>: vetor de 256}` |
32
+ | `f3_istft.bank.pt` | estilos de referência por emoção e por tipo de frase |
33
+ | `packs/*.pt` | estilos base: neutro (`pm_alex`), pergunta (`q`), `animado`, `suave` |
34
+ | `train_student.py` | script de treino com `--style-labels` |
35
+ | `samples/` | uma frase escrita para cada emoção |
36
+
37
+ ## Como usar
38
+
39
+ O estilo de uma frase é `pack[len(fonemas) - 1] + labels["emo_<emoção>"] * EMO_GAIN`, passado ao `KModel.forward_with_tokens`.
40
+
41
+ ## Treino
42
+
43
+ - 60.000 passos, lr 1e-4 com decaimento, partindo do modelo iSTFT de 16,7M (`b3_istft`).
44
+ - Áudio professor:
45
+ - [VoxCPM2](https://huggingface.co/openbmb/VoxCPM2) (Apache 2.0): 12 mil frases por emoção, filtradas com Whisper.
46
+ - Para o modelo de partida, [Pocket TTS](https://huggingface.co/kyutai/pocket-tts) da Kyutai (pesos CC-BY-4.0).
47
+ - A voz de referência vem do conjunto `marcosremar2/gemini-dataset-rasalgethi`.
48
+ - Medido no Mac: 97–100% das palavras reconhecidas pelo Whisper e 0,02–0,07 s de processamento por segundo de áudio.
49
+
50
+ ## Uso responsável
51
+
52
+ Conforme os termos do VoxCPM2, não use para se passar por uma pessoa, para fraude ou para desinformação, e marque o áudio gerado como feito por IA. Os termos da Google sobre o áudio do Gemini usado como referência ainda não foram revisados.
53
+
54
+ ## Créditos
55
+
56
+ Kokoro (hexgrad, Apache 2.0), VoxCPM2 (OpenBMB, Apache 2.0), Pocket TTS (Kyutai, CC-BY-4.0).
config_v3_istft.json ADDED
@@ -0,0 +1,170 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "istftnet": {
3
+ "upsample_kernel_sizes": [
4
+ 20,
5
+ 12
6
+ ],
7
+ "upsample_rates": [
8
+ 10,
9
+ 6
10
+ ],
11
+ "gen_istft_hop_size": 5,
12
+ "gen_istft_n_fft": 20,
13
+ "resblock_dilation_sizes": [
14
+ [
15
+ 1,
16
+ 3,
17
+ 5
18
+ ],
19
+ [
20
+ 1,
21
+ 3,
22
+ 5
23
+ ]
24
+ ],
25
+ "resblock_kernel_sizes": [
26
+ 3,
27
+ 7
28
+ ],
29
+ "upsample_initial_channel": 128,
30
+ "hidden_channels": 256,
31
+ "out_channels": 128
32
+ },
33
+ "dim_in": 64,
34
+ "dropout": 0.2,
35
+ "hidden_dim": 160,
36
+ "max_conv_dim": 512,
37
+ "max_dur": 50,
38
+ "multispeaker": true,
39
+ "n_layer": 3,
40
+ "n_mels": 80,
41
+ "n_token": 178,
42
+ "style_dim": 128,
43
+ "text_encoder_kernel_size": 5,
44
+ "plbert": {
45
+ "hidden_size": 768,
46
+ "num_attention_heads": 12,
47
+ "intermediate_size": 2048,
48
+ "max_position_embeddings": 512,
49
+ "num_hidden_layers": 12,
50
+ "dropout": 0.1
51
+ },
52
+ "vocab": {
53
+ ";": 1,
54
+ ":": 2,
55
+ ",": 3,
56
+ ".": 4,
57
+ "!": 5,
58
+ "?": 6,
59
+ "—": 9,
60
+ "…": 10,
61
+ "\"": 11,
62
+ "(": 12,
63
+ ")": 13,
64
+ "“": 14,
65
+ "”": 15,
66
+ " ": 16,
67
+ "̃": 17,
68
+ "ʣ": 18,
69
+ "ʥ": 19,
70
+ "ʦ": 20,
71
+ "ʨ": 21,
72
+ "ᵝ": 22,
73
+ "ꭧ": 23,
74
+ "A": 24,
75
+ "I": 25,
76
+ "O": 31,
77
+ "Q": 33,
78
+ "S": 35,
79
+ "T": 36,
80
+ "W": 39,
81
+ "Y": 41,
82
+ "ᵊ": 42,
83
+ "a": 43,
84
+ "b": 44,
85
+ "c": 45,
86
+ "d": 46,
87
+ "e": 47,
88
+ "f": 48,
89
+ "h": 50,
90
+ "i": 51,
91
+ "j": 52,
92
+ "k": 53,
93
+ "l": 54,
94
+ "m": 55,
95
+ "n": 56,
96
+ "o": 57,
97
+ "p": 58,
98
+ "q": 59,
99
+ "r": 60,
100
+ "s": 61,
101
+ "t": 62,
102
+ "u": 63,
103
+ "v": 64,
104
+ "w": 65,
105
+ "x": 66,
106
+ "y": 67,
107
+ "z": 68,
108
+ "ɑ": 69,
109
+ "ɐ": 70,
110
+ "ɒ": 71,
111
+ "æ": 72,
112
+ "β": 75,
113
+ "ɔ": 76,
114
+ "ɕ": 77,
115
+ "ç": 78,
116
+ "ɖ": 80,
117
+ "ð": 81,
118
+ "ʤ": 82,
119
+ "ə": 83,
120
+ "ɚ": 85,
121
+ "ɛ": 86,
122
+ "ɜ": 87,
123
+ "ɟ": 90,
124
+ "ɡ": 92,
125
+ "ɥ": 99,
126
+ "ɨ": 101,
127
+ "ɪ": 102,
128
+ "ʝ": 103,
129
+ "ɯ": 110,
130
+ "ɰ": 111,
131
+ "ŋ": 112,
132
+ "ɳ": 113,
133
+ "ɲ": 114,
134
+ "ɴ": 115,
135
+ "ø": 116,
136
+ "ɸ": 118,
137
+ "θ": 119,
138
+ "œ": 120,
139
+ "ɹ": 123,
140
+ "ɾ": 125,
141
+ "ɻ": 126,
142
+ "ʁ": 128,
143
+ "ɽ": 129,
144
+ "ʂ": 130,
145
+ "ʃ": 131,
146
+ "ʈ": 132,
147
+ "ʧ": 133,
148
+ "ʊ": 135,
149
+ "ʋ": 136,
150
+ "ʌ": 138,
151
+ "ɣ": 139,
152
+ "ɤ": 140,
153
+ "χ": 142,
154
+ "ʎ": 143,
155
+ "ʒ": 147,
156
+ "ʔ": 148,
157
+ "ˈ": 156,
158
+ "ˌ": 157,
159
+ "ː": 158,
160
+ "ʰ": 162,
161
+ "ʲ": 164,
162
+ "↓": 169,
163
+ "→": 171,
164
+ "↗": 172,
165
+ "↘": 173,
166
+ "ᵻ": 177
167
+ },
168
+ "predictor_hidden": 256,
169
+ "decoder_dim_in": 512
170
+ }
f3_istft.bank.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5bad31f4a604733aa882825ab70d520d2eadccdbbfb3cf57aee493c2e453dbbb
3
+ size 2361941
f3_istft.kmodel.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1e05784d4e1d806dd5d2887fa92979272718c9d7786af404fe6251498a081286
3
+ size 67162501
f3_istft.labels.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:dc3c164880e66d2b484b31307d8e1ccdac1221beedc6dbf0f9ab773bd46f64f3
3
+ size 11248
f3_istft.pth ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed90ffd57d3ba33b57ad95def99231abf45463c9c9820aa4de72a2bd64dd1abe
3
+ size 67169492
packs/animado.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6f1377ee94f003dda2060b6f58cc246aab2395c1dd704aa596d8bf5d711e015d
3
+ size 523817
packs/pm_alex.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:81cfa698e58565157952599f21db1df2ab2497fd5dad8c964c0d8dc4d0301cf0
3
+ size 523817
packs/q.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9e34fa511674f84ce3c677b5a9279350889a7eaf7be5b282c40adbd6c37432ff
3
+ size 523583
packs/suave.pt ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:598904a1be8a81f7ccd8291254359f168e7c503e9383c2f44d5b93a9e5cadfd1
3
+ size 523739
samples/f3_istft.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:9723cf76ca1682cc8f5771bec7dce8c225eefe3fbd7bbdee16c10372347c9b56
3
+ size 277244
samples/f3_istft_emo_animado.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:768a9ec6c1411955c012b0c8c5d3d1afc508a17ad741c42148ac2e1972d14296
3
+ size 219644
samples/f3_istft_emo_raiva.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8e7407aef6c9cd43709775c069393c0b28b1e3a8ca025fecb9a0e43fded9775a
3
+ size 213644
samples/f3_istft_emo_suave.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e056036bf5aab1e8cc2170bb368d2410afb3e879821c2d8f8a280f0acdadcc90
3
+ size 361244
samples/f3_istft_emo_surpresa.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f33477006400cd1ca05677da8b16b88570f1b7a67039054595bc805a1c596277
3
+ size 328844
samples/f3_istft_emo_triste.wav ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8fb78256298f1f27622c6c0bb155e57172c35e27b5c4467ce29606f3fbdd0930
3
+ size 313244
samples/frases.json ADDED
@@ -0,0 +1,8 @@
 
 
 
 
 
 
 
 
 
1
+ [
2
+ "Bom dia. A reunião começa às nove horas, na sala do segundo andar.",
3
+ "Ele foi embora ontem e nem se despediu. A casa ficou tão vazia sem ele.",
4
+ "A gente ganhou! Eu não acredito, foi o melhor jogo da minha vida!",
5
+ "Fica tranquila, está tudo bem. Respira devagar, eu estou aqui com você.",
6
+ "Eu já falei mil vezes para não mexer nas minhas coisas! Chega!",
7
+ "O quê? Você vai casar no mês que vem? Desde quando vocês estão juntos?"
8
+ ]
train_student.py ADDED
@@ -0,0 +1,609 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Stage 1: distill Nabra-82M into the 5.25M Kokoro student.
2
+
3
+ The student is trained on the TEACHER's audio using the TEACHER's own durations, so the
4
+ alignment is correct by construction. That is the whole reason for distilling: kitten-arabic
5
+ established that with mismatched durations this architecture tops out around WER 0.85 and
6
+ 62,000 further steps change nothing, because free-running inference beats teacher-forcing --
7
+ the durations, not the capacity, were the ceiling.
8
+
9
+ Adversarial loss is NOT optional. The same project measured reconstruction loss falling from
10
+ 6.445 to 2.996 while the audio degraded to pure static: multi-resolution STFT and mel distance
11
+ constrain magnitude but not phase, so the optimizer can win on the objective and lose the
12
+ waveform. MPD + MSD from StyleTTS2 are what make the objective mean something.
13
+
14
+ Block names mirror KModel (bert/bert_encoder/predictor/text_encoder/decoder) so a finished
15
+ student loads straight into KPipeline with the teacher's G2P and voicepack.
16
+ """
17
+ from __future__ import annotations
18
+ import _env # noqa: F401
19
+ import argparse, json, math, os, random, time
20
+ from pathlib import Path
21
+
22
+ import numpy as np, torch, torch.nn as nn, torch.nn.functional as F
23
+ from torch.utils.data import Dataset, DataLoader
24
+ from transformers import AlbertConfig
25
+
26
+ from kokoro.istftnet import Decoder
27
+ from kokoro.modules import ProsodyPredictor, TextEncoder, CustomAlbert
28
+ from style_encoder import StyleEncoder
29
+ from styletts2_gan.discriminators import (MultiPeriodDiscriminator,
30
+ MultiResSpecDiscriminator, WavLMDiscriminator)
31
+ from styletts2_gan.losses_styletts2 import (GeneratorLoss, DiscriminatorLoss,
32
+ MultiResolutionSTFTLoss, WavLMLoss)
33
+
34
+
35
+ class MelL1(nn.Module):
36
+ """L1 on log-mel. The existing multi-resolution STFT loss is spectral convergence on
37
+ LINEAR-frequency bins, so every bin counts equally in Hz; mel spacing instead weights by
38
+ where hearing resolves detail.
39
+
40
+ Measured motivation: against the teacher the student runs +0.025 energy in 0-1 kHz and
41
+ -0.025 in 1-4 kHz -- the consonant/F2-F3 band -- with a 12.28 dB mel L1 gap, while
42
+ intelligibility is already fine (WER 0.064, median 0.000). So the deficit is timbre in a
43
+ specific band, which is exactly what a mel-domain term penalises and a linear-bin one
44
+ under-weights."""
45
+
46
+ def __init__(self, sr=24000, n_fft=1024, hop=256, n_mels=80):
47
+ super().__init__()
48
+ import torchaudio
49
+ self.mel = torchaudio.transforms.MelSpectrogram(
50
+ sample_rate=sr, n_fft=n_fft, hop_length=hop, n_mels=n_mels, power=1.0)
51
+
52
+ def forward(self, y_hat, y):
53
+ a = torch.log(self.mel(y_hat) + 1e-5)
54
+ b = torch.log(self.mel(y) + 1e-5)
55
+ return F.l1_loss(a, b)
56
+
57
+ R1R2_EVERY = 8
58
+ SPF = 600 # samples per alignment frame, MEASURED in dump_teacher.py; --sr rewrites it
59
+ AUDIO_SR = 0 # set from --audio-sr when the files differ from the model's rate
60
+ SPACE = 16
61
+ FRAME_HZ = 40 # the alignment grid, held constant across sample rates so a duration means
62
+ # the same thing whether the decoder emits 600 or 400 samples per frame
63
+ NB = "/home/ahmed-wasfy/Documents/projects/kikiri-tts/training/nabra"
64
+
65
+
66
+ def build_student(cfg_path):
67
+ c = json.load(open(cfg_path))
68
+ m = nn.Module()
69
+ m.bert = CustomAlbert(AlbertConfig(vocab_size=c["n_token"], **c["plbert"]))
70
+ predictor_hidden = c.get("predictor_hidden", c["hidden_dim"])
71
+ m.bert_encoder = nn.Linear(m.bert.config.hidden_size, predictor_hidden)
72
+ m.predictor = ProsodyPredictor(style_dim=c["style_dim"], d_hid=predictor_hidden,
73
+ nlayers=c["n_layer"], max_dur=c["max_dur"],
74
+ dropout=c["dropout"])
75
+ m.text_encoder = TextEncoder(channels=c["hidden_dim"],
76
+ kernel_size=c["text_encoder_kernel_size"],
77
+ depth=c["n_layer"], n_symbols=c["n_token"])
78
+ decoder_dim_in = c.get("decoder_dim_in", c["hidden_dim"])
79
+ if "decoder_dim_in" in c:
80
+ m.feature_proj = nn.Sequential(nn.Conv1d(c["hidden_dim"], decoder_dim_in, 1), nn.SiLU(),
81
+ nn.Conv1d(decoder_dim_in, decoder_dim_in, 1))
82
+ m.decoder = Decoder(dim_in=decoder_dim_in, style_dim=c["style_dim"],
83
+ dim_out=c["n_mels"], disable_complex=True, **c["istftnet"])
84
+ return m, c
85
+
86
+
87
+ class DistDataset(Dataset):
88
+ """Teacher audio from the packed i16 memmap, or -- when a row carries "wav" -- Jenny's real
89
+ recording with the DTW-transferred durations from align_real.py.
90
+
91
+ For the real branch the clip is cut to sum(dur)*SPF samples, because that is exactly the
92
+ span the durations describe; anything past it has no label."""
93
+
94
+ def __init__(self, rows, bin_path, crop):
95
+ self.rows, self.bin_path, self.crop = rows, bin_path, crop
96
+ self._mm = {}
97
+
98
+ def _mem(self, path=None):
99
+ # a row may name its own bin file, so a bilingual index can span two teacher dumps
100
+ path = path or self.bin_path
101
+ if path not in self._mm:
102
+ self._mm[path] = np.memmap(path, dtype=np.int16, mode="r")
103
+ return self._mm[path]
104
+
105
+ def __len__(self):
106
+ return len(self.rows)
107
+
108
+ def __getitem__(self, i):
109
+ r = self.rows[i]
110
+ if "wav" in r:
111
+ import soundfile as sf
112
+ # "start" lets one long recording supply many training segments without being cut
113
+ # into files on disk; v9 clips run to 123 s and are segmented at aligner-found pauses.
114
+ need = int(sum(r["dur"])) * SPF
115
+ off = int(r.get("start", 0)) * SPF
116
+ if AUDIO_SR and AUDIO_SR != SPF * FRAME_HZ:
117
+ import soxr
118
+ nspf = AUDIO_SR // FRAME_HZ # frames are 40 Hz at either rate
119
+ a, _ = sf.read(r["wav"], dtype="float32",
120
+ start=int(r.get("start", 0)) * nspf,
121
+ frames=int(sum(r["dur"])) * nspf)
122
+ if a.ndim > 1:
123
+ a = a.mean(axis=1)
124
+ a = soxr.resample(a, AUDIO_SR, SPF * FRAME_HZ).astype(np.float32)
125
+ else:
126
+ a, _ = sf.read(r["wav"], dtype="float32", start=off, frames=need)
127
+ if a.ndim > 1:
128
+ a = a.mean(axis=1)
129
+ if len(a) < need:
130
+ a = np.pad(a, (0, need - len(a)))
131
+ a = a[:need]
132
+ elif "offset" not in r:
133
+ a = np.zeros(0, dtype=np.float32)
134
+ else:
135
+ a = np.asarray(self._mem(r.get("bin"))[r["offset"]: r["offset"] + r["samples"]],
136
+ dtype=np.float32) / 32767.0
137
+ return {"ids": r["ids"], "dur": r["dur"], "audio": a, "lang": r.get("lang", "en")}
138
+
139
+
140
+ def collate(batch):
141
+ B = len(batch)
142
+ L = max(len(b["ids"]) for b in batch)
143
+ ids = torch.zeros(B, L, dtype=torch.long)
144
+ dur = torch.zeros(B, L, dtype=torch.float)
145
+ lens = torch.tensor([len(b["ids"]) for b in batch], dtype=torch.long)
146
+ for i, b in enumerate(batch):
147
+ ids[i, : len(b["ids"])] = torch.tensor(b["ids"])
148
+ dur[i, : len(b["dur"])] = torch.tensor(b["dur"], dtype=torch.float)
149
+ return {"ids": ids, "dur": dur, "lens": lens,
150
+ "lang": [b.get("lang", "en") for b in batch],
151
+ "audio": [torch.from_numpy(b["audio"]) for b in batch]}
152
+
153
+
154
+ def build_aln(dur, lens, F_max, device):
155
+ """[B,L] integer durations -> [B,L,F] hard monotonic alignment."""
156
+ B, L = dur.shape
157
+ aln = torch.zeros(B, L, F_max, device=device)
158
+ for b in range(B):
159
+ pos = 0
160
+ for j in range(int(lens[b])):
161
+ d = int(dur[b, j])
162
+ if d <= 0 or pos >= F_max:
163
+ continue
164
+ aln[b, j, pos: min(pos + d, F_max)] = 1.0
165
+ pos += d
166
+ return aln
167
+
168
+
169
+ def main():
170
+ ap = argparse.ArgumentParser()
171
+ ap.add_argument("--steps", type=int, default=60000)
172
+ ap.add_argument("--batch", type=int, default=16)
173
+ ap.add_argument("--crop", type=int, default=40) # frames; 40 * 600 = 1.0 s
174
+ ap.add_argument("--lr", type=float, default=2e-4)
175
+ ap.add_argument("--lr-disc", type=float, default=2e-4)
176
+ ap.add_argument("--lr-decay", action="store_true",
177
+ help="cosine-decay the generator LR to 10%% of peak. The 15.34M student "
178
+ "diverged at a CONSTANT 2e-4: its duration L1 stepped from 0.33 to 0.76 "
179
+ "between 20.5k and 21k and never recovered, leaving predictions "
180
+ "uncorrelated with the true durations (r=+0.019 against the 7.48M's "
181
+ "+0.381). The acoustic path kept improving throughout, so nothing in the "
182
+ "headline losses showed it.")
183
+ ap.add_argument("--dur-weight", type=float, default=5.0)
184
+ ap.add_argument("--stft-weight", type=float, default=5.0)
185
+ ap.add_argument("--disc-warmup", type=int, default=500)
186
+ ap.add_argument("--slm-weight", type=float, default=0.0,
187
+ help="WavLM feature-matching loss. Every other term here is MAGNITUDE-domain "
188
+ "(log-STFT, mel, silence RMS), so nothing but the GAN constrains phase "
189
+ "-- and our GAN is MPD (waveform) plus MultiResSpec (magnitude, "
190
+ "phase-blind). Measured: the student matches the teacher on band "
191
+ "energy, F0 variation, harmonic ratio, modulation spectrum and "
192
+ "frame-rate buzz, yet sounds robotic. SLM losses are what StyleTTS2 "
193
+ "credits for naturalness, and this repo vendored them unused.")
194
+ ap.add_argument("--slm-adv-weight", type=float, default=0.0,
195
+ help="SLM adversarial term on top of feature matching")
196
+ ap.add_argument("--mel-weight", type=float, default=0.0,
197
+ help="weight on the log-mel L1 term (see MelL1)")
198
+ ap.add_argument("--sil-weight", type=float, default=0.0,
199
+ help="penalise output energy where the TEACHER is silent. Kokoro-82M emits "
200
+ "true digital silence in pauses (frame RMS 2e-5); the 5.42M student "
201
+ "floors at 5.4e-3, 270x higher, which is audible as a constant hiss. "
202
+ "Multi-resolution STFT is computed on log-magnitude, where -45 dB vs "
203
+ "-90 dB is a small loss difference and a large perceptual one, so "
204
+ "nothing in the existing objective punishes it.")
205
+ ap.add_argument("--sil-thresh", type=float, default=1e-3,
206
+ help="frame RMS below which the teacher counts as silent")
207
+ ap.add_argument("--init-from", default=None,
208
+ help="load weights from a finished run and start a fresh schedule; --resume "
209
+ "cannot do this because it restores a scheduler that is already done")
210
+ ap.add_argument("--r1r2-weight", type=float, default=0.0)
211
+ ap.add_argument("--index", default="dist/index.jsonl")
212
+ ap.add_argument("--bin", default="dist/audio.i16.bin")
213
+ ap.add_argument("--config", default="config_student.json")
214
+ ap.add_argument("--out", default="runs/distill")
215
+ ap.add_argument("--holdout", type=int, default=200)
216
+ ap.add_argument("--eval-every", type=int, default=1000)
217
+ ap.add_argument("--ckpt-every", type=int, default=2000)
218
+ ap.add_argument("--sr", type=int, default=24000,
219
+ help="output sample rate. 16000 for corpora that are natively 16 kHz: "
220
+ "upsampling them to 24 kHz invents nothing above 8 kHz and misreports "
221
+ "the bandwidth. Sets samples-per-frame to sr/40, and the mel, silence "
222
+ "and WavLM losses with it.")
223
+ ap.add_argument("--audio-sr", type=int, default=0,
224
+ help="the wav files' own rate, when it differs from --sr. The audio is then "
225
+ "resampled on read. Durations are in 40 Hz FRAMES, which mean the same "
226
+ "thing at any rate, so an alignment built at one rate is valid at the "
227
+ "other and does not need redoing.")
228
+ ap.add_argument("--init-loose", action="store_true",
229
+ help="load only the tensors whose shapes match. Needed when the decoder is "
230
+ "rebuilt for another sample rate: the 16 kHz upsample stack differs from "
231
+ "the 24 kHz one in exactly two tensors, and the other 99.6%% transfers.")
232
+ ap.add_argument("--bilingual", action="store_true",
233
+ help="rows carry a 'lang' field; condition English rows on af_heart and "
234
+ "Arabic rows on af_msa so one student learns both, with language "
235
+ "selectable at inference by choosing the pack")
236
+ ap.add_argument("--mode", choices=["joint", "text", "decoder"], default="joint")
237
+ ap.add_argument("--teacher", default=None)
238
+ ap.add_argument("--feat-weight", type=float, default=5.0)
239
+ ap.add_argument("--prefix-prob", type=float, default=0.0)
240
+ ap.add_argument("--lookahead-words", type=int, default=2)
241
+ ap.add_argument("--resume", action="store_true")
242
+ ap.add_argument("--style-lr", type=float, default=0.0)
243
+ ap.add_argument("--style-labels", default="")
244
+ ap.add_argument("--probe", action="store_true")
245
+ a = ap.parse_args()
246
+
247
+ dev = "cuda" if torch.cuda.is_available() else "cpu"
248
+ torch.manual_seed(42)
249
+ global SPF, AUDIO_SR
250
+ SPF = a.sr // FRAME_HZ
251
+ AUDIO_SR = a.audio_sr
252
+ if AUDIO_SR and AUDIO_SR != a.sr:
253
+ print(f"[*] reading {AUDIO_SR} Hz audio, resampling to {a.sr} Hz on the fly", flush=True)
254
+ if a.sr != 24000:
255
+ print(f"[*] {a.sr} Hz: {SPF} samples per frame at {FRAME_HZ} Hz", flush=True)
256
+ out = Path(a.out); out.mkdir(parents=True, exist_ok=True)
257
+
258
+ rows = [json.loads(l) for l in open(a.index, encoding="utf-8")]
259
+ rows = [r for r in rows if sum(r["dur"]) >= a.crop + 2]
260
+ random.Random(1234).shuffle(rows)
261
+ train_rows, eval_rows = rows[: -a.holdout], rows[-a.holdout:]
262
+ print(f"[*] train {len(train_rows):,} | held out {len(eval_rows):,}", flush=True)
263
+
264
+ model, cfg = build_student(a.config)
265
+ model.to(dev)
266
+ n = sum(p.numel() for p in model.parameters())
267
+ print(f"[*] student {n/1e6:.2f}M", flush=True)
268
+
269
+ pack = torch.load(os.environ["STUDENT_PACK"], map_location="cpu", weights_only=True)
270
+ PACKS = {"ar": pack}
271
+ for name, path in json.loads(os.environ.get("STUDENT_PACK_MAP", "{}")).items():
272
+ PACKS[name] = torch.load(path, map_location="cpu", weights_only=True)
273
+ if a.bilingual:
274
+ import _env # noqa: F401
275
+ from kokoro import KPipeline
276
+ PACKS["en"] = KPipeline(lang_code="a", repo_id="hexgrad/Kokoro-82M",
277
+ model=False).load_voice("af_heart").cpu()
278
+ if os.path.exists("pack_cs.pt"):
279
+ # the code-switch corpus is a THIRD voice (239.5 Hz vs 292.6 Arabic / 197.5 English),
280
+ # so it gets its own style vector instead of overwriting one of theirs
281
+ PACKS["cs"] = torch.load("pack_cs.pt", map_location="cpu", weights_only=True)
282
+ print(f"[*] bilingual packs: {sorted(PACKS)}", flush=True)
283
+ teacher = None
284
+ if a.mode != "joint":
285
+ from kokoro import KModel
286
+ teacher = KModel(repo_id="hexgrad/Kokoro-82M", config=a.teacher + "/config.json", model=a.teacher + "/kokoro.pth").to(dev).eval()
287
+ teacher.decoder = None
288
+ mpd, msd = MultiPeriodDiscriminator().to(dev), MultiResSpecDiscriminator().to(dev)
289
+ gen_loss = GeneratorLoss(mpd, msd).to(dev)
290
+ disc_loss = DiscriminatorLoss(mpd, msd).to(dev)
291
+ stft_loss = MultiResolutionSTFTLoss().to(dev)
292
+ mel_loss = MelL1(sr=a.sr).to(dev)
293
+ wl = wd = opt_wd = None
294
+ if a.slm_weight > 0 or a.slm_adv_weight > 0:
295
+ # WavLM-base-plus emits 13 hidden states x 768 dims; the discriminator's first conv takes
296
+ # slm_hidden * slm_layers = 9984 channels. Passing (1024, 512, 256) builds a 524288-channel
297
+ # conv and fails on the first real batch.
298
+ wd = WavLMDiscriminator(slm_hidden=768, slm_layers=13, initial_channel=64).to(dev)
299
+ wl = WavLMLoss("microsoft/wavlm-base-plus", wd, a.sr, 16000).to(dev)
300
+ for p_ in wl.wavlm.parameters():
301
+ p_.requires_grad = False
302
+ wl.wavlm.eval()
303
+ opt_wd = torch.optim.AdamW(wd.parameters(), lr=a.lr_disc, betas=(0.8, 0.99))
304
+ print(f"[*] SLM: WavLM feature matching (w={a.slm_weight}) "
305
+ f"adversarial (w={a.slm_adv_weight})", flush=True)
306
+
307
+ opt_g = torch.optim.AdamW((model.decoder if a.mode == "decoder" else model).parameters(), lr=a.lr, betas=(0.8, 0.99), weight_decay=1e-2)
308
+ styler = StyleEncoder(sr=a.sr).to(dev) if a.style_lr > 0 and a.mode == "joint" else None
309
+ if styler is not None:
310
+ opt_g.add_param_group({"params": styler.parameters(), "lr": a.style_lr})
311
+ labels = [label for label in a.style_labels.split(",") if label]
312
+ label_style = torch.nn.Embedding(len(labels) + 1, 256, padding_idx=0).to(dev) if labels and a.mode == "joint" else None
313
+ if label_style is not None:
314
+ torch.nn.init.zeros_(label_style.weight)
315
+ opt_g.add_param_group({"params": label_style.parameters(), "lr": a.style_lr or a.lr})
316
+ sched_g = (torch.optim.lr_scheduler.CosineAnnealingLR(opt_g, T_max=a.steps, eta_min=a.lr * 0.1)
317
+ if a.lr_decay else None)
318
+ opt_d = torch.optim.AdamW(list(mpd.parameters()) + list(msd.parameters()),
319
+ lr=a.lr_disc, betas=(0.8, 0.99), weight_decay=1e-2)
320
+
321
+ dl = DataLoader(DistDataset(train_rows, a.bin, a.crop), batch_size=a.batch, shuffle=True,
322
+ collate_fn=collate, num_workers=int(os.environ.get("WORKERS", 4)), drop_last=True, pin_memory=True)
323
+
324
+ def style_rows(lens, langs):
325
+ return torch.cat([PACKS.get(lg, pack)[max(min(int(l) - 3, pack.shape[0] - 1), 0)]
326
+ for l, lg in zip(lens, langs)]).to(dev)
327
+
328
+ def prefix_cuts(ids, lens):
329
+ cuts, keeps = lens.clone(), lens.clone()
330
+ for i in range(ids.shape[0]):
331
+ spaces = (ids[i, :lens[i]] == SPACE).nonzero().flatten().tolist()
332
+ if random.random() >= a.prefix_prob or len(spaces) <= a.lookahead_words + 1:
333
+ continue
334
+ k = random.randint(a.lookahead_words + 1, len(spaces) - 1)
335
+ cuts[i], keeps[i] = spaces[k], spaces[k - a.lookahead_words]
336
+ return cuts, keeps
337
+
338
+ def prepare(b):
339
+ ids = b["ids"].to(dev); lens = b["lens"].to(dev); durt = b["dur"].to(dev)
340
+ B, L = ids.shape
341
+ tmask = torch.arange(L, device=dev)[None].expand(B, -1) + 1 > lens[:, None]
342
+ langs = b.get("lang", ["ar"] * B)
343
+ ref = style_rows(lens, langs)
344
+ if styler is not None:
345
+ ref = ref + styler(b["audio"])
346
+ if label_style is not None:
347
+ ref = ref + label_style(torch.tensor([labels.index(lg) + 1 if lg in labels else 0 for lg in langs], device=dev))
348
+ frames = durt.sum(dim=1).long()
349
+ aln = build_aln(durt, lens, int(frames.max()), dev)
350
+ return ids, lens, durt, tmask, ref[:, 128:], ref[:, :128], frames, aln
351
+
352
+ def text_side(net, ids, lens, tmask, s_pred, aln):
353
+ d_en = net.bert_encoder(net.bert(ids, attention_mask=(~tmask).int())).transpose(-1, -2)
354
+ d = net.predictor.text_encoder(d_en, s_pred, lens, tmask)
355
+ F0, N = net.predictor.F0Ntrain(d.transpose(-1, -2) @ aln, s_pred)
356
+ feats = net.text_encoder(ids, lens, tmask)
357
+ if hasattr(net, "feature_proj"):
358
+ feats = net.feature_proj(feats)
359
+ return d, F0, N, feats
360
+
361
+ def duration_loss(d, durt, ignored):
362
+ x, _ = model.predictor.lstm(d)
363
+ dur_pred = torch.sigmoid(model.predictor.duration_proj(x)).sum(dim=-1)
364
+ return (F.l1_loss(dur_pred, durt, reduction="none") * (~ignored)).sum() / (~ignored).sum()
365
+
366
+ def text_losses(b):
367
+ ids, lens, durt, tmask, s_pred, _, frames, aln = prepare(b)
368
+ with torch.no_grad():
369
+ _, t_F0, t_N, t_feats = text_side(teacher, ids, lens, tmask, s_pred, aln)
370
+ cuts, keeps = prefix_cuts(ids, lens)
371
+ L = int(cuts.max())
372
+ positions = torch.arange(L, device=dev)[None]
373
+ inside, scored = positions < cuts[:, None], positions < keeps[:, None]
374
+ ids, durt = ids[:, :L] * inside, durt[:, :L] * inside
375
+ T = int(durt.sum(1).max())
376
+ aln = aln[:, :L, :T] * inside[:, :, None]
377
+ d, F0, N, feats = text_side(model, ids, cuts, ~inside, style_rows(cuts, b.get("lang", ["ar"] * len(cuts)))[:, 128:], aln)
378
+ scored_frames = (torch.arange(2 * T, device=dev)[None] < 2 * (durt * scored).sum(1)[:, None]).float()
379
+ masked = lambda loss, mask: (loss * mask).sum() / mask.sum()
380
+ l_f0 = masked(F.smooth_l1_loss(F0, t_F0[:, :2 * T], reduction="none"), scored_frames) / 10
381
+ l_n = masked(F.smooth_l1_loss(N, t_N[:, :2 * T], reduction="none"), scored_frames)
382
+ l_feat = masked(F.l1_loss(feats, t_feats[:, :, :L], reduction="none").mean(1), scored.float())
383
+ return duration_loss(d, durt, ~scored), l_f0, l_n, l_feat
384
+
385
+ def run_batch(b, probe=False):
386
+ ids, lens, durt, tmask, s_pred, s_dec, frames, aln = prepare(b)
387
+ B = ids.shape[0]
388
+ if a.mode == "decoder":
389
+ with torch.no_grad():
390
+ _, F0, N, feats = text_side(teacher, ids, lens, tmask, s_pred, aln)
391
+ l_dur = torch.zeros((), device=dev)
392
+ else:
393
+ d, F0, N, feats = text_side(model, ids, lens, tmask, s_pred, aln)
394
+ l_dur = duration_loss(d, durt, tmask)
395
+ asr = feats @ aln
396
+ up = F0.shape[-1] / asr.shape[-1]
397
+ if probe:
398
+ print(f" asr {tuple(asr.shape)} F0 {tuple(F0.shape)} up={up:.2f} "
399
+ f"frames={frames.tolist()[:4]}", flush=True)
400
+ assert abs(up - round(up)) < 1e-6, f"F0/asr ratio not integral: {up}"
401
+ up = round(up)
402
+
403
+ # Random crop: the decoder is convolutional, so a window is enough and lets the batch
404
+ # be rectangular. Audio, asr and F0/N are cropped on the SAME frame window.
405
+ C = a.crop
406
+ asr_c = torch.empty(B, asr.shape[1], C, device=dev)
407
+ F0_c = torch.empty(B, C * up, device=dev); N_c = torch.empty(B, C * up, device=dev)
408
+ y = torch.empty(B, 1, C * SPF, device=dev)
409
+ for i in range(B):
410
+ hi = max(int(frames[i]) - C, 0)
411
+ f0 = random.randint(0, hi)
412
+ asr_c[i] = asr[i, :, f0:f0 + C]
413
+ F0_c[i] = F0[i, f0 * up:(f0 + C) * up]
414
+ N_c[i] = N[i, f0 * up:(f0 + C) * up]
415
+ w = b["audio"][i]
416
+ seg = w[f0 * SPF:(f0 + C) * SPF]
417
+ if seg.shape[0] < C * SPF:
418
+ seg = F.pad(seg, (0, C * SPF - seg.shape[0]))
419
+ y[i, 0] = seg.to(dev)
420
+ y_hat = model.decoder(asr_c, F0_c, N_c, s_dec)
421
+ if y_hat.dim() == 2:
422
+ y_hat = y_hat.unsqueeze(1)
423
+ y_hat = y_hat[..., : y.shape[-1]]
424
+ if probe:
425
+ print(f" y {tuple(y.shape)} y_hat {tuple(y_hat.shape)}", flush=True)
426
+ return y, y_hat, l_dur
427
+
428
+ if a.probe and a.mode != "text":
429
+ b = next(iter(dl))
430
+ y, y_hat, l_dur = run_batch(b, probe=True)
431
+ print(f"[+] probe OK dur_L1 {l_dur.item():.3f} "
432
+ f"stft {stft_loss(y_hat.squeeze(1), y.squeeze(1)).item():.3f}", flush=True)
433
+ return
434
+
435
+ step, best = 0, math.inf
436
+ if a.init_from:
437
+ c = torch.load(a.init_from, map_location="cpu", weights_only=False)
438
+ src = c["model"] if "model" in c else c
439
+ # KModel checkpoints are stored NESTED as {block: {name: tensor}}; the trainer's own
440
+ # last.pt is flat. Flatten so --init-from accepts either, instead of silently matching
441
+ # nothing and training from scratch.
442
+ if src and all(isinstance(v, dict) for v in src.values()):
443
+ src = {f"{b}.{k}": t for b, d in src.items() for k, t in d.items()}
444
+ # Checkpoints saved under DataParallel/DDP carry a "module." prefix inside each block
445
+ # (Nabra-82M does). Left in place it matches nothing and the model trains from scratch.
446
+ if any(".module." in k for k in src):
447
+ src = {k.replace(".module.", ".", 1): v for k, v in src.items()}
448
+ if a.init_loose:
449
+ tgt = model.state_dict()
450
+ fit = {k: v for k, v in src.items() if k in tgt and tgt[k].shape == v.shape}
451
+ skipped = [k for k in tgt if k not in fit]
452
+ model.load_state_dict(fit, strict=False)
453
+ n_fit = sum(tgt[k].numel() for k in fit)
454
+ n_all = sum(v.numel() for v in tgt.values())
455
+ print(f"[*] loose init: {n_fit/1e6:.2f}M of {n_all/1e6:.2f}M "
456
+ f"({100*n_fit/n_all:.1f}%) loaded, {len(skipped)} tensors left random",
457
+ flush=True)
458
+ for k in skipped:
459
+ print(f" random: {k}", flush=True)
460
+ else:
461
+ model.load_state_dict(src)
462
+ if "mpd" in c: mpd.load_state_dict(c["mpd"]); msd.load_state_dict(c["msd"])
463
+ print(f"[*] init weights (+discriminators) from {a.init_from}", flush=True)
464
+ ck = out / "last.pt"
465
+ if a.resume and ck.exists():
466
+ c = torch.load(ck, map_location="cpu", weights_only=False)
467
+ model.load_state_dict(c["model"]); mpd.load_state_dict(c["mpd"])
468
+ msd.load_state_dict(c["msd"]); opt_g.load_state_dict(c["opt_g"])
469
+ opt_d.load_state_dict(c["opt_d"]); step = c["step"]
470
+ if styler is not None and "styler" in c:
471
+ styler.load_state_dict(c["styler"])
472
+ if label_style is not None and "label_style" in c:
473
+ label_style.load_state_dict(c["label_style"])
474
+ if sched_g is not None:
475
+ if "sched_g" in c:
476
+ sched_g.load_state_dict(c["sched_g"])
477
+ else: # checkpoint predates the scheduler
478
+ for _ in range(step):
479
+ sched_g.step()
480
+ print(f"[*] resumed at step {step}", flush=True)
481
+
482
+ if a.mode == "text":
483
+ model.train()
484
+ step, acc, t0 = 0, {}, time.time()
485
+ while step < a.steps:
486
+ for b in dl:
487
+ if step >= a.steps:
488
+ break
489
+ l_dur, l_f0, l_n, l_feat = text_losses(b)
490
+ opt_g.zero_grad(set_to_none=True)
491
+ (a.dur_weight * l_dur + l_f0 + l_n + a.feat_weight * l_feat).backward()
492
+ torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
493
+ opt_g.step()
494
+ if sched_g is not None:
495
+ sched_g.step()
496
+ step += 1
497
+ for k, v in (("dur", l_dur), ("f0", l_f0), ("n", l_n), ("feat", l_feat)):
498
+ acc[k] = acc.get(k, 0.0) + float(v)
499
+ if step % 100 == 0 or a.probe:
500
+ print(f" step {step}/{a.steps} " + " ".join(f"{k} {v/(1 if a.probe else 100):.4f}" for k, v in acc.items()) +
501
+ f" {step * a.batch / (time.time() - t0):.0f} clip/s", flush=True)
502
+ acc = {}
503
+ if a.probe and step >= 3:
504
+ return
505
+ if step % a.ckpt_every == 0 or step == a.steps:
506
+ torch.save(model.state_dict(), out / "student.pth")
507
+ print(f"[+] text side done at step {step}", flush=True)
508
+ return
509
+
510
+ t0 = time.time(); acc = {}
511
+ model.train()
512
+ while step < a.steps:
513
+ for b in dl:
514
+ if step >= a.steps:
515
+ break
516
+ y, y_hat, l_dur = run_batch(b)
517
+
518
+ opt_d.zero_grad(set_to_none=True)
519
+ d_l = disc_loss(y, y_hat.detach())
520
+ l_r1r2 = torch.zeros((), device=dev)
521
+ if a.r1r2_weight > 0 and step % R1R2_EVERY == 0:
522
+ real_wave = y.detach().requires_grad_(True)
523
+ fake_wave = y_hat.detach().requires_grad_(True)
524
+ for disc in (mpd, msd):
525
+ real_out, fake_out = disc(real_wave, fake_wave)[:2]
526
+ grads = torch.autograd.grad(sum(o.mean() for o in real_out + fake_out), [real_wave, fake_wave], create_graph=True)
527
+ l_r1r2 = l_r1r2 + sum(g.pow(2).sum(dim=(1, 2)).mean() for g in grads)
528
+ d_l = d_l + R1R2_EVERY * a.r1r2_weight / 2 * l_r1r2
529
+ d_l.backward(); torch.nn.utils.clip_grad_norm_(
530
+ list(mpd.parameters()) + list(msd.parameters()), 10.0)
531
+ opt_d.step()
532
+
533
+ opt_g.zero_grad(set_to_none=True)
534
+ l_stft = stft_loss(y_hat.squeeze(1), y.squeeze(1))
535
+ l_mel = (mel_loss(y_hat.squeeze(1), y.squeeze(1)) if a.mel_weight > 0
536
+ else torch.zeros((), device=dev))
537
+ l_slm = torch.zeros((), device=dev)
538
+ if wl is not None:
539
+ if a.slm_weight > 0:
540
+ l_slm = wl(y.squeeze(1), y_hat.squeeze(1))
541
+ if a.slm_adv_weight > 0 and step >= a.disc_warmup:
542
+ l_slm = l_slm + a.slm_adv_weight / max(a.slm_weight, 1e-9) * \
543
+ wl.generator(y_hat.squeeze(1))
544
+ l_sil = torch.zeros((), device=dev)
545
+ if a.sil_weight > 0:
546
+ H = a.sr // 100 # 10 ms
547
+ n = y.shape[-1] // H
548
+ yr = y[..., : n * H].reshape(y.shape[0], n, H)
549
+ hr = y_hat[..., : n * H].reshape(y_hat.shape[0], n, H)
550
+ yrms = (yr.pow(2).mean(-1) + 1e-12).sqrt()
551
+ tsil = (yrms < a.sil_thresh).float()
552
+ if tsil.sum() > 0:
553
+ # RMS, not mean-square: the floor is ~5e-3, so squared it is ~3e-5 and
554
+ # would need a weight of ~10,000 to matter beside an stft term near 1.9.
555
+ # In RMS the weight reads directly as "how many dB do I care about silence".
556
+ #
557
+ # Hinged at the TARGET's own floor. Kokoro emits true digital silence
558
+ # (1e-6), so against the teacher this is the original loss to 4 decimals.
559
+ # Jenny is a real microphone: her quiet frames sit at 2.75e-4, and driving
560
+ # the student below its own target's room tone is not a fix, it is a
561
+ # different artefact (pumping between speech and dead air).
562
+ hrms = (hr.pow(2).mean(-1) + 1e-12).sqrt()
563
+ l_sil = (F.relu(hrms - yrms) * tsil).sum() / tsil.sum()
564
+ # The discriminator is noise until it can tell real from fake; connecting it to the
565
+ # generator immediately injects that noise into the weights.
566
+ l_g = gen_loss(y, y_hat) if step >= a.disc_warmup else torch.zeros((), device=dev)
567
+ loss = a.stft_weight * l_stft + a.dur_weight * l_dur + l_g \
568
+ + a.sil_weight * l_sil + a.mel_weight * l_mel \
569
+ + a.slm_weight * l_slm
570
+ loss.backward(); torch.nn.utils.clip_grad_norm_(model.parameters(), 5.0)
571
+ if styler is not None:
572
+ torch.nn.utils.clip_grad_norm_(styler.parameters(), 5.0)
573
+ if label_style is not None:
574
+ torch.nn.utils.clip_grad_norm_(label_style.parameters(), 5.0)
575
+ opt_g.step()
576
+ if sched_g is not None:
577
+ sched_g.step()
578
+ step += 1
579
+
580
+ for k, v in (("stft", l_stft), ("dur", l_dur), ("gen", l_g), ("disc", d_l),
581
+ ("sil", l_sil), ("mel", l_mel), ("slm", l_slm), ("r1r2", l_r1r2)):
582
+ acc[k] = acc.get(k, 0.0) + float(v)
583
+ if step % 100 == 0:
584
+ r = step * a.batch / (time.time() - t0)
585
+ print(f" step {step}/{a.steps} " +
586
+ " ".join(f"{k} {v/100:.3f}" for k, v in acc.items()) +
587
+ f" {r:.0f} clip/s", flush=True)
588
+ acc = {}
589
+ if a.ckpt_every and step % a.ckpt_every == 0:
590
+ tmp = out / "last.pt.tmp"
591
+ torch.save({"model": model.state_dict(), "mpd": mpd.state_dict(),
592
+ "msd": msd.state_dict(), "opt_g": opt_g.state_dict(),
593
+ "opt_d": opt_d.state_dict(), "step": step, "config": cfg,
594
+ **({"sched_g": sched_g.state_dict()} if sched_g else {}),
595
+ **({"styler": styler.state_dict()} if styler is not None else {}),
596
+ **({"label_style": label_style.state_dict()} if label_style is not None else {})}, tmp)
597
+ os.replace(tmp, ck)
598
+ torch.save(model.state_dict(), out / "student.pth")
599
+ if styler is not None:
600
+ torch.save(styler.state_dict(), out / "styler.pth")
601
+ if label_style is not None:
602
+ torch.save({label: label_style.weight[i + 1].detach().cpu() for i, label in enumerate(labels)}, out / "label_style.pt")
603
+ if step % 4000 == 0:
604
+ torch.save(model.state_dict(), out / f"student_{step}.pth")
605
+ print(f"[+] done at step {step}", flush=True)
606
+
607
+
608
+ if __name__ == "__main__":
609
+ main()