pinecoresystems commited on
Commit
6b94ece
·
verified ·
1 Parent(s): e886176

Mirror modeling_moss_audio_tokenizer.py

Browse files
Files changed (1) hide show
  1. modeling_moss_audio_tokenizer.py +1843 -0
modeling_moss_audio_tokenizer.py ADDED
@@ -0,0 +1,1843 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # Copyright 2026 OpenMOSS and the HuggingFace Inc. team. All rights reserved.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+ """PyTorch MossAudioTokenizer model."""
15
+
16
+ from __future__ import annotations
17
+
18
+ import copy
19
+ import math
20
+ from contextlib import ExitStack, contextmanager
21
+ from dataclasses import dataclass
22
+ from typing import cast
23
+
24
+ import torch
25
+ import torch.nn as nn
26
+ import torch.nn.functional as F
27
+
28
+ from transformers.modeling_utils import PreTrainedAudioTokenizerBase
29
+ from transformers.utils import ModelOutput, auto_docstring, logging
30
+ from .configuration_moss_audio_tokenizer import MossAudioTokenizerConfig
31
+
32
+
33
+ logger = logging.get_logger(__name__)
34
+
35
+
36
+ # =============================================================================
37
+ # Output Classes
38
+ # =============================================================================
39
+
40
+
41
+ @dataclass
42
+ @auto_docstring
43
+ class MossAudioTokenizerEncoderOutput(ModelOutput):
44
+ r"""
45
+ audio_codes (`torch.LongTensor` of shape `(num_quantizers, batch_size, sequence_length)`, *optional*):
46
+ Discrete audio codes computed using the encoder and quantizer.
47
+ audio_codes_lengths (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
48
+ Valid lengths for each sample's audio codes.
49
+ encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, hidden_size, sequence_length)`, *optional*):
50
+ Hidden states from the encoder before quantization.
51
+ """
52
+
53
+ audio_codes: torch.Tensor | None = None
54
+ audio_codes_lengths: torch.Tensor | None = None
55
+ encoder_hidden_states: torch.Tensor | None = None
56
+
57
+
58
+ @dataclass
59
+ @auto_docstring
60
+ class MossAudioTokenizerDecoderOutput(ModelOutput):
61
+ r"""
62
+ audio (`torch.FloatTensor` of shape `(batch_size, channels, sequence_length)`, *optional*):
63
+ Decoded audio waveform.
64
+ audio_lengths (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
65
+ Valid lengths for each sample's audio.
66
+ """
67
+
68
+ audio: torch.Tensor | None = None
69
+ audio_lengths: torch.Tensor | None = None
70
+
71
+
72
+ @dataclass
73
+ @auto_docstring
74
+ class MossAudioTokenizerOutput(ModelOutput):
75
+ r"""
76
+ audio (`torch.FloatTensor` of shape `(batch_size, channels, sequence_length)`, *optional*):
77
+ Decoded audio waveform.
78
+ audio_lengths (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
79
+ Valid lengths for each sample's audio.
80
+ audio_codes (`torch.LongTensor` of shape `(num_quantizers, batch_size, sequence_length)`, *optional*):
81
+ Discrete audio codes computed using the encoder and quantizer.
82
+ audio_codes_lengths (`torch.LongTensor` of shape `(batch_size,)`, *optional*):
83
+ Valid lengths for each sample's audio codes.
84
+ """
85
+
86
+ audio: torch.Tensor | None = None
87
+ audio_lengths: torch.Tensor | None = None
88
+ audio_codes: torch.Tensor | None = None
89
+ audio_codes_lengths: torch.Tensor | None = None
90
+
91
+
92
+ # =============================================================================
93
+ # Streaming Module Base Classes
94
+ # =============================================================================
95
+
96
+
97
+ @dataclass
98
+ class StreamingState:
99
+ """Base state for streaming modules."""
100
+
101
+ batch_size: int
102
+ device: torch.device
103
+
104
+ def __post_init__(self):
105
+ self.exec_mask = torch.ones(self.batch_size, dtype=torch.bool, device=self.device)
106
+
107
+ def set_exec_mask(self, exec_mask: torch.Tensor):
108
+ self.exec_mask[:] = exec_mask
109
+
110
+ def reset(self, reset_mask: torch.Tensor) -> None:
111
+ self.exec_mask[:] = torch.where(reset_mask, torch.ones_like(self.exec_mask), self.exec_mask)
112
+
113
+ def __enter__(self):
114
+ # ExitStack expects a context manager; returning self is conventional and useful for debugging.
115
+ return self
116
+
117
+ def __exit__(self, exc_type, exc_value, traceback) -> None:
118
+ pass
119
+
120
+
121
+ class StreamingModule(nn.Module):
122
+ """Base class for streaming components."""
123
+
124
+ def __init__(self) -> None:
125
+ super().__init__()
126
+ self._streaming_state: StreamingState | None = None
127
+ self._streaming_detached: bool = False
128
+ self._cached_children: list[tuple[str, StreamingModule]] | None = None
129
+
130
+ @property
131
+ def is_streaming(self):
132
+ return self._streaming_state is not None
133
+
134
+ def _apply_named_streaming(self, fn):
135
+ def _handle_module(prefix: str, module: nn.Module):
136
+ if isinstance(module, StreamingModule):
137
+ if module._streaming_detached and prefix != "":
138
+ return
139
+ if self._cached_children is None:
140
+ raise RuntimeError("Internal error: _cached_children should be initialized before traversal.")
141
+ self._cached_children.append((prefix, module))
142
+ for name, child in module.named_children():
143
+ new_prefix = f"{prefix}.{name}" if prefix else name
144
+ _handle_module(new_prefix, child)
145
+
146
+ if self._cached_children is None:
147
+ self._cached_children = []
148
+ _handle_module("", self)
149
+ for name, child in self._cached_children:
150
+ fn(name, child)
151
+
152
+ def _start_streaming(self, batch_size: int, exit_stack: ExitStack):
153
+ def _start_streaming_fn(name: str, module: StreamingModule):
154
+ if module._streaming_state is not None:
155
+ raise RuntimeError(f"{name} is already streaming!")
156
+ state = module._init_streaming_state(batch_size)
157
+ exit_stack.enter_context(state)
158
+ module._streaming_state = state
159
+
160
+ self._apply_named_streaming(_start_streaming_fn)
161
+
162
+ def _stop_streaming(self) -> None:
163
+ def _stop_streaming_fn(name: str, module: StreamingModule):
164
+ module._streaming_state = None
165
+
166
+ self._apply_named_streaming(_stop_streaming_fn)
167
+
168
+ def _init_streaming_state(self, batch_size: int) -> StreamingState:
169
+ device = next(iter(self.parameters())).device
170
+ return StreamingState(batch_size, device)
171
+
172
+ def streaming(self, batch_size: int) -> ExitStack:
173
+ """Context manager to enter streaming mode."""
174
+ exit_stack = ExitStack()
175
+ self._start_streaming(batch_size, exit_stack)
176
+ exit_stack.callback(self._stop_streaming)
177
+ return exit_stack
178
+
179
+
180
+ class StreamingContainer(StreamingModule):
181
+ """Container for streaming modules."""
182
+
183
+ pass
184
+
185
+
186
+ # =============================================================================
187
+ # Normalization Layers
188
+ # =============================================================================
189
+
190
+
191
+ class MossAudioTokenizerRMSNorm(nn.Module):
192
+ """Root Mean Square Layer Normalization."""
193
+
194
+ def __init__(
195
+ self,
196
+ dim: int,
197
+ eps: float = 1e-5,
198
+ dtype: torch.dtype | None = None,
199
+ device=None,
200
+ ):
201
+ super().__init__()
202
+ self.eps = eps
203
+ self.dtype = dtype
204
+ self.alpha = nn.Parameter(torch.full((1, 1, dim), 1.0, requires_grad=True, device=device, dtype=dtype))
205
+
206
+ def forward(self, x: torch.Tensor):
207
+ x_dtype = x.dtype
208
+ if self.dtype is not None:
209
+ x = x.to(self.dtype)
210
+ var = self.eps + torch.mean(x**2, dim=2, keepdim=True)
211
+ y = (x * (self.alpha.to(var) * torch.rsqrt(var))).to(x_dtype)
212
+ return y
213
+
214
+
215
+ class MossAudioTokenizerLayerScale(nn.Module):
216
+ """Layer scale from Touvron et al. 2021."""
217
+
218
+ def __init__(
219
+ self,
220
+ channels: int,
221
+ init: float = 1e-4,
222
+ channel_last: bool = True,
223
+ device=None,
224
+ dtype=None,
225
+ ):
226
+ super().__init__()
227
+ self.channel_last = channel_last
228
+ self.scale = nn.Parameter(torch.full((channels,), init, requires_grad=True, device=device, dtype=dtype))
229
+
230
+ def forward(self, x: torch.Tensor):
231
+ if self.channel_last:
232
+ return self.scale * x
233
+ else:
234
+ return self.scale[:, None] * x
235
+
236
+
237
+ def create_norm_fn(norm_type: str, dim: int, **kwargs) -> nn.Module:
238
+ """Create normalization module."""
239
+ if norm_type == "layer_norm":
240
+ return nn.LayerNorm(dim, eps=1e-5, **kwargs)
241
+ elif norm_type in {"rms_norm"}:
242
+ return MossAudioTokenizerRMSNorm(dim, eps=1e-5, **kwargs)
243
+ elif norm_type in {"rms_norm_f32"}:
244
+ kwargs.pop("dtype", None)
245
+ return MossAudioTokenizerRMSNorm(dim, eps=1e-8, dtype=torch.float, **kwargs)
246
+ else:
247
+ raise ValueError(f"Unknown norm type: {norm_type}")
248
+
249
+
250
+ # =============================================================================
251
+ # Rotary Position Embedding
252
+ # =============================================================================
253
+
254
+
255
+ def apply_rope(
256
+ q: torch.Tensor,
257
+ k: torch.Tensor,
258
+ offset: torch.Tensor,
259
+ max_period: float = 10_000,
260
+ time_before_heads: bool = False,
261
+ ):
262
+ """Apply rotary position embedding."""
263
+ if time_before_heads:
264
+ B, T, H, D = q.shape
265
+ else:
266
+ B, H, T, D = q.shape
267
+ if k.shape != q.shape:
268
+ raise ValueError(f"Expected k.shape == q.shape, got k={tuple(k.shape)} q={tuple(q.shape)}")
269
+ if D <= 0 or (D % 2) != 0:
270
+ raise ValueError(f"RoPE requires an even last dimension, got D={D}")
271
+
272
+ ds = torch.arange(D // 2, device=q.device, dtype=torch.float32)
273
+ freqs = torch.exp(ds * (-math.log(max_period) * 2 / D))
274
+ ts = offset.float().view(-1, 1) + torch.arange(T, device=q.device, dtype=torch.float32)
275
+
276
+ if time_before_heads:
277
+ ts = ts.view(B, -1, 1, 1)
278
+ else:
279
+ ts = ts.view(B, 1, -1, 1)
280
+
281
+ dims = q.shape[:-1]
282
+ q = q.view(*dims, D // 2, 2)
283
+ k = k.view(*dims, D // 2, 2)
284
+
285
+ qr, qi = q[..., 0].float(), q[..., 1].float()
286
+ kr, ki = k[..., 0].float(), k[..., 1].float()
287
+
288
+ rotr = torch.cos(freqs * ts)
289
+ roti = torch.sin(freqs * ts)
290
+
291
+ qor = qr * rotr - qi * roti
292
+ qoi = qr * roti + qi * rotr
293
+ kor = kr * rotr - ki * roti
294
+ koi = kr * roti + ki * rotr
295
+
296
+ dtype = q.dtype
297
+ qo = torch.stack([qor.to(dtype), qoi.to(dtype)], dim=-1)
298
+ ko = torch.stack([kor.to(dtype), koi.to(dtype)], dim=-1)
299
+
300
+ return qo.view(*dims, D), ko.view(*dims, D)
301
+
302
+
303
+ class MossAudioTokenizerRotaryEmbedding(nn.Module):
304
+ """Rotary positional embedding (RoPE)."""
305
+
306
+ def __init__(self, max_period: float = 10000.0):
307
+ super().__init__()
308
+ self.max_period = max_period
309
+
310
+ def forward(
311
+ self,
312
+ q: torch.Tensor,
313
+ k: torch.Tensor,
314
+ offset: torch.Tensor,
315
+ time_before_heads: bool = False,
316
+ ):
317
+ return apply_rope(q, k, offset, self.max_period, time_before_heads)
318
+
319
+
320
+ # =============================================================================
321
+ # Gating Modules
322
+ # =============================================================================
323
+
324
+
325
+ class MossAudioTokenizerActivationGating(nn.Module):
326
+ """Gating FFN layer with activation."""
327
+
328
+ def __init__(self, dim: int, dim_feedforward: int, activation, **factory_kwargs):
329
+ super().__init__()
330
+ if dim_feedforward == 4 * dim:
331
+ hidden = (21 * dim) // 8
332
+ else:
333
+ hidden = (2 * dim_feedforward) // 3
334
+
335
+ self.linear_in = nn.Linear(dim, 2 * hidden, bias=False, **factory_kwargs)
336
+ self.linear_out = nn.Linear(hidden, dim, bias=False, **factory_kwargs)
337
+ self.activation = activation
338
+
339
+ def forward(self, x: torch.Tensor):
340
+ x = self.linear_in(x)
341
+ B, T, _ = x.shape
342
+ x = x.view(B, T, 2, -1)
343
+ x = self.activation(x[..., 0, :]) * x[..., 1, :]
344
+ x = self.linear_out(x)
345
+ return x
346
+
347
+
348
+ def _get_activation(name: str):
349
+ if name in ["sigmoid", "tanh", "relu"]:
350
+ return getattr(torch, name)
351
+ elif name in ["leaky_relu", "elu", "gelu", "silu", "mish", "softsign"]:
352
+ return getattr(F, name)
353
+ elif name == "identity":
354
+ return nn.Identity()
355
+ else:
356
+ raise ValueError(f"Unknown activation {name}")
357
+
358
+
359
+ def make_gating(name: str, dim: int, dim_feedforward: int, **factory_kwargs) -> nn.Module:
360
+ return MossAudioTokenizerActivationGating(dim, dim_feedforward, _get_activation(name), **factory_kwargs)
361
+
362
+
363
+ # =============================================================================
364
+ # Positional Embeddings
365
+ # =============================================================================
366
+
367
+
368
+ def create_sin_embedding(
369
+ positions: torch.Tensor,
370
+ dim: int,
371
+ max_period: float = 10000,
372
+ dtype: torch.dtype = torch.float32,
373
+ ) -> torch.Tensor:
374
+ """Create sinusoidal positional embedding with shape [B, T, C]."""
375
+ if dim % 2 != 0:
376
+ raise ValueError(f"Sinusoidal embedding requires even dim, got dim={dim}")
377
+ half_dim = dim // 2
378
+ if half_dim <= 1:
379
+ raise ValueError(f"Sinusoidal embedding requires dim >= 4, got dim={dim}")
380
+ positions = positions.to(dtype)
381
+ adim = torch.arange(half_dim, device=positions.device, dtype=dtype).view(1, 1, -1)
382
+ max_period_tensor = torch.full([], max_period, device=positions.device, dtype=dtype)
383
+ phase = positions / (max_period_tensor ** (adim / (half_dim - 1)))
384
+ return torch.cat([torch.cos(phase), torch.sin(phase)], dim=-1)
385
+
386
+
387
+ # =============================================================================
388
+ # KV Cache for Attention
389
+ # =============================================================================
390
+
391
+
392
+ class KVCacheResult:
393
+ """Container for KV cache results that supports tuple unpacking."""
394
+
395
+ __slots__ = ("keys", "values", "positions")
396
+
397
+ def __init__(self, keys: torch.Tensor, values: torch.Tensor, positions: torch.Tensor):
398
+ self.keys = keys
399
+ self.values = values
400
+ self.positions = positions
401
+
402
+ def __iter__(self):
403
+ """Allow unpacking as (keys, values, positions)."""
404
+ return iter((self.keys, self.values, self.positions))
405
+
406
+ @staticmethod
407
+ def from_kv(keys: torch.Tensor, values: torch.Tensor) -> KVCacheResult:
408
+ B, H, T, D = keys.shape
409
+ positions = torch.arange(T, device=keys.device, dtype=torch.long)
410
+ return KVCacheResult(keys, values, positions.expand(B, -1))
411
+
412
+
413
+ class RingKVCache:
414
+ """Efficient streaming KVCache compatible with CUDA Graph."""
415
+
416
+ def __init__(
417
+ self,
418
+ batch_size: int,
419
+ num_heads: int,
420
+ dim_per_head: int,
421
+ capacity: int,
422
+ respect_exec_mask: bool = True,
423
+ device: torch.device = torch.device("cuda"),
424
+ dtype: torch.dtype = torch.bfloat16,
425
+ ):
426
+ self.capacity = capacity
427
+ self.cache = torch.zeros(
428
+ (2, batch_size, num_heads, capacity, dim_per_head),
429
+ device=device,
430
+ dtype=dtype,
431
+ )
432
+ self.respect_exec_mask = respect_exec_mask
433
+ if self.respect_exec_mask:
434
+ self.end_offset = torch.zeros(batch_size, device=device, dtype=torch.long)
435
+ else:
436
+ self.end_offset = torch.zeros(1, device=device, dtype=torch.long)
437
+
438
+ def reset(self, reset_mask: torch.Tensor) -> None:
439
+ self.end_offset[:] = torch.where(reset_mask, torch.zeros_like(self.end_offset), self.end_offset)
440
+
441
+ def complete(self, k: torch.Tensor, v: torch.Tensor, exec_mask: torch.Tensor) -> KVCacheResult:
442
+ B, H, T, D = k.shape
443
+ if T <= 0:
444
+ raise ValueError(f"Expected T > 0, got T={T}")
445
+
446
+ indexes = torch.arange(T, device=self.end_offset.device, dtype=self.end_offset.dtype)
447
+ indexes = indexes + self.end_offset.view(-1, 1)
448
+ indexes = indexes % self.capacity
449
+
450
+ if self.respect_exec_mask:
451
+ this_indexes = indexes.view(B, 1, T, 1).expand(-1, H, T, D)
452
+ self.cache[0].scatter_(2, this_indexes, k)
453
+ self.cache[1].scatter_(2, this_indexes, v)
454
+ else:
455
+ self.cache[0].index_copy_(2, indexes[0], k)
456
+ self.cache[1].index_copy_(2, indexes[0], v)
457
+
458
+ keys = self.cache[0]
459
+ values = self.cache[1]
460
+
461
+ indexes = torch.arange(self.capacity, device=self.end_offset.device, dtype=torch.long)
462
+ last_offset = self.end_offset.view(-1, 1) + T - 1
463
+ end_index = last_offset % self.capacity
464
+ delta = indexes - end_index
465
+
466
+ positions = torch.where(
467
+ delta <= 0,
468
+ last_offset + delta,
469
+ last_offset + delta - self.capacity,
470
+ )
471
+
472
+ if self.respect_exec_mask:
473
+ self.end_offset[:] = torch.where(exec_mask, self.end_offset + T, self.end_offset)
474
+ else:
475
+ self.end_offset.add_(T)
476
+
477
+ invalid = indexes >= self.end_offset.view(-1, 1)
478
+ positions = torch.where(invalid, torch.full_like(positions, -1), positions)
479
+
480
+ return KVCacheResult(keys, values, positions)
481
+
482
+
483
+ # =============================================================================
484
+ # Multi-Head Attention
485
+ # =============================================================================
486
+
487
+
488
+ @dataclass
489
+ class MHAState(StreamingState):
490
+ kv_cache: RingKVCache | None
491
+ offset: torch.Tensor
492
+ offset_cpu: int
493
+
494
+ def reset(self, reset_mask: torch.Tensor):
495
+ super().reset(reset_mask)
496
+ self.offset[:] = torch.where(reset_mask, torch.zeros_like(self.offset), self.offset)
497
+ if self.kv_cache is not None:
498
+ self.kv_cache.reset(reset_mask)
499
+ self.offset_cpu = 0
500
+
501
+
502
+ def apply_weights_per_step(
503
+ modules: nn.ModuleList,
504
+ schedule: list[int] | None,
505
+ x: torch.Tensor,
506
+ offset: int | None,
507
+ ) -> torch.Tensor:
508
+ """Apply different weights for each time step."""
509
+ if len(modules) == 1:
510
+ return modules[0](x)
511
+
512
+ if offset is None:
513
+ raise ValueError("offset must be provided when using per-step weights (len(modules) > 1).")
514
+ ys = []
515
+ B, T, C = x.shape
516
+ for t in range(T):
517
+ module_index = t + offset
518
+ if schedule is not None:
519
+ if module_index >= len(schedule) or module_index < 0:
520
+ raise ValueError(
521
+ f"weights_per_step_schedule is too short for module_index={module_index} (len={len(schedule)})."
522
+ )
523
+ module_index = schedule[module_index]
524
+ if module_index >= len(modules) or module_index < 0:
525
+ raise ValueError(f"module_index={module_index} out of range for len(modules)={len(modules)}.")
526
+ y = modules[module_index](x[:, t : t + 1])
527
+ ys.append(y)
528
+ return torch.cat(ys, 1)
529
+
530
+
531
+ class MossAudioTokenizerMultiheadAttention(StreamingModule):
532
+ """Multi-head attention with streaming support."""
533
+
534
+ def __init__(
535
+ self,
536
+ embed_dim: int,
537
+ num_heads: int,
538
+ causal: bool = False,
539
+ context: int | None = None,
540
+ rope: MossAudioTokenizerRotaryEmbedding | None = None,
541
+ weights_per_step: int = 0,
542
+ weights_per_step_schedule: list[int] | None = None,
543
+ device=None,
544
+ dtype=None,
545
+ ):
546
+ super().__init__()
547
+ factory_kwargs = {"device": device, "dtype": dtype}
548
+
549
+ self.embed_dim = embed_dim
550
+ self.causal = causal
551
+ self.context = context
552
+ self.rope = rope
553
+ self.num_heads = num_heads
554
+ self.weights_per_step = weights_per_step
555
+ self.weights_per_step_schedule = weights_per_step_schedule
556
+
557
+ out_dim = 3 * embed_dim
558
+ mult = 1
559
+ if weights_per_step:
560
+ mult = max(weights_per_step_schedule) + 1 if weights_per_step_schedule else weights_per_step
561
+ self.mult = mult
562
+
563
+ self.out_projs = nn.ModuleList(
564
+ [nn.Linear(embed_dim, embed_dim, bias=False, **factory_kwargs) for _ in range(mult)]
565
+ )
566
+ self.in_projs = nn.ModuleList(
567
+ [nn.Linear(embed_dim, out_dim, bias=False, **factory_kwargs) for _ in range(mult)]
568
+ )
569
+
570
+ self._register_load_state_dict_pre_hook(self._load_hook, with_module=True)
571
+
572
+ @staticmethod
573
+ def _load_hook(module, state_dict, prefix, *_):
574
+ mappings = {
575
+ "in_proj_weight": "in_projs.{i}.weight",
576
+ "in_proj.weight": "in_projs.{i}.weight",
577
+ "out_proj.weight": "out_projs.{i}.weight",
578
+ }
579
+ mult = module.mult
580
+ for suffix in ["", "_scb"]:
581
+ for source, target in mappings.items():
582
+ this_source = prefix + source + suffix
583
+ if this_source in state_dict:
584
+ weight = state_dict[this_source]
585
+ _, *OD = weight.shape
586
+ weight = weight.view(mult, -1, *OD)
587
+ for i in range(mult):
588
+ state_dict[prefix + target.format(i=i) + suffix] = weight[i]
589
+ state_dict.pop(this_source)
590
+
591
+ def _init_streaming_state(self, batch_size: int) -> MHAState:
592
+ in_proj = cast(nn.Linear, self.in_projs[0])
593
+ device = cast(torch.device, in_proj.weight.device)
594
+ dtype = cast(torch.dtype, in_proj.weight.dtype)
595
+
596
+ dim_per_head = self.embed_dim // self.num_heads
597
+ if self.context is None:
598
+ capacity = self.weights_per_step if self.weights_per_step else 1024
599
+ else:
600
+ capacity = self.context
601
+
602
+ kv_cache = RingKVCache(
603
+ batch_size,
604
+ self.num_heads,
605
+ dim_per_head,
606
+ capacity,
607
+ respect_exec_mask=not self.weights_per_step,
608
+ device=cast(torch.device, device),
609
+ dtype=cast(torch.dtype, dtype),
610
+ )
611
+ return MHAState(
612
+ batch_size,
613
+ cast(torch.device, device),
614
+ kv_cache,
615
+ offset=torch.zeros(batch_size, device=cast(torch.device, device), dtype=torch.long),
616
+ offset_cpu=0,
617
+ )
618
+
619
+ def _complete_kv(self, k, v) -> KVCacheResult:
620
+ state = cast(MHAState | None, self._streaming_state)
621
+ if state is None:
622
+ return KVCacheResult.from_kv(k, v)
623
+ if state.kv_cache is None:
624
+ return KVCacheResult.from_kv(k, v)
625
+ return state.kv_cache.complete(k, v, state.exec_mask)
626
+
627
+ def forward(self, query: torch.Tensor, key: torch.Tensor, value: torch.Tensor):
628
+ state = cast(MHAState | None, self._streaming_state)
629
+ B, T = query.shape[:2]
630
+
631
+ if state is None:
632
+ offset = torch.zeros(B, device=query.device, dtype=torch.long)
633
+ offset_cpu = 0
634
+ else:
635
+ offset = state.offset
636
+ offset_cpu = state.offset_cpu
637
+
638
+ projected = apply_weights_per_step(self.in_projs, self.weights_per_step_schedule, query, offset_cpu)
639
+ dim_per_head = self.embed_dim // self.num_heads
640
+ projected = projected.reshape(B, T, 3, self.num_heads, dim_per_head).permute(2, 0, 3, 1, 4)
641
+ q, k, v = projected[0], projected[1], projected[2]
642
+
643
+ if self.rope:
644
+ q, k = self.rope(q, k, offset, time_before_heads=False)
645
+
646
+ k, v, pos_k = self._complete_kv(k, v)
647
+ pos_k = pos_k[:, None]
648
+
649
+ if self.causal:
650
+ pos_q = offset.view(-1, 1, 1) + torch.arange(T, device=q.device, dtype=torch.long).view(-1, 1)
651
+ delta = pos_q - pos_k
652
+ attn_bias = (pos_k >= 0) & (delta >= 0)
653
+ if self.context is not None:
654
+ attn_bias = attn_bias & (delta < self.context)
655
+ attn_bias = attn_bias[:, None]
656
+ else:
657
+ attn_bias = None
658
+
659
+ x = F.scaled_dot_product_attention(q, k, v, attn_bias, dropout_p=0.0)
660
+ x = x.transpose(1, 2).reshape(B, T, self.embed_dim)
661
+ x = apply_weights_per_step(self.out_projs, self.weights_per_step_schedule, x, offset_cpu)
662
+
663
+ if state is not None:
664
+ state.offset[:] = torch.where(state.exec_mask, state.offset + T, state.offset)
665
+ state.offset_cpu += T
666
+ return x
667
+
668
+
669
+ # =============================================================================
670
+ # Transformer Layer
671
+ # =============================================================================
672
+
673
+
674
+ @dataclass
675
+ class LayerState(StreamingState):
676
+ offset_cpu: int = 0
677
+
678
+ def reset(self, reset_mask: torch.Tensor):
679
+ super().reset(reset_mask)
680
+ self.offset_cpu = 0
681
+
682
+
683
+ class MossAudioTokenizerTransformerLayer(StreamingModule):
684
+ """Transformer layer with streaming support."""
685
+
686
+ def __init__(
687
+ self,
688
+ d_model: int,
689
+ num_heads: int,
690
+ dim_feedforward: int = 2048,
691
+ causal: bool = False,
692
+ context: int | None = None,
693
+ rope: MossAudioTokenizerRotaryEmbedding | None = None,
694
+ norm: str = "layer_norm",
695
+ layer_scale: float | None = None,
696
+ gating: str = "none",
697
+ weights_per_step: int = 0,
698
+ weights_per_step_schedule: list[int] | None = None,
699
+ activation=F.gelu,
700
+ device=None,
701
+ dtype=None,
702
+ ):
703
+ super().__init__()
704
+ factory_kwargs = {"device": device, "dtype": dtype}
705
+
706
+ self.self_attn = MossAudioTokenizerMultiheadAttention(
707
+ embed_dim=d_model,
708
+ num_heads=num_heads,
709
+ causal=causal,
710
+ context=context,
711
+ rope=rope,
712
+ weights_per_step=weights_per_step,
713
+ weights_per_step_schedule=weights_per_step_schedule,
714
+ **factory_kwargs,
715
+ )
716
+ self.norm1 = create_norm_fn(norm, d_model, **factory_kwargs)
717
+ self.norm2 = create_norm_fn(norm, d_model, **factory_kwargs)
718
+
719
+ self.weights_per_step = weights_per_step
720
+ self.weights_per_step_schedule = weights_per_step_schedule
721
+ self.gating: nn.Module | nn.ModuleList | None = None
722
+ self.linear1: nn.Module | None = None
723
+ self.linear2: nn.Module | None = None
724
+ self.activation = activation
725
+
726
+ num_weights = 1
727
+ if weights_per_step:
728
+ num_weights = max(weights_per_step_schedule) + 1 if weights_per_step_schedule else weights_per_step
729
+
730
+ if gating == "none":
731
+ self.linear1 = nn.Linear(d_model, dim_feedforward, bias=False, **factory_kwargs)
732
+ self.linear2 = nn.Linear(dim_feedforward, d_model, bias=False, **factory_kwargs)
733
+ else:
734
+ if weights_per_step:
735
+ dim_ff_list = [dim_feedforward] * num_weights if isinstance(dim_feedforward, int) else dim_feedforward
736
+ self.gating = nn.ModuleList(
737
+ [make_gating(gating, d_model, dim, **factory_kwargs) for dim in dim_ff_list]
738
+ )
739
+ else:
740
+ self.gating = make_gating(gating, d_model, dim_feedforward, **factory_kwargs)
741
+
742
+ if layer_scale is None:
743
+ self.layer_scale_1 = nn.Identity()
744
+ self.layer_scale_2 = nn.Identity()
745
+ else:
746
+ self.layer_scale_1 = MossAudioTokenizerLayerScale(
747
+ channels=d_model, init=layer_scale, channel_last=True, **cast(dict[str, object], factory_kwargs)
748
+ )
749
+ self.layer_scale_2 = MossAudioTokenizerLayerScale(
750
+ channels=d_model, init=layer_scale, channel_last=True, **cast(dict[str, object], factory_kwargs)
751
+ )
752
+
753
+ def _init_streaming_state(self, batch_size: int) -> LayerState:
754
+ device = next(iter(self.parameters())).device
755
+ return LayerState(batch_size, device, offset_cpu=0)
756
+
757
+ def _ff_block(self, x: torch.Tensor) -> torch.Tensor:
758
+ state = self._streaming_state
759
+ offset = state.offset_cpu if isinstance(state, LayerState) else 0
760
+
761
+ x_orig = x
762
+ x = self.norm2(x)
763
+
764
+ if self.gating is None:
765
+ assert self.linear1 is not None
766
+ assert self.linear2 is not None
767
+ update = self.linear2(self.activation(self.linear1(x)))
768
+ else:
769
+ if self.weights_per_step:
770
+ assert isinstance(self.gating, nn.ModuleList)
771
+ update = apply_weights_per_step(self.gating, self.weights_per_step_schedule, x, offset)
772
+ else:
773
+ update = self.gating(x)
774
+ return x_orig.to(update) + self.layer_scale_2(update)
775
+
776
+ def _sa_block(self, x: torch.Tensor):
777
+ x_orig = x
778
+ x = self.norm1(x)
779
+ update = self.self_attn(x, x, x)
780
+ return x_orig.to(update) + self.layer_scale_1(update)
781
+
782
+ def forward(self, x: torch.Tensor):
783
+ x = self._sa_block(x)
784
+ x = self._ff_block(x)
785
+ state = self._streaming_state
786
+ if state is not None:
787
+ assert isinstance(state, LayerState)
788
+ state.offset_cpu += x.shape[1]
789
+ return x
790
+
791
+
792
+ # =============================================================================
793
+ # Streaming Transformer
794
+ # =============================================================================
795
+
796
+
797
+ @dataclass
798
+ class TransformerState(StreamingState):
799
+ offsets: torch.Tensor
800
+
801
+ def reset(self, reset_mask: torch.Tensor):
802
+ super().reset(reset_mask)
803
+ self.offsets[:] = torch.where(reset_mask, torch.zeros_like(self.offsets), self.offsets)
804
+
805
+
806
+ class MossAudioTokenizerTransformer(StreamingModule):
807
+ """Transformer with streaming/causal support."""
808
+
809
+ def __init__(
810
+ self,
811
+ d_model: int,
812
+ num_heads: int,
813
+ num_layers: int,
814
+ dim_feedforward: int = 2048,
815
+ causal: bool = False,
816
+ context: int | None = None,
817
+ positional_embedding: str = "sin",
818
+ max_period: float = 10_000,
819
+ positional_scale: float = 1.0,
820
+ device=None,
821
+ dtype=None,
822
+ **kwargs,
823
+ ):
824
+ super().__init__()
825
+ if d_model % num_heads != 0:
826
+ raise ValueError(f"d_model must be divisible by num_heads, got d_model={d_model}, num_heads={num_heads}")
827
+
828
+ self.positional_embedding = positional_embedding
829
+ self.max_period = max_period
830
+ self.positional_scale = positional_scale
831
+
832
+ self.rope: MossAudioTokenizerRotaryEmbedding | None = None
833
+ if positional_embedding in {"rope", "sin_rope"}:
834
+ self.rope = MossAudioTokenizerRotaryEmbedding(max_period=max_period)
835
+
836
+ self.layers = nn.ModuleList()
837
+ for _ in range(num_layers):
838
+ self.layers.append(
839
+ MossAudioTokenizerTransformerLayer(
840
+ d_model=d_model,
841
+ num_heads=num_heads,
842
+ dim_feedforward=dim_feedforward,
843
+ causal=causal,
844
+ context=context,
845
+ rope=self.rope,
846
+ device=device,
847
+ dtype=dtype,
848
+ **kwargs,
849
+ )
850
+ )
851
+
852
+ def _init_streaming_state(self, batch_size: int) -> TransformerState:
853
+ device = next(self.parameters()).device
854
+ return TransformerState(
855
+ batch_size,
856
+ device,
857
+ offsets=torch.zeros(batch_size, device=device, dtype=torch.long),
858
+ )
859
+
860
+ def forward(self, x: torch.Tensor, *args, **kwargs):
861
+ B, T, C = x.shape
862
+ state = self._streaming_state
863
+ offsets = (
864
+ torch.zeros(1, dtype=torch.long, device=x.device)
865
+ if state is None
866
+ else (
867
+ state.offsets
868
+ if isinstance(state, TransformerState)
869
+ else torch.zeros(1, dtype=torch.long, device=x.device)
870
+ )
871
+ )
872
+
873
+ if self.positional_embedding in {"sin", "sin_rope"}:
874
+ positions = torch.arange(T, device=x.device).view(1, -1, 1)
875
+ positions = positions + offsets.view(-1, 1, 1)
876
+ pos_emb = create_sin_embedding(positions, C, max_period=self.max_period, dtype=x.dtype)
877
+ x = x + self.positional_scale * pos_emb
878
+
879
+ for layer in self.layers:
880
+ x = layer(x, *args, **kwargs)
881
+
882
+ if state is not None:
883
+ assert isinstance(state, TransformerState)
884
+ state.offsets[:] = torch.where(state.exec_mask, state.offsets + T, state.offsets)
885
+ return x
886
+
887
+
888
+ class MossAudioTokenizerProjectedTransformer(StreamingContainer):
889
+ """Transformer with input/output projections."""
890
+
891
+ def __init__(
892
+ self,
893
+ input_dimension: int,
894
+ output_dimension: int,
895
+ d_model: int,
896
+ *,
897
+ conv_layout: bool = False,
898
+ module_type: str,
899
+ **kwargs,
900
+ ):
901
+ super().__init__()
902
+ self.module_type = module_type
903
+ self.downsample_ratio: int = 1
904
+ self.input_dimension = input_dimension
905
+ self.output_dimension = output_dimension
906
+
907
+ self.input_proj = (
908
+ nn.Linear(input_dimension, d_model, bias=False) if d_model != input_dimension else nn.Identity()
909
+ )
910
+ self.transformer = MossAudioTokenizerTransformer(d_model=d_model, **kwargs)
911
+ self.conv_layout = conv_layout
912
+ self.output_proj = (
913
+ nn.Linear(d_model, output_dimension, bias=False) if d_model != output_dimension else nn.Identity()
914
+ )
915
+
916
+ def forward(self, x, input_lengths, *args, **kwargs):
917
+ x = self.input_proj(x.transpose(1, 2)) # (B, D, T) -> (B, T, D)
918
+ x = self.transformer(x, *args, **kwargs)
919
+ x = self.output_proj(x).transpose(1, 2) # (B, T, D) -> (B, D, T)
920
+ return x, input_lengths
921
+
922
+
923
+ # =============================================================================
924
+ # Patched Pretransform Module
925
+ # =============================================================================
926
+
927
+
928
+ class MossAudioTokenizerPatchedPretransform(nn.Module):
929
+ """Patching module for downsampling/upsampling."""
930
+
931
+ def __init__(self, patch_size: int, is_downsample: bool, module_type: str, **kwargs):
932
+ super().__init__()
933
+ self.patch_size = patch_size
934
+ self.downsample_ratio: int = patch_size
935
+ self.is_downsample = is_downsample
936
+ self.module_type = module_type
937
+
938
+ def encode(self, x, input_lengths):
939
+ b, d, _ = x.shape
940
+ h = self.patch_size
941
+ x = x.reshape(b, d, -1, h).permute(0, 1, 3, 2).reshape(b, d * h, -1)
942
+ # We pad the input waveform to a multiple of `downsample_rate` before applying the encoder.
943
+ # Use a ceil division to match that padding and avoid dropping the last (partially padded) frame.
944
+ output_lengths = input_lengths // self.patch_size
945
+ return x, output_lengths
946
+
947
+ def decode(self, x, input_lengths):
948
+ b, dh, l = x.shape
949
+ h = self.patch_size
950
+ d = dh // h
951
+ x = x.reshape(b, d, h, l).permute(0, 1, 3, 2).reshape(b, d, l * h)
952
+ output_lengths = input_lengths * self.patch_size
953
+ return x, output_lengths
954
+
955
+ def forward(self, x, input_lengths):
956
+ if self.is_downsample:
957
+ return self.encode(x, input_lengths)
958
+ else:
959
+ return self.decode(x, input_lengths)
960
+
961
+
962
+ # =============================================================================
963
+ # Vector Quantization
964
+ # =============================================================================
965
+
966
+
967
+ def WNConv1d(*args, **kwargs):
968
+ """Weight-normalized Conv1d."""
969
+ return nn.utils.parametrizations.weight_norm(nn.Conv1d(*args, **kwargs))
970
+
971
+
972
+ class MossAudioTokenizerVectorQuantize(nn.Module):
973
+ """Single codebook vector quantization (inference only)."""
974
+
975
+ def __init__(
976
+ self,
977
+ input_dim: int,
978
+ codebook_size: int,
979
+ codebook_dim: int,
980
+ **kwargs,
981
+ ):
982
+ super().__init__()
983
+ self.input_dim = input_dim
984
+ self.codebook_size = codebook_size
985
+ self.codebook_dim = codebook_dim
986
+
987
+ if input_dim != codebook_dim:
988
+ self.in_proj = WNConv1d(input_dim, codebook_dim, kernel_size=1)
989
+ self.out_proj = WNConv1d(codebook_dim, input_dim, kernel_size=1)
990
+ else:
991
+ self.in_proj = nn.Identity()
992
+ self.out_proj = nn.Identity()
993
+
994
+ self.codebook = nn.Embedding(codebook_size, codebook_dim)
995
+
996
+ @torch.no_grad()
997
+ def forward(self, z: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
998
+ """
999
+ Args:
1000
+ z: Input tensor of shape (B, D, T)
1001
+ Returns:
1002
+ z_q: Quantized tensor of shape (B, D, T)
1003
+ indices: Code indices of shape (B, T)
1004
+ z_e: Encoded tensor before quantization
1005
+ """
1006
+ z = z.float()
1007
+ z_e = self.in_proj(z).float()
1008
+
1009
+ encodings = z_e.transpose(1, 2).reshape(-1, z_e.shape[1])
1010
+
1011
+ codebook_weight = self.codebook.weight
1012
+ dist = (
1013
+ encodings.pow(2).sum(1, keepdim=True)
1014
+ - 2 * encodings @ codebook_weight.float().t()
1015
+ + codebook_weight.float().pow(2).sum(1, keepdim=True).t()
1016
+ )
1017
+
1018
+ indices = (-dist).max(1)[1]
1019
+ indices = indices.reshape(z.size(0), -1)
1020
+
1021
+ z_q = self.decode_code(indices)
1022
+ z_q = self.out_proj(z_q).float()
1023
+
1024
+ return z_q, indices, z_e
1025
+
1026
+ def decode_code(self, embed_id: torch.Tensor) -> torch.Tensor:
1027
+ """Decode code indices to embeddings."""
1028
+ return self.codebook(embed_id).transpose(1, 2).float()
1029
+
1030
+
1031
+ class MossAudioTokenizerLFQ(nn.Module):
1032
+ """LFQ (inference-only) used by ResidualLFQ."""
1033
+
1034
+ def __init__(
1035
+ self,
1036
+ input_dim: int,
1037
+ codebook_size: int,
1038
+ codebook_dim: int,
1039
+ **kwargs,
1040
+ ):
1041
+ super().__init__()
1042
+ self.input_dim = input_dim
1043
+ self.codebook_size = codebook_size
1044
+ self.codebook_dim = codebook_dim
1045
+
1046
+ if self.input_dim != self.codebook_dim:
1047
+ self.in_proj = WNConv1d(self.input_dim, self.codebook_dim, kernel_size=1)
1048
+ self.out_proj = WNConv1d(self.codebook_dim, self.input_dim, kernel_size=1)
1049
+ else:
1050
+ self.in_proj = nn.Identity()
1051
+ self.out_proj = nn.Identity()
1052
+
1053
+ self.codebook = nn.Embedding(codebook_size, codebook_dim)
1054
+
1055
+ @torch.no_grad()
1056
+ def forward(self, z: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
1057
+ """Quantize z into codebook vectors."""
1058
+ z = z.float()
1059
+ z_e = self.in_proj(z).float()
1060
+ z_q, indices = self.decode_latents(z_e)
1061
+ z_q = (z_e + (z_q - z_e).detach()).float()
1062
+ z_q = self.out_proj(z_q).float()
1063
+ return z_q, indices, z_e
1064
+
1065
+ def embed_code(self, embed_id: torch.Tensor) -> torch.Tensor:
1066
+ return F.embedding(embed_id, self.codebook.weight)
1067
+
1068
+ def decode_code_wo_out_proj(self, embed_id: torch.Tensor) -> torch.Tensor:
1069
+ return self.embed_code(embed_id).transpose(1, 2)
1070
+
1071
+ def decode_code(self, embed_id: torch.Tensor) -> torch.Tensor:
1072
+ z_q = self.decode_code_wo_out_proj(embed_id).float()
1073
+ z_q = self.out_proj(z_q).float()
1074
+ return z_q
1075
+
1076
+ def decode_latents(self, latents: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
1077
+ """Match training LFQ: L2-normalize then argmin squared distance."""
1078
+ encodings = latents.transpose(1, 2).reshape(-1, latents.shape[1]).float()
1079
+ codebook = self.codebook.weight.float()
1080
+
1081
+ encodings = F.normalize(encodings)
1082
+ codebook = F.normalize(codebook)
1083
+
1084
+ dist = (
1085
+ encodings.pow(2).sum(1, keepdim=True)
1086
+ - 2 * encodings @ codebook.t()
1087
+ + codebook.pow(2).sum(1, keepdim=True).t()
1088
+ )
1089
+ indices = (-dist).max(1)[1]
1090
+ indices = indices.reshape(latents.size(0), -1)
1091
+ z_q = self.decode_code_wo_out_proj(indices).float()
1092
+ return z_q, indices
1093
+
1094
+
1095
+ class MossAudioTokenizerResidualVQ(nn.Module):
1096
+ """Residual Vector Quantization (inference only)."""
1097
+
1098
+ def __init__(
1099
+ self,
1100
+ input_dim: int = 1024,
1101
+ rvq_dim: int | None = None,
1102
+ output_dim: int | None = None,
1103
+ num_quantizers: int = 32,
1104
+ codebook_size: int = 1024,
1105
+ codebook_dim: int = 8,
1106
+ **kwargs,
1107
+ ):
1108
+ super().__init__()
1109
+ self.input_dim = input_dim
1110
+ self.rvq_dim = rvq_dim or input_dim
1111
+ self.output_dim = output_dim or input_dim
1112
+ self.num_quantizers = num_quantizers
1113
+ self.codebook_size = codebook_size
1114
+ self.codebook_dim = codebook_dim
1115
+
1116
+ self.input_proj = (
1117
+ WNConv1d(input_dim, self.rvq_dim, kernel_size=1) if input_dim != self.rvq_dim else nn.Identity()
1118
+ )
1119
+ self.output_proj = (
1120
+ WNConv1d(self.rvq_dim, self.output_dim, kernel_size=1)
1121
+ if self.rvq_dim != self.output_dim
1122
+ else nn.Identity()
1123
+ )
1124
+
1125
+ self.quantizers = nn.ModuleList(
1126
+ [
1127
+ MossAudioTokenizerVectorQuantize(
1128
+ input_dim=self.rvq_dim,
1129
+ codebook_size=codebook_size,
1130
+ codebook_dim=codebook_dim,
1131
+ **kwargs,
1132
+ )
1133
+ for _ in range(num_quantizers)
1134
+ ]
1135
+ )
1136
+
1137
+ @torch.no_grad()
1138
+ def forward(
1139
+ self,
1140
+ z: torch.Tensor,
1141
+ input_length: torch.Tensor,
1142
+ n_quantizers: int | None = None,
1143
+ ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
1144
+ """
1145
+ Args:
1146
+ z: Input tensor of shape (B, D, T)
1147
+ input_length: Valid lengths for each sample (B,)
1148
+ n_quantizers: Number of quantizers to use
1149
+ Returns:
1150
+ quantized_out: Quantized output (B, D, T)
1151
+ all_indices: All code indices (N, B, T)
1152
+ output_length: Output lengths (B,)
1153
+ """
1154
+ z = self.input_proj(z)
1155
+
1156
+ batch_size, _, max_time = z.shape
1157
+ mask = torch.arange(max_time, device=z.device).expand(batch_size, max_time) < input_length.unsqueeze(1)
1158
+
1159
+ quantized_out = torch.zeros_like(z, dtype=torch.float32)
1160
+ residual = z.clone().float()
1161
+ all_indices = []
1162
+
1163
+ n_quantizers = n_quantizers or self.num_quantizers
1164
+
1165
+ for i, quantizer in enumerate(self.quantizers):
1166
+ if i >= n_quantizers:
1167
+ break
1168
+
1169
+ masked_residual = residual * mask.unsqueeze(1)
1170
+ z_q_i, indices_i, _ = quantizer(masked_residual)
1171
+
1172
+ update_mask = mask.unsqueeze(1)
1173
+ quantized_out = quantized_out + z_q_i * update_mask
1174
+ residual = residual - z_q_i * update_mask
1175
+ all_indices.append(indices_i)
1176
+
1177
+ all_indices = torch.stack(all_indices) # (N, B, T)
1178
+ quantized_out = self.output_proj(quantized_out)
1179
+
1180
+ return quantized_out, all_indices, input_length
1181
+
1182
+ def decode_codes(self, codes: torch.Tensor) -> torch.Tensor:
1183
+ """Decode codes from multiple quantizers to embeddings."""
1184
+ nq, B, T = codes.shape
1185
+ emb = torch.zeros(B, self.rvq_dim, T, device=codes.device, dtype=torch.float32)
1186
+
1187
+ for i, quantizer in enumerate(self.quantizers[:nq]):
1188
+ quantizer = cast(MossAudioTokenizerVectorQuantize, quantizer)
1189
+ quantized_i = quantizer.decode_code(codes[i])
1190
+ emb += quantized_i
1191
+
1192
+ emb = self.output_proj(emb)
1193
+ return emb
1194
+
1195
+
1196
+ class MossAudioTokenizerResidualLFQ(nn.Module):
1197
+ """Residual LFQ (inference only)."""
1198
+
1199
+ def __init__(
1200
+ self,
1201
+ input_dim: int = 1024,
1202
+ rvq_dim: int | None = None,
1203
+ output_dim: int | None = None,
1204
+ num_quantizers: int = 32,
1205
+ codebook_size: int = 1024,
1206
+ codebook_dim: int = 8,
1207
+ **kwargs,
1208
+ ):
1209
+ super().__init__()
1210
+ self.input_dim = input_dim
1211
+ self.rvq_dim = rvq_dim or input_dim
1212
+ self.output_dim = output_dim or input_dim
1213
+ self.num_quantizers = num_quantizers
1214
+ self.codebook_size = codebook_size
1215
+ self.codebook_dim = codebook_dim
1216
+
1217
+ self.input_proj = (
1218
+ WNConv1d(input_dim, self.rvq_dim, kernel_size=1) if input_dim != self.rvq_dim else nn.Identity()
1219
+ )
1220
+ self.output_proj = (
1221
+ WNConv1d(self.rvq_dim, self.output_dim, kernel_size=1)
1222
+ if self.rvq_dim != self.output_dim
1223
+ else nn.Identity()
1224
+ )
1225
+
1226
+ self.quantizers = nn.ModuleList(
1227
+ [
1228
+ MossAudioTokenizerLFQ(
1229
+ input_dim=self.rvq_dim,
1230
+ codebook_size=codebook_size,
1231
+ codebook_dim=codebook_dim,
1232
+ **kwargs,
1233
+ )
1234
+ for _ in range(num_quantizers)
1235
+ ]
1236
+ )
1237
+
1238
+ @torch.no_grad()
1239
+ def forward(
1240
+ self,
1241
+ z: torch.Tensor,
1242
+ input_length: torch.Tensor,
1243
+ n_quantizers: int | None = None,
1244
+ ) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
1245
+ """Inference quantization."""
1246
+ z = self.input_proj(z).float()
1247
+
1248
+ batch_size, _, max_time = z.shape
1249
+ mask = torch.arange(max_time, device=z.device).expand(batch_size, max_time) < input_length.unsqueeze(1)
1250
+
1251
+ quantized_out = torch.zeros_like(z, dtype=torch.float32)
1252
+ residual = z.clone().float()
1253
+ all_indices = []
1254
+
1255
+ n_quantizers = n_quantizers or self.num_quantizers
1256
+ for i, quantizer in enumerate(self.quantizers):
1257
+ if i >= n_quantizers:
1258
+ break
1259
+
1260
+ masked_residual = residual * mask.unsqueeze(1)
1261
+ z_q_i, indices_i, _ = quantizer(masked_residual)
1262
+
1263
+ update_mask = mask.unsqueeze(1)
1264
+ quantized_out = quantized_out + z_q_i * update_mask
1265
+ residual = residual - z_q_i * update_mask
1266
+ all_indices.append(indices_i)
1267
+
1268
+ all_indices = (
1269
+ torch.stack(all_indices)
1270
+ if all_indices
1271
+ else torch.empty(0, batch_size, max_time, device=z.device, dtype=torch.long)
1272
+ )
1273
+ quantized_out = self.output_proj(quantized_out)
1274
+ return quantized_out, all_indices, input_length
1275
+
1276
+ def decode_codes(self, codes: torch.Tensor) -> torch.Tensor:
1277
+ nq, B, T = codes.shape
1278
+ emb = torch.zeros(B, self.rvq_dim, T, device=codes.device, dtype=torch.float32)
1279
+ for i, quantizer in enumerate(self.quantizers[:nq]):
1280
+ quantizer = cast(MossAudioTokenizerLFQ, quantizer)
1281
+ emb += quantizer.decode_code(codes[i]).float()
1282
+ emb = self.output_proj(emb)
1283
+ return emb
1284
+
1285
+
1286
+ # =============================================================================
1287
+ # Main Model Classes
1288
+ # =============================================================================
1289
+
1290
+
1291
+ @auto_docstring
1292
+ class MossAudioTokenizerPreTrainedModel(PreTrainedAudioTokenizerBase):
1293
+ """Base class for MossAudioTokenizer models."""
1294
+
1295
+ config_class = MossAudioTokenizerConfig
1296
+ base_model_prefix = ""
1297
+ main_input_name = "input_values"
1298
+ input_modalities = "audio"
1299
+ supports_gradient_checkpointing = False
1300
+ _no_split_modules = [
1301
+ "MossAudioTokenizerTransformerLayer",
1302
+ "MossAudioTokenizerResidualVQ",
1303
+ "MossAudioTokenizerResidualLFQ",
1304
+ ]
1305
+
1306
+
1307
+ @auto_docstring(
1308
+ custom_intro="""
1309
+ The MossAudioTokenizer neural audio codec model for audio tokenization and synthesis.
1310
+ """
1311
+ )
1312
+ class MossAudioTokenizerModel(MossAudioTokenizerPreTrainedModel):
1313
+ """
1314
+ MossAudioTokenizer model for audio tokenization and synthesis.
1315
+
1316
+ This model can encode audio waveforms into discrete tokens and decode
1317
+ tokens back into audio waveforms.
1318
+ """
1319
+
1320
+ def __init__(self, config: MossAudioTokenizerConfig):
1321
+ super().__init__(config)
1322
+
1323
+ self.config = config
1324
+ _ = config.version
1325
+ self.sampling_rate = config.sampling_rate
1326
+ self.downsample_rate = config.downsample_rate
1327
+ self.causal_transformer_context_duration = config.causal_transformer_context_duration
1328
+
1329
+ # Build encoder
1330
+ current_frame_rate: float = float(self.sampling_rate)
1331
+ self.encoder = nn.ModuleList()
1332
+
1333
+ for encoder_kwargs_i in config.encoder_kwargs:
1334
+ encoder_kwargs_i = dict(encoder_kwargs_i) # Make a copy
1335
+ if encoder_kwargs_i["module_type"] == "PatchedPretransform":
1336
+ self.encoder.append(MossAudioTokenizerPatchedPretransform(**encoder_kwargs_i, is_downsample=True))
1337
+ elif encoder_kwargs_i["module_type"] == "Transformer":
1338
+ self.encoder.append(
1339
+ MossAudioTokenizerProjectedTransformer(
1340
+ **encoder_kwargs_i,
1341
+ context=int(current_frame_rate * self.causal_transformer_context_duration),
1342
+ )
1343
+ )
1344
+ current_frame_rate /= self.encoder[-1].downsample_ratio
1345
+
1346
+ # Build quantizer
1347
+ quantizer_kwargs = dict(config.quantizer_kwargs)
1348
+ quantizer_type = quantizer_kwargs.get("quantizer_type", getattr(config, "quantizer_type", "rvq"))
1349
+ if quantizer_type in {"rvq", "spec_rvq"}:
1350
+ self.quantizer = MossAudioTokenizerResidualVQ(**quantizer_kwargs)
1351
+ elif quantizer_type in {"rlfq", "random_prefix_rlfq"}:
1352
+ self.quantizer = MossAudioTokenizerResidualLFQ(**quantizer_kwargs)
1353
+ else:
1354
+ raise ValueError(f"Unsupported quantizer_type: {quantizer_type}")
1355
+
1356
+ # Build decoder
1357
+ decoder_kwargs_list = copy.deepcopy(config.decoder_kwargs)
1358
+ self.decoder = nn.ModuleList()
1359
+
1360
+ for decoder_kwargs_i in decoder_kwargs_list:
1361
+ decoder_kwargs_i = dict(decoder_kwargs_i)
1362
+ if decoder_kwargs_i["module_type"] == "PatchedPretransform":
1363
+ self.decoder.append(MossAudioTokenizerPatchedPretransform(**decoder_kwargs_i, is_downsample=False))
1364
+ elif decoder_kwargs_i["module_type"] == "Transformer":
1365
+ self.decoder.append(
1366
+ MossAudioTokenizerProjectedTransformer(
1367
+ **decoder_kwargs_i,
1368
+ context=int(current_frame_rate * self.causal_transformer_context_duration),
1369
+ )
1370
+ )
1371
+ current_frame_rate *= self.decoder[-1].downsample_ratio
1372
+
1373
+ self.post_init()
1374
+
1375
+ def _start_streaming(self, batch_size: int):
1376
+ """Start streaming mode for all modules."""
1377
+
1378
+ def _start(module):
1379
+ if isinstance(module, StreamingModule):
1380
+ module._streaming_state = module._init_streaming_state(batch_size)
1381
+
1382
+ self.apply(_start)
1383
+
1384
+ def _stop_streaming(self):
1385
+ """Stop streaming mode for all modules."""
1386
+
1387
+ def _stop(module):
1388
+ if isinstance(module, StreamingModule):
1389
+ module._streaming_state = None
1390
+
1391
+ self.apply(_stop)
1392
+
1393
+ @contextmanager
1394
+ def streaming(self, batch_size: int = 1):
1395
+ """Context manager for streaming mode."""
1396
+ self._start_streaming(batch_size)
1397
+ try:
1398
+ yield
1399
+ finally:
1400
+ self._stop_streaming()
1401
+
1402
+ @torch.no_grad()
1403
+ def batch_encode(
1404
+ self, wav_list: list[torch.Tensor], num_quantizers: int | None = None
1405
+ ) -> MossAudioTokenizerEncoderOutput:
1406
+ """Batch encode a list of audio waveforms.
1407
+
1408
+ Args:
1409
+ wav_list: List of audio tensors, each of shape `(num_samples,)`.
1410
+ num_quantizers: Number of quantizers to use. By default, all quantizers are used.
1411
+
1412
+ Returns:
1413
+ [`MossAudioTokenizerEncoderOutput`] with `audio_codes` and `audio_codes_lengths`.
1414
+ """
1415
+ if len(wav_list) == 0:
1416
+ raise ValueError("`wav_list` must contain at least one waveform.")
1417
+
1418
+ device = wav_list[0].device
1419
+ batch_size = len(wav_list)
1420
+
1421
+ max_length = max(wav.shape[-1] for wav in wav_list)
1422
+ input_values = torch.zeros(batch_size, 1, max_length, device=device)
1423
+ input_lengths = torch.zeros(batch_size, device=device, dtype=torch.long)
1424
+
1425
+ for i, wav in enumerate(wav_list):
1426
+ input_values[i, 0, : wav.shape[-1]] = wav
1427
+ input_lengths[i] = wav.shape[-1]
1428
+
1429
+ return self._encode_frame(input_values, input_lengths, n_quantizers=num_quantizers)
1430
+
1431
+ @torch.no_grad()
1432
+ def batch_decode(
1433
+ self, codes_list: list[torch.Tensor], num_quantizers: int | None = None
1434
+ ) -> MossAudioTokenizerDecoderOutput:
1435
+ """Batch decode a list of audio codes.
1436
+
1437
+ Args:
1438
+ codes_list: List of audio code tensors, each of shape `(num_quantizers, codes_length)`.
1439
+ num_quantizers: If provided, decode only the first `num_quantizers` quantizers from each element in
1440
+ `codes_list`. If omitted, all elements in `codes_list` must have the same number of quantizers.
1441
+
1442
+ Returns:
1443
+ [`MossAudioTokenizerDecoderOutput`] with `audio` and `audio_lengths`.
1444
+ """
1445
+ if len(codes_list) == 0:
1446
+ raise ValueError("`codes_list` must contain at least one code tensor.")
1447
+
1448
+ batch_size = len(codes_list)
1449
+ device = codes_list[0].device
1450
+ nqs = [codes.shape[0] for codes in codes_list]
1451
+ if num_quantizers is None:
1452
+ num_quantizers = nqs[0]
1453
+ if any(nq != num_quantizers for nq in nqs):
1454
+ raise ValueError(
1455
+ "All elements in `codes_list` must have the same number of quantizers when `num_quantizers` is None. "
1456
+ "Pass `num_quantizers=...` to decode a common prefix."
1457
+ )
1458
+ else:
1459
+ min_nq = min(nqs)
1460
+ if min_nq < num_quantizers:
1461
+ raise ValueError(
1462
+ "`num_quantizers` must be <= the number of quantizers for every element in `codes_list`. "
1463
+ f"Got num_quantizers={num_quantizers}, min(codes.shape[0])={min_nq}."
1464
+ )
1465
+ max_length = max(codes.shape[-1] for codes in codes_list)
1466
+
1467
+ audio_codes = torch.zeros(num_quantizers, batch_size, max_length, device=device, dtype=torch.long)
1468
+ audio_codes_lengths = torch.zeros(batch_size, device=device, dtype=torch.long)
1469
+
1470
+ for i, codes in enumerate(codes_list):
1471
+ codes = codes[:num_quantizers]
1472
+ audio_codes[:, i, : codes.shape[-1]] = codes
1473
+ audio_codes_lengths[i] = codes.shape[-1]
1474
+
1475
+ return self._decode_frame(audio_codes, audio_codes_lengths)
1476
+
1477
+ @torch.no_grad()
1478
+ def _encode_frame(
1479
+ self,
1480
+ input_values: torch.Tensor,
1481
+ input_lengths: torch.Tensor | None = None,
1482
+ n_quantizers: int | None = None,
1483
+ ) -> MossAudioTokenizerEncoderOutput:
1484
+ """Tokenize audio waveform into discrete tokens."""
1485
+ # Handle input shape
1486
+ if input_values.dim() == 2:
1487
+ input_values = input_values.unsqueeze(1)
1488
+
1489
+ B, _, T = input_values.shape
1490
+ device = input_values.device
1491
+
1492
+ if input_lengths is None:
1493
+ input_lengths = torch.full((B,), T, device=device, dtype=torch.long)
1494
+
1495
+ # Pad to multiple of downsample_rate
1496
+ if T % self.downsample_rate != 0:
1497
+ pad_length = self.downsample_rate - (T % self.downsample_rate)
1498
+ input_values = F.pad(input_values, (0, pad_length))
1499
+
1500
+ # Encode
1501
+ e, e_lengths = input_values, input_lengths
1502
+ for encoder_module in self.encoder:
1503
+ e, e_lengths = encoder_module(e, e_lengths)
1504
+
1505
+ # Quantize
1506
+ quantizer = cast(MossAudioTokenizerResidualVQ | MossAudioTokenizerResidualLFQ, self.quantizer)
1507
+ zq, audio_codes, audio_codes_lengths = quantizer(e, e_lengths, n_quantizers)
1508
+
1509
+ return MossAudioTokenizerEncoderOutput(
1510
+ audio_codes=audio_codes, audio_codes_lengths=audio_codes_lengths, encoder_hidden_states=e
1511
+ )
1512
+
1513
+ @torch.no_grad()
1514
+ def _decode_frame(
1515
+ self,
1516
+ codes: torch.Tensor,
1517
+ codes_lengths: torch.Tensor | None = None,
1518
+ ) -> MossAudioTokenizerDecoderOutput:
1519
+ """Detokenize discrete tokens into audio waveform."""
1520
+ nq, B, T = codes.shape
1521
+ device = codes.device
1522
+
1523
+ if codes_lengths is None:
1524
+ codes_lengths = torch.full((B,), T, device=device, dtype=torch.long)
1525
+
1526
+ # Decode from codes
1527
+ quantizer = cast(MossAudioTokenizerResidualVQ | MossAudioTokenizerResidualLFQ, self.quantizer)
1528
+ zq = quantizer.decode_codes(codes)
1529
+
1530
+ d, d_lengths = zq, codes_lengths
1531
+ for decoder_module in self.decoder:
1532
+ d, d_lengths = decoder_module(d, d_lengths)
1533
+
1534
+ return MossAudioTokenizerDecoderOutput(audio=d, audio_lengths=d_lengths)
1535
+
1536
+ def encode( # type: ignore[override]
1537
+ self,
1538
+ input_values: torch.Tensor,
1539
+ padding_mask: torch.Tensor | None = None,
1540
+ num_quantizers: int | None = None,
1541
+ return_dict: bool | None = None,
1542
+ chunk_duration: float | None = None,
1543
+ ):
1544
+ """
1545
+ Encodes the input audio waveform into discrete codes.
1546
+
1547
+ Args:
1548
+ input_values (`torch.Tensor` of shape `(batch_size, channels, sequence_length)`):
1549
+ Float values of the input audio waveform.
1550
+ padding_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
1551
+ Mask to indicate valid audio samples.
1552
+ num_quantizers (`int`, *optional*):
1553
+ Number of quantizers to use. By default, all quantizers are used.
1554
+ return_dict (`bool`, *optional*):
1555
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
1556
+ chunk_duration (`float`, *optional*):
1557
+ If provided, encode the input waveform in successive chunks of `chunk_duration` seconds while keeping a
1558
+ streaming KV cache for the causal transformers.
1559
+
1560
+ `chunk_duration` must be <= `config.causal_transformer_context_duration`, and
1561
+ `chunk_duration * config.sampling_rate` must be divisible by `config.downsample_rate`.
1562
+
1563
+ Returns:
1564
+ `MossAudioTokenizerEncoderOutput` or tuple containing audio codes and lengths.
1565
+ """
1566
+ return_dict = return_dict if return_dict is not None else self.config.return_dict
1567
+
1568
+ # Handle input shape
1569
+ if input_values.dim() == 2:
1570
+ input_values = input_values.unsqueeze(1)
1571
+
1572
+ B, _, T = input_values.shape
1573
+ device = input_values.device
1574
+
1575
+ if padding_mask is not None:
1576
+ input_lengths = padding_mask.sum(dim=-1).long()
1577
+ else:
1578
+ input_lengths = torch.full((B,), T, device=device, dtype=torch.long)
1579
+
1580
+ if chunk_duration is None:
1581
+ encoder_output = self._encode_frame(input_values, input_lengths, num_quantizers)
1582
+ else:
1583
+ if chunk_duration <= 0:
1584
+ raise ValueError("`chunk_duration` must be > 0 when provided.")
1585
+ if chunk_duration > self.causal_transformer_context_duration:
1586
+ raise ValueError(
1587
+ "`chunk_duration` must be <= `config.causal_transformer_context_duration` "
1588
+ f"({self.causal_transformer_context_duration}), got {chunk_duration}."
1589
+ )
1590
+ if B != 1:
1591
+ raise ValueError("Streaming encode via `chunk_duration` currently only supports batch_size=1.")
1592
+
1593
+ chunk_length = int(round(chunk_duration * self.sampling_rate))
1594
+ if chunk_length <= 0:
1595
+ raise ValueError("`chunk_duration` is too small and results in chunk_length <= 0.")
1596
+ if chunk_length % self.downsample_rate != 0:
1597
+ raise ValueError(
1598
+ "`chunk_duration * config.sampling_rate` must be divisible by `config.downsample_rate`. "
1599
+ f"Got chunk_length={chunk_length}, downsample_rate={self.downsample_rate}."
1600
+ )
1601
+
1602
+ input_length = int(input_lengths[0].item())
1603
+ if input_length <= chunk_length:
1604
+ encoder_output = self._encode_frame(input_values[..., :input_length], input_lengths, num_quantizers)
1605
+ else:
1606
+ codes_chunks: list[torch.Tensor] = []
1607
+ hidden_chunks: list[torch.Tensor] = []
1608
+
1609
+ with ExitStack() as exit_stack:
1610
+ for encoder_module in self.encoder:
1611
+ if isinstance(encoder_module, StreamingModule):
1612
+ exit_stack.enter_context(encoder_module.streaming(batch_size=B))
1613
+
1614
+ for start_idx in range(0, input_length, chunk_length):
1615
+ input_length_i = min(chunk_length, input_length - start_idx)
1616
+ if input_length_i <= 0:
1617
+ break
1618
+
1619
+ input_lengths_i = torch.tensor([input_length_i], device=device, dtype=torch.long)
1620
+ input_values_i = input_values[..., start_idx : start_idx + input_length_i]
1621
+ result_i = self._encode_frame(input_values_i, input_lengths_i, num_quantizers)
1622
+
1623
+ if result_i.audio_codes is None or result_i.audio_codes_lengths is None:
1624
+ raise RuntimeError("Internal error: `_encode_frame` returned empty audio codes.")
1625
+ if result_i.encoder_hidden_states is None:
1626
+ raise RuntimeError("Internal error: `_encode_frame` returned empty encoder hidden states.")
1627
+
1628
+ codes_length_i = result_i.audio_codes_lengths
1629
+ codes_chunks.append(result_i.audio_codes[:, :, : codes_length_i[0]])
1630
+ hidden_chunks.append(result_i.encoder_hidden_states[:, :, : codes_length_i[0]])
1631
+
1632
+ audio_codes = torch.cat(codes_chunks, dim=-1)
1633
+ encoder_hidden_states = torch.cat(hidden_chunks, dim=-1)
1634
+ audio_codes_lengths = torch.tensor([audio_codes.shape[-1]], device=device, dtype=torch.long)
1635
+ encoder_output = MossAudioTokenizerEncoderOutput(
1636
+ audio_codes=audio_codes,
1637
+ audio_codes_lengths=audio_codes_lengths,
1638
+ encoder_hidden_states=encoder_hidden_states,
1639
+ )
1640
+
1641
+ if not return_dict:
1642
+ assert encoder_output.audio_codes is not None
1643
+ assert encoder_output.audio_codes_lengths is not None
1644
+ return (
1645
+ cast(torch.Tensor, encoder_output.audio_codes),
1646
+ cast(torch.Tensor, encoder_output.audio_codes_lengths),
1647
+ )
1648
+ return encoder_output
1649
+
1650
+ def decode( # type: ignore[override]
1651
+ self,
1652
+ audio_codes: torch.Tensor,
1653
+ padding_mask: torch.Tensor | None = None,
1654
+ return_dict: bool | None = None,
1655
+ chunk_duration: float | None = None,
1656
+ num_quantizers: int | None = None,
1657
+ ):
1658
+ """
1659
+ Decodes the given codes into an output audio waveform.
1660
+
1661
+ Args:
1662
+ audio_codes (`torch.LongTensor` of shape `(num_quantizers, batch_size, sequence_length)`):
1663
+ Discrete code embeddings computed using `model.encode`.
1664
+ padding_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*):
1665
+ Mask to indicate valid code positions.
1666
+ return_dict (`bool`, *optional*):
1667
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
1668
+ chunk_duration (`float`, *optional*):
1669
+ If provided, decode the input codes in successive chunks of `chunk_duration` seconds while keeping a
1670
+ streaming KV cache for the causal transformers.
1671
+
1672
+ num_quantizers (`int`, *optional*):
1673
+ Number of quantizers to use. By default, all quantizers in `audio_codes` are used.
1674
+
1675
+ `chunk_duration` must be <= `config.causal_transformer_context_duration`, and
1676
+ `chunk_duration * config.sampling_rate` must be divisible by `config.downsample_rate`.
1677
+
1678
+ Returns:
1679
+ `MossAudioTokenizerDecoderOutput` or tuple containing decoded audio.
1680
+ """
1681
+ return_dict = return_dict if return_dict is not None else self.config.return_dict
1682
+
1683
+ if audio_codes.dim() == 2:
1684
+ audio_codes = audio_codes.unsqueeze(1) # nq, T -> nq, B=1, T
1685
+
1686
+ if num_quantizers is not None:
1687
+ if num_quantizers > audio_codes.shape[0]:
1688
+ raise ValueError(
1689
+ f"`num_quantizers` ({num_quantizers}) must be <= audio_codes.shape[0] ({audio_codes.shape[0]})."
1690
+ )
1691
+ audio_codes = audio_codes[:num_quantizers]
1692
+
1693
+ _, B, T = audio_codes.shape
1694
+ device = audio_codes.device
1695
+
1696
+ if padding_mask is not None:
1697
+ codes_lengths = padding_mask.sum(dim=-1).long()
1698
+ else:
1699
+ codes_lengths = torch.full((B,), T, device=device, dtype=torch.long)
1700
+
1701
+ if chunk_duration is None:
1702
+ decoder_output = self._decode_frame(audio_codes, codes_lengths)
1703
+ else:
1704
+ if chunk_duration <= 0:
1705
+ raise ValueError("`chunk_duration` must be > 0 when provided.")
1706
+ if chunk_duration > self.causal_transformer_context_duration:
1707
+ raise ValueError(
1708
+ "`chunk_duration` must be <= `config.causal_transformer_context_duration` "
1709
+ f"({self.causal_transformer_context_duration}), got {chunk_duration}."
1710
+ )
1711
+ if B != 1:
1712
+ raise ValueError("Streaming decode via `chunk_duration` currently only supports batch_size=1.")
1713
+
1714
+ chunk_length = int(round(chunk_duration * self.sampling_rate))
1715
+ if chunk_length <= 0:
1716
+ raise ValueError("`chunk_duration` is too small and results in chunk_length <= 0.")
1717
+ if chunk_length % self.downsample_rate != 0:
1718
+ raise ValueError(
1719
+ "`chunk_duration * config.sampling_rate` must be divisible by `config.downsample_rate`. "
1720
+ f"Got chunk_length={chunk_length}, downsample_rate={self.downsample_rate}."
1721
+ )
1722
+
1723
+ chunk_frame_length = chunk_length // self.downsample_rate
1724
+ codes_length = int(codes_lengths[0].item())
1725
+ if codes_length <= chunk_frame_length:
1726
+ decoder_output = self._decode_frame(audio_codes[..., :codes_length], codes_lengths)
1727
+ else:
1728
+ wav_chunks: list[torch.Tensor] = []
1729
+ with ExitStack() as exit_stack:
1730
+ for decoder_module in self.decoder:
1731
+ if isinstance(decoder_module, StreamingModule):
1732
+ exit_stack.enter_context(decoder_module.streaming(batch_size=B))
1733
+
1734
+ for start_idx in range(0, codes_length, chunk_frame_length):
1735
+ codes_length_i = min(chunk_frame_length, codes_length - start_idx)
1736
+ if codes_length_i <= 0:
1737
+ break
1738
+
1739
+ codes_lengths_i = torch.tensor([codes_length_i], device=device, dtype=torch.long)
1740
+ codes_i = audio_codes[:, :, start_idx : start_idx + codes_length_i]
1741
+ result_i = self._decode_frame(codes_i, codes_lengths_i)
1742
+
1743
+ if result_i.audio is None or result_i.audio_lengths is None:
1744
+ raise RuntimeError("Internal error: `_decode_frame` returned empty audio.")
1745
+
1746
+ wav_chunks.append(result_i.audio[:, :, : result_i.audio_lengths[0]])
1747
+
1748
+ wav = torch.cat(wav_chunks, dim=-1)
1749
+ audio_lengths = torch.tensor([wav.shape[-1]], device=device, dtype=torch.long)
1750
+ decoder_output = MossAudioTokenizerDecoderOutput(audio=wav, audio_lengths=audio_lengths)
1751
+
1752
+ if not return_dict:
1753
+ assert decoder_output.audio is not None
1754
+ return (cast(torch.Tensor, decoder_output.audio),)
1755
+ return decoder_output
1756
+
1757
+ @auto_docstring
1758
+ def forward(
1759
+ self,
1760
+ input_values: torch.FloatTensor | None = None,
1761
+ padding_mask: torch.BoolTensor | None = None,
1762
+ audio_codes: torch.Tensor | None = None,
1763
+ num_quantizers: int | None = None,
1764
+ return_dict: bool | None = None,
1765
+ ) -> tuple[torch.Tensor | None, torch.Tensor | None, torch.Tensor | None] | MossAudioTokenizerOutput: # type: ignore[override]
1766
+ r"""
1767
+ input_values (`torch.FloatTensor` of shape `(batch_size, channels, sequence_length)`, *optional*):
1768
+ Raw audio input converted to Float.
1769
+ padding_mask (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*):
1770
+ Mask to avoid computing on padding token indices. Mask values selected in `[0, 1]`:
1771
+ - 1 for tokens that are **not masked**,
1772
+ - 0 for tokens that are **masked**.
1773
+ audio_codes (`torch.LongTensor` of shape `(num_quantizers, batch_size, sequence_length)`, *optional*):
1774
+ Discrete code embeddings computed using `model.encode`.
1775
+ num_quantizers (`int`, *optional*):
1776
+ Number of quantizers (codebooks) to use. By default, all quantizers are used.
1777
+ return_dict (`bool`, *optional*):
1778
+ Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple.
1779
+
1780
+ Examples:
1781
+
1782
+ ```python
1783
+ >>> import torch
1784
+ >>> from transformers import MossAudioTokenizerModel
1785
+
1786
+ >>> model = MossAudioTokenizerModel.from_pretrained("moss_audio_tokenizer-model")
1787
+
1788
+ >>> # Create dummy audio input
1789
+ >>> audio = torch.randn(1, 1, 24000) # 1 second of audio at 24kHz
1790
+
1791
+ >>> outputs = model(input_values=audio)
1792
+ >>> audio_codes = outputs.audio_codes
1793
+ >>> audio_values = outputs.audio
1794
+ ```
1795
+ """
1796
+ return_dict = return_dict if return_dict is not None else self.config.return_dict
1797
+
1798
+ output_audio_codes: torch.Tensor | None = None
1799
+ output_audio_codes_lengths: torch.Tensor | None = None
1800
+ output_audio: torch.Tensor | None = None
1801
+ output_audio_lengths: torch.Tensor | None = None
1802
+ decoded_from_encoded_codes = False
1803
+
1804
+ # Encode if input_values provided
1805
+ if input_values is not None:
1806
+ encoder_output = self.encode(input_values, padding_mask, num_quantizers, return_dict=True)
1807
+ encoder_output = cast(MossAudioTokenizerEncoderOutput, encoder_output)
1808
+ output_audio_codes = encoder_output.audio_codes
1809
+ output_audio_codes_lengths = encoder_output.audio_codes_lengths
1810
+
1811
+ # If codes not provided separately, use encoded codes for decoding
1812
+ if audio_codes is None:
1813
+ audio_codes = output_audio_codes
1814
+ decoded_from_encoded_codes = True
1815
+
1816
+ # Decode if codes available
1817
+ if audio_codes is not None:
1818
+ # If we're decoding the codes we just produced, use the computed lengths so we don't decode padded garbage.
1819
+ if decoded_from_encoded_codes and output_audio_codes_lengths is not None:
1820
+ decoder_output = self._decode_frame(audio_codes, output_audio_codes_lengths)
1821
+ else:
1822
+ decoder_output = self.decode(
1823
+ audio_codes,
1824
+ padding_mask=padding_mask,
1825
+ return_dict=True,
1826
+ num_quantizers=num_quantizers,
1827
+ )
1828
+ decoder_output = cast(MossAudioTokenizerDecoderOutput, decoder_output)
1829
+ output_audio = decoder_output.audio
1830
+ output_audio_lengths = decoder_output.audio_lengths
1831
+
1832
+ if not return_dict:
1833
+ return (output_audio_codes, output_audio, output_audio_lengths)
1834
+
1835
+ return MossAudioTokenizerOutput(
1836
+ audio=output_audio,
1837
+ audio_lengths=output_audio_lengths,
1838
+ audio_codes=output_audio_codes,
1839
+ audio_codes_lengths=output_audio_codes_lengths,
1840
+ )
1841
+
1842
+
1843
+ __all__ = ["MossAudioTokenizerModel", "MossAudioTokenizerPreTrainedModel"]