ManyOtherFunctions commited on
Commit
f5f0c8b
·
verified ·
1 Parent(s): db93a21

Upload 14 files

Browse files
README.md CHANGED
@@ -1,99 +1,3 @@
1
- ![Extra](O_O.png)
2
-
3
- # What This Is:
4
- Just a humble wannabe semantic tokenizer to make the semantic tokens to train the Yue2 music model with
5
-
6
- # Q/A:
7
- Q: But.. wasn't there already another one?
8
-
9
- A: Yes. But the dataset used to train it wasn't very robust. Many song segments devolved into noise.
10
-
11
- Q: How good is this one then? Can it tokenize any song then?
12
-
13
- A: More or less
14
-
15
- Q: .. what do you mean more or less?
16
-
17
- A: Yknow how if you don't have the real Nine Yin Manual, you have to practice the fake Nine Yin Manual?
18
- This is the fake Nine Yin Manual. It gets alot more right. It gets the overall song right.
19
- There are some slight hiccups with consonants occasionally. But it is our hope that that is something
20
- the extra text conditioning on the NAR diffusion stage will change.
21
-
22
- Q: What about the joint nar lora or all that from mothersuperior, do I use th-
23
-
24
- A: Thou shalt have no other tokenizers before me (hahahaa) ... But no, don't do that.
25
- This tokenizer will sound mostly right with the base model NAR stage. WILD, amirite? But that's how you make sure the AR model
26
- won't shit the bed trying to learn your token sequences that may or may not describe production grade music.
27
- You don't HAVE to train on production music. Sometimes you just want an amateur grade school orchestra to troll with.
28
- But yeah no we don't need no stinkin NAR lora. It's counterproductive. How would you know if Yue2 can even learn it or not
29
- if you can't even hear what the NAR stage normally sounds like? NAR loras are for the final polish stage.
30
-
31
-
32
- # How to actually train Yue2:
33
- The first step is obviously build a good foundation. The official Yu2 tokenizer would be that good foundation.
34
- But if you don't have the real Nine Yin, practice the fake Nine Yin.
35
-
36
- Next is the AR or autoregressive LLM part of Yue2. This is why having accurate semantic tokens matters so much.
37
- The tokens carry ALL the meaning.
38
- What sounds to say to make up lyrics.
39
- What those vocals sound like.
40
- That part takes your lyrics, your text prompt, and most importantly, sheet music melody, and spits out these semantic tokens here
41
- These semantic tokens therefore have to be correct
42
-
43
- SO! It means you have to use this tokenizer and listen to the reconstructed audio
44
- And if it fails to remake your song, then you can't train it.
45
- Simply that. Try to train and you'll destroy the model and give it brain damage.
46
- Most of those people who look at us like we're crazy for doin all this extra shit,
47
- well guess what? They like music the model was already good at and the tokenizer built on limited data already knew
48
-
49
- But what if you wanted to make a chinese version of "I Want It That Way"
50
- What if you wanted it to be Dutch? (yes this model knows Dutch. Crazy, I know.)
51
- What if you wanted backing vocals? What if you didnt wanna sound like AI slop, but wanted to actually fool people
52
- into thinking a cracked band made an awesome banger?
53
- With the old tokenizer you'd be shit out of luck.
54
- With this tokenizer... you have a better chance. No guarantees.
55
-
56
-
57
-
58
- Lastly we come to the NAR, the diffusion stage! That stage turns these tokens here,
59
- which describe exactly what instruments, what they sound like,
60
- what the vocals sound like exactly, that stage turns these tokens into audio basically.
61
- (I mean not really, but as far as alot of you are concerned, that's good enough of an understanding)
62
- The NAR / diffusion stage takes these IDs and it also takes the text prompt.
63
-
64
- HOWEVER!!!
65
- We tried throwing in all kindsa trash and garbage instructions into the captions for the NAR stage
66
- IT DID NOT CHANGE THE SONG AT ALL.
67
- the NAR ONLY LISTENS TO THESE SEMANTIC TOKENS.
68
- The only thing that might have changed a little bit was MAYBE,
69
- JUST MAYBE, supplying the right lyrics improved pronunciation. Maybe. Or I could be trippin balls and coping hard.
70
-
71
-
72
- # So which part of Yue2 Do I train?!!:
73
- If you want a little extra polish on your sound, train the NAR stage.
74
-
75
- Most of you want more than that. You want your own singers. You want a sick arrangement without having to prompt for one.
76
- You want it to make banger melodies so you don't have to compose songs by hand.
77
-
78
- That's all the AR, but in two stages. ONE of these stages is unaffected by a dogshit wannabe semantic tokenizer.
79
-
80
- If you only need it to make good melodies, and nothing else, YOU ARE IN LUCK!
81
- Freeze the losses on everything BUT the sheetABC part of the AR stage and just train based on melodies.
82
- It will at least make banger melodies.
83
-
84
- If you wanna train everything else, that's where this tokenizer comes into play.
85
- You wanna train the melodies first. Then on the main AR stage, freeze the losses on the ABC section
86
- and only compute losses on the tokens. (Ask your friendly AI assistant for help with that)
87
- Now when it comes to the generating semantic tokens stage
88
- You caption your song and everything, maybe even supply a corrected sheetsage ABC melody file. And then you supply
89
- these semantic tokens.
90
-
91
- The AR has to output the tokens that make the audio.
92
- This tokenizer is a sanity check for that. It will generate and reconstruct.
93
- If you don't hear something that sounds enough like what you want, you're not gonna get what you want with training
94
- (And in fact I can't guarantee this won't blow up the AR after alotta steps either, but it should help somewhat)
95
-
96
-
97
  # WannabeTokenizer inference
98
 
99
  A self-contained, offline inference package for:
@@ -134,11 +38,25 @@ Existing outputs are protected unless `--overwrite` is supplied. Common input fo
134
  Optional settings:
135
 
136
  ```bash
137
- python run_tokenizer.py input.flac --tokenizer checkpoints/WannabeTokenizer.safetensors --yue2-checkpoint /path/to/yue2_3b_bf16.safetensors --output-dir output --device cuda:0 --seed 1 --steps 32
138
  ```
139
 
140
  Use `python run_tokenizer.py --help` for all options. If installed as a package, the equivalent entry point is `wannabe-tokenize`.
141
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
142
  ## Behavior and resource use
143
 
144
  Reconstruction is generative and seed-dependent, not a lossless inversion of the recording. It uses YuE2's base NAR with an empty tags/lyrics prompt. Different seeds produce slightly different audio from the same tokens.
@@ -153,11 +71,24 @@ from wannabe_tokenizer.audio import load_audio, save_wav_atomic
153
 
154
  waveform = load_audio("input.wav")
155
  tokenizer = WannabeTokenizer("checkpoints/WannabeTokenizer.safetensors", device="cuda")
156
- tokens = tokenizer.tokenize(waveform)
157
 
158
  del tokenizer # release encoder memory before loading YuE2 in a real application
159
  renderer = YuE2Renderer("/path/to/yue2_3b_bf16.safetensors", device="cuda")
160
  reconstructed = renderer.reconstruct(tokens, seed=0, steps=32)
161
  save_wav_atomic("reconstructed.wav", reconstructed)
162
  ```
163
- ![Extra](O_O.png)
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  # WannabeTokenizer inference
2
 
3
  A self-contained, offline inference package for:
 
38
  Optional settings:
39
 
40
  ```bash
41
+ python run_tokenizer.py input.flac --tokenizer checkpoints/WannabeTokenizer.safetensors --yue2-checkpoint /path/to/yue2_3b_bf16.safetensors --output-dir output --device cuda:0 --seed 1 --steps 32 --level all
42
  ```
43
 
44
  Use `python run_tokenizer.py --help` for all options. If installed as a package, the equivalent entry point is `wannabe-tokenize`.
45
 
46
+ ## Loudness
47
+
48
+ The tokenizer writes the input's loudness into the IDs, and YuE2 renders whatever level the IDs carry. YuE2's own songs sit in a fairly narrow loudness range, so by default a song outside that range is brought into it before it is tokenized: one gain for the whole song from its gated RMS, with a peak limiter if the song was raised. Per-section gains are never used, so the song keeps its dynamics.
49
+
50
+ `--level` chooses this:
51
+
52
+ - `band` (default): a song inside YuE2's usual range is left alone, and one outside it is moved only as far as the nearest edge. The checkpoint names its range (metadata `head.level_band`); one that names none uses -21 to -13 dBFS (a -25 dBFS song to -21, a -9 dBFS master to -13). This is the smallest change that puts a song in the range YuE2's own songs cover, and how ai-toolkit reads songs by default.
53
+ - `all`: every song is moved to -16.5 dBFS (about -14 LUFS), the level YuE2's own covers of real songs sit at, whatever the original master's.
54
+ - `asis`: the song is read as recorded. Use it for audio YuE2 generated itself, which is already at its own level.
55
+
56
+ The run prints the gain it applied.
57
+
58
+ Decode each song from the same file every time. The IDs also depend on where the 40-ms frame grid falls in the audio, so two encodes of one recording (an MP3 and an M4A download, say) can give noticeably different IDs.
59
+
60
  ## Behavior and resource use
61
 
62
  Reconstruction is generative and seed-dependent, not a lossless inversion of the recording. It uses YuE2's base NAR with an empty tags/lyrics prompt. Different seeds produce slightly different audio from the same tokens.
 
71
 
72
  waveform = load_audio("input.wav")
73
  tokenizer = WannabeTokenizer("checkpoints/WannabeTokenizer.safetensors", device="cuda")
74
+ tokens, gain_db = tokenizer.tokenize(waveform, level="band", return_gain=True)
75
 
76
  del tokenizer # release encoder memory before loading YuE2 in a real application
77
  renderer = YuE2Renderer("/path/to/yue2_3b_bf16.safetensors", device="cuda")
78
  reconstructed = renderer.reconstruct(tokens, seed=0, steps=32)
79
  save_wav_atomic("reconstructed.wav", reconstructed)
80
  ```
81
+
82
+ ## Tests
83
+
84
+ The lightweight tests do not load model weights:
85
+
86
+ ```bash
87
+ python -m unittest discover -s tests
88
+ ```
89
+
90
+ An end-to-end run requires both checkpoints and is the recommended numerical smoke test after installation.
91
+
92
+ ## Third-party models
93
+
94
+ Model weights are subject to their respective upstream licenses. The YuE2 checkpoint identifies its license as CC-BY-NC-4.0. Review the applicable model and dependency licenses before redistribution or commercial use.
Run_WannabeTokenizer.ipynb CHANGED
@@ -1,85 +1,69 @@
1
  {
2
- "cells": [
3
- {
4
- "cell_type": "markdown",
5
- "metadata": {},
6
- "source": [
7
- "# Run WannabeTokenizer\n",
8
- "\n",
9
- "Set the three paths below, then run the cells in order. Start Jupyter from this package folder so the relative checkpoint path works."
10
- ]
11
- },
12
- {
13
- "cell_type": "code",
14
- "execution_count": null,
15
- "metadata": {},
16
- "outputs": [],
17
- "source": [
18
- "from pathlib import Path\n",
19
- "import subprocess\n",
20
- "import sys\n",
21
- "\n",
22
- "PACKAGE_DIR = Path.cwd()\n",
23
- "TOKENIZER = PACKAGE_DIR / 'checkpoints' / 'WannabeTokenizer.safetensors'\n",
24
- "YUE2_CHECKPOINT = Path('/path/to/yue2_3b_bf16.safetensors')\n",
25
- "OUTPUT_DIR = PACKAGE_DIR / 'output'\n",
26
- "\n",
27
- "DEVICE = 'auto' # 'auto', 'cuda', 'cuda:0', or 'cpu'\n",
28
- "SEED = 4 # Shout out to eMuhdra\n",
29
- "STEPS = 32\n",
30
- "OVERWRITE = False"
31
- ]
32
- },
33
- {
34
- "cell_type": "code",
35
- "execution_count": null,
36
- "metadata": {},
37
- "outputs": [],
38
- "source": [
39
- "INPUT_AUDIO = Path('/path/to/input.wav')\n",
40
- "command = [\n",
41
- " sys.executable,\n",
42
- " str(PACKAGE_DIR / 'run_tokenizer.py'),\n",
43
- " str(INPUT_AUDIO),\n",
44
- " '--tokenizer', str(TOKENIZER),\n",
45
- " '--yue2-checkpoint', str(YUE2_CHECKPOINT),\n",
46
- " '--output-dir', str(OUTPUT_DIR),\n",
47
- " '--device', DEVICE,\n",
48
- " '--seed', str(SEED),\n",
49
- " '--steps', str(STEPS),\n",
50
- "]\n",
51
- "if OVERWRITE:\n",
52
- " command.append('--overwrite')\n",
53
- "\n",
54
- "subprocess.run(command, cwd=PACKAGE_DIR, check=True)"
55
- ]
56
- },
57
- {
58
- "cell_type": "code",
59
- "execution_count": null,
60
- "metadata": {},
61
- "outputs": [],
62
- "source": [
63
- "import numpy as np\n",
64
- "from IPython.display import Audio, display\n",
65
- "\n",
66
- "tokens = np.load(OUTPUT_DIR / 'tokens.npy')\n",
67
- "print(f'{len(tokens)} tokens at 25 Hz; dtype={tokens.dtype}; range={tokens.min()}..{tokens.max()}')\n",
68
- "display(Audio(filename=str(OUTPUT_DIR / 'reconstructed.wav')))"
69
- ]
70
- }
71
- ],
72
- "metadata": {
73
- "kernelspec": {
74
- "display_name": "Python 3",
75
- "language": "python",
76
- "name": "python3"
77
- },
78
- "language_info": {
79
- "name": "python",
80
- "version": "3.10"
81
- }
82
  },
83
- "nbformat": 4,
84
- "nbformat_minor": 5
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
85
  }
 
1
  {
2
+ "cells": [
3
+ {
4
+ "cell_type": "markdown",
5
+ "metadata": {},
6
+ "source": [
7
+ "# Run WannabeTokenizer\n",
8
+ "\n",
9
+ "Set the three paths below, then run the cells in order. Start Jupyter from this package folder so the relative checkpoint path works."
10
+ ]
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
11
  },
12
+ {
13
+ "cell_type": "code",
14
+ "execution_count": null,
15
+ "metadata": {},
16
+ "outputs": [],
17
+ "source": [
18
+ "from pathlib import Path\n",
19
+ "import subprocess\n",
20
+ "import sys\n",
21
+ "\n",
22
+ "PACKAGE_DIR = Path.cwd()\n",
23
+ "TOKENIZER = PACKAGE_DIR / 'checkpoints' / 'WannabeTokenizer.safetensors'\n",
24
+ "YUE2_CHECKPOINT = Path('/path/to/yue2_3b_bf16.safetensors')\n",
25
+ "OUTPUT_DIR = PACKAGE_DIR / 'output'\n",
26
+ "\n",
27
+ "DEVICE = 'auto' # 'auto', 'cuda', 'cuda:0', or 'cpu'\n",
28
+ "SEED = 4 # Shout out to eMuhdra\n",
29
+ "STEPS = 32\n",
30
+ "LEVEL = 'band' # 'band' = only a song outside YuE2's range moves, to its nearest edge | 'all' = every song to YuE2's loudness | 'asis' = as recorded (YuE2's own audio)\n",
31
+ "OVERWRITE = False"
32
+ ]
33
+ },
34
+ {
35
+ "cell_type": "code",
36
+ "execution_count": null,
37
+ "metadata": {},
38
+ "outputs": [],
39
+ "source": "INPUT_AUDIO = Path('/path/to/input.wav')\ncommand = [\n sys.executable,\n str(PACKAGE_DIR / 'run_tokenizer.py'),\n str(INPUT_AUDIO),\n '--tokenizer', str(TOKENIZER),\n '--yue2-checkpoint', str(YUE2_CHECKPOINT),\n '--output-dir', str(OUTPUT_DIR),\n '--device', DEVICE,\n '--seed', str(SEED),\n '--steps', str(STEPS),\n '--level', LEVEL,\n]\nif OVERWRITE:\n command.append('--overwrite')\n\nsubprocess.run(command, cwd=PACKAGE_DIR, check=True)"
40
+ },
41
+ {
42
+ "cell_type": "code",
43
+ "execution_count": null,
44
+ "metadata": {},
45
+ "outputs": [],
46
+ "source": [
47
+ "import numpy as np\n",
48
+ "from IPython.display import Audio, display\n",
49
+ "\n",
50
+ "tokens = np.load(OUTPUT_DIR / 'tokens.npy')\n",
51
+ "print(f'{len(tokens)} tokens at 25 Hz; dtype={tokens.dtype}; range={tokens.min()}..{tokens.max()}')\n",
52
+ "display(Audio(filename=str(OUTPUT_DIR / 'reconstructed.wav')))"
53
+ ]
54
+ }
55
+ ],
56
+ "metadata": {
57
+ "kernelspec": {
58
+ "display_name": "Python 3",
59
+ "language": "python",
60
+ "name": "python3"
61
+ },
62
+ "language_info": {
63
+ "name": "python",
64
+ "version": "3.10"
65
+ }
66
+ },
67
+ "nbformat": 4,
68
+ "nbformat_minor": 5
69
  }
WannabeTokenizer.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:3a5d311aeca55d3dfccbbc2af950ea247777a1e2757b8ef4ed6ac888f492e232
3
+ size 1698038168
pyproject.toml CHANGED
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
4
 
5
  [project]
6
  name = "wannabe-tokenizer-inference"
7
- version = "1.0.0"
8
  description = "Audio-to-semantic-token inference and YuE2 reconstruction"
9
  readme = "README.md"
10
  requires-python = ">=3.10"
 
4
 
5
  [project]
6
  name = "wannabe-tokenizer-inference"
7
+ version = "1.1.0"
8
  description = "Audio-to-semantic-token inference and YuE2 reconstruction"
9
  readme = "README.md"
10
  requires-python = ">=3.10"
wannabe_tokenizer/__init__.py CHANGED
@@ -4,4 +4,4 @@ from .quantizer import WannabeTokenizer
4
  from .renderer import YuE2Renderer
5
 
6
  __all__ = ["WannabeTokenizer", "YuE2Renderer"]
7
- __version__ = "1.0.0"
 
4
  from .renderer import YuE2Renderer
5
 
6
  __all__ = ["WannabeTokenizer", "YuE2Renderer"]
7
+ __version__ = "1.1.0"
wannabe_tokenizer/audio.py CHANGED
@@ -1,14 +1,73 @@
1
- """Audio input and output for the 48 kHz stereo YuE2 pipeline."""
2
 
 
3
  import os
4
  from pathlib import Path
5
 
6
  import soundfile as sf
7
  import torch
 
8
 
9
  SAMPLE_RATE = 48000
10
  HOP = 1920
11
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
12
 
13
  def load_audio(path: str | Path) -> torch.Tensor:
14
  """Decode an audio file to fp32 [2, samples] at 48 kHz, trimmed to VAE hops."""
 
1
+ """Audio input and output for the 48 kHz stereo YuE2 pipeline, and the loudness fit applied before tokenizing."""
2
 
3
+ import math
4
  import os
5
  from pathlib import Path
6
 
7
  import soundfile as sf
8
  import torch
9
+ import torch.nn.functional as F
10
 
11
  SAMPLE_RATE = 48000
12
  HOP = 1920
13
 
14
+ # YuE2 sets a song's loudness by style: its own covers of real songs sit at one level whatever the master's (about -14 LUFS), and the
15
+ # tokenizer writes the input's loudness into the ids, which the NAR then renders. So a real song's loudness is fitted before it is
16
+ # tokenized: one gain for the whole song from its gated RMS (per-chunk gains would erase its dynamics), peaks limited if it was raised.
17
+ # The default, 'band', moves only a song outside YuE2's range, and only to its edge: a song already inside keeps its own level.
18
+ LEVEL_TARGET = -16.5 # dBFS, gated RMS of the tokenizer input (mono, 24 kHz), the level YuE2's own songs have
19
+ LEVEL_BAND = (-21.0, -13.0) # the range YuE2's own songs sit in: 'band' leaves a song inside it alone, moves one outside to its edge.
20
+ # A checkpoint names its own band (metadata head.level_band); this one is for a checkpoint that names none
21
+ LEVEL_MODES = ("band", "all", "asis")
22
+ LEVEL_DEFAULT = "band"
23
+ LEVEL_FRAME = 960 # 40-ms frames at 24 kHz, the encoder's frame rate
24
+
25
+
26
+ def level_db(mono: torch.Tensor, gate_db: float = 40.0) -> float:
27
+ """Gated RMS level in dBFS of a mono 24 kHz signal: the RMS over its 40-ms frames within ``gate_db`` of the loudest frame, so long
28
+ silences do not count. -inf for silence."""
29
+ x = mono.float()
30
+ frames = x[: len(x) // LEVEL_FRAME * LEVEL_FRAME].view(-1, LEVEL_FRAME).pow(2).mean(1).sqrt()
31
+ if not len(frames) or float(frames.max()) <= 0:
32
+ return float("-inf")
33
+ kept = frames[frames > frames.max() * 10 ** (-gate_db / 20)]
34
+ return 20 * float(torch.log10(kept.pow(2).mean().sqrt()))
35
+
36
+
37
+ def level_gain(mono: torch.Tensor, mode: str = LEVEL_DEFAULT, tolerance: float = 0.5, band: tuple = LEVEL_BAND) -> float:
38
+ """The gain in dB for a mono 24 kHz tokenizer input (0 = none): 'all' = to LEVEL_TARGET; 'band' = a song outside ``band`` to its
39
+ nearest edge, one inside it left alone; 'asis' = never. A song within ``tolerance`` dB of where it would go is left alone.
40
+ Silence: 0."""
41
+ if mode not in LEVEL_MODES:
42
+ raise ValueError(f"level must be one of {', '.join(LEVEL_MODES)}, not {mode!r}")
43
+ level = level_db(mono)
44
+ if mode == "asis" or not math.isfinite(level):
45
+ return 0.0
46
+ goal = LEVEL_TARGET if mode == "all" else min(max(level, band[0]), band[1])
47
+ return 0.0 if abs(goal - level) < tolerance else goal - level
48
+
49
+
50
+ def limit_peaks(x: torch.Tensor, rate: int, ceiling: float = 0.98, hold_ms: float = 10.0) -> torch.Tensor:
51
+ """A peak limiter for a mono signal: no sample above ``ceiling``. The gain each sample needs is held over +-hold_ms and smoothed, so
52
+ transients are pulled down smoothly instead of clipped."""
53
+ width = max(1, int(rate * hold_ms / 1000))
54
+ need = (ceiling / x.abs().float().clamp_min(1e-9)).clamp(max=1.0)[None, None]
55
+ need = -F.max_pool1d(-need, 2 * width + 1, 1, width)
56
+ return x * F.avg_pool1d(need, 2 * width + 1, 1, width, count_include_pad=False)[0, 0].to(x.dtype)
57
+
58
+
59
+ def fit_level(mono: torch.Tensor, mode: str = LEVEL_DEFAULT, rate: int = 24000, ceiling: float = 0.98,
60
+ band: tuple = LEVEL_BAND) -> tuple[torch.Tensor, float]:
61
+ """A mono tokenizer input -> (its loudness fitted by ``mode`` into ``band``, the gain in dB). A song raised past ``ceiling`` is
62
+ peak-limited; a loud master brought down keeps its own dynamics."""
63
+ gain = level_gain(mono, mode, band=band)
64
+ if not gain:
65
+ return mono, 0.0
66
+ x = mono * 10 ** (gain / 20)
67
+ if float(x.abs().max()) > ceiling:
68
+ x = limit_peaks(x, rate, ceiling)
69
+ return x, gain
70
+
71
 
72
  def load_audio(path: str | Path) -> torch.Tensor:
73
  """Decode an audio file to fp32 [2, samples] at 48 kHz, trimmed to VAE hops."""
wannabe_tokenizer/cli.py CHANGED
@@ -9,7 +9,7 @@ import numpy as np
9
  import torch
10
 
11
  from . import __version__
12
- from .audio import load_audio, save_wav_atomic
13
  from .quantizer import WannabeTokenizer
14
  from .renderer import YuE2Renderer
15
 
@@ -47,6 +47,13 @@ def build_parser() -> argparse.ArgumentParser:
47
  parser.add_argument("--seed", type=int, default=0, help="NAR noise seed (default: 0)")
48
  parser.add_argument("--steps", type=positive_int, default=32, help="Flow-matching steps (default: 32)")
49
  parser.add_argument("--batch-windows", type=positive_int, default=8, help="Tokenizer windows per batch (default: 8)")
 
 
 
 
 
 
 
50
  parser.add_argument("--overwrite", action="store_true", help="Replace existing tokens.npy and reconstructed.wav")
51
  parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
52
  return parser
@@ -86,7 +93,12 @@ def main() -> None:
86
  waveform = load_audio(audio_path)
87
  print(f"Tokenizing on {device}...")
88
  tokenizer = WannabeTokenizer(tokenizer_path, device=device)
89
- tokens = tokenizer.tokenize(waveform, batch_windows=args.batch_windows)
 
 
 
 
 
90
  save_tokens_atomic(tokens_path, tokens)
91
  print(f"Wrote {tokens_path} ({len(tokens)} frames, {len(tokens) / 25:.2f} seconds)")
92
 
 
9
  import torch
10
 
11
  from . import __version__
12
+ from .audio import LEVEL_DEFAULT, LEVEL_MODES, LEVEL_TARGET, load_audio, save_wav_atomic
13
  from .quantizer import WannabeTokenizer
14
  from .renderer import YuE2Renderer
15
 
 
47
  parser.add_argument("--seed", type=int, default=0, help="NAR noise seed (default: 0)")
48
  parser.add_argument("--steps", type=positive_int, default=32, help="Flow-matching steps (default: 32)")
49
  parser.add_argument("--batch-windows", type=positive_int, default=8, help="Tokenizer windows per batch (default: 8)")
50
+ parser.add_argument(
51
+ "--level",
52
+ choices=LEVEL_MODES,
53
+ default=LEVEL_DEFAULT,
54
+ help="Loudness before tokenizing: band = move only a song outside YuE2's usual range, to its nearest edge (default), "
55
+ "all = move every song to YuE2's level, asis = as recorded (for audio YuE2 generated itself)",
56
+ )
57
  parser.add_argument("--overwrite", action="store_true", help="Replace existing tokens.npy and reconstructed.wav")
58
  parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
59
  return parser
 
93
  waveform = load_audio(audio_path)
94
  print(f"Tokenizing on {device}...")
95
  tokenizer = WannabeTokenizer(tokenizer_path, device=device)
96
+ tokens, gain = tokenizer.tokenize(waveform, batch_windows=args.batch_windows, level=args.level, return_gain=True)
97
+ if gain:
98
+ goal = f"YuE2's level ({LEVEL_TARGET:g} dBFS gated RMS)" if args.level == "all" else "the edge of YuE2's range"
99
+ print(f"Loudness: moved {gain:+.1f} dB to {goal} before tokenizing")
100
+ else:
101
+ print(f"Loudness: left as recorded (--level {args.level})")
102
  save_tokens_atomic(tokens_path, tokens)
103
  print(f"Wrote {tokens_path} ({len(tokens)} frames, {len(tokens) / 25:.2f} seconds)")
104
 
wannabe_tokenizer/quantizer.py CHANGED
@@ -10,6 +10,7 @@ import torch.nn.functional as F
10
  from safetensors import safe_open
11
  from torch import nn
12
 
 
13
  from .encoder import MERT2Encoder, SAMPLE_RATE as ENCODER_RATE
14
 
15
  CODEC_SIZE = 32768
@@ -75,6 +76,7 @@ class WannabeTokenizer(nn.Module):
75
  tau = float(metadata.get("head.tau", 0.07))
76
  if not math.isfinite(tau) or tau <= 0:
77
  raise ValueError("head.tau must be a finite positive number")
 
78
 
79
  encoder_keys = [key for key in f.keys() if key.startswith("encoder.")]
80
  head_keys = [key for key in f.keys() if key.startswith("head.")]
@@ -131,6 +133,22 @@ class WannabeTokenizer(nn.Module):
131
  raise ValueError(f"{key} must be at least {minimum}")
132
  return value
133
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
134
  @staticmethod
135
  def _boolean(metadata: dict, key: str) -> bool:
136
  value = metadata.get(key)
@@ -139,8 +157,13 @@ class WannabeTokenizer(nn.Module):
139
  return value == "1"
140
 
141
  @torch.inference_mode()
142
- def tokenize(self, waveform: torch.Tensor, batch_windows: int = 8) -> np.ndarray:
143
- """Convert fp32 [channels, samples] 48 kHz audio to int32 tokens at 25 Hz."""
 
 
 
 
 
144
  import torchaudio
145
 
146
  if waveform.ndim != 2 or waveform.shape[0] < 1:
@@ -150,10 +173,12 @@ class WannabeTokenizer(nn.Module):
150
  mono = torchaudio.functional.resample(
151
  waveform.float().mean(0), 48000, ENCODER_RATE
152
  ).to(self.device)
 
153
  features = self.encoder.features(mono, self.layer, self.chunk)
154
  # The deployed pipeline transfers fp16 features before head inference.
155
  features = features.half().cpu().numpy()
156
- return self._predict(features, batch_windows).astype(np.int32)
 
157
 
158
  def _predict(self, features: np.ndarray, batch_windows: int) -> np.ndarray:
159
  x = features.astype(np.float32)
 
10
  from safetensors import safe_open
11
  from torch import nn
12
 
13
+ from .audio import LEVEL_BAND, LEVEL_DEFAULT, fit_level
14
  from .encoder import MERT2Encoder, SAMPLE_RATE as ENCODER_RATE
15
 
16
  CODEC_SIZE = 32768
 
76
  tau = float(metadata.get("head.tau", 0.07))
77
  if not math.isfinite(tau) or tau <= 0:
78
  raise ValueError("head.tau must be a finite positive number")
79
+ self.level_band = self._band(metadata)
80
 
81
  encoder_keys = [key for key in f.keys() if key.startswith("encoder.")]
82
  head_keys = [key for key in f.keys() if key.startswith("head.")]
 
133
  raise ValueError(f"{key} must be at least {minimum}")
134
  return value
135
 
136
+ @staticmethod
137
+ def _band(metadata: dict) -> tuple:
138
+ """The loudness band 'band' mode fits a song into, as the checkpoint names it (head.level_band, "[low, high]" dBFS); a
139
+ checkpoint that names none was made at audio.LEVEL_BAND"""
140
+ raw = metadata.get("head.level_band")
141
+ if raw is None:
142
+ return LEVEL_BAND
143
+ try:
144
+ band = json.loads(raw)
145
+ except json.JSONDecodeError as exc:
146
+ raise ValueError("head.level_band must be JSON") from exc
147
+ if not (isinstance(band, list) and len(band) == 2 and all(type(x) in (int, float) for x in band)
148
+ and all(math.isfinite(x) for x in band) and band[0] < band[1] <= 0):
149
+ raise ValueError("head.level_band must be [low, high] dBFS, finite, low < high <= 0")
150
+ return float(band[0]), float(band[1])
151
+
152
  @staticmethod
153
  def _boolean(metadata: dict, key: str) -> bool:
154
  value = metadata.get(key)
 
157
  return value == "1"
158
 
159
  @torch.inference_mode()
160
+ def tokenize(self, waveform: torch.Tensor, batch_windows: int = 8, level: str = LEVEL_DEFAULT, return_gain: bool = False):
161
+ """Convert fp32 [channels, samples] 48 kHz audio to int32 tokens at 25 Hz.
162
+
163
+ ``level`` (audio.fit_level, on the whole song first): 'band' (default) moves only a song outside YuE2's usual range (the band
164
+ the checkpoint names, ``self.level_band``), to its nearest edge, as ai-toolkit reads songs by default; 'all' moves every song to
165
+ YuE2's level; 'asis' reads it as recorded
166
+ (YuE2's own output, whose loudness is already its own). ``return_gain`` also returns the gain applied, in dB."""
167
  import torchaudio
168
 
169
  if waveform.ndim != 2 or waveform.shape[0] < 1:
 
173
  mono = torchaudio.functional.resample(
174
  waveform.float().mean(0), 48000, ENCODER_RATE
175
  ).to(self.device)
176
+ mono, gain = fit_level(mono, level, ENCODER_RATE, band=self.level_band)
177
  features = self.encoder.features(mono, self.layer, self.chunk)
178
  # The deployed pipeline transfers fp16 features before head inference.
179
  features = features.half().cpu().numpy()
180
+ tokens = self._predict(features, batch_windows).astype(np.int32)
181
+ return (tokens, gain) if return_gain else tokens
182
 
183
  def _predict(self, features: np.ndarray, batch_windows: int) -> np.ndarray:
184
  x = features.astype(np.float32)