Upload 14 files
Browse files- README.md +30 -99
- Run_WannabeTokenizer.ipynb +66 -82
- WannabeTokenizer.safetensors +3 -0
- pyproject.toml +1 -1
- wannabe_tokenizer/__init__.py +1 -1
- wannabe_tokenizer/audio.py +60 -1
- wannabe_tokenizer/cli.py +14 -2
- wannabe_tokenizer/quantizer.py +28 -3
README.md
CHANGED
|
@@ -1,99 +1,3 @@
|
|
| 1 |
-

|
| 2 |
-
|
| 3 |
-
# What This Is:
|
| 4 |
-
Just a humble wannabe semantic tokenizer to make the semantic tokens to train the Yue2 music model with
|
| 5 |
-
|
| 6 |
-
# Q/A:
|
| 7 |
-
Q: But.. wasn't there already another one?
|
| 8 |
-
|
| 9 |
-
A: Yes. But the dataset used to train it wasn't very robust. Many song segments devolved into noise.
|
| 10 |
-
|
| 11 |
-
Q: How good is this one then? Can it tokenize any song then?
|
| 12 |
-
|
| 13 |
-
A: More or less
|
| 14 |
-
|
| 15 |
-
Q: .. what do you mean more or less?
|
| 16 |
-
|
| 17 |
-
A: Yknow how if you don't have the real Nine Yin Manual, you have to practice the fake Nine Yin Manual?
|
| 18 |
-
This is the fake Nine Yin Manual. It gets alot more right. It gets the overall song right.
|
| 19 |
-
There are some slight hiccups with consonants occasionally. But it is our hope that that is something
|
| 20 |
-
the extra text conditioning on the NAR diffusion stage will change.
|
| 21 |
-
|
| 22 |
-
Q: What about the joint nar lora or all that from mothersuperior, do I use th-
|
| 23 |
-
|
| 24 |
-
A: Thou shalt have no other tokenizers before me (hahahaa) ... But no, don't do that.
|
| 25 |
-
This tokenizer will sound mostly right with the base model NAR stage. WILD, amirite? But that's how you make sure the AR model
|
| 26 |
-
won't shit the bed trying to learn your token sequences that may or may not describe production grade music.
|
| 27 |
-
You don't HAVE to train on production music. Sometimes you just want an amateur grade school orchestra to troll with.
|
| 28 |
-
But yeah no we don't need no stinkin NAR lora. It's counterproductive. How would you know if Yue2 can even learn it or not
|
| 29 |
-
if you can't even hear what the NAR stage normally sounds like? NAR loras are for the final polish stage.
|
| 30 |
-
|
| 31 |
-
|
| 32 |
-
# How to actually train Yue2:
|
| 33 |
-
The first step is obviously build a good foundation. The official Yu2 tokenizer would be that good foundation.
|
| 34 |
-
But if you don't have the real Nine Yin, practice the fake Nine Yin.
|
| 35 |
-
|
| 36 |
-
Next is the AR or autoregressive LLM part of Yue2. This is why having accurate semantic tokens matters so much.
|
| 37 |
-
The tokens carry ALL the meaning.
|
| 38 |
-
What sounds to say to make up lyrics.
|
| 39 |
-
What those vocals sound like.
|
| 40 |
-
That part takes your lyrics, your text prompt, and most importantly, sheet music melody, and spits out these semantic tokens here
|
| 41 |
-
These semantic tokens therefore have to be correct
|
| 42 |
-
|
| 43 |
-
SO! It means you have to use this tokenizer and listen to the reconstructed audio
|
| 44 |
-
And if it fails to remake your song, then you can't train it.
|
| 45 |
-
Simply that. Try to train and you'll destroy the model and give it brain damage.
|
| 46 |
-
Most of those people who look at us like we're crazy for doin all this extra shit,
|
| 47 |
-
well guess what? They like music the model was already good at and the tokenizer built on limited data already knew
|
| 48 |
-
|
| 49 |
-
But what if you wanted to make a chinese version of "I Want It That Way"
|
| 50 |
-
What if you wanted it to be Dutch? (yes this model knows Dutch. Crazy, I know.)
|
| 51 |
-
What if you wanted backing vocals? What if you didnt wanna sound like AI slop, but wanted to actually fool people
|
| 52 |
-
into thinking a cracked band made an awesome banger?
|
| 53 |
-
With the old tokenizer you'd be shit out of luck.
|
| 54 |
-
With this tokenizer... you have a better chance. No guarantees.
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
Lastly we come to the NAR, the diffusion stage! That stage turns these tokens here,
|
| 59 |
-
which describe exactly what instruments, what they sound like,
|
| 60 |
-
what the vocals sound like exactly, that stage turns these tokens into audio basically.
|
| 61 |
-
(I mean not really, but as far as alot of you are concerned, that's good enough of an understanding)
|
| 62 |
-
The NAR / diffusion stage takes these IDs and it also takes the text prompt.
|
| 63 |
-
|
| 64 |
-
HOWEVER!!!
|
| 65 |
-
We tried throwing in all kindsa trash and garbage instructions into the captions for the NAR stage
|
| 66 |
-
IT DID NOT CHANGE THE SONG AT ALL.
|
| 67 |
-
the NAR ONLY LISTENS TO THESE SEMANTIC TOKENS.
|
| 68 |
-
The only thing that might have changed a little bit was MAYBE,
|
| 69 |
-
JUST MAYBE, supplying the right lyrics improved pronunciation. Maybe. Or I could be trippin balls and coping hard.
|
| 70 |
-
|
| 71 |
-
|
| 72 |
-
# So which part of Yue2 Do I train?!!:
|
| 73 |
-
If you want a little extra polish on your sound, train the NAR stage.
|
| 74 |
-
|
| 75 |
-
Most of you want more than that. You want your own singers. You want a sick arrangement without having to prompt for one.
|
| 76 |
-
You want it to make banger melodies so you don't have to compose songs by hand.
|
| 77 |
-
|
| 78 |
-
That's all the AR, but in two stages. ONE of these stages is unaffected by a dogshit wannabe semantic tokenizer.
|
| 79 |
-
|
| 80 |
-
If you only need it to make good melodies, and nothing else, YOU ARE IN LUCK!
|
| 81 |
-
Freeze the losses on everything BUT the sheetABC part of the AR stage and just train based on melodies.
|
| 82 |
-
It will at least make banger melodies.
|
| 83 |
-
|
| 84 |
-
If you wanna train everything else, that's where this tokenizer comes into play.
|
| 85 |
-
You wanna train the melodies first. Then on the main AR stage, freeze the losses on the ABC section
|
| 86 |
-
and only compute losses on the tokens. (Ask your friendly AI assistant for help with that)
|
| 87 |
-
Now when it comes to the generating semantic tokens stage
|
| 88 |
-
You caption your song and everything, maybe even supply a corrected sheetsage ABC melody file. And then you supply
|
| 89 |
-
these semantic tokens.
|
| 90 |
-
|
| 91 |
-
The AR has to output the tokens that make the audio.
|
| 92 |
-
This tokenizer is a sanity check for that. It will generate and reconstruct.
|
| 93 |
-
If you don't hear something that sounds enough like what you want, you're not gonna get what you want with training
|
| 94 |
-
(And in fact I can't guarantee this won't blow up the AR after alotta steps either, but it should help somewhat)
|
| 95 |
-
|
| 96 |
-
|
| 97 |
# WannabeTokenizer inference
|
| 98 |
|
| 99 |
A self-contained, offline inference package for:
|
|
@@ -134,11 +38,25 @@ Existing outputs are protected unless `--overwrite` is supplied. Common input fo
|
|
| 134 |
Optional settings:
|
| 135 |
|
| 136 |
```bash
|
| 137 |
-
python run_tokenizer.py input.flac --tokenizer checkpoints/WannabeTokenizer.safetensors --yue2-checkpoint /path/to/yue2_3b_bf16.safetensors --output-dir output --device cuda:0 --seed 1 --steps 32
|
| 138 |
```
|
| 139 |
|
| 140 |
Use `python run_tokenizer.py --help` for all options. If installed as a package, the equivalent entry point is `wannabe-tokenize`.
|
| 141 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 142 |
## Behavior and resource use
|
| 143 |
|
| 144 |
Reconstruction is generative and seed-dependent, not a lossless inversion of the recording. It uses YuE2's base NAR with an empty tags/lyrics prompt. Different seeds produce slightly different audio from the same tokens.
|
|
@@ -153,11 +71,24 @@ from wannabe_tokenizer.audio import load_audio, save_wav_atomic
|
|
| 153 |
|
| 154 |
waveform = load_audio("input.wav")
|
| 155 |
tokenizer = WannabeTokenizer("checkpoints/WannabeTokenizer.safetensors", device="cuda")
|
| 156 |
-
tokens = tokenizer.tokenize(waveform)
|
| 157 |
|
| 158 |
del tokenizer # release encoder memory before loading YuE2 in a real application
|
| 159 |
renderer = YuE2Renderer("/path/to/yue2_3b_bf16.safetensors", device="cuda")
|
| 160 |
reconstructed = renderer.reconstruct(tokens, seed=0, steps=32)
|
| 161 |
save_wav_atomic("reconstructed.wav", reconstructed)
|
| 162 |
```
|
| 163 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# WannabeTokenizer inference
|
| 2 |
|
| 3 |
A self-contained, offline inference package for:
|
|
|
|
| 38 |
Optional settings:
|
| 39 |
|
| 40 |
```bash
|
| 41 |
+
python run_tokenizer.py input.flac --tokenizer checkpoints/WannabeTokenizer.safetensors --yue2-checkpoint /path/to/yue2_3b_bf16.safetensors --output-dir output --device cuda:0 --seed 1 --steps 32 --level all
|
| 42 |
```
|
| 43 |
|
| 44 |
Use `python run_tokenizer.py --help` for all options. If installed as a package, the equivalent entry point is `wannabe-tokenize`.
|
| 45 |
|
| 46 |
+
## Loudness
|
| 47 |
+
|
| 48 |
+
The tokenizer writes the input's loudness into the IDs, and YuE2 renders whatever level the IDs carry. YuE2's own songs sit in a fairly narrow loudness range, so by default a song outside that range is brought into it before it is tokenized: one gain for the whole song from its gated RMS, with a peak limiter if the song was raised. Per-section gains are never used, so the song keeps its dynamics.
|
| 49 |
+
|
| 50 |
+
`--level` chooses this:
|
| 51 |
+
|
| 52 |
+
- `band` (default): a song inside YuE2's usual range is left alone, and one outside it is moved only as far as the nearest edge. The checkpoint names its range (metadata `head.level_band`); one that names none uses -21 to -13 dBFS (a -25 dBFS song to -21, a -9 dBFS master to -13). This is the smallest change that puts a song in the range YuE2's own songs cover, and how ai-toolkit reads songs by default.
|
| 53 |
+
- `all`: every song is moved to -16.5 dBFS (about -14 LUFS), the level YuE2's own covers of real songs sit at, whatever the original master's.
|
| 54 |
+
- `asis`: the song is read as recorded. Use it for audio YuE2 generated itself, which is already at its own level.
|
| 55 |
+
|
| 56 |
+
The run prints the gain it applied.
|
| 57 |
+
|
| 58 |
+
Decode each song from the same file every time. The IDs also depend on where the 40-ms frame grid falls in the audio, so two encodes of one recording (an MP3 and an M4A download, say) can give noticeably different IDs.
|
| 59 |
+
|
| 60 |
## Behavior and resource use
|
| 61 |
|
| 62 |
Reconstruction is generative and seed-dependent, not a lossless inversion of the recording. It uses YuE2's base NAR with an empty tags/lyrics prompt. Different seeds produce slightly different audio from the same tokens.
|
|
|
|
| 71 |
|
| 72 |
waveform = load_audio("input.wav")
|
| 73 |
tokenizer = WannabeTokenizer("checkpoints/WannabeTokenizer.safetensors", device="cuda")
|
| 74 |
+
tokens, gain_db = tokenizer.tokenize(waveform, level="band", return_gain=True)
|
| 75 |
|
| 76 |
del tokenizer # release encoder memory before loading YuE2 in a real application
|
| 77 |
renderer = YuE2Renderer("/path/to/yue2_3b_bf16.safetensors", device="cuda")
|
| 78 |
reconstructed = renderer.reconstruct(tokens, seed=0, steps=32)
|
| 79 |
save_wav_atomic("reconstructed.wav", reconstructed)
|
| 80 |
```
|
| 81 |
+
|
| 82 |
+
## Tests
|
| 83 |
+
|
| 84 |
+
The lightweight tests do not load model weights:
|
| 85 |
+
|
| 86 |
+
```bash
|
| 87 |
+
python -m unittest discover -s tests
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
An end-to-end run requires both checkpoints and is the recommended numerical smoke test after installation.
|
| 91 |
+
|
| 92 |
+
## Third-party models
|
| 93 |
+
|
| 94 |
+
Model weights are subject to their respective upstream licenses. The YuE2 checkpoint identifies its license as CC-BY-NC-4.0. Review the applicable model and dependency licenses before redistribution or commercial use.
|
Run_WannabeTokenizer.ipynb
CHANGED
|
@@ -1,85 +1,69 @@
|
|
| 1 |
{
|
| 2 |
-
|
| 3 |
-
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
|
| 9 |
-
|
| 10 |
-
|
| 11 |
-
},
|
| 12 |
-
{
|
| 13 |
-
"cell_type": "code",
|
| 14 |
-
"execution_count": null,
|
| 15 |
-
"metadata": {},
|
| 16 |
-
"outputs": [],
|
| 17 |
-
"source": [
|
| 18 |
-
"from pathlib import Path\n",
|
| 19 |
-
"import subprocess\n",
|
| 20 |
-
"import sys\n",
|
| 21 |
-
"\n",
|
| 22 |
-
"PACKAGE_DIR = Path.cwd()\n",
|
| 23 |
-
"TOKENIZER = PACKAGE_DIR / 'checkpoints' / 'WannabeTokenizer.safetensors'\n",
|
| 24 |
-
"YUE2_CHECKPOINT = Path('/path/to/yue2_3b_bf16.safetensors')\n",
|
| 25 |
-
"OUTPUT_DIR = PACKAGE_DIR / 'output'\n",
|
| 26 |
-
"\n",
|
| 27 |
-
"DEVICE = 'auto' # 'auto', 'cuda', 'cuda:0', or 'cpu'\n",
|
| 28 |
-
"SEED = 4 # Shout out to eMuhdra\n",
|
| 29 |
-
"STEPS = 32\n",
|
| 30 |
-
"OVERWRITE = False"
|
| 31 |
-
]
|
| 32 |
-
},
|
| 33 |
-
{
|
| 34 |
-
"cell_type": "code",
|
| 35 |
-
"execution_count": null,
|
| 36 |
-
"metadata": {},
|
| 37 |
-
"outputs": [],
|
| 38 |
-
"source": [
|
| 39 |
-
"INPUT_AUDIO = Path('/path/to/input.wav')\n",
|
| 40 |
-
"command = [\n",
|
| 41 |
-
" sys.executable,\n",
|
| 42 |
-
" str(PACKAGE_DIR / 'run_tokenizer.py'),\n",
|
| 43 |
-
" str(INPUT_AUDIO),\n",
|
| 44 |
-
" '--tokenizer', str(TOKENIZER),\n",
|
| 45 |
-
" '--yue2-checkpoint', str(YUE2_CHECKPOINT),\n",
|
| 46 |
-
" '--output-dir', str(OUTPUT_DIR),\n",
|
| 47 |
-
" '--device', DEVICE,\n",
|
| 48 |
-
" '--seed', str(SEED),\n",
|
| 49 |
-
" '--steps', str(STEPS),\n",
|
| 50 |
-
"]\n",
|
| 51 |
-
"if OVERWRITE:\n",
|
| 52 |
-
" command.append('--overwrite')\n",
|
| 53 |
-
"\n",
|
| 54 |
-
"subprocess.run(command, cwd=PACKAGE_DIR, check=True)"
|
| 55 |
-
]
|
| 56 |
-
},
|
| 57 |
-
{
|
| 58 |
-
"cell_type": "code",
|
| 59 |
-
"execution_count": null,
|
| 60 |
-
"metadata": {},
|
| 61 |
-
"outputs": [],
|
| 62 |
-
"source": [
|
| 63 |
-
"import numpy as np\n",
|
| 64 |
-
"from IPython.display import Audio, display\n",
|
| 65 |
-
"\n",
|
| 66 |
-
"tokens = np.load(OUTPUT_DIR / 'tokens.npy')\n",
|
| 67 |
-
"print(f'{len(tokens)} tokens at 25 Hz; dtype={tokens.dtype}; range={tokens.min()}..{tokens.max()}')\n",
|
| 68 |
-
"display(Audio(filename=str(OUTPUT_DIR / 'reconstructed.wav')))"
|
| 69 |
-
]
|
| 70 |
-
}
|
| 71 |
-
],
|
| 72 |
-
"metadata": {
|
| 73 |
-
"kernelspec": {
|
| 74 |
-
"display_name": "Python 3",
|
| 75 |
-
"language": "python",
|
| 76 |
-
"name": "python3"
|
| 77 |
-
},
|
| 78 |
-
"language_info": {
|
| 79 |
-
"name": "python",
|
| 80 |
-
"version": "3.10"
|
| 81 |
-
}
|
| 82 |
},
|
| 83 |
-
|
| 84 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 85 |
}
|
|
|
|
| 1 |
{
|
| 2 |
+
"cells": [
|
| 3 |
+
{
|
| 4 |
+
"cell_type": "markdown",
|
| 5 |
+
"metadata": {},
|
| 6 |
+
"source": [
|
| 7 |
+
"# Run WannabeTokenizer\n",
|
| 8 |
+
"\n",
|
| 9 |
+
"Set the three paths below, then run the cells in order. Start Jupyter from this package folder so the relative checkpoint path works."
|
| 10 |
+
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11 |
},
|
| 12 |
+
{
|
| 13 |
+
"cell_type": "code",
|
| 14 |
+
"execution_count": null,
|
| 15 |
+
"metadata": {},
|
| 16 |
+
"outputs": [],
|
| 17 |
+
"source": [
|
| 18 |
+
"from pathlib import Path\n",
|
| 19 |
+
"import subprocess\n",
|
| 20 |
+
"import sys\n",
|
| 21 |
+
"\n",
|
| 22 |
+
"PACKAGE_DIR = Path.cwd()\n",
|
| 23 |
+
"TOKENIZER = PACKAGE_DIR / 'checkpoints' / 'WannabeTokenizer.safetensors'\n",
|
| 24 |
+
"YUE2_CHECKPOINT = Path('/path/to/yue2_3b_bf16.safetensors')\n",
|
| 25 |
+
"OUTPUT_DIR = PACKAGE_DIR / 'output'\n",
|
| 26 |
+
"\n",
|
| 27 |
+
"DEVICE = 'auto' # 'auto', 'cuda', 'cuda:0', or 'cpu'\n",
|
| 28 |
+
"SEED = 4 # Shout out to eMuhdra\n",
|
| 29 |
+
"STEPS = 32\n",
|
| 30 |
+
"LEVEL = 'band' # 'band' = only a song outside YuE2's range moves, to its nearest edge | 'all' = every song to YuE2's loudness | 'asis' = as recorded (YuE2's own audio)\n",
|
| 31 |
+
"OVERWRITE = False"
|
| 32 |
+
]
|
| 33 |
+
},
|
| 34 |
+
{
|
| 35 |
+
"cell_type": "code",
|
| 36 |
+
"execution_count": null,
|
| 37 |
+
"metadata": {},
|
| 38 |
+
"outputs": [],
|
| 39 |
+
"source": "INPUT_AUDIO = Path('/path/to/input.wav')\ncommand = [\n sys.executable,\n str(PACKAGE_DIR / 'run_tokenizer.py'),\n str(INPUT_AUDIO),\n '--tokenizer', str(TOKENIZER),\n '--yue2-checkpoint', str(YUE2_CHECKPOINT),\n '--output-dir', str(OUTPUT_DIR),\n '--device', DEVICE,\n '--seed', str(SEED),\n '--steps', str(STEPS),\n '--level', LEVEL,\n]\nif OVERWRITE:\n command.append('--overwrite')\n\nsubprocess.run(command, cwd=PACKAGE_DIR, check=True)"
|
| 40 |
+
},
|
| 41 |
+
{
|
| 42 |
+
"cell_type": "code",
|
| 43 |
+
"execution_count": null,
|
| 44 |
+
"metadata": {},
|
| 45 |
+
"outputs": [],
|
| 46 |
+
"source": [
|
| 47 |
+
"import numpy as np\n",
|
| 48 |
+
"from IPython.display import Audio, display\n",
|
| 49 |
+
"\n",
|
| 50 |
+
"tokens = np.load(OUTPUT_DIR / 'tokens.npy')\n",
|
| 51 |
+
"print(f'{len(tokens)} tokens at 25 Hz; dtype={tokens.dtype}; range={tokens.min()}..{tokens.max()}')\n",
|
| 52 |
+
"display(Audio(filename=str(OUTPUT_DIR / 'reconstructed.wav')))"
|
| 53 |
+
]
|
| 54 |
+
}
|
| 55 |
+
],
|
| 56 |
+
"metadata": {
|
| 57 |
+
"kernelspec": {
|
| 58 |
+
"display_name": "Python 3",
|
| 59 |
+
"language": "python",
|
| 60 |
+
"name": "python3"
|
| 61 |
+
},
|
| 62 |
+
"language_info": {
|
| 63 |
+
"name": "python",
|
| 64 |
+
"version": "3.10"
|
| 65 |
+
}
|
| 66 |
+
},
|
| 67 |
+
"nbformat": 4,
|
| 68 |
+
"nbformat_minor": 5
|
| 69 |
}
|
WannabeTokenizer.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:3a5d311aeca55d3dfccbbc2af950ea247777a1e2757b8ef4ed6ac888f492e232
|
| 3 |
+
size 1698038168
|
pyproject.toml
CHANGED
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "wannabe-tokenizer-inference"
|
| 7 |
-
version = "1.
|
| 8 |
description = "Audio-to-semantic-token inference and YuE2 reconstruction"
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
|
|
|
| 4 |
|
| 5 |
[project]
|
| 6 |
name = "wannabe-tokenizer-inference"
|
| 7 |
+
version = "1.1.0"
|
| 8 |
description = "Audio-to-semantic-token inference and YuE2 reconstruction"
|
| 9 |
readme = "README.md"
|
| 10 |
requires-python = ">=3.10"
|
wannabe_tokenizer/__init__.py
CHANGED
|
@@ -4,4 +4,4 @@ from .quantizer import WannabeTokenizer
|
|
| 4 |
from .renderer import YuE2Renderer
|
| 5 |
|
| 6 |
__all__ = ["WannabeTokenizer", "YuE2Renderer"]
|
| 7 |
-
__version__ = "1.
|
|
|
|
| 4 |
from .renderer import YuE2Renderer
|
| 5 |
|
| 6 |
__all__ = ["WannabeTokenizer", "YuE2Renderer"]
|
| 7 |
+
__version__ = "1.1.0"
|
wannabe_tokenizer/audio.py
CHANGED
|
@@ -1,14 +1,73 @@
|
|
| 1 |
-
"""Audio input and output for the 48 kHz stereo YuE2 pipeline."""
|
| 2 |
|
|
|
|
| 3 |
import os
|
| 4 |
from pathlib import Path
|
| 5 |
|
| 6 |
import soundfile as sf
|
| 7 |
import torch
|
|
|
|
| 8 |
|
| 9 |
SAMPLE_RATE = 48000
|
| 10 |
HOP = 1920
|
| 11 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 12 |
|
| 13 |
def load_audio(path: str | Path) -> torch.Tensor:
|
| 14 |
"""Decode an audio file to fp32 [2, samples] at 48 kHz, trimmed to VAE hops."""
|
|
|
|
| 1 |
+
"""Audio input and output for the 48 kHz stereo YuE2 pipeline, and the loudness fit applied before tokenizing."""
|
| 2 |
|
| 3 |
+
import math
|
| 4 |
import os
|
| 5 |
from pathlib import Path
|
| 6 |
|
| 7 |
import soundfile as sf
|
| 8 |
import torch
|
| 9 |
+
import torch.nn.functional as F
|
| 10 |
|
| 11 |
SAMPLE_RATE = 48000
|
| 12 |
HOP = 1920
|
| 13 |
|
| 14 |
+
# YuE2 sets a song's loudness by style: its own covers of real songs sit at one level whatever the master's (about -14 LUFS), and the
|
| 15 |
+
# tokenizer writes the input's loudness into the ids, which the NAR then renders. So a real song's loudness is fitted before it is
|
| 16 |
+
# tokenized: one gain for the whole song from its gated RMS (per-chunk gains would erase its dynamics), peaks limited if it was raised.
|
| 17 |
+
# The default, 'band', moves only a song outside YuE2's range, and only to its edge: a song already inside keeps its own level.
|
| 18 |
+
LEVEL_TARGET = -16.5 # dBFS, gated RMS of the tokenizer input (mono, 24 kHz), the level YuE2's own songs have
|
| 19 |
+
LEVEL_BAND = (-21.0, -13.0) # the range YuE2's own songs sit in: 'band' leaves a song inside it alone, moves one outside to its edge.
|
| 20 |
+
# A checkpoint names its own band (metadata head.level_band); this one is for a checkpoint that names none
|
| 21 |
+
LEVEL_MODES = ("band", "all", "asis")
|
| 22 |
+
LEVEL_DEFAULT = "band"
|
| 23 |
+
LEVEL_FRAME = 960 # 40-ms frames at 24 kHz, the encoder's frame rate
|
| 24 |
+
|
| 25 |
+
|
| 26 |
+
def level_db(mono: torch.Tensor, gate_db: float = 40.0) -> float:
|
| 27 |
+
"""Gated RMS level in dBFS of a mono 24 kHz signal: the RMS over its 40-ms frames within ``gate_db`` of the loudest frame, so long
|
| 28 |
+
silences do not count. -inf for silence."""
|
| 29 |
+
x = mono.float()
|
| 30 |
+
frames = x[: len(x) // LEVEL_FRAME * LEVEL_FRAME].view(-1, LEVEL_FRAME).pow(2).mean(1).sqrt()
|
| 31 |
+
if not len(frames) or float(frames.max()) <= 0:
|
| 32 |
+
return float("-inf")
|
| 33 |
+
kept = frames[frames > frames.max() * 10 ** (-gate_db / 20)]
|
| 34 |
+
return 20 * float(torch.log10(kept.pow(2).mean().sqrt()))
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def level_gain(mono: torch.Tensor, mode: str = LEVEL_DEFAULT, tolerance: float = 0.5, band: tuple = LEVEL_BAND) -> float:
|
| 38 |
+
"""The gain in dB for a mono 24 kHz tokenizer input (0 = none): 'all' = to LEVEL_TARGET; 'band' = a song outside ``band`` to its
|
| 39 |
+
nearest edge, one inside it left alone; 'asis' = never. A song within ``tolerance`` dB of where it would go is left alone.
|
| 40 |
+
Silence: 0."""
|
| 41 |
+
if mode not in LEVEL_MODES:
|
| 42 |
+
raise ValueError(f"level must be one of {', '.join(LEVEL_MODES)}, not {mode!r}")
|
| 43 |
+
level = level_db(mono)
|
| 44 |
+
if mode == "asis" or not math.isfinite(level):
|
| 45 |
+
return 0.0
|
| 46 |
+
goal = LEVEL_TARGET if mode == "all" else min(max(level, band[0]), band[1])
|
| 47 |
+
return 0.0 if abs(goal - level) < tolerance else goal - level
|
| 48 |
+
|
| 49 |
+
|
| 50 |
+
def limit_peaks(x: torch.Tensor, rate: int, ceiling: float = 0.98, hold_ms: float = 10.0) -> torch.Tensor:
|
| 51 |
+
"""A peak limiter for a mono signal: no sample above ``ceiling``. The gain each sample needs is held over +-hold_ms and smoothed, so
|
| 52 |
+
transients are pulled down smoothly instead of clipped."""
|
| 53 |
+
width = max(1, int(rate * hold_ms / 1000))
|
| 54 |
+
need = (ceiling / x.abs().float().clamp_min(1e-9)).clamp(max=1.0)[None, None]
|
| 55 |
+
need = -F.max_pool1d(-need, 2 * width + 1, 1, width)
|
| 56 |
+
return x * F.avg_pool1d(need, 2 * width + 1, 1, width, count_include_pad=False)[0, 0].to(x.dtype)
|
| 57 |
+
|
| 58 |
+
|
| 59 |
+
def fit_level(mono: torch.Tensor, mode: str = LEVEL_DEFAULT, rate: int = 24000, ceiling: float = 0.98,
|
| 60 |
+
band: tuple = LEVEL_BAND) -> tuple[torch.Tensor, float]:
|
| 61 |
+
"""A mono tokenizer input -> (its loudness fitted by ``mode`` into ``band``, the gain in dB). A song raised past ``ceiling`` is
|
| 62 |
+
peak-limited; a loud master brought down keeps its own dynamics."""
|
| 63 |
+
gain = level_gain(mono, mode, band=band)
|
| 64 |
+
if not gain:
|
| 65 |
+
return mono, 0.0
|
| 66 |
+
x = mono * 10 ** (gain / 20)
|
| 67 |
+
if float(x.abs().max()) > ceiling:
|
| 68 |
+
x = limit_peaks(x, rate, ceiling)
|
| 69 |
+
return x, gain
|
| 70 |
+
|
| 71 |
|
| 72 |
def load_audio(path: str | Path) -> torch.Tensor:
|
| 73 |
"""Decode an audio file to fp32 [2, samples] at 48 kHz, trimmed to VAE hops."""
|
wannabe_tokenizer/cli.py
CHANGED
|
@@ -9,7 +9,7 @@ import numpy as np
|
|
| 9 |
import torch
|
| 10 |
|
| 11 |
from . import __version__
|
| 12 |
-
from .audio import load_audio, save_wav_atomic
|
| 13 |
from .quantizer import WannabeTokenizer
|
| 14 |
from .renderer import YuE2Renderer
|
| 15 |
|
|
@@ -47,6 +47,13 @@ def build_parser() -> argparse.ArgumentParser:
|
|
| 47 |
parser.add_argument("--seed", type=int, default=0, help="NAR noise seed (default: 0)")
|
| 48 |
parser.add_argument("--steps", type=positive_int, default=32, help="Flow-matching steps (default: 32)")
|
| 49 |
parser.add_argument("--batch-windows", type=positive_int, default=8, help="Tokenizer windows per batch (default: 8)")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 50 |
parser.add_argument("--overwrite", action="store_true", help="Replace existing tokens.npy and reconstructed.wav")
|
| 51 |
parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
|
| 52 |
return parser
|
|
@@ -86,7 +93,12 @@ def main() -> None:
|
|
| 86 |
waveform = load_audio(audio_path)
|
| 87 |
print(f"Tokenizing on {device}...")
|
| 88 |
tokenizer = WannabeTokenizer(tokenizer_path, device=device)
|
| 89 |
-
tokens = tokenizer.tokenize(waveform, batch_windows=args.batch_windows)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 90 |
save_tokens_atomic(tokens_path, tokens)
|
| 91 |
print(f"Wrote {tokens_path} ({len(tokens)} frames, {len(tokens) / 25:.2f} seconds)")
|
| 92 |
|
|
|
|
| 9 |
import torch
|
| 10 |
|
| 11 |
from . import __version__
|
| 12 |
+
from .audio import LEVEL_DEFAULT, LEVEL_MODES, LEVEL_TARGET, load_audio, save_wav_atomic
|
| 13 |
from .quantizer import WannabeTokenizer
|
| 14 |
from .renderer import YuE2Renderer
|
| 15 |
|
|
|
|
| 47 |
parser.add_argument("--seed", type=int, default=0, help="NAR noise seed (default: 0)")
|
| 48 |
parser.add_argument("--steps", type=positive_int, default=32, help="Flow-matching steps (default: 32)")
|
| 49 |
parser.add_argument("--batch-windows", type=positive_int, default=8, help="Tokenizer windows per batch (default: 8)")
|
| 50 |
+
parser.add_argument(
|
| 51 |
+
"--level",
|
| 52 |
+
choices=LEVEL_MODES,
|
| 53 |
+
default=LEVEL_DEFAULT,
|
| 54 |
+
help="Loudness before tokenizing: band = move only a song outside YuE2's usual range, to its nearest edge (default), "
|
| 55 |
+
"all = move every song to YuE2's level, asis = as recorded (for audio YuE2 generated itself)",
|
| 56 |
+
)
|
| 57 |
parser.add_argument("--overwrite", action="store_true", help="Replace existing tokens.npy and reconstructed.wav")
|
| 58 |
parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
|
| 59 |
return parser
|
|
|
|
| 93 |
waveform = load_audio(audio_path)
|
| 94 |
print(f"Tokenizing on {device}...")
|
| 95 |
tokenizer = WannabeTokenizer(tokenizer_path, device=device)
|
| 96 |
+
tokens, gain = tokenizer.tokenize(waveform, batch_windows=args.batch_windows, level=args.level, return_gain=True)
|
| 97 |
+
if gain:
|
| 98 |
+
goal = f"YuE2's level ({LEVEL_TARGET:g} dBFS gated RMS)" if args.level == "all" else "the edge of YuE2's range"
|
| 99 |
+
print(f"Loudness: moved {gain:+.1f} dB to {goal} before tokenizing")
|
| 100 |
+
else:
|
| 101 |
+
print(f"Loudness: left as recorded (--level {args.level})")
|
| 102 |
save_tokens_atomic(tokens_path, tokens)
|
| 103 |
print(f"Wrote {tokens_path} ({len(tokens)} frames, {len(tokens) / 25:.2f} seconds)")
|
| 104 |
|
wannabe_tokenizer/quantizer.py
CHANGED
|
@@ -10,6 +10,7 @@ import torch.nn.functional as F
|
|
| 10 |
from safetensors import safe_open
|
| 11 |
from torch import nn
|
| 12 |
|
|
|
|
| 13 |
from .encoder import MERT2Encoder, SAMPLE_RATE as ENCODER_RATE
|
| 14 |
|
| 15 |
CODEC_SIZE = 32768
|
|
@@ -75,6 +76,7 @@ class WannabeTokenizer(nn.Module):
|
|
| 75 |
tau = float(metadata.get("head.tau", 0.07))
|
| 76 |
if not math.isfinite(tau) or tau <= 0:
|
| 77 |
raise ValueError("head.tau must be a finite positive number")
|
|
|
|
| 78 |
|
| 79 |
encoder_keys = [key for key in f.keys() if key.startswith("encoder.")]
|
| 80 |
head_keys = [key for key in f.keys() if key.startswith("head.")]
|
|
@@ -131,6 +133,22 @@ class WannabeTokenizer(nn.Module):
|
|
| 131 |
raise ValueError(f"{key} must be at least {minimum}")
|
| 132 |
return value
|
| 133 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 134 |
@staticmethod
|
| 135 |
def _boolean(metadata: dict, key: str) -> bool:
|
| 136 |
value = metadata.get(key)
|
|
@@ -139,8 +157,13 @@ class WannabeTokenizer(nn.Module):
|
|
| 139 |
return value == "1"
|
| 140 |
|
| 141 |
@torch.inference_mode()
|
| 142 |
-
def tokenize(self, waveform: torch.Tensor, batch_windows: int = 8
|
| 143 |
-
"""Convert fp32 [channels, samples] 48 kHz audio to int32 tokens at 25 Hz.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 144 |
import torchaudio
|
| 145 |
|
| 146 |
if waveform.ndim != 2 or waveform.shape[0] < 1:
|
|
@@ -150,10 +173,12 @@ class WannabeTokenizer(nn.Module):
|
|
| 150 |
mono = torchaudio.functional.resample(
|
| 151 |
waveform.float().mean(0), 48000, ENCODER_RATE
|
| 152 |
).to(self.device)
|
|
|
|
| 153 |
features = self.encoder.features(mono, self.layer, self.chunk)
|
| 154 |
# The deployed pipeline transfers fp16 features before head inference.
|
| 155 |
features = features.half().cpu().numpy()
|
| 156 |
-
|
|
|
|
| 157 |
|
| 158 |
def _predict(self, features: np.ndarray, batch_windows: int) -> np.ndarray:
|
| 159 |
x = features.astype(np.float32)
|
|
|
|
| 10 |
from safetensors import safe_open
|
| 11 |
from torch import nn
|
| 12 |
|
| 13 |
+
from .audio import LEVEL_BAND, LEVEL_DEFAULT, fit_level
|
| 14 |
from .encoder import MERT2Encoder, SAMPLE_RATE as ENCODER_RATE
|
| 15 |
|
| 16 |
CODEC_SIZE = 32768
|
|
|
|
| 76 |
tau = float(metadata.get("head.tau", 0.07))
|
| 77 |
if not math.isfinite(tau) or tau <= 0:
|
| 78 |
raise ValueError("head.tau must be a finite positive number")
|
| 79 |
+
self.level_band = self._band(metadata)
|
| 80 |
|
| 81 |
encoder_keys = [key for key in f.keys() if key.startswith("encoder.")]
|
| 82 |
head_keys = [key for key in f.keys() if key.startswith("head.")]
|
|
|
|
| 133 |
raise ValueError(f"{key} must be at least {minimum}")
|
| 134 |
return value
|
| 135 |
|
| 136 |
+
@staticmethod
|
| 137 |
+
def _band(metadata: dict) -> tuple:
|
| 138 |
+
"""The loudness band 'band' mode fits a song into, as the checkpoint names it (head.level_band, "[low, high]" dBFS); a
|
| 139 |
+
checkpoint that names none was made at audio.LEVEL_BAND"""
|
| 140 |
+
raw = metadata.get("head.level_band")
|
| 141 |
+
if raw is None:
|
| 142 |
+
return LEVEL_BAND
|
| 143 |
+
try:
|
| 144 |
+
band = json.loads(raw)
|
| 145 |
+
except json.JSONDecodeError as exc:
|
| 146 |
+
raise ValueError("head.level_band must be JSON") from exc
|
| 147 |
+
if not (isinstance(band, list) and len(band) == 2 and all(type(x) in (int, float) for x in band)
|
| 148 |
+
and all(math.isfinite(x) for x in band) and band[0] < band[1] <= 0):
|
| 149 |
+
raise ValueError("head.level_band must be [low, high] dBFS, finite, low < high <= 0")
|
| 150 |
+
return float(band[0]), float(band[1])
|
| 151 |
+
|
| 152 |
@staticmethod
|
| 153 |
def _boolean(metadata: dict, key: str) -> bool:
|
| 154 |
value = metadata.get(key)
|
|
|
|
| 157 |
return value == "1"
|
| 158 |
|
| 159 |
@torch.inference_mode()
|
| 160 |
+
def tokenize(self, waveform: torch.Tensor, batch_windows: int = 8, level: str = LEVEL_DEFAULT, return_gain: bool = False):
|
| 161 |
+
"""Convert fp32 [channels, samples] 48 kHz audio to int32 tokens at 25 Hz.
|
| 162 |
+
|
| 163 |
+
``level`` (audio.fit_level, on the whole song first): 'band' (default) moves only a song outside YuE2's usual range (the band
|
| 164 |
+
the checkpoint names, ``self.level_band``), to its nearest edge, as ai-toolkit reads songs by default; 'all' moves every song to
|
| 165 |
+
YuE2's level; 'asis' reads it as recorded
|
| 166 |
+
(YuE2's own output, whose loudness is already its own). ``return_gain`` also returns the gain applied, in dB."""
|
| 167 |
import torchaudio
|
| 168 |
|
| 169 |
if waveform.ndim != 2 or waveform.shape[0] < 1:
|
|
|
|
| 173 |
mono = torchaudio.functional.resample(
|
| 174 |
waveform.float().mean(0), 48000, ENCODER_RATE
|
| 175 |
).to(self.device)
|
| 176 |
+
mono, gain = fit_level(mono, level, ENCODER_RATE, band=self.level_band)
|
| 177 |
features = self.encoder.features(mono, self.layer, self.chunk)
|
| 178 |
# The deployed pipeline transfers fp16 features before head inference.
|
| 179 |
features = features.half().cpu().numpy()
|
| 180 |
+
tokens = self._predict(features, batch_windows).astype(np.int32)
|
| 181 |
+
return (tokens, gain) if return_gain else tokens
|
| 182 |
|
| 183 |
def _predict(self, features: np.ndarray, batch_windows: int) -> np.ndarray:
|
| 184 |
x = features.astype(np.float32)
|