SheetSage consistently messes up covers around the 50 second mark
I have been finding that whenever I do cover songs with YuE2, Sheetsage's transcription gets messed up around 50 seconds in, which causes vocals to be sung early, which then throws off the timing of the rest of the song. Is it possible there is some issue with the way the model is trained, or am I using it incorrectly?
+1, not sure if this is some ComfyUI problem but in the current state it's basically unusable
Thansk for your feedback. Could you share an example song (like a youtube link) that reproduces this? Better if you can also share the ABC output.
Also, which codebase/comfyui workflow did you use to run it?
Best,
Junyan
any instrumental track, default ComfyUI workflow 🤷♂️ btw is this normal that "Ins" is the actual melody and "Vocal" is only chords?
Thanks. Some examples would be helpful.
Known issue: SheetSage2 does not handle songs with constant tempo changes very well. Occasional tempo changes are fine, but free-rhythm passages are currently outside its scope.
"Ins" is the actual melody and "Vocal" is only chords is as expected for songs without any vocal.
Chiptune example: Draconus Main theme
Note the hallucinated pause & two notes z24z4e2e2 in the very beginning.
Sheetsage likes to insert extra beats here, the last time I tried it, I got some weird 4/4 <-> 2/4 <-> 4/4 switch around 40-50 seconds, this time got an extra beat ~2 times later, at 1:40ish. Even more mistakes later, but that might be another question
(oh well, HF does not allow txt attach, IWOT)
X:1
T:
M:4/4
L:1/32
Q:1/4=151
V: Vocal clef=treble name="Vocal Melody" snm="Vocal"
V: Ins clef=treble name="Ins Melody" snm="Inst."
K:Cm
% intro
V: Vocal
z24z4"Cm"z4|"Cm"z32|"Cm"z32|"D#"z32|
V: Ins
z24z4e2e2|e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|
V: Vocal
"D#"z32|"Fm"z32|"G#"z32|"A#"z32|
V: Ins
c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|
V: Vocal
"Fm"z32|"Cm"z32|"Cm"z32|"D#"z32|
V: Ins
c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|
V: Vocal
"D#"z32|"Fm"z32|"G#"z32|"A#"z32|
V: Ins
c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|
V: Vocal
"Fm"z32|"Cm"z32|"Cm"z32|
V: Ins
c4g4B4g4c4g4d4g4|C4C4C4C4C4C4C4C4|C4C4C4C4C4C4C4C4|
% verse
V: Vocal
"Cm"z32|"D#"z16"A#"z16|"Gm"z32|"G#"z16"A#"z16|
V: Ins
c24c4d4|e8g8f8e8|f4c24-c4-|c4z4a4d2e2d8B8|
V: Vocal
"Cm"z32|"D#"z16"A#"z16|"Gm"z32|"G#"z16"A#"z16|
V: Ins
c24c4d4|e8g8a8b8|a4g24-g4-|g4z4c'2b2a2g2f2e2d2c2B2A2C2B,2|
V: Vocal
"Fm"z32|"Gm"z32|"Fm"z32|"G"z32|
V: Ins
F12F4G4A12|G4F12E4D12|F12F4G4A12|G8=B8d8B8|
V: Vocal
"Fm"z32|"Gm"z32|"Fm"z32|"G"z32|
V: Ins
F12F4G4A12|G4F12E4D12|F12F4G4A12|G4=B4d4g4b4d'4b4g4|
V: Vocal
"Cm"z32|"D#"z16"A#"z16|"Gm"z32|"G#"z16"A#"z16|
V: Ins
c24c4d4|e8g8f8e8|f4c24-c4-|c4z4a4d2e2d8B8|
V: Vocal
"Cm"z32|"D#"z16"A#"z16|"Gm"z32|"G#"z16"A#"z16|
V: Ins
c24c4d4|e8g8a8b8|a4g24-g4-|g4z4c'2b2a2g2f2e2d2c2B2A2C2B,2|
V: Vocal
"Fm"z32|"Gm"z32|"Fm"z32|"G"z32|
V: Ins
F12F4G4A12|G4F12E4D12|F12F4G4A12|G8=B8d8B8|
V: Vocal
"Fm"z32|"Gm"z32|"Fm"z32|"G"z32|
V: Ins
F12F4G4A12|G4F12E4D12|F12F4G4A12|G4=B4d4g4b4d'4b4g4|
% interlude
V: Vocal
"Cm"z32|"Cm"z16"A#7"z16|"A#7"z32|"Cm7"z32|
V: Ins
C2B,2C2G,2F,2G,2C2B,2C2G,2F,2G,2C2B,2C2G,2|F,2G,2C2B,2C2G,2F,2G,2B,2A,2B,2F,2E,2F,2B,2A,2|B,2F,2E,2F,2B,2A,2B,2F,2E,2F,2B,2A,2F,2F,2E,2F,2|C2B,2C2G,2F,2G,2C2B,2C2G,2F,2G,2C2B,2C2G,2|
V: Vocal
"Cm7"z16"Fm7"z16|"Fm7"z32|"Cm7"z32|"Cm7"z16"Fm7"z16|
V: Ins
F,2G,2C2B,2C2G,2F,2G,2B,2A,2B,2F,2E,2F,2B,2A,2|B,2F,2E,2F,2B,2A,2B,2F,2E,2F,2B,2A,2B,2F,2E,2F,2|C2B,2C2G,2F,2G,2C2B,2C2G,2F,2G,2C2B,2C2G,2|F,2G,2C2B,2C2G,2F,2G,2B,2A,2B,2F,2E,2F,2B,2A,2|
V: Vocal
"Fm7"z32|"Cm7"z32|"Cm7"z32|"Fm7"z32|
V: Ins
B,2F,2E,2F,2B,2A,2B,2F,2E,2F,2B,2A,2B,2F,2E,2F,2|B,2B,2C2G,2F,2G,2C2B,2C2B,2C2G,2F,2G,2C2B,2|C2B,2C2G,2F,2G,2C2B,2C2B,2C2G,2F,2G,2c2c2|c2c2c2c2c2c2c2c2B2B2B2B2B2B2B2B2|
V: Vocal
"Fm7"z32|
V: Ins
G2G2G2G2G2G2G2G2F2F2F2F2F2F2F2F2|
% verse
V: Vocal
"Cm"z32|"Cm"z16"Gm"z16|"Gm"z32|"Cm"z32|
V: Ins
c4d2e4f6e2d4e6d2c2-|c2d6c2B6c8z2G6-|G32|c4d2e4f6e2d4e6d2c2-|
V: Vocal
"Cm"z16"Gm"z16|"Gm"z32|"Fm"z32|"Fm"z16"Cm"z16|
V: Ins
c2d6c2B6g8z2g2z4|z8g4z4g4g4g4f4|F6G6A6B6c6d2-|d4e6f6e8d2e2c4-|
V: Vocal
"Cm"z32|"Gm"z32|"Gm"z16"G"z16|"G"z32|
V: Ins
c24-c4z4|G24A8-|A4G6d6=B8z2G2A4|z2E2^F2F2=A4A2G2=E2G2=B2B2B2A2F2A2|
V: Vocal
"Cm"z32|"Cm"z16"Gm"z16|"Gm"z32|"Cm"z32|
V: Ins
c4d2e4f6e2d4e6d2c2-|c2d6c2B6c8z2G6-|G24-G2z6|c4d2e4f6e2d4e6d2c2-|
V: Vocal
"Cm"z16"Gm"z16|"Gm"z32|"Fm"z32|"Fm"z16"Cm"z16|
V: Ins
c2d6c2B6g8z2g2z4|z8g4z4f4g4g4f4|F6G6A6B6c6d2-|d4e6f6e8d2e2c4-|
V: Vocal
"Cm"z32|"Gm"z32|"Gm"z16"G"z16|"G"z32|
V: Ins
c24-c4z4|G24A8-|A4G6d6=B8z2G2A4|z2^F2F2F2=A4A2G2G2G2=B2B2B2A2F2A2|
% interlude
V: Vocal
"Cm"z32|"Cm"z16"Gm"z16|"Gm"z32|"Cm"z32|
V: Ins
c'2b2g2^f2=f2e2c2B2G2^F2=F2E2C2G2C2G2|C2G2C2B2C2B2C2B2c'2b2g2^f2=f2e2c2B2|G2^F2=F2E2C2G2C2G2C2G2C2B2C2B2C2B2|c'2b2g2^f2=f2e2c2B2G2^F2=F2E2C2G2C2G2|
V: Vocal
"Cm"z16"Gm"z16|"Gm"z32|"F"z32|"F"z16"Cm"z16|
V: Ins
C2G2C2B2C2B2C2B2c'2b2g2^f2=f2e2c2B2|G2^F2=F2E2C2G2C2G2C2G2C2B2C2B2C2B2|F,2G,2B,2C2E2F2^F2G2B2c2e2=f2b2f2b2f2|b2f2=a2f2a2f2a2f2c'2b2g2^f2=f2e2c2B2|
V: Vocal
"Cm"z32|"Gm"z32|"Gm"z32|"Gm"z32|
V: Ins
G2^F2=F2E2C2G2C2G2C2G2C2B2C2B2C2B2|B,2=A,2G,2F,2E,2F,2^F,2G,2B,2C2E2=F2^F2G2B2c2|e2f2e2c2B2G2^F2=F2c2B2G2c2B2G2c2B2|G2c2B2G2c2B2G2G2G2G2c2B2G2^F2=F2E2|
% verse
V: Vocal
"Cm"z32|"Cm"z16"Gm"z16|"Gm"z32|"Cm"z32|
V: Ins
c4d2e4f6e2d4e6d2c2-|c2d6c2B6c8z2G6-|G24-G2z6|c4d2e4f6e2d4e6d2c2-|
V: Vocal
"Cm"z16"Gm"z16|"Gm"z32|"Fm"z32|"Fm"z16"Cm"z16|
V: Ins
c2d6c2B6g8z2g2z4|z8g4z4g4g4g4f4|F6G6A6B6c6d2-|d4e6f6e8d2e2c4-|
V: Vocal
"Cm"z32|"Gm"z32|"Gm"z16"G"z16|"G"z32|
V: Ins
c24-c4z4|G24A8-|A4G6d6=B8z2G2A4|z2^F2E2F2=A4A2G2=E2G2=B2B2B2A2F2A2|
% interlude
V: Vocal
"Cm"z32|"Cm"z32|
V: Ins
C4C4C4C4C4C4C4C4|C4C4C4C4C4C4C4C4|
% outro
V: Vocal
"Cm"z32|"Cm"z32|"D#"z32|"D#"z32|
V: Ins
e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|
V: Vocal
"Fm"z32|"G#maj7"z32|"A#"z32|"Cm"z32|
V: Ins
e4g4f4g4e4g4d4g4|c4g4B4g4c4g4d4g4|e4g4f4g4e4g4d4g4|g4z24z4|
V: Vocal
"Cm"z32|
V: Ins
Z|
Thanks for sharing the sample. I also observed extra predicted beats on this track.
- Current SheetSage2 make errors in beat, meter, and melody prediction, including extra beats and notes.
- For precise alignment with the original audio, I recommend the original MIDI export, which better preserves the predicted note timestamps. Current ABC export uses a single average tempo for the whole track. Incorrect beat counts can affect that tempo and cause alignment drift.
I have attached the ABC and full MIDI (melody and chords) from my run for reference.
https://drive.google.com/file/d/1ZQIkc0jySYjP-JrddWif8j489Np1erZh/view?usp=sharing
https://drive.google.com/file/d/13dq3U6IllWLgu2nQiB6_LMGp9OjqVAY0/view?usp=sharing
I think I've found the cause of the timing going wrong after the first 30–50 s. It's numerical, not a model error, and it's a small fix.
Cause. RotaryEmbedding.forward in modeling_mert2.py builds the rotary tables with torch.einsum("i,j->ij", positions, self.inv_freq). Under CUDA bf16 autocast, the setting in benchmark_results.json and presumably the training setting, that einsum runs in bfloat16. bfloat16 rounds the frame positions: in steps of 32 by the end of a 300 s window. It also rounds the angles, which go up to 7,500 radians. The model has learned to rely on those rounded tables.
Any run where that einsum isn't autocast gets exact fp32 tables instead, and the timing drifts after about the first 30 s of each window. Going by the code, that's every non-CUDA device (inference_autocast returns a null context), dtype="fp32" on CUDA, and loaders that cast the model to fp16/bf16 without autocast. I'd guess the ComfyUI workflow is one of these, but I haven't checked it.
Test. RWC-MDB-P-2001 No. 9, released code, paper preset, full prompts, CPU fp32. Scored against the RWC annotations, with only RotaryEmbedding patched:
| beat F1 (70 ms) | vocal pitch-class F1 (50 ms) | chord majmin | |
|---|---|---|---|
| as released | 0.33 | 0.22 | 0.86 |
| bf16-rounded rotary tables | 0.99 | 0.65 | 0.95 |
Across 24 RWC Popular songs, the same change in an MLX port takes vocal pitch-class F1 from 0.21 to 0.84, in line with the 82.51 on the model card.
Workaround (after loading the model):
import types, torch
def rotary_bf16(self, hidden_states):
length = hidden_states.shape[1]
positions = torch.arange(length, dtype=torch.float32, device=hidden_states.device).to(torch.bfloat16)
freqs = torch.einsum("i,j->ij", positions, self.inv_freq.to(torch.bfloat16)) # as under bf16 autocast
angles = torch.cat((freqs, freqs), dim=-1)
cos, sin = angles.cos()[:, None, None, :], angles.sin()[:, None, None, :]
return cos.to(hidden_states.dtype).permute(1, 0, 2, 3), sin.to(hidden_states.dtype).permute(1, 0, 2, 3)
for m in model.modules():
if type(m).__name__ == "RotaryEmbedding":
m.forward = types.MethodType(rotary_bf16, m)
Suggested fix: compute the tables with this explicit bf16 rounding on every device and dtype, so that every setup gets the same tables the model was trained with. I'm happy to open a PR.
@Makazi @Zueuk , if you can try the snippet on the tracks that drifted for you, that would confirm it on your setups.