Multi-Task Multi-Frame Visual Piano Transcription
Abstract
Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.
Community
We introduce V2N (Video to Notes), the first complete visual piano transcription (VPT) system. From top-view video alone, with no audio, it predicts onset, offset, key hold, and velocity for every note.
Why video? When the sustain pedal is down, a note keeps sounding long after the key lifts, so audio systems predict pedal-extended offsets rather than the physical key release. The camera sees the key itself, including how hard it is struck. Prior visual systems, though, focus on onset from short windows, so offset accuracy lags onset by a wide margin and note-level velocity has not been reported. V2N closes both gaps with a shared temporal backbone and task-specific heads for onset, offset, key hold, and velocity, trained with per-frame supervision instead of only at the window center.
Ablations show multi-task supervision enables offset and velocity while improving onset, and longer temporal context helps further. V2N sets new state of the art on PianoVAM and R3. To appear at ISMIR 2026.
🎬 Demo: https://huggingface.co/spaces/PianoVAM/V2N
📄 Paper: https://arxiv.org/abs/2608.03419
💻 Code: https://github.com/yonghyunk1m/V2N
Get this paper in your agent:
hf papers read 2608.03419 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 1
Collections including this paper 0
No Collection including this paper