Our paper collects the first large-scale 3D hand motion dataset containing 10 hours, 153 pieces of piano music performed by 15 elite-level pianists, along with synchronized audio and key pressing events; we also proposes a method that can control a physically simulated hand to play novel pieces "unheard" from the training set.
3D Dataset Visualization (with ♫)
Piano Sonata No. 2 (1st Movement)Goldberg Variations, BWV 988: AriaScales
Concerto in D Minor, BWV 1052: I. AllegroRhapsody in BlueA Slow Boat to China
Here we show a few sample motions in our dataset using a 3D interactive visualizer. Please download the dataset to see full recordings of all 153 pieces.
Synthesized Results
Für EliseRondo Alla TurcaMerry Christmas Mr. LawrenceMazurka in B flat major Op. 7 No.
1
Results from our method. We train policies to play new music scores unseen from the dataset in physics simulation.
Overview Video
Dataset Collection
Our markerless motion capture system first reconstructs initial 3D motions by triangulating 2D keypoints from multi-view videos. Then the motions are regularized by fitting MANO parameters. Finally, they are refined via inverse kinematics using the MIDI key-pressing data recorded by specialized sensors.
Reconstructed motion (shown as white hands) accurately matches the original video in multiple camera angles.
Method
Given a novel sheet music in MIDI format, we first create a reference motion ensemble of 3D hands using a trained diffusion model and motion retrieval from the dataset. We then train a control policy along with two discriminators to imitate the reference motion ensemble while achieving musical accuracy in physics simulation.
Acknowledgments
We thank Yifeng Jiang and Jiaman Li for providing detailed feedback on the paper. This work was supported in
part by the Wu-Tsai Human Performance Alliances, Stanford Institute for Human-Centered Artificial
Intelligence and Roblox. We thank the 15 pianist volunteers for their essential contributions to this study.
To protect their privacy, they remain unnamed, but their participation was invaluable to our research.
BibTeX
@inproceedings{wang2024piano,
title = {FürElise: Capturing and Physically Synthesizing Hand Motions of Piano Performance},
author = {Ruocheng Wang and Pei Xu and Haochen Shi and Elizabeth Schumann and C. Karen Liu},
booktitle = {SIGGRAPH Asia 2024},
year = {2024}
}
