NVIDIA's Latest Deep Learning Model: Automatically Choreographing Dance Based on Music

2019-12-11

NVIDIA researchers have collaborated with the University of California and Mercer University to develop a deep learning-based model that can automatically choreograph diverse, stylistically consistent, and beat-matching new dance movements.


In a paper published this week in 2019, NVIDIA researchers stated, "This is a challenging but interesting generative task with the potential to assist and expand artistic and sports content creation, such as performing arts, artistic gymnastics, and figure skating."The core of this work is a decompose-to-compose framework, which first learns how to move and then learns how to assemble.



In the top-down decomposition stage, the team uses a motion beat detector to normalize dance units segmented from actual dance sequences. They then train DU-VAE to model dance units. In the bottom-up composition stage, given a pair of music and dance, the team uses MM-GAN to learn how to organize dance units based on the given music. During the testing stage, researchers extract style and beat from the input music and cyclically synthesize a series of dance units. Finally, a beat shaper is applied to the generated dance unit sequence to render the output dance.


To train the generative adversarial networks used in the system, the team collected dance videos representing three representative dance categories, including ballet, zouk, and hip-hop. The team obtained a total of over 361,000 clips or approximately 71 hours of dance footage.


For pose handling, the team used OpenPose, an open-source, real-time multi-person system developed by Carnegie Mellon University that can jointly detect body, hand, face, and foot keypoints on a single image.


[video width="1152" height="720" mp4="https://www.atyun.com/uploadfile/2019/12/Dancing-to-Music-NeurIPS-2019.mp4"][/video]

This work was done using the PyTorch deep learning framework and NVIDIA V100 GPU. For inference, the same GPU used during training was used. In future work, the team plans to incorporate more dance styles, such as pop dance and partner dance. The researchers state in the paper, "In this work, we propose a decompose-to-compose learning framework for synthesizing music dance. In the top-down decomposition stage, we teach the model how to generate and separate basic dance units. In the bottom-up composition stage, we guide the model to meaningfully compose patterns of basic dance movements conditioned on input music. We leverage motion beats and music beats to make the generated dance temporally aligned with the accompanying music. Extensive qualitative and quantitative evaluations demonstrate that the synthesized method is not only realistic and diverse but also exhibits style consistency and beat matching. In future work, we will continue to collect and absorb more dance styles, such as pop dance and partner dance."


The source code and models have been released on GitHub.

RECENT AI NEWS

RECENT AI TOOLS