Start moving and grooving with a production-ready, music-driven dance motion synthesis system. The research in SIGGRAPH 2021 Technical Paper “ChoreoMaster: Choreography-oriented Music-driven Dance Synthesis” presents a method — ChoreoMaster — that can automatically generate a high-quality dance motion sequence to accompany the input music in terms of style, rhythm, and structure. In this interview, researcher Kang Chen dives into the inspiration behind the project and encourages others to continue their hard work and submit to the Technical Papers program.
SIGGRAPH: Share some background on ChoreoMaster. What inspired this research?
Kang Chen (KC): I’m currently leading a research team at NetEase Games AI Lab, which is a research and development (R&D) department in China’s second-largest game company. Our main job is to explore the applications of cutting-edge AI and graphics technologies in the game industry, and ChoreoMaster is one of our successful attempts in this area.
The first thing I learned at a game company is that the production of high-quality 3D animation is incredibly costly and inefficient, especially for dance animations because of the extra cost for inviting a professional artist to choreograph a dance and an experienced dancer to perform it. One minute of dance animation can easily cost thousands of (U.S.) dollars. So, we were thinking: Why not build an automated dance synthesis tool? At first, we thought it would be an easy task, because, as graphics guys, we knew a bunch of motion-synthesis methods. However, after some failed attempts, we realized that there was a huge gap between a research prototype and a practical productive system. None of the existing algorithms could be successfully adapted for this purpose. Since we all agreed that it was definitely a valuable research topic, we then decided to dig deeper toward this direction.
SIGGRAPH: Break down how you developed your choreography-oriented choreomusical embedding framework approach. How many people were involved? How long did it take?
KC: Compared with a typical SIGGRAPH project, ChoreoMaster had a much longer development time. We started the research very early (i.e., late 2018) and invested many development resources into it. Altogether there were seven people fully involved, including three researchers, one engineer, one choreographer, and two artists — not counting the support staff who helped in building a high-quality dance animation dataset (e.g., web crawler engineers, MoCap artists, data labelers, etc.) and numerous animation artists from different game studios in NetEase who gave us lots of worthy feedback and constructive suggestions in improving ChoreoMaster.
The idea of a choreography-oriented choreomusical embedding framework didn’t come from thin air. We have tested almost all known technical solutions in 3D animation synthesis, however, none could robustly produce acceptable dance animations by professional artists. Our first system prototype was built upon a traditional graph-based solution, which could fluidly piece together a set of dance movement units, but the result overall did not look like a well-composed artform. So, we switched to modern deep-generative, dance-motion synthesis frameworks, but the poor controllability and unstable performances fundamentally prevented the application of such methods in practical production environments. After many failed attempts, we found that combining modern deep features with the traditional graph-based framework was currently the best practical solution to this problem. Then, through multiple rounds of iteration with professional artists, we further integrated various choreographic rules into our algorithm and eventually developed the proposed choreography-oriented choreomusical embedding framework.
SIGGRAPH: What challenges did you face while developing this research?
KC: We have faced many difficulties during the development. Fortunately, investing more development resources solved most of them. For instance, lacking data was the first big problem. We then spent around 1.5 years and hundreds of thousands of (U.S.) dollars in collecting and processing data, then finally built a high-quality dance motion dataset — big enough for carrying on this research.
The problems that I would consider true challenges were always related to the quality of the synthesis results. The goal of this research was to make a productive tool, not just an academic paper or a fancy prototype, which means dance motions generated by our algorithm must get the recognition from professional artists. This was much harder than we had expected. When we tried to sell this technology to animation artists in our company, a typical response was, “It’s amazing and promising, but the results still need to be improved to meet the quality standard of game assets.” However, when we asked them to be more specific, they could not tell which part was problematic either. All we learned was that the generated dances failed to give them the right feel, though those results already looked quite good in our technical [team’s] eyes. We got stuck several times because we didn’t know where the problem lied, not to mention how to improve the algorithm. The turning point came when we had a consultation with an experienced choreographer, who impressed us with many examples showing a well-composed dance should follow certain rules. We then began systematic study of the theory of choreography and even hired a full-time choreographer for this research. With more and more choreographic rules integrated into the algorithm, the generated results were finally recognized by all professional artists in our company.
SIGGRAPH: How do you envision this research being used in the future? What problems does it solve?
KC: The direct application of this research is to produce high-quality, 3D dance animation assets for game and film projects. Conventionally, producing a dance animation for a complete piece of music (i.e., around 3–4 minutes) would cost thousands of (U.S.) dollars and take at least one month. In contrast, using ChoreoMaster, the same task can be done within minutes. Actually, ChoreoMaster has already been widely used within NetEase Games, and has successfully produced hours of dance assets for many game projects. We hope this technology will one day change the industry’s conventional dance animation production pipeline.
Further, ChoreoMaster can also be used as a computer-assisted dance composing tool. It significantly lowers the bar of choreography. Actually, we have received numerous positive feedback from various users — even professional artists may derive some inspiration from the novel results generated by ChoreoMaster.
SIGGRAPH: What excites you most about your research?
KC: To me, the most exciting part of this research is that dances synthesized by our algorithm can be practically used in real game products. As a graphics researcher and a lover of video games, nothing can give me a greater sense of accomplishment than that. This is also the reason why I chose to do research in a game company rather than a university.
SIGGRAPH: What did you enjoy most about participating in SIGGRAPH 2021?
KC: As part of the graphics community, I have attended many SIGGRAPH/SIGGRAPH Asia conferences since 2012. Actually, I enjoy almost all parts of SIGGRAPH. For example, the Technical Papers program — which is the “main course” to me — always gives me inspiration for innovative new research topics, Exhibitor Sessions offer a good channel to explore the latest hotspots in the graphics industry, and the Computer Animation Festival Electronic Theater (my favorite part) is always a visual feast. I really hope SIGGRAPH will become better and better.
SIGGRAPH: What advice do you have for someone who wants to submit to Technical Papers for a future SIGGRAPH conference?
KC: I would say that the graphics community is open and inclusive. If you have a new idea that really works, just give it a shot. At the same time, it is also important to notice that SIGGRAPH is the top-ranked and most influential academic conference in computer graphics, which means sufficient evaluation of the method and proper comparisons with existing works are always necessary to get your paper accepted. So, be sure to leave plenty of time for evaluations and comparisons.
Missed out on SIGGRAPH 2021? On-demand registration is open through 18 October so you can access the latest in computer graphics and interactive technology, including research like ChoreoMaster!
Kang Chen is a researcher at NetEase Games AI Lab. He is currently leading a research team to explore potential applications of AI and graphics technologies in the game industry. His research interests include 3D modelling, geometry processing, motion capture, animation, etc. He has published six SIGGRAPH/SIGGRAPH Asia papers in these areas. Chen received a Ph.D. in computer graphics from Tsinghua University under the supervision of Professor Shi-Min in 2017, and Bachelor’s degree in computer science from Nanjing University in 2012.
