© All copyrights for the representative image are retained by the authors. All publicly available datasets used in the results have been attributed in the paper.
Most hand-tracking methods rely on a learned, data-driven model called MANO. One problem with MANO is it doesn’t account for unseen hand shapes that deviate from the training set, which affects hand-tracking accuracy.
That’s why graduate student Pratik Kalshetti and Prof. Parag Chaudhuri of the Indian Institute of Technology Bombay (IIT Bombay) worked hard to introduce a new hand-shape model called aMANO (“adaptive MANO”). Their research took second place in the Graduate Division of the ACM Student Research Competition and was featured in the SIGGRAPH Posters interactive forum at SIGGRAPH 2022 in Vancouver.
In this interview, Pratik talks about his love for computer graphics, his journey as a graduate student, and his work on aMANO:
SIGGRAPH: What excites or inspires you about the computer graphics industry?
Pratik Kalshetti (PK): Computer graphics inspires me because it allows us to visualize our ideas and thus enable a much more intuitive communication experience. For example, it helps storytellers render their stories in images, it aids teachers in going beyond traditional 2D explanations, and, as a student, it helps me understand and convey my research ideas via 3D animations. And now, with the advent of virtual reality, there are newer ways of communicating our ideas and interacting with the virtual world. I am excited to be a part of this fantastic computer graphics community!
SIGGRAPH: As a student at IIT Bombay, you have been working on hand tracking for virtual reality with Prof. Parag Chaudhuri. Can you talk about the importance of that collaboration?
PK: I began my journey as a graduate student at IIT Bombay with Parag, where he introduced me to the problem of hand tracking and its importance in virtual reality. It was my first exposure to computer graphics as well as to research. He guided me as I paved my path in understanding the field of hand tracking and eventually developing a hand-tracking system. As part of the ViGIL lab at IIT Bombay, he provided ample computing resources and required cameras for my research. Thanks to Parag’s support, I could attend my first SIGGRAPH as a student volunteer in 2018, which stimulated me to pursue research in computer graphics. Over these years, I have been fortunate to have Parag as my Ph.D. supervisor.
SIGGRAPH: In your Posters project at SIGGRAPH 2022, you introduced a new hand-shape model called aMANO (“adaptive MANO”) that builds upon limitations of existing hand-tracking solutions. In essence, how is aMANO different?
PK: Most existing hand-tracking methods rely on a learned hand shape model, MANO. However, this data-driven hand model cannot adapt to unseen hand shapes that have significant deviations from the training set, and this adversely affects tracking that is based on the model. We tackle this problem by introducing a new shape model that augments MANO’s shape space with local scale adaptation. Specifically, we use a set of local scale parameters that scale each of the bones in the hand model and a modified skinning function to handle the local scale adaptation. As a result, aMANO gracefully adapts MANO to unseen hand sizes, which aids in accurate tracking.
SIGGRAPH: Can you talk about the importance of being able to calibrate the mesh used in hand-shape modeling to include substantially different hand sizes?
PK: A user-specific (or “calibrated”) hand-shape model is essential for accurate hand tracking. The calibration process involves registering a template hand model to the input depth point cloud using non-linear optimization. This calibration process requires a good initialization of the template hand model. However, if the input point cloud is of a substantially different hand size (for example, a child’s hand), then the optimization procedure gets stuck in a local minimum and cannot converge to a correct solution. Our aMANO hand-shape model allows precise control to scale its bones to match estimated bone lengths from a pose estimation method, and thus provides a good initialization. As a result, we can calibrate aMANO to users with different hand sizes, including that of a child’s hand, which was not possible until now.
SIGGRAPH: Congratulations for taking Second Place in the Graduate Division of the ACM Student Research Competition! How did it feel to be recognized by a panel of judges and the ACM SIGGRAPH community for your work?
PK: We had been working on this idea for quite some time and believed that it had the potential to make a substantial impact. It has been very rewarding to have the idea acknowledged as interesting and important by an expert panel of judges. The Student Research Competition provides a great platform to showcase our work and receive valuable feedback from the SIGGRAPH community.
SIGGRAPH: What do you hope to be doing a few years from now?
PK: I have always wanted to work on important problems impacting a large and diverse group of people. I am particularly interested in solutions that revolve around computer graphics and vision, and my Ph.D. journey has been crucial to acquiring the skills and mindset to tackle challenging problems. I hope to work with a team whose focus is aligned with this goal.
Innovators like Pratik and Parag brought ideas and excitement to SIGGRAPH 2022. There’s still time to experience the event in the virtual conference platform, where registrants can access research from Technical Papers to Posters and hundreds of hours of on-demand content until 31 October.
Pratik Kalshetti is in his sixth year of the M.Tech.+Ph.D. dual degree program in computer science and engineering at the Indian Institute of Technology Bombay. He is working on hand tracking for virtual reality, advised by Prof. Parag Chaudhuri. His research interests are computer graphics and vision. He won the Qualcomm Innovation Fellowship India 2017 and was also awarded TCS Research Scholar Fellowship in 2019. At SIGGRAPH 2022, he won second place in the ACM Student Research Competition. (He also won third place in that category in 2019.) Pratik completed his B.Tech. in computer science and engineering from the Indian Institute of Technology Jodhpur. He likes to swim and play badminton.
Parag Chaudhuri is an associate professor in the Department of Computer Science and Engineering at IIT Bombay. His current research centers around the simulation of virtual characters and natural phenomena to populate virtual worlds. His team develops methods to create, interact with, and understand visual data. Prior to joining IIT Bombay, he was a postdoctoral researcher at MIRALab, University of Geneva, and he holds a Ph.D. from IIT Delhi. He loves teaching computer graphics to enthusiastic students.
