“Virtual Reality Mirror Therapy Rehabilitation for Post-Stroke Patients” © 2019 UMBC
At SIGGRAPH 2019, Levan Sulimanov (University of Maryland, Baltimore County) and Dr. Marc Olano (University of Maryland, Baltimore County and SIGGRAPH 2020 Real-Time Live! Chair) shared a poster titled “Virtual Reality Mirror Therapy Rehabilitation for Post-Stroke Patients.” The research presented turns traditional mirror therapy into virtual reality (VR), providing new exercises and a customizable experience for users.
We caught up with Sulimanov to learn what inspired this research and how mirror therapy in VR can assist users in the future.
SIGGRAPH: Share a bit about the process for developing your poster. What inspired you to pursue the specifice topic? What sparked the idea to modify traditional mirror therapy?
Levan Sulimanov (LS): The initial idea was focused on implementing software aid for students with ADHD, or
attention deficit hyperactivity disorder, using augmented reality (AR) to decrease the number of distractive objects in the scene by virtually covering them with overlaying materials.
Further discussions led us to talking with physical therapists, who shared information about ongoing projects for pose tracking during exercises using Kinect. As a result, mirror therapy was mentioned. The desire to use AR in medical use switched to using VR to help people who have suffered from strokes.
This idea had originally been put on hold due to [timing] and was raised again when Dr. Olano shared that Unreal Engine was giving out a grant for developing a research project using Epic Games’ game engine. From there, our idea was chosen and we received a grant from Epic Games to proceed with the development.
SIGGRAPH: How did you develop your research? How long did it take? What was the biggest challenge?
LS: The research implementation was developed over the course of three months and focused on three main steps:
- Transposing mirrored hand mesh onto the scene to follow its host (the one from which the mirrored hand is reflected from).
- Learning about traditional mirror therapy techniques and methods.
- Designing new levels and exercises in VR, inspired by traditional mirror therapy uses.
The biggest challenge was choosing the movements of the reflected hand and choosing appropriate exercises that would both fit the therapy layout and practical use of exercising the hand enough to recover. Also, another challenge was finding an approval toward experimenting with such applications and using them on actual patients.
SIGGRAPH: In “Virtual Reality Mirror Therapy,” patients are “surrounded by a customizable virtual environment.” Why is it important for this environment to be customized for each user?
LS: A customizable environment allowed us to create a more dynamic use of such therapy, since physical therapists could easily move and change the layout based on their patient’s needs. For example, if a patient did not have a finger on one hand, this could easily be reimplemented inside the engine to create better immersion. Alternatively, if doctors wanted to change the table design or spawn additional objects in the scene, this would not require any monetary purchases, allowing the process be fluid and fast.
SIGGRAPH: What excites you most about “Virtual Reality Mirror Therapy”?
LS: I am most excited by its immersion, which allowed us to proceed with new exercises that were not previously available in traditional mirror therapy. Patients now can do cross-hand movements and interact with both of their hands anywhere in the scene due to the removal of a physical mirror from the scene.
VR mirror therapy also allows us to proceed with outpatient (at-home use) practices, due to the availability of learning and observing the patient in a virtual environment at a distance. This means that a patient can proceed with therapy at home and therapists can observe progress from the office.
VR mirror therapy also allows us to reduce the cost for such therapy and increase the number of objects easily available in the scene, together with any desired updates in the scene or game-play that could be reimplemented and updated on the go.
SIGGRAPH: What does working with adaptive and assistive technology mean to you?
LS: I enjoy bringing to life new innovations that could actually reduce disabilities present in people.
SIGGRAPH: How do you envision “Virtual Reality Mirror Therapy” helping users in the future?
LS: People who suffered from a stroke could be prescribed with VR mirror therapy and receive a VR headset and VR-capable computer (either for rent or to purchase).
Once the devices are set up, with or without the help of professional installation, patients can proceed with therapy and receive updates from within a VR environment, if needed, and be monitored by physical therapists or additional software, similar to pose tracking that can inform them of progress in exercises.
SIGGRAPH: What is your favorite memory from the SIGGRAPH conference?
LS: I enjoyed the Experience Hall where people could experience and try out new technologies, such as force feedback gloves and Arque (the robotic tail for aiding in lifting heavy objects and balancing).
Do you have cutting-edge research to share with the SIGGRAPH community? Submit your latest work to the SIGGRAPH 2020 Posters program by 28 April.
Marc Olano has been doing research on uses of graphics hardware for real-time 3D graphics and non-graphics applications for over 25 years. He is currently is an associate professor in the CSEE department at the University of Maryland, Baltimore County (UMBC) and director of UMBC’s game development track and 3D photogrammetric scanning facility. He also is editor in chief of the Journal of Computer Graphics Techniques and chair of the Real-Time Live! program for SIGGRAPH 2020. Olano received his Ph.D. from the University of North Carolina for the first high-level shading language for graphics hardware, now one of the defining features of all graphics accelerators. After his Ph.D., he worked at Silicon Graphics on some of the earliest commercial efforts to develop programmable graphics hardware. Since coming to UMBC, his work has focused on applications of this now-ubiquitous hardware for computation, visualization, and, of course, games. In support of the games aspects of that research focus, he spent a year at Firaxis games working on “Civilization V,” a year at Epic Games working with the Unreal Engine 4 rendering team, and has also worked with Activision, 2K, Big Huge Games, and Oxide Games.
Levan Sulimanov is a recent graduate of the University of Maryland, Baltimore County, who spent his college years focusing on research development in graphics and machine learning. Currently, he works as research engineer and specializes in machine learning and computer vision.
