photo by John Fujii © 2015 ACM SIGGRAPH
As I sat down to write this, I reflected on a story I wrote exactly one year ago for the ACM SIGGRAPH blog and how the state of virtual reality (VR) has evolved over the past 12 months. At this time last year, I was mainly focused on the sociological model of the diffusion process and innovation curve. Given the progress we have seen for virtual and augmented reality (AR) not only as a tool, but a medium, over the past year, I am increasingly excited about the many directions creators of immersive content are taking.
To be a successful medium and a successful technology today, we often think that projects must rake in dollars and immediately become broadly adopted. Interestingly, virtual and immersive realities go way back to 1968 with the first known head mounted display (HMD) by Ivan Sutherland and Bob Sproull, and even before then if you consider “diorama” theatre and stage magic. The immersive realities have had a long, rocky journey — thanks in part to the very multifaceted nature of the medium and the technological model of constant iteration. Yet it is tempting to be in a rush, searching for the next “killer app” and to be the first VR experience to “finally do it right.” If we stop for a moment to consider that VR may be one of the most incredibly complex modes of communication and sensory experiences we have ever experimented with, it doesn’t make much sense to rush towards an invisible finish line.
Classic cinema and filmmaking as a language took over 100 years to develop, and some of the most iconic films ever made, like “Citizen Kane,” actually lost considerable money in its day. This is because, at the time, these films were focused on artistry and developing a new language that had never been seen before, and it took time to understand the underlying cognitive factor and aesthetics of the medium. Today, technology is developing at a rate more rapid than we can take the time to explore in-depth for content creation. What this means is that there is often temptation to build for the technology, rather than fully understand the cognitive, sensory, and psychological processes that drive our experiences. That said, there are several outfits that are doing exactly this. At last year’s VR Village, we saw numerous projects that involved full-body interaction and sensory experiences to explore the potential of the medium. And, this year, there are topics that may have the potential to “up the game” once again. The areas of art and narrative are well-covered by most press and presentations already, and story-based VR has tremendous potential (as proven by last year’s VR Village contributor, Google, through “Pearl” — the first VR project nominated for a “Best Short Film (Animated)” Academy Award). So for now, let’s focus more on some of the lower profile, but extremely engaging areas of interest:
A new language for social interaction
Language and social interactions constantly evolve, often unnoticed. The “emoji” is now considered by many to be its own language and it makes sense: in written language where we need to express ourselves in minimal words, what better way to soften your tone or express embarrassment than with a smiley icon? And, emojis do not even need to be human faces or bodies to work. Virtual and augmented reality allows for another mode of communication and telepresence, where due to strong properties of immersive “presence,” we connect with others through the technology interface and content. Collaborative, multi-user experiences in VR have been explored before and continue to evolve, with users drawing and creating art together, exploring environments with multiple people, and playing multi-user mixed reality gaming. As HMDs and AR glasses become wireless and nomadic, users will have more freedom to move and interact with their environment, unbound by computer cables.
Listening to our inner voices
VR doesn’t simply let us get in touch with one another, the very nature of it being an immersive, insightful medium allows for more “inner activity.” As we become more comfortable with wearable and tangible computing devices, we discover opportunities to learn more about our internal workings. Take fitness trackers and consumer electroencephalography (EEG) as an example: We are constantly in pursuit of self-improvement, clarity, and focus; hence, a desire to find out more about how we function and what we can do to improve. VR and AR have a tremendous potential to do exactly this, and feed all this information, in a filtered manner, back to us. The wearable nature of HMDs, glasses, and wireless audio input means we can combine bio- and neuro-feedback interfaces naturally into the experience. Imagine wearing a display and seeing your environment respond to your brain’s frequency bands, or sync your breathing to another user to complete a virtual task. In terms of well-being and insight, VR is able to do this exceptionally well. Our lab is currently researching how experiences in VR can lead to cognitive shift and a more caring view of our planet. And, a team at the University of Barcelona is exploring out-of-body experiences in VR that help reduce anxiety around death.
Improving health care and patient-practitioner communication
We are fortunate to be living in an era where we are living longer, yet this comes at a cost to society. The strain of demand on healthcare providers is greater than ever before, and a high-paced lifestyle ensures many of us at some point will need to overcome illness or injury that we can barely pause to recover from. In the event that it becomes chronic or fails to respond to treatment, this battle is only made worse. We have seen successful immersive applications in the past that train medical staff, and VR games that help distract children from pain. Another excellent application will be in bridging patient-medical practitioner communication, through a form of immersive telepresence or through education aimed at helping to create empathy (for the practitioner) or explaining/visualizing medical procedures and treatments (for the patient).
Learning and visualization
One pet project of mine is creation of virtual, immersive memory palaces that can help train users on the art of mnemonic memory techniques. As a practitioner of method of loci since childhood, the idea that an individual can be trained to practice memory techniques virtually that help build the working memory and recall is fascinating to me. As many individuals who practice mnemonic techniques know, it takes considerable practice and time to learn but resulting benefits include a highly effective ability to absorb, retain, and recall memories and learned information. Yet, as our accessibility to technology increases, we have a tendency to engage in cognitive offloading which can impede our ability to learn and retain information. Recent studies show that we are less motivated to use our memories to search for information if we have access to the internet, and this information is soon forgotten once a task is over. Building a memory room and learning how to use and route to specific locations to place memories in a VR-immersive environment may be an excellent way to turn technology into an ally.
Experiencing the impossible
We’ve relied on paintings, literature, and oral tradition to describe sites on our planet that used to exist, and either are gone forever or are no longer in their original states. From ancient ruins to architectural wonders, VR and AR can give us the ability to revisit history. Exploration of archaeology in VR has been growing, with one VR Village contributor in 2016 utilizing many methods — including laser scanners, photometry, and 360-degree video — to allow users to explore a site. Also, a startup recently made headlines for its work in reconstructing ancient architecture in VR. It is hard to imagine the scale, workmanship, and beauty that must have been present in visiting these places. Now, with VR, we might get to understand more about them. Hopefully soon, we will even be able to travel to them in a physically impossible way, such as flying like a bird with your friends and family.
The call for submissions for the SIGGRAPH 2017 VR Village is open through 21 March 2017. Submit a project to show off your work and expand the possibilities of VR’s future.
