Screenshot from the Unity Project © 2021 Illustration by author.
We caught up with Aki Järvinen, Ph.D. and lead immersive technologist at Digital Catapult, to learn more about his upcoming SIGGRAPH 2021 Labs session, “Rapid Prototyping Approach for Audio Augmented Reality User Experiences”.
Via an innovative use of game engine software and consumer technology to build audio-only AR experiences, Aki’s session will focus on equipping participants with the ability to be creative with this entryway into spatial audio. Read on to learn more about his session and background and discover what inspires Aki.
SIGGRAPH: Your SIGGRAPH Labs session is meant to be an interactive, hands-on class. What are you most excited about workshopping with SIGGRAPH 2021 participants and why?
Aki Järvinen: Last year I delivered a similar class on sketching XR designs and it was very encouraging in that my approach used the visual collaboration tool Mural and engaged the participants well — I instantly got volunteers to contribute to the live session, so it’s really that kind of engagement and curiosity that I’m looking forward the most. Also, people are generally tech-savvy [at SIGGRAPH], which helps in getting to the heart of the matter more quickly and makes the session more productive for all parties.
SIGGRAPH: What is a key technique you hope participants bring back to the office?
AJ: Really just becoming more aware of the possibilities of spatial audio, which is one of the learning objectives. Beyond that, encouraging people to engage with software that might at first appear somewhat intimidating, such as a game engine. If you stick to [that software] for a while, you don’t need to be a programmer to communicate your ideas with rapid, yet functional, prototypes. Plus, learning more about prototyping as communication, and more specifically in the context of augmenting a physical space with audio.
SIGGRAPH: Are there examples of audio-only augmented reality (AR) projects that you think are especially well done or exciting for their advancements?
AJ: In general, I find that the creative work in this space is the most interesting. I have been very inspired by Darkfield’s work, both in their location-based and mobile incarnations. While they don’t frame their work as “AR”, and thus far have not leveraged the spatial aspect of audio in full, their work does augment reality in quite a special way, and they have a consistent theme across their various projects.
The way Darkfield asks the audience to suspend their disbelief — for instance, by asking them to get into a specific setting (such as a park bench) — and close their eyes, I find quite clever because it does away with some technical, unsolved issues, such as how ambient audio from the surroundings blends with the audio from the AR application.
I also find the work that Bose started with its Frames sunglasses a few years ago inspiring, even if it did not work out for them commercially.
It seems clear that Apple is investing quite a bit in spatial audio, too. My workshop is aimed at inspiring ideas around that.
SIGGRAPH: What trends in AR and UX are you excited to see develop over the coming years?
AJ: I think that what tends to be forgotten in AR discussions is the question of interaction, or input, methods. We are fascinated with the visual aspects and the challenges with, for example, how to solve problems related to physics in getting to the holy grail of light, all-day wearable AR glasses, that interaction methods do not get quite the same attention.
Smartphone AR UX is not ergonomic at all, and some of the current AR glasses still use laser pointer-type interaction methods, which essentially address planar surfaces for interaction and consequently do not lend themselves to fully embracing the spatial nature and promise of AR.
So, whenever I see glimpses of research and development around voice and gestures, it piques my interest, even if, on the other hand, hand tracking has its own UX challenges. And, of course, use of audio and haptics in providing user feedback.
SIGGRAPH: What are some resources every AR developer who joins your class should be aware of going into it?
AJ: The BBC in the UK has published a bunch of articles on their audio AR experiments. A good place to start is their article, “Recommendations for Designing Audio Augmented Reality Experiences.” I also recommend checking out Darkfield Radio, a mobile app, for inspiration.
Finally, anyone who wants to follow along and try creating their own prototypes can download the Unity project I will be using during the session via GitHub.
SIGGRAPH: What advice to you have for someone looking to teach in SIGGRAPH Labs during a future conference?
AJ: In most workshops that I run the aim is to produce practical outputs, and, therefore, I try to demonstrate the capability of the technology or an example of an output very early on, in the hopes of that it awakens the audience’s curiosity and keeps them engaged. Now that in 2021 we are still delivering these activities mostly remotely, leveraging interactive tools (like Mural or Miro). To make the session interactive is key.
I have also had success, now two years in a row, with submitting an approach that is not in itself “cutting edge” from a technology point of view — the technology it is targeting is relatively accessible (VR headsets, AR-capable smartphones especially) rather than something being developed in a R&D lab. Instead, what I have done with the technology in order to explore design and UX issues, in a hands-on format, has been acknowledged by the SIGGRAPH jurors as novel and innovative enough to include in the conference. For me, that is encouraging and I feel it also makes SIGGRAPH more inclusive and attractive to folks who do not necessarily have a primarily technology or computer science background.
Register now for SIGGRAPH 2021 to experience sessions like Aki’s and so many more on the latest in computer graphics and interactive techniques.
Aki Järvinen, Ph.D., is an immersive researcher, designer, and technologist with 20 years of experience with interactive media. He works at Digital Catapult in London helping UK companies adopt immersive technologies into their business. Learn more about his thoughts on audio-based augmented reality via his blog, Design Superpowers.
