Image Credit: © 2023 3FINERY LTD.
The world of artificial intelligence is evolving every day and with that, the opportunities are endless. SIGGRAPH sat down with Llogari Casas and Professor Kenny Mitchell of the SIGGRAPH 2023 Real-Time Live! presentation “Intermediated Reality With an AI 3D Printed Character” to discuss “Intermediated Reality” and what SIGGRAPH 2023 participants can expect from their presentation.
SIGGRAPH: The world of artificial intelligence is evolving every day. What inspired the research and development of this real-time technology?
Llogari Casas and Kenny Mitchell (LC and KM): The inspiration behind the research and development of this Real-Time Live! breakthrough comes from a surprisingly timely convergence of many factors and advancements across a number of technologies to make a singular fun experience to showcase.
First, for sure, the rapid progress in AI and machine learning algorithms, specifically in areas like computer vision, speech recognition, conversational AI, text-to-speech, and more, has unlocked exciting new possibilities for us to create interactive and expressive characters. This lightning pace served as the catalyst to crack the combination of mixing AI with Intermediated Reality, our augmented reality (AR) telepresence communication with visual appearance redistribution invention. Our patented solution allows for the highest level of realism in animated facial expressions and movements, mostly by efficiently utilizing live video optical appearance directly from the device’s embedded camera.
Second, the entertainment industry’s pursuit of new and captivating digital experiences has been instrumental in shaping our direction. Intermediated Reality presents a new way to engage with expressive virtual characters through cloud networked remote embodied communication that conveys a new sense of presence with personality to everyone. With our discovery of delivering AI-driven conversational characters with our expressive facial animation and AR methods combined with great quality 3D printing, we can craft compelling interactive narratives, virtual companions, and digital guides that engage users more physically visceral than before. This opportune combination of our own unique methods that we developed for 3FINERY LTD, together with the super pace of the AI community’s advancements, coupled with the desire to create immersive and emotionally engaging experiences, has enabled us to showcase this technology to the Real-Time Live! audience.
SIGGRAPH: Give our audience a behind-the-scenes description of how you developed this work. What problems does it solve?
LC and KM: The technology’s foundation is Intermediated Reality, a technique developed during Llogari’s Ph.D. with Professor Kenny Mitchell’s supervision. Initially prototyped many years ago in Unity and more recently optimized in Swift and Metal on Apple’s iOS, Intermediated Reality employs live video optical appearance most efficiently from the device’s embedded camera to warp animated facial expressions and movements, achieving a high level of realism, and then transmit those expressions to remote devices all connected to the cloud.
To deep dive into our prototype showing at SIGGRAPH 2023, we kick off the speech recognition processing using the ML chip embedded locally on the mobile device, transcribing spoken words into text and then send it to a Large Language Model (LLM) server in the cloud. The response is directed to a local speech synthesizer model to generate audio with enhanced quality speech generation. At the same time, we correlate the response’s textual tokens to character animation expressions so that we realize a procedurally animated face based on the AI-generated response. These visual voice expression tokens smoothly articulate with their surrounding tokens, ensuring natural blending and physical consistency of our animated faces forming words with mouth movements in sync with the audio generated voice. This ensemble of great techniques empowers us to deliver real-time interactions with expressive AI avatars, bringing characters to life physically with dynamic responses and animations, delivering an engaging and immersive experience.
SIGGRAPH: The opportunities in the world of artificial intelligence are endless. How do you envision this technology evolving and impacting the real-time space?
LC and KM: As AI and 3D printing continue to advance, the evolution of this technology could lead to personalized and emotionally engaging experiences, where users could have real-time conversations and interactions with lifelike characters brought to life, such as their favorite animated famous personalities or historical figures. Nonetheless, the impact of this technology in the real-time space extends beyond super engaging new forms of entertainment. Its potential in remote communication and telepresence opens possibilities for businesses, education, and healthcare. Organizations could use AI-driven avatars to facilitate virtual meetings and collaborations, enhancing remote team interactions. In education, students could engage with AI-powered historical figures or virtual tutors, making learning more interactive and personalized. Additionally, we hope this technology can address social isolation challenges by providing companionship to those in need of virtual interaction. As we bring our technology to more people, it could become an integral part of many daily lives, transforming how we connect with others and interact with the world around us in real-time.
SIGGRAPH: What can SIGGRAPH 2023 participants expect from your demo during Real-Time Live!?
LC and KM: At SIGGRAPH 2023, participants will have the unique opportunity to witness an exclusive Real-Time Live! demonstration featuring expressive toys brought to life in AR with conversational artificial intelligence (AI) responses. Intermediated Reality is all about communicating through remote embodied characters as a kind of telepresent puppetry for AR. Here, it is empowered through never-before-seen generative AI personalities. With the video feed direct from the phone, we replace costly traditional rendering computations with simple optics deformed to the 3D-printed characters’ facial animation with uniquely low computational overhead, empowering the audience to see a real-time, physically embodied avatar chat as though they were right in front of them. We hope it will also be a lot of fun. We’re really excited to share out what we’ve got lined up.
SIGGRAPH: Can you talk about social telepresence? What is it and how does your technology promote or enhance it?
LC and KM: Social telepresence refers to the feeling of being physically present or connected with others in a remote environment through technology. We aim to create a sense of shared presence and engagement, as if individuals are together in the same physical space, even though they may be geographically distant. The concept of social telepresence is closely related to virtual communication and teleconferencing technologies that seek to bridge the gap between physical separation and social interaction.
Our Intermediated Reality technology enhances social telepresence by creating lifelike AI-driven avatars that foster a sense of shared presence and engagement in remote interactions. These avatars exhibit realistic facial expressions, a sometimes unnerving-seeming emergence of emotional intelligence and personalization, with the aim of leading to more authentic and meaningful interactions with users. In professional settings, Intermediated Reality could facilitate remote collaboration, making virtual meetings and teamwork more engaging and immersive.
Additionally, we can develop more to address social isolation by providing virtual companionship through empathetic AI avatars, enabling individuals to engage in interactive and emotionally fulfilling conversations, even when physically apart. Even as distinct technologies, the impact potential is compelling. With our combination of AI, augmented reality, and 3D printing in Intermediated Reality technology, we promote a deeper connection and sense of social presence, transforming the way we communicate and interact in real-time across all domains.
SIGGRAPH: Looking ahead, do you have any plans for further improvements or expansions to your technology? Are there any new features or enhancements you would like to explore?
LC and KM: At 3FINERY LTD, we are close to releasing our social play platform called, MagiPeeps®. This platform allows users to embody attuned avatars that reflect their individuality and identity, embodying each other thru MagiPeeps®. It’s like a kind of mirror. Whether through games or chatting huddled together spatially on your phone’s screen or integrating with Intermediated Reality to bring a new augmented life to your toys, we want to enable folks to explore new interaction capabilities for more meaningful and authentic communications wherever we are and in ways people feel comfortable and empowered to appear with others however they choose.
We place these beginnings with MagiPeeps®, as the first step, showing imaginative capabilities that can be built upon in all kinds of directions and domains. Particularly, as generative AI develops with better quality and efficiency in already seen ways, and in unseen potential directions, we see folks attending enhanced virtual meetings for the best-ever actionable outcomes reached optimally and the most cordial diplomatic engagements even in tough opposing positions reaching accords expeditiously. Of course, we imagine the most fun social gatherings ever, interacting with others in a more immersive and engaging way than before.
This year’s Real-Time Live! is a can’t miss event! Lucky for you, you won’t miss it! Join us in person at SIGGRAPH 2023, or catch Real-Time Live! streaming on Twitch on Tuesday, 8 August at 6 pm PDT!
Llogari Casas is a computer scientist and entrepreneur in the field of augmented reality (AR). He holds a degree in multimedia from Universitat Politècnica de Catalunya and a master’s degree in computer animation from Kingston University London. He obtained his Ph.D. from Edinburgh Napier University as part of a Horizon 2020 Marie Curie Fellowship, a highly regarded research fellowship established by the European Commission.
One of Casas’ notable achievements is the development of Intermediated Reality, a groundbreaking tool that brings real-world objects to life using augmented reality. This tool enables objects such as statues in museums, characters, and consumer products to interact with users through mobile devices. Unlike traditional augmented reality, which overlays virtual objects onto reality, Casas’ innovation focuses on animating real objects and creating believable animations. The animations even include the deformation of an object’s shadow as it moves.
Casas has also embraced entrepreneurship and played a pivotal role in co-founding two ventures. He is a co-founder of 3FINERY LTD, a spin-out company from Edinburgh Napier University that specializes in delivering augmented reality solutions for mobile devices. Additionally, Casas is a co-founder of Flash Park SL, a startup focused on providing innovative solutions for efficiently locating parking spots in busy cities through a network of renewable energy-powered parking sensors. In recognition of his achievements, Casas was awarded the Young Computer Researcher award by the Spanish Scientific Society of Informatics in 2021.
Professor Kenny Mitchell is chair of Video Game Technology at Edinburgh Napier University and a technical director of rendering at Roblox, providing solutions for video games, immersive technology, and media. He has accumulated over 50 patent inventions while leading technology for the BAFTA-award-winning BoomBlox with Steven Spielberg, “Harry Potter” video games, “Star Wars” rides, and “Finding Dory” and “Pirates of the Caribbean” movies. A co-founder of 3FINERY LTD bringing objects to life and CTO of Cobra Simulation disrupting immersive software, he is co-editor in chief of ACM Games: Research and Practice and associate editor of ACM Computer Graphics & Interactive Techniques and Elsevier Computers & Graphics journals. He has built and managed large research groups and projects with international consortia of collaborating companies and universities.
