Image Credit: Representative image of the fictional VR landscape controlled by brainwave data, featuring pulsating lanterns, energy orb, illuminated threads of thoughts, ambient particles, and fog.
Take a look back at some of the incredible, immersive content that was showcased during SIGGRAPH’s 50th conference. We sat down with Tiange Wang and Xin Feng, the creators of “SinkInSync: A Brainwave Synchronization Experience in VR Towards Enhancing Social Connectedness,” a cutting-edge piece of technology that explores the potential of VR enhancing social connectedness in an era dominated by remote interactions.
SIGGRAPH: The idea behind this project is truly fascinating. What was the inspiration behind this technology?
Tiange Wang (TW) and Xin Feng (XF): Our project drew inspiration from a fascinating phenomenon known as “embodied simulation” or the “mirror effect.” It’s the natural tendency that we synchronize with others in various ways, from matching footsteps during a stroll to mirroring gestures during conversations. We were also intrigued by studies showing how people sync heart rates and breathing while watching emotional films together or sharing intimate moments.
Delving into past research, we found that these synchronizations extend beyond the physical realm and also produce significant social effects. For instance, engaging in eye contact, physical contact, and collaborative teamwork has been demonstrated to effectively promote inter-brain synchronization. Such activities not only enhance interpersonal connections but also nurture a sense of unity and empathy among individuals. However, in a world of rising remote relationships and telepresence, we could no longer have real eye and physical contact as much as we used to. It signals both an opportunity and a hypothesis: the potential to harness the power of virtual inter-brain synchronization to strengthen bonds and enhance social connectedness among people.
Therefore, our project originated from the inquiry: How might we foster mutual bonds and social connectedness through brainwave entrainment mechanisms? How might we enhance mental health and well-being via the experience of virtual inter-brain synchronization? These questions have become the catalyst for our technological exploration, aimed at bridging the emotional divide that physical distance creates in our progressively digital world.
SIGGRAPH: You mentioned that this explores the potential of VR in enhancing social connectedness. Can you elaborate on this concept for our readers?
TW and XF: In an era dominated by remote interactions, the richness of meaningful physical connections has begun to fade. Our project steps into this landscape to explore the profound capabilities of virtual reality in reviving and reimagining these lost connections. Consider the prospect of utilizing virtual reality not just as a tool for entertainment or escapism, but as a means to bridge emotional gaps and recreate the depth of genuine human bonds. By employing real-time, immersive visualizations of brainwave patterns and facilitating the experience of harmonizing these brainwaves across vast distances, our goal is to pioneer a novel dimension of social connectedness. In this space, the barriers imposed by physical separation are surpassed, enabling deep and intimate connections to thrive, regardless of geographical boundaries.
SIGGRAPH: While using “SinkInSync,” what should someone interacting with this technology be looking for in a user’s brainwave data?
TW and XF: With “SinkInSync,” the real-time brainwave data of a user’s paired partner — be it a romantic partner, friend, colleague, even a stranger, or their past selves — continuously alters the virtual 3D environment experienced by the user. The purpose of this immersive journey is to guide participants toward a transformative journey that delves into self-awareness, enhances interpersonal understanding, and fosters unique multimodal interactions.
The visual elements within the scene provide users with an intuitive understanding and immersive experience of their partner’s brainwave data. The colors help the user identify the dominant brainwave frequency and, consequently, the mental state of their partner user while the pulsations visually represent the level of mental activity and facilitate the synchronization of mental rhythms between users. Furthermore, when the users’ brainwaves synchronize within a specific band, ethereal flowers in the corresponding color bloom beneath their feet. This vibrant display celebrates their neural resonance and connection, creating memorable moments that users eagerly anticipate, where their brainwaves align harmoniously.
SIGGRAPH: What advice do you have for those planning to submit to the SIGGRAPH 2024 Immersive Pavilion?
TW and XF: Feedback and Iterations: When we started the “SinkInSync” project at MIT Media Lab back in May 2022, our initial prototype underwent considerable evolution to become what it is today. While our technical stack and pipeline remained largely unchanged, both the design and immersive experience underwent major transformations. Based on invaluable user testing and peer feedback, we pivoted our approach to develop the second version of “SinkInSync” that eventually made it to SIGGRAPH. Designed with a more focused and intentional approach, this version features abstract 3D elements in an ethereal, fictional landscape. This approach resulted in a significantly enhanced experience compared to our first prototype. Our primary lesson from this experience is that embracing iterative refinements based on feedback can greatly enhance and elevate a project.
Push the edges of engaging, innovative user experiences: The projects showcased at the SIGGRAPH 2023 Immersive Pavilion offered a unique and innovative user experience. They stretched the boundaries of our senses through multimodal interfaces, weaving participatory storytelling and immersive narratives. To stand out, it would be great for participants to think beyond conventions, experiment with novel technologies, and introduce fresh, unconventional ideas that push the boundaries of immersive experiences.
SIGGRAPH: Looking ahead, do you have any plans for further improvements or expansions to this technology? Are there any new features or enhancements you would like to explore?
TW and XF: The current phase of installation and user testing has revealed “SinkInSync’s” potential for diverse real-life applications, profoundly impacting social connectedness and emotional well-being. From relationship therapy to autism therapy, meditation support, past-self immersion, and innovative interactive gaming, “SinkInSync” showcases adaptability across multiple domains. In our next developmental phase, we aim to delve deeper into these specific use cases across various settings, fostering collaborations with different labs to amplify its effectiveness.
After we showcased at the SIGGRAPH 2023 Immersive Pavilion, we also modified our implementation to elevate the paired gamification element: When users’ brainwaves synchronize, vibrant bursts of flowers emerge from beneath their feet, celebrating their connection. This emphasis aims to amplify the message of fostering meaningful social bonds, encouraging users to explore this alternative mode of communication for richer interactions.
SIGGRAPH 2024 will be here before you know it, and we are currently accepting submissions for many of our programs. Be sure to take the advice from Tiange and Xin and submit your innovation to present at SIGGRAPH 2024 before it’s too late!
Tiange Wang is an award-winning and internationally exhibiting designer and creative technologist. Her work investigates innovative interactive paradigms among humans, technology, and the environment. Tackling subjects like climate change, holistic well-being, embodied experiences, and speculative futures, she employs mediums including multimodal interaction, creative data visualization, multimedia art, AR/VR, installation, food, software, AI, and games.
Presently, Tiange is a software designer at IDEO, where she crafts data-driven experiences spanning web/mobile platforms, XR, bespoke objects, and emerging technology, guiding clients through transformative journeys across sectors such as health, climate, media, and consumer products.
Tiange is the recipient of UX Design Award, A’ Design Award, and Muse Creative Award, among others. Her work has been exhibited and presented at Vancouver International Film Festival, SIGGRAPH, CHI Play, MOOD Museum of Design, Hebei Design Museum, Aedes Architekturforum, Boston Design Week, NYCxDesign Festival, SF Design Week, Cambridge Science Festival, and featured on Dezeen and World Design Week.
Tiange holds a Master of Architecture from Harvard Graduate School of Design and a Bachelor of Arts in Architecture with High Honors from the University of California, Berkeley.
Xin Feng is a creative technologist and multidisciplinary designer based in California. With a master’s degree in design technology from Harvard University, she is currently working at Meta Reality Labs. Prior to that, she also worked at Tencent Game studio as a technical artist, combining art with technologies to empower users with innovative open world experience.
Xin embraces a dual identity, fulfilling roles both as a coder and a designer. Her work spans a mixture of game development, interaction design, XR prototyping, and web development, with an emphasis on human-centered multimodal experiences that connect the digital and physical realms. Her research areas cover social wellbeing in XR, affective computing, VR embodiment, and immersive storytelling.
Xin has been recognized with Muse Creative Awards, Vega Digital Awards, and other honors. Her project was showcased at Vancouver International Film Festival, SIGGRAPH Immersive Pavilion, Hong Kong/Shenzhen Biennale of Urbanism and Architecture, NYCxDesign Festival, and SF Design Week.
