Image Credit: © Weidi Zhang
“Cangjie’s Poetry” is an intelligent, multimodal system that interprets images through Chinese characters to produce new symbols. Read on to hear what its creators — Weidi Zhang and Donghao Ren — had to say about this incredible art installation, which also happens to be the winner of the SIGGRAPH 2021 Art Gallery Best in Show award.
SIGGRAPH: Tell us about the process of developing “Cangjie’s Poetry”. What inspired your team to create this real-time poetry writing system?
Weidi Zhang (WZ): We were interested in the constant conversations between humans and machines nowadays. Humans use programming languages that are compiled into binary digits that machines can interpret. However, Intelligent machines today are not only observers of the world, but they also make their own decisions. We wondered if AI imitates human beings to create a symbolic system to communicate based on their understandings of the universe. This would allow them to start actively interacting with us. We wondered how this will recontextualize and redefine our coexistence in this intertwined reality? Our inspiration comes from an ancient Chinese legendary historian (c.2650 BCE), who invented Chinese characters based on the characteristics of everything on the earth. We think it is intriguing to train an AI system to constantly observe the real world via a camera and reconstruct the live streaming of surroundings using unique symbols with meanings, just like Cangjie did thousands of years ago.
Donghao Ren (DR): It’s very interesting to use a machine-learning algorithm to achieve an artistic goal. We developed this machine learning model that can generate novel characters from real-world images. The artistic goal has a lot more freedom than common machine learning tasks that have specific metrics. There isn’t a clear specification of what the input and output of the model should be. We explored a lot of models and, in the end, it’s more about how the system looks and feels like an installation. I think there is still a lot more that we can explore.
SIGGRAPH: You say that this chapter, specifically, is a “semantic human-machine experience that evokes feelings”. Can you tell us more about that?
WZ: The previous chapter of the “Cangjie” series focuses more on the visual experience that
celebrates the machine’s creation. Compared to the previous chapter, “Cangjie’s Poetry” implements a computer vision system to generate descriptive sentences based on its observations of surroundings in the installation space in real-time, which attaches meanings to these new symbols (pseudo-Chinese characters constructed by Chinese strokes). We specifically used English as the language to make this work more approachable to international audiences. We want to engage our audiences in the conversation with our intelligent system (Cangjie) as much as possible.
SIGGRAPH: Did you come across any specific difficulties in making this intelligent system?
DR: Our biggest difficulty during this project was developing the machine-learning algorithm to produce characters from real-world images captured by a camera. We experimented with a few approaches, starting with a basic autoencoder, then tried VAE (Variational Autoencoders), and, finally, we picked the BiGAN model because it generates more interesting and better-looking characters. It took a lot of iterations — and trial-and-error — to arrive at the final model.
SIGGRAPH: What do you hope the audience takes away from their experience as they interact with the system?
WZ: I hope the audience engages with the visualization of new symbols and intelligent poetry. Our goal was to build semantic connections between reality and abstraction with empathy, and I believe the audience feels that. Moreover, I hope they feel the intimate relationship between themselves and the machine. This connection can help reflect on the meaning of language and the possible future of coexistence between humans and machines.
SIGGRAPH: What do you find most exciting about the final product you presented to the SIGGRAPH 2021 community?
WZ: We built a virtual gallery to present our works to the SIGGRAPH 2021 community. Due to the pandemic, we did not get the chance to present this project physically, but we still got to share the story and virtual version of this work with the audience. We were so excited to hear the feedback from the audience and how they related to it no matter what cultural background they came from.
SIGGRAPH: What’s next for “Cangjie’s Poetry”? Can we expect to see another chapter in the future?
WZ and DR: We’d really love to see it shown as a physical exhibition. Because of pandemic regulations, we have not had many chances to see real-time feedback between the audience and a physical art gallery. The idea of the next chapter will evolve after our physical exhibition in Santa Barbara this October.
SIGGRAPH: Now that SIGGRAPH 2021 is over, how does it feel having won the Best in Show award?
WZ: It is such an honor to win the Best in Show award and exhibit virtually with the other amazing artists in SIGGRAPH’s Art Gallery. The aesthetic and inspiration of this project are deeply related to our cultural background: Chinese mythology, history of language, and Eastern ink wash painting, among others. I feel grateful that both the concept and visual executions were delivered to our audience from diverse cultural backgrounds and engaged the audience in different ways. I also want to thank my Ph.D. advisor, Professor George Legrady, the Experimental Visualization Lab, Professor Marko Pelhjan, and Professor Curtis Roads in the Media Arts and Technology Program at the University of California, Santa Barbara. This project would not have been possible without their support.
SIGGRAPH: What is something about your project that you would like to tell the SIGGRAPH audience, that you didn’t get a chance to?
WZ and DR: We would like to thank our audiences for interacting with our project. We also invite the audience to participate in future in-person exhibitions of this project. It will surely be a very different experience from what you saw at SIGGRAPH 2021.
Learn more about this project when you access SIGGRAPH 2021 on demand through 29 October! Registration closes on 18 October.
Weidi Zhang is an LA-based new media artist, visual designer, and researcher. Her current research and media art practices investigate the possibilities of semantic meanings and non-linear narrativity of assemblage art in the context of Intelligence system design, immersive media, and interactive generative art. Her works are featured at different international venues, such as SIGGRAPH/SIGGRAPH Asia, ISEA, Times Art Museum, IEEE VISAP, Athens Digital Arts Festival (ADAF), Japan Media Art Festival, and others. Her immersive audiovisuals have been selected and screened in International Fulldome Festivals, such as Metaxis Festival Planetarium 1 (Russia), Jena Fulldome Fest (Zeiss-Planetarium), and Dome Under Fest ( the Melbourne Planetarium / Victoria Museum). Currently, she is a Ph.D. candidate in the Media Arts and Technology Program and a researcher in the Experimental Visualization Lab at the University of California, Santa Barbara. She holds her MFA degree in Art + Technology from California Institute of the Arts.
Donghao Ren received a B.S. from Peking University, then a Ph.D. degree in computer science from the University of California, Santa Barbara. His research focus is on information visualization and machine learning. During his time at UC Santa Barbara, he worked on user interfaces for visualization authoring, as well as visualizations for machine learning. He is currently working at Apple.
