Copyright Painting generated by Metro Re-illustrated of the city of Shenzhen.
SIGGRAPH is now publishing select blog posts in both English and a language of the contributor’s choosing. Read the Mandarin version here, and continue reading the version in English below.
SIGGRAPH 2023 Posters presentation, “Metro Re-illustrated: Incremental Generation of Stylized Paintings Using Neural Networks,” explores the fusion of technology and visual artistry, showcasing an interactive system for labeling time-series data on city metro railway maps and a neural painter generating oil painting-like strokes on a virtual canvas. Here, we caught up with the presenters, who share the inspiration behind their research and its potential applications for urban planning, art, or other fields.
SIGGRAPH: What inspired you to explore the incremental generation of stylized oil paintings of city metro maps using neural networks? Why does this research matter?
Bo Shui (BS), Chufan Shi (CS), and Xiaomei Nie (XN): “Metro Re-illustrated” is inspired by our concern for the human imprint in urban development. The development of rail transit systems is one of the important humanistic features of urban modernization. Metro routes are not only a navigation tool for quickly traveling through the city, but their changes also tell the story of urban expansion and development. Under the influence of different natural and cultural environments, the development of urban transportation systems also shows different characteristics. Using neural networks to generate stylized maps of change over time allows us to represent the dynamic nature of urban development in a visually compelling way.
This work explores a new form of data visualization by applying machine learning to media arts. Furthermore, it may provide urban planners and policymakers with a new way of observing and communicating the evolution of metro systems in urban space, providing a medium for potentially participatory discussions in urban development.
SIGGRAPH: What techniques are employed by the neural painter to generate oil painting-like strokes on a virtual canvas?
BS, CS, and XN: The neural painter employs a technique that simulates the stroke-based nature of oil painting. We provide the reference images for the neural painter with data labeling. It incrementally applies virtual strokes to a canvas, utilizing an iterative process that calculates the loss between the currently generated image and the reference metro map image. This approach allows the neural network to continuously update its parameters and generate strokes that effectively blend colors and layer each stroke, reflecting the physical qualities of actual oil paintings with stroke layering and color blending.
SIGGRAPH: What are the key components of the neural network architecture used in the incremental generation of stylized oil paintings of city metro maps?
BS, CS, and XN: The key components of the work include the data labeling system, the stroke generation algorithm, and the animation encoding process. The data labeling system collects and structures time-series data of city metro lines and generates reference images. The stroke generation algorithm, a part of the neural painter, draws each stroke on the virtual canvas based on the reference images generated from the labeled data. The animation encoding process creates a time-lapse video that showcases the progression of the painting from a blank canvas to the final artwork, illustrating the development of the city metro lines over time.
SIGGRAPH: How does the system ensure the accuracy and consistency of representing the city metro maps?
BS, CS, and XN: We ensure accuracy and consistency by using a meticulous data-labeling process that requires users to interact with actual metro maps and label the coordinates of stations and lines. This generates detailed JSON files that hold time-series data of the metro systems. Metro maps commonly used in our daily life are usually simplified and scaled according to typesetting needs. During the stroke generation, these data are cross-referenced with satellite images to ensure that the positioning of the metro lines corresponds accurately to the real-world geography of urban development.
SIGGRAPH: What potential applications or implications does this incremental generation of stylized oil paintings of city metro maps using neural networks have for urban planning, art, or other fields?
BS, CS, and XN: The potential applications of this technique are varied. In urban planning, the visualizations could aid in understanding and communicating the growth of metro systems and their integration into the city fabric. For art, this project expands the frontiers of digital media art, representing a convergence of cartography, urbanism, and painting through the lens of machine learning. Furthermore, in education, it could engage students in geography and urban studies with a captivating visual tool. And in public facilities or transit hubs, such artworks could enhance the aesthetics of the environment while also providing informative content about the history and current state of metro systems.
Have you “mapped” out the research you plan to submit to SIGGRAPH 2024? Submit to Posters by 25 April for your chance to showcase your research in Denver!
Bo Shui is a master’s degree student in electronic and information engineering at Tsinghua University. Concurrently, he is also a Gold Microsoft Student Ambassador since 2024. His research interest includes human-computer interaction and visualization. In 2023, he completed an internship at Intel where he conducted user experience engineering and interaction design for AIGC products. His work aims at bridging individuals and the ever-evolving digital landscape with innovative design, which has been presented and exhibited at SIGGRAPH, ACM MM, ACM IDC, Design Intelligence Award, SIRF, Child-Friendly Design Award/Expo, and UN-Habitat Smart Winter Olympics, and Sustainable Development Report.
Chufan Shi is a student pursuing a master’s degree in electronic and information engineering at Tsinghua University. He has gained hands-on experience through internships at Tencent AI Lab and Intel, working on natural language processing and deep learning projects. He has contributed to publications in prestigious conferences such as EMNLP and SIGGRAPH.
Dr. Xiaomei Nie is an associate professor at Tsinghua University. Her research interests focus on cross-disciplinary studies of digital media, social innovation design, and contemporary art and design. She has published three books and numerous articles and has won multiple academic and teaching awards. Before joining the Tsinghua faculty, Dr. Nie served as a brand director in a large enterprise for many years. In addition to her academic pursuits, Dr. Nie is also an artist and designer.
