© Berreth, Polyak, and FitzGerald
A duel SIGGRAPH 2020 Talks and Groovy Graphics selection, “Story-Go-Round: Augmented Reality Storytelling in the Multidisciplinary Classroom” is a interdisciplinary project that pushes students to become comfortable with ambiguity — a modern challenge not so easily overcome with the internet as an always-present safety net. We caught up with creators and educators Patrick FitzGerald, Todd Berreth, and Emil Polyak to learn more about their inspiration, ideation, and implementation.
SIGGRAPH: Talk a bit about the process for developing your project, “Story-Go-Round: Augmented Reality Storytelling in the Multidisciplinary Classroom.” What piqued your interest in augmented reality storytelling?
Patrick FitzGerald (PF): We’ve always been focused on promoting taking creative chances in the classroom, encouraging students to work with ambiguity, and ultimately helping students develop stronger creative confidence.
Todd Berreth (TB): I don’t think our project started necessarily with the intention to pursue augmented reality storytelling. It evolved in that direction as we pursued a number of opportunities, learning objectives, and innovations that we tried to integrate and emphasize in our design studio.
The particular course in which we utilize the technology is an upper-level undergraduate art and design studio, where some students are approaching interactive media design, game development, coding, electronics, and other technical domains for the first time. Almost all of our students are excellent visual artists, illustrators, animators, and crafters, so we wanted to leverage those comforts and skill sets, along with their analog skills, in an environment like Unity as they began to experiment with digital making and coding.
We looked for hybrid analog and digital spaces that added richness to the interplay between traditional craft and emerging technical affordances, and found inspiration within the game industry, including State of Play’s “Lumino City“, and animation projects like Wes Anderson’s “Isle of Dogs” and “Fantastic Mr. Fox“, all of which combined a precious, handcrafted aesthetic with technical innovation.
Part of this project was also inspired by previous experiments that Emil and Pat pursued in a former studio project — “Paper Town VR” — which allowed students to build multi-plane environments in VR using hand-illustration and physical making.
Emil Polyak (EP): There are many elements in digital media production that benefit greatly from experience with real-world subjects. This includes lighting, textures and materials, shape and form, camera lenses, and other aspects. As educators in this highly virtualized era, we are tempted to shift the focus exclusively to digital tools; however, this limits the design that students produce to obvious choices offered by software, and the results become alike and repetitive. We wanted to create an assignment that functions by combining two made-up worlds: one that is physically produced and another that is digitally animated and controlled.
SIGGRAPH: Let’s get technical. How did you develop the project? How many people were involved? How long did it take? What was the biggest challenge? Our audience loves details!
TB: For the most part, the platform was developed over the course of an academic semester, which is 16 weeks in our case. The faculty focused on developing the general technical systems, the electronics, cabinet, scanning software, Unity software templates, etc., staying at least a few days ahead of the needs of the students as they developed their individual projects.
The students were first asked to consider the affordances of the system before it was developed — the form-factor of the wooden spools, the cyclical rotating behavior, the potential arcade controls — and also explore conceptual themes related to developing stories and games focused on environmental sustainability. They had to consider cradle-to-cradle design and the cyclical nature of things. The students asked themselves, “What are the poetic potentials therein, and how can we invent and develop a technical system to express this potential?”
EP: As artists, we tend to use our “sixth sense” and snap into design mode in seemingly ordinary scenarios, such as when we see a substantial amount of empty wooden spools thrown out by a cable company that had just replaced the network connections in our building. In terms of development, we have worked on several zoetrope-type projects in the past, and the stacked up cable spools reminded us of cartridges, cassette tapes, and floppy discs.
SIGGRAPH: How do you link the “carousel-like” stage to a game engine? How did you come up with this method?
TB: There are two applications that run on the Story-Go-Round computer: a spool controller/scanner application (SCSA), and a Unity-based “game” application. There’s also a small control program running on an Arduino in the cabinet.
The SCSA is an intermediary in the system. It relays game controller information from the Unity app via Open Sound Control (OSC) to the Arduino board via serial connection, which controls the precise rotation via stepper motor/gearbox. The SCSA relays positional information from the Arduino via OSC, while also capturing a live video feed via 4K video, depth camera, or playing a recorded spool video, and shares this with the Unity app via shared GPU textures using Spout.
We came up with this method out of necessity and simplicity. The first capability we needed for our students to see was what their spools would look like through a camera — to “scan” them — and then be able to edit those scans using digital video tools like Adobe After Effects. So we developed this application right away, which allowed for frame-by-frame capture, or quick video capture, and general positional control of the spools.
In parallel, we were developing Unity templates to give the students later in the semester. We wanted these templates to be relatively lightweight without incorporating all of the spool control code.
SIGGRAPH: How did you manage to create a seamless blend of technology, art, and design?
PF: We are lucky in that our department has faculty and students with wide range of skills, including coding, electronics, physical making, illustration, and animation. We also have a knowledge and love for the history of animation and gaming. We thought the idea of combining a rotating physical set (zoetrope) and a nod to early gaming cartridges with new augmented reality ideas would be a charming project. We also tried to expose the inner workings of the built mechanism so that the “magic” of seeing and controlling the very physical set with the monitor would create an improbable but integrated combination.
EP: We also wanted to embrace the interdisciplinary nature of art and design. It is perhaps something that as a modern culture we’ve lost contact with a bit. If you look back in time, you realize that most great minds — from Leonardo da Vinci to Thomas Edison and many others — combined technology, art, and design without drawing any lines between them. Imagination, vision, and openness to experimentation are all fundamental elements of innovation, including those elements that are functional, as a well as those that speak through aesthetics. Often, one inspires the other.
SIGGRAPH: What will be different between your Talks session and your Groovy Graphics presentation?
TB: In the Groovy Graphics presentation, we plan to focus on the particulars of the assignment: the templates we created to jump-start development and how we accommodated differing levels of technical expertise and interest domains. We also want to discuss the difficulties — the trials and tribulations of the experience — as well as future directions for the assignment, including accommodating classrooms where the faculty is missing some of the domains required for the assignment.
The Talk will focus more on our designing platforms for teaching creativity and storytelling within emerging technology domains, and facilitating multidisciplinary learning.
PF: We will also discuss some of the unexpected turns, challenges, and adaptations the project acquired over the course of the semester.
SIGGRAPH: How can students benefit from your project?
PF: There is a lot of ambiguity in working with emerging media. As we mentioned, a main goal of ours is to help students create a “track record” of success, despite not having foresight into the problem’s solution. Additionally, the students worked in teams. This is often a struggle, but the experience of working through these situations together, wearing different hats, making collective decisions (somehow!), and ending up with a respect for the disparate skills of each team member are all great learning outcomes for our multidisciplinary students.
EP: These days, students can find all available information on using any software on the internet. Most of this information is a step-by-step process, asking the student to simply repeat the given process. We wanted to ask our students to take a break from this practice and invent new steps. Asking them to leave the comfort zone that the internet offers pushed them to a different creative mindset where exploration and curiosity go hand-in-hand.
TB: Similar to what Emil and Pat mentioned, when students try to develop a multi-faceted project like this, which crosses multiple technical and artistic domains, and try to quickly learn the skills they need to implement their vision, they get to experience the remixing, mashup, open-source making culture, which is so prominent in the way we work today. How else can young designers best learn the duct-tape and jerry-rigging skills which will allow them to see potential opportunities in disparate things, and find new ways of hacking them together to create something new, even if what they create is “ridiculous”, and often not practical?
SIGGRAPH: What’s next for “Story-Go-Round”? How can this be applied to future design thinking?
PF: One idea is to utilize this as an outreach teaching tool with depth-sensing software that could create a real-time map for occlusion detection for handmade sets. Template walk cycle software would allow young students to “amend” the supplied walk cycles, creating a very customized and satisfying STEAM experience.
EP: This project is an inspiration for further exploration of mechanical, electronic, and virtual coexistence in storytelling and play. The opportunities in the future are exciting, as AR glasses make a breakthrough and depth cameras become more mainstream. The “magic” that will allow a meaningful interaction between the two worlds, in both ways, is also an extremely exciting design challenge.
SIGGRAPH: Share a favorite SIGGRAPH memory.
TB: This is my second SIGGRAPH, as I also attended to present an Art Paper in 2018 in Vancouver. I have to say my favorite memory is meeting for a first beer and fries with my session-mates and some past colleagues, and realizing how many of us knew the same people and had enjoyed each other’s work in past conferences or other venues, along with the very quick comradery of Birds of a Feather. I fondly remember having taxi adventures in a strange city with this group, trying to find our way to a party, and a behind-the-scenes tour of the VIVO Media Arts Centre … fabulous!
PF: This is my fourth SIGGRAPH. My favorite memory stems from when we presented The Long View as an interactive projection [in the SIGGRAPH 2013 Art Gallery]. Our team had difficulties with the physical set up in the gallery space, and solving these problems with a ticking “opening” clock was more exciting than I was hoping for. Eventually it worked, and it was a thrill to see folks from around the world interacting with our work.
EP: My very first SIGGRAPH experience was when I briefly visited the United States for the very first time for the sole reason of attending SIGGRAPH. I connected with many incredibly talented people from all over the world, and realized that the spirit of this conference is not only about technology or dissemination of research, but even moreso about the passion that connects us and the conversations that have no borders. The new format online will allow more people to be part of it; however, the in-person parties will be missed greatly!
SIGGRAPH: What advice do you have for someone looking to submit to the Talks or Groovy Graphics programs for a future SIGGRAPH conference?
PF: Collaboration! Find partners that have different skill sets than yours AND who you enjoy working with. In our model, we co-taught a studio, which was a unique situation, and we knew we accomplished things that could not have been done individually. In all honesty, we had SIGGRAPH in the back of our minds, knowing that we would try to submit this project if it was an aesthetic and technical success. But, we did not mention this to students for some time, so that they could create and take chances without the extra burden and pressure of a potential SIGGRAPH submission.
I would also suggest reading the new SIGGRAPH conference details as soon as they are published. Sometimes, you need time to let your ideas gestate before deciding on a project that might be of interest to the SIGGRAPH community.
SIGGRAPH 2020 registration is now open! Register here for the virtual conference and explore more groundbreaking Talks and Groovy Graphics selections.
Todd Berreth is an artist, designer, architect, and technologist, and is an assistant professor in the Art + Design department at North Carolina State University. He is a member of the Chancellor’s Visual Narrative faculty cluster and has a research interest in exploring how emerging technologies, immersive and pervasive media, AI/machine learning, and other advances can help us see our world in new ways, alter our physical environments, and expand our narrative and art-making capabilities. He has presented artwork locally, nationally, and internationally, including through ACM SIGGRAPH, Leonardo/ISAST, IEEE Visweek, and the China Science and Technology Museum, among other venues.
Emil Polyak received a master’s degree in cross-disciplinary art and design from the University of New South Wales in Australia. With a background in 3D animation, interactive media, and digital arts, Polyak is exploring intersections of science, engineering, and art. His work focuses on research-informed design practice with emerging and experimental creative technologies to expand disciplinary boundaries and to create meaningful experiences. The overarching goal of his scholarship is to reimagine creative practice with the possibilities and implications of computational creativity. Prior to joining Drexel University, he taught at New York University’s Media Design School and at North Carolina State University.
Patrick FitzGerald received a master’s degree in painting from Cranbrook Academy of Art. With a background in illustration, animation, experimental interactive installation, and design, his work crosses many disciplinary boundaries. His collaborative work is focused on the creation of experimental creative installations that utilize animation, new forms of interaction, and projected spaces. This work has been exhibited across the United States and Japan. A member of the Experience Design Lab for the College of Design, FitzGerald’s research and teaching span the full range of artistic and multimedia production, from illustration to interactive 3D animation and installation.
