‘Paint the World’ With Your Digital Paintbrush

SIGGRAPH 2024 is excited to showcase the newest jaw-dropping technology in Denver, 28 July–1 August. There is so much to look forward to, and we cannot wait to begin offering some sneak peeks at what you can expect. In this discussion, we sat down with SIGGRAPH 2024 Appy Hour contributor Elliot Dickman to discuss his presentation “Paint the World.” Grab your paint, paintbrushes, and a blank canvas, and join us on this creative deep dive.

SIGGRAPH: For those who are not familiar with your project, “Paint the World,” can you tell us about the background of this work and what its goal(s) are?

Elliot Dickman (ED): Paint the World is an augmented reality (AR) digital painting app. The idea is that users can use their phone as a sort of digital paintbrush, so when you move your device around the real world it leaves a digital trail of paint that stays anchored to its location in the environment, and then you can step back and look at it on your device. It’s really fun to play around with, and there are a bunch of different brush settings and materials that are already built into the app for some artistic variation. You can even take pictures of your creations from inside the app and share them with friends.

SIGGRAPH: What inspired you to pursue developing this application?

ED: The idea really started because I wanted to learn more about RealityKit and Swift development, and I wanted to challenge myself with a new project. I thought an AR painting app would be a good place to start, and I was inspired by Tilt Brush (a VR painting app developed by Google) and the old Disney Channel “Wand” bumpers that you’re familiar with if you grew up watching Disney Channel in the 2000s. I initially figured that someone must have built a similar app and have some resources or documentation out there, but when I couldn’t find anything I decided that it was a great opportunity to try building it myself.

SIGGRAPH: Were there any barriers you encountered during this process? If so, how did you overcome those challenges?

ED: There were definitely challenges! I have a lot of experience with both procedural geometry and programming, but this was the first time I’d worked directly with generating 3D meshes at a really low level. A lot of what I needed to implement I had known on a theoretical level, like how Bezier functions work and how to draw a 3D mesh from a set of points, but actually writing out the code in such a way that it does exactly what you want it to do, dealing with all sorts of edge cases, and implementing customization options that I wanted to give to the user, that was a really iterative process. I spent a lot of time on the Math StackExchange forums combing through functions and translating those to code, and ultimately the whole project has been a very iterative process.

SIGGRAPH: What was your favorite part about working on this project? What creative doors were opened once this project was completed?

ED: I would say that each step in the development process was very rewarding. This was one of those projects where I had something in mind and was able to sit down, work through it, and then the thing that I had envisioned was there in my hands. It’s a great feeling. I also really enjoyed being able to take some of those theoretical concepts that I had learned in math and computer science classes in college and actually apply them to something that I was building.

The whole concept of AR painting is really versatile as an artistic medium. There were plenty of times while I was developing the app where I was playing around with it and thought, for example, “I really wish I could move this object by dragging it,” or “I wish I could draw this kind of brush stroke,” and I think the creative aspect of using the app really guided the development process. Early on when I was showing it off to friends, a lot of them would do things I hadn’t even thought of. One person immediately used it to mock up what a room would look like with different furniture by drawing out the shape of the furniture in the space with the app, for instance. So even though the app is in a pretty basic proof-of-concept state right now, it’s still been awesome to see what people do with it.

SIGGRAPH: What does the future hold for your project? How do advancements in AR shape the future of your project?

ED: There’s definitely more work to be done. Some features that I consider to be pretty fundamental still need to be built out before I would consider it a market-ready application, like saving and loading scene files that you create. From more of a big-picture perspective, I think this is just one of many things that represents the democratization of mixed-reality applications. You don’t necessarily need a fancy headset to have access to these kinds of programs, you just need a phone. The development of applications like this one and advancements in AR technology overall sort of feed into each other. AR technologies — both hardware and software — are necessary to be able to build and run these apps, and the more that people build apps with that XR tech stack, the more innovation there will be in that space.

SIGGRAPH: What should attendees look forward to in interacting with your app at SIGGRAPH 2024?  

ED: There will hopefully be a build up on TestFlight by the time SIGGRAPH comes around, so attendees will be able to download the app on their own devices and the whole conference (or the whole city!) can be their canvas. All you need is an iPhone, so I’m excited to see what people come up with and I think it’ll be a lot of fun.

This is only a small glimpse at the amazing content you can look forward to at SIGGRAPH 2024. Don’t delay and register today to secure your spot at SIGGRAPH 2024.

---