Image Credit: NVIDIA
SIGGRAPH steps back into a standout SIGGRAPH 2025 hands-on Lab, where computer graphics and robotics converged — and digital twin technology reimagined how intelligent systems are built, tested, and brought to life.
Ji Yuan Feng and Rishabh Chadha share how OpenUSD, NVIDIA Isaac Sim, and ROS unite to create high-fidelity, collaborative virtual environments for software-in-the-loop testing. From constructing rich, layered 3D worlds to simulating and validating robot behavior, this project highlighted both the technical workflows and the broader shift toward simulation-first development — where visual computing and physical intelligence meet to accelerate innovation in robotics.
SIGGRAPH: What inspired you to create a hands‑on Lab focused on digital twins for robotics simulation?
Ji Yuan Feng and Rishabh Chadha (JYF & RC): Robots are expensive and can be dangerous to test in the real world. Because of this, we are seeing simulation take a more important role in robotics development, particularly in training, validation, and scalable synthetic data generation. In those first phases of testing, we can try a robot’s behavior in a variety of simulated situations before ever transferring to the real robot.
An accurate digital twin representation of the robot in simulation is foundational in training robot brains, such as how a robot walks or manipulates objects, that can easily transfer to the real world.
So, we wanted to use this hands-on Lab to share techniques on how to build realistic digital twin environments in which to test robots and incorporate digital twins and simulation in robotics workflows for validation.
SIGGRAPH: Your session featured OpenUSD, NVIDIA Isaac Sim, and ROS. Why did you choose this specific toolchain for the hands-on session?
JYF & RC: Software-in-the-loop testing involves several steps: Prepare the robot asset, configure it, and communicate with the “brain” using middleware.
First, you bring a robot into simulation and configure sensors, joints, and other attributes that help it interact with the simulated world. As a common scene description framework, OpenUSD makes it easy to work with these modular assets and enables interoperability between different applications. With Isaac Sim, an open simulation framework built on Omniverse, we can add and configure sensors — all in OpenUSD.
Finally, ROS is the middleware that lets us connect the “brain” of our robot to the simulation. For example, we show how to publish IMU (inertial measurement unit) data from our simulated robot to the robot controller node.
With this pipeline, we aim to seamlessly transfer the robot from simulation to reality. When we want to make changes to the software, we can easily test it again in simulation before deploying to the real world.
SIGGRAPH: How does OpenUSD enable richer or more collaborative virtual environment creation compared to other frameworks?
JYF & RC: OpenUSD enables richer, more collaborative virtual environments by serving as a shared, layered scene description standard that multiple tools and teams can edit non-destructively at the same time. It defines a common way to describe scenes and assets, so DCC tools, game engines, and simulators can all work from the same sources instead of using fragile custom exporters. OpenUSD features such as payloads let teams keep huge worlds responsive by deferring heavy assets, loading only lightweight summaries until detailed geometry and materials are needed. Layering and good asset structuring then allow each department to maintain its own override layers on top of shared assets, making large libraries reusable, versioned, and easy to customize without stepping on each other’s work.
SIGGRAPH: What are the key steps involved in integrating robot models into Isaac Sim, and where do beginners often struggle?
JYF & RC: The typical integration pipeline begins with converting source files — ranging from existing OpenUSD assets in the Isaac library to CAD models from software such as SolidWorks or robotics-specific formats such as URDF and MJCF — directly into a USD file. Once converted, the user must author the USD file to handle joint tuning and collision filtering to ensure a stable and accurate simulation, while optionally leveraging properties to optimize performance in larger environments.
A primary area where beginners struggle is the tuning step, as this process is highly specific to the robot, task, and simulation environment. Beyond setting basic controller parameters, newcomers frequently face “exploding” models caused by unmanaged self-collisions between overlapping joints or choosing optimal collision shapes while trying to achieve precise grasping. To help bridge these gaps, a featured workshop at GTC 2026 on tuning dexterous hands provided deep-dive guidance on stabilizing high-degree-of-freedom manipulators for complex interaction tasks.
SIGGRAPH: Digital twin adoption is rapidly expanding — what emerging trends or technologies are you most excited about?
JYF & RC: Digital twin adoption is accelerating, and two exciting directions include Real2Sim pipelines and large-scale robot learning. Real2Sim workflows turn real sensor data (RGB-D, lidar, video) into simulation-ready 3D worlds with minimal manual modeling, using techniques such as 3D Gaussian splatting and automated asset generation to preserve appearance and geometry for training and validation. This makes it practical to import 3D reconstruction scenes (for example, from systems such as NVIDIA Omniverse NuRec-style pipelines) directly into simulators so digital twins can continuously mirror changing factories, warehouses or city blocks instead of staying as static, CAD-only models.
The second big trend is moving from validating a single robot in a pristine digital twin to stress-testing entire heterogeneous fleets at scale, including humanoids, mobile bases, conveyors, and industrial arms. Emerging platforms combine high-fidelity physics, domain-aligned rendering and automated scenario generation to run thousands of parallel simulations, enabling policy learning and safety validation across many robots, layouts, and failure modes before deployment. This fleet-scale view turns the digital twin from a visualization tool into a core operational substrate for optimizing throughput, reliability, and coordination across whole factories and logistics networks.
SIGGRAPH: Where do you see the biggest opportunities for graphics techniques to reshape robotics development over the next few years?
JYF & RC: The biggest opportunities for graphics techniques to reshape robotics lie in the convergence of photorealistic rendering, learned physics, and generative world models.
Neural rendering techniques, such as 3D Gaussian splatting and NVIDIA RTX-powered path tracing, are moving beyond simple aesthetics to provide robots with high-fidelity visual inputs that narrow the perception gap between simulation and the real world. This is being further revolutionized by neural simulation, a major theme from the SIGGRAPH 2025 special address, which replaces traditional analytical solvers with learned dynamics models such as NeRD (Neural Robot Dynamics). These models allow for stable, contact-rich interactions and long-horizon predictions that can be fine-tuned directly from real-world data, effectively solving the “sim-to-real” problem for complex articulations. Complementing these are world foundation models such as NVIDIA Cosmos, which provide a generative “imagination” for robotics pipelines. Unlike classical simulators that require manual scene assembly, Cosmos can generate diverse, physics-aware environments and rare “corner case” scenarios, serving as a high-level reasoning and data-generation engine that works alongside Isaac Sim to train more generalized and robust robotic brains.
Ready to explore techniques like these in action? Register for SIGGRAPH 2026 to discover what’s next at the intersection of graphics and robotics. Secure early registration pricing through 8 May.
Ji Yuan (Steven) Feng is a robotics engineer at NVIDIA with expertise in simulation development, narrowing the sim-to-real gap by creating high-fidelity virtual environments to accelerate robotic testing and deployment. He is particularly passionate about reinforcement learning and imitation learning, exploring control strategies that enable safer, more human-like interactions.
Ji Yuan is completing his master’s degree in electrical engineering at Stanford University. Previously, He earned his bachelor’s degree in Mechatronics Engineering from the University of Waterloo. He also conducted human-robot interaction research at the Active Robotics Interaction Lab under Dr. Yue Hu, with a publication on IEEE Robotics and Automation Letters.
Rishabh Chadha is a technical marketing engineer at NVIDIA focusing on robot simulation. He has a Masters degree in Robotics from Worcester Polytechnic Institute. His interests primarily include robot perception, physics engines and deep learning.
