Introduction 

Any roboticist will tell you that developing and refining robots in simulation is a challenge. From finding the right tools, setting up environments, building a robot model, to integrating your SW stack, each step takes up valuable engineering resources. One major source of this time sink is the complexity of building a seemingly simple virtual testing ground. This process typically requires a number of engineers to make use of multiple tools, often for the first time, before a useful asset is developed. To speak nothing of maintaining the solution as project continues and the codebase evolves. We have implemented a USD importer that allows us to instantiate scenarios at run time without needing to recompile a UE4 game. This significantly improves pipeline integration for robotics teams and reduces ramp up time on projects. 

Today’s Solutions 

‍Robotics teams today use a variety of tools to meet their needs, and two of the major ones are Gazebo and gaming engines. Gazebo has been a popular choice for 3D robotics simulation for years. The XML Simulation Description Format (SDF) file is used to define the visual and physical characteristics of machines and environments. ROS serves as the machine interface and RViz is used for inspecting logged data. While this toolset provides familiarity to roboticist through the ROS structure, is has several shortcomings: 

• Limited ability for teams to internally create the large, complex, photoreal worlds that are especially critical to perception sensors. This problem also stems from inadequate inter-change support with digital content creation (DCC) and CAD tools.
• Machine definition is at a component level. This fact makes it difficult to model a more complicated sub-system’s behavior like a four wheel base with gears and transmission. As a result, this kind of behavior definition gets absorbed in the ROS interface layer.
• Limited ability to connect multiple robots or include live human input. This is particularly important as robots take on greater challenges and leave the lab for the real world.
• The SDF data format is flat which makes it very difficult for teams to collaborate, maintain multiple variations, and separate machine specification from characterization.

Recently, game engines such as UE4 and Unity 3D, have seen greatly increased use in the robotics field due to their photoreal rendering capabilities, ability to handle complex scenes, and integrated physics engines. However, these tools were designed to support massive projects in a very different application space. It is unsurprising then, that these engines are challenging for robotics teams to use given the complexity of the tools themselves and the foreign paradigms that drive their design. 

Solutions such as AirSim try to ease on-boarding for roboticists using these game engnines. However, these solutions lack a robust data model and clear separation between the game engine executable and project-team specific assets. Everything gets complied into a shippable "game" in order to run the simulation making changes to the simulation require returning to the game editor. This is far from optimal for robotics pipelines and workflows that require iterative design changes, span large teams, and can persist for years. 

The Power of a Data Model 

‍The data model used to describe your environment, robot and scenario is an often overlooked but highly critical aspect of your simulation solution. Without a strong and widely supported data model, bringing assets into your platform adds significant friction; workflows get convoluted as the line between platform execution and project/customer specific assets is blurred; and on-boarding customers, especially without deep graphics and simulation experience, becomes challenging. We have adopted USD as the core data model for Duality's platform specifically to avoid these problems. 

Pixar’s Universal Scene Description (USD) 

‍USD was developed by Pixar specifically to address the need for a standardized way to define and manage complex scenes created iteratively by collaborative teams of creative and technical artists. When you compare the complexity of Andy's Toy Story world with more recent films such as the Land of the Dead in Coco, the exponential growth in scene complexity is immediately apparent.                   

We will cover the three core concepts of USD that are critical to setting up a robust, scalable simulation pipeline for robotics. 

‍1. Hierarchical 3D inter-change data format: 

‍Since its release, USD has quickly become the data format of choice in visual effects and animated film production. All major DCC tools such as Autodesk Maya can export USD. Nvidia's Omniverse collaboration platform demonstrates how a USD based pipeline with multiple contributors and tools can be used across large teams and projects. 

‍2. Layer Composition: 

‍USD allows multiple artists and engineers to work on the same assets by working in distinct files, called layers. A well defined, flexible and efficient layer compositing system allows all of these "opinions" to be resolved into a final scene definition. In addition to supporting team collaboration, this mechanism can have unique applications for robotics. For example, a machine definition can be created using component and sub-system product specifications while characterization for specific physical machines can be layered on top of this base specification in a non-destructive way. Similarly generative design exploration can be setup as a layer that explores a specific set of parameters such as sensor position or resolution. 

‍3. Active Support and Vibrant Open Source Ecosystem: 

Given its critical use in large entertainment projects with budgets in excess of $100M, USD is actively supported by Pixar & Disney's engineering teams, significant partners like Autodesk, Apple, Nvidia and a broader community of open source developers. While USD itself is a pure data model, several useful tools have been developed to support USD in pipelines. This includes usdview for viewing, Hydra for rendering, Omniverse for interactive USD editing, and USD Manager as a visual editor. Apple's USDZ format implements compression for augmented reality applications and efficient network transport.

USD and the Duality platform 

‍USD is the core data model for our platform. We have implemented a USD importer that allows us to instantiate scenarios at run time without needing to recompile a UE4 game. This significantly improves pipeline integration for robotics teams and reduces ramp up time on projects. We define scenarios in a standardized way as being made up of "World" and "Characters" scope. The hierarchical structure provides a unified way to specify the 3D transforms, visual meshes & textures, as well as physical & sensor parameters. Simulation results are also stored in layers following the same hierarchy making it straightforward to replay results using usdview while also converting the data to other formats such as rosbagsusing USD's Python binding. 

Our USD Importer 

‍We have developed a USD importer that allows us to instantiate scenarios at run time without needing to recompile a UE4 game. By separating the executable from the simulation context, a number of benefits are delivered to our customers. Teams can now deploy a lightweight simulation executable that can support virtually any team need via an update to the USD. Scenario modifications and design exploration no longer require use of Unreal Editor, lowering the cognitive load on team members. Further, teams benefit from the improved execution of packaged games without needing to wait for the packaging and deployment process. Together, this significantly improves pipeline integration for robotics teams and reduces ramp up time on projects. We take advantage of UE4's Blueprint visual scripting system to create sub-systems templates of bases, attachments and sensors. These components act as building blocks to allow our customers to easily create their exact robot in Unreal. Users can specify component hierarchy, physical constraints and even higher level behavior such as gears and transmission, all within a clear, clean, comprehensive structure. Our Bot Injector populates these hierarchical sub-system templates in real time with machine specific properties and visual appearance from the USD file to instantiate a digital twin. We believe this approach really lends itself to how modern robots are created and allows a robotics team to create a digital twin of their complex robot in hours rather than months. 

Conclusion 

At Duality we are very excited about the potential of combining USD, UE4 and our unique platform to help robotics teams take advantage of simulation for diverse applications ranging from data collection, testing to marcom and training.

We would like to thank Epic for supporting this work with a Mega Grant, the Autodesk Technology Centers for supporting us through their residency program and our founding customers for co-developing this solution with us. Pixar for open sourcing this amazing technology.