Until now spatial simulation has generally involved a tradeoff between scale and fidelity. The ability to simulate and learn from complex, city-scale scenarios carries transformative potential for fields ranging from civil engineering to policy decision making. And with increasing adoption of autonomous systems in strategic planning, comprehensive simulation-based testing and validation has become imperative - for both safety and efficacy reasons.

To achieve this full-scale testing, a simulator must be able to maintain a high level of complexity on a massive scale with a large populace of concurrent agents. The reason for this is that complex systems require more that just an evaluation of their top-down, command and control strategy on a large scale. They also need continuous, realistic feedback of how their environment (human and machine) responds to and progressively shapes those controls and policies. This continuous feedback serves to help adapt the overall strategy to successfully function in the real world. But the scale-fidelity tradeoff has been a barrier to doing this successfully.               

So why has sustaining high-level simulation performance at an increasingly large scale been such a challenge? Simply put, simulating large swaths of the physical world requires vast computational resources, which in turn limits the scale of feasible simulations. Furthermore, as scale grows, the time and human investment to set up the simulation can increase disproportionately, putting the project out of reach. But Duality’s recent collaboration with Amazon Web Services (AWS) opens the door to a promising future: a new approach to move beyond the scaling limitations. Duality and AWS first showcased this approach at AWS re:Invent 2022, with a demo titled: Rush Hour Emergency and Response Simulation in Smart Cities.

Video: Highlights from Rush Hour Emergency and Response Simulation in Smart Cities Demo presented at AWS re:Invent 2022.

The simulation illustrates what Falcon, Duality’s digital twin simulator, can do when integrated with AWS SimSpace Weaver, a fully managed compute service that helps customers build, operate, and oversee large-scale spatial simulations. To demonstrate the features of our collaboration, we posed a hypothetical question: Can city planners use simulation to gain valuable insights for improving ambulance response times in the real world? Our demo offers a successful proof of concept for how such a simulation can be implemented. The EMS use case demonstrates high fidelity physics, photoreal rendering, virtual sensors, and real time performance with over a million concurrent agents. Additionally, it spotlights how Falcon with AWS SimSpace Weaver can take reality-based environments and systems developed for the Epic Games’ Unreal Engine ecosystem and make them instantly available for massive scale spatial simulation.

Video: With AWS SimSpace Weaver integration, Falcon can run its entire library of virtual sensors in a true city-scale environment.

In the following sections we break down our demo and introduce the Falcon-AWS SimSpace Weaver infrastructure. We walk through our customization of the immense Unreal Engine 5 City Sample environment (designed by our partner, and Unreal Engine creator, Epic Games) and illustrate how this open world experience, in conjunction with MassAI and MetaHumans, was repurposed to function as an emergency response simulator. Lastly, we discuss the technical underpinning of the integration between Falcon, Unreal Engine, and AWS SimSpace Weaver, which opens the door to truly massive, high-fidelity simulation. 

‍

Rush Hour Emergency and Response Simulation in Smart Cities Demo

For saving lives during medical emergencies, improving EMS response times remains the best predictor of successful outcomes. In our ever-evolving cities, this is a constant challenge. As populations grow, neighborhoods develop, traffic patterns change, and gridlocks become common at rush hour. City-scale simulation can help emergency preparedness planners better understand the obstacles faced by ambulance drivers and test out interventions to improve performance.

For this demo, we simulated a single EMS unit responding to a 911 call originating in a randomly chosen part of the city. Several routes are calculated using an A* algorithm, and their unique traffic complications are identified along with respective ETAs.                    

A participant, via high-level controls, teleoperates a semi-autonomous EMS vehicle along the selected route, navigating the traffic and pedestrian conditions, while a timer keeps track of their progress.                  

To illustrate the potential for testing out coordinated interventions, a second agent in the form of a traffic officer preemptively attempts to clear congested intersections to create a path for the ambulance. When the ambulance arrives on scene, its final time is recorded and compared to previous runs as well as the estimated times.

As mentioned at the outset: a key feature of Falcon is the ability to easily utilize any environment and system developed for the vast Unreal Engine ecosystem. This means that instead of building every detail from scratch, high quality assets are often available to be used or modified - a significant benefit for efficient simulation deployment. For our demo, we chose the City Sample environment, which is based on The Matrix Awakens: An Unreal Engine 5 Experience.              

Built as a photo-realistic, interactive open world using Unreal Engine 5, The Matrix Awakens is a sprawling environment with tens of thousands of buildings, MassAI vehicles, and MetaHuman characters exhibiting realistic behaviors. Developed by Epic Games, this virtual metropolis was efficiently created via a procedural approach with SideFX’s Houdini.                    

Starting with this project, Duality’s team designed and implemented some significant modifications in order to adapt City Sample to this use case. The list includes: implementing traffic-aware route optimization, building teleoperation controls and driving rules for the semi-autonomous ambulance, creating a novel traffic officer mechanic for rerouting vehicles, adding sensors to the traffic officer and ambulance for external data analysis, designing a new traffic light dynamic, and modifying visual assets.

The final result is a real-time, city-scale simulation with all the capabilities of Falcon aimed at gathering perception data from the ambulance and learning from the accumulated results of multitudes of runs.  

How Does Falcon Work with AWS SimSpace Weaver?

An enterprise metaverse powered by a high-fidelity digital twin simulator like Falcon is the natural path for realizing this type of massive simulation. Falcon’s ability to build photoreal environments at run-time, leverage vast libraries of Unreal Engine ecosystem assets, and render high quality perception data with a user-friendly interface and API directly addresses the requirements of our customers. However, scale limited by resource constraints, especially in the cloud-computing sphere, has long stood as a barrier. Integration with AWS SimSpace Weaver presents a robust and cost-effective path for clearing the scaling hurdle.

AWS SimSpace Weaver handles provisioning Amazon Elastic Compute Cloud (Amazon EC2) instances as well as managing the network and compute infrastructure. Instead of running a spatial simulation as a single resource heavy instance, AWS SimSpace Weaver divides the spatial domain of the simulation environment into equally sized partitions and allows individual instances of Falcon to run in every partition and coordinate simulation state with neighboring partitions.

Not unlike an air traffic controller, AWS SimSpace Weaver keeps track of and manages all the state information representing each simulated entity – resulting in all the distinct instances of the simulation functioning as a unified, seamless whole. As digital twins move from one partition to another, the relevant Falcon instance takes charge of simulating that entity. AWS SimSpace Weaver handles data replication to neighboring partitions and client agents that subscribe to a particular area of interest or state information for specific digital twins.                

Since each partition is simulated within a separate instance of Falcon, the computational resources required per partition are bounded. Furthermore, AWS SimSpace Weaver supports grouping partitions into multiple servers for even greater scalability. By leveraging the massive cloud infrastructure of AWS, the available computational resources can now scale with the needs of the spatial simulation no matter how demanding: the full capabilities of Falcon with high fidelity perception sensors and physics can concurrently run on as many partitions, and on as many servers as needed.

As noted above, AWS SimSpace Weaver is a managed compute service and not itself a simulation engine. This means that it must be integrated with a spatial application that performs the simulation calculations. Falcon, as a best-in-class digital twin simulator, fits perfectly within this architecture. Once fully integrated, Falcon has a novel way of handling all of the data transmission within the simulation and between simulation and the end-user. While building this integration was a significant part of the project, it was done in record time due to Falcon’s open API and data model. Specifically, there are three processes involved: the headless instances of Falcon Sim, Python Custom Comms apps, and the Falcon interactive Client app.

All transmission of data is done by coordinating between these processes and the AWS SimSpace Weaver SDK. In broad terms, at each tick, the headless Falcon instances simulate the state of the entities in their respective partitions and register the updated states with AWS SimSpace Weaver. The Custom Comms apps exists to facilitate communication between individual Client and the multi-partition simulation infrastructure. It enables the interaction with the simulated digital twins as well as visualizing relevant areas of interest within the overall simulation. The Client receives updated digital twin state information, updates its visualization of the scene, and finally sends the updated state information for the digital twin that the Client controls to the relevant Agent app. Finally, the Custom Comms app registers the updated state information with AWS SimSpace Weaver so that it can be factored into the next simulation tick.

To the end user, whether teleoperating a digital twin or controlling it with an autonomy stack, the result presents no differently than running our standard version of Falcon.‍

We’re thrilled by the result of our collaboration with AWS and the potential of AWS SimSpace Weaver to enable Falcon’s capabilities at an increasingly larger scale. When paired with high quality digital twins, the Falcon-Unreal-AWS SimSpace Weaver integration opens a new level of possibility in leveraging the enterprise metaverse for solving complex real-world problems within realistic budgets and schedules for our customers. At Duality, our singular focus is on helping our customers leverage digital twins to solve high impact problems, and this collaboration represents a significant step in that direction.