Today we’re announcing our first release of 2024 — Falcon product suite version 4.1. Our last release marked a major expansion for the Falcon product suite with the debut of FalconCloud and the early release version of FalconEditor. In the 4.1 release we have issued updates to all three of our core products, and added exciting new features that address customer use cases — making Falcon even more powerful and flexible all the while ensuring that it remains the most accessible digital twin simulation platform to date.

In this blog we highlight select new features arriving with the 4.1 release. You can access the complete release notes along with full Falcon documentation here: https://falcon.duality.ai/secure/documentation/release-notes-4-1

‍(Note for current users: Upgrading to Falcon 4.1 should proceed smoothly as all Falcon 3.3 digital twins and scenarios are forward compatible with the 4.1 release)

FalconSim - V4.1.0201 (Windows and Linux)

‍Interactive Python Console

‍The key to Falcon’s flexibility is our powerful Python API. It enables users to quickly control and customize digital twins and the simulation at runtime. It allows full access of Falcon and Unreal Engine to any tool that users need to integrate for the most accurate simulation results (e.g. controllers, MATLAB models, ROS/ROS2, PX4 code, co-simulation tools, etc.). Now we’ve made it even more seamless to use by introducing an interactive Python console integrated with FalconSim, enabling users to easily modify their simulation world during the simulation run.

‍Real-time Video Streaming of Sensor Data Encoded in h264 Format

‍The recent introduction of our novel Capture Sensor (which offers all of Duality’s camera sensors in a single sensor) helped optimize Falcon sensor performance several fold. Now we’re making it much easier to see the live feed of that sensor data. Falcon 4.1 offers an option to encode the image data in h264 format right after it is rendered (and while still on GPU memory) and immediately stream it in real-time.

Auto Complete Tools for Python Now Available in Falcon

‍Ease of writing Python code in Falcon has been greatly improved with the addition of auto complete tools supporting all available commands specific to Falcon, and all commands from the Unreal Engine that can be accessed inside of Falcon.

‍USD Formatting for System Twins now supports DTES

‍The Digital Twin Encapsulation Standard (DTES) evolved by Duality AI is a vital tool for simulation-ready digital twins. The DTES data structure introduced in Falcon 3.3 helps ensure truly interoperable digital twins that carry all the information they need to be instantly useful in predictive simulation. With Falcon 4.1 we now offer complete DTES schemas for all three digital twin types: environments, systems, and items.

‍Expanded Capture Sensor Capabilities

‍As mentioned above, the Capture Sensor offers all of Duality’s camera sensors in a single sensor, greatly optimizing sensor performance, simplifying sensor configuration, and providing novel data that previously could not be accessed. It can be queried for individual sensing modalities, such as an RGB color image, but also provides additional scene information (thanks to Duality’s custom build of the Unreal Engine). With this update we’re continuing to improve the Capture Sensor by giving it more capabilities that make it easier to configure and replicate individual sensors.

FalconCloud

FalconCloud makes FalconSim fully accessible from a web browser, enabling users to run high-fidelity simulations at any scale without any special hardware or software installations. FalconCloud makes it easy for teams to share simulation assets and resources and acts as the central point for learning how to use FalconSim (including hosting all documentation). New updates are focused on furthering FalconCloud accessibility and bringing more diverse public scenarios to the cloud for all users.

Simplified Scenario Launching

‍This update improves the users experience by consolidating scenario launching process steps, requiring fewer actions needed from the user.

Python API Documentation for FalconSim

‍Along with the auto complete tools released for FalconSim, FalconCloud now also hosts comprehensive API documentation of all the commands users can call inside of their python scripts, making Falcon onboarding that much easier for new users. The new documentation can be found HERE.

Exciting New Public Scenarios

‍All public scenarios are available to all FalconCloud users. (Note: if you are not a current registered user, you can simply make a free FalconCloud account to access all public scenarios.)

Embodied AI Scenarios:

‍Generative AI models are broadening the horizons of what robots can do. But their behavior can be unpredictable, and even dangerous. Digital twin simulation in Falcon is the perfect tool for safely studying, testing, and tuning the performance of these models. These new scenarios enable anyone to experiment with various AI models to see their behavior first-hand.

‍Infrastructure Inspection: Users prompt a LLM-enabled (users select GPT4 or LLAMA)drone with natural language inputs. These LLMs convert prompts into running code in order to navigate an expansive rural landscape filled with solar panels, utility poles, trees and various other elements.

Downtown - Visual Reasoning: Users control a bipedal mannequin equipped with ViperGPT, a framework that leverages code-generation models to compose vision-and-language models into subroutines to produce a result for any query. Users query the model about features in its field of view and answers present as natural language replies, bounding boxes, or segmentation masks.

Embodied AMR Maze: Users engage with an Autonomous Mobile Robot (AMR) within a simple maze setting. The AMR is equipped with GPT4, and the maze features digital twins of various objects that the AMR can reason about an maneuver towards.

City Park Visual Reasoning: Users operate a bipedal mannequin in a photorealistic park. They prompt the GroundingDINO object detection model to identify objects and features in its field of view.

Machine Learning Scenarios

We have also introduced scenarios that exhibit FalconCloud's capabilities to generate high-quality synthetic data vital for ML model training and testing.

Forest Inspection - Quadcopter: Demonstration of Falcons's ability to handle complex, physics-based tasks in a simulated environment. Users experience the intricate physics-based controls of a quadcopter while piloting it to inspect electric poles and capture images of any defects.

Grocery Store Checkout: This object detection example scenario lets users drop items into a shopping basket, and generate images of every iteration. Large generated data sets are intended for training ML algorithms to identify individual products without need for taking real photographs.

FalconEditor - Alpha Release V4.1.01222 (Windows)

The new version of FalconEditor is in alpha release. Recent updates include DTES support for system twins, UX/UI updates, and more scenario editing tools. With vital new features FalconEditor is rapidly approaching its general release stage, and the goal of enabling users to visually compose complete simulation scenarios for any use case.