KEF Robotics: Tethered Drones Learn Safe Navigation of Complex Environments

KEF Robotics provides state-of-the-art autonomy software to fly aircraft without a human pilot. KEF’s mission is ambitious and timely — to improve the safety and reliability of aircraft, expand their range and utility, and allow them to fly without GPS in uncooperative environments.

In service of this mission, KEF’s algorithms leverage passive, cameras-only sensor suites which enables autonomous flight across a wide variety of platforms and use cases. This made the company the ideal candidate for SBIR funding agreement with the US Army, which tasked them with developing autonomy software for a Tethered Unmanned Aircraft System (TeUAS).

Background

In a TeUAS, a drone is connected to a ground vehicle via a cable (or tether) that transmits power and data.

Tethered drones can stay aloft for much longer periods of time compared to conventional drones. But the tether is also a source of vulnerability due to hazards such as power lines, among many others. (Image courtesy of KEF Robotics)

Tethered drones provide specific benefits over their more familiar untethered counterparts. Since power is delivered via the tethering cable, these drones can stay aloft indefinitely, providing a perpetual lookout and improved radio transmission range. And since they don’t need to store or process any data onboard, they can carry more powerful sensors, transmitters, and payloads. Due to these benefits, TeUAS play a valuable role in security and communication - a radio tower on demand.

Challenge

While the tether offers numerous benefits for stationary assets, it becomes a point of vulnerability when flown above a moving ground vehicle. As the vehicle navigates various environments, the tether can be caught by frequent hazards that can impact and damage it along with the drone. Any TeUAS operating in the real world needs to reliably recognize these hazards and navigate the drone to avoid them as the vehicle drives through any type of environment.

In order to operate in the real world, tethered drones need to reliably detect quickly approaching hazards and adjust their path accordingly, while keeping pace and maintaining connection with the ground vehicle. (Image courtesy of KEF Robotics)

Goal

The KEF team took on the task of developing TeUAS autonomy software that could fly above a ground vehicles while reliably avoiding structures that would impact the tether or the drone. To show real world utility, the program required demonstrations in diverse operational environments.

Approach and Workflow

"We're excited by the potential for simulation to dramatically increase the velocity at which we can validate our aerial autonomy software, and identified Duality's Falcon tool as the best visual scene generator in the trade space" - Fraser Kitchell, CEO, KEF Robotics

Accomplishing this ambitious goal requires software capable of rapidly analyzing camera images to detect distant obstacles in time for the system to plan routes that avoid them. KEF Robotics employs a large neural network for this demanding task. Neural networks are powerful tools frequently leveraged in this way, but deploying these nets in complex scenarios requires significant development, training and testing efforts. Choosing models pre-trained on large-scale, real-world object detection, segmentation, and captioning datasets (e.g. COCO, Cityscapes, etc.), accelerates this work by providing a baseline. However, these datasets lack the classes and variety required to meet KEFs performance targets. Synthetic data generated in Falcon presents a way to efficiently close this performance gap.

Example of Falcon environment with obstacles commonly faced by tethered drones - power lines, road signs, and more. Leveraging synthetic data generated from diverse lighting conditions is vital for learning reliable hazard avoidance in any real-world scenario.

Duality provided the KEF team with fit-for-purpose, customizable, high-fidelity digital twins of environments (rural and urban) and systems (drone and vehicle), as well as smart assets representing varieties of requisite hazards. These smart assets (e.g. power lines, road and guide signs, trees, etc.) were then deployed by KEF’s robotics engineers to create the varieties of scenarios needed to drive system performance to reach the program’s goals.

Examples of rural and urban environments used for simulation in Falcon

By integrating their autonomy software with the Falcon simulator, KEF’s team was able to:

Run simulations of realistic and domain-appropriate scenarios.
Generate synthetic data from large-scale, performant, photoreal, and endlessly customizable environments.
Explore and optimize their drone’s sensor suite configuration.
Fine tune their models performance by training with intentionally generated data.

The end result enabled KEF Robotics to exhibit successful performance in BOTH simulation and real-world testing.

"In the past, KEF often faced a trade-off between simulating expansive environments and maintaining high fidelity. With Duality's Falcon simulator, this compromise is no longer a concern. Falcon enables us to rigorously test our autonomy algorithms in photorealistic environments an order of magnitude larger than any we’ve previously used.” - Paul Frivold, Robotics Research Engineer, KEF Robotics

KEF Robotics leveraged Duality’s photorealistic environments created to match their intended domain of operation. One example is the Iowa farmland environment based on real geospatial data. Top: Overview of resulting virtual environment (left) and an aerial view (right). Bottom: Roadways in the environment with customizable lighting and weather conditions.

A key part of this approach for generating the kind of data KEF required was the use of Duality’s GIS-pipeline to develop site twins: environment digital twins that replicate a specific location in the real world. For this program, two site twins were developed, re-creating two unique locations in the Driftless Region of the midwest United States. These site twins provided KEF with realistic and domain-appropriate testing and training environments, with accurate statistical and geographical distributions of features as they present in the real world.

Falcon's tunable worlds enabled KEF to rapidly generate variations of scenarios necessary for successful detection and avoidance of obstacles a tethered drone may face. In addition to vast varieties of hazards, KEF engineers could specify locations and varieties of power lines needed for a scenario (e.g. distance, height, line gauge and slack, etc.) and Falcon would automatically spawn the poles and power lines with realistic physics and appearance.

“Duality’s USD based approach to modifying their environments lets us quickly add, move, and customize objects of interest like power lines. The Falcon simulator represents a significant advancement in our simulation capabilities, balancing both scale and detail without compromise." - Paul Frivold, Robotics Research Engineer, KEF Robotics

Realistic environments were the starting point, but robots need to be trained and tested in more than a single, static world. Duality’s tunable worlds allowed the KEF team to specifically or statistically control any relevant features, including obstacles that their systems must detect and avoid. For example: the first simulation run may contain a street corner shaded by a tree, while on the next run that tree may have moved 2 feet east. On the third run, the tree could be gone while a traffic cone now sits in the middle of the lane. These types of variation control were made easier by a custom system for placing power lines - a major hazard of interest. KEF engineers could specify locations and varieties of power lines needed for a scenario (e.g. distance, height, line gauge and slack, etc.) and Falcon would automatically spawn the poles and power lines with realistic physics and appearance. This kind of iterative scenario control is critical to building, training and testing the sorts of advanced systems that KEF is developing today.

In addition to obstacle and lighting variations, Falcon makes it easy to generate synethic data from neccesary weather conditions.

To fully leverage these environments, KEF made use of Falcon’s runtime API to conduct co-simulation of their validated physics model as well as to connect to their flight hardware. The API also enabled KEF to run multi-machine simulations with a human-in-the-loop controlling the ground vehicle. This type of integration of simulation with their existing, full development process helped the KEF team to more rapidly field a flight stack and test with human users for results that even more closely mirrored real world operation.

KEF’s work resulted in autonomy software that demonstrated safe navigation around diverse structures that pose a hazard to the drone and tether. Accomplishing this goal in simulation ensured a more resilient TeUAS by the time field testing was ready to commence, thus reducing risk of damage to expensive hardware and limiting the need for field-based problem solving.

‍

Curious what Falcon can do for your project? Let's talk!
Or try FalconCloud for yourself with a free trial.

‍