Problem Statement

Duality is a simulation platform that allows engineers to create human-robot collaborative scenarios to speed up deployment time in robotic projects. The goal of this project is to create an end-to-end example demonstrating the capabilities of Duality’s platform. Given a specific GPS location, for example by analyzing satellite imagery, we will use digital twins of consumer off-the-shelf (COTS) Quadcopter and digital camera to determine more precisely the coverage of fire at that location by flying in a low altitude grid pattern.

Project Description

The goal of this project is to check an area for forest fires, and if one is found, assess the severity and size of the forest fire. To do this, we created a geo-accurate, photo-realistic environment of terrain using satellite imagery of a forest within Duality’s simulator, DuSim. A drone is then placed within the environment, and flies to designated GPS coordinates within the forest. The drone takes pictures (640 x 480 px) and sends these images to Google’s Vision API. Should there be a forest fire, the drone will perform a more thorough investigation of the area to assess how widespread the fire is.

Code Description

A high-level view of the architecture used in the code is below:

ForestFireDemo.py is the Python script used to control the drone through DuSim. This uses a Python API to send actuator commands and obtain sensor output from the simulation scenario. The Python script receives GPS coordinates as input to scan for forest fires and controls the drone’s orientation and its resulting GPS location through PID loops.

The script has two modes:

1. Scanning Mode – the drone autonomously flies to the GPS coordinates specified by the user and takes a picture of the forest at a high altitude (100-200 meters above the ground). It then sends this picture to Google Vision’s API and determines if there is a forest fire. If so, move to investigation mode. Otherwise, move to the next GPS coordinate to scan.
2. Investigation Mode – the drone flies to a lower altitude closer to the forest and takes pictures in a snaking grid-like fashion. If the label “Fire” is associated with a picture, this part of the grid is deemed on fire. The total percentage of the investigated area that is on fire is then estimated from this count.

A high-level view of the logic within ForestFireDemo.py is below:

Some other notes on the script:

• Taking a picture requires fetching the texture buffer from GPU memory.
• Fetching the labels from Google’s Vision API requires starting an asynchronous thread. This is because the images would otherwise block the main thread, suspending the simulation.

• The script can save taken images to disk if the user desires
• The time, location, and labels of each picture are recorded within a log
• If desired, teleoperate can be set to True to control the drone manually

As an example, ForestFireDemo demonstrates how DuSim may be used to create your own control logic, actuate a machine, and receive sensor output from a scenario. DuSim performs the heavy lifting – all the Python process needs to do is send actuator commands and receive sensor output through DuSim. By adapting to a specified hardware controller and camera interface, the logic of ForestFireDemo.py script that controls a “digital twin” can be used with a real drone.

Results

The scenario we used was a simple terrain with forests and shrubs that could be set on fire:

The drone first takes a picture at a high altitude in scan mode to determine whether there is a forest fire or not (clearly, there is!):           

Afterwards, the drone enters investigation mode and takes picture in a grid-like fashion:

Improvements and Next Steps

Here are some improvements we hope to make in the future:

1. As it is, the area estimate is just that: an estimate. It can be made an exact area by utilizing more precise bounding boxes for fires located in pictures.
2. It would be nice to be able to stitch all the low-resolution images into a high resolution map of the forest fire. This will require stitching calculations with camera parameters and altitude of the drone using Google’s Elevation API.
3. The drone currently has a fixed altitude for both scanning and investigation mode. Using Google’s Elevation API, this could be changed to be a dynamic altitude, as the drone could hover a fixed distance from the ground.
4. Currently, the move_to_latlngalt function only uses PID loops to move to a desired altitude and attitude. Currently, the latitude and longitude control logic uses only a P term of the PID loop. Navigation of the Quadcopter could be improved by using nested PID loops that also include latitude and longitude targets.

Conclusion and Thanks

Overall, I had a lot of fun with the project and at Duality! I learned a lot about the fine details of drone movement, path-finding, simulation, control logic, physics, and thread handling to make this project work. Hopefully, this end-to-end example showcases some of DuSim’s is capability. I am thankful that I was able to work at Duality this semester!

Special thanks to Jason’s vision for Duality, Mike for the path-finding and PID loops, Aditya for the fire scenario, and Apurva for being a helpful, insightful, and overall awesome mentor!

Sources

• ‍Source Code‍
• Google Vision API‍
• Conversion from GPS to meters