WP4-39: Difference between revisions

From COMP4DRONES
Jump to navigation Jump to search
(Created page with "=Simulated data aggregator supporting intelligent decision in computer vision components= {|class="wikitable" | ID|| WP4-39 |- | Contributor || HI-IBERIA |- | Levels || Functional |- | Require || AirSim built on Unreal Engine |- | Provide || Simulator-based data aggregator built over AirSim generating high amounts of training data (RGB images) to support any computer vision component for intelligent decision in drones. |- | Input || Simulation scenario par...")
 
Line 25: Line 25:
|-
|-
|  C4D building block || (Simulated) Data Acquisition
|  C4D building block || (Simulated) Data Acquisition
|
|-
|  TRL || 6
|  TRL || 6
|}
|}


==Detailed Description==
==Detailed Description==
Computer vision is a significant driver in the new era of drone applications, but developing and testing computer vision components for drones in real world is an expensive and time-consuming process. This issue is further exacerbated by the fact that drones are often unsafe and expensive to operate during the training phase. Additionally, one of the key challenges with these techniques is the high sample complexity - the amount of training data needed to learn useful behaviors is often prohibitively high, but in fact, it is needed to collect a large amount of annotated training data in a variety of conditions and environments in order to utilize recent advances in machine intelligence and deep learning. This unfortunately involves not only developing the proposed ML algorithms but also requires vast amounts of time dedicated for the development of an infrastructure able to generate training models in a variety of environments. Consequently, the design, deployment, and evaluation of  computer vision components for drones becomes a complex and costly endeavor for researchers and engineers, since it requires the exploration of multiple conditions and parameters through repeatable and controllable experiments.
In the context of C4D, the proposed component is a simulator-based data aggregator built over AirSim which intends to generate high amounts of training data with the objective of supporting any computer vision component for intelligent decision in drones. This component will allow speeding up the constructive process of a civil infrastructure while saving costs, by reducing the need to perform multiple data collection campaigns to get real training data. Additionally, this component could assess the need of different cameras/sensors (RGB, LIDAR, ...) integrated in the real drone, starting from the simulated clouds of points generated by the component. This would allow detecting which kind of camera could provide more suitable results for the analysis before launching the real drone flight.
In the context of UC2 – Construction, the specific role of the simulated data aggregator is to generate a vast amount of annotated training data which allows us to train the convolutional neural networks which will be implemented in the Computer Vision Component for Drones, which is being developed within the task 3.3. Particularly, this simulated data aggregator will generate a vast amount of RGB images (or point-cloud as a flat array of floats along with the timestamp of the capture and position) which intend to represent the real scenario from scenario-tailored input data, that is, the specific drone configuration parameters, the digital terrain model and the 3D models (CAD files) of the objects to detect, being traffic signs in the particular case of the UC2 - Demo 1. Such RGB images will serve to train the convolutional neural networks, and hence, the DL algorithms of the Computer Vision Component for Drones in an early way.
Considering the C4D Reference Architecture Context, the role of the simulator-based data aggregator is clearly referred to the '''''Data acquisition''''' building block in the ''Payload management'' block. Particularly, this component contributes to the reference architecture by simulating the behaviour of the '''''Data acquisition''''' building block, since it allows generating simulated payload data which serves as training payload data for  the ''Data Analytics'' building block in the ''Data Management'' block.
Additionally, the simulator-based data aggregator contributes to the ''UC2-D1-FUN-09-Specialized software for georeferenced point-cloud creation'' as well as to the following requirements of the UC2-DEM1:
* UC2-DEM1-FUN-01 - The drone system shall capture a high-density point cloud.
* UC2-DEM1-FUN-02 - The drone system shall capture RGB data of the surface.
* UC2-DEM1-FUN-03 - The point cloud shall be in an open format such as LAS.
Therefore, such component also addresses the following KPIs for the UC2-DEM1:
* UC2-D1-KPI-T1 - Recognition of work elements through AI: Detection of main work elements position in the road through point cloud
* UC2-D1-KPI-T2 - Recognition of work elements through AI: Detection of the total number of elements.
==Technical specification==
From a technical point of view, the simulated data aggregator has been implemented over already existing tools: '''Unreal Engine''' and '''AirSim'''. '''Unreal Engine''' is a complete suite of creation tools for game development, architectural and automotive visualization, linear film and television content creation, broadcast and live event production, training and simulation, and other real-time applications. '''AirSim''' is a simulator for drones, cars and more, built on '''Unreal Engine'''. It is open-source, cross platform, and supports software-in-the-loop simulation with popular flight controllers. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment.
The recommended hardware requirements specified for the simulation tools are:
* '''Unreal Engine:'''
** Operating System: Windows 10 64-bit
** Processor Quad-core Intel or AMD, 2.5 GHz or faster
** Memory 8 GB RAM
** Graphics Card DirectX 11 or 12 compatible graphics card
** RHI Version:
*** DirectX 11: Latest drivers
*** DirectX 12: Latest drivers
*** Vulkan: AMD (21.11.3+) and NVIDIA (496.76+)
* '''AirSim Simulator:'''
** Operating System: Windows 10 64bit
** CPU: Intel Core i7
** GPU: Nvidia GTX 1080
** RAM: 32 GB
However, for the implementation of the Unreal+AirSim simulator server under the Comp4Drones domain, the technical specifications required are the following regarding software and hardware infrastructures:
*'''Software:'''
** Windows 10
** Python 3.7.9
** Unreal engine Version 4.25
** AirSim 1.3.0
*'''Hardware:'''
** Operating System: Windows 10 64bit
** CPU: Intel Core i5
** GPU: Nvidia GTX 980
** RAM: 32 GB
Regarding the interfaces, there are two interfaces: the scenery creator based on Unreal Engine and AirSim for the simulation itself. Each of the interfaces requires specific inputs and provides specific outputs as it is descripted below:


==Application and Improvements==
==Application and Improvements==

Revision as of 14:52, 9 May 2022

Simulated data aggregator supporting intelligent decision in computer vision components

ID WP4-39
Contributor HI-IBERIA
Levels Functional
Require AirSim built on Unreal Engine
Provide Simulator-based data aggregator built over AirSim generating high amounts of training data (RGB images) to support any computer vision component for intelligent decision in drones.
Input Simulation scenario parameters:
  • Digital terrain model imported in Unreal plugin (optional, but recommendable)
  • Objects to be detected in CAD format (Traffic signs in CAD format)

Drone configuration parameters in JSON format:

  • Vision angle
  • Inclination
  • Viewing depth
  • Flight height
Output Simulated RGB images (or point-cloud as a flat array of floats)
C4D building block (Simulated) Data Acquisition
TRL 6

Detailed Description

Computer vision is a significant driver in the new era of drone applications, but developing and testing computer vision components for drones in real world is an expensive and time-consuming process. This issue is further exacerbated by the fact that drones are often unsafe and expensive to operate during the training phase. Additionally, one of the key challenges with these techniques is the high sample complexity - the amount of training data needed to learn useful behaviors is often prohibitively high, but in fact, it is needed to collect a large amount of annotated training data in a variety of conditions and environments in order to utilize recent advances in machine intelligence and deep learning. This unfortunately involves not only developing the proposed ML algorithms but also requires vast amounts of time dedicated for the development of an infrastructure able to generate training models in a variety of environments. Consequently, the design, deployment, and evaluation of computer vision components for drones becomes a complex and costly endeavor for researchers and engineers, since it requires the exploration of multiple conditions and parameters through repeatable and controllable experiments.

In the context of C4D, the proposed component is a simulator-based data aggregator built over AirSim which intends to generate high amounts of training data with the objective of supporting any computer vision component for intelligent decision in drones. This component will allow speeding up the constructive process of a civil infrastructure while saving costs, by reducing the need to perform multiple data collection campaigns to get real training data. Additionally, this component could assess the need of different cameras/sensors (RGB, LIDAR, ...) integrated in the real drone, starting from the simulated clouds of points generated by the component. This would allow detecting which kind of camera could provide more suitable results for the analysis before launching the real drone flight.

In the context of UC2 – Construction, the specific role of the simulated data aggregator is to generate a vast amount of annotated training data which allows us to train the convolutional neural networks which will be implemented in the Computer Vision Component for Drones, which is being developed within the task 3.3. Particularly, this simulated data aggregator will generate a vast amount of RGB images (or point-cloud as a flat array of floats along with the timestamp of the capture and position) which intend to represent the real scenario from scenario-tailored input data, that is, the specific drone configuration parameters, the digital terrain model and the 3D models (CAD files) of the objects to detect, being traffic signs in the particular case of the UC2 - Demo 1. Such RGB images will serve to train the convolutional neural networks, and hence, the DL algorithms of the Computer Vision Component for Drones in an early way.

Considering the C4D Reference Architecture Context, the role of the simulator-based data aggregator is clearly referred to the Data acquisition building block in the Payload management block. Particularly, this component contributes to the reference architecture by simulating the behaviour of the Data acquisition building block, since it allows generating simulated payload data which serves as training payload data for the Data Analytics building block in the Data Management block.

Additionally, the simulator-based data aggregator contributes to the UC2-D1-FUN-09-Specialized software for georeferenced point-cloud creation as well as to the following requirements of the UC2-DEM1:

  • UC2-DEM1-FUN-01 - The drone system shall capture a high-density point cloud.
  • UC2-DEM1-FUN-02 - The drone system shall capture RGB data of the surface.
  • UC2-DEM1-FUN-03 - The point cloud shall be in an open format such as LAS.

Therefore, such component also addresses the following KPIs for the UC2-DEM1:

  • UC2-D1-KPI-T1 - Recognition of work elements through AI: Detection of main work elements position in the road through point cloud
  • UC2-D1-KPI-T2 - Recognition of work elements through AI: Detection of the total number of elements.

Technical specification

From a technical point of view, the simulated data aggregator has been implemented over already existing tools: Unreal Engine and AirSim. Unreal Engine is a complete suite of creation tools for game development, architectural and automotive visualization, linear film and television content creation, broadcast and live event production, training and simulation, and other real-time applications. AirSim is a simulator for drones, cars and more, built on Unreal Engine. It is open-source, cross platform, and supports software-in-the-loop simulation with popular flight controllers. It is developed as an Unreal plugin that can simply be dropped into any Unreal environment. The recommended hardware requirements specified for the simulation tools are:

  • Unreal Engine:
    • Operating System: Windows 10 64-bit
    • Processor Quad-core Intel or AMD, 2.5 GHz or faster
    • Memory 8 GB RAM
    • Graphics Card DirectX 11 or 12 compatible graphics card
    • RHI Version:
      • DirectX 11: Latest drivers
      • DirectX 12: Latest drivers
      • Vulkan: AMD (21.11.3+) and NVIDIA (496.76+)
  • AirSim Simulator:
    • Operating System: Windows 10 64bit
    • CPU: Intel Core i7
    • GPU: Nvidia GTX 1080
    • RAM: 32 GB

However, for the implementation of the Unreal+AirSim simulator server under the Comp4Drones domain, the technical specifications required are the following regarding software and hardware infrastructures:

  • Software:
    • Windows 10
    • Python 3.7.9
    • Unreal engine Version 4.25
    • AirSim 1.3.0
  • Hardware:
    • Operating System: Windows 10 64bit
    • CPU: Intel Core i5
    • GPU: Nvidia GTX 980
    • RAM: 32 GB

Regarding the interfaces, there are two interfaces: the scenery creator based on Unreal Engine and AirSim for the simulation itself. Each of the interfaces requires specific inputs and provides specific outputs as it is descripted below:

Application and Improvements