WP3-19 1: Difference between revisions

From COMP4DRONES
Jump to navigation Jump to search
(Created page with "=Hyperspectral payload= {|class="wikitable" | ID|| WP3-19_1 |- | Contributor || IMEC |- | Levels || Function |- | Require || Power, GPS coordinates, data & control channel to ground controller |- | Provide || Data analytics & spectral data |- | Input || Start/end capture signal, exposure time, GPS coordinates, system time |- | Output || Current captured hyperspectral frame, classified images, data analytics, spectral hyperspectral data |- | C4D buildin...")
 
 
(3 intermediate revisions by one other user not shown)
Line 18: Line 18:
|-
|-
|  TRL || 6
|  TRL || 6
|-
|| Contact || Murali.Jayapala at imec.be
|}
|}


[[File:wp3-14_1_01.png|frame|center|Building block diagram for WP3-14_1]]
[[File:wp3-19_1_01.png|frame|center|Building block diagram for WP3-19_1]]


==Detailed Description==
==Detailed Description==
Essentially, the Hyperspectral payload captures hyperspectral images, tags the images together with GPS coordinates, stores the images locally, processes the data to provide certain analytics, and sends the raw and classified/processed images to ground controller.


The growing popularity of small civilian drones has generated a wide array of complex and unprecedented challenges related to the risks posed to security, safety, potential to disrupt people’s privacy and interfere with activities on the ground and with the manned aircrafts. This necessitates the development of technologies that allow UAVs to safely navigate according to the regulations. Geofencing technique can be used to specify no-fly zones for drones. It is a technique that defines virtual boundaries in a specific geographical area. Once these virtual boundaries are defined, the drone should never be able to penetrate through these boundaries. In other words there should be a potential barrier that prevents drones from crossing the geofence boundary and redirect their trajectory to avoid any conflict. The drones must not only respect the geofence but should also be capable of detecting and avoiding any obstacle in their path. The obstacles could be stationary (e.g., buildings) or moving objects (e.g. birds, other drones or manned aircrafts). A drone could either be autonomous or remotely controlled by an operator. In both cases, the potential barriers must repel the drone automatically and generate a warning of alert whenever it is near the restricted area or any obstacle in path that could cause a collision.
The hyperspectral payload can be coupled together with a certain drone system. An example of the integration is shown in the figures below. The integration is on three aspects: physical connection, control interface and the data connection between drone and payload. The physical connection is usually done via a gimbal. The control connection is done via a serial port over which commands can be sent to the payload to set the relevant parameters for the sensors in the payload. The data connection is done via Ethernet/HDMI connection, where the drone controller can request specific frames/info from the sensors that can eventually be sent to a ground controller. This information can also be used to synchronize between payload data and other sensors (e.g., GPS) or other payloads connected to the drone. The images from this sensor are unique in that they can provide hyper/multi-spectral images of up to 40 spectral bands in the range 450-900 nm.


[[File:wp3-14_1_02.png|frame|center|Geofencing example]]
==Contributions and Improvements==
Currently, there are no real lightweight hyperspectral UAV cameras which have more than 4/5 bands. Such a camera would be a real breakthrough in the domain of UAV precision agriculture. Parrot’s sequoia multispectral camera with about 4-5 spectral bands is the leading state of the art in this domain. However, with 4-5 spectral bands only simple agriculture indices like NDVI can be extracted. Tetra cam’s 3-filter camera or multi-camera systems supporting up to 12 bands are other alternatives. However, multi camera systems lead to much more bulkier systems with additional complexity of software to register images from different cameras to obtain the same spatial field of view, which could potentially lead to loss in image quality. For our target applications more spectral information would be required (>10 bands in VISNIR) to provide accurate diagnostic and actionable information. Our proposed camera can enable such applications. Headwall’s micro-hyperspec is another camera intended for UAV platforms, which uses conventional grating-- based solutions for the spectral unit. This leads to a bulkier camera than our proposed solution, making this unsuitable for lightweight drones. Micro-hyperspecs cameras can weigh up to 1kg or more, making this perhaps more suitable for larger drones/UAVs.


==Specifications and contribution==
Compared to other multispectral payload systems, this system has a significant improvement in the number of spectral bands, typically from 5-10spectral bands to up to 40 spectral bands. This enables higher precision and accuracies in current inspections and also enables new inspection methods (e.g., inspection of soil quality, which conventionally would have required sending the samples to a lab).


The purpose of this component is to provide a mechanism which ensures that the collision between the drones and obstacles never happens and geofence is always preserved. From a technical viewpoint, preventing drones from violating the boundaries defined by the geofence system can be considered as a constrained control problem. Constrained control addresses the problem of enforcing constraints satisfaction at all times while ensuring that control objectives are achieved. It is required that the geofence information is provided to the drone and that information could be updated online while the drone is in operation. In order to avoid collision with an obstacle, it is assumed that drone is equipped with proximity sensors and can detect any relative position of both non-cooperative and cooperative entities within a sensing range. A cooperative entity is another drone which has the same capability. In contrary, a non-cooperative entity is an entity without collision avoidance system. Sensing capability is required to sense the presence of any other entities in its close vicinity which may lead to a collision. These sensors only give the relative position of any entity in its range in the local frame and do not provide position information in the global frame. It is to note that this assumption is used for the purpose of collision avoidance only.


==Design and Implementation==
==Design and Implementation==
A potential barrier for geofencing and collision avoidance can be achieved using Artificial Potential Function (APF). In APF, a drone is considered as a point in a potential field. This drone experiences a repulsion force from the obstacles or geofence and therefore, instead of colliding with them, it steers away from them. Typically, potential functions are based on the relative distance between drone and the obstacle or the geofence and do not require any global information. Based on the practical aspects, an ideal potential function must have the following properties:
The core design of the payload and its integration with the drone system is shown in Figure below.
*The range of the potential field must be bounded. Usually, it depends on the range of obstacle sensors mounted on the agent.
 
*The value of the potential field and the corresponding repulsion must be infinity at the boundary of the obstacle/geofence and must decrease with the increase in the distance
 
*First and second derivatives of the potential function must exist in order to have a smooth repulsion force
[[File:wp3-19_1_02.png|1000px|thumb|center|System architecture of UAV payload with compute enabled system]]
 
A first prototype payload has been built by Airobot to be able successfully perform first data collection flights. The pictures below show the integration on the Airobot Mapper drone and of the first test flight.
 
[[File:wp3-19_1_03.png|frame|center|Prototype payload on Airobot Mapper]]
 
As a second prototype, IMEC-BG has implemented a second iteration of the payload. The payload consists of two multispectral sensors in the spectral range 470-900nm, a jetson Tx2 to enable onboard computation and about 1TB of storage to collect spectral data during flight. In addition to the payload integration with a drone as described by Airobot, IMEC has done initial integration (both hardware and software) of this payload with a DJI M600 drone. This integration was done to show the modular and flexible aspect of our payload and data acquisition software blocks. Current software development for this version of the prototype has two parts (1) firmware/acquisition software running on the payload/jetson system and (2) ground controller software running on a tablet which is connected to a drone controller. The key functionality of firmware block is to capture the data from the two multispectral sensors and perform initial pre-processing steps and store them on onboard disk. The key functionality of the ground controller software is to enable user to control camera parameters and to provide a live preview of the images from spectral sensors.
 
[[File:wp3-19_1_04.png|frame|center| Prototype IMEC payload]]


APF based repulsion mechanism is combined with the control algorithm, for instance it could be combined with the position control of the drone. The repulsive force remains zero when the distance is greater than some predefined value and position controller works normally. However, when the distance becomes less than the threshold, the repulsive force comes into play and lesser the distance more will be the repulsive force. Figure 55 shows a graph of distance based potential function.
Finally, Airobot has been working together with IMEC on working out the detailed design to integrate their payload. The mechanical, electrical and software interface has been defined. To speed up the development a setup has been created so that the software development can be done without needing physical access to the drone.


[[File:wp3-14_1_03.png|frame|center|Potential function for collision avoidance and geo fencing]]


The detection range is considered as 2 meters and the protection radius around the drone is 0.5 meters. It can be observed that the value of the potential field increases as the distance between the fence/obstacle and the drone. This will act as the repulsive force and push the drone away from the fence/obstacle.
[[File:wp3-19_1_05.png|1000px|thumb|center|Architecture of interface with IMEC payload ]]

Latest revision as of 08:36, 10 March 2023

Hyperspectral payload

ID WP3-19_1
Contributor IMEC
Levels Function
Require Power, GPS coordinates, data & control channel to ground controller
Provide Data analytics & spectral data
Input Start/end capture signal, exposure time, GPS coordinates, system time
Output Current captured hyperspectral frame, classified images, data analytics, spectral hyperspectral data
C4D building block Hyperspectral payload
TRL 6
Contact Murali.Jayapala at imec.be
Building block diagram for WP3-19_1

Detailed Description

Essentially, the Hyperspectral payload captures hyperspectral images, tags the images together with GPS coordinates, stores the images locally, processes the data to provide certain analytics, and sends the raw and classified/processed images to ground controller.

The hyperspectral payload can be coupled together with a certain drone system. An example of the integration is shown in the figures below. The integration is on three aspects: physical connection, control interface and the data connection between drone and payload. The physical connection is usually done via a gimbal. The control connection is done via a serial port over which commands can be sent to the payload to set the relevant parameters for the sensors in the payload. The data connection is done via Ethernet/HDMI connection, where the drone controller can request specific frames/info from the sensors that can eventually be sent to a ground controller. This information can also be used to synchronize between payload data and other sensors (e.g., GPS) or other payloads connected to the drone. The images from this sensor are unique in that they can provide hyper/multi-spectral images of up to 40 spectral bands in the range 450-900 nm.

Contributions and Improvements

Currently, there are no real lightweight hyperspectral UAV cameras which have more than 4/5 bands. Such a camera would be a real breakthrough in the domain of UAV precision agriculture. Parrot’s sequoia multispectral camera with about 4-5 spectral bands is the leading state of the art in this domain. However, with 4-5 spectral bands only simple agriculture indices like NDVI can be extracted. Tetra cam’s 3-filter camera or multi-camera systems supporting up to 12 bands are other alternatives. However, multi camera systems lead to much more bulkier systems with additional complexity of software to register images from different cameras to obtain the same spatial field of view, which could potentially lead to loss in image quality. For our target applications more spectral information would be required (>10 bands in VISNIR) to provide accurate diagnostic and actionable information. Our proposed camera can enable such applications. Headwall’s micro-hyperspec is another camera intended for UAV platforms, which uses conventional grating-- based solutions for the spectral unit. This leads to a bulkier camera than our proposed solution, making this unsuitable for lightweight drones. Micro-hyperspecs cameras can weigh up to 1kg or more, making this perhaps more suitable for larger drones/UAVs.

Compared to other multispectral payload systems, this system has a significant improvement in the number of spectral bands, typically from 5-10spectral bands to up to 40 spectral bands. This enables higher precision and accuracies in current inspections and also enables new inspection methods (e.g., inspection of soil quality, which conventionally would have required sending the samples to a lab).


Design and Implementation

The core design of the payload and its integration with the drone system is shown in Figure below.


System architecture of UAV payload with compute enabled system

A first prototype payload has been built by Airobot to be able successfully perform first data collection flights. The pictures below show the integration on the Airobot Mapper drone and of the first test flight.

Prototype payload on Airobot Mapper

As a second prototype, IMEC-BG has implemented a second iteration of the payload. The payload consists of two multispectral sensors in the spectral range 470-900nm, a jetson Tx2 to enable onboard computation and about 1TB of storage to collect spectral data during flight. In addition to the payload integration with a drone as described by Airobot, IMEC has done initial integration (both hardware and software) of this payload with a DJI M600 drone. This integration was done to show the modular and flexible aspect of our payload and data acquisition software blocks. Current software development for this version of the prototype has two parts (1) firmware/acquisition software running on the payload/jetson system and (2) ground controller software running on a tablet which is connected to a drone controller. The key functionality of firmware block is to capture the data from the two multispectral sensors and perform initial pre-processing steps and store them on onboard disk. The key functionality of the ground controller software is to enable user to control camera parameters and to provide a live preview of the images from spectral sensors.

Prototype IMEC payload

Finally, Airobot has been working together with IMEC on working out the detailed design to integrate their payload. The mechanical, electrical and software interface has been defined. To speed up the development a setup has been created so that the software development can be done without needing physical access to the drone.


Architecture of interface with IMEC payload
Retrieved from "https://c4d.lias-lab.fr/index.php?title=WP3-19_1&oldid=885"