ARTIFICIAL NEURAL NETWORK LEARNING METHOD AND DEVICE FOR AIRCRAFT LANDING ASSISTANCE

Abstract

A neural network learning method for aircraft landing assistance, the method includes receiving a set of labeled learning data comprising sensor data associated with a ground truth representing at least a landing runway and an approach light bar; running an artificial neural network deep learning algorithm on the learning data set, the deep learning algorithm using a cost function called runway threshold trapezium, parameterized for the recognition of a runway threshold and of approach light bars; and generating a trained artificial intelligence model for landing runway recognition.

Claims

1. A neural network learning method for generating a trained artificial intelligence model, the method comprising: receiving a set of labeled learning data comprising sensor data associated with a ground truth representing at least a landing runway and an approach light bar; running an artificial neural network deep learning algorithm on the learning data set, said deep learning algorithm being based on an iterative computation to optimize a cost function called runway threshold trapezium, parameterized for the recognition of a trapezoidal quadrilateral defined by a landing runway threshold and approach light bars; and generating a trained artificial intelligence model for landing runway recognition.

2. The method according to claim 1, wherein the step of running a deep learning algorithm is implemented on a convolutional neural network.

3. The method according to claim 1, wherein the step of running a deep learning algorithm comprises several iterations of the prediction error computation on the learning data set in order to optimize said cost function.

4. The method according to claim 3, wherein iterations for learning are terminated when the error computation is equal to or below a predefined error threshold.

5. The method according to claim 1, wherein the step of running a deep learning algorithm comprises a step of recognition of a trapezoidal quadrilateral defined by the runway threshold and a wider row of lamps positioned before the runway threshold, notably positioned at 300 metres before the runway threshold.

6. The method according to claim 1, wherein the step of receiving learning data consists in receiving real data or receiving simulated data.

7. A neural network learning device comprising hardware and software means for implementing the steps of the neural network learning method for generating a trained artificial intelligence model, according to claim 1.

8. A use of a trained artificial intelligence model obtained by the method of claim 1 in a landing assistance system in an inference phase.

9. A landing assistance system, notably of SVS, SGVS, EVS, EFVS or CVS type, comprising means for implementing a trained artificial intelligence model generated according to the neural network learning method according to claim 1.

10. The landing assistance system according to claim 9, further comprising means for implementing a trained artificial intelligence model for the recognition of the light bar and of its axis.

11. An aircraft comprising a landing assistance system according to claim 9.

12. A computer program comprising code instructions for executing the steps of the neural network learning method for generating a trained artificial intelligence model, according to claim 1, when said program is run by a processor.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Other features, details and advantages of the invention will emerge on reading the description which is given with reference to the attached drawings that are given by way of example and which represent, respectively:

[0048] FIG. 1 an architecture making it possible to implement the method of the invention;

[0049] FIG. 2 a convolutional neural network-based image processing architecture according to an embodiment of the invention;

[0050] FIG. 3 a method for generating a trained artificial intelligence model for runway and approach light bar recognition according to an embodiment of the invention;

[0051] FIG. 4 a representation of an encompassing trapezoidal box generated by the cost function of the learning algorithm of the invention;

[0052] FIG. 5 a general architecture of a visualization system that allows implementing a trained artificial intelligence model obtained by the learning method of the invention;

[0053] FIG. 6a and FIG. 6b illustrate, on an IR image, the result of image processing according to the method of the invention.

DETAILED DESCRIPTION

[0054] FIG. 1 illustrates an architecture 100 of a system allowing for the processing of sensor images by an artificial neural network algorithm, according to an embodiment of the invention.

[0055] The system generally comprises “image supplier” sources 110, 112 capable of sending sensor data to a learning base 102 coupled to an image processing module 104 configured to implement a deep learning algorithm and generate trained artificial intelligence (AI) models.

[0056] The learning database 102 should contain a very large quantity of data representing a maximum number of possible situations, encompassing different approaches to different runways with different approach light bars for different weather conditions. In order to implement the deep network learning method of the invention and learn to recognize the landing runway in the data, the database is composed of a plurality of labeled or tagged data sets, in which each set of labeled data corresponds to a pair (sensor data, ground truth VT). A ground truth VT, within the meaning of the present invention, is a description of the different elements of interest that have to be recognized in the sensor data. Such elements represent at least a landing runway and an approach light bar.

[0057] The data in the learning base originate from multiple sources 110, 112. They can be real sensor images, whether they are images in the visible range, in the infrared range or radar images. The images are taken by at least one sensor oriented towards the front of an aircraft and capable of supplying information characteristic of a landing runway before the eye of the pilot can see it. This sensor can be an IR camera fixed and positioned in the nose of the aircraft and oriented along the longitudinal axis of the aircraft and usually slightly downwards, supplying a continuous stream of black and white images. The more recent sensors generally combine several specific cameras for different wavelength ranges in the infrared and the visible in order to maximize the capacity of the sensor to detect elements of interest in degraded visibility. Despite these recent advances, this type of sensor does not make it possible to always, in degraded visibility, detect the landing runway before the regulatory minima, typically before the aircraft is below a height of 200 ft above the runway threshold for a category I ILS approach. To mitigate this drawback, the use of active sensors, such as millimetric radars or lidars for example, is being studied. These sensors have a much better capacity to detect elements of interest regardless of the weather conditions. Their drawback is a narrower field, a particular nuisance in cross winds, a lower resolution and a low capacity to generate an image that can readily be interpreted by a pilot. These sensors supply a stream of 3D data (elevation angle, azimuth, distance). The implementation of an automatic landing runway recognition algorithm according to the invention is particularly advantageous with this type of sensor.

[0058] The learning base 102 is enriched with images supplied by a multitude of different sensors equipping a plurality of aircraft performing either real flights, or flights purely for taking images. Each image is associated with parameters of the corresponding flight, and notably the 3D position and 3D orientation parameters of the aircraft at the moment of image capture.

[0059] In addition, aircraft parameters can be added, such as, for example, heading error, height, or even the distance DME with respect to the runway. In the case of an infrared (IR) image, the associated ground truth is an image of the same size with a specific colour for each element of the image for which recognition is desired to be learned.

[0060] Advantageously, simulated sensor data can be added to the learning base in addition to the real data. The simulated data are supplied by an image simulator device 112, capable of generating simulated learning data.

[0061] The image suppliers within the meaning of the present invention are understood to be a set of devices in which each device is capable of supplying one or more sensor images. The image suppliers can be a fleet of aircraft, an aircraft within the meaning of the present invention having a generic definition covering any flying vehicle, whether it is an aeroplane, a helicopter, a drone, a balloon, piloted or unmanned.

[0062] In one embodiment, the learning data are distributed over a decentralized network of distributed ledger or DLT (“Distributed Ledger Technology”) type which is composed of a plurality of computation entities (processors, computers) and in which a ledger is simultaneously stored and synchronized on the entities of the network. The network can evolve through the addition of new information previously validated by the entirety of the network, and the updating of a distributed ledger is reflected over all of the network. Each device or entity of the network permanently has the latest version of the ledger.

[0063] In a particular embodiment, the learning data are distributed over a blockchain, in which each block is linked to the preceding one by a hash key. A block chain is a distributed database secured by cryptographic techniques. Transactions exchanged over the blockchain are grouped together in “blocks” at regular time intervals, securely through cryptography, and form a chain.

[0064] In one embodiment, mechanisms for securely time stamping sensor images can be implemented upon the addition of new learning data in the learning base, each sensor image being associated with corresponding flight parameters, sensor parameters and notably identification, 3D position and 3D orientation parameters of the aircraft at the moment of image capture.

[0065] In one embodiment, the integrity of the images to be added to the learning base can be validated by a consensus validation algorithm. Notably, the verification of the quality of the sensor images which have to be made available in the base can be done via validation rules dependent on the image suppliers, in particular dependent on the quality of each image supplier, the quality covering the quality of the aircraft and of the crew members, for example for procedures. The validation rules also take account of the quality of actors specializing in image processing who can also validate the quality of the photographs and their use for training the image processing algorithms.

[0066] Returning to FIG. 1, the image processing module 104 is configured to implement a deep learning artificial intelligence algorithm on the learning data 102 and generate trained artificial intelligence models (or AI models).

[0067] In one embodiment, the trained AI models can be embedded in aircraft, either sent before a flight or downloaded during a flight, for operational use in landing assistance.

[0068] In one embodiment, the trained AI models can be stored in a database of trained AI models 106.

[0069] The issue addressed by the present invention stems from the fact that the lighting system of the runways of the airports is the essential tool for the safety of the aircraft and their passengers. The lamps which make up the lighting system and their associations allow the pilots to identify the runway in the landing phase, to put the wheels down at the right point, to remain in the axis of the runway and to assess the distance to the end of the runway.

[0070] In conditions of degraded visibility, the approach light bar is generally detectable before the landing runway. However, that is not always the case and sometimes, it is in fact the landing runway which can be detectable first, for example in the cases where the light bar is composed of LEDs instead of incandescent lamps, an IR sensor can then detect the runway before the light bar. That is also the case with a radar which, from far away, can detect the contrast between the ground and the asphalt of the runway well before the contrast between the ground and the metal of the light bar whose reflecting surfaces are too small from far away.

[0071] In order to obtain the earliest possible detection, it is then important to train the neural network to recognize the different types of light bars, and also recognize the landing runway.

[0072] So, the image processing module of the invention 104 implements an artificial neural network deep learning algorithm for runway and light bar detection. In an advantageous embodiment, the algorithm is based on a convolutional neural network CNN.

[0073] In the computer vision domain, the objective of the deep learning is to model the data with a high level of abstraction. Synthetically, there are two phases: a learning phase and an inference phase. The learning phase allows defining and generating a trained AI model which meets the operational need. This model is then used in the operational context in the inference phase. The learning phase is therefore of prime importance. In order to obtain the best model, the learning phase demands the prior collection of a large database that is most representative of the operational context and the data to have been labeled with respect to a ground truth (VT).

[0074] The ground truth is a reference image which represents a result expected after a segmentation operation. In the context of the invention, the ground truth of an image represents at least a runway and an approach light bar and the visible ground. The result of a segmentation of an image is compared with the reference image or ground truth in order to assess the performance of the classification algorithm.

[0075] Learning is considered efficient if it allows defining a predictive model which matches the learning data well but which is also capable of successfully predicting data which have not been seen during learning. If the model does not match the learning data, the model suffers from underlearning. If the model matches the learning data too well and is unable to generalize, the model suffers from overlearning. Thus, from the many labeled images of the learning base 102, the learning phase allows defining the architecture of the neural network and the hyperparameters (the number of layers, the types of layers, the learning pitch, etc.) and to search for the best parameters (the weightings of the layers and between the layers) which best model the different labels (runway/light bar). On each iteration of the learning, the neural network propagates (extracting/abstracting characteristics specific to the objects of interest) and estimates the presence and the position of the objects. From this estimation and from the ground truth, the learning algorithm calculates a prediction error and propagates it backwards in the network in order to update the parameters of the model.

[0076] FIG. 2 illustrates an architecture of an image processing module implementing a convolutional neural network CNN, according to an embodiment of the invention. The person skilled in the art will be able to refer to the existing literature to obtain more details on the known structure of the networks of CNN type.

[0077] A processing step consists in recognising the light bar as a trapezoidal quadrilateral defined by the runway threshold and a wider row of lamps positioned before the runway threshold. Indeed, on the approach light bars, whatever their type, there is a wider row of lamps, at 1000 feet, or approximately 300 metres, before the runway threshold.

[0078] This step is based on the use of a deep neural network which, in the context of the invention, allows detecting a quadrilateral of trapezoidal type, of which the bottom side corresponds to the wider row of lamps 300 metres before the runway threshold and the top side corresponds to the runway threshold.

[0079] In one embodiment, the deep neural network for the light bar recognition step is a convolutional neural network based on an architecture of YOLO type, in particular of YOLO v3 type, described in the document “YOLO v3: An Incremental Improvement—Redmon et al. 2018”.

[0080] FIG. 2 illustrates an architecture of a CNN network of YOLO v3 type suited to the use case of runway and light bar recognition, with its different hyperparameters (numbers of layers with detection layers, convolutional layers, re-sampling layers). In the context of the runways and light bars, the learning is performed with a specific cost function that addresses the new need for aircraft landing assistance, to detect the runway threshold and the row of lamps 300 metres before the threshold in order to know its relative orientation with respect to the aircraft.

[0081] Advantageously, the new cost function is based on the trapezoidal characteristics of a runway threshold and the wider row of lamps before the runway threshold. The neural network seeks to model the parameters of the trapezium, of which the bottom side corresponds to the wider row of lamps before the runway threshold and the top side corresponds to the runway threshold. The detection of the trapezium is performed to different scales (scales 1, 2, 3 in the example illustrated).

[0082] In one embodiment, the learning method can comprise an initial step based on the visibility of the light bar.

[0083] The initial step consists in recognizing the light bar and its axis when the aircraft is far away and it is not possible to detect the different parts of this light bar. This step is based on the use of a second deep neural network which allows segmenting the light bar instance or object. This step is optional, and the learning method of the invention can implement only the runway recognition step previously described.

[0084] In one embodiment, the model of the neural network used for learning light bar recognition uses a “Mask R-CNN—resNet 101” (Regions with CNN features—101 layers) architecture which is described in the document “Mask R-CNN—Kaiming et al. 2017”. In a concrete embodiment using this model, learning by transfer followed by finer learning was performed to adapt to the runway and light bar use case, in which 1900 labeled synthetic images (with runway and light bar seen from different positions and in different day/night conditions) with 30 000 iterations were used to perform the learning.

[0085] FIG. 3 illustrates a method 300 for generating a model trained by deep neural network for runway and approach light bar recognition according to an embodiment of the invention.

[0086] The method begins with the reception 302 of a set of labeled learning data. A set of learning data in the context of the invention consists of a plurality of labeled data, the labeled data corresponding to a plurality of sensor images or radar data in which each image is associated with its ground truth, that is to say a description of the different elements to be recognized in each sensor image. In the case of an image derived from an infrared sensor, the associated ground truth is an image of the same size with a specific colour assigned to each element of the image for which there is a desire for the network to learn to recognize. The ground truth is generated in the phase of construction of the learning database, either manually using a specific tool, or automatically.

[0087] In a next phase 304, the method allows for the execution, on the input data, of a deep learning algorithm based on artificial neural networks. In one embodiment, the deep learning algorithm is based on a convolutional neural network CNN as illustrated in FIG. 2.

[0088] For an algorithm to be able to learn effectively, an efficiency computation function is defined, generally called cost function, whose result decreases all the more as the model being trained predicts values which approximate observations. In a conventional detection network, the detection is defined by a rectangle (width, height, position on x and position on y). This cost function used in the known learning algorithms, such as the neural networks CNN, presents drawbacks in the aeronautical context. Indeed, it does not allow for a detection that is sufficiently accurate to meet the requisite safety requirements in this field. So, a specific cost function was developed by the inventors so as to adapt to the aeronautical use case of landing runway and light bar recognition. This cost function, called “runway threshold trapezium”, is parameterized to make it possible to more effectively detect a landing runway and the approach light bar, in particular in weather conditions resulting in reduced or degraded visibility, where a standard “rectangular box” cost function does not reliably make this possible. The learning phase comprises error computation iterations on the set of learning data in order to optimize the cost function and converge towards a low error and a high accuracy of the trained model.

[0089] FIG. 4 illustrates a representation of a runway threshold trapezium 402 generated by the cost function of the invention during the deep learning process. The runway threshold trapezium (and therefore the cost function) is more complex than a rectangular box because it has to take account of the position and the size of the large base of the trapezium (xB, yB, wB) representing the runway threshold, the position and the size of the small base of the trapezium (xb, yb, wb) representing the wider row of lamps before the threshold (300 metres), the space between these bases (h) and the angle (θ) between the large base and the horizontal axis. These different variables are taken into account in the new cost function to assess the difference between the prediction/detection by the neural network and the ground truth.

[0090] In the inference phase, this specific trapezium form is identified by the embedded AI engine. The coordinates in pixels of the corners of the trapezium which is identified are transformed into 3D coordinates to make it possible to calculate the relative position of the aircraft with respect to the threshold and to the runway axis in terms of distance, of height and of lateral drift.

[0091] Returning to FIG. 3, after having finished the iterations necessary to the learning, that is to say after having optimized the cost function and obtained an error calculation that is equal to or below a predefined error threshold, the method allows generating, in a subsequent step 306, a model trained for runway recognition.

[0092] Advantageously, the trained model can be used in an embedded landing assistance system.

[0093] FIG. 5 illustrates a general architecture of a visualization system 500 making it possible to implement, in inference phase, a trained model obtained according to the neural network learning method of the invention.

[0094] In a first implementation, a validated AI model (architecture and the hyperparameters learned) can be incorporated in a system embedded on board an aircraft comprising at least one sensor of the same type as that used for the learning. The embedded system 500 comprises a terrain database (BDT) 502, a database of elements of interest (BDEI) 504, a module for generating a synthetic view 506 in 3D towards the front of the aircraft (SVS) from the position and the attitude of the aircraft received by sensors 508, and sensors 510, an analysis module 512 comprising at least one AI module generated according to the method of the invention in a deep learning network learning phase and an SVS display device 514 for the crew of the aircraft. This display device 514 can be a head-down display (HDD), a head-up transparent screen (HUD), a head-worn transparent screen (HWD), the windscreen of the aircraft. Advantageously, the usual piloting symbology showing the pilot parameters of the aircraft (attitude, heading, speed, altitude, vertical speed, speed vector, etc.) is superimposed on the 3D synthetic view. The analysis module 512 can be configured to correct the position of the landing runway shown on the SVS 506. In one embodiment, in which the embedded AI model comprises a model relating to the recognition of the runway axis (obtained in learning phase according to the initial step described previously), in a first stage, the coordinates in pixels of the segment (the axis) detected are sent to the analysis module which calculates, from these coordinates, parameters of the aircraft (attitude and position), of the position of the runway derived from the database and parameters of the sensor, the heading error and the position error perpendicularly to the landing runway.

[0095] In a second stage, from the coordinates in pixels of the trapezium which is detected, the analysis module calculates, as previously, the heading error and the position error perpendicularly to the landing runway, but this time, it also calculates the altitude error and the position error longitudinally to the landing runway. All these calculations are performed by comparison with the position data of the landing runway in the terrain database 502, the position and the attitude received by the sensors of the aircraft 508, and the parameters of the sensor 510 (horizontal field, vertical field, orientation, etc.). These calculated errors are then used to improve the accuracy of the input data of the 3D synthetic view generation module which thus shows a view corrected using the detection by the AI model of the landing runway in sensor 510 data. Such a system is called CVS (Combined Vision System) and allows continuing under landing minima if the runway is detected by a sensor.

[0096] In another implementation, the embedded system comprises an automatic piloting (PA) device which receives a position and an attitude of the aircraft whose accuracy has been improved as described in the preceding paragraph by the AI model, thus allowing for a lowering of the decision minima and/or an automatic landing of the aircraft. In this implementation, the synthetic view generation module SVS is optional.

[0097] FIGS. 6a and 6b illustrate, on an IR image, the result of an image processing using a trained AI model obtained according to the method of the invention with the version with two recognition steps. In a first stage, a fairly distant view allows the system, using the longitudinal axis of the runway 602, to adjust, if necessary, the SVS in terms of heading and laterally to the landing runway then, on approach, in a second stage, the runway threshold trapezium 604 is detected, allowing the system to identify the runway threshold and the approach light bars, and, if necessary, adjust the SVS, still in terms of heading and laterally to the landing runway, but also in terms of altitude and longitudinally to the landing runway.

[0098] Thus, the present description illustrates an implementation of the invention that is preferential, but which is not limiting. Examples are chosen to allow for a good understanding of the principles of the invention and a concrete application, but are in no way exhaustive and should allow the person skilled in the art to make modifications and add variant implementations while retaining the same principles.

[0099] The invention can be implemented from hardware and/or software elements. It can be available as computer program product on a computer-readable medium and comprises code instructions for executing the steps of the methods in their different embodiments.