METHOD AND DEVICE FOR ASSISTING IN LANDING AN AIRCRAFT UNDER POOR VISIBILITY CONDITIONS

Abstract

Method and device for assisting with landing an aircraft under poor visibility conditions are provided. The method allows sensor data to be received during a phase of approach toward a runway when the runway and/or an approach lighting system are not visible to the pilot from the cockpit; then, in the received sensor data, data of interest characteristic of the runway and/or the approach lighting system to be determined; then, on the basis of the data of interest, the coordinates of a target area to be computed; and, on a head-up display, a guiding symbol representative of the target area to be displayed, the guiding symbol being displayed before the aircraft reaches the decision height, in order to provide the pilot with a visual cue in which to search for the runway and/or approach lighting system.

Claims

1. A method for assisting with landing an aircraft under poor visibility conditions, the method comprising at least the steps of: receiving during a phase of approach toward a runway data generated by a sensor, said runway and/or an approach lighting system not being visible to the pilot from the cockpit; determining, in the received sensor data, data of interest characteristic of said runway and/or of said approach lighting system; computing, on the basis of the data of interest, the coordinates of a target area; and displaying, on a head-up display, a guiding symbol representative of the target area, said guiding symbol being displayed before the aircraft reaches the decision height, in order to provide the pilot with a visual cue in which to search for said runway and/or approach lighting system.

2. The method as claimed in claim 1, wherein the step of receiving sensor data consists in receiving data from a sensor located on-board the aircraft and looking forward, said sensor being chosen from the group consisting of FLIR guidance sensors, a multi-spectral camera, a LIDAR and a millimeter-wave radar.

3. The method as claimed in claim 1, wherein the step of determining data of interest consists in executing an artificial-intelligence algorithm on the received sensor data, the algorithm implementing an artificial-intelligence model trained for image processing, said model being obtained in a training phase by deep learning.

4. The method as claimed in claim 3, wherein the deep learning is based on convolutional neural networks.

5. The method as claimed in claim 1, wherein the step of computing a target area consists in determining, on the basis of the characteristics of the sensor and of the attitude of the aircraft corresponding to the sensor data, heading and elevation coordinates of the target area.

6. The method as claimed in claim 1, in addition comprising, after the step of computing a target area, a step of sending the coordinates of the target area to the head-up display.

7. The method as claimed in claim 5, wherein the coordinates of the target area correspond to two opposite corners of a rectangle framing the data of interest characteristic of said runway and/or said approach lighting system, and wherein the guiding symbol that is displayed is said framing rectangle.

8. The method as claimed in claim 1, wherein the step of head-up display consists in displaying the framing symbol on a fixed head-up display of the cockpit and/or on a head-up display worn by the pilot.

9. A device for assisting with landing an aircraft, especially under poor visibility conditions, said device comprising means for implementing the steps of the method for assisting with landing an aircraft under poor visibility conditions as claimed in claim 1.

10. The device as claimed in claim 9, wherein the data allowing the target area to be computed are generated by a first sensor, the device in addition comprising a second sensor able to deliver an image displayable on the head-up device worn by the pilot, the guiding symbol computed on the basis of the data of the first sensor being displayed in said image delivered by the second sensor.

11. A human-machine interface comprising means for displaying a guiding symbol obtained using the method of claim 1.

12. A system for assisting with landing, especially of SVS, SGVS, EVS, EFVS or CVS type, comprising a device for assisting with landing an aircraft, especially under poor visibility conditions, as claimed in claim 9.

13. An aircraft comprising a device for assisting with landing, especially under poor visibility conditions, as claimed in claim 9.

14. A computer program comprising code instructions for executing the steps of the method for assisting with landing an aircraft, especially under poor visibility conditions, as claimed in claim 1, when said program is executed by a processor.

Description

[0034] Other features, details and advantages of the invention will become apparent on reading the description, which is given with reference to the appended drawings, which are given by way of example and which show, respectively:

[0035] FIG. 1 a known landing symbology for a head-up display of an EVS system;

[0036] FIG. 2 a method for assisting with landing an aircraft, allowing a guiding symbol to be obtained for a head-up display, according to one embodiment of the invention;

[0037] FIG. 3 a head-up display of an EVS system with a guiding symbol obtained using the method of the invention displayed;

[0038] FIG. 4 a symbol according to the invention displayed on an IR image; and

[0039] FIG. 5 a general architecture of a display system allowing the method of the invention to be implemented.

[0040] FIG. 2 illustrates the steps of a method 200 for assisting with landing an aircraft, allowing a guiding symbol to be obtained for a head-up display, according to one embodiment of the invention.

[0041] The method begins with receipt 202 of sensor data generated by a forward-looking sensor located on-board the aircraft. The method of the invention applies to any type of sensor (FLIR guidance sensor delivering an IR image, multi-spectral camera, LIDAR, millimeter-wave radar, etc.).

[0042] The technical problem that the invention solves is that of assisting with detection of the runway of an aircraft by the pilot in a sensor image or with the naked eye before the decision height is dropped below, especially under poor visibility conditions. The invention allows a new guiding symbol generated using a technique for automatically detecting the approach lighting system (ALS) or the runway in data generated by a sensor (sensor data) to be displayed. Advantageously, the displayed symbol is perfectly consistent with the outside world and indicates to the pilot the area in which the runway and/or the approach lighting system will appear, before they can be seen by the pilot with the naked eye, i.e. through direct vision. This allows the pilot to identify where to look for the expected visual references, but also to indicate to him when to look for them, especially if visibility conditions are poor, as, once the system has identified the runway and/or the lighting system and has displayed the symbol, the pilot expects to be able to identify them/it himself visually.

[0043] In a following step (204), after the sensor data have been received, the method allows data of interest that are characteristic of the runway and/or of the approach lighting system to be determined in the received sensor data. The sensor may be an IR or multispectral sensor the image of which is presented to the pilot, or be an active second sensor (which in principle is more effective than an IR sensor), such as a millimeter-wave radar for example, the data of which is not displayable to the pilot because they are difficult to interpret.

[0044] In one embodiment, the data of interest are determined by implementing a conventional algorithm for detecting patterns and straight lines. Patent application FR3 049 744 of the Applicant describes one example of such a conventional detecting algorithm. In one embodiment, the algorithm consists in computing a box encompassing detected elements of interest, said box taking the form of a rectangle the coordinates in pixels of two opposite corners of which correspond to the smallest X and Y coordinates of the pixels belonging to the detected elements, and to the largest X and Y coordinates of the pixels belonging to the detected elements, respectively. The area of the rectangle may be increased by a few percent, for example 10%, while remaining centered on the initial rectangle.

[0045] In one preferred embodiment, the step of determining data of interest consists in executing an artificial-intelligence algorithm on the received sensor data, the algorithm implementing an artificial-intelligence model trained for image processing, said model being trained to detect runways and approach lighting systems. The trained model is a model, hosted on-board the aircraft in operational use, that was obtained in a training phase, and in particular by deep learning using an artificial neural network, for detecting runways and approach lighting systems. In one advantageous embodiment, the artificial neural network is a convolutional neural network (CNN).

[0046] A conventional CNN-based model may be employed to detect and segment runways and approach lighting systems, for example using a Mask RCNN architecture (regions with CNN features)—ResNet 101 (101 layers) [Mask R-CNN—Kaiming et al. 2017]. Transfer learning (then more in-depth learning) may be employed to tailor this model to the use case of runways and approach lighting systems.

[0047] After a training phase, in which the CNN models are trained, it is important to validate the trained model via a phase of testing on data. In order to test the robustness of the model with respect to the variabilities with which it will be confronted in an operational environment (various meteorological conditions, various runways, various approach lighting systems), these data will not have been used in the training phase. A plurality of training and testing iterations may be necessary to obtain a valid and generic CNN model meeting the operational need. The validated model (i.e. its architecture and the learned hyperparameters) may be integrated into a system located on-board an aircraft comprising at least one sensor of the same type as the sensor used for training.

[0048] In the field of computer vision, the objective of deep learning is to model, with a high level of abstraction, data. In brief, there are two phases: a training phase and an inference phase. The training phase allows a trained AI model that meets the operational need to be defined and generated. This model is then used in the operational context, in the inference phase. The training phase is therefore essential. Training is considered to have succeeded if it allows a predictive model to be defined that not only fits the training data well, but that is also capable of correctly predicting data that it did not see during training. If the model does not fit the training data, the model suffers from underfitting. If the model fits the training data too well and is not capable of generalizing, the model suffers from overfitting.

[0049] In order to obtain the best model, the training phase requires a large database that is as representative of the operational context as possible to have been generated and the data of the database to have been labeled with regard to a ground truth (GT).

[0050] The ground truth is a reference image that represents an expected result of a segmenting operation. In the context of the invention, the ground truth of an image contains at least one runway and one approach lighting system, and the visible ground. The result of a segmentation of an image is compared with the reference image or ground truth, in order to evaluate the performance of the classifying algorithm.

[0051] Thus, on the basis of many labeled images, the training phase allows the architecture of the neural network and the associated hyperparameters (the number of layers, the types of layers, the training step size, etc.) to be defined, then, in successive iterations, the best parameters (the weightings of the layers and between the layers), i.e. the parameters that model the various labels (runway/approach lighting) best, to be found. In each iteration of the training, the neural network propagates (extracts/abstracts characteristics specific to the objects of interest) and estimates whether the objects are present and if so their positions. On the basis of this estimate and of the ground truth, the learning algorithm computes a prediction error and backpropagates it through the network in order to update the parameters of the model.

[0052] A training database must contain a very large number of data representing a maximum of possible situations, encompassing, in the context of the invention, various approaches to various runways with various approach lighting systems for various meteorological conditions. In order to implement a deep-learning method and to learn to recognize a runway in the received sensor data, the database that is constructed contains a plurality of labeled or tagged datasets, each labeled dataset corresponding to one (sensor datum/ground truth) pair. In the operational context of the present invention, a ground truth is a description of various elements of interest that have to be recognized in the sensor data, including at least one runway and one approach lighting system.

[0053] Returning to FIG. 2, after the step of determining data of interest in the received sensor data, the method allows an area in which the approach lighting system and/or the runway have been detected to be computed.

[0054] In one embodiment, the computed target area is a rectangle framing the identified elements of interest. On the basis of the characteristics of the sensor and of the attitude of the aircraft corresponding to the sensor data, the method allows the heading and elevation coordinates of two opposite corners of a framing rectangle to be computed.

[0055] After the target area has been computed, the coordinates are sent to a head-up display (or head-down display in a head-down EVS or CVS device), which may or may not be worn, and the area is displayed in a following step (208) as a guiding symbol that is consistent with the outside world, and that takes the form of a framing rectangle. FIG. 3 illustrates a head-up display of an EVS system with a guiding symbol (302) obtained using the method of the invention displayed.

[0056] The display of the symbol allows the path to the runway to be validated before visual acquisition of the latter by the pilot. The display of the symbol thus assists the pilot with acquisition of the mandatory visual references before the DH, since he knows that he must look inside the rectangle.

[0057] In one embodiment, the head-up device may also display an SVS, EVS, or CVS view. In the last two cases (EVS or CVS), the sensor image displayed is the one that fed to the AI search.

[0058] In another embodiment, such as illustrated in FIG. 4, the method allows head-up display of an IR image in which the pilot is able to search for the required visual references with the assistance of a framing symbol (402), a rectangle for example, generated following detection of the runway, by the CNN model or by any other runway- and ALS-detecting algorithm, in data from an active sensor, a millimeter-wave radar for example. This embodiment is particularly advantageous because active sensors have enhanced detection capabilities with respect to IR sensors under poor visibility conditions; however, the data delivered by these sensors are not easily interpretable with the human eye. In this variant of embodiment, the aircraft benefits from the EFVS decrease in landing minima.

[0059] In another embodiment, the on-board sensor is a simple visible camera and its image is not presented to the pilot. Only the guiding symbol generated via the method of the invention is presented to the pilot, this symbol assisting him with visual detection of the runway, for example during VFR flights with reduced visibility (VFR being the acronym of view flight rules). This embodiment does not benefit from any decrease in landing minima.

[0060] FIG. 5 illustrates a general architecture of a display system 500 allowing the method of the invention to be implemented.

[0061] In one preferred implementation, an AI model that has been validated (architecture and the learned hyperparameters validated) is integrated into a system located on-board an aircraft comprising at least one sensor of the same type as the sensor used for training. The on-board system 500 also comprises: a terrain database (BDT) 502; a database (BDEI) 504 of elements of interest; a module (SVS) 506 for generating in 3D a synthetic forward-looking view from the position and the attitude of the aircraft as determined by sensors 508; sensors 510; an analyzing module 512 comprising at least one validated AI model; and a display device 514 for displaying the SVS view to the aircrew of the aircraft. The display device 514, or human-machine interface, may be a head-down display (HDD), a see-through head-up display (HUD), a see-through head-worn display (HWD), or the windshield of the aircraft. Advantageously, the usual guidance symbology showing guidance cues of the aircraft (attitude, heading, speed, altitude, vertical speed, velocity vector, etc.) is superimposed on the synthetic 3D view. The analyzing module 512 may be configured to correct the position of the runway presented on the SVS 506.

[0062] Thus, the present description illustrates one preferred but non-limiting implementation of the invention. A number of examples were provided with a view to allowing a good comprehension of the principles of the invention and a concrete application; however, these examples are in no way exhaustive, and anyone skilled in the art should be able to make modifications thereto and implement variants thereof while remaining faithful to said principles.

[0063] The invention may be implemented by means of hardware and/or software elements. It may be provided in the form of a computer program product, on a computer-readable medium, and comprises code instructions for executing the steps of the methods in their various embodiments.