DRONE INCLUDING A FRONT-VIEW CAMERA WITH ATTITUDE-INDEPENDENT CONTROL PARAMETERS, IN PARTICULAR AUTO-EXPOSURE CONTROL

Abstract

The drone comprises a camera (14), an inertial unit (46) measuring the drone angles, and an extractor module (52) delivering image data of a mobile capture area of reduced size dynamically displaced in a direction opposite to that of the angle variations measured by the inertial unit. Compensator means (52) receive as an input the current drone attitude data and acting dynamically on the current value (54) of an imaging parameter such as auto-exposure, white balance or autofocus, calculated as a function of the image data contained in the capture area.

Claims

1. A drone (10) comprising: a camera (14) linked to the drone body, comprising: a hemispheric-field lens of the fisheye type, pointing in a fixed direction with respect to the drone body; and a digital sensor collecting the image (I) formed by the lens and delivering raw image data; an inertial unit (16), adapted to measure the Euler angles (φ, θ, ψ) characterizing the instantaneous attitude of the drone with respect to an absolute terrestrial reference system and delivering as an output current drone attitude data; extractor means (52), adapted to define, in said image (I) formed over the extent of the sensor, the position of a capture area (ZC) of reduced size; control means (48, 50, 52), receiving as an input the current drone attitude data and adapted to dynamically modify the position and the orientation of the capture area (ZC) in said image (I) in a direction opposite to that of the changes of values of the angles measured by the inertial unit; and reprojection means (52), receiving as an input image data of a user area (ZU.sub.B) extracted from the capture area (ZC) and delivering as an output corresponding straightened image data (ZU.sub.R), compensated for the geometric distortions introduced by the fisheye lens, characterized in that the camera (14) further includes: means for the dynamic control of at least one imaging parameter among: auto-exposure, white balance and autofocus, and in that the drone further comprises: analysis means, adapted to define in the capture area (ZC) at least one reduced-definition thumbnail, and to deliver (54) a current value of said imaging parameter as a function of the image data contained in said thumbnail; and compensator means (52), such means receiving as an input the current drone attitude data delivered by the inertial unit and being adapted to dynamically interact with the analysis means, as a function of these current attitude data, in a direction opposite to the variations, liable to be caused by the instantaneous variations of attitude of the drone, of said value of the imaging parameter, delivered by the analysis means, so as to keep to said imaging parameter a value that is substantially independent of the instantaneous variations of attitude of the drone.

2. The drone of claim 1, wherein the analysis means are further adapted to exclude from said image data contained in the thumbnail coming from the capture area (ZC) the raw image data that are located outside (X) the region of the image formed by the lens on the sensor.

3. The drone of claim 1, wherein the compensator means receive as an input (104) the image data comprised in the thumbnail coming from the capture area (ZC), delivered by the extractor means.

4. The drone of claim 3, wherein: the analysis means comprise means adapted to define dynamically in each image a plurality of regions of interest ROIs (ROI.sub.1 . . . ROI.sub.7) distributed over the capture area (ZC) with a corresponding thumbnail for each ROI, and to deliver a current value of said imaging parameter for each respective thumbnail; and the compensator means comprise means adapted to interact dynamically with the analysis means by modification of the size and/or position of the ROIs in the capture area as a function of the current drone attitude data.

5. The drone of claim 4, wherein the compensator means comprise means adapted to previously exclude from the definition of the ROIs those of the ROIs that are located outside the current user area (ZU.sub.B) included in the capture area (ZC).

6. The drone of claim 5, wherein the compensator means comprise means adapted to allocate (106) to each ROI a peculiar weighting value that is function of the more or less great extent of the overlapping of the ROI with the current user area (ZU.sub.B) defined inside the capture area, this value being maximum for the ROI entirely included in the current user area and lower for the overlapping ROIs extending both inside and outside the current user area.

7. The drone of claim 5, wherein the compensator means comprise means adapted to allocate to each ROI a peculiar weighting value that is function of the more or less great surface of the ROI.

8. The drone of claim 3, wherein: the analysis means comprise means adapted to define in each image a grid (GR) of regions of interest ROIs (ROI(i,j)) distributed in a uniform and predetermined manner over the capture area (ZC) with a corresponding thumbnail for each ROI, and to deliver a current value of said imaging parameter for each respective thumbnail; and the compensator means comprise means adapted to interact dynamically with the analysis means by allocating (106) to each ROI a peculiar weighting value that is function of the extent of the overlapping of the ROI with the current user area (ZU.sub.B) defined inside the capture area (ZC), this value being maximum for the ROIs included in the current user area, minimum for the ROIs external to the current user area, and intermediate for the overlapping ROIs extending both inside and outside the current user area.

9. The drone of claim 1, wherein the compensator means receive as a input (206) the straightened image data (ZU.sub.R), compensated for the geometric distortions introduced by the fisheye lens, delivered by the reprojection means.

10. The drone of claim 9, wherein: the analysis means comprise means adapted to define dynamically in each image a plurality of regions of interest ROIs distributed over the straightened image (ZU.sub.R) with a corresponding thumbnail for each ROI, and to deliver a current value of said imaging parameter for each respective thumbnail; and the compensator means comprise means adapted to interact dynamically with the analysis means by modification of the size and/or position of the ROIs in the straightened image (ZU.sub.R) as a function of the current drone attitude data.

Description

[0045] An example of implementation of the present invention will now be described, with reference to the appended drawings in which the same references denote identical of functionally similar elements throughout the figures.

[0046] FIG. 1 is an overall view showing the drone and the associated remote-control device allowing the remote piloting thereof.

[0047] FIGS. 2a and 2b illustrate the modifications of the viewing directions of the camera caused by a forward tilting of the drone, for example during a phase of acceleration.

[0048] FIG. 3 illustrates in (a1)-(a3) the successive steps of windowing and correction of the distortions applied to the image formed on the sensor of the drone camera so as to produce a straightened image, and in (b1)-(b3) the corresponding steps when the drone attitude is modified, with the camera inclined towards the ground.

[0049] FIG. 4 illustrates in (a) and (b) the brightness histograms obtained by analysis of the capture areas (a2) and (b2), respectively, of FIG. 3.

[0050] FIG. 5 illustrates as a block diagram the different modules participating to the implementation of the invention.

[0051] FIG. 6 is a flow diagram explaining the main successive steps of implementation of the invention, in a first and a second embodiment of the latter.

[0052] FIG. 7 illustrates the analysis areas of the capture area according to the first embodiment of the invention, based on regions of interest defined automatically inside this capture area.

[0053] FIG. 8 illustrates the analysis areas of the capture area according to the second embodiment of the invention, based on the making of a grid in this capture area.

[0054] FIG. 9 is a flow diagram explaining the main successive steps of implementation of the invention, in a third embodiment of the latter.

[0055] Examples of implementation of the present invention will now be described.

[0056] In FIG. 1, the reference 10 generally denotes a drone, which is for example a quadricopter such as the model Bebop Drone of Parrot SA, Paris, France. This drone includes four coplanar rotors 12 whose motors are piloted independently from each other by an integrated navigation and attitude control system. It is provided with a front-view camera 14 permitting to obtain an image of the scene towards which the drone is directed.

[0057] The drone also includes a vertical view camera (not shown) pointing downward, adapted to capture successive images of the overflown land and used in particular to evaluate the speed of the drone with respect to the ground. Inertial sensors (accelerometers and gyrometers) permit to measure with a certain accuracy the angular speeds and the attitude angles of the drone, i.e. the Euler angles (pitch φ, roll θ and yaw ψ) describing the inclination of the drone with respect to a horizontal plane in a fixed terrestrial reference system. An ultrasound telemeter arranged under the drone moreover provides a measurement of the altitude with respect to the ground.

[0058] The drone 10 is piloted by a remote-control device 16 provided with a touch screen 18 displaying the image on board the front camera 14, with in superimposition a certain number of symbols allowing the activation of piloting commands by simple contact of a user's finger 20 on the touch screen 18. The device 16 is provided with means for radio link with the drone, for example of the Wi-Fi (IEEE 802.11) local network type, for the bidirectional exchange of data from the drone 10 to the device 16, in particular for the transmission of the image captured by the camera 14, and from the device 16 to the drone 10 for the sending of piloting commands.

[0059] The remote-control device 16 is also provided with inclination sensors permitting to control the drone attitude by imparting to the device corresponding inclinations about the roll and pitch axes, it being understood that the two longitudinal and transverse components of the horizontal speed of the drone 10 will be closely linked to the inclination about the two respective pitch and roll axes. The piloting of the drone consists in making it evolve by: [0060] a) rotation about a pitch axis 22, to make it move forward or rearward; [0061] b) rotation about a roll axis 24, to shift it to the right or to the left; [0062] c) rotation about a yaw axis 26, to make the main axis of the drone pivot to the right or to the left; and [0063] d) translation downward or upward by changing the gas control, so as to reduce or increase, respectively, the altitude of the drone.

[0064] When these piloting commands are applied by the user from the remote-control device 16, the commands a) and b) of pivoting about the pitch 22 and roll 24 axes are obtained by inclinations of the device 16 about its longitudinal axis 28 and its transverse axis 30, respectively: for example, to make the drone move forward, it is just needed to incline the remote-control device 16 forward by tilting it about the axis 28, to move it aside to the right, it is just needed to incline the remote-control device 16 by tilting it to the right about the axis 30, etc. The commands c) and d) themselves result from actions applied by contact of the user's finger 20 on corresponding specific areas of the touch screen 18.

[0065] The drone has also an automatic and autonomous system of hovering flight stabilization, activated in particular as soon as the user removes his finger from the touch screen of the device, or automatically at the end of the take-off phase, or in case of interruption of the radio link between the device and the drone.

[0066] FIG. 2a schematically illustrates, in profile, the attitude of the drone when the latter is stationary, in a lift condition.

[0067] The field covered by a front camera 14 of the conventional type, for example a camera covering a field of 54° and whose sight axis δ is centred to the horizon, is schematized in 36.

[0068] If, as illustrated in FIG. 2b, the drone moves forward with a non-zero horizontal speed, by design, the axis 26 of the drone will be inclined forward by an angle φ (pitch angle) with respect to the vertical V. This forward inclination, schematized by the arrow 38, involves an inclination of same value, schematized by the arrow 40, of the axis δ of the camera with respect to the plane of the horizon HZ. It is hence understood that, over the evolutions of the drone, the accelerations, decelerations thereof, etc., the axis δ oscillates permanently about the direction of the horizon HZ, which will translate in the image into permanent upward and downward oscillation movements.

[0069] Comparably, if the drone moves aside to the right or to the left, this movement will be accompanied by a pivoting about the roll axis 24, which will translate in the image into rotations in one direction or the other of the scene captured by the camera.

[0070] To compensate for these drawbacks, it has been proposed, as explained in the above-mentioned EP 2 933 775 A1 (published on 21 Oct. 2015), to provide the camera with a hemispherical-field lens of the fisheye type covering a field of about 180° as schematized in 42 in FIG. 2a. The image captured by the camera provided with this fisheye lens will of course be subjected to the same oscillation and rotation movements as a conventional camera but only a part of the field captured by this camera will be used by selecting a particular window corresponding to the angular sector 36 captured by a conventional camera, and that will be dynamically displaced in the hemispherical image in the direction opposite to the movements of the drone as determined by the inertial central, so as to annihilate the oscillations that would otherwise be observed in the image.

[0071] Hence, in the case illustrated in FIG. 2b, where the drone dives downward by a pitch angle φ (arrow 38) with respect to the vertical V, the capture window will be displaced upward (arrow 44) by an angle of same value, hence returning back towards the horizon HZ the central axis of the sector 36 of a “virtual camera” covering the field corresponding to the image area.

[0072] The problem of the invention will now be described with reference to FIG. 3.

[0073] FIG. 3 shows in (a1) an example of scene, as collected by the sensor of the video camera provided with the fisheye lens.

[0074] As can be observed, the image I of this scene includes very strong geometric distortions, inherent to the hemispheric or quasi-hemispheric coverage of the fisheye lens, straightened on the planar surface of the sensor. Only a part of this image I produced by the fisheye lens will be used. This part is determined as a function i) of the direction in which the “virtual camera” points, ii) of the field of view of the latter (schematized in 36 in FIGS. 2a and 2b), and iii) of its width/height ratio. It is hence defined a “capture area” ZC containing raw pixel data including the raw “useful area” ZU.sub.B that will correspond to the field of the “virtual camera” after compensation for the geometric distortions introduced by the fisheye lens.

[0075] It will be noted that it is not useful to capture all the pixels of the image I formed on the sensor, but only a fraction of these latter, corresponding to the capture area ZC, for example a window ZC of about 2 Mpixels extracted from an image I of HD quality (1920×1080 pixels), produced by a sensor whose resolution will typically be of 14 Mpixels (4608×3288 pixels). Hence, only the really required pixel data of the capture area ZC are transferred, data that may then be refreshed at a cadence of 30 frames/second with no particular difficulty. A high-resolution sensor can hence be chosen, while keeping a high image flowrate.

[0076] Views (a2) and (a3) of FIG. 3 illustrate the processing operations performed on the pixel data of the capture area ZC to obtain the final image compensated for the geometric distortions: based on the pixel data transferred from the capture area ZC (view (a2)), an algorithm extracts the pixel data of the raw useful area ZU.sub.B and applies thereto a triangular meshing (technique known in itself) then straightens the image by stretching each triangle, to give a straightened useful image ZU.sub.R (view (a3)) with rectified pixel data. The strongly curved horizontal lines of the fisheye image will then be corrected to make them rectilinear and to produce an image corresponding to a natural vision, devoid of geometric distortions.

[0077] Views (b1)-(b3) of FIG. 3 are similar to views (a1)-(a3), in a configuration where the attitude of the drone is modified due to the inclination forward of the latter (rotation about its pitch axis), an inclination that follows for example a switching from a still configuration, in hovering flight, to a configuration of moving forward (the linear speed of the drone being all the more great that its inclination is strong).

[0078] As illustrated in (b1), to compensate for this downward inclination of the drone, the capture area ZC is moved towards the top of the image, hence in the direction opposite to the inclination of the drone. If the relative position of the raw useful area ZU.sub.B remains substantially the same inside the capture area ZC (to allow the following of the scene aimed at), the capture area will on the other hand now include a far more significant part of ground S than of sky C: if then comparing views (a2) and (b2), it is observed that, in the initial configuration (view(a2)), the sky/ground proportion is about 50/50%, whereas in the modified configuration (view(b2)), the sky/ground proportion is about 25/75%. Moreover, is it is strongly displaced upward, the capture area may include areas X that are located outside the region of the circular image formed by the fisheye lens on the sensor.

[0079] On the other hand, the final image ZU.sub.R of the straightened useful area (view (b3)) will be substantially identical to what is was (view (a3)) before the tilting of the drone forwards.

[0080] FIG. 4 illustrates the histograms obtained by analysis of the brightness of the pixels of the thumbnail coming from the capture area ZC, in (a) in the case of view (a2) and in (b) in the case of view (b2), respectively.

[0081] As can be seen in this figure, the tilting of the drone forward is translated into a significant modification of the brightness histogram, with an offset towards the left of the mean value M, due to the increase of the ground/sky ratio in the image of the area ZC.

[0082] The auto-exposure algorithm will interpret this change of the mean value M as a darkening of the image, which will be automatically compensated by an increase of the time of exposure and/or of the camera sensitivity.

[0083] That way, the final images (a3) and (b3) respectively obtained (image of the straightened useful area ZU.sub.R), although they display to the user the same framing of the scene, will differ from each other by their exposure setting, the image of view (b3) being clearer than that of view (a3).

[0084] The present invention has for object to correct this defect.

[0085] FIG. 5 illustrates as a block diagram the different modules participating to the implementation of the invention.

[0086] The front camera 14 of the drone delivers a raw image signal corresponding to the image I. This camera, mechanically linked to the drone body, is subjected to angular displacements that are measured by an inertial unit (IMU) 12 linked to the drone body and hence to the camera. The rotations of the camera are given by the pitch angle φ, the roll angle θ and the yaw angle ψ, describing the inclination of the drone in the three dimensions with respect to a fixed terrestrial reference system (Euler angles). These data are applied to an angle prediction module 48 piloting a module of calculation of the position of the capture area ZC in the image I. A video processing module 52 receives as an input the raw image signal I and performs various operations of windowing as a function of the position of the captured area ZC calculated by the module 50, of image stabilization, of extraction and straightening of the useful area, to deliver as an output to the user a useful image signal ZU.sub.R to be transmitted to the user, and possibly displayed and recorded.

[0087] The module 52 also performs the control (schematized by the return 54) of the camera operation parameters, in particular the control of the auto-exposure (EA), of the white balance AWB and of the automatic focusing AF. The module 52 also ensures the correction, according to the invention, of the above-mentioned defect about the automatic calculation of these camera operation parameters, as will be described hereinafter.

[0088] FIG. 6 is a flow diagram explaining the main successive steps of implementation of the invention, in a first and a second embodiment of the latter. This flow diagram 100 comprises an initial step (block 102) of collection of the raw image I by the camera, followed by a step (block 104) of extraction from this raw image of the capture area ZC as a function of the drone attitude data delivered by the inertial unit IMU.

[0089] The following step (block 106), characteristic of the invention, consists in an analysis of the image data of the capture area, in the manner that will be exposed in details hereinafter with reference to FIGS. 7 and 8, for the delivery of camera control parameters, in particular the auto-exposure parameter.

[0090] The content of the capture area ZC is then subjected to a processing (block 108) of extraction of the raw useful area ZU.sub.B and of reprojection of this raw useful area ZU.sub.B to give the straightened useful area ZU.sub.R, corresponding to the final straightened image delivered to the user.

[0091] With reference to FIGS. 7 and 8, two possible embodiments of the data analysis operated by the block 106 will now be described in more details.

[0092] By the way, it will be noted that this analysis is performed on the basis of the thumbnail coming from the image initially contained in the capture area ZC (downstream from block 104), before the reprojection step (block 108), hence on a deformed version of the image.

[0093] FIG. 7 illustrates the areas of analysis of the capture area according to the first embodiment of the invention, such analysis being operated based on regions of interest ROIs defined automatically inside the thumbnail coming from the capture area.

[0094] In this embodiment, the image analysis device defines (according to techniques known in themselves, that won't be described in more details) a plurality of regions of interest ROIs, which are geometrical selections of areas of reduced size in the image to be analysed, a brightness histogram being established for each of these areas. The auto-exposure algorithm analyses and compares the histograms corresponding to the different ROIs and adjusts accordingly the level of exposure, according to techniques of analysis also known by themselves.

[0095] Characteristically of the invention, the ROIs are distributed in the thumbnail coming from the capture area so as to be located totally or partially inside the raw useful area ZU.sub.B, i.e. if the ROI definition algorithm generates ROIs outside the raw useful area ZU.sub.B, these latter will be excluded from the subsequent analysis for the auto-exposure control. In any case, the pixel data located outside the region of the image formed on the sensor by the lens (regions X of the views (b1) and (b2) of FIG. 3) are excluded from the analysis. In other words, the pixels located outside the image circle are ignored for the calculation of the auto-exposure.

[0096] Moreover, each of the regions of interest ROI.sub.1 . . . ROI.sub.n is allocated with a weighting value taking into account the more or less great extent of the overlapping of the concerned ROI with the raw user useful area ZU.sub.B defined inside the capture area: the weighting will be maximal for the ROIs entirely included in the area ZU.sub.B, null for the ROIs entirely located outside the area ZU.sub.B (which comes to exclude them from the analysis), and intermediate for the ROIs partially included in the area ZU.sub.B (the weighting being all the more high that the proportion of the surface area of the ROI located inside the area ZU.sub.B is high).

[0097] FIG. 8 illustrates a second embodiment, in which the ROIs are no longer defined dynamically and with a variable size, but as a regular grid in the thumbnail coming from the capture area ZC, with a grid GR each elementary square or rectangular ROI(I,j) of which will be affected by a variable weighting as a function of its more or less overlapping character with the area ZU.sub.B, in the same manner as in the first embodiment, exposed in relation with FIG. 7.

[0098] FIG. 9 is a flow diagram explaining the main successive steps of implementation of the invention, in a third embodiment of the latter.

[0099] In this third embodiment, the image data analysis is not performed on the thumbnail coming from the deformed, initial version of the image (capture area ZC and raw useful area ZU.sub.B) with a weighting applied to each region of interest, but on the thumbnail coming from the straightened image, after the reprojection step.

[0100] On the flow diagram 200, the blocks 202 (image collection), 204 (extraction of the capture area ZC) and 206 (extraction and reprojection of the user area) are similar to the respective blocks 102, 104 and 108 of FIG. 6.

[0101] On the other hand, the data analysis step for the control of the camera auto-exposure parameters (block 208) is operated downstream from block 206, i.e. on the straightened version of the image. The auto-exposure then operates conventionally (automatic definition of the ROIs, etc.) without it is required to apply to each ROI a weighting value reflecting the position of this ROI with respect to the raw useful area ZU.sub.B.