Mobile appratus, in particular a rotary-wing drone, provided with a video camera delivering sequences of images dynamically corrected for the wobble effect

Abstract

The apparatus comprises a camera (10) with a digital sensor read by a mechanism of the rolling shutter type delivering video data (S.sub.cam) line by line. An exposure control circuit (22) adjusts dynamically the exposure time (t.sub.exp) as a function of the level of illumination of the scene that is captured. A gyrometer unit (12) delivers a gyrometer signal (S.sub.gyro) representative of the instantaneous variations of attitude (φ, θ, ψ) of the camera, and a processing circuit (18) that receives the video data (S.sub.cam) and the gyrometer signal (S.sub.gyro) delivers as an output video data processed and corrected for artefacts introduced by vibrations specific to the apparatus. An anti-wobble filter (24) dynamically modifies the gain of the gyrometer signal as a function of the exposure time (t.sub.exp), so as to reduce the gain of the filter when the exposure time increases, and vice versa.

Claims

1. A mobile apparatus, in particular a motorized flying device such as a drone, including: a camera comprising a lens, a digital sensor on which is formed an image of a scene during a given exposure time (t.sub.exp), and a mechanism for reading the sensor of the rolling-shutter type delivering as an output. line by line, video data (S.sub.cam); an exposure control circuit, comprising a servomechanism adapted to dynamically adjust the exposure time (t.sub.exp) as a function of the level of illumination of the scene that is captured; a gyrometer unit adapted to deliver a gyrometer signal (S.sub.gyro) representative of the instantaneous variations of the attitude (φ, θ, ψ) of the mobile apparatus; and an image processing circuit, comprising an anti-wobble filter, receiving as an input the video data (S.sub.cam) and the gyrometer signal (S.sub.gyro), and delivering as an output video data processed and corrected for artefacts introduced by vibrations specific to the mobile apparatus, characterized in that the anti-wobble filter is adapted to dynamically modify the gain of the gyrometer signal applied at the input of the image processing circuit as a function of the exposure time (t.sub.exp) determined by the exposure control circuit, so as to reduce the gain of the filter when the exposure time increases, and vice versa.

2. The mobile apparatus of claim 1, wherein the gain variation law of the filter is a monotonic variation law.

3. The mobile apparatus of claim 2, wherein the gain variation law of the filter is a variation law that is bound between a maximum gain of one and a minimum gain of zero.

4. The mobile apparatus of claim 2, wherein the anti-wobble filter (24) is a digital filter adapted to generate, for a given image, a low-pass Butter-worth filter whose order is predefined and whose cut-off frequency (F.sub.c) varies as a function of the exposure time (t.sub.exp) determined for said given image.

5. The mobile apparatus of claim 4, wherein the law of variation of the cut-off frequency (F.sub.c) of the Butterworth filter as a function of the exposure time (t.sub.exp) is a law of the type F.sub.c=1/2.t.sub.exp.

6. The mobile apparatus of claim 4, wherein the digital filter is applied twice to the gyrometer signal, in the forward direction and in the return direction.

7. The mobile apparatus of claim 1, wherein the camera is adapted to deliver as an output the video data for the duration of an acquisition window (A), and wherein means are provided. which are adapted to apply the anti-wobble filter for a given image for the duration of a processing window (P, A, P′) starting before the window of acquisition of this image.

8. The mobile apparatus of claim 7, wherein said means are adapted to apply the anti-wobble filter for the duration of said processing window (P, A, P′) ending after the window of acquisition of the image.

Description

(1) An exemplary embodiment of the invention will now be described, with reference to the appended drawings in which the same references denote identical or functionally similar elements throughout the figures.

(2) FIGS. 1a to 1c illustrate the artefacts of the jelly and wobble type, which can be observed on the image of a checkerboard, before the application of any corrective filtering.

(3) FIGS. 2a to 2d illustrate the images of a high-contrast vertical bar. obtained by application of various distortion correction techniques, with a short exposure time.

(4) FIGS. 3a to 3d illustrate the same respective images as those of FIGS. 2a to 2d, but with a long exposure time, close to the period of the vibrations undergone by the camera.

(5) FIG. 4 illustrates, as a block-diagram, the different elements of correction and filtering of the artefacts of the wobble type, according to the invention.

(6) FIG. 5 illustrates the optimized gain/exposure time characteristic for the characteristic vibration frequency undergone by the camera, as experimentally recorded and approximated by a mathematical function allowing to model this optimum characteristic.

(7) FIG. 6 is a chronogram showing the action of the dynamic filter of the invention, in the case of an image captured with a short exposure time.

(8) FIG. 7 is similar to FIG. 6, for a long exposure time.

(9) In FIG. 1 are illus rated the deformations referred to as jelly and wobble, which can be observed on the image of a checkerboard shown in FIG. 1a.

(10) These artefacts are specific to a sensor of the rolling-shutter type (and not global-shutter type), where the lines constituting the image are not acquired at the same time for all the pixels of the image, but successively line after line (or group of lines after group of lines), as the scanning of the sensor goes along in the direction D.sub.B, perpendicular to the lines C. The movements of the drone and the vibrations occurring during the capture of an image therefore generate elsewhere in the image deformations that won't be the same from one line to the following one.

(11) The jelly effect, visible in FIG. 1b, appears in the presence of movements of the camera of high amplitude but relatively low frequency, for example in the presence of important rotations of the drone: when the latter rotates, between the beginning and the end of the exploration of the sensor by the rolling shutter, the image is displaced by several pixels (in the example of FIG. 1b, by about one square of the checkerboard between the top and the bottom of the image). This phenomenon can be compensated for by adapting at each line l.sub.i of the image a suitable offset as the scanning D.sub.B of the sensor goes along, this line-by-line correction allowing to cancel the jelly artefact introduced by the fast rotation of the drone.

(12) FIG. 1c illustrates the other type of artefact, called “wobble effect”, which is the effect to which the present invention relates.

(13) Unlike the jelly effect, which is a low-frequency and high-amplitude effect caused by the rotations of the drone to move, the wobble effect is mainly caused by the vibrations of the motor, which introduce oscillations of high-frequency (typically of the order of 120 Hz) and low-amplitude.

(14) The wobble effect is partially corrected by a suitable mechanical damping of the camera support allowing to filter the vibrations of the motor, as described for example in the WO 2011/058255 A1 (Parrot SA).

(15) This mechanical filtering is, in practice, not sufficient and the residues of the wobble effect must be eliminated, which may be performed using measurements delivered by the inertial unit of the drone, representative of the instantaneous rotations of the drone caused by the vibrations, and by applying the corrections line by line.

(16) Indeed, the gyrometers of the inertial unit allow to have an indication of the precise attitude of the drone for each of the lines, hence allowing an “intra-image” correction, line by line, so as to readjust the successive lines relative to each other in order to reproduce an image that is the closest possible to the captured scene, i.e. the checkerboard of FIG. 1a in the present example.

(17) The above-mentioned French application 14 56302 of Jul. 2, 2014, describes such a technique for correcting the jelly and wobble effects using the signals delivered by the gyrometers of the drone inertial unit, technique that will be described hereinafter with reference to FIG. 4.

(18) This technique is however not devoid of drawbacks in certain circumstances, as will be now explained with reference to FIGS. 2 and 3.

(19) In FIG. 2, a high-contrast scene with a straight line has been simulated, with: FIG. 2a. the original scene, i.e. the actual representation serving as a reference; FIG. 2b, the image obtained on a rolling-shutter sensor of a camera filming the scene of FIG. 2a, and subjected to an important vibration: the initial straight line appears curled due to the fact that the lines l.sub.i are not acquired simultaneously, but scanned successively in the direction D.sub.B, according to the technique that is characteristic of a rolling-shutter camera; FIG. 2c, the image of FIG. 2b after correction by application of the inverse movement of the vibration, as measured by gyrometers integral with the camera: as can be observed, the undulation has disappeared, to the only cost of a slight thickening of the line, that is visually not hampering for the perception of the image; FIG. 2d, the image of the scene of FIG. 2a that would have been obtained on a global-shutter sensor of a camera filming the scene of FIG. 2a; the lines of the sensor are herein read by blocks, simultaneously, and are hence not subjected to the wobble phenomenon. This type of image is the ideal to which it will be searched to come closer.

(20) The representations of FIG. 2 have been made with a very short exposure time, of the order of 1 ms, corresponding to imaging conditions typical of those which can be met outside by sunny weather.

(21) FIG. 3 illustrates the results that would be obtained with a longer exposure time, for example 8 ms, close to the period of the vibration undergone by the camera (i.e. 8.33 ms for a vibration of 120 Hz).

(22) It can be observed in FIG. 3b that the wobble has been naturally filtered by the camera: the undulation has disappeared and the artefact occurs only as a progressive blurring, which is not visually too hampering. This result is moreover perfectly comparable, as illustrated in FIG. 3d, to what would have been obtained with a global-shutter sensor, which would also show a blurring by moving at the contrast area.

(23) On the other hand, as illustrated in FIG. 3c, the image on which the inverse movement undergone by the camera and measured by the gyrometer has been applied gives a result of bad quality, very far from the ideal constituted by the image of FIG. 3d, which would have been obtained with a global-shutter sensor.

(24) This example shows the limits of the anti-wobble correction by application of an inverse gyrometer signal: of course, the gyrometers measure all the movements undergone by the camera during the scanning of the sensor image but, according to the frequency of these movements and the exposure time, these movements are liable to generate more or less visible deformations, which have for consequence to over-correct certain movements and to generate new artefacts, By applying a correction, it is hence paradoxically obtained an image of less quality (FIG. 3c) than if another correction had been applied (FIG. 3b).

(25) That is the problem that the present invention aims to solve.

(26) FIG. 4 illustrates, as a block-diagram, the different elements of an image processing circuit allowing to correct the wobble effect on the image signal S.sub.cam delivered by a camera 10 comprising a sensor of the rolling-shutter type.

(27) This camera is subjected to angular displacements, of high amplitude and low frequency (displacement of the drone). as well as low amplitude and high frequency (vibration of the motors). These movements undergone by the camera are measured by an inertial unit 12 whose measurements are applied to a circuit 14 for estimating the drone attitude, delivering a gyrometer signal S.sub.gyro representative of the instantaneous rotations of the drone and hence of the camera, the inertial unit 12 and the camera 10 being mechanically integral to each other. These rotations are given by the pitch angle φ, the roll angle θ and the yaw angle ψ, describing the inclination of the drone in the three dimensions with respect to a fixed terrestrial reference system (Euler angle).

(28) The gyrometer signal S.sub.gyro is applied to a circuit 18 for stabilizing and correcting the artefacts of the image signal S.sub.cam, to obtain as an output a corrected and stabilized image signal that can then be transmitted to the user to be visualized on a screen. recorded in a digital memory, etc.

(29) The camera 10 and the inertial unit 12 are piloted by a common clock circuit 16, the respective working frequency of the inertial unit and the camera being sub-multiples of the frequency CLK of this clock 16. In other words, the camera 10 and the inertial unit 12 are configured so that F.sub.gyro=K.F.sub.cam, F.sub.gyro being the acquisition frequency of the gyrometers (typically, F.sub.gyro=990 Hz), F.sub.cam being the frequency of acquisition of the images by the video camera (typically , F.sub.cam=30 Hz), and K being a positive integer (typically, K=33). The fact that K is an integer and that the basis clock is the same for the gyrometers and the camera ensures that there will always be K samples of the gyrometer signal S.sub.gyro per image S.sub.cam, with no drift, the angle measurements always falling at the same instant of time.

(30) However, if this mechanism ensures that the signal S.sub.gyro delivered by the gyrometer sensors and the signal S.sub.cam delivered by the camera are synchronous, it gives no guarantee about the phase concordance of these signals. To guarantee a perfect synchronisation, a hardware component 20 measures the time interval Δ between the gyrometer S.sub.gyro and video S.sub.cam signals with a great accuracy. The common clock 16 and the phase-shift measurement hardware circuit 20 hence allow to link the gyrometer and video signals in time with a very high accurateness, to within a clock cycle. It will be noted that a single measurement is sufficient, because the clocks have been set so as not to drift.

(31) This configuration is that described in the above-mentioned application FR 14 56302 but, as shown with reference to FIGS. 2 and 3, in the case of long exposure time, it does not allow to obtain a satisfying correction of the wobble but, on the contrary, introduces new artefacts by over-correction effect.

(32) For that purpose, the present invention proposes to dynamically modulate the effect of the gyrometer correction, as a function of the exposure time of the image.

(33) The exposure time t.sub.exp is determined, image by image, by an exposure control circuit 22, comprising a servomechanism allowing to dynamically adjust the exposure time according to the level of illumination of the scene captured by the camera 10. As a function of this level of illumination, determined by analysis of the signal S.sub.cam, the circuit 22 pilots the camera so as to vary the exposure time in order to maintain the level of illumination in the best conditions of reproduction of the image by the sensor.

(34) Characteristically, the invention provides to insert between the inertial unit 12 and the digital image correction circuit 18 a filter 24 that is dynamically controlled as a function of the exposure time t.sub.exp determined by the circuit 22. This filtering is an adaptive filtering with, for each image, the generation of a digital filter whose cut-off frequency will vary inversely with the exposure time. so as to all the more attenuate the gyrometer signal—and hence the correction that the latter introduces in the image—that the exposure time is high, so as not to introduce a false correction degrading the result.

(35) It will now be explained, with reference to FIG. 5, the way the transfer function of this filter is defined.

(36) In a first time, the optimum attenuation to be apply to the data S.sub.gyro coming from the gyrometer is determined experimentally, for different exposures, as a function of the frequency of the propellers (in an example of implementation, in fixed point, the propellers rotate at 120 Hz, and that is this value of frequency that will be used to adjust the characteristics of the dynamic filter).

(37) For that purpose, the drone is caused to fly in front of a test chart, the result produced for different values of attenuation of the signal S.sub.gyro is examined, and the amplitude of the undulations of the image is measured: when this amplitude is minimum, it is considered that the optimum gain value A.sub.i has been found.

(38) The different values A.sub.1, A.sub.2 . . . A.sub.7 recorded for increasing exposure times of 3 ms, 4 ms . . . 10 ms give the optimum gain curve shown in FIG. 5.

(39) As can be seen, for exposure times lower than 3 ms, the gain is equal to one, i.e. the totality of the signal S.sub.gyro reflecting the vibration of the propellers is applied to the artefact correction circuit 18, with no filtering. On the other hand, for exposure times equal to or higher than 8 ms, the signal S.sub.gyro is totally filtered, i.e. the gyrometer signal is not taken into account at all to correct the artefacts. Between both, for an average exposure time of 5 ms, the best gain is of about 0.5, which means that the amplitude of the correction signal S.sub.gyro must be attenuated by a half.

(40) The following step consists in modelling the characteristic experimentally recorded, i.e. determining the law of generation of the digital filter that will be applied to each image.

(41) It may advantageously be chosen a filter of the Butterworth type, for the quality of its gain response, which is as constant as possible in the pass-band.

(42) To make up for the phase-shift problems, the filter is applied twice to the gyrometer data S.sub.gyro, first in the forward direction, then in the return direction. More precisely: the gyrometer signal produces a series of samples e.sub.0, e.sub.1 . . . e.sub.n; the filter is applied a first time to this sample, giving filtered values F.sub.0, F.sub.1 . . . F.sub.n; the order of these values is inverted, giving a sequence F.sub.n . . . F.sub.1, F.sub.0; the filter is applied a second time, producing new filtered values F′.sub.4 . . . F′.sub.1, F′.sub.1; and the order of these samples is inverted to give the values F′.sub.0, F′.sub.1 . . . F′.sub.n, which will be finally used by the artefact correction circuit (filtered signal S.sub.gyro).

(43) This double filtering requires to have samples of the raw gyrometer signal S.sub.gyro before and after the capture of the image. The anti-wobble processing will hence be delayed by a few milliseconds, the time to cumulate enough samples of the gyrometer signal in a buffer.

(44) The synthesis of a low-pass Butterworth filter involves two input parameters: its cut-off frequency, beyond which the signals have to be attenuated, and its order, which determines the width of the transition band between the passband and the cut band, i.e. the more or less abrupt character of this transition.

(45) By choosing an order 2 and fixing for the cut-off frequency a law of the type:
F.sub.c=1/2.t.sub.exp,

(46) F.sub.c being the cut-off frequency of the Butterworth filter, and

(47) t.sub.exp being the current value of the exposure time,

(48) the continuous curve of the modelled gain illustrated in FIG. 5 is obtained, which approximates in a perfectly suitable manner the characteristic experimentally recorded.

(49) FIGS. 6 and 7 are chronograms showing the action of the dynamic filter of the invention, in the case of an image captured with a short exposure time and with a long exposure time, respectively.

(50) On these figures. the area A corresponds to the image acquisition window, and the curve “S.sub.gyro (raw)” corresponds to the gyrometer signal before application of the filtering.

(51) To operate the correction according to the invention, it is required to collect an history of the samples of the gyrometer signal on a wider window (anterior samples P and posterior samples P′) than the window A corresponding to the only data of the image.

(52) FIG. 6 illustrates a case where the exposure time is short, with the filter partially applied so as to reduce the effect of the vibrations. By comparing the two curves “S.sub.gyro (raw)” and “S.sub.gyro (filtered)”, it may be observed that the low-frequency undulations have been preserved and the high-frequency ones are partially attenuated, the whole with no phase-shift.

(53) FIG. 7 is similar to FIG. 6, in the case of a very long exposure time: in this case, the high frequency of the gyrometer signal is totally suppressed by the filter. On the other hand, the low-frequency variations are preserved. These slow variations, representative of a jelly effect, will be able to be corrected by a conventional technique of adjustment line by line of the successive lines of the image.