A METHOD FOR CALCULATING INFORMATION RELATIVE TO A RELATIVE SPEED BETWEEN AN OBJECT AND A CAMERA, A CONTROL METHOD FOR A VEHICLE, A COMPUTER PROGRAM, A COMPUTER-READABLE RECORDING MEDIUM, AN OBJECT MOTION ANALYSIS SYSTEM AND A CONTROL SYSTEM

Abstract

A computer-implemented method for calculating information relative to a relative speed between an objectand a camera, based on two images of the object acquired by the camera. The method comprises:determining a value of an optical flowbetween the two images and, altogether with or after the determination of the value the optical flow, determining at least one parameter of the transformation, using the optical flow; andcalculating information relative to a relative speed between the object and the camera, based on said at least one parameter of the transformation.

Claims

1. A computer-implemented method for calculating information relative to a relative speed between an object and a camera, based on an initial image I.sub.k and a final image I.sub.k+1 of the object, derived from image frames outputted by the camera, the method comprising: determining a value of an optical flow (F) between the initial image (I.sub.k) and the final image (I.sub.k+1), the optical flow defining, for pixels (x, y) of the initial image, an estimated motion (u, v) thereof during a transformation transforming the initial image (I.sub.k) into the final image (I.sub.k+1); and, altogether with or after the determination of the value the optical flow, determining at least one parameter (s, tx, ty) of the transformation, using the optical flow; and calculating information (RVD, TtC) relative to a relative speed between the object and the camera, based on said at least one parameter (s, tx, ty) of the transformation.

2. The computer-implemented method of claim 1 further comprising: determining the value of the optical flow between the initial image and the final image; a transformed pixel being a pixel obtained by applying the transformation to a corresponding antecedent pixel; coordinates of a transformed pixel being therefore obtained by adding a value (u, v) of the optical flow F for the corresponding antecedent pixel to coordinates (x, y) of the corresponding antecedent pixel; and solving a set of equations expressing the coordinates (x+u, y+v) of transformed pixels as a function of the coordinates (x, y) of corresponding antecedent pixels, using at least one parameter (s, tx, ty) of the transformation, and thereby determining said at least one parameter (s, tx, ty) of the transformation.

3. The computer-implemented method according to claim 1, wherein a transformed pixel being a pixel obtained by applying the transformation to a corresponding antecedent pixel; coordinates of a transformed pixel being therefore obtained by adding a value (u, v) of the optical flow F for the corresponding antecedent pixel to coordinates (x, y) of the corresponding antecedent pixel; a first equation being an optical flow equation expressing the coordinates (x+u, y+v) of a transformed pixel as a function of the coordinates (x, y) of a corresponding antecedent pixel, using said at least one parameter (s, tx, ty) of the transformation; a second equation being an optical flow equation expressing, for a considered pixel (x, y), a relationship between spatio-temporal information $\left({{\nabla }^{\prime }}_x\text{I,}{{\nabla }^{\prime }}_y\text{I,}{{\nabla }^{″}}_t\text{I}\right)$ about pixel intensities I at the considered pixel, spatio-temporal information $\left({{\nabla }^{\prime }}_x\text{I,}{{\nabla }^{\prime }}_y\text{I,}{{\nabla }^{″}}_t\text{I}\right)$ about pixel intensities at another pixel which is offset from the considered pixel by an estimated offset value (u .sub.i, v.sub.i), and said at least one parameter (s, tx, ty) of the transformation; the method further comprising: determining initial value of said at least one parameter ((s,tx,ty).sub.0) of the transformation and the optical flow (u.sub.0, v.sub.0); and iteratively executing steps of: determining updated values of said at least one parameter ((s,tx,ty).sub.i+1) of the transformation, by solving a set of second equations established for considered pixels of the initial image based on an estimated value (u.sub.i, v.sub.i) of the optical flow (F) for said considered pixels; determining whether a new iteration has to be performed; and determining updated estimated values (u.sub.i+1, v.sub.i+1) of the optical flow (F) for said considered pixels, using a set of first equations for said considered pixels and the values of the at least one parameter of the transformation ((s,tx,ty).sub.i+1).

4. The computer-implemented method of claim 3, wherein said set of second equations, for each of said considered pixels, expresses a relation between spatio-temporal information in the initial image (I.sub.k) at said considered pixel (x), and spatio-temporal information at said another pixel (x + u) in the final image (I.sub.k+1).

5. The computer-implemented method of claim 3, wherein the transformation is a similarity defined by transformation parameters s, tx, and ty; and the set of second equations is defined as: $\text{s}\cdot \left({{\nabla }^{\prime }}_{\text{x}}\text{I}\cdot \text{x}+{{\nabla }^{\prime }}_{\text{y}}\text{I}\cdot \text{y}\right)+\text{tx}\cdot {{\nabla }^{\prime }}_{\text{x}}\text{I}+\text{ty}\cdot {{\nabla }^{\prime }}_{\text{y}}\text{I}+{{\nabla }^{″}}_{\text{t}}\text{I}=0$ in which for a current iteration i, bi-local spatial derivatives ∇.sub.x′I and ∇.sub.y′I and bi-local temporal derivatives ${{\nabla }^{″}}_{\text{t}}\text{I}$ are defined as: ${{\nabla }^{\prime }}_{\text{x}}\text{I}\left(\text{x, y}\right)=\left({\nabla }_{\text{x}}{\text{I}}_{\text{k}}\left(\text{x, y}\right)+{\nabla }_{\text{x}}{\text{I}}_{\text{k+1}}\left({\text{x+u}}_{\text{i}}{\text{, y+v}}_{\text{i}}\right)\right)/2$ ${{\nabla }^{\prime }}_{\text{y}}\text{I}\left(\text{x, y}\right)=\left({\nabla }_{\text{y}}{\text{I}}_{\text{k}}\left(\text{x, y}\right)+{\nabla }_{\text{y}}{\text{I}}_{\text{k+1}}\left({\text{x+u}}_{\text{i}}{\text{, y+v}}_{\text{i}}\right)\right)/2$ $\begin{array}{l}{{\nabla }^{″}}_{\text{t}}{\text{I}}_{\text{k}}\left(\text{x, y}\right)=\\ {\text{I}}_{\text{k+1}}\left(\text{x}+{\text{u}}_{\text{i}}\text{, y}+{\text{v}}_{\text{i}}\right)-{\text{I}}_{\text{k}}\left(\text{x, y}\right){{\nabla }^{\prime }}_{\text{x}}\text{I}\left(\text{x, y}\right)\cdot \text{x}-{{\nabla }^{\prime }}_{\text{y}}\text{I}\left(\text{x, y}\right)\cdot \text{y}\end{array}$ in which u.sub.i, v.sub.i are values of the optical flow F for the current iteration i.

6. The computer-implemented method of claim 1, wherein the transformation is a perspective transformation, a similarity transformation (s) combined with a translation (tx,ty), an affine transformation, or a rotation of the object combined with a translation.

7. A control method for controlling at least one device, the control method comprising steps of: acquiring camera images of an environment of at least one camera with said at least one camera; identifying an object in at least a pair of said camera images acquired by one of said at least one camera; extracting an initial image (I.sub.k) and a final image (I.sub.k+1) of the object, which are portions of said pair of camera images defined by a bounding box of said identified object; calculating information (RVD,TtC) relative to a relative speed between an object and said at least one camera, using a method according to claim 1; controlling said at least one device based on said information relative to the relative speed between the object and said at least one camera.

8. A computer program which is stored on a computer readable storage media, and which is suitable for being performed on a processor, the program including instructions adapted to perform the steps of a method according to claim 1 when it is run on the processor.

9. A computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform the computer program according to claim 8.

10. An object motion analysis system, the object motion analysis system comprising an electronic control unit configured, based on an initial image I.sub.k and a final image I.sub.k+1 of the object, derived from image frames outputted by a camera: to determine a value of an optical flow (F) between the initial image (I.sub.k) and the final image (I.sub.k+1), the optical flow defining, for pixels (x,y) of the initial image, an estimated motion (u,v) thereof during a transformation transforming the initial image (I.sub.k) into the final image (I.sub.k+1); and, altogether with or after the determination of the value the optical flow (u,v), to determine at least one parameter (s, tx, ty) of the transformation, using the optical flow; and to calculate information (RVD, TtC) relative to a relative speed between the object and the camera, based on said at least one parameter (s,tx,ty) of the transformation.

11. The object motion analysis system according to claim 10, wherein the electronic control unit is configured: to determine the value of the optical flow between the initial image and the final image; a transformed pixel being a pixel obtained by applying the transformation to a corresponding antecedent pixel, coordinates of a transformed pixel being therefore obtained by adding a value (u,v) of the optical flow F for the corresponding antecedent pixel to coordinates (x,y) of the corresponding antecedent pixel; and to solve a set of equations expressing the coordinates (x+u,y+v) of transformed pixels as a function of the coordinates (x,y) of corresponding antecedent pixels, using said at least one parameter (s, tx, ty) of the transformation, and to determine thereby said at least one parameter (s,tx,ty) of the transformation.

12. The object motion analysis system according to claim 11, wherein the electronic control unit is configured to solve said set of equations by robust fitting a linear model using a RANSAC method or an IRLS method.

13. The object motion analysis system according to claim 10, wherein a transformed pixel being a pixel obtained by applying the transformation to a corresponding antecedent pixel, coordinates of a transformed pixel being therefore obtained by adding a value (u(x,y), v(x,y)) of the optical flow F for the corresponding antecedent pixel to coordinates (x, y) of the corresponding antecedent pixel; a first equation being an equation expressing the coordinates (x+u,y+v) of a transformed pixel as a function of the coordinates (x, y) of a corresponding antecedent pixel, using said at least one parameter (s, tx, ty) of the transformation; a second equation being an optical flow equation expressing, for a considered pixel (x, y), a relationship between spatio-temporal information $\left({{\nabla }^{\prime }}_x\text{I,}{{\nabla }^{\prime }}_y\text{I,}{{\nabla }^{″}}_t\text{I}\right)$ about pixel intensities at the considered pixel, spatio-temporal information $\left({{\nabla }^{\prime }}_x\text{I,}{{\nabla }^{\prime }}_y\text{I,}{{\nabla }^{″}}_t\text{I}\right)$ about pixel intensities I at another pixel which is offset from the considered pixel by an estimated offset value (u .sub.i, v.sub.i), and said at least one parameter (s,tx,ty) of the transformation; the electronic control unit is configured to execute an initial step of: determining initial values of said at least one parameter ((s,tx,ty).sub.0) of the transformation and the optical flow (u.sub.0, v.sub.0); and the electronic control unit is configured to further iteratively execute steps of: determining updated values of said at least one parameter ((s,tx,ty).sub.i+1) of the transformation, by solving a set of second equations established for considered pixels of the initial image, and based on an estimated value (u.sub.i, v.sub.i) of the optical flow F for said considered pixels; determining whether a new iteration has to be performed; and determining updated estimated values (u.sub.i+1,v.sub.i+1) of the optical flow F for said considered pixels, using a set of first equations for said considered pixels and the values of the at least one parameter of the transformation ((s, tx, ty).sub.i+1).

14. The object motion analysis system according to claim 10, wherein the transformation is a perspective transformation, a similarity transformation (S) combined with a translation (tx, ty), an affine transformation, or a rotation of the object combined with a translation.

15. A control system comprising an object motion analysis system according to claim 10, wherein the control system comprises at least one camera configured to acquire images (I.sub.k,I.sub.k+1); the electronic control unit is configured: to acquire camera images of an environment of said at least one camera with said at least one camera; to identify an object in at least a pair of said camera images acquired by the at least one camera; to extract an initial image (I.sub.k) and a final image (I.sub.k+1) of the object, which are portions of said pair of camera images defined by a bounding box of said identified object; to calculate information (RVD, TtC) relative to a relative speed between an object and the at least one camera, using a method according to claim 1; and to control at least one device of the control system based on said information relative to the relative speed between the object and said at least one camera.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0112] The present disclosure may be better understood and its numerous other objects and advantages will become apparent to those skilled in the art by reference to the accompanying drawing wherein like reference numerals refer to like elements in the several figures and in which:

[0113] FIG. 1 is a schematic drawing showing essentially of a car equipped with an automated driving system according to the present disclosure;

[0114] FIG. 2 is a block diagram showing steps of a method for determining information relative to a relative speed between a moving object and the camera of a vehicle according to a first embodiment of a method according to the present disclosure; and

[0115] FIG. 3 is a block diagram showing steps of a method for determining information relative to a relative speed between a moving object and the camera of a vehicle according to a second embodiment of a method according to the present disclosure.

DESCRIPTION OF AN EXAMPLE EMBODIMENT

[0116] An automated driving system 10 (as an example of a control system according to the present disclosure) configured to implement the proposed method for controlling at least one device, is now going to be described.

[0117] FIG. 1 schematically represents a car 100 equipped with an automated driving system 10 which forms an exemplary embodiment of a control system according to the present disclosure. Car 100 moves on a road (arrow A); in the opposite direction, another car 200 is coming, in the other lane.

[0118] The automated driving system 10 (or, in short, the system 10) comprises an object motion analysis system including an electronic control unit 20, a camera 30, as well as several other not represented sensors.

[0119] The images produced by camera 30 are transmitted to the electronic control unit 20 (ECU 20).

[0120] The ECU 20 has globally the hardware architecture of a computer. The ECU 20 comprises a processor 22, a random access memory (RAM) 24, a read only memory (ROM) 26, an interface 28.

[0121] The hardware elements of ECU 20 are optionally shared with other units of the automated driving system 10 and/or other systems of the car 100.

[0122] The interface 28 includes in particular a tactile display and various displays mounted in or on the dashboard of the car.

[0123] The interface 28 comprises a driver interface with a (not-shown) display to transmit information to the driver of the car 100, and interface connections with actuators and other vehicle devices of the car. In particular, interface 28 comprises a connection with several driving actuators of the car 100, namely, the engine 32, the steering column 34, and the brakes 36.

[0124] A computer program configured to partly assume the driving task by performing lateral and longitudinal control of the vehicle is stored in memory 26. This program is configured to calculate information relative to a relative speed between car 100 and the surrounding objects, detected in the images transmitted by camera 30. This program is further configured to output the commands which, at least during some driving periods, control driving actuators of the host vehicle.

[0125] This program, and the memory 26, are examples respectively of a computer program and a non-transitory computer-readable medium pursuant to the present disclosure.

[0126] The read-only memory 26 of the ECU 20 indeed constitutes a non-transitory computer readable medium according to the present disclosure, readable by the processor 22. It stores instructions which, when executed by a processor, cause the processor 22 to perform the control method according to the present disclosure.

[0127] More specifically, the program stored in memory 26 includes instructions for executing a method for controlling several devices of the vehicle 100 based on information relative to a relative speed between the vehicle and the ‘independently moving objects’ (IMO) moving outside the vehicle. In the present embodiment, the car 200 coming in the opposite direction is taken as an example of an IMO outside vehicle 100. In the present embodiment, the automated driving system 10 constitutes an exemplary object motion analysis system according to the present disclosure.

[0128] To perform its function, system 10 uses the images provided by camera 30, processes these images in ECU 20, and controls the vehicle 100 on the basis of information relative to a relative speed between the vehicle and the ‘independently moving objects’ detected around the vehicle, calculated by ECU 20.

[0129] In accordance with the present disclosure, the vehicle 100 can be controlled during driving pursuant in particular to one of the control methods illustrated by FIGS. 2 and 3, which represent exemplary embodiments of a control method according to the present disclosure.

[0130] In the control method of FIG. 2, the parameter(s) of the transformation transforming an initial image into the final image are calculated in a direct, two-step method. By contrast, in the control method of FIG. 3, this or these parameter(s) are calculated iteratively.

[0131] The steps of the control methods illustrated by FIGS. 2 and 3 will now be described.

Steps S10-S30: Preparation

[0132] In the control methods illustrated by FIGS. 2 and 3, the control method first comprises the following preparation steps: S10) Camera images of the environment of the vehicle are acquired successively by camera 30. For each of these successive acquisitions, the value of a main index k is incremented. In practice, camera 30 continuously monitors the scene in front of car 100, and thus transmits image frame at a rate of 30 frames per second to the electronic control unit 20.

[0133] S20) In each of these images, the electronic control unit identifies the objects which are present. This identification is made using neural network and focuses in identifying in particular pedestrians, other vehicles, etc.

[0134] In the present example, an upcoming car 200 is identified in successive images transmitted by camera 30.

[0135] S30) When an object has been identified in an image, the image of the object is extracted from the camera image. In this purpose, a bounding box is determined for the object in each of these camera images. A sub-image showing the object in the image is then cropped out from the camera image so as to be defined by the bounding box of the object.

[0136] Pursuant to this process, two images (I.sub.k and I.sub.k+.sub.1 in the present example, where k and k + 1 correspond to two successive acquisitions of camera 30) are thus extracted from frames acquired by camera 30, and represent an initial image and a final image of the object with the meaning of the present disclosure.

[0137] Each of these images forms a matrix of pixel intensities I(x, y).

Step S40: Calculation of Information Relative to a Relative Speed

[0138] S40) Then, information relative to a relative speed between the object and the camera is calculated in two steps S41 and S42.

Step S41

[0139] At step S41, the optical flow F between the two images I.sub.k and I.sub.k+.sub.1 is determined; then, once the optical flow F has been determined, the parameter(s) of the transformation transforming the initial image (I.sub.k) into the final image (I.sub.k+.sub.1) are calculated using the optical flow F.

[0140] The optical flow defines, for pixels (x, y) of the initial image (I.sub.k), an estimated motion (u, v) of the pixels between the initial image (I.sub.k) and the final image (I.sub.k+.sub.1). As mentioned before, generally speaking, the optical flow is defined by a function F which provides (u, v) as a function of (x, y):

[00009]$\left(\text{u,v}\right)=\text{F}\left(\text{x,y}\right)$

[0141] In this expression, the couple (x, y) represents the coordinates of a pixel in the initial image. Usually, the optical flow is modelled by making some assumptions to simplify calculations.

[0142] In some embodiments, for instance the embodiments considered here in relation with FIGS. 2 and 3 (other possibilities are of course possible within the scope of the present disclosure), it is assumed that the transformation of the image is a

[00010]$\begin{array}{cc}\left\{\begin{array}{l}x+u=s\cdot x+tx\\ y+v=s\cdot y+ty\end{array}\right)& \text{­­­(1)}\end{array}$

similarity of ratio s combined with a translation (tx, ty). Therefore, the position of a transformed pixel is defined by the following equations:

[0143] More generally, in the methods according to the present disclosure, any kind of function can be used to define the transformation of the image. For instance, the transformation can be defined as a perspective transformation, an affine transformation, a rotation of the rigid object combined with a translation, etc.

[0144] Accordingly, in the present embodiment the optical flow F is expressed by the set of equations (1′) below, which is equivalent to system (1):

[00011]$\begin{array}{cc}\left\{\begin{array}{r}u=\left(s-1\right)\cdot x+tx\\ v=\left(s-1\right)\cdot y+ty\end{array}\right)& \text{­­­(1′)}\end{array}$

[0145] The system (1′) can be established for any pixel (x, y) of the initial image whose optical flow (u, v) can be calculated.

[0146] Step 41 can be implemented in different ways. Two example implementations of step S41 will be presented in more detail below: the ‘direct method’, illustrated by FIG. 2, and the ‘iterative method’, illustrated by FIG. 3.

Step S41 - Direct Method (FIG. 2)

[0147] In this implementation of the control method, step S41 comprises two sub-steps.

[0148] In a first sub-step S411, the optical flow F is determined. Here, the optical flow F can be calculated using any known method.

[0149] Then, in a sub-step S412, the parameter(s) of the transformation are determined.

[0150] For instance, if the transformation is a similarity combined with a translation, defined by parameters s,tx,ty as envisaged above, by combining all systems (1′) established for all the pixels (x, y) of the initial image whose optical flow could be calculated, a large set of equations is obtained. This set of equations comprises only three unknowns, s, tx and ty.

[0151] This set of equations is solved during step S412 by any known method. It can be solved for instance classically by a least squares method; however, in some examples, a robust fitting method is used, for instance a RANSAC method, an IRLS method, etc.

Step S41 - Iterative Method (FIG. 3)

[0152] At step S41 as mentioned before, the parameter(s) (s,tx,ty) of the transformation is or are determined, based on an initial image (I.sub.k) and a final image (I.sub.k+.sub.1).

[0153] In this implementation of step S41, step S41 comprises an initialization sub-step S415, followed by several sub-steps which are executed iteratively. At each iteration, an index i, which is the index of the local loop nested in step S41, is incremented.

[0154] First, in the initialization step S415 (i = 0), initial values are set for the estimated optical flow F.

[0155] In the iterative method, the optical flow F is constrained (or defined) by the parameter(s) of the transformation. The initialization of the flow is therefore carried out by initializing the parameter(s) of the transformation.

[0156] Accordingly, the initialization consists in setting initial values for the parameter(s) for the transformation. In the present exemplary embodiment, the following values can for instance be chosen:

[00012]${\left(\text{s, tx, ty}\right)}_0=\left(1,0,0\right).$

[0157] Consequently, u.sub.0 = v.sub.o = 0 for each pixel in the sub-image (Here, in the variables (s, tx, ty).sub.0, u.sub.0 and v.sub.0, the subscript refers to i, with i=0).

[0158] The initialization of the optical flow is not restricted to u.sub.0 = v.sub.0 = 0. Any reasonable initial values of u.sub.0 and v.sub.0 may be used. For instance, it is possible to initialize their values by executing the above-described direct method.

[0159] Then, in a step S416, updated values (s, tx, ty).sub.i+1 of the parameter(s) of the transformation are calculated.

[0160] In application of the principle of the ‘bi-local algorithm’ presented before, step S416 is based on the following calculations:

[0161] For the pixels of the initial image I.sub.k where they can be calculated, bi-local spatial derivatives are introduced, which are denoted as ∇.sub.x′I and ∇.sub.y′I, for the current iteration i, and are defined as follows:

[00013]$\begin{array}{cc}{{\nabla }^{\prime }}_{\text{X}}\text{I}\left(\text{x,y}\right)=\left({\nabla }_{\text{X}}{\text{I}}_{\text{k}}\left(\text{x,y}\right)+{\nabla }_{\text{X}}{\text{I}}_{\text{k+1}}\left(\text{x}+{\text{u}}_{\text{i}},\text{y}+{\text{v}}_{\text{i}}\right)\right)/2& \text{­­­(2a)}\end{array}$

[00014]$\begin{array}{cc}{{\nabla }^{\prime }}_{\text{y}}\text{I}\left(\text{x,y}\right)=\left({\nabla }_{\text{y}}{\text{I}}_{\text{k}}\left(\text{x,y}\right)+{\nabla }_{\text{y}}{\text{I}}_{\text{k+1}}\left(\text{x}+{\text{u}}_{\text{i}},\text{y}+{\text{v}}_{\text{i}}\right)\right)/2& \text{­­­(2b)}\end{array}$

[0162] In the above equations, the values u.sub.i, v.sub.i are the values of the optical flow F for the current iteration (for the pending value of index i).

[0163] In addition, the following bi-local temporal derivative ∇.sub.t″ is also introduced. It is defined as follows:

[00015]$\begin{array}{cc}\begin{array}{l}{\nabla }_{\text{t}}{}^{\prime \text{}\prime }{\text{I}}_{\text{k}}\left(\text{x,y}\right)=\\ {\text{I}}_{\text{k+1}}\left(\text{x}+{\text{u}}_{\text{i}},\text{y}+{\text{v}}_{\text{i}}\right)-{\text{I}}_{\text{k}}\left(\text{x,y}\right)-{\nabla }_{\text{x}}{}^{\prime }\text{I}\left(\text{x,y}\right)\cdot \text{x}-{\nabla }_{\text{y}}{}^{\prime }\text{I}\left(\text{x,y}\right)\cdot \text{y}\end{array}& \text{­­­(2c)}\end{array}$

[0164] The above bi-local temporal derivative ∇.sub.t″ can also be expressed in simplified form as follows:

[00016]${\nabla }_t{}^{\prime \text{}\prime }{\text{I}}_k={\text{I}}_{k\text{+1}}\left(\text{x}+{\text{u}}_{\text{i}},\text{y}+{\text{v}}_{\text{i}}\right)-{\text{I}}_t\left(\text{x,y}\right)-{\nabla }_{\text{x}}{}^{\prime }\text{I}\cdot \text{x}-{\nabla }_{\text{y}}{}^{\prime }\text{I}\cdot \text{y}$

[0165] The bi-local spatial derivatives ∇.sub.x′I and ∇.sub.y′I, and the bi-local temporal derivative ∇.sub.t″ I,are examples of spatio-temporal information according to the present disclosure.

[0166] On the numerical side, for the implementation, the above spatial gradients ∇.sub.x′I and ∇.sub.y′I are evaluated by a simple [-1, 0, 1] convolution mask. The evaluation of the spatio-temporal information at (x + u.sub.i, y + v.sub.i ) is carried out using bilinear interpolation.

[0167] In order to formulate the bi-local optical flow, the basic (local) optical flow equation is used, based for instance on the Horn-Schunck algorithm, to express the conservation of brightness intensity of a pixel across successive frames. This yields:

[00017]$\begin{array}{cc}{\nabla }_{\text{x}}\text{I}\left(\text{x,y}\right)\cdot \text{u}+{\nabla }_y\text{I}\left(\text{x,y}\right)\cdot \text{v}+{\nabla }_t\text{I}\left(\text{x,y}\right)=0.& \text{­­­(2d1)}\end{array}$

[0168] Of course, other criteria than the conservation of brightness intensity of a pixel across frames can be used to calculate the bi-local formulation (which defines the second equations 2d.sub.1).

[0169] As an example, it is possible to add the possibility of an additive and multiplicative scaling on the intensity change. Furthermore, the Horn and Schunck equation above only models first-order effects, ignoring the higher order terms in the Taylor expansion (equation 2d.sub.2). Also from this perspective it is possible to add second and higher order terms in to the optical flow equation.

[0170] Given an estimate (u.sub.i, v.sub.i) of the optical flow at a considered pixel, the above equation (2d.sub.1) can be reformulated using the current frame I.sub.k and the next frame Ik.sub.+1, the next frame I.sub.k+1 being ‘warped’ with this initial estimate. The bi-local optical flow equation can thus be formulated as follows, where the equation (2d.sub.1) is applied to the residual flow (u - u.sub.i, v - v.sub.i):

[00018]$\begin{array}{cc}\begin{array}{l}{\nabla }_x\text{I}\left(\text{x,y}\right)\cdot \left(\text{u}-{\text{u}}_{\text{i}}\right)+{\nabla }_y\text{I}\left(\text{x,y}\right)\cdot \left(\text{v}-{\text{v}}_{\text{i}}\right)+{\text{I}}_{\text{k+1}}\left(\text{x}+{\text{u}}_{\text{i}},\text{y}+{\text{v}}_{\text{i}}\right)-\\ {\text{I}}_{\text{k}}\left(\text{x,y}\right)=0\end{array}& \text{­­­(2d)}\end{array}$

[0171] The above equation is explained as follows:

[0172] Note: In the description below, optical flow estimates are noted u′, without referring to the iterative procedure, since the mathematical description is generic and does not depend on the iteration.

[0173] Let us consider the vectors x = (x, y), u = (u, v), and an intensity image I that translates over a distance u and is being observed at location x in two consecutive frames k and k + 1 (the analysis is presented here for two frames, but can be applied to more than two frames): [0174] The rule of conservation of the brightness across successive frames imposes that 1.sub.k+1(x) = I.sub.k (x - u). For small displacements the Taylor expansion can be used to approximate this equation as and thus, When omitting the second order effects, equation (2d.sub.2) is equivalent to the 1D equation ∇I.sub.k (x). u + ∇.sub.tI (x) = 0, which is equivalent to the scalar formulation of equation (2d.sub.1) relating the spatial and temporal information at x. The temporal gradient ∇.sub.t expresses the change of intensity between two chosen consecutive frames. [0175] In addition, starting from equation (2d.sub.2), one can also consider k + 1 as reference and express an observed motion - u from frame k + 1 to k, i.e. I.sub.k(x) = I.sub.k+1(x + u). This yields: From (2d.sub.2) and (2d.sub.3), it follows that: ∇I.sub.k = ∇I.sub.k+1, which confirms that under brightness conservation hypothesis, the gradients remain constant. [0176] Given an estimate u′ for the actual motion u, and du = u′- u the residual motion, it follows from equation (2d.sub.2) and (2d.sub.3): Equation (2d.sub.4) formulates a bi-local relationship between the residual motion du, and the spatio-temporal information at I.sub.k+1(x + u) and I.sub.t(x).

[0177] In this formulation, there is the implicit assumption that du is small, but there is no constraint on the magnitude of u′ or u. Equation (2d.sub.5) describes the equivalent forward/backward warping between frames I.sub.k and I.sub.k+.sub.1 with respective motions u′ and du.

[0178] Equation (2d.sub.4) is equivalent to a scalar formulation of equation (2d).

[0179] The above equations can then be simplified by replacing therein the optical flow (u, v) by its expression as a function of (x, y) and the parameter(s) of the transformation. In the proposed embodiment, the transformation is the similarity transformation defined by equations (1), whose parameters are (s, tx, ty).

[0180] Consequently, the bi-local optical flow equation (2d) can be formulated in terms of parameters (s, tx, ty) as:

[00023]$\begin{array}{cc}\text{s}\cdot \left({{\nabla }^{\prime }}_{\text{x}}\text{I}\cdot \text{x}+{{\nabla }^{\prime }}_{\text{y}}\text{I}\cdot \text{y}\right)+\text{tx}\cdot {{\nabla }^{\prime }}_{\text{x}}\text{I}+\text{ty}\cdot {{\nabla }^{\prime }}_{\text{y}}\text{I}+{\nabla }_{\text{t}}{}^{\prime \text{}\prime }\text{I}=0& \text{­­­(2d′)}\end{array}$

[0181] The equations (2d′) constitute second equations with the meaning of the present disclosure.

[0182] These second equations are written for all pixels where they can be calculated. A set of equations is thus obtained, which expresses relationships between spatio-temporal information about the pixel intensities I at considered pixels, spatio-temporal information about the pixel intensities I at another pixel for each of said considered pixels, and the parameter(s) (s, tx, ty) of the transformation.

[0183] More specifically, for each of said considered pixels, these second equations express a relation between spatio-temporal information in the initial image I.sub.k at said considered pixel (x), and spatio-temporal information at said another pixel (x + u) in the final image (I.sub.k+1).

[0184] Said another pixel is offset from the considered pixel by an estimated offset value, which is based on a current value (u.sub.i, v.sub.i) of the optical flow for the considered pixel. In the present embodiment, the estimated offset value is equal to the current estimate (u.sub.i, v.sub.i) of the optical flow.

[0185] The set of equations (2d′) is a linear system (for parameters (s, tx, ty)) and is then solved by any appropriate method, thereby providing a new (or updated) estimate (s, tx, ty)i+.sub.1 for the parameters s, tx and ty of the transformation.

[0186] Then, based on this updated estimate (s, tx, ty).sub.i+1, and using equations (1′), the updated value (u.sub.i+1, v.sub.i+1) of the optical flow is calculated.

[0187] Then, at step S417 th new estimate for the parameters (s, tx, ty).sub.i+1 is compared with the prior estimate (s, tx, ty).sub.i. If the difference exceeds a predetermined threshold, it is concluded that the algorithm has not converged yet.

[0188] The algorithm therefore jumps to step S418. In step S418, updated values (u.sub.i+1, v.sub.i+1) are calculated for the optical flow, based on equations (1′), for each pixel (x, y) of the initial image where these values can be calculated.

[0189] Then, the algorithm continues at step S416, described above.

[0190] Conversely, if at step S417, it is determined that the difference between (s, tx, ty).sub.i+1 and (s, tx, ty).sub.i is below the predetermined threshold, it is concluded that the algorithm has sufficiently converged. It is therefore concluded that the values (s, tx, ty).sub.i+1 are the values of the parameter(s) of the transformation transforming the initial image into the final image.

Step S42

[0191] The algorithm then continues at step S42.

[0192] In step S42, the values of the parameter(s) of the transformation calculated at step S412 are used to calculate information relative to a relative speed between the object and the camera.

[0193] This information can be very simple. For instance, depending on the value of the similarity ratio s, it can be determined whether the detected object is moving away (s < 1) or toward (s > 1) the vehicle. Such information is information relative to the relative speed between the vehicle and the object and is sometimes sufficient to decide to stop tracking the object (if it moves away), or conversely to increase the level of attention paid to the object, if it gets closer.

[0194] Actually, the similarity ratio s is the visual expansion parameter: it characterizes how fast the image of the object grows or conversely shrinks in the images acquired by camera 30.

[0195] As another information relative to a relative speed between the object and the camera which can be possibly extracted from the parameter(s) of the transformation, based on the similarity ratio s and an estimate D (which can be a rough estimate) of the distance from the vehicle 100 to the car 200, the relative velocity in depth RVD of the object relative to the vehicle can be calculated using the following formula:

[00024]$\text{RVD}=\text{D}\cdot \left(1-\text{s}\right)/\left({\text{T}}_{\text{1}}-{\text{T}}_0\right)$

where T.sub.0 and T.sub.1 are the instants at which the initial image I.sub.k and the final image I.sub.k+1 were acquired.

[0196] As another information relative to a relative speed between the object and the camera which can be possibly extracted from the parameter(s) of the transformation, based on the relative velocity in depth RVD, the time to collision with the object (the car 200) can be estimated using formula below:

[00025]$\text{TtC}=\text{D}/\text{RVD}$

[0197] It is further possible to consider the above equations of RVD and TtC over different time steps T.sub.0, ..., T.sub.n, assuming for instance a constant relative velocity over the period. This combination leads to a set of equations:

[00026]$\text{RVD}={\text{D}}_0\cdot \left(1-{\text{s}}_k\right)/\left({\text{T}}_k-{\text{T}}_0\right),k=1,.Math.,n$

in which S.sub.k are the expansion rates at the different time steps T.sub.k, k = 1, ..., n, and are all calculated relative to the initial frame at T.sub.0.

[0198] This joint homogeneous set of equations is then directly solved to calculate TtC: Being given an estimate for D, RVD is calculated as:

[00027]$\text{RVD}=\text{D}/\text{TtC}.$

[0199] This particular formulation stabilizes the numerical output of D and RVD through TtC.

[0200] Moreover, in addition to the relative velocity RVD, the relative acceleration RAC can possibly also be taken into account in the formulation to calculate the time to collision TtC.

[0201] In this case, the velocity is not supposed to be constant; the distance D.sub.k to the object over different time steps T.sub.0, ..., T.sub.n, and given the expansion S.sub.k for each time step, can be expressed by

[00028]${\text{D}}_{\text{k}}={\text{D}}_0-\text{RVD}\left({\text{T}}_k-{\text{T}}_0\right)-\text{RAC}{\left({\text{T}}_k-{\text{T}}_0\right)}^2/2={\text{s}}_k{\text{D}}_0$

[0202] (The above equation can be considered as a Taylor expansion of the kinematic behavior of the car: It is of course possible to include additional term(s)).

[0203] The above homogeneous set of equations can be solved to determine the values of RVD/D and RAC/D. Since there are two unknowns, at least 3 observations (e.g. k=1,2, in addition to 0 used as reference) must be used to arrive at a solution.

[0204] If a proper estimate exists for the distance D.sub.g (e.g. through means of the car width and camera intrinsics), the time to collision TtC can then be calculated based on the values of RVD and RAC.

Final Step S50: Control of at Least One Vehicle Device

[0205] S50) Finally, the electronic control unit 20 outputs commands to control at least one vehicle device (32, 34, 36) of car 100, based on said information relative to the relative speed between the object and the camera, for instance based the visual expansion s, the relative velocity in depth RVD of car 200, and/or the time-to-collision TtC with car 200.

[0206] Although the embodiments of the methods according to the present disclosure have been presented only with respect to the acquisition of two images of an object, derived from camera images acquired by a (single) camera, it should be understood that these methods are usually implemented iteratively, on the basis of a succession of images acquired at successive time steps by one or more cameras.

[0207] Accordingly, in some embodiments, step S41 can be executed several times, based on multiple pairs of initial and final image. Several values of the parameter(s) of the transformation are thus obtained. Then, at step S42, the information relative to a relative speed between the object and the camera can be calculated based on these different values.

[0208] In addition, the above-defined methods can be executed to calculate information relative to a relative speed between an object and a camera based not only on a single camera, but based on a plurality of (two or more) cameras. In this case, all the cameras must be maintained in a fixed position relative to each other. In addition, the cameras must have overlapping fields of views, so that the object appear in the images of the different cameras.

[0209] In this case, step S41 can be executed based on pairs of initial and final images, wherein said pairs of initial and final images are based on camera images of the different cameras. Several values of the parameter(s) of the transformation are thus obtained. Then, at step S42, the information relative to a relative speed between the object and the camera can be calculated based on these different values.