METHOD AND DEVICE FOR STABILIZATION OF A SURROUND VIEW IMAGE

Abstract

A method and a device for image stabilization of an image sequence of one or more cameras of a vehicle are disclosed. Images of the surroundings of the vehicle are recorded and image points of the recorded images are projected to a unit sphere, where the projection includes applying a lens distortion correction. For each of the one or more cameras, a vanishing point is calculated using the projected image points on the unit sphere and a motion of the vanishing point on the unit sphere is tracked. The motion of the vanishing point is used to calculate a corrected projection to a ground plane.

Claims

1. A method for image stabilization of an image sequence of one or more cameras of a vehicle, the method comprising: recording images of surroundings of the vehicle from one or more cameras of the vehicle; projecting image points of the recorded images to a unit sphere, wherein the projection comprises applying a lens distortion correction; calculating, for each of the one or more cameras, at least one vanishing point from the projected image points on the unit sphere; tracking a motion of the at least one vanishing point; and using the motion of the at least one vanishing point to calculate a corrected projection to a ground plane.

2. The method of claim 1, further comprising: deriving, for each of the one or more cameras, a position deviation of the position of an image plane from the motion of the vanishing point and filtering the position deviation over time.

3. The method of claim 1, further comprising: detecting edges in the recorded images and projecting image points of the detected edges onto the unit sphere.

4. The method of claim 1, wherein the distortion correction comprises applying a lens distortion correction that is based on a polynomial fish-eye model.

5. The method of claim 1, wherein the recalculation of the corrected projection to a ground plane comprises deriving a scaling of the recorded images using the filtered position.

6. The method of claim 1, further comprising: applying a lens distortion correction to obtain a corrected image; detecting at least two corresponding straight lines in a corrected image; deriving a position of a vanishing point of the at least two straight lines on a unit sphere; tracking a motion of the vanishing point on the unit sphere; deriving a change in orientation of the one or more cameras from the tracked motion of the vanishing point; and deriving a scaling of the recorded images using the derived change in orientation.

7. The method of claim 5, wherein the image scaling comprises: deriving a true distance between a camera of the surround view system and a ground surface; and computing a scaling factor for an image pixel based on the derived true distance between the camera and the ground surface.

8. The method of claim 6, wherein the vanishing point of the detected straight lines is derived by projecting the detected lines on a unit sphere and determining the intersection of corresponding great circles.

9. The method of claim 1, wherein the tracking of the vanishing point comprises a tracking with a prediction filter.

10. The method of claim 1, wherein input values of the prediction further comprise an input from a motion sensor of the vehicle.

11. A computer program product for executing a method for image stabilization of an image sequence of one or more cameras of a vehicle, the method comprising: recording images of surroundings of the vehicle from one or more cameras of the vehicle; projecting image points of the recorded images to a unit sphere, wherein the projection comprises applying a lens distortion correction; calculating, for each of the one or more cameras, at least one vanishing point from the projected image points on the unit sphere; tracking a motion of the at least one vanishing point; and using the motion of the at least one vanishing point to calculate a corrected projection to a ground plane.

12. An image correction system for a motor vehicle comprising a computation unit with an input connection for receiving consecutive camera images from one or more cameras, the computation unit comprising an input connection for receiving images of surroundings of a vehicle from one or more cameras of the vehicle and a processing unit that is operative to: project image points of the recorded images to a unit sphere, wherein the projection comprises applying a lens distortion correction; calculate, for each of the one or more cameras, vanishing points from the projected image points on the unit sphere; track a motion of the vanishing points; calculate, for each of the one or more cameras, an error in a position of an image plane and filtering the position over time; and using the filtered position to calculate a corrected projection to a ground plane.

13. The image correction system of claim 12, wherein the processing unit is further operative to: apply a lens distortion correction to obtain a corrected image; detect at least two corresponding straight lines in the corrected image; derive a position of a vanishing point of the at least two corresponding straight lines on a unit sphere; track a motion of the vanishing point on the unit sphere; derive a change in orientation of the one or more cameras from the tracked motion of the vanishing point; and and to derive a scaling of the recorded images using the derived change in orientation.

14. A motor vehicle with an image correction system of claim 12, wherein one or more cameras of the motor vehicle are connected to the computation unit of the image correction system.

Description

DESCRIPTION OF DRAWINGS

[0052] The application is now explained in further detail with respect to the following Figures in which:

[0053] FIG. 1 shows a vehicle with a surround view system,

[0054] FIG. 2 illustrates a projection to a ground plane of an image point recorded with the surround view system of FIG. 1,

[0055] FIG. 3 illustrates in further detail the ground plane projection of FIG. 2,

[0056] FIG. 4 illustrates in further detail the ground plane projection of FIG. 3,

[0057] FIG. 5 shows a determination of a vanishing point,

[0058] FIG. 6 shows a determination of a vanishing point in further detail,

[0059] FIG. 7 shows a motion of the vanishing point in spherical coordinates,

[0060] FIG. 8 shows a procedure for deriving an image point of a corrected image, and

[0061] FIG. 9 shows a procedure for an image stabilization according to the present disclosure.

[0062] Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

[0063] FIG. 1 shows a vehicle 10 with a surround view system 11. The surround view system 11 comprises a front view camera 12, a right side view camera 13, a left side view camera 14 and a rear view camera 15. The cameras 11-14 are connected to a CPU of a controller, which is not shown in FIG. 1. The controller is connected to further sensors and units, such as a velocity sensor, a steering angle sensor, a GPS unit, and acceleration and orientation sensors.

[0064] FIGS. 2, 3 and 4 show a projection to a ground plane 16. FIG. 2 shows a projection of an image point to a ground plane 16. An angle of inclination θ relative to the vertical may be estimated from a location of the image point on the image sensor of the right side view camera 13. If the image point corresponds to a feature of the road, the location of the corresponding object point is the projection of the image point onto the ground plane.

[0065] In the example of FIG. 2, the camera 13 has an elevation H above the ground plane. Consequently, the correspond object point is located at a distance H*cos(θ) from the right side of the vehicle 10. If an image point corresponds an object point on the ground plane, a projection of the image point to the ground plane represents the real position of an object point in the surroundings. An angle θ of incidence is derived from a location of the image point on the camera sensor. A location Y of the projection is then derived using the height H of the camera sensor above street level as Y=H*cos(θ).

[0066] FIG. 3 shows an isometric view of the affine projection of FIG. 2. In FIG. 4, a point in a view port plane 17 is denoted by p=(u, v) and a corresponding point in the ground plane 16 is denoted by P=(X, Y). A distance between the view port plane 17 and a projection center C is denoted by the letter “f”.

[0067] FIG. 4 shows an enlarged section of FIG. 3 in which a central image point p0 is denoted by the coordinates (u0, v0) an image point p is denoted by coordinates (u, v) and a projected image point P on the ground plane is denoted by coordinates (Xc, Yc, Zc).

[0068] In a further aspect, FIG. 4 may be seen as a projection of the world point (Xc, Yc, Zc) to an image plane. The 3D point (Xc, Yc, Zc) with the word coordinates Xc, Yc, Zc can be mapped to an image point according to the following equations:

[00002]$\begin{array}{cc}u=f.Math.\frac{Y_c}{Z_c}+u_0& \left(1\right)\\ v=f.Math.\frac{X_c}{Z_c}+v_0& \left(2\right)\end{array}$

[0069] A camera ray is defined as a normal vector. Thus, a mapping to a camera ray corresponds to a mapping to a point on the surface of a unit sphere. It can be computed by normalizing the vector (u, v, 1).

[0070] FIG. 5 shows a determination of a vanishing point. A virtual image plane 17, in which line distortions of a camera sensor image have been compensated by a distortion correction, includes a first vertical line 18 and a second vertical line 19.

[0071] When the lines 18, 19 are projected onto a unit sphere 23 by using a central projection for which the center of projection is at the center of the unit sphere 23, the projections of the parallel lines 18, 19 form segments of respective great circles 20, 21.

[0072] According to the present disclosure, a Jacobian Matrix, which is used in an extended Kalman filter, such as the state matrix and the prediction matrix, may be derived from a normalized projection camera model. The Jacobian matrices may be easier to calculate by using a constraint on the radius.

[0073] The intersections of the great circles 20, 21 are the vanishing points of the parallel lines 18, 19. In FIG. 5 one intersection 22 of two intersections of the great circles 20, 21 is shown and a direction 24 of a corresponding vanishing line is indicated.

[0074] In some implementations, the vanishing points on the unit sphere 23 are tracked over time using a Kalman filter. In some examples, a distance between vanishing points corresponds to a length of a great circle segment on the unit sphere between the vanishing points and a statistical error is computed based on the distance.

[0075] The derived motion of the vanishing points corresponds to a change of camera orientation. In some implementations, a change in camera orientation is compensated using the derived change of camera orientation. In some examples, small amplitude pitch and roll movements, which, among others, are caused by the vehicle's suspension are tracked and compensated for. Yaw movements may be tracked, which are caused, among others, by the steering and a sliding motion of the wheels on a street surface.

[0076] In some implementations, the parallel lines of one vanishing point correspond to lateral boundaries of the street. The parallel lines may correspond to demarcations on the street.

[0077] FIG. 6 shows a further example of a transformation of line segments to segments of great circles and the determination of a vanishing point. The image planes 17 of FIGS. 5 and 6 represent a virtual image plane in computer memory in which lens distortions, which are present in the original sensor data of the camera sensors, have been reduced or removed by applying a distortion correction transformation.

[0078] A first line segment 25, which is the image of a first street border, is mapped to a segment 26 of a first great circle 27. A second line segment 28, which is the image of a second street border, is mapped to a segment 29 of a second great circle 30. The first and second great circles 27, 30 intersect at a vanishing point 31. This intersection point may be determined by computing an intersection of the great circles 27, 30 or also by extending the lines 25, 28, computing their intersection point 32 and mapping the intersection point 32 to the vanishing point 31.

[0079] In general, it is not known whether two given straight lines in a camera image correspond to parallel object lines and have a common vanishing point. Therefore, the computation of a vanishing point from two arbitrary line segments may also yield a “false vanishing point”. False vanishing points may be avoided or eliminated in various ways. For example, the image information can be analyzed and the line segments be identified as street border. Furthermore, in some examples, a stored digital map may is used to determine that the street borders are straight and are suitable for the determination of a vanishing point.

[0080] Further methods to avoid false vanishing points may include among others, the use of three-dimensional information from distance measurements or from stereoscopic images, inferences about an object's orientation based on a computational analysis of the camera image, and the computational determination of vanishing point clusters on the unit sphere.

[0081] In some implementations, an artificial neural network is trained to detect parallel object lines, such as lines corresponding to street borders, street markings, guardrails, guide posts, lamp posts, horizontal and vertical features of buildings or other straight object features.

[0082] FIG. 7 shows spherical coordinates of a locus of a derived vanishing point over time. The spherical coordinates are chosen such that Φ corresponds to a yaw angle and θ corresponds to an elevation angle. FIG. 7 illustrates a situation in which a camera, which is fixed to the vehicle, oscillates around an equilibrium elevation angle and turns together with the vehicle by a yaw angle of about 90 degrees.

[0083] Yaw, pitch and roll angles, which are also known as “Tait-Bryan angles” may be obtained from the angles of the spherical coordinates θ, Φ on the unit sphere by the transformation

[00003]$\begin{array}{cc}\alpha ={\tan }^{-1}\left(\frac{z}{y}\right)={\tan }^{-1}\left(\frac{\sin \left(\theta \right)}{\cos \left(\theta \right).Math.\sin \left(\varphi \right)}\right)& \left(3\right)\\ \beta ={\tan }^{-1}\left(\frac{z}{x}\right)={\tan }^{-1}\left(\frac{\sin .Math..Math.\left(\theta \right)}{\cos \left(\theta \right).Math.\cos \left(\varphi \right)}\right)& \left(4\right)\\ \gamma =\varphi & \left(5\right)\end{array}$

if Φ is measured in the x, y plane and Φ=0 corresponds to (x, y)=(1, 0).

[0084] FIG. 8 illustrates a computational model for deriving an image point x″ from an object point or, conversely for deriving an object point from an image point.

[0085] In a first transformation step 30, an object point X is projected onto a view port plane. In some examples, the transformation of step 30 is modelled as a rotation R and a translation C of the original object point X onto the view port plane.

[0086] In a second transformation step 31, a transformation h distorts the image according to properties of an imaging lens. The reverse transformation h−1 according to step 34 is referred to as distortion correction transformation. The image correction according to step 34 reduces or removes lens distortions according to a pre-defined mathematical model of the lens distortions.

[0087] In some implementations, in which a fish-eye objective is modeled, the distortion correction transformation includes a transformation according to a polynomial fish-eye model such as, among others, an odd order polynomial model, a PFET (polynomial fish-eye transformation) model, or a division model. Other lens models for removing barrel-type distortions include, among others, the Kannala-Brandt model, and lens models according to Grossberg and Nayar and according to Gennery.

[0088] Furthermore, the distortion correction transformation may include a Hough transform. According to the Hough transform every point in an x-y plane is associated with a parameterized curve r(t)=x*cos(t)+y*sin(t). The parameters of the Hough transform, such as origin and orientation of the coordinate system, are determined by minimizing a distortion measure and/or by detecting lines in the picture and matching the lines to the parameterized curves. The distortion measure is related to a curvature of the image lines.

[0089] In some implementations, lines are detected by processing the image using an edge detector and detecting lines in the processed image. The processed image is then transformed into Hough space, which is spanned by the distance r and the angle t. A high intensity in Hough space corresponds to a well detected line.

[0090] In a third transformation step 32, a projection ∂ is applied which projects the point x′ onto a corresponding image pixel x″. The projection ∂ of step 32 may include irreversible components, such as an averaging step, which models the mapping of image points to a pixel, and reversible components. The inverse of the reversible portion of the transformation ∂ according to step 33 is referred to as reverse projection ∂.sup.−1.

[0091] The arrows 33, 34 indicate the respective reverse transformation of the transformations 32 and 31. In a given realization, which may involve information loss such as image clipping and numerical errors, the transformations 31 and 32 are only approximately reversible.

[0092] By contrast, the projection according to step 30 is in general not reversible and the reconstruction of an object point X requires further assumptions, such as the recognition of an object and assumption about its orientation, and/or further measurements, such as distance measurements.

[0093] In order to remove distortions from an image of image pixels x″, the reverse projection is applied in step 33 and the distortion correction is applied in step 34. The resulting two dimensional image is then used to determine straight lines and their corresponding vanishing points, as shown in FIG. 5.

[0094] In some implementations, the function h, which models the lens, is used to extend a perspective projection according to a pinhole camera model. In projective or homogeneous coordinates, the pinhole camera model may be written as:

[00004]$\begin{array}{cc}{\mathrm{sm}}^{\prime }=A\left[RC\right].Math.M^{\prime },\mathrm{or}& \left(6\right)\\ s\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& c_1\\ r_{21}& r_{22}& r_{23}& c_2\\ r_{31}& r_{32}& r_{33}& c_3\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]& \left(7\right)\end{array}$

[0095] Herein, [RIC] stands for a block matrix that is composed of the rotation matrix R and the translation vector C, (X, Y, Z) are the coordinates of a 3D point in the world coordinate space, (u, v) are the coordinates of the projection point in pixel units, A is a camera matrix, or a matrix of intrinsic parameters, (cx, cy) is a principal point that is usually at the image center, and fx, fy are the focal lengths expressed in pixel units.

[0096] The matrix A provides an example of the abovementioned instric transformation ∂. In general, the function ∂ is a generic intrinsic transformation ∂=∂ (fx, 0, cx; 0, fy, cy).

[0097] For z≠0, the above transformation can be written in three dimensions as:

[00005]$\begin{array}{cc}\left[\begin{array}{c}x\\ y\\ z\end{array}\right]=R\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]+t& \left(8\right)\\ x^{\prime }=\frac{x}{z},y^{\prime }=\frac{y}{z}& \left(9\right)\\ u=f_x*x^{\prime }+c_x,v=f_y*y^{\prime }+c_y.& \left(10\right)\end{array}$

[0098] The lens distortion function h may be integrated in this framework by inserting a transformation (x″,y″)=h(x′,y′) and computing u and v as a function of x″ and y″ instead.

[0099] An example of a function h is given by:

[00006]$\begin{array}{cc}\left[\begin{array}{c}{x}^{″}\\ y^{″}\end{array}\right]=\left[\begin{array}{c}{h}_1\left(x^{\prime }\right)\\ h_2\left(y^{\prime }\right)\end{array}\right]=\left[\begin{array}{c}\frac{{\theta }_d}{r}*x^{\prime }\\ \frac{{\theta }_d}{r}*y^{\prime }\end{array}\right]& \left(11\right)\\ \mathrm{where}& \\ {\theta }_d=\theta \left(1+k_1.Math.{\theta }^2+k_2.Math.{\theta }^4+k_3.Math.{\theta }^6+k_4.Math.{\theta }^8\right),& \left(12.Math.a\right)\\ \theta =\mathrm{atan}\left(r\right),r^2=a^2+b^2,a=\frac{x}{z},b=\frac{y}{z}.& \left(12.Math.b\right)\end{array}$

[0100] In some implementations, the polynomial in θ includes also even powers of θ and/or a lower or higher order approximation.

[0101] According to the computational method of FIG. 8, a 3D world point Xc is projected onto the surface of a fisheye lens, which is represented by a polynomial function h. Thereby, the point Xs is projected into a point Xm in the sensor plane. The projection onto the sensor plane is affected by the lens distortion and this effect is modelled by the polynomial function h.

[0102] By using this camera model, each image point Xm may be mapped into a camera ray, which is a unit vector on the surface of a sphere with radius one. Computationally, the projection may be realized by normalization to a unit vector.

[0103] In some implementations, edges are determined in the distorted fisheye image and points of the detected edges are then projected into the camera rays using a camera module which implements the distortion correction. Each line is approximately mapped to a great circle or a segment of a great circle on the unit sphere. The vanishing points are determined by fitting circles. The calculations are simplified because the circles are great circles on the unit sphere, which have radius one.

[0104] In some implementations, the camera model is realized by generating a lookup-table for the mapping. The points on the unit sphere are easy to track as the motion is a circle in the surface of the sphere.

[0105] FIG. 9 shows a procedure for image stabilization according to the present disclosure.

[0106] In a step 40, one or more camera images are acquired from camera image sensors. In a step 41, a suitable image area is selected. For example, an image portion corresponding to a sky area may be clipped. In a step 42, image distortions are removed using an image transformation that is based on a lens model.

[0107] In a step 43, the corrected image is analyzed computationally and lines that correspond to parallel object lines are detected. In some examples, the lines are detected by and edge detection algorithm. The digitalized image may first be transformed by a suitable integral transformation, such as Hough transformation or a Radon transformation, and the straight lines may be detected as features of the transformed image.

[0108] In a step 44, the line segments are mapped to corresponding great circles on a unit sphere and, in a step 45, an intersection point of the unit circles is determined. In a step 46, a motion of the intersection point is determined. In some examples, a filter is used, for examples, a Kalman filter, to interpolate and predict the motion of the vanishing point.

[0109] If there is more than one vanishing point and those vanishing points all correspond to stationary objects, the motions of the vanishing points are similar. In some implementations, the motions of the vanishing points are averaged to obtain an average motion. The resulting motion is used as an average of the camera motion. In a step 47, the derived motion of the vanishing points is used to stabilize the camera image.

[0110] The sequence of the steps in FIG. 9 are provided by way of example. The steps in a given realization may be different from the sequence of FIG. 9. For example, in a further example, straight lines are detected first and a vanishing point is detected only after mapping the lines to corresponding great circles, for example by attributing a vanishing point to a set of lines if the corresponding vanishing points of each pair of lines in the set of lines cluster on the unit sphere.

[0111] In this way, a vanishing point may be identified without prior identification of the line segments as street borders or as other objects with a known orientation. The clustering of vanishing points works especially well for objects with many horizontal and vertical lines, which are provided by many types of buildings and artificial structures.

[0112] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.