Method and Device for Displaying Vehicle Surroundings

Abstract

The invention relates to a device (100) as well as a method for displaying vehicle surroundings in a vehicle (1) during an instantaneous second time point (t). The method comprises the steps: providing (S1) a first blind area image (I_B[t]), which contains an image synthesis of a blind area (B) of the vehicle surroundings, which blind area is arranged outside of a field of view (141F, 141R, 141LL, 141LR) of a vehicle camera (140F, 140R, 140LL, 140LR), at a first time point (t−1) preceding the second time point (t), arranging (S2) each first blind area pixel (IB_SP[t−1]) of the blind area (B) in a new position estimated for the second time point (t), determining (S3) whether the new position of each first blind area pixel (IB_SP[t−1]) at the second time point (t) still lies within the blind area (B), and producing (S4) a respective second blind area pixel (IB_SP[t]) for the second time point (t) by synthesizing (S5A) each first blind area pixel (IB_SP[t−1]) on the basis of motion data (v_x, v_y) of the vehicle (1) if the new position is determined to lie within the blind area (B).

Claims

1. A method for displaying vehicle surroundings in a vehicle (1) during an instantaneous second time point (t), having the steps: providing (S1) a first blind area image (I_B[t]), which contains an image synthesis of a blind area (B) of the vehicle surroundings, which blind area is arranged outside of a field of view (141F, 141R, 141LL, 141LR) of a vehicle camera (140F, 140R, 140LL, 140LR), at a first time point (t−1) preceding the second time point (t), arranging (S2) each first blind area pixel (IB _SP[t−1]) of the blind area (B) in a new position estimated for the second time point (t), determining (S3) whether the new position of each first blind area pixel (IB_SP[t−1]) at the second time point (t) still lies within the blind area (B), and producing (S4) a respective second blind area pixel (IB_SP [t]) for the second time point (t) by synthesizing (S5A) each first blind area pixel (IB _SP[t−1]) on the basis of motion data (v_x, v_y) of the vehicle (1) if the new position is determined to lie within the blind area (B).

2. The method according to claim 1, wherein the respective second blind area pixel (IB_SP[t]) is produced (S4) for the second time point (t) by synthesizing (S5B) each first blind area pixel (IB_SP[t−1]) on the basis of at least one captured camera image (I_F[t−1], I_R[t−1], I_LL[t−1], I _LR[t−1]) which is provided at the first time point (t−1) if the new position is determined to lie outside of the blind area (B).

3. The method according to claim 2, wherein it is determined on the basis of a motion vector (SP_V) obtained from the motion data (v_x, v_y) of each first blind area pixel (IB_SP[t−1]), from which vehicle camera from a multiplicity of vehicle cameras (140F, 140R, 140LL, 140LR), which each have different fields of view (141F, 141R, 141LL, 141LR) , the camera image (I_F[t−1], I_R[t−1], I_LL[t−1], I_LR[t−1]) is provided.

4. The method according to claim 3, wherein the camera image (I_F[t−1], I_R[t−1], I_LR[t−1], I_RR[t−1]) of the respective camera (140F, 140R, 140LL, 140LR), in the field of view (141F, 141R, 141LL, 141LR) of which the motion vector (SP_V) is arranged, is selected to be provided.

5. The method according to claim 3, wherein exactly one camera image (I_F[t−1], I_R[t−1], I_LR[t−1], I_RR[t−1]) per vehicle camera (140F, 140R, 140LL, 140LR) is retained to be provided.

6. The method according to claim 1, wherein exactly one blind area image (IB_SP[t−1]) of the first time point (t−1) is retained to be provided.

7. The method according to claim 1, wherein a second blind area image (IB_SP[t]) produced from the respective second blind area pixels (IB_SP[t]) for the second time point (t) and at least one instantaneous camera image (I_F[t], I_R[t], I_LL[t], I_LR[t]) of the second time point (t) are combined for displaying in the vehicle (1).

8. The method according to claim 1, wherein at the initial start a display image (I) of the vehicle surroundings is displaced so far into the field of view (141F, 141R, 141LL, 141LR) that the blind area (B) is hidden.

9. The method according to claim 1, wherein the blind area (B) is arranged in a ground level which is driven over by the vehicle (1).

10. A device (100) for displaying vehicle surroundings in a vehicle (1) during an instantaneous second time point (t), having at least one vehicle camera (140F, 140R, 140LL, 140LR) for detecting a corresponding field of view (141F, 141R, 141LL, 141LR) of the vehicle surroundings, a storage apparatus (130), in which exactly one first blind area image (I_B[t−1]) is retained, which contains an image synthesis of a blind area (B) of the vehicle surroundings, which blind area is arranged outside of the field of view (141F, 141R, 141LL, 141LR), at a first time point (t−1) preceding the second time point (t), and a data processing apparatus (120) which is designed i) to arrange each first blind area pixel (IB_SP[t−1]) of the blind area (B) in a new position estimated for the second time point (t), ii) to determine whether the new position of each first blind area pixel (IB_SP[t−1]) at the second time point (t) still lies within the blind area (B), and iii) to produce a respective second blind area pixel (IB_SP[t]) for the second time point (t) by synthesizing (S5) each first blind area pixel (IB_SP[t−1]) on the basis of motion data (v_x, v_y) of the vehicle (1) if the new position is determined to lie within the blind area (B).

Description

[0028] Advantageous embodiments of the invention are explained in greater detail below, with reference to the appended figures, wherein:

[0029] FIG. 1 shows a vehicle having a driving assistance system according to an aspect, which has a device for displaying vehicle surroundings according to a further aspect of the invention,

[0030] FIG. 2A shows a diagram of the production of a blind area image, wherein a previous, synthesized blind area image serves as the basis,

[0031] FIG. 2B shows a diagram of the production of a blind area image, wherein a previous, captured camera image serves as the basis,

[0032] FIG. 2C shows a diagram of the production of a blind area image, wherein a previous, captured camera image serves as the basis, and

[0033] FIG. 3 show a flowchart of a method according to a further aspect of the invention.

[0034] The figures are merely schematic representations and only serve to explain the invention. The same or similarly acting elements are consistently provided with the same reference numerals.

[0035] FIG. 1 shows a vehicle 1 standing on a ground level, which can substantially move in the directions x and y and which has a device 100 in the form of a driving assistance system or respectively a surround view system. This makes it possible for a vehicle passenger to have a panoramic view of the instantaneous vehicle surroundings displayed to him in the interior of the vehicle as a display or respectively pictorial reproduction.

[0036] To this end, the device 100 has a display apparatus 110 in the form of a screen, which is arranged in the interior of the vehicle 1 and visually represents a display image I. Moreover, the device 100 has a data processing apparatus 120 having at least one processor (which is not described in greater detail), which interacts with the display device 110, and a storage apparatus 130 which likewise interacts therewith. In addition, the device 100 has a multiplicity of vehicle cameras 140F, 140R, 140LL, 140LR which are mounted at different positions of the vehicle 1 and have different viewing angles or respectively fields of view 141F, 141R, 141LL, 141LR. Thus, the vehicle camera 140F is arranged on the vehicle front, the vehicle camera 140R is arranged on the rear of the vehicle, the vehicle camera 140LL is arranged laterally on the left and the vehicle camera 140LR is arranged laterally on the right. The fields of view 141F, 141R, 141LL, 141LR can be detected as the respective camera image I_F, I_R, I_LL, I_LR and are directly reproduced on the display apparatus 110 and, if applicable, are (temporarily) stored in the storage apparatus 130. For displaying, the camera images I_F, I_R, I_LL, I_LR are composed or respectively combined by the data processing apparatus 120 into the display image I (see FIGS. 2A-2C).

[0037] It is obvious from FIG. 1 that in the ground level, due to the vehicle 1 located thereon or respectively the body thereof, a blind area B is produced, which does not lie in any of the fields of view 141F, 141R, 141LL, 141LR and which cannot therefore be detected by any of the vehicle cameras 140F, 140R, 140LL, 140LR, since the body obscures the respective view. Consequently, no direct camera image can be reproduced for the blind area B as a blind area image I_B of the display image I on the display apparatus 110.

[0038] In order to nevertheless represent the blind area B with a substantially photo-realistic display in the form of the blind area image I_B on the display apparatus 110, the device 100 can be operated with the method described below on the basis of FIGS. 2A-2C and 3.

[0039] In principle, the vehicle surroundings are to be displayed on the display apparatus 110 substantially in real time at an instantaneous time point t, that is to say the display image I is to contain the vehicle surroundings at the instantaneous time point t. This is indicated below as well as in FIGS. 2A-2C by a time reference [t] in the reference numerals. Accordingly, a first time point t−1 preceding the instantaneous second time point t is referenced in FIGS. 2A-2C with [t−1].

[0040] When the device 100 is initially started, the blind area B is hidden in the display image I of the display apparatus 110 in an optional step S0, by displacing the display image I of the vehicle surroundings so far into one of the fields of view 141F, 141R, 141LL, 141LR of the vehicle cameras 140F, 140R, 140LL, 140LR that the blind area B is thus hidden. The respective field of view 141F, 141R, 141LL, 141LR can be selected e.g. as a function of the motion direction x, y of the vehicle, the motion data thereof such as a vehicle speed v_x, v_y, the steering angle, the gear selection of the transmission, etc. To put it more simply, the display image I runs ahead of the actual vehicle surroundings at time point t e.g. by a vehicle length so that the blind area B is hidden. An initial blind area image I_B[t−1] can thus be stored and retained, which, instead of the blind area B, contains an image section of the respective camera image IF[t−1], IR[t−1], ILL[t−1], ILR[t−1].

[0041] In a step S1, the blind area image I_B[t−1] of the time point t−1 is then provided at the instantaneous time point t from the storage apparatus 130, which blind area image only contains the image content explained above in the first run-through following the initial start of the device 100, but which is constantly updated during operation as described below. If the vehicle 1 continues moving at e.g. 15 m/s in the x-direction, the image content of the blind area image I_B is updated with a corresponding image refresh rate and contains the image content at the preceding time point t−1 when it is provided for the respective instantaneous time point t.

[0042] In a step S2, each first blind area pixel IB_SP[t−1] of the blind area image I_B[t−1] is arranged by the data processing apparatus 120 in a new position estimated or respectively predicted for the instantaneous time point t by e.g. a transformation or similar, for which reason the motion data v_x, v_y of the vehicle 1 are in particular considered. These are made available e.g. by a vehicle system, e.g. a vehicle bus, or are determined from the camera images I_F, I_R, I_LL, I_LR. The new position can be estimated or respectively predicted e.g. by a suitable image processing method. If the vehicle 1 has continued moving at e.g. v_x=15 m/s in the x-direction, a corresponding motion vector SP_V is determined for each first blind area pixel IB_SP[t−1] and this is accordingly arranged as a second blind area pixel IB_SP[t] in its new position, as indicated in FIG. 2A.

[0043] It is then determined in a step S3 by the data processing apparatus 120 whether the new position of the respective blind area pixel IB_SP at the instantaneous time point t still lies within the blind area B. In the exemplary embodiment shown in FIG. 2A, a respective blind area pixel IB_SP shifted or respectively transformed by the motion vector SP_V still lies within the blind area B.

[0044] In this case, a respective second blind area pixel IB_SP[t] is produced, e.g. rendered, for the instantaneous time point t in the data processing apparatus 120, in a step S4, by motion compensating each first blind area pixel IB_SP[t−1], on the basis of the motion data of the vehicle 1 in a step S5A, i.e. in particular shifting it by the motion vector SP_V. As indicated in FIG. 2A for two exemplary blind area pixels IB_SP, this is repeated frequently until such time as each first blind area pixel IB_SP [t−1] is preferably arranged in its new position and a second blind area image I_B[t] is produced, e.g. rendered, therefrom, and is displayed in the display image I [t] in place of the blind area B. The second blind area image I_B[t] is then stored in the storage apparatus 130 and is provided in the next run-through of the method beginning with step S1 as a first blind area image I_B[t−1], which has been updated in terms of its content, for a further second time point t+n.

[0045] This can be generalized as follows:

I_B[x, y, t]=I_B[x+v_x, y+v_y, t−1],

[0046] wherein I_B is the blind area image, v_x and v_y are the motion data in the x- or respectively y-direction, t−1 is the first time point and t is the instantaneous second time point.

[0047] FIG. 2B shows another possible case of the determination in step S3, in which the new position of each first blind area pixel IB_SP[t−1] at the instantaneous time point t no longer lies within, but instead outside of the blind area B. Thus, in the exemplary embodiment shown in FIG. 2B, the respective blind area pixel IB_SP which has been shifted or respectively transformed by the motion vector SP_V lies in the field of view 141F of the vehicle camera 140F due to the motion of the vehicle in the x-direction.

[0048] In this case, the respective second blind area pixel IB_SP[t] for the instantaneous time point t is produced, e.g. rendered, in step S4, by synthesizing, e.g. rendering, each first blind area pixel IB_SP[t] on the basis of the captured camera image I_F[t−1] which is provided at the first time point t−1, in a step S5B.

[0049] FIG. 2C shows this case of the determination in step S3 using the example of cornering to the right, which comprises a motion both in the x- and y-directions having the motion data v_x, v_y. As shown in FIG. 2C, the respective blind area pixel IB_SP which is shifted or respectively transformed by the motion vector SP_V now lies in the field of view 141LR of the vehicle camera 140LR, due to the motion of the vehicle in the x-direction.

[0050] In this case, the respective second blind area pixel IB_SP[t] is produced, e.g. rendered, for the instantaneous time point t in step S4, by synthesizing, e.g. rendering, each first blind area pixel IB[t−1] on the basis of the captured camera image I_LR[t−1] which is provided at the first time point t−1 in step S5B. As indicated in FIG. 2B for two exemplary blind area pixels IB_SP, this is repeated frequently until such time as each first blind area pixel IB_SP [t−1] is preferably arranged in its new position and a second blind area image I_B[t] is produced, e.g. rendered, therefrom, and is displayed in the display image I [t] in place of the blind area B. The second blind area image I_B[t] is then stored in the storage apparatus 130 and provided in the next run-through of the method beginning with step S1 as a first blind area image I_B[t−1] which has been updated in terms of its content for a further second time point t+n.

[0051] In steps S5B according to FIGS. 2B and 2C, the blind area pixel IB_SP[t] to be displayed is accordingly produced by an image synthesis, e.g. rendering, of a captured camera image I_F[t−1] or respectively I_LR[t−1] at time point t−1. This principle can of course be easily transferred to the remaining camera images I_R and I_LL.

[0052] This can be generalized as follows:

I_B[x, y, t]=I_F, I_R, I_LL, I_LR[x+v_x, y+v_y, t−1],

[0053] wherein I_B is the blind area image, I_F-I_LR are the camera images of the vehicle cameras 140E-140LR, v_x and v_y are the motion data in the x- or respectively y-direction, t−1 is the first time point and t is the instantaneous second time point.

[0054] In an optional step S6, the display image I for the display apparatus 110 is then combined from the second blind area image I_B[t] and the instantaneous camera images I_F[t], I_R[t], I_LL[t], I_LR[t] of the second time point t.

[0055] FIG. 3 shows the method described above having the optional step S0, the steps S1-S5 as well as the optional step S6, which is once again summarized in a flowchart.