MIRROR REPLACEMENT SYSTEM AND METHOD FOR DISPLAYING IMAGE AND/OR VIDEO DATA OF THE SURROUNDINGS OF A MOTOR VEHICLE

Abstract

A mirror replacement system (12) for a motor vehicle (10) is described, having at least one processor unit (28), an optical sensor unit (14) which generates image and/or video data of the environment of the motor vehicle (10), and a screen (38) which is arranged so as to display at least part of the generated image and/or video data. The mirror replacement system (12) comprises a viewing direction sensor (30) which is arranged so as to determine the viewing direction of the vehicle driver (36) on the screen (38) or the position of the head of the vehicle driver (36). The processor unit (28) is arranged so as to adapt the image and/or video data to be displayed on the screen (38) based on the detected viewing direction or position of the head so that the image and/or video data displayed on the screen (38) are adapted to the position of the head or to the viewing direction of the vehicle driver (36). A method of displaying image and/or video data is furthermore described.

Claims

1. A mirror replacement system (12) for a motor vehicle (10), having at least one processor unit (28), an optical sensor unit (14) which generates image and/or video data of the environment of the motor vehicle (10), and a screen (38) which is arranged so as to display at least part of the generated image and/or video data, wherein the mirror replacement system (12) comprises a viewing direction sensor (30) which is arranged so as to detect the viewing direction of the vehicle driver (36) on the screen (38) or the position of the head of the vehicle driver (36), wherein the processor unit (28) is arranged so as to adapt the image and/or video data to be displayed on the screen (38) based on the detected viewing direction or the position of the head so that the image and/or video data displayed on the screen (38) are adapted to the position of the head or to the viewing direction of the vehicle driver (36).

2. The mirror replacement system (12) according to claim 1, wherein the optical sensor unit (14) comprises at least one camera (16-20), in particular three cameras (16-20).

3. The mirror replacement system (12) according to claim 1, wherein the viewing direction sensor (30) comprises at least one infrared sensor (32, 34).

4. The mirror replacement system (12) according to claim 1, wherein the mirror replacement system (12) is arranged so as to detect the three-dimensional position and/or a change in position of the head of the vehicle driver (36) in all three spatial directions.

5. The mirror replacement system (12) according to claim 1, wherein the mirror replacement system (12) is arranged so as to scale the image and/or video data to be displayed based on a detected change in position of the head in the vehicle direction, the scaling of the image and/or video data to be displayed being linearly related to the change in position of the head.

8. The mirror replacement system (12) according to claim 1, wherein the mirror replacement system (12) is arranged so as to determine a scaling based on an origin which deviates in the vehicle direction from the neutral head position of the vehicle driver (36), so that a zoomed-out view of the displayed image and/or video data on the screen (38) is obtained in the neutral head position, in particular wherein an offset of approximately 0.75 m from the origin is provided.

7. The mirror replacement system (12) according to claim 1, wherein characterized in that the mirror replacement system (12) is arranged so as to shift the image and/or video data to be displayed based on a detected change in the viewing direction or a change in position of the head transversely to the vehicle direction, the shifting of the image and/or video data to be displayed being linearly related to the change in the viewing direction or to the change in position of the head.

8. The mirror replacement system (12) according to claim 1, wherein the mirror replacement system (12) is arranged so as to determine a shift factor and/or a zoom factor for adapting the displayed image and/or video data which is/are dependent on the viewing direction or the position of the head.

9. The mirror replacement system (12) according to claim 8, wherein the processor unit (28) is arranged so as to smooth the shift factor and/or the zoom factor exponentially.

10. The mirror replacement system (12) according to claim 8, wherein the processor unit (28) comprises a fragment shader (40) and is arranged so as to determine the image and/or video data to be displayed taking the shift factor and/or the zoom factor into account, in particular wherein the fragment shader (40) additionally takes the smoothed shift factor, the smoothed zoom factor, the height and/or the width of the images or videos to be displayed into account.

11. The mirror replacement system (12) according to claim 1, wherein the mirror replacement system (12) is arranged so as to use threshold values for the detected change in the viewing direction, the change in position or the position of the head, wherein the processor unit 28 does not change the image and/or video data to be displayed until they are exceeded.

12. The mirror replacement system (12) according to claim 1, wherein the mirror replacement system (12) is arranged so as to carry out non-linear transformations of the generated image and/or video data so that a scaling of the image and/or video data to be displayed is linearly related to the change in the viewing direction, the change in position or the position of the head.

13. A method of displaying image and/or video data of the environment of a motor vehicle (10) by a mirror replacement system (12) for a motor vehicle (10), comprising the following steps: recording image and/or video data by an optical sensor unit (14), detecting the viewing direction of the vehicle driver (36) on a screen (38) or the position of the head of the vehicle driver (36), processing the recorded image and/or video data by a processor unit (28), showing at least part of the generated image and/or video data by the screen, wherein the viewing direction of the vehicle driver (36) on the screen (38) or the position of the head of the vehicle driver (36) is used in the processing of the recorded image and/or video data, so that the image and/or video data displayed on the screen (38) are adapted to the position of the head or to the viewing direction of the vehicle driver (36).

14. The method according to claim 13, wherein the three-dimensional position and/or a change in position of the head of the vehicle driver (36) is/are detected in all three spatial directions.

15. The method according to claim 13, wherein the image and/or video data to be displayed are scaled linearly based on a detected change in position of the head in the vehicle direction, in particular wherein a scaling factor between 0.0005 and 0.002 per cm of a head movement is used, for example a scaling factor of 0.00125 per cm.

16. The method according to claim 13, wherein the image and/or video data to be displayed are shifted linearly based on a detected change in the viewing direction or a change in position of the head transversely to the vehicle direction, in particular wherein a scaling factor between 0.05 and 0.2 per cm of a head movement is used, for example a scaling factor of 0.1 per cm.

17. The method claim 13, wherein the scaling is determined based on an origin which deviates in the vehicle direction from the neutral head position of the vehicle driver (36), so that a slightly zoomed-out view of the displayed image and/or video data on the screen (38) is obtained in the neutral head position, in particular wherein an offset of approximately 0.75 m from the origin is provided.

18. The method according to claim 13, wherein a shift factor and/or a zoom factor is/are determined for the image and/or video data to be displayed, which is/are dependent on the viewing direction or the position of the head.

19. The method according to claim 18, wherein the shift factor and/or the zoom factor are/is smoothed exponentially.

20. The method according to claim 18, wherein the image and/or video data to be displayed are calculated in a fragment shades (40) taking the shift factor and/or the zoom factor into account, in particular wherein in addition, the smoothed shift factor, the smoothed zoom factor, the height and/or the width of the images or videos to be displayed are/is taken into account.

21. The method according to claim 13, wherein threshold values are used for the detected change in the viewing direction, the change in position or the position of the head, the image and/or video data to be displayed being not changed until they are exceeded.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Further advantages and characteristics of the mirror replacement system according to the invention and of the method according to the invention of displaying image and/or video data of the environment of a motor vehicle will become apparent from the description below and from the drawings to which reference is made and in which:

[0069] FIG. 1 shows a schematic representation of a motor vehicle comprising a mirror replacement system according to the invention,

[0070] FIG. 2 shows a schematic representation of a mirror replacement system according to one embodiment, and

[0071] FIG. 3 shows a schematic representation of a flow chart of a method according to the invention.

DETAILED DESCRIPTION

[0072] FIG. 1 shows a motor vehicle 10 including a mirror replacement system 12 which replaces a conventional physical mirror system of physical mirrors.

[0073] The mirror replacement system 12 has an optical sensor unit 14, which in the embodiment shown comprises three wide-angle cameras 16-20, which are provided to replace the two side mirrors and the inside mirror.

[0074] The first camera 16 is assigned to the left side 22 of the motor vehicle 10, whereas the second camera 18 is assigned to the right side 24 of the motor vehicle 10. Furthermore, the third camera 20 is assigned to the rear side 26 of the motor vehicle 10. The respective cameras 16-20 are for example positioned on the respective sides 22-26 of the motor vehicle 10. Therefore, the first camera 16 detects the left lateral and rear environment of the motor vehicle 10, whereas the second camera 18 detects the right lateral and rear environment of the motor vehicle 10. In contrast thereto, the third camera 20 detects the rear area of the motor vehicle 10, in particular the area directly behind the rear end of the motor vehicle 10.

[0075] The three cameras 16-20 each detect an area of the environment of the motor vehicle 10, the corresponding areas of the left and right cameras 16, 18 at least partially overlapping with the area of the rear camera 20, so that there is a left overlap area and a right overlap area. Data from two cameras 16-20 are thus detected simultaneously in the respective overlap area.

[0076] The image and/or video data of the environment of the motor vehicle 10 detected by the cameras 16-20, i.e. the optical sensor unit 14, are transmitted to a processor unit 28, which is part of the mirror replacement system 12. The processor unit 28 processes the received data accordingly, as explained below.

[0077] In addition, the mirror replacement system 12 comprises a viewing direction sensor 30, which in the embodiment shown is formed by two infrared sensors 32, 34, which are located in the vehicle interior.

[0078] Alternatively or in addition, the viewing direction sensor 30 can comprise at least one camera, for example an ROB camera, which is directed towards the vehicle driver.

[0079] The viewing direction sensor 30 is generally arranged so as to detect the position of the head of the vehicle driver 36 or the viewing direction of the vehicle driver 36 on a screen 38, which is also part of the mirror replacement system 12. Eye-tracking techniques can be used for this purpose.

[0080] The image and/or video data detected or generated by the optical sensor unit 14 are displayed on the screen 38, which is why the screen 38 is coupled to the processor unit 28. The detected image or video data are thus shown to the vehicle driver, enabling him to monitor the environment of the motor vehicle 10.

[0081] The viewing direction sensor 30 is also coupled to the processor unit 28 so that the data detected by the viewing direction sensor 30 are evaluated and accordingly taken into account by the processor unit 28. This will be explained in more detail below with reference to FIG. 3, when the method of displaying image and/or video data of the environment of the motor vehicle 10 by means of the mirror replacement system 12 is described.

[0082] FIG. 1 also shows that the screen 38 is positioned centrally between the two infrared sensors 32, 34.

[0083] Generally, the two infrared sensors 32, 34 have a defined distance to each other and to the head of the vehicle driver; at least in the neutral head position.

[0084] The screen 38 is also located in a plane which extends in the x and y direction, i.e. transversely to the vehicle direction, as is usually the case with an inside mirror or a dashboard. In this respect, the z-direction is the vehicle direction of the motor vehicle 10. The vehicle direction also substantially corresponds to the direction of travel of the motor vehicle 10.

[0085] FIG. 2 shows an alternative embodiment of the mirror replacement system 12, which comprises two processor units 28a, 28b. The first processor unit serves to calculate the viewing direction of the vehicle driver or the position of the head of the vehicle driver 36, the correspondingly calculated tracking data being transmitted to the second processor unit 28b, which additionally receives the data detected by the optical sensor unit 14, in particular the corresponding raw data, Based on the tracking data received from the first processor unit 28a and the data received from the optical sensor unit 14, the second processor unit 28b calculates the image to be displayed correspondingly, which is then transmitted to the screen 38 for display.

[0086] In this respect, the second embodiment of the mirror replacement system 12 shown in FIG. 2 differs from the first embodiment shown in FIG. 1 only in that two separate processor units 28a, 28b are provided, as a result of which the respective processor units 28a, 28b can be better adapted to the specific requirements, i.e. to the calculation steps to be performed.

[0087] With reference to FIG. 3, it is explained below how the mirror replacement system 12 calculates and accordingly displays the image and/or video data of the environment of the motor vehicle 10 to be displayed.

[0088] First, image and/or video data of the environment of the motor vehicle 10 are recorded by means of the optical sensor unit 14, i.e. the respective cameras 16 20. This results in a left data set of the first camera 16, a right data set of he second camera 18 and a middle or rear data set of the rear camera 20.

[0089] The correspondingly recorded image and/or video data, i.e. the respective data sets, are transmitted to the at least one processor unit 28, where they are further processed.

[0090] In addition, the viewing direction of the vehicle driver 36 on the screen 38 or the position of the head of the vehicle driver 36 is detected by means of the viewing direction sensor 30. The three-dimensional position of the head can be detected so that the position in the x, y and z directions, i.e. in the vehicle direction (z direction) and transversely to the vehicle direction (x direction and/or y direction) is determined.

[0091] The corresponding data of the viewing direction sensor 30 are forwarded to the at least one processor unit 28, which processes the recorded image and/or video data of the optical sensor unit 14 accordingly. The processor unit 28 also determines the viewing direction of the vehicle driver on the screen 28 or the (three-dimensional) position of the head of the vehicle driver 36 on the basis of the acquired data of the viewing direction sensor 30.

[0092] When processing the recorded image and/or video data which are to be displayed, the processor unit 28 uses the determined viewing direction of the vehicle driver on the screen 28 or the (three-dimensional) position of the head of the vehicle driver 36 in order to accordingly adapt the section to be displayed.

[0093] On the basis of the detected position of the head of the vehicle driver 36 or his viewing direction, the section to be displayed on the screen 38 is selected accordingly such that the latter is adapted to the position of the head or to the viewing direction of the vehicle driver 36. The generation of a natural or naturally perceived mirror image for the driver 36 on the screen 38 is thus obtained, which depends on the viewing direction or on the position of the head of the vehicle driver 36 and in particular changes or accordingly adapts variably with a change in position or in the viewing direction.

[0094] That is to say, if the vehicle driver 36 moves his head or changes the viewing direction, this is detected by the viewing direction sensor 30 and the processor unit 28, so that the section shown on the screen 38 is adapted accordingly. The corresponding change in position of the head is detected in all three spatial directions to create a natural and intuitive mirror image for the vehicle driver 36.

[0095] The mirror replacement system 12, in particular the processor unit 28, is arranged so as to linearly scale the image and/or video data to be displayed on the basis of the detected change in position of the head in the z direction, i.e. in the vehicle direction. This results in a particularly natural sensation for the vehicle driver.

[0096] A scaling factor between 0.0005 and 0.002 per cm is for example used, in particular a scaling factor of 0.00125 per cm, to calculate a corresponding scaling of the image and/or video data when the vehicle driver 36 moves towards or away from the screen 38 in the vehicle direction.

[0097] Furthermore, the mirror replacement system 12, in particular the processor unit 28, can be arranged so as to determine the scaling starting from an origin which deviates from the neutral head position of the vehicle driver 36 in the z-direction, i.e. in the vehicle direction, so that a slightly zoomed-out view of the displayed image and/or video data is obtained on the screen 38 when the vehicle driver 36 is in his neutral seating position, which is associated with the neutral head position.

[0098] Generally, a zoom factor is thus determined via which the movement of the head of the vehicle driver 36 in the z-direction, i.e. in the direction of the vehicle, is converted accordingly. The conversion is linear, an offset being provided so that a zoomed-out view is obtained when the driver 36 is in his neutral seating position. The zoom factor can therefore be represented by the following formula:

z=0.750.00125/cm *z

[0099] Here, the parameter z corresponds to the actual distance of movement of the driver 36 in the z-direction, i.e. in the vehicle direction, which is correspondingly converted with z.

[0100] Here, the scaling factor of 0.00125 per cm and the offset of 0.75 are provided so that the origin of the scaling has been moved forward by 0.75 m, the neutral head position being approximately 0.9 m. A shift factor can be calculated in an analogous manner, wherein for this purpose, the processor unit 28 shifts the image and/or video data to be displayed linearly on the basis of a detected change in the viewing direction or change in position of the head in the x-direction and/or y-direction. A scaling factor between 0.05 and 0.2 per cm can be used, for example a scaling factor of 0.1 per cm, so that the following formulae are obtained for the respective shift factors:

x=0.1/cm*x or y=0.1/cm*y

[0101] The corresponding origin of the x-axis or y-axis is placed in a centered image on the neutral head position of the vehicle driver 36. In the neutral head position, values of 0 are thus obtained.

[0102] The calculated shift factors can then be limited to a defined range of values, for example to a range of values from 1 to 1, so that a maximum shifting of the image or video data is obtained.

[0103] Generally, the linear shifting or linear scaling of the image and/or video data on the basis of the detected change in position or in the viewing direction ensures that a natural sensation for the vehicle driver is achieved.

[0104] At the same time, low translation factors (scaling factors) are used so that confusion of the vehicle driver 36 during the journey is avoided.

[0105] In addition, the calculated shift factors and the calculated zoom factor can be smoothed exponentially, so that jumps in the displayed images or videos and noise are avoided. However, since the displayed information must be quickly adapted to incoming movements, an exponential smoothing is advantageous.

[0106] To this end, the actual values of the corresponding factor are updated in each frame using the set point of the corresponding factor. Depending on the screen 38 used, this may be the case, for example, 60 times per second, i.e. for a 60 Hz screen. The exponential smoothing can for example be represented by the following formulae:

x=x+0.1*(xx),

y=y+0.1*(yy) and

z=z+0.2*(zz)

[0107] The double apostrophized parameters (x, y, z) are the actual values, which are updated accordingly on the basis of their set points (x, y, z).

[0108] Furthermore, the processor unit 28 can include a fragment shader 40, which calculates the texture coordinates (u, v) of the individual pixels of the images or videos to be displayed. For this purpose, the fragment shader 40 uses the previously determined shift factors or the zoom factor, in particular the smoothed shift factors or the smoothed zoom factor.

[0109] In addition, the fragment shader 40 can take the height (img_height) or width (img_width) of the images or videos to be displayed into account to vary the corresponding texture coordinates (u, v). The fragment shader 40 can then use the following formulae:

u=img_width(u*z ((1z)*0.5)*(1+x)*img_width) and

v=v*z+(1z)*0.5*img_height

[0110] At the same time, the processor unit 28 can apply thresholds in the generation or processing of the image and/or video data, which are provided for the detected change in the viewing direction, change in position or position of the head, Based on the threshold values, it is ensured that the displayed image and/or video data do not cause an unsettled impression on the vehicle driver 36 because a certain minimum distance must first be exceeded in terms of amount before a change in the displayed image and/or video data, i.e. a shifting or scaling of the image and/or video data occurs.

[0111] Furthermore, it may be provided that the processor unit 28 is arranged so as to supplement the image and/or video data provided by the optical sensor unit 14 with information. For example, the information can be maneuver lines which indicate to the driver 36 where the motor vehicle 10 will move based on a selected steering angle when reversing.

[0112] Other information that can be displayed for example includes warning signs, symbols and/or highlights.

[0113] The additionally displayed information can also be adapted accordingly in terms of its display based on the viewing direction of the vehicle driver 36 or the position of the head, so that not only the image and/or video data acquired by the optical sensor unit 14 is adapted based on the viewing direction or the position of the head of the vehicle driver 36, but also the additionally displayed information.

[0114] This creates a natural sensation for a so-caned augmented reality, which is similar to a real (physical) mirror.

[0115] This therefore results in an intuitively comprehensible mirror replacement system 12, which provides a natural representation of the detected environment of the motor vehicle 10 on the screen 38, as the displayed information accordingly adapts to the viewing direction or the position of the head of the vehicle driver 36.