DEPTH MEASUREMENT THROUGH DISPLAY

Abstract

Disclosed herein is a display device including at least one illumination source configured for projecting at least one illumination beam on at least one scene; at least one optical sensor having at least one light sensitive area, where the optical sensor is configured for measuring at least one reflection light beam generated by the scene in response to illumination by the illumination beam; at least one translucent display configured for displaying information, where the illumination source and the optical sensor are placed in direction of propagation of the illumination light beam in front of the display, and at least one control unit, where the control unit is configured for turning off the display in an area of the illumination source during illumination and/or in an area of the optical sensor during measuring.

Claims

1. A display device comprising: at least one illumination source configured for projecting at least one illumination beam on at least one scene; at least one optical sensor having at least one light sensitive area, wherein the optical sensor is configured for measuring at least one reflection light beam generated by the scene in response to illumination by the illumination beam; at least one translucent display configured for displaying information, wherein the illumination source and the optical sensor are placed in direction of propagation of the illumination light beam in front of the display, and at least one control unit, wherein the control unit is configured for turning off the display in an area of the illumination source during illumination and/or in an area of the optical sensor during measuring.

2. The display device according to claim 1, wherein the translucent display is a full size display having display material extending over the full size of the display.

3. The display device according to claim 1, wherein the display is configured for showing a black area in the area of the illumination source and/or in an area of the optical sensor when the control unit has turned off the display in the area of the illumination source during illumination and/or in the area of the optical sensor during measuring.

4. The display device according to claim 1, wherein the control unit is configured for turning off the display in the area of the illumination source such that the display in the area of the illumination source functions as an adjustable notch and/or for turning off the display in the area of the optical sensor such that the display in the area of the optical sensor functions as the adjustable notch, wherein the adjustable notch is configured to be active during illumination and/or measuring and inactive otherwise.

5. The display device according to claim 1, wherein the display device is configured for performing a face recognition using the optical sensor, wherein the control unit is configured for issuing an indication during performing face recognition indicating that face recognition is active, wherein the translucent display is configured for displaying said indication during performing face recognition.

6. The display device according to claim 1, wherein the optical sensor comprises at least one CMOS sensor.

7. The display device according to claim 1, wherein the illumination source comprises at least one infrared light source.

8. The display device according to claim 1, wherein the illumination source comprises at least one laser projector configured for generating at least one illumination pattern.

9. The display device according to claim 1, wherein the illumination source comprises at least one flood illumination light-emitting diode.

10. The display device according to claim 1, wherein the display, the illumination source and the optical sensor are synchronized.

11. The display device according to claim 1, wherein the display is or comprises at least one organic light-emitting diode (OLED) display, wherein, when the control unit has turned off the display in the area of the illumination source, the OLED display is non-active in the area of the illumination source and/or, when the control unit has turned off the display in the area of the optical sensor, the OLED display is non-active in the area of the optical sensor.

12. The display device according to claim 1, wherein the illumination source is configured for projecting at least one illumination pattern comprising a plurality of illumination features on the at least one scene, wherein the optical sensor is configured for determining at least one first image comprising a plurality of reflection features generated by the scene in response to illumination by the illumination features, wherein the display device further comprises at least one evaluation device, wherein the evaluation device is configured for evaluating the first image, wherein the evaluation of the first image comprises identifying the reflection features of the first image and sorting the identified reflection features with respect to brightness, wherein each of the reflection features comprises at least one beam profile, wherein the evaluation device is configured for determining at least one longitudinal coordinate z.sub.DPR for each of the reflection features by analysis of their beam profiles, wherein the evaluation device is configured for unambiguously matching of reflection features with corresponding illumination features by using the longitudinal coordinate z.sub.DPR, wherein the matching is performed with decreasing brightness of the reflection features starting with the brightest reflection feature, wherein the evaluation device is configured for classifying a reflection feature being matched with an illumination feature as real feature and for classifying a reflection feature not being matched with an illumination feature as false feature, wherein the evaluation device is configured for rejecting the false features and for generating a depth map for the real features by using the longitudinal coordinate z.sub.DPR.

13. The display device according to claim 12, wherein the evaluation device is configured for determining at least one second longitudinal coordinate z.sub.triang for each of the reflection features using triangulation and/or depth-from-defocus and/or structured light techniques.

14. The display device according to claim 13, wherein the evaluation device is configured for determining a combined longitudinal coordinate of the second longitudinal coordinate z.sub.triang and the longitudinal coordinate z.sub.DPR, wherein the combined longitudinal coordinate is a mean value of the second longitudinal coordinate z.sub.triang and the longitudinal coordinate z.sub.DPR, wherein the combined longitudinal coordinate is used for generating the depth map.

15. The display device according to claim 12, wherein the evaluation device is configured for determining the beam profile information for each of the reflection features by using depth-from-photon-ratio technique.

16. The display device according to claim 12, wherein the evaluation device is configured for determining at least one material property m of the object by evaluating the beam profile of at least one of the reflection features.

17. The display device according to claim 1, wherein the display device is a mobile device selected from the group consisting of: a television device, a cell phone, a smart phone, a game console, a tablet computer, a personal computer, a laptop, a tablet, a virtual reality device, and another type of portable computer.

18. A method for measuring through a translucent display, wherein at least one display device according to claim 1 is used, wherein the method comprises the following steps: a) illuminating at least one scene by using at least one illumination light beam generated by at least one illumination source, wherein the illumination source is placed in direction of propagation of the illumination beam in front of the display; b) measuring at least one reflection light beam generated by the scene in response to illumination by the illumination beam by using at least one optical sensor, wherein the optical sensor has at least one light sensitive area, wherein the optical sensor is placed in direction of propagation of the illumination beam in front of the display; and c) controlling the display by using at least one control unit, wherein the display is turned off in an area of the illumination source during illumination and/or in an area of the optical sensor during measuring.

19. A method of using the display device of claim 1, the method comprising using the display device for a purpose of use selected from the group consisting of: a position measurement in traffic technology; an entertainment application; a security application; a surveillance application; a safety application; a human-machine interface application; a tracking application; a photography application; an imaging application or camera application; a mapping application for generating maps of at least one space; a homing or tracking beacon detector for vehicles; an outdoor application; a mobile application; a communication application; a machine vision application; a robotics application; a quality control application; a manufacturing application; and an automotive application.

Description

BRIEF DESCRIPTION OF THE FIGURES

[0194] Further optional details and features of the invention are evident from the description of preferred exemplary embodiments which follows in conjunction with the dependent claims. In this context, the particular features may be implemented in an isolated fashion or in combination with other features. The invention is not restricted to the exemplary embodiments. The exemplary embodiments are shown schematically in the figures. Identical reference numerals in the individual figures refer to identical elements or elements with identical function, or elements which correspond to one another with regard to their functions.

[0195] Specifically, in the figures:

[0196] FIG. 1 shows an embodiment of a display device according to the present invention; and

[0197] FIGS. 2A to 2C show an embodiment of synchronizing the display, the illumination source and the optical sensor.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0198] FIG. 1 shows an embodiment of a display device 1 of the present invention in a highly schematic fashion. For example, the display device 1 may be a mobile device selected from the group consisting of: a television device, a cell phone, a smart phone, a game console, a tablet computer, a personal computer, a laptop, a tablet, a virtual reality device, or another type of portable computer.

[0199] The display device 1 comprises at least one translucent display 2 configured for displaying information. The display device 1 comprises at least one optical sensor 4 having at least one light sensitive area. The display device 1 comprises at least one illumination source 5 configured for projecting at least one illumination beam on at least one scene. The scene may be an arbitrary object or spatial region. The scene may comprise the at least one object and a surrounding environment.

[0200] The illumination source 5 is configured for projecting the illumination beam, in particular at least one illumination pattern comprising a plurality of illumination features, on the scene. The illumination source 5 may be adapted to directly or indirectly illuminating the scene, wherein the illumination beam is reflected or scattered by surfaces of the scene and, thereby, is at least partially directed towards the optical sensor. The illumination source 5 may be adapted to illuminate the scene, for example, by directing a light beam towards the scene, which reflects the light beam.

[0201] The illumination source 5 may comprise at least one light source. The illumination source 5 may comprise a plurality of light sources. The illumination source 5 may comprise an artificial illumination source, in particular at least one laser source and/or at least one incandescent lamp and/or at least one semiconductor light source, for example, at least one light-emitting diode, in particular an organic and/or inorganic light-emitting diode. The illumination source 5 may comprise at least one infrared light source. The illumination source 5 may be configured for generating the at least one illumination light beam, in particular the at least one illumination pattern, in the infrared region. Using light in the near infrared region allows that light is not or only weakly detected by human eyes and is still detectable by silicon sensors, in particular standard silicon sensors.

[0202] The illumination source 5 may comprise at least one laser projector. The laser projector may be a vertical-cavity surface-emitting laser (VCSEL) projector in combination with refractive optics. However, other embodiments are feasible. The laser projector may comprises at least one laser source and at least one diffractive optical element (DOE). The illumination source 5 may be or may comprise at least one multiple beam light source. For example, the illumination source 5 may comprise at least one laser source and one or more diffractive optical elements (DOEs). Specifically, the illumination source 5 may comprise at least one laser and/or laser source. Various types of lasers may be employed, such as semiconductor lasers, double heterostructure lasers, external cavity lasers, separate confinement heterostructure lasers, quantum cascade lasers, distributed bragg reflector lasers, polariton lasers, hybrid silicon lasers, extended cavity diode lasers, quantum dot lasers, volume Bragg grating lasers, Indium Arsenide lasers, transistor lasers, diode pumped lasers, distributed feedback lasers, quantum well lasers, interband cascade lasers, Gallium Arsenide lasers, semiconductor ring laser, extended cavity diode lasers, or vertical cavity surface-emitting lasers. Additionally or alternatively, non-laser light sources may be used, such as LEDs and/or light bulbs. The illumination source 5 may comprise one or more diffractive optical elements (DOEs) adapted to generate the illumination pattern. For example, the illumination source 5 may be adapted to generate and/or to project a cloud of points, for example the illumination source 5 may comprise one or more of at least one digital light processing projector, at least one LCoS projector, at least one spatial light modulator; at least one diffractive optical element; at least one array of light emitting diodes; at least one array of laser light sources. On account of their generally defined beam profiles and other properties of handleability, the use of at least one laser source as the illumination source 5 is particularly preferred. The illumination source 5 may be integrated into a housing of the display device.

[0203] The illumination source 5 configured for generating at least one illumination pattern. The illumination pattern may comprise a plurality of illumination features. The illumination pattern may be selected from the group consisting of: at least one point pattern; at least one line pattern; at least one stripe pattern; at least one checkerboard pattern; at least one pattern comprising an arrangement of periodic or non periodic features. The illumination pattern may comprise regular and/or constant and/or periodic pattern such as a triangular pattern, a rectangular pattern, a hexagonal pattern or a pattern comprising further convex tilings. The illumination pattern may exhibit the at least one illumination feature selected from the group consisting of: at least one point; at least one line; at least two lines such as parallel or crossing lines; at least one point and one line; at least one arrangement of periodic or non-periodic feature; at least one arbitrary shaped featured. The illumination pattern may comprise at least one pattern selected from the group consisting of: at least one point pattern, in particular a pseudo-random point pattern; a random point pattern or a quasi random pattern; at least one Sobol pattern; at least one quasiperiodic pattern; at least one pattern comprising at least one pre-known feature at least one regular pattern; at least one triangular pattern; at least one hexagonal pattern; at least one rectangular pattern at least one pattern comprising convex uniform tilings; at least one line pattern comprising at least one line; at least one line pattern comprising at least two lines such as parallel or crossing lines. For example, the illumination source 5 may be adapted to generate and/or to project a cloud of points. The illumination source 5 may comprise the at least one light projector adapted to generate a cloud of points such that the illumination pattern may comprise a plurality of point pattern. The illumination source 5 may comprise at least one mask adapted to generate the illumination pattern from at least one light beam generated by the illumination source 5.

[0204] The optical sensor 4 is configured for measuring at least one reflection light beam generated by the scene in response to illumination by the illumination beam. The display device 1 may comprise a single camera comprising the optical sensor 4. The display device 1 may comprise a plurality of cameras each comprising an optical sensor 4 or a plurality of optical sensors 4. The display device 1 may comprise a plurality of optical sensors 4 each having a light sensitive area.

[0205] The display device 1 may be configured for performing at least one distance measurement such as based on time-of-flight (ToF) technology and/or based one depth-from-defocus technology and/or based on depth-from-photon-ratio technique, also called beam profile analysis. With respect to depth-from-photon-ratio (DPR) technique reference is made to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the full content of which is included by reference. The optical sensor 4 may be or may comprise at least one distance sensor.

[0206] The optical sensor 4 specifically may be or may comprise at least one photodetector, preferably inorganic photodetectors, more preferably inorganic semiconductor photodetectors, most preferably silicon photodetectors. Specifically, the optical sensor 4 may be sensitive in the infrared spectral range. All pixels of the matrix or at least a group of the optical sensors of the matrix specifically may be identical. Groups of identical pixels of the matrix specifically may be provided for different spectral ranges, or all pixels may be identical in terms of spectral sensitivity. Further, the pixels may be identical in size and/or with regard to their electronic or optoelectronic properties. Specifically, the optical sensor 4 may be or may comprise at least one inorganic photodiode which are sensitive in the infrared spectral range, preferably in the range of 700 nm to 3.0 micrometers. Specifically, the optical sensor 4 may be sensitive in the part of the near infrared region where silicon photodiodes are applicable specifically in the range of 700 nm to 1100 nm. Infrared optical sensors which may be used for optical sensors may be commercially available infrared optical sensors, such as infrared optical sensors commercially available under the brand name Hertzstueck™ from trinamiX™ GmbH, D-67056 Ludwigshafen am Rhein, Germany. Thus, as an example, the optical sensor 4 may comprise at least one optical sensor of an intrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge photodiode, an InGaAs photodiode, an extended InGaAs photodiode, an InAs photodiode, an InSb photodiode, a HgCdTe photodiode. Additionally or alternatively, the optical sensor may comprise at least one optical sensor of an extrinsic photovoltaic type, more preferably at least one semiconductor photodiode selected from the group consisting of: a Ge:Au photodiode, a Ge:Hg photodiode, a Ge:Cu photodiode, a Ge:Zn photodiode, a Si:Ga photodiode, a Si:As photodiode. Additionally or alternatively, the optical sensor 4 may comprise at least one photoconductive sensor such as a PbS or PbSe sensor, a bolometer, preferably a bolometer selected from the group consisting of a VO bolometer and an amorphous Si bolometer.

[0207] For example, the optical sensor 4 may be or may comprise at least one element selected from the group consisting of a photodiode, a photocell, a photoconductor, a phototransistor or any combination thereof. For example, the optical sensor 4 may be or may comprise at least one element selected from the group consisting of a CCD sensor element, a CMOS sensor element, a photodiode, a photocell, a photoconductor, a phototransistor or any combination thereof. Any other type of photosensitive element may be used. The photosensitive element generally may fully or partially be made of inorganic materials and/or may fully or partially be made of organic materials. Most commonly, one or more photodiodes may be used, such as commercially available photodiodes, e.g. inorganic semiconductor photodiodes.

[0208] The display device 1 comprises the at least one translucent display 2 configured for displaying information. The illumination source 5 and the optical sensor 4 are placed in direction of propagation of the illumination light beam in front of the translucent display 2. The translucent display 2 may be or may comprise at least one screen. The screen may have an arbitrary shape, preferably a rectangular shape. The information displayed by the display 2 may be arbitrary information such as at least one image, at least one diagram, at least one histogram, at least one graphic, text, numbers, at least one sign, an operating menu, and the like.

[0209] The translucent display 2 may be a full size display having display material extending over the full size of the display 2. The translucent display 2 may be recess free or cutout free. The translucent display 2 may have an entire active display area. The translucent display 2 may be designed such that the entire display area is activatable. The translucent display 2 may have a continuous distribution of display material. The translucent display 2 may be designed without any recesses or cutouts. For example, the display device 1 may comprise a front side having a display area such as a rectangular display area at which the translucent display 2 is arranged.

[0210] The display area may be completely covered by the translucent display, in particular by the display material, and specifically without any recesses or notches. This may allow increasing the display size, in particular the area of the display device 1 configured for displaying information. For example, the whole and/or entire front size of the display device 1 may be covered by the display material, wherein, however, a frame enclosing the display 2 may be possible.

[0211] The translucent display 2 may be or may comprise at least one organic light-emitting diode (OLED) display. The OLED display may be configured for emitting visible light.

[0212] The display device comprises the at least one control unit 8. The control unit 8 is configured for is configured for turning off the display 2 in an area of the illumination source 5 during illumination and/or in an area of the optical sensor 4 during measuring. The control unit 8 may be configured for controlling at least one further component of the display device 1 such as the illumination source 5 and/or the optical sensor 2 and/or the display 2, in particular by using at least one processor and/or at least one application-specific integrated circuit. Thus, as an example, the control unit 8 may comprise at least one data processing device having a software code stored thereon comprising a number of computer commands. The control unit 8 may provide one or more hardware elements for performing one or more of the named operations and/or may provide one or more processors with software running thereon for performing one or more of the named operations. Thus, as an example, the control unit 8 may comprise one or more programmable devices such as one or more computers, application-specific integrated circuits (ASICs), Digital Signal Processors (DSPs), or Field Programmable Gate Arrays (FPGAs) which are configured to perform the above-mentioned controlling. Additionally or alternatively, however, the control unit 8 may also fully or partially be embodied by hardware.

[0213] The control unit 8 is configured for turning off the display in an area of the illumination source 5 during illumination and/or in an area of the optical sensor 4 during measuring. The turning off the display 2 in an area may comprise adjusting in particular preventing and/or interrupting and/or stopping power supply to the certain area of the display 2. As outlined above, the display 2 may comprise at least one OLED display. When the control unit 8 has turned off the display 2 in the area of the illumination source 5, the OLED display may be non-active in the area of the illumination source 5. When the control unit 8 has turned off the display 2 in the area of the optical sensor 4, the OLED display may be non-active in the area of the optical sensor 4. The control unit 8 may be configured for turning off the area of display 2 while measurement is active.

[0214] The illumination source 5 may comprise a radiation area in which the illumination beam, in particular the illumination pattern, is radiated towards the scene. The radiation area may be defined by an opening angle of the illumination source 5. The illumination source 5 and the optical sensor 4 may be arranged in a defined area. The illumination source 5 and the optical sensor 4 may be arranged in a fixed position with respect to each other. For example, the illumination source 5 and the optical sensor 4 may be arranged next to each other, in particular having a fixed distance. The illumination source 5 and the optical sensor 4 may be arranged such that the area of the translucent display 2 covered by the radiation area and the light sensitive area is minimal.

[0215] The display 2 may be configured for showing a black area 6 in the area of the illumination source 5 and/or in an area of the optical sensor 4 when the control unit 8 has turned off the display 2 in the area of the illumination source 4 during illumination and/or in the area of the optical sensor 4 during measuring. The black area 6 may be an area not emitting light and/or a reduced amount of light in comparison with other areas of the display 2. For example, the black area 6 may be a darkened area. Specifically, the control unit 8 is configured for turning off the display 2 in the area of the illumination source 5 such that the display 2 in the area of the illumination source 5 functions as an adjustable notch and/or for turning off the display 2 in the area of the optical sensor 4 such that the display 2 in the area of the optical sensor 4 functions as the adjustable notch. The adjustable notch may be configured to be active during illumination and/or measuring and inactive otherwise. The adjustable notch may function as a virtual notch which is active during measurement such as during face unlock, when the display device 1 is not in use and which is non-active when no optical sensor 4, in particular no front sensor, is needed. For the used OLED display this may mean that there is no activity in the display 2 at all. This may allow to ensure that no color of any pixel may be changed by the IR light. Additionally the display device 1, in particular the control unit 8 and/or a further processing device and/or a further optical element, may be configured for correcting the color in the display, e.g. perceived flickering of the IR-Laser.

[0216] The adjustable notch may comprise harsh edges. In other embodiments, however, the adjustable notch may be realized with brightness gradients to avoid any harsh fringes. The display device 1 may comprise brightness reducing elements configured for introducing a brightness gradient to the edge of the display 2 where the optical sensor 4 is usualy positioned to avoid any harsh fringes. This may allow to provide a reduced brightness in the area of the adjustable notch.

[0217] The control unit 8 may be configured for synchronizing the display 2 and the illumination source in such a way that they do not interfere with one another, the so-called toggle mode.

[0218] The control unit 8 may be configured for issuing an indication when the optical sensor 4 and/or the illumination source 5 are active. The translucent display 2 may be configured for displaying said indication when the optical sensor 4 and/or the illumination source 5 are active. For example, the display device 1 may be configured for performing a face recognition using the illumination source 5 and the optical sensor 4. Method and techniques for face recognition are generally known to the skilled person. The control unit 8 may be configured for issuing an indication during performing face recognition indicating that face recognition is active. The translucent display 2 may be configured for displaying said indication during performing face recognition. For example, the indication may be at least one warning element. The indication may be one or more of an icon and/or a logo and/or a symbol and/or an animation which indicates that the optical sensor 4 and/or the illumination source 5, in particular the face recognition, are active. For example, the black area 6 may comprise an identification mark that secure authentication is active. This may allow the user to recognize that he is in a safe environment e.g. for payment or signing or the like. For example, the warning element may change color and/or appearance for indicating that the face recognition is active. The indication may further allow the user to recognize that that the optical sensor, in particular the camera, is turned on to avoid spying. The control unit 8 and/or a further secure zone may be configured for issuing at least one command to display in the black area at least one watermark. The watermark may be a symbol which cannot be mimicked by the low-security app, e.g. from a store.

[0219] The arrangement of the illumination source 5 and optical sensor 4 in a direction of propagation of light reflected by the scene, behind the translucent display 2, however, may result in that diffraction grating of the display generates multiple laser points on the scene and also in an image captured by the optical sensor 4. Thereby these multiple spots on the image may not include any useful distance information. The display device 1 may comprise at least one evaluation device 10 configured for finding and evaluating reflection features of zero order of diffraction grating, i.e. real features, and may neglect the reflection features of the higher orders, i.e. false features.

[0220] The illumination source 5 may be configured for projecting at least one illumination pattern comprising a plurality of illumination features on the at least one scene. The optical sensor 4 may be configured for determining at least one first image comprising a plurality of reflection features generated by the scene in response to illumination by the illumination features. The display device 1 further may comprise the at least one evaluation device 10 configured for evaluating the first image, wherein the evaluation of the first image comprises identifying the reflection features of the first image and sorting the identified reflection features with respect to brightness. Each of the reflection features may comprise at least one beam profile. The evaluation device 10 may be configured for determining at least one longitudinal coordinate z.sub.DPR for each of the reflection features by analysis of their beam profiles. The evaluation device 10 may be configured for unambiguously matching of reflection features with corresponding illumination features by using the longitudinal coordinate z.sub.DPR. The matching may be performed with decreasing brightness of the reflection features starting with the brightest reflection feature. The evaluation device 10 may be configured for classifying a reflection feature being matched with an illumination feature as real feature and for classifying a reflection feature not being matched with an illumination feature as false feature. The evaluation device 10 may be configured for rejecting the false features and for generating a depth map for the real features by using the longitudinal coordinate z.sub.DPR.

[0221] The evaluation device 10 may be configured for performing at least one image analysis and/or image processing in order to identify the reflection features. The image analysis and/or image processing may use at least one feature detection algorithm. The image analysis and/or image processing may comprise one or more of the following: a filtering; a selection of at least one region of interest; a formation of a difference image between an image created by the sensor signals and at least one offset; an inversion of sensor signals by inverting an image created by the sensor signals; a formation of a difference image between an image created by the sensor signals at different times; a background correction; a decomposition into color channels; a decomposition into hue; saturation; and brightness channels; a frequency decomposition; a singular value decomposition; applying a blob detector; applying a corner detector; applying a Determinant of Hessian filter; applying a principle curvature-based region detector; applying a maximally stable extremal regions detector; applying a generalized Hough-transformation; applying a ridge detector; applying an affine invariant feature detector; applying an affine-adapted interest point operator; applying a Harris affine region detector; applying a Hessian affine region detector; applying a scale-invariant feature transform; applying a scale-space extrema detector; applying a local feature detector; applying speeded up robust features algorithm; applying a gradient location and orientation histogram algorithm; applying a histogram of oriented gradients descriptor; applying a Deriche edge detector; applying a differential edge detector; applying a spatio-temporal interest point detector; applying a Moravec corner detector; applying a Canny edge detector; applying a Laplacian of Gaussian filter; applying a Difference of Gaussian filter; applying a Sobel operator; applying a Laplace operator; applying a Scharr operator; applying a Prewitt operator; applying a Roberts operator; applying a Kirsch operator; applying a high-pass filter; applying a low-pass filter; applying a Fourier transformation; applying a Radon-transformation; applying a Hough-transformation; applying a wavelet-transformation; a thresholding; creating a binary image. The region of interest may be determined manually by a user or may be determined automatically, such as by recognizing a feature within the image generated by the optical sensor.

[0222] For example, the illumination source 5 may be configured for generating and/or projecting a cloud of points such that a plurality of illuminated regions is generated on the optical sensor 4, for example the CMOS detector. Additionally, disturbances may be present on the optical sensor such as disturbances due to speckles and/or extraneous light and/or multiple reflections. The evaluation device 10 may be adapted to determine at least one region of interest, for example one or more pixels illuminated by the light beam which are used for determination of the longitudinal coordinate of the object. For example, the evaluation device 10 may be adapted to perform a filtering method, for example, a blob-analysis and/or an edge filter and/or object recognition method.

[0223] The evaluation device 10 may be configured for performing at least one image correction. The image correction may comprise at least one background subtraction. The evaluation device 10 may be adapted to remove influences from background light from the beam profile, for example, by an imaging without further illumination.

[0224] Each of the reflection features comprises at least one beam profile. The beam profile may be selected from the group consisting of a trapezoid beam profile; a triangle beam profile; a conical beam profile and a linear combination of Gaussian beam profiles. The evaluation device 10 is configured for determining beam profile information for each of the reflection features by analysis of their beam profiles.

[0225] The evaluation device 10 is configured for determining at least one longitudinal coordinate z.sub.DPR for each of the reflection features by analysis of their beam profiles. For example, the analysis of the beam profile may comprise at least one of a histogram analysis step, a calculation of a difference measure, application of a neural network, application of a machine learning algorithm. The evaluation device 10 may be configured for symmetrizing and/or for normalizing and/or for filtering the beam profile, in particular to remove noise or asymmetries from recording under larger angles, recording edges or the like. The evaluation device 10 may filter the beam profile by removing high spatial frequencies such as by spatial frequency analysis and/or median filtering or the like. Summarization may be performed by center of intensity of the light spot and averaging all intensities at the same distance to the center. The evaluation device 10 may be configured for normalizing the beam profile to a maximum intensity, in particular to account for intensity differences due to the recorded distance. The evaluation device 10 may be configured for removing influences from background light from the beam profile, for example, by an imaging without illumination.

[0226] The evaluation device 10 may be configured for determining the longitudinal coordinate z.sub.DPR for each of the reflection features by using depth-from-photon-ratio technique. With respect to depth-from-photon-ratio (DPR) technique reference is made to WO 2018/091649 A1, WO 2018/091638 A1 and WO 2018/091640 A1, the full content of which is included by reference.

[0227] The evaluation device 10 may be configured for determining the beam profile of each of the reflection features. The determining the beam profile may comprise identifying at least one reflection feature provided by the optical sensor 4 and/or selecting at least one reflection feature provided by the optical sensor 4 and evaluating at least one intensity distribution of the reflection feature. As an example, a region of the image may be used and evaluated for determining the intensity distribution, such as a three-dimensional intensity distribution or a two-dimensional intensity distribution, such as along an axis or line through the image. As an example, a center of illumination by the light beam may be determined, such as by determining the at least one pixel having the highest illumination, and a cross-sectional axis may be chosen through the center of illumination. The intensity distribution may an intensity distribution as a function of a coordinate along this cross-sectional axis through the center of illumination. Other evaluation algorithms are feasible.

[0228] The analysis of the beam profile of one of the reflection features may comprise determining at least one first area and at least one second area of the beam profile. The first area of the beam profile may be an area A1 and the second area of the beam profile may be an area A2. The evaluation device 10 may be configured for integrating the first area and the second area. The evaluation device 10 may be configured to derive a combined signal, in particular a quotient Q, by one or more of dividing the integrated first area and the integrated second area, dividing multiples of the integrated first area and the integrated second area, dividing linear combinations of the integrated first area and the integrated second area. The evaluation device may configured for determining at least two areas of the beam profile and/or to segment the beam profile in at least two segments comprising different areas of the beam profile, wherein overlapping of the areas may be possible as long as the areas are not congruent. For example, the evaluation device 10 may be configured for determining a plurality of areas such as two, three, four, five, or up to ten areas. The evaluation device 10 may be configured for segmenting the light spot into at least two areas of the beam profile and/or to segment the beam profile in at least two segments comprising different areas of the beam profile. The evaluation device 10 may be configured for determining for at least two of the areas an integral of the beam profile over the respective area. The evaluation device 10 may be configured for comparing at least two of the determined integrals. Specifically, the evaluation device 10 may be configured for determining at least one first area and at least one second area of the beam profile. The first area of the beam profile and the second area of the beam profile may be one or both of adjacent or overlapping regions. The first area of the beam profile and the second area of the beam profile may be not congruent in area. For example, the evaluation device 10 may be configured for dividing a sensor region of the CMOS sensor into at least two sub-regions, wherein the evaluation device may be configured for dividing the sensor region of the CMOS sensor into at least one left part and at least one right part and/or at least one upper part and at least one lower part and/or at least one inner and at least one outer part.

[0229] Additionally or alternatively, the display device 1 may comprise at least two optical sensors 4, wherein the light-sensitive areas of a first optical sensor and of a second optical sensor may be arranged such that the first optical sensor is adapted to determine the first area of the beam profile of the reflection feature and that the second optical sensor is adapted to determine the second area of the beam profile of the reflection feature. The evaluation device 10 may be adapted to integrate the first area and the second area. T

[0230] In one embodiment, A1 may correspond to a full or complete area of a feature point on the optical sensor. A2 may be a central area of the feature point on the optical sensor. The central area may be a constant value. The central area may be smaller compared to the full area of the feature point. For example, in case of a circular feature point, the central area may have a radius from 0.1 to 0.9 of a full radius of the feature point, preferably from 0.4 to 0.6 of the full radius.

[0231] The evaluation device 10 may be configured to derive the quotient Q by one or more of dividing the first area and the second area, dividing multiples of the first area and the second area, dividing linear combinations of the first area and the second area. The evaluation device 10 may be configured for deriving the quotient Q by

[00019]$Q=\frac{\int {\int }_{A1}E\left(x,y\right)\mathrm{dxdy}}{\int {\int }_{A2}E\left(x,y\right)\mathrm{dxdy}}$

wherein x and y are transversal coordinates, A1 and A2 are the first and second area of the beam profile, respectively, and E(x,y) denotes the beam profile.

[0232] The evaluation device 10 may be configured for using at least one predetermined relationship between the quotient Q and the longitudinal coordinate for determining the longitudinal coordinate. The predetermined relationship may be one or more of an empiric relationship, a semi-empiric relationship and an analytically derived relationship. The evaluation device 10 may comprise at least one data storage device for storing the predetermined relationship, such as a lookup list or a lookup table.

[0233] The evaluation device 10 may be configured for executing at least one depth-from-photon-ratio algorithm which computes distances for all reflection features with zero order and higher order.

[0234] The evaluation of the first image comprises sorting the identified reflection features with respect to brightness. The sorting may comprise assigning a sequence of the reflection features for further evaluation with respect to brightness, in particular starting with the reflection feature having maximum brightness and subsequent the reflection features with decreasing brightness. The robustness of the determining of the longitudinal coordinate z.sub.DPR can be increased if the brightest reflection features are preferred for DPR computation. This is mainly because reflection features with zero order of diffraction grating are always brighter than false features with a higher order.

[0235] The evaluation device 10 is configured for unambiguously matching of reflection features with corresponding illumination features by using the longitudinal coordinate z.sub.DPR. The longitudinal coordinate determined with the depth-from-photon-ratio technique can be used for solving the so called correspondence problem. In that way, distance information per reflection feature can be used to find the correspondence of the known laser projector grid.

[0236] The illumination feature corresponding to the reflection feature may be determined using epipolar geometry. For description of epipolar geometry reference is made, for example, to chapter 2 in X. Jiang, H. Bunke: “Dreidimensionales Computersehen” Springer, Berlin Heidelberg, 1997. Epipolar geometry may assume that an illumination image, i.e. an image of the non-distorted illumination pattern, and the first image may be images determined at different spatial positions and/or spatial orientations having a fixed distance. The distance may be a relative distance, also denoted as baseline. The illumination image may be also denoted as reference image. The evaluation device 10 may be adapted to determine an epipolar line in the reference image. The relative position of the reference image and first image may be known. For example, the relative position of the reference image and the first image may be stored within at least one storage unit of the evaluation device. The evaluation device 10 may be adapted to determine a straight line extending from a selected reflection feature of the first image to a real world feature from which it originates. Thus, the straight line may comprise possible object features corresponding to the selected reflection feature. The straight line and the baseline span an epipolar plane. As the reference image is determined at a different relative constellation from the first image, the corresponding possible object features may be imaged on a straight line, called epipolar line, in the reference image. The epipolar line may be the intersection of the epipolar plane and the reference image. Thus, a feature of the reference image corresponding to the selected feature of the first image lies on the epipolar line.

[0237] Depending on the distance to the object of the scene having reflected the illumination feature, the reflection feature corresponding to the illumination feature may be displaced within the first image. The reference image may comprise at least one displacement region in which the illumination feature corresponding to the selected reflection feature would be imaged. The displacement region may comprise only one illumination feature. The displacement region may also comprise more than one illumination feature. The displacement region may comprise an epipolar line or a section of an epipolar line. The displacement region may comprise more than one epipolar line or more sections of more than one epipolar line. The displacement region may extend along the epipolar line, orthogonal to an epipolar line, or both. The evaluation device 10 may be adapted to determine the illumination feature along the epipolar line. The evaluation device 10 may be adapted to determine the longitudinal coordinate z for the reflection feature and an error interval±ε from the combined signal Q to determine a displacement region along an epipolar line corresponding to z±ε or orthogonal to an epipolar line. The measurement uncertainty of the distance measurement using the combined signal Q may result in a displacement region in the second image which is non-circular since the measurement uncertainty may be different for different directions. Specifically, the measurement uncertainty along the epipolar line or epipolar lines may be greater than the measurement uncertainty in an orthogonal direction with respect to the epipolar line or lines. The displacement region may comprise an extend in an orthogonal direction with respect to the epipolar line or epipolar lines. The evaluation device 10 may be adapted to match the selected reflection feature with at least one illumination feature within the displacement region. The evaluation device 10 may be adapted to match the selected feature of the first image with the illumination feature within the displacement region by using at least one evaluation algorithm considering the determined longitudinal coordinate z.sub.DPR. The evaluation algorithm may be a linear scaling algorithm. The evaluation device 10 may be adapted to determine the epipolar line closest to and/or within the displacement region. The evaluation device may be adapted to determine the epipolar line closest to the image position of the reflection feature. The extent of the displacement region along the epipolar line may be larger than the extent of the displacement region orthogonal to the epipolar line. The evaluation device 10 may be adapted to determine an epipolar line before determining a corresponding illumination feature. The evaluation device 10 may determine a displacement region around the image position of each reflection feature. The evaluation device may be adapted to assign an epipolar line to each displacement region of each image position of the reflection features, such as by assigning the epipolar line closest to a displacement region and/or within a displacement region and/or closest to a displacement region along a direction orthogonal to the epipolar line. The evaluation device 10 may be adapted to determine the illumination feature corresponding to the reflection feature by determining the illumination feature closest to the assigned displacement region and/or within the assigned displacement region and/or closest to the assigned displacement region along the assigned epipolar line and/or within the assigned displacement region along the assigned epipolar line.

[0238] Additionally or alternatively, the evaluation device 10 may be configured to perform the following steps: [0239] Determining a displacement region for the image position of each reflection feature; [0240] Assigning an epipolar line to the displacement region of each reflection feature such as by assigning the epipolar line closest to a displacement region and/or within a displacement region and/or closest to a displacement region along a direction orthogonal to the epipolar line; [0241] Assigning and/or determining at least one illumination feature to each reflection feature such as by assigning the illumination feature closest to the assigned displacement region and/or within the assigned displacement region and/or closest to the assigned displacement region along the assigned epipolar line and/or within the assigned displacement region along the assigned epipolar line.

[0242] Additionally or alternatively, the evaluation device 10 may be adapted to decide between more than one epipolar line and/or illumination feature to be assigned to a reflection feature such as by comparing distances of reflection features and/or epipolar lines within the illumination image and/or by comparing error weighted distances, such as E-weighted distances of illumination features and/or epipolar lines within the illumination image and assigning the epipolar line and/or illumination feature in shorter distance and/or E-weighted distance to the illumination feature and/or reflection feature.

[0243] As outlined above, due to diffraction grating a plurality of reflection features, e.g. for each illumination feature one real feature and a plurality of false features, are generated. The matching is performed with decreasing brightness of the reflection features starting with the brightest reflection feature. No other reflection feature can be assigned to the same matched illumination feature. In due of the display artifacts, the false features which are generated are generally darker than the real features. By sorting the reflection features by brightness, brighter reflection features are preferred for the correspondence matching. If a correspondence of an illumination feature is already used, a false feature cannot be assigned to a used, i.e. matched, illumination feature.

[0244] FIGS. 2A to 2C show an embodiment of synchronizing the display 2, the illumination source 5 and the optical sensor 4.

[0245] As shown in FIG. 2A, the display device 1 may comprise the illumination source 5 comprising at least one projector 12 configured for generating at least one illumination pattern, e.g. a Laser projector module, an additional flood illumination 14 for illuminating the scene and the optical sensor 4, e.g. an IR camera module having a shutter. The display device 1 may be configured such that these components are placed in direction of propagation of the illumination light beam in front of the display 2.

[0246] The translucent display 2 may be at least one OLED display. The OLED display may have a transmission of about 25% or more. However, even embodiments of OLED display with less transmission may be possible. FIG. 2C shows an embodiment of the OLED display. Indicated with reference number 16 are potential positions for the IR camera module, projector 12 and flood illumination 14. The display 2 may have a resolution V×H with V being the vertical extension and H the height. The OLED display may comprise a plurality of pixels arranged in a matrix arrangement V×H. The OLED may update and/or refresh it's content line by line from top to bottom of the matrix. In FIG. 2C are indicated the first line, e.g. line 0, with reference number 18 and the final line V with reference number 20. The update direction is indicated with reference number 22. The control unit 8 may be configured for synchronizing the display 2, projector 12, flood illumination 14 and optical sensor 4. Elements of control unit 8 are shown in FIG. 2, e.g. as part of SoC 26 and/or as elements of the display 2, optical sensor 4 and illumination sources 12,14. The display 2, in particular a display driver, may be configured for emitting at least one signal indicating that an update and/or a refresh wraps around from the final line 20 to the first line 18. The display driver may be part of the control unit. For example, when the update and/or the refresh wraps around from the final line 20 to the first line 18, a Vertical SYNC (VSYNC) signal, also denoted as display VSYNC 24, may be emitted by the display 2.

[0247] The emission of light through the OLED display may be timed shortly before the content get's updated and/or refreshed, in particular overwritten. This may allow minimizing visible distortion. The optical sensor 4 may be synchronized with the projector 12 and flood illumination 14. The optical sensor 4 may be active, i.e. in a mode for capturing images and/or detecting light, during the illumination. For example, the synchronization of optical sensor 4 and illumination source 5 may be realized as shown in FIG. 2A.

[0248] As shown in FIG. 2A, the control unit may comprise a system on a chip (SoC) 26. The SoC 26 may comprise a display interface 28. The SoC 26 may comprise at least one application programming interface (API) 30 connected to at least one application 32. The SoC 26 may further comprise at least one image signal processor (ISP) 34. The optical sensor 4 may be connected to the SoC 26, in particular to the ISP 34 and/or API 30 via connection 35. The connection 35 may be configured for one or more of power control, providing a clock signal (CLK), transfer of image signals. Additionally or alternatively, the connection 35 may be embodied as Inter-Integrated Circuit (I2C). Additionally or alternatively, the connection may be embodied as image data interface such as MIP.

[0249] The application 32 may request 40 illumination by one or more of the illumination sources 12, 14. The SoC 26, via API 30, may power the optical sensor 4 via connection 35. The optical sensor 4 may emit a VSYNC signal, also denoted as camera VSYNC 36 to the SoC 26 and a strobe signal 38 to the illumination sources 12, 14. The SoC 26, via API 30, may issue in response to the camera VSYNC 36 trigger signals 41, 42 to the illumination sources 12, 14 for activating the illumination sources, respectively. In case the respective trigger signal 41, 42 and the strobe signal 38 are received by the respective illumination source 12, 14, in particular by an AND logical gate, a respective driver 43 of the illumination source 12, 14 drives the illumination. The signals of the optical sensor 4 may be transferred to the SoC 26 e.g. to API 30 and ISP 34 by connection 35, and may be provided 44, e.g. for further evaluation, to the application 32, e.g. together with meta data and the like.

[0250] As further shown in FIG. 2A, the optical sensor 4 and the display 2 may be synchronized. The display 2 may be connected to the display interface 28 via at least one Display Serial Interface (DSI) 46, in particular MIPI Display Serial Interface (MIPI DSI®). The display interface 28 may transfer at least one SW-signal 48 to the optical sensor 4. The display 2 may be operated in two operation modes, i.e. “video mode” or “command mode”. In the video mode the VSYNC signal may be issued from the display 2. In the command mode the VSYNC signal may be generated and issued by the SoC, in particular may be generated by software. Thus, the VSNC signal of the display 2 may be issued by the display itself and/or by SoC 26.

[0251] The display device 1 may be configured for passing the display VSYNC 24 to the optical sensor 4 as trigger signal to synchronize the display VSYNC 24 to the end of a camera frame exposure. Depending on the optical sensor's 4 trigger requirements the display VSYNC 24 may be adapted, in particular conditioned, before passing it to the optical sensor 4 to full fill the requirements. For example, the frequency of the display VSYNC 24 may be conditioned to half the frequency.

[0252] FIG. 2B shows the development of the display VSYNC 24, strobe signal 38, camera VSYNC, and trigger signals 41 and 42 as a function of time, in particular in 1/frames per second (FPS). The exposure of the camera is shown to happen directly before display VSYNC 24, i.e. directly before refresh of the first lines where the transparent areas of positions 16 are located.

LIST OF REFERENCE NUMBERS

[0253] 1 display device [0254] 2 translucent display [0255] 4 optical sensor [0256] 5 illumination source [0257] 6 black area [0258] 8 control unit [0259] 10 evaluation device [0260] 12 projector [0261] 14 flood illumination [0262] 16 positions [0263] 18 line 0 [0264] 20 line V [0265] 24 display VSYNC [0266] 26 SoC [0267] 28 display interface [0268] 30 API [0269] 32 application [0270] 34 ISP [0271] 35 connection [0272] 36 camera VSYNC [0273] 38 strobe signal [0274] 40 request [0275] 41 trigger signal [0276] 42 trigger signal [0277] 44 providing [0278] 46 DSI [0279] 48 SW signal