DEPTH MAP GENERATOR

Abstract

The invention describes a depth map generator comprising an array comprising a plurality of individually addressable array elements, wherein an array element comprises a semiconductor emitter or a reflector arranged to reflect light emitted by a semiconductor emitter; a driver realised to switch an array element according to a predefined illumination pattern; a number of image sensors, wherein an image sensor is arranged to detect light reflected from a scene irradiated by the irradiation pattern; and a processing unit realised to compute a depth map of the scene on the basis of a light pattern detected by an image sensor. The invention further describes an irradiation arrangement for use in such a depth map generator; a method of generating a depth map of a scene; and a device comprising such a depth map generator.

Claims

1. A depth map generator comprising: an array comprising a plurality of individually addressable array elements, wherein each of the plurality of individually addressable array elements comprises at least one of a semiconductor emitter and a reflector arranged to reflect light emitted by a semiconductor emitter; a driver realised to switch the plurality of individually addressable array elements according to a predefined illumination pattern; a number of image sensors, wherein the number of image sensors are arranged to detect light reflected from a scene illuminated by the predefined illumination pattern; and a processing unit realised to compute a depth map of the scene on the basis of a light pattern detected by the number of image sensors.

2. The depth map generator according to claim 1, wherein a semiconductor emitter is an infrared LED.

3. The depth map generator according to claim 1, wherein the predefined illumination pattern defines a number of active regions such that each of the number of active region is bounded by at least two inactive regions.

4. The depth map generator according to claim 1, wherein the array comprises a plurality of individually addressable semiconductor emitters, and further comprises an primary optical element arranged to shape the light emitted by the semiconductor emitters of the array.

5. The depth map generator according to claim 1, wherein the array comprises a plurality of individually addressable micro-mirrors.

6. The depth map generator according to claim 1, wherein the array comprises at least 100 individually addressable array elements.

7. The depth map generator according to claim 1, wherein each of the number of image sensors comprises an array of photosensors, wherein the array of photosensors are sensitive to the wavelength of light originating from the array.

8. The depth map generator according to claim 1, comprising two of the number of image sensors arranged on opposite sides of the array.

9. An illumination arrangement for use in a depth map generator, comprising: an array of individually addressable array elements, wherein each of the individually addressable array elements comprises at least one of a semiconductor emitter and a reflector arranged to reflect light emitted by a semiconductor emitter; a driver realised to switch the plurality of individually addressable array elements according to a predefined illumination pattern; a number of image sensor arrays, wherein the number of image sensor arrays are arranged to detect light reflected from a scene illuminated by the predetermined illumination pattern; a processing unit realised to compute a depth map of the scene on the basis of a light pattern detected by the number of image sensors: a first interface for connection to the driver of the depth map generator; and a second interface for connection to the processing unit of the depth map generator.

10. A method of generating a depth map of a scene, the method comprises: arranging a depth map generator in front of the scene, the depth map generating including an array comprising a plurality of individually addressable array elements, wherein each of the plurality of individually addressable array elements comprises at least one of a semiconductor emitter and a reflector arranged to reflect light emitted by a semiconductor emitter; choosing an illumination pattern to illuminate the scene; switching, via a driver, the plurality of individually addressable array elements of the array according to the illumination pattern; and computing a depth map of the scene on the basis of the reflected light detected by a plurality of image sensor arrays.

11. The method according to claim 10, wherein the scene is illuminated by a sequence of the illumination patterns.

12. The method according to claim 11, wherein the sequence of the illumination patterns are chosen to successively illuminate the scene in at least one of a vertical direction and a horizontal direction.

13. The method according to claim 11, wherein the sequence of the illumination patterns is the inverse of a preceding illumination pattern.

14. The method according to claim 10, comprising a step of adjusting the illumination pattern on the basis of a computed depth map.

15. The depth map generator according to claim 1 for use in a device, wherein the device is realised as at least one of a mobile phone, a handheld electronic device, a portable electronic device, a machine vision device, a robot, a drone, and an automotive vision device.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0028] FIG. 1 illustrates the underlying principle of the invention in a first embodiment of the inventive illumination arrangement;

[0029] FIG. 2 shows various irradiation patterns;

[0030] FIG. 3 shows a calibration pattern;

[0031] FIG. 4 shows a second embodiment of the inventive illumination arrangement;

[0032] FIG. 5 shows an embodiment of the inventive depth map generator;

[0033] FIG. 6 shows a third embodiment of the inventive illumination arrangement.

[0034] In the drawings, like numbers refer to like objects throughout. Objects in the diagrams are not necessarily drawn to scale.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0035] FIG. 1 illustrates the underlying principle of the invention. The diagram shows an emitter array E and an image sensor S of a camera at some distance from a scene 3. The scene 3 is simply illustrated as a line, but can be a face, a scene in front of an automobile, etc. In this embodiment, the emitter array E comprises a relatively large array of infrared light-emitting diodes or VCSELs, arranged for example in a 1010 array. The emitter array E is arranged at a distance HE behind a lens FE, and the image sensor S is arranged at a distance HS behind a lens FS. For the sake of simplicity, the imaging lenses FE, FS are shown in a common plane, but such an arrangement is not strictly necessary. The centre of each lens FE, FS corresponds to the centre of the respective array E, S. The distance B is measured from the centre of the first lens FE to the centre of the camera lens FS. The field of view of the emitter array should exceed the camera's field of view. The image sensor S can comprise an array of infrared-sensitive photodiodes (photosensors), for example at least ten times the number of emitter pixels. This ensures that the size of a detected disparity is a fraction of the line pitch of the on-scene irradiation pattern.

[0036] The array of emitters irradiates a scene in its field of view, i.e. the emitter array casts an irradiation pattern into its field of view. The image sensor captures an image of the irradiated scene. The diagram shows the path of a light ray L from an emitter pixel EP to point P in the scene, from which it is reflected as light ray L and detected by image sensor pixel(s) SP. In simplified ray optics, rays L, L pass unchanged through the centres of the respective lenses FE, FS. Therefore, each pixel in the irradiated scene originates from a pixel of the emitter array, so that an emitter pixel EP subtends an angle 2 to its corresponding scene pixel. The scene pixels are imaged by the image sensor S, so that a scene pixel P subtends an angle 1 to its imaged pixel SP. A right angle is subtended between the illuminated pixel P in the scene and a point V in the plane of the lenses FE, FS. Recognising that

[00001]$\begin{array}{cc}B=V_S+V_E& \left(1\right)\\ \tan .Math..Math..Math..Math.2=\frac{H}{V_E}& \left(2\right)\\ \tan .Math..Math..Math..Math.1=\frac{H}{V_S}& \left(3\right)\end{array}$

allows H to be expressed as:

[00002]$\begin{array}{cc}H=B.Math.\frac{\tan .Math..Math..Math..Math.1.Math..Math.\tan .Math..Math..Math..Math.2}{\tan .Math..Math..Math..Math.1+\tan .Math..Math..Math..Math.2}& \left(4\right)\end{array}$

The principle of similar triangles can then be applied in determining the distance to point P as follows: the position of the emitting pixel E.sub.P is known (from its position in the irradiation pattern), so that

[00003]$\begin{array}{cc}\tan .Math..Math..Math..Math.2=\frac{H_E}{B_E}& \left(5\right)\end{array}$

in which H.sub.E is the distance to lens F.sub.E, and B.sub.E is the distance from the array centre to the emitting pixel E.sub.P. Similarly, the position B.sub.S of the image sensor pixel E.sub.S can be determined (by comparing the sensed image to the irradiation pattern), so that

[00004]$\begin{array}{cc}\tan .Math..Math..Math..Math.1=\frac{H_S}{B_S}& \left(6\right)\end{array}$

[0037] These values can be substituted into equation (4) to solve for H. These computations are repeated for each pixel of the irradiation pattern.

[0038] As explained above, the invention is based on the insight that the distance to an illuminated point on a scene can be established by identifying the position of the corresponding detecting photosensor(s) in a sensor array. An offset between the expected position of a detected pixel to the actual position of the detected pixel is used to calculate the distance to the illuminated point. This works best with a black & white irradiation pattern, in which white represents emitters that are on. FIG. 2 shows various possible irradiation patterns as applied to an array of individually addressable array elements: a horizontal stripe pattern X1, a vertical stripe pattern X2, a chessboard pattern X3, and a random dot pattern X4. In each case, the emitter array pixels EON that are on are neighboured by two or more array pixels EOFF that are off. Of course, rows or columns of a striped pattern may comprise be more than one pixel wide. Such illumination patterns make it relatively easy to identify a specific pixel in the sensor array, even if the corresponding light rays were deflected from their expected position by the surface contour of the scene.

[0039] FIG. 1 corresponds to a first embodiment of the inventive depth map generator 1 that comprises a single camera with an image sensor S and an emitter array E. As explained above, light reflected from an irradiated point will be detected by photodiodes at some point on the image sensor S. The distance H to the irradiated point (i.e. the depth of point P in the scene 3) can be calculated as explained above. Of course, the irradiation pattern can comprise n active emitters, resulting in corresponding images on each of the sensors S1, S2, so that n distances to the n irradiated points are calculated. In this way, a depth map can be deduced for all irradiated points in the scene.

[0040] It shall be noted that the diagram is not to scale. Generally, the distance between emitter array E and image sensor S may be relatively small, for example from a few tens of millimetres to a few centimetres, while the distance to a scene can be in the order of 0.25 m to 3 m for consumer products such as mobile devices, and may exceed 3 m in the case of applications such as automotive imaging systems.

[0041] To correctly interpret an imaged irradiation pattern, the system carries out a calibration procedure to establish a relationship between the emitter array and any image sensor. A calibration procedure can be used to compensate for unavoidable inaccuracies in the arrangement of emitter array and image sensors. FIG. 3 shows a calibration pattern XC in which a set of emitters at the centre of the emitter array E is switched on, along with a set of emitters at each outer corner of the emitter array. Each set of emitters may comprise a single emitter or a sub-array, for example four emitters in a 22 sub-array or a 33 sub-array, etc. These five points fix the outer corners and centre point of the emitter array. In a calibration procedure, the device containing the depth map generator is arranged at a distance in front of a suitable surface such as a white wall and the calibration pattern XC is cast onto the surface and imaged by the sensor. Knowing the shape of the calibration pattern XC, and assuming that the surface is flat and that the imaged pattern is free of distortion, the system can establish an absolute reference between pattern position and distance of objects in the scene.

[0042] FIG. 4 shows a second embodiment of the inventive depth map generator 1. Again, the diagram is not to scale. Here, the depth map generator 1 comprises an emitter array E and two cameras, represented here by two image sensors S1, S2, each with an imaging lens FS1, FS2. The two image sensors S1, S2 are arranged at a distance B from each other. This distance B may be measured between the midpoints of the sensor arrays S1, S2, i.e. between the midpoints of the corresponding imaging lenses FS1, FS2. The lenses FS1, FS2 are arranged at distances HS1, HS2 from the image sensors S1, S2. Light from the emitter array E is shaped by its imaging lens FE so that the emitter array E can illuminate a certain field of view (that is preferably large than each camera field of view to be able to create depth maps as large as the FOV of each camera). Each camera images the scene 3 onto its sensor S1, S2. The diagram shows the case for a single active emitter EP illuminating a point P on the object or scene 3. Light is reflected from that point P in the scene 3 and is detected by certain photosensors of each image sensor S1, S2. The reflected light will fall at different positions on each of the two image sensors S1, S2. In other words, the image pixel positions of the detecting photosensor(s) S1P of sensor S1 will be different from the image pixel positions of the detecting photosensor(s) S2P of sensor S2.

[0043] The distance H to the illuminated point (i.e. the depth of point P in the scene 3) can be calculated as explained above, recognising that a scene pixel P subtends an angle 1 to its imaged pixel S1P in the first image sensor S1, and the same scene pixel P subtends an angle 2 to its imaged pixel S2P in the second image sensor S2.

[0044] FIG. 5 is a block diagram of an embodiment of the inventive depth map generator 1. The relevant units and modules of the depth map generator 1 are shown, namely an emitter matrix E to emit light R onto a scene; a driver 11 to control the emitters of the emitter matrix according to an irradiation pattern X1, . . . , X4; an image sensor S to detect reflected light R from the scene; a processor 12 configured to computer a depth map DM using e data from the image sensor S and known system parameters, and a controller 13 to control the driver 11 and to communicate with other components of a device that incorporates this depth map generator 1. An irradiation arrangement comprising the emitter matrix E and one or more image sensors S can be provided on a PCB with suitable connectors 110, 120 for connecting to the driver 11 and processor 12.

[0045] FIG. 6 shows a third embodiment of the inventive illumination arrangement. In the same way as the embodiment of FIG. 4, an array of individually addressable array elements is arranged between two image sensors S1, S2. In this embodiment, an array R of micro-mirrors is arranged to deflect light L originating from an infrared semiconductor emitter EX such as a VCSEL or LED. The diagram illustratesgreatly simplifieda row of reflectors in such an array R, and indicates one reflector RP tilted (in response to a signal from the driver) to deflect the light L onto the scene 3. In such a realisation, the semiconductor emitter EX is always on during the depth map generation procedure, while the micro-mirrors are rapidly tilted to either cast the light onto the scene 3 (such micro-mirrors act as active array pixels), or to deflect the light such that it does not arrive at the scene 3 (such micro-mirrors act as inactive array pixels). The absence of any refracting lens element between the semiconductor emitter EX and the scene 3 or target means that there is essentially negligible light loss, so that the accuracy of the procedure is favourably high. The method explained in FIG. 4 above is used to compute the depth map of the scene 3.

[0046] Although the present invention has been disclosed in the form of preferred embodiments and variations thereon, it will be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention. For example, embodiments of the inventive depth map generator may comprise two emitter arrays and a single camera; an assembly in which two cameras are arranged in line and with an emitter array perpendicularly offset from that line (this arrangement permits the use of alternating irradiation patterns of vertical lines and horizontal lines); an active emitter array with an active lens to allow sub emitter-lens-pitch shifts of the irradiation pattern to increase depth map resolution; two emitters with different emission wavelengths, visible line pattern, etc.

[0047] For the sake of clarity, it is to be understood that the use of a or an throughout this application does not exclude a plurality, and comprising does not exclude other steps or elements. The mention of a unit or a module does not preclude the use of more than one unit or module.

REFERENCE SIGNS

[0048] depth map generator 1 [0049] driver 11 [0050] driver interface 110 [0051] processing unit 12 [0052] processor interface 120 [0053] scene 3 [0054] emitter array E, R [0055] imaging lens F.sub.E, F.sub.S [0056] semiconductor emitter E.sub.P [0057] semiconductor emitter E.sub.X [0058] micro-mirror R.sub.P [0059] active array region E.sub.ON [0060] inactive array region E.sub.OFF [0061] image sensor array S, S1, S2 [0062] photosensor S.sub.P [0063] point P [0064] illumination pattern X1, . . . , X4, XC [0065] emitted light L [0066] reflected light L [0067] distance to illuminated point H [0068] distance to lens H.sub.E, H.sub.S, H.sub.S1, H.sub.S2 [0069] array separation B [0070] distance to pixel B.sub.E, B.sub.S [0071] angle 1, 2 [0072] depth map DM