Method of obtaining a digital image

Abstract

Devices, systems and methods are described. A method includes driving a first light source to generate a visible light pulse having a first intensity for a first duration to illuminate a scene. An image sensor is used to record a visible-spectrum image of the scene illuminated by the first light pulse. A second light source is driven to generate a second light pulse having a second intensity that exceeds the first intensity by at least 100% for a second duration that is at most 10% of the first duration to illuminate the scene during the first duration. An image sensor is used to record a monochrome image of the scene illuminated by the second light pulse. Image processing is performed on a visible-spectrum image and the monochrome image to obtain the digital image of the scene.

Claims

1. A method of obtaining a digital image of a scene, the method comprising: driving a first light source using a first drive current having a first current density of about 1.0 A/mm2 to generate a visible light pulse having a first intensity for a first duration to illuminate the scene, the first duration being at least a sensor integration time; using an image sensor to record a visible-spectrum image of the scene illuminated by the first light pulse; driving a second light source using a second drive current having a second current density of more than about 1.5 A/mm2 to generate a second light pulse having a second intensity that exceeds the first intensity by at least about 100% for a second duration that is at most about 10% of the first duration to illuminate the scene during the first duration, the second duration being at most about 5.0 ms; using an image sensor to record a monochrome image of the scene illuminated by the second light pulse, the image sensor used to record the monochrome image being the same or different than the image sensor used to record the visible-spectrum image; and performing image processing on the visible-spectrum image and the monochrome image to obtain the digital image of the scene.

2. The method according to claim 1, wherein the image processing is performed on a sequence of visible-spectrum images and monochrome images recorded with respective light pulses.

3. The method according to claim 1, wherein the image processing comprises identifying a common image region corresponding to an imaged subject present in the visible-spectrum image and the monochrome image.

4. The method according to claim 1, further comprising identifying a motion blur image region in the visible-spectrum image.

5. The method according to claim 4, further comprising correcting color values in at least the motion blur image region.

6. A system comprising: a visible-spectrum light source configured to emit visible-spectrum light in a visible spectrum; a second light source that is configured to emit at least one type of light selected from the visible-spectrum light in the visible-spectrum and infrared-spectrum light in an infrared-spectrum; a driver configured to provide a first drive current having a first current density of about 1.0 A/mm2 to a first light source to generate a first light pulse having a first light intensity over a first duration and to provide a second drive current having a second current density of more than about 1.5 A/mm2 to the second light source to generate a second light pulse having a second light intensity that exceeds the first light intensity by at least about 100% over a second duration that is at most about 10% of the first duration, the first light pulse being a visible-spectrum light pulse, the first duration being at least a sensor integration time, the second duration being at most about 5.0 MS; at least one image sensor having pixels that are sensitive to at least the visible-spectrum light; and an image processing unit configured to combine information from a visible-spectrum image and a monochrome image to obtain a digital image of a scene.

7. The system according to claim 6, wherein the second light source comprises one or more light-emitting diodes configured to emit the infrared-spectrum light.

8. The system according to claim 6, wherein the second light source comprises one or more vertical cavity surface-emitting laser diodes.

9. The system according to claim 6, wherein the at least one image sensor comprises a single image sensor that comprises pixels that are sensitive to the visible-spectrum and pixels that are sensitive to the infrared-spectrum.

10. The system according to claim 9, wherein the single image sensor further comprises a filter that is at least partially transmissive to the infrared-spectrum.

11. The system according to claim 6, wherein the at least one image sensor comprises two image sensors, each comprising only pixels that are sensitive to the visible-spectrum.

12. The system according to claim 6, wherein the at least one image sensor comprises two image sensors, and a first one of the two image sensors comprises pixels that are sensitive to the visible-spectrum, and a second one of the two image sensors comprises pixels that are sensitive to the infrared-spectrum.

13. The system according to claim 12, wherein the first one of the two image sensors comprises a filter that is transmissive only in the visible-spectrum, and the second one of the two image sensors comprises a filter that is transmissive only in the infrared-spectrum.

14. The system according to claim 6, wherein the sensor integration time is approximately 50 ms.

15. The system according to claim 6, wherein the second duration is at most about 1.0 ms.

16. A wireless device comprising: a camera flash comprising: a visible-spectrum light source configured to emit visible-spectrum light in a visible spectrum, a second light source that is configured to emit at least one type of light selected from the visible-spectrum light in the visible-spectrum and infrared-spectrum light in an infrared-spectrum, and a driver configured to provide a first drive current having a first current density of about 1.0 A/mm2 to a first light source to generate a first light pulse having a first light intensity over a first duration and to provide a second drive current having a second current density of more than about 1.5 A/mm2 to the second light source to generate a second light pulse having a second light intensity that exceeds the first light intensity by at least about 100% over a second duration that is at most about 10% of the first duration, the first light pulse being a visible-spectrum light pulse, the first duration being at least a sensor integration time, the second duration being at most about 5.0 ms; a camera comprising at least one image sensor having pixels that are sensitive to at least the visible-spectrum light; and an image processing unit configured to combine information from a visible-spectrum image and a monochrome image to obtain a digital image of a scene.

17. The device of claim 16, wherein the device is a smartphone.

18. The device of claim 16, wherein at least one of the first light source and the second light source comprises one or more light-emitting diodes.

19. The device of claim 16, wherein: the pixels comprise visible-spectrum pixels that are sensitive to the visible-spectrum light and infrared sensitive pixels that are sensitive to the infrared-spectrum, the infrared sensitive pixels are about 25% of a total number of pixels in the at least one image sensor, and the infrared sensitive pixels are distributed between the visible-spectrum pixels.

Description

BRIEF DESCRIPTION OF THE DRAWING(S)

(1) FIG. 1 is a simplified block diagram to illustrate the principle of the invention;

(2) FIG. 2 shows a main flash and a motion-freeze flash generated during the inventive method;

(3) FIG. 3 shows exemplary curves of intensity vs. current density for a visible-spectrum semiconductor light source and an infrared-emitting semiconductor light source;

(4) FIG. 4 shows an embodiment of the inventive imaging arrangement;

(5) FIG. 5 shows a different embodiment of the inventive imaging arrangement;

(6) FIG. 6 shows a different embodiment of the inventive imaging arrangement;

(7) FIG. 7 shows an exemplary set of sensitivity peaks vs. wavelength.

(8) In the drawings, like numbers refer to like objects throughout. Objects in the diagrams are not necessarily drawn to scale.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT(S)

(9) FIG. 1 is a simplified representation of a poorly-lit scene D with a moving subject B, and two images M.sub.RGB, M.sub.mono obtained using the inventive method. A first image M.sub.RGB is a colour image M.sub.RGB obtained during the visible-spectrum light pulse. This colour image M.sub.RGB will record a blurred version B.sub.blur of the moving object B, since the RGB sensor and the visible-spectrum flash are realised to increase the integration time (exposure time) as necessary to obtain sufficient information. In the inventive method, a second, monochrome image M.sub.mono is obtained during the significantly shorter light pulse. This monochrome or “motion-freeze” image M.sub.mono is essentially a monochrome snapshot that is devoid of colour information but which is better at capturing the outline or shape of any moving objects. In this example, the body B moving through the scene D is captured as a comparatively sharp shape B.sub.mf in the infrared image M.sub.mono.

(10) The second light source could be realised as a visible-spectrum LED that can be overdriven over a brief period, i.e. to provide a high-intensity light pulse over a very short time. However, in this exemplary embodiment, it may be assumed that the second light source 12 is an infrared-emitting light source, and that the “monochrome” light pulse comprises wavelengths in the infrared range.

(11) In an image processing step 140, the RGB values of the image pixels of the colour image M.sub.RGB are adjusted by a suitable luminance factor to improve the image signal-to-noise ratio. The algorithm comprises several main steps including the identification of blurred objects B.sub.blur in the colour image (only one exemplary blurred object is shown for the sake of clarity) using information from the monochrome image M.sub.mono, performing brightness correction over the pixels in that image region B.sub.blur, and performing colour adjustment in any corrected image region.

(12) An initial step may be to consider the brightness of the image. Each image pixel X has a red (R.sub.X), a green (G.sub.X) and a blue (B.sub.X) value. In one approach, a “brightness image” may simply comprise the G.sub.X values of the pixels of the RGB image. Alternatively, the red, green and blue colour values of each image pixel X of the colour image M.sub.RGB can be adjusted as follows
R.sub.X.fwdarw.R.sub.X−Y.sub.X  (1)
G.sub.X.fwdarw.Y.sub.X  (2)
B.sub.X.fwdarw.B.sub.X−Y.sub.X  (3)
where Y.sub.X is a “grey value” of luminance for that pixel and is computed using the equation:
Y.sub.X=0.2R.sub.X+0.7G.sub.X+0.1B.sub.X  (4)

(13) The result of this adjustment on each image pixel is a monochrome corrected brightness image (also referred to as “RGB-Y image” or “G image”) described above. These adjustments can be beneficial especially in the case of an embodiment using only a single image sensor with an IR notch filter. In such an embodiment, the red, green and blue sensor pixels will also—to a small extent—integrate the infrared light arriving at the sensor. For colour reproduction, the red, green and blue components can be corrected using information from the IR image. In the above equations (1) and (3), such correction is performed by subtracting the IR signal.

(14) The brightness image and the IR image are then analysed using pattern recognition or edge detection techniques. Where an edge is identified in the brightness image as well as the IR image M.sub.mono, no correction is needed in the corresponding region of the RGB image M.sub.RGB. Where an edge is blurred in the brightness image but sharp in the IR image M.sub.mono, the RGB image M.sub.RGB will be corrected using information from the IR image M.sub.mono. The grey value Y of equation (2) is therefore further adjusted for each pixel as follows
Y′.sub.X=α.Math.Y.sub.X+(1−α).Math.β.Math.IR.sub.X  (5)
where α is a pixel-dependent matching parameter, δ is a secondary matching factor to correct for the different reflectance of infrared light compared to visible light, and IR.sub.X is the appropriate pixel value in the motion-freeze image M.sub.mono. The matching parameter α will be 1.0 for a non-blurred regions of the image and can be as low as 0.0 in a motion-blur region. In an alternative approach, the matching parameter α could be varied between 0.5 and 0.0 to use the IR information for the complete image, if e.g. noise conditions make this preferable.

(15) For colour reconstruction, information from the RGB image M.sub.RGB will remain unchanged for any non-blurred image regions, i.e. α=1.0 for each pixel in a non-blurred region (see equations (6)-(8) below). For motion-blur regions B.sub.blur, several image processing techniques can be applied. For example, the difference in size of a moving object as recorded in the RGB image M.sub.RGB compared to its size in the IR image M.sub.mono will give an indication of the speed and direction of the movement and thereby give an indication of the “mixing ratio” between moving object B and background. For an object B moving across the scene D, the colour in a motion blur region B.sub.blur is essentially a blend of object colour and background colour, and more background is “mixed in” when the object B is moving quickly. Similarly, if the object B is moving towards the camera, the flash light will become more relevant and will highlight the object colour more than the background colour. The colour of any pixel X in blurred area of the image can be reconstructed from RGB values R.sub.BG, G.sub.BG, B.sub.BG of neighbouring background pixels (i.e. background pixels that are adjacent to the blurred image region), and image regions of the moving object B can be assigned the colour of the blurred region, corrected—for the mixing in of a suitable background colour. For the image pixels of the blurred object B.sub.blur, colour is preferably reconstructed within regions corresponding to the moving object B identified in the “common” region R, as well as in regions R.sub.blur that actually belong to the background.

(16) Motion towards or away from the camera can be identified by observing size and brightness changes over the object's trajectory or path, for example its brightness will increase if it is moving towards the camera. Object recognition can also be performed to assist in assigning a realistic colour to any corrected image region.

(17) Within the object, i.e. within edges of the shape B.sub.mf detected in the IR motion freeze image M.sub.mono:
R′.sub.X.fwdarw.αR.sub.X+(1−α)(R.sub.X−γR.sub.BG)  (6)
G′.sub.X.fwdarw.αG.sub.X+(1−α)(G.sub.X−γG.sub.BG)  (7)
B′.sub.X.fwdarw.αB.sub.X+(1−α)(B.sub.X−γB.sub.BG)  (8)

(18) The term γ expresses the above-mentioned mixing ratio between the moving object and the background. In any blurred region of the image that is outside of the object, such interpolated/extrapolated RGB values can also be used to reconstruct the background colour. The background color for a pixel in a motion blur region R.sub.blur (and to the side of the object) can be assumed to be the same as the colour of an adjacent pixel outside the motion blur region R.sub.blur. This simple assumption may suffice, or processing steps may be included to adjust such pixel colours to take into account any variations in the background. A resulting pixel value of the optimized image M.sub.opt is the set of equations (6)-(8).

(19) The optimized image M.sub.opt will therefore show the object Box to a satisfactory degree of sharpness and colour, and will include image regions R.sub.edit (derived from motion blur R.sub.blur regions as shown in colour image M.sub.RGB) that have been “edited” or corrected, for example to reconstruct the background.

(20) An alternative approach to optimising the background is to combine information from images taken before or after the actual photo event carried out by the user. Such techniques are already used in smartphones to improve image quality.

(21) In general, the form of the imaged object may be slightly elongated in the motion-freeze image, so that the shape of the object in the final image M.sub.opt may also be slightly elongated. Additional processing steps may be applied on an image sequence, if desired, to determine the object trajectory and speed from which the object's “real” shape may be deduced.

(22) In a prior art imaging arrangement which uses an RGB sensor and a visible-spectrum flash F.sub.RGB that are optimised for imaging well-lit scenes, a moving object in a dimly-lit scene will result in a poor-quality image such as colour image M.sub.RGB, in which a moving object B is imaged as a distorted and blurred shape B.sub.blur.

(23) FIG. 2 is a graph of optical power (Y-axis, in [W]) against time (X-axis, in milliseconds) for a main flash F.sub.RGB and a motion-freeze flash F.sub.mono generated in an imaging sequence using an embodiment of the inventive imaging arrangement, and used to capture the images M.sub.RGB, M.sub.mono described in FIG. 1. The main flash F.sub.RGB extends in this exemplary embodiment over a relatively long duration t.sub.RGB, over at least the relevant image sensor integration time t.sub.int, so that the image sensor can record the incoming light to generate a digital image. The motion-freeze flash F.sub.mono is only generated for a very short time t.sub.mono, but has a significantly higher optical power. In practice, the integration time t.sub.int may be shorter than the flash duration t.sub.RGB. In a rolling shutter application, the duration t.sub.RGB may be twice the frame length (not necessarily the same as the sensor integration time).

(24) FIG. 3 is a graph of optical power (Y-axis, in [W]) against current density (X-axis, in [A/mm.sup.2]) for a visible-spectrum semiconductor light source (curve 110) and an infrared-emitting semiconductor light source (curve 120). The diagram shows that the light output of a blue LED (acting as pump for a visible light LED device) saturates at high current densities, i.e. the light output cannot increase beyond a certain upper limit. In contrast, the light output of the infrared-emitting semiconductor light source saturates at much higher current densities. This makes it possible to generate the significantly higher motion-freeze flash of FIG. 2.

(25) FIG. 4 shows an embodiment of the inventive imaging arrangement 1, incorporated in a smartphone 4. Here, the imaging arrangement 1 comprises a single camera C and therefore also a single image sensor S. The sensor S is represented in a simplified manner by its exemplary arrangement of sensor pixels P.sub.R, P.sub.G, P.sub.B, P.sub.IR as indicated at the side of the diagram. In this embodiment, the image sensor S is a four-channel sensor, comprising red-sensitive sensor pixels P.sub.R, green-sensitive sensor pixels P.sub.G, blue-sensitive sensor pixels P.sub.B, and also infrared-sensitive sensor pixels P.sub.IR. Here, green-sensitive sensor pixels comprise about 50% of the total, infrared-sensitive sensor pixels comprise about 25% of the total; while red-sensitive and blue-sensitive sensor pixels each comprise about 12.5%.

(26) The image sensor is equipped with a filter, for example by depositing a four-channel colour filter mosaic on the pixels of the image sensor die. The IR filter for pixels that are to respond to IR wavelengths may be formed by depositing red, green and blue filter layers onto those pixels P.sub.IR.

(27) The smartphone 4 also comprises a white LED light source 11 and an infrared light source 12. To capture a colour image M.sub.RGB, a driver 13 causes the white LED light source 11 to generate a visible-spectrum light pulse F.sub.RGB (the main flash) as explained in FIG. 2. During the sensor integration time, visible light reflected from a scene arrives at the sensor S and reaches the RGB sensor pixels P.sub.R, P.sub.G, P.sub.B. The infrared-sensitive sensor pixels P.sub.IR will be unaffected by the visible light reflected from the scene.

(28) To capture a motion-freeze image M.sub.mono, the driver 13 causes the infrared light source 12 to generate an infrared-spectrum light pulse F.sub.mono (the motion-freeze flash) as explained in FIG. 2. The infrared light reflected from a scene arrives at the sensor S and reaches the infrared-sensitive sensor pixels P.sub.IR. Because the RGB sensor pixels P.sub.R, P.sub.G, P.sub.B will also detect infrared light, this can be corrected for by taking the difference between the RGB signal and the IR signal.

(29) The colour image M.sub.RGB and the motion-freeze image M.sub.mono are then forwarded to an image processing unit 14 which will use the information provided by the images M.sub.RGB, M.sub.mono to obtain the optimized image M.sub.opt described in FIG. 1, i.e. an image that is free of motion blur, or at least with reduced motion blur.

(30) FIG. 5 shows another embodiment of the inventive imaging arrangement 1, again incorporated in a smartphone 4. Here, the imaging arrangement 1 comprises two cameras C1, C2 and therefore also two image sensors S1, S2. As described in FIG. 4, the smartphone 4 also comprises a white LED light source 11 and an infrared light source 12. Here also, each sensor S1, S2 is represented in a simplified manner by its exemplary arrangement of sensor pixels P.sub.R, P.sub.G, P.sub.B as indicated at the side of the diagram. In this embodiment, each image sensor S1, S2 is a three-channel sensor, comprising red-sensitive sensor pixels P.sub.R, green-sensitive sensor pixels P.sub.G, and blue-sensitive sensor pixels P.sub.B. Here, green-sensitive sensor pixels comprise about 50% of the total, while red-sensitive and blue-sensitive sensor pixels each comprise about 25%. An exemplary arrangement of sensor pixels of the image sensors S1, S2 is shown in the enlarged portion of the diagram.

(31) One image sensor S1 is used to capture a colour image, and the other image sensor S2 is used to capture a monochrome infrared image. To this end, the first image sensor S1 is equipped with a colour filter mosaic applied to the image sensor die so that each sensor pixel is covered by a corresponding R, G, B filter as appropriate. The second image sensor S2 is equipped with a filter that is partially transmissive in the infrared region (so that its transmission spectrum overlaps with the emission spectrum of the infrared-spectrum light pulse F.sub.mono). In this way, the sensor pixels of the second image sensor will record an infrared image. Because the second image sensor S2 is also sensitive to visible light, its integration time is reduced or shortened to allow the short motion-freeze exposure with the IR spectrum light pulse F.sub.mono.

(32) As described in FIG. 4 above, the driver 13 drives the light sources 11, 12 to generate a visible-spectrum light pulse F.sub.RGB (the main flash) over the integration time of the first sensor S1, and to generate an infrared-spectrum light pulse F.sub.mono (the motion-freeze flash) over the much shorter integration time of the second sensor S2. The first image sensor S1 provides the colour image M.sub.RGB, and the second image sensor S2 provides the motion-freeze monochrome or infrared image M.sub.mono. The images M.sub.RGB, M.sub.mono are then forwarded to an image processing unit 14 which will use the information provided by the images M.sub.RGB, M.sub.mono to obtain the optimized image M.sub.opt described in FIG. 1, i.e. an image that is free of motion blur, or at least with reduced motion blur.

(33) FIG. 6 shows another embodiment of the inventive imaging arrangement 1, again incorporated in a smartphone 4. Here, the imaging arrangement 1 comprises a visible-spectrum camera C1 and an infrared camera C.sub.IR, and therefore also two image sensors S1, S.sub.IR. Here also, each sensor S1, S.sub.IR is represented in a simplified manner by its exemplary arrangement of sensor pixels P.sub.R, P.sub.G, P.sub.B, P.sub.IR as indicated at the side of the diagram. Similar to the two embodiments described above, the device 4 also comprises a white LED light source 11 and an infrared light source 12. In this embodiment, the first image sensor S1 is a three-channel sensor, comprising red-sensitive sensor pixels P.sub.R, green-sensitive sensor pixels P.sub.G, and blue-sensitive sensor pixels P.sub.B. Here, green-sensitive sensor pixels comprise about 50% of the total, while red-sensitive and blue-sensitive sensor pixels each comprise about 25%. The second image sensor S.sub.IR is an infrared image sensor, comprising only infrared-sensitive sensor pixels P.sub.IR. An exemplary arrangement of sensor pixels of the image sensors S1, S.sub.IR is shown in the enlarged portion of the diagram.

(34) The first image sensor S1 is used to capture a colour image M.sub.RGB, and the infrared image sensor S.sub.IR is used to capture a monochrome infrared image M.sub.mono. To this end, the first image sensor S1 is equipped with a colour filter mosaic applied to the image sensor die so that each sensor pixel is covered by a corresponding R, G, B filter as appropriate. The infrared image sensor S.sub.IR is equipped with a filter that is only transmissive in the infrared region. In this way, the sensor pixels of the infrared image sensor will record an infrared image M.sub.mono.

(35) As described in FIG. 4 and FIG. 5 above, the driver 13 drives the light sources 11, 12 to generate a visible-spectrum light pulse F.sub.RGB (the main flash) over the integration time of the first sensor S1, and to generate a short and powerful infrared-spectrum light pulse F.sub.mono (the motion-freeze flash) during the visible-spectrum light pulse F.sub.RGB. The first image sensor S1 provides the colour image M.sub.RGB, and the infrared image sensor S.sub.IR provides the motion-freeze monochrome or infrared image M.sub.mono. The images M.sub.RGB, M.sub.mono are then forwarded to an image processing unit 14 which will use the information provided by the images M.sub.RGB, M.sub.mono to obtain the optimized image M.sub.opt described in FIG. 1, i.e. an image that is free of motion blur, or at least with reduced motion blur.

(36) The performance of the imaging arrangement 1 can be optimized by matching the light source spectra to the sensitivities of the image sensors. FIG. 7 shows an exemplary set of spectra 70_B, 70_G, 70_R, 70_IR, with amplitude plotted against wavelength λ [nm]. Preferably, the peak sensitivity of an image sensor pixel coincides with a peak spectrum amplitude of the relevant light source. For example, the peak sensitivity of a IR-sensitive sensor pixel P.sub.IR in an image sensor will be as close as possible to the peak amplitude of the IR part of the spectrum 70_IR generated by the IR light source 12.

(37) Although the present invention has been disclosed in the form of preferred embodiments and variations thereon, it will be understood that numerous additional modifications and variations could be made thereto without departing from the scope of the invention. For example, the distribution of RGB and IR image sensor pixels could be different than shown in FIGS. 4-6 above; multiple monochrome images can be captured in the manner of a stroboscope; a moving object can be tracked by an infrared “follow-me” light to improve illumination of the moving object (for example using a segmented IR LED die); an active cooling step could precede an IR flash pulse in the case of temperature-limited IR LED driving conditions, etc.

(38) For the sake of clarity, it is to be understood that the use of “a” or “an” throughout this application does not exclude a plurality, and “comprising” does not exclude other steps or elements. The mention of a “unit” or a “module” does not preclude the use of more than one unit or module.

REFERENCE SIGNS

(39) imaging arrangement 1 visible-spectrum light source 11 infrared-spectrum light source 12 intensity graph 110, 120 driver 13 image processor 14 image processing step 140 device 4 sensitivity curve 70_R, 70_G, 70_B, 70_IR visible-spectrum image M.sub.RGB infrared-spectrum image M.sub.mono digital image M.sub.opt camera C, C1, C2, C.sub.IR image sensor S, S1, S2, S.sub.IR RGB sensor pixels P.sub.R, P.sub.G, P.sub.B infrared sensor pixel P.sub.IR scene D moving object X blurred imaged object B.sub.blur motion-freeze imaged object B.sub.mf imaged object B.sub.OK common image region R motion blur region R.sub.blur corrected image regions R.sub.edit matching parameter α visible-spectrum light pulse F.sub.RGB infrared-spectrum light pulse F.sub.mono pulse duration t.sub.RGB, t.sub.mono sensor integration time t.sub.int