IMAGING DEVICE AND METHOD OF MULTI-SPECTRAL IMAGING

Abstract

An imaging device is specified, the imaging device including a detector array a plurality of pixels, the pixels including a plurality of subpixel types, a micromirror array with a plurality of mirror elements, and an internal light source, wherein at least one of the subpixel types is configured to detect a first radiation; the mirror elements are configured to deflect in response to a second radiation, the internal light source is configured to illuminate the detector array with a third radiation; at least one of the subpixel types is configured to detect the third radiation deflected by the micromirror array. Furthermore, a method of multi-spectral imaging is specified.

Claims

1. An imaging device comprising: a detector array with a plurality of pixels, the pixels comprising a plurality of subpixel types, a micromirror array with a plurality of mirror elements, and an internal light source, wherein at least one of the subpixel types is configured to detect a first radiation wherein the first radiation includes radiation in the visible spectral range and comes from a scene to be imaged by the imaging device; the mirror elements are configured to deflect in response to a second radiation, the internal light source is configured to illuminate the micromirror array with a third radiation; at least one of the subpixel types is configured to detect the third radiation deflected by the micromirror array.

2. (canceled)

3. The imaging device according to claim 1, wherein the second radiation includes thermal radiation.

4. The imaging device according to claim 1, wherein the imaging device comprises a first lens configured to direct the first radiation onto the detector array and a second lens configured to direct the second radiation onto the micromirror array, the first lens and the second lens being arranged side by side in a top view of the imaging device.

5. The imaging device according to claim 4, wherein a first beam splitter is arranged between the detector array and the first lens, and wherein a second beam splitter is arranged between the internal light source and the micromirror array.

6. The imaging device according to claim 1, wherein the detector array and the internal light source are mounted side by side on a common substrate.

7. The imaging device according to claim 1, wherein the at least one subpixel type configured to detect the third radiation is sensitive to at least part of the first radiation as well.

8. The imaging device according to claim 1, wherein the at least one subpixel type configured to detect the third radiation is insensitive to the first radiation.

9. The imaging device according to claim 1, wherein at least one subpixel type is configured to detect near infrared radiation included in the first radiation.

10. The imaging device according to claim 1, wherein the internal light source is configured to emit the third radiation with a dot pattern.

11. The imaging device according to claim 1, wherein the internal light source includes an emitter and a dot pattern generator arranged downstream of the emitter.

12. The imaging device according to claim 1, wherein the internal light source includes an emitter array configured to emit a plurality of individual light beams.

13. The imaging device according to claim 1, wherein the imaging device is configured to be operable in a low power mode, wherein only a subset of the plurality of subpixels is operated in the low power mode.

14. The imaging device according to claim 13, wherein only one of the subpixel types is operated in the low power mode.

15. A method of multi-spectral imaging comprising including a) providing an imaging device comprising: a detector array with a plurality of pixels, the pixels comprising a plurality of subpixel types, a micromirror array with a plurality of mirror elements, the mirror elements being configured to deflect in response to a second radiation, and an internal light source; b) obtaining a first image using at least one subpixel type responsive to a first radiation, wherein the first radiation includes radiation in the visible spectral range and comes from a scene to be imaged by the imaging device; c) illuminating the micromirror array with a third radiation emitted by the internal light source; d) detecting the third radiation reflected by the micromirror array using at least one subpixel type responsive to the third radiation; and e) obtaining a second image corresponding to the second radiation based on the detected third radiation.

16. The method according to claim 15, wherein step e) includes comparing a detected dot pattern of the third radiation with a calibrated dot pattern.

17. The method according to claim 16, wherein step e) includes determining an intensity of the second radiation for the pixels of the detector array based on deviations between the detected dot pattern and the calibrated dot pattern.

18. The method according to claim 15, wherein steps b) and c) are performed simultaneously using different subpixel types for the first radiation and the third radiation.

19. The method according to claim 15, wherein steps b) and c) are performed using time-multiplexing based on at least one subpixel type for the first radiation and the third radiation.

20. The method according to claim 15, wherein the method is performed using an imaging device comprising: a detector array with a plurality of pixels, the pixels comprising a plurality of subpixel types, a micromirror array with a plurality of mirror elements, and an internal light source, wherein at least one of the subpixel types is configured to detect a first radiation, wherein the first radiation includes radiation in the visible spectral range; the mirror elements are configured to deflect in response to a second radiation, the internal light source is configured to illuminate the micromirror array with a third radiation: and at least one of the subpixel types is configured to detect the third radiation deflected by the micromirror array.

21. The imaging device according to claim 3, wherein the second radiation includes radiation with a wavelength between 7 m and 14 m.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In the figures:

[0070] FIG. 1 shows an exemplary embodiment of an imaging device;

[0071] FIG. 2 shows a schematic representation of the imaging device operation according to an exemplary embodiment;

[0072] FIG. 3 shows an exemplary embodiment of a micromirror array;

[0073] FIGS. 4A and 4B show an exemplary embodiment of a mirror element in top view (FIG. 4A) and in side view (FIG. 4B);

[0074] FIG. 4C and 4D show an exemplary embodiment of a mirror element in top view (FIG. 4C) and in side view (FIG. 4D);

[0075] FIGS. 5A and 5B show an exemplary embodiment of a calibrated dot pattern (FIG. 5A) and a corresponding detected dot pattern (FIG. 5B) during operation;

[0076] FIG. 6 shows an exemplary embodiment of an imaging device;

[0077] FIG. 7 shows an exemplary embodiment of an imaging device;

[0078] FIG. 8 shows an exemplary embodiment of an imaging device;

[0079] FIG. 9 shows an exemplary embodiment of an imaging device;

[0080] FIG. 10A shows an exemplary embodiment of a filter array with corresponding transmission curves for the filters as a function of the wavelength in FIG. 10B;

[0081] FIG. 11A shows an exemplary embodiment of a filter array with corresponding transmission curves for the filters as a function of the wavelength in FIG. 11B;

[0082] FIG. 12A shows an exemplary embodiment of a filter array with corresponding transmission curves for the filters as a function of the wavelength in FIG. 12B;

[0083] FIG. 13A shows an exemplary embodiment of a filter array with corresponding transmission curves for the filters as a function of the wavelength in FIG. 13B;

[0084] FIG. 14 shows an exemplary embodiment of a method of multispectral imaging.

DETAILED DESCRIPTION

[0085] The elements illustrated in the figures and their size relationships among one another are not necessarily true to scale. Rather, individual elements or layer thicknesses may be represented with an exaggerated size for the sake of better representability and/or for the sake of better understanding.

[0086] An exemplary embodiment of an imaging device 1 is schematically illustrated in FIG. 1. The imaging device 1 comprises a detector array 2 with a plurality of pixels 25 wherein the pixels 25 each comprise a plurality of subpixel types 21. A filter array 29 is arranged on the detector array in order to obtain different spectral sensitivity distributions for the subpixel types 21. For easier representation the pixels and subpixels are not explicitly shown in FIG. 1. Possible arrangements of subpixels 21 within the pixels 25 are described in connection with FIGS. 10A to 13B.

[0087] The imaging device 1 further comprises a micromirror array 3 with a plurality of mirror elements 31. The micromirror array 3 may be one-dimensional or two-dimensional.

[0088] The imaging device 1 further comprises internal light source 4.

[0089] At least one of the subpixel types 21, for example three subpixel types, is/are configured to detect a first radiation R1. For example, one subpixel type 21 is configured to detect radiation in the blue spectral range, one subpixel type 21 is configured to detect radiation in the green spectral range and one subpixel type 21 is configured to detect radiation in the red spectral range.

[0090] The radiation R1 is directed onto the detector array 2 using a first lens 51. The first lens 51 is illustrated as a single lens in FIG. 1, but it may also comprise two or more lenses.

[0091] The imaging device 1 further comprises a second lens 52 arranged laterally beside the first lens 51. The second lens 52 is configured to direct a second radiation R2, for example radiation in the longwave infrared range onto the micromirror array 3.

[0092] The first lens 51 and/or the second lens 52 may be of a single lens or a multi-lens configuration. The first lens 51 and/or the second lens 52 may be configured as a conventional lens of transmissive bulk material or as a metalens. For example, the metalens may comprise a dielectric material such as titanium dioxide, niobium pentoxide or silicon nitride or a semiconductor material such as silicon or a metal. The metalens may comprise structures like pillars or slots or holes, H, U, V, plus(+) or cross-shaped structures. For example, a height of the structures is between 500 nm and 700 nm inclusive. For example, a maximum lateral extent or diameter of the structures is between 40 nm and 400 nm inclusive. For example, a period of the structures is between 180 nm and 450 nm inclusive.

[0093] Furthermore, the second lens 52 may also be a microlens array. The microlens array may also be implemented as an array of metalenses.

[0094] The mirror elements 31 are configured to deflect in response to the second radiation R2. The internal light source 4 is configured to illuminate the micromirror array with a third radiation R3. At least one of the subpixel types 21 is configured to detect the third radiation R3 reflected by the micromirror array 3.

[0095] A simultaneous readout of a thermal image belonging to radiation R2 and an optical image belonging to radiation R1 can be obtained from the same detector array 2 which is configured as a CMOS image sensor, for example.

[0096] The thermal image is read out from a difference in a spot position of a dot pattern compared to a calibrated dot pattern belonging to a calibrated position of the mirror elements 31.

[0097] This is illustrated in FIGS. 5A and 5B, showing a calibrated dot pattern 71 in FIG. 5A and a corresponding detected dot pattern 72 in FIG. 5B which is detected during regular operation. The thermal energy of the second radiation R2 causes the mirror elements 31 of the micromirror array 3 to bend. The difference in the individual spot positions of the dot pattern on the detector array 2 allows the corresponding thermal image belonging to the second radiation R2 to be calculated.

[0098] As shown in FIG. 1, a lens 43 may be placed between the internal light source 4 and the micromirror array 3. This facilitates obtaining sharp spots on the detector array 2, resulting in an improved quality of the obtained thermal image.

[0099] For example, the internal light source 4 emits with a peak emission wavelength in the near infrared range so that the detector array 2 with a photosensitive region based on silicon is able to detect the third radiation R3.

[0100] However, the internal light source 4 may also emit in a different spectral range, for instance in the ultraviolet range as long as the radiation can be detected by at least one subpixel type 21 of the detector array 2.

[0101] In the exemplary embodiment of FIG. 1, an optical filter 53 is arranged in the beam path of the second radiation R2 on its way to the micromirror array 3, for example downstream of the second lens 53. For example, the optical filter 3 transmits radiation in the longwave infrared range and blocks radiation in the visible range. However, such an optical filter 53 is optional. This likewise applies to all further subsequently described exemplary embodiments.

[0102] The imaging device 1 uses two lenses, namely the first lens 51 and the second lens 52, to create separate imaging channels for the first radiation R1, for example visible radiation, and the second radiation R2, for example thermal radiation in the longwave infrared spectral ranges. The first lens 51 and the second lens 52 are spaced apart from one another and designed and aligned such that they both image the same scene.

[0103] The first lens 51 and/or the second lens 52 may be optionally coated with an anti-reflector coating for better performance in the respective spectral range. The optical filter 53 may be used to block any radiation other than the longwave infrared radiation from impinging on the micromirror array 3.

[0104] For example, the mirror elements 31 are cantilevers. The optical filter 53 may help to reduce the measurement noise in the longwave infrared image.

[0105] The first lens is placed above the detector array 2 to form an imaging path for the first radiation R1, for example visible radiation in the spectral range between 400 nm and 700 nm. The first radiation R1 to be detected may further include radiation in the near infrared spectral range.

[0106] The first lens 51 may be placed directly above the detector array 2 or with an offset to satisfy design conditions to align both the images belonging to the first R1 and the second radiation R2 respectively. The captured visible images belonging to the first radiation R1 may serve as a frame of reference to setup and/or align the imaging assembly used to detect the second radiation R2.

[0107] An exemplary embodiment of a micromirror array 3 is illustrated in FIG. 3. The mirror elements 31 are micro cantilevers arranged in a rectangular grid. The mirror elements 31 each extend away from a frame 39 and are free to reflect in response to impinging thermal radiation. Thus, the mirror elements 31 represent micro-optomechanical (MOM) devices such as bimaterial micro-cantilevers, for instance bimetallic micro-cantilevers.

[0108] The size and the pitch of the mirror elements 31 is determined by the required resolution and sensitivity of the image belonging to the second radiation, for instance a thermal image. The mirror elements 31 may also be arranged one-dimensionally in a row. Further details of the mirror elements 31 are described in connection with FIGS. 4A to 4D.

[0109] The light of the internal light source 4, i.e. the third radiation R3 is illuminated on the side of the micromirror array 3 facing away from the second lens 52. The third radiation R3 is reflected by the micromirror array 3 onto the detector array 2.

[0110] When the side of the mirror element 31 facing the second lens 52 is illuminated by radiation R2, for example thermal energy, the mirror elements 31, for example cantilevers, bend, thereby deflecting the spots of the third radiation R3 to a different position on the detector array 2. This is illustrated in FIGS. 5A and 5B which show how the arrangement of the spots on the detector array 2 changes as a response to the incident thermal energy mediated by the bending of the micro-cantilever array in response to the incident thermal energy. By tracking the change in position of each spot with time, the thermal image can be reconstructed simultaneously with the visible image from the same detector array 2. The sensitivity of the micromirror array and overall path length of the deflected internal light determines the limits of the movement of the spots on the detector array 2.

[0111] FIG. 2 schematically represents a summary of the operation of the device. The detector array 2 is illuminated by radiation R1 from the scene, for instance visible light. Second radiation R2, for instance in the longwave infrared radiation, from the same scene is focused on the micromirror array (block S1). This causes the mirror elements 31 to bend (block S2). Thus the longwave infrared radiation causes the mirror elements 31 to bend (block S3). The micromirror array 3 is irradiated by radiation R3 from an internal light source 4. The deflected third radiation R3 impinges onto the detector array 2, where at least one type of subpixel 21 is sensitive to the radiation R3 of the internal light source 4.

[0112] FIGS. 4A to 4D illustrate exemplary embodiments of possible mirror elements 31 in top view (FIGS. 4A and 4C) and in side view (FIGS. 4B and 4D). The mirror elements 31 comprise a first material 35 and a second material 36. The frame is likewise made from the second material 36. For example the first material is gold and the second material is silicon and/or silicon nitride. Other materials may likewise be used as first and second material as long as there is a sufficient difference in the thermal coefficient of linear expansion for the materials used. In the exemplary embodiment illustrated in FIG. 4A the mirror element 31 essentially has a rectangular shape that extends away from the frame 39. In the exemplary embodiment of FIG. 4C the mirror element 31 has a triangular shape.

[0113] The geometry of the mirror elements 31 may be modified in wide limits as long as impinging thermal energy results in a deflection of the mirror element 31.

[0114] Optionally, the imaging device 1 is configured to be operable in a low power mode. For example, only a subset of the plurality of subpixels is operated in the low power mode. For example, the subset is a random selection of the subpixels or corresponds to a predefined selection. The subset may include 10% or less, or 5% or less, or 1% or less of the total number of subpixels of the imaging device 1. The reduced number of operated subpixels allows for a significant reduction of the power consumption compared to a regular operation mode where all of the subpixels are operated. If, for example, a 4040 matrix of pixels 25 is assigned to one mirror element 31, the number of activated pixels during the low power mode may be 10 or less, for example 2, 3, 5 or 8, so that less than 1% of the pixels assigned to the mirror element 31 are operated in the low power mode. Of course, the number of mirror elements 31, the number of pixels 25 assigned to one mirror element 31, and the number of pixels per mirror element 31 activated during the low power mode may be varied within broad limits.

[0115] For example, only the subpixel type sensitive to the third radiation R3 is operated in the low power mode, so that a change in the thermal radiation, for example due to an approaching animal or human being can trigger a switching into the regular operation mode, for example to perform resolution imaging in mono or full color so as to now enable high quality imaging for object identification, recognition, etc. The low power mode can be used for human presence monitoring, in particular followed by imaging, for instance.

[0116] However, in addition to or instead of thermal radiation, radiation in the visible spectral range may also be detected in the low power mode and trigger a switching event using the respective subpixel types.

[0117] The described low power mode may also apply for the further exemplary embodiments.

[0118] The exemplary embodiments illustrated in FIGS. 6 and 7 differ in the way a radiation pattern is formed for the third radiation R3 provided by the internal light source 4. In the exemplary embodiment of FIG. 6 the internal light source 4 comprises an emitter 41. The emitter 41 produces a uniform illumination that illuminates a dot pattern generator 42, such as a diffractive optical element.

[0119] In the exemplary embodiment of FIG. 7, the emitter of the internal light source 4 is an emitter array 44 that produces an array of collimated light beams. For example, the emitter array 44 is an array of laser diodes or vertical cavity surface emitting laser (VCSEL) diodes. In this case an additional dot pattern generator may be dispensed with.

[0120] The exemplary embodiment of FIG. 8 essentially corresponds to the embodiments described in connection with FIGS. 1, 6 and 7. Here, the internal light source 4 and the detector array 2 are arranged on a common substrate 9. For example, the internal light source 4 and the detector array 2 are configured as surface mounted devices.

[0121] Unlike in the previous exemplary embodiments, the imaging device 1 further comprises a first beam splitter 61 arranged between the detector array 2 and the first lens 51 and a second beam splitter 62 arranged between the internal light source 4 and the second lens 52. This results in a very compact setup of the imaging device 1 so that the imaging device is also suited for use in handheld devices such as smartphones or for home automation devices that require implementation in compact packages.

[0122] The first beam splitter 61 and/or the second beam splitter 62 may be embodied as dichroic or non-dichroic beam splitters. For example, the first beam splitter 61 is configured such that it allows for passage of the first radiation R1, for example visible radiation, and for reflection of the third radiation R3 which may have a longer wavelength than the first radiation R1. The angle of incidence on the first beam splitter 61 is about 45 for the first radiation R1 as well as for the third radiation R3.

[0123] As shown in FIG. 8, the third radiation R3 emitted by the internal light source 4 illuminates the micromirror array 3 through the second beam splitter 61. The radiation reflected by the micromirror array 3 is reflected at least in part by the second beam splitter 62 and subsequently by the first beam splitter 61 so that it impinges onto the detector array 2. The first radiation R1 is transmitted through the first beam splitter 61 and impinges onto the detector array 2 as well.

[0124] For example, the first beam splitter 61 and the second beam splitter 62 are arranged at an angle of 45 with respect to a mounting side of the substrate 9.

[0125] For example, the first beam splitter 61 and/or the second beam splitter 62 are implemented as thin film coatings on suitable substrates.

[0126] The exemplary embodiment shown in FIG. 9 essentially corresponds to that of FIG. 8.

[0127] In this exemplary embodiment the detector array 2 is configured to detect near infrared portions of the first radiation R1 (labelled NIR in FIG. 9). Consequently, the imaging device 1 may provide a full color image for the visible radiation, a second image for the radiation R2 in the longwave infrared range and a third image belonging to the near infrared radiation NIRI of the first radiation R1. An example of a suitable filter array is described in connection with FIGS. 13A and 13B.

[0128] Direct detection of near infrared radiation may also be used for the exemplary embodiments described in connection with FIGS. 1, 6 and 7.

[0129] FIG. 10A, together with FIG. 10B, illustrates an exemplary embodiment of a filter array 29 for the detector array 2. In this exemplary embodiment the detector array 2 comprises three different subpixel types 21 for detecting radiation in the blue spectral range (labelled B), in the green spectral range (labelled G) and in the red spectral range (labelled R). The three subpixel types 21 are arranged in the so-called Bayer mosaic pattern (BGGR) having one subpixel for blue radiation, one subpixel for red radiation and two subpixels for green radiation.

[0130] As illustrated in the transmission curves of FIG. 10B, the color filter transmitting the red radiation also transmits the third radiation R3 in the near infrared range emitted by the internal light source 4. The blue color filter B and the green color filter G likewise transmit in the near infrared wavelength regime NIR. Thus, the transmission curves of the blue color filter and the green color filter have two transmission maxima spectrally spaced from one another.

[0131] With this filter array 29, near infrared light emitted by the internal light source 4 can be detected by all of the subpixel types 21. This allows for a high-resolution detection of the radiation pattern of the third radiation R3 impinging onto the detector array 2.

[0132] In this configuration the internal light source 4 may be operated in pulsed (on/off) operation at a certain frequency. When the internal light source 4 is off, then no third radiation R3 of the internal light source 4 is present on the detector array so that during this timeframe the detector array 2 will only record the images for the radiation R1, for instance in the visible spectral range. When the internal light source 4 in the near infrared is on, then the near infrared spots appear on the detector array. During this timeframe, the imaging device 1 records the changes in the near infrared spot location, thereby estimating the thermal images. Thus, in the on/off cycle the imaging device 1 is able to record both the first image in the visible range and the second image in the thermal range in quick succession. This approach is particularly suited for cost and/or power sensitive applications.

[0133] In the exemplary embodiment shown in FIGS. 11A and 11B, the detector array 2 comprises an additional subpixel type 21 which is sensitive for the radiation of the internal light source 4 in the near infrared spectral range (labelled NIR). In this implementation all color channels are available at the same time so that the imaging device 1 performs capturing of both thermal and visible images. Such an implementation allows for high frame rate applications.

[0134] The exemplary embodiment shown in FIGS. 12A and 12B essentially corresponds to that of FIGS. 11A and 11B.

[0135] In contrast to the previous exemplary embodiment, the internal light source 4 does not emit in the near infrared spectral range, but rather in the ultraviolet spectral range. Thus, the detector array 2 comprises, in addition to the subpixel types 21 for visible radiation (R, G, B) a subpixel type (UV) to detect ultraviolet radiation.

[0136] The exemplary embodiment shown in FIGS. 13A and 13B allows simultaneous imaging to be performed in the visible spectral range, the near infrared range and the longwave infrared range. In addition to the subpixel types 21 for the visible spectral range (R, G, B), a subpixel type NIR1 sensitive in the near infrared and a further subpixel type NIR2 sensitive in the near infrared with a different NIR wavelength are provided. By choosing two NIR channels, one of the channels is dedicated to the thermal imaging via the internal light source 4, while the other NIR channel is dedicated to imaging at NIR wavelength provided in first radiation R1.

[0137] For example, a peak wavelength of the internal light source 4 emits at 780 nm, whereas the NIR2 channel is chosen for imaging at a peak detection wavelength of 940 nm. Of course, simultaneous imaging in the visible, the near infrared and the longwave infrared range can also be obtained using an internal light source 4 for emitting in the ultraviolet spectral range as described in connection with FIGS. 12A and 12B. In this case, only one subpixel type 21 sensitive in the NIR is required.

[0138] In the exemplary embodiments described above, the detector array and the micromirror array are embodied as two-dimensional arrays. However, the micromirror array 3 and/or the detector array 2 may also be implemented as one-dimensional line scan array.

[0139] An exemplary embodiment of a method for multi-spectral imaging is illustrated in FIG. 14. A detector array 2 with a plurality of pixels 25 is provided, wherein the pixels comprise a plurality of subpixel types 21. The imaging device 1 further comprises a micromirror array 3 with a plurality of mirror elements 31, wherein the mirror elements 31 are configured to deflect in response to a second radiation R2. The imaging device further comprises an internal light source 4 (step S11).

[0140] In a step S12 a first image is obtained using at least one subpixel type responsive to a first radiation R1.

[0141] In a step S13, the micromirror array is illuminated with a third radiation R3 emitted by the internal light source 4.

[0142] In a step S14 the third radiation R3 reflected by the micromirror array 3 is detected using at least one subpixel type 21 responsive to the third radiation R3.

[0143] In a step S15 a second image corresponding to the second radiation R2 is obtained based on the detected third radiation R3.

[0144] As described in connection with FIGS. 5A and 5B, the step of obtaining the second image may include comparing a detected dot pattern of the third radiation R3 with a calibrated dot pattern.

[0145] The steps S12 and S13 may be performed in the time domain in an alternating manner as described in connection with FIGS. 10A and 10B or simultaneously as described in connection with FIGS. 11A to 13B.

[0146] Using the method, first and second images in different spectral ranges, for instance in the visible spectral range and in the longwave infrared range, can be obtained using the same detector array. In addition, a third image in the near infrared spectral range may be provided.

[0147] The invention described herein is not restricted by the description given with reference to the exemplary embodiments. Rather the invention encompasses any novel feature and any combination of features including in particular any combination of features in the claims, even if this feature or this combination is not itself explicitly indicated in the claims or exemplary embodiments.