Intensity-normalized image sensor

Abstract

An image sensor has a plurality of rows and columns of pixels, including RGB and bandpass I filters in a predetermined pattern shifted between adjacent columns so that none of the RGBI filters is adjacent the same type of filter. Each pixel includes a photodiode, a transfer gate and a floating diffusion. The transfer gate for all pixels in a pattern is controlled by the same signal, which can be a separate synchronous control signal controlled based on a predefined integration period or an asynchronous signal generated internally by the bandpass filter I and that is compared to a predefined voltage level indicative of a predetermined intensity at filter I. Upon activation of either signal, the integration period for the pixels ends and the charge on the floating diffusion for the R, G and B pixels is digitized in relation to the bandpass pixel I using a ratio-to-digital converter.

Claims

1. An image sensor comprising: a plurality of rows and columns of pixels, each pixel including a corresponding filter including a photodiode configured-to generate a signal in response to detecting one of red, green, blue or a frequency band of visible light incident on said pixel, said pixels including red (R), green (G), blue (B) and bandpass (I) filters, said bandpass (I) filters operable at the frequency band calibrated so that the associated pixel will accept more light than the pixels associated with the red, green or blue filters, said pixels arranged in a pre-defined sequence such that in each column a pattern RIG (for red, bandpass and green filters), and a pattern BIG (for blue, bandpass and green filters) are alternated along a given column and such that the pre-defined sequence of the RIG and BIG patterns is shifted between adjacent columns so that no particular type of the R,G,B and I filters is adjacent the same type of filter; and electrical circuitry for detecting the signal generated by the corresponding filter of each pixel and generating a digital output signal for further video processing, said electrical circuitry including a transfer gate and a floating diffusion corresponding to each photodiode of the RIG pattern and the BIG pattern, the electrical circuitry configured and adapted to integrate the signal received from each photodiode in the pattern during an integration period as a charge accumulated in the corresponding floating diffusion, the transfer gate responsive to a control signal to permit or prevent the signal from the each photodiode passing to a corresponding floating diffusion, wherein said control signal is an asynchronous signal generated by the bandpass filter (I) for a corresponding pixel in the pattern.

2. The image sensor of claim 1, wherein said electrical circuitry includes a comparator comparing the asynchronous signal generated by the bandpass filter (I) and a predefined voltage level indicative of a pre-determined intensity at filter (I), the comparator generating the control signal based on the comparison.

3. The image sensor of claim 2, wherein the asynchronous signal generated by the bandpass filter (I) is the accumulated charge of the transfer gate associated with the bandpass filter (I).

4. An image sensor comprising: a plurality of rows and columns of pixels, each pixel including a corresponding filter including a photodiode configured-to generate a signal in response to detecting one of red, green, blue or a frequency band of visible light incident on said pixel, said pixels including red (R), green (G), blue (B) and bandpass (I) filters, said bandpass (I) filters operable at the frequency band calibrated so that the associated pixel will accept more light than the pixels associated with the red, green or blue filters, said pixels arranged in a pre-defined sequence such that in each column a pattern RIG (for red, bandpass and green filters), and a pattern BIG (for blue, bandpass and green filters) are alternated along a given column and such that the pre-defined sequence of the RIG and BIG patterns is shifted between adjacent columns so that no particular type of the R,G,B and I filters is adjacent the same type of filter; and electrical circuitry for detecting the signal generated by the corresponding filter of each pixel and generating a digital output signal for further video processing, said electrical circuitry including a transfer gate and a floating diffusion corresponding to each photodiode of the RIG pattern and the BIG pattern, the electrical circuitry configured and adapted to integrate the signal received from each photodiode in the pattern during an integration period as a charge accumulated in the corresponding floating diffusion, the transfer gate responsive to a control signal to permit or prevent the signal from the each photodiode passing to a corresponding floating diffusion, wherein the accumulated charge on the floating diffusion for each of the (R) and (G) pixels of all of the RIG patterns and each of the (B) and (G) pixels of all of the RIG patterns is digitized in relation to the bandpass pixel (I) using a ratio-to-digital converter to produce a digitized signal corresponding to the relative intensity of the (R), (B) and (G) pixels, said digitized signals combined to form the digital output signal for further video processing.

5. The image sensor of claim 4, wherein the accumulated charge on the floating diffusion for the bandpass pixel (I) for each of the RIG patterns and BIG patterns is digitized to provide a digitized signal corresponding to the intensity of the light received at said bandpass pixel to be combined with the digitized signals from the (R), (G) and (B) pixels to form the digital output signal for further video processing.

6. The image sensor of claim 5, wherein the accumulated charge on the floating diffusion for the bandpass pixel (I) is digitized by an analog-to-digital converter.

7. The image sensor of claim 1, wherein the bandpass filter for the bandpass pixel (I) is configured to pass all visible light frequencies.

8. An image sensor comprising: at least one pixel including a corresponding photodiode adapted to generate a signal in response to detecting a predetermined frequency of light incident on said pixel; at least one bandpass pixel distinct from said at least one pixel and including a corresponding bandpass photodiode adapted to generate a signal in response to detecting a predetermined frequency band of light incident on said control pixel, the frequency band calibrated so that the control pixel will accept more light than the each of the at least one pixel; and electrical circuitry for detecting the signal generated by each corresponding photodiode of said at least one pixel and said at least one bandpass pixel and for generating a digital output signal for further video processing, said electrical circuitry including a transfer gate and a floating diffusion corresponding to each photodiode of said at least one pixel and said at least one bandpass pixel, the electrical circuitry configured and adapted to integrate the signal received from each photodiode during an integration period as a charge accumulated in the corresponding floating diffusion, the transfer gate responsive to a control signal to permit or prevent the signal from the each photodiode passing to a corresponding floating diffusion, wherein said control signal is an asynchronous signal generated by the bandpass photodiode of one of the at least one bandpass pixel corresponding to the photodiode of one or more of said at least one pixel.

9. The image sensor of claim 8, wherein said control signal is an externally generated synchronous control signal provided at a predetermined interval to produce a predefined integration period.

10. The image sensor of claim 8, wherein said electrical circuitry includes a comparator comparing the asynchronous signal generated by the bandpass photodiode of one of the at least one bandpass pixel and a predefined voltage level indicative of a pre-determined intensity at the bandpass photodiode of said one of the at least one bandpass pixel, the comparator generating the control signal based on the comparison.

11. The image sensor of claim 10, wherein the asynchronous signal generated by the bandpass photodiode of said one of the at least one bandpass pixel is the accumulated charge of the transfer gate associated with the bandpass photodiode of said one of the at least one bandpass pixel.

12. An image sensor comprising: at least one pixel including a corresponding photodiode adapted to generate a signal in response to detecting a predetermined frequency of light incident on said pixel; at least one bandpass pixel distinct from said at least one pixel and including a corresponding bandpass photodiode adapted to generate a signal in response to detecting a predetermined frequency band of light incident on said control pixel, the frequency band calibrated so that the control pixel will accept more light than the each of the at least one pixel; and electrical circuitry for detecting the signal generated by each corresponding photodiode of said at least one pixel and said at least one bandpass pixel and for generating a digital output signal for further video processing, said electrical circuitry including a transfer gate and a floating diffusion corresponding to each photodiode of said at least one pixel and said at least one bandpass pixel, the electrical circuitry configured and adapted to integrate the signal received from each photodiode during an integration period as a charge accumulated in the corresponding floating diffusion, the transfer gate responsive to a control signal to permit or prevent the signal from the each photodiode passing to a corresponding floating diffusion, wherein the accumulated charge on the floating diffusion for one or more of said at least one pixel is digitized in relation to one of the at least one bandpass pixel using a ratio-to-digital converter to produce a digitized signal corresponding to the relative intensity of the one or more of said at least one pixel, said digitized signal of each of said one or more of said at least one pixel combined to form the digital output signal for further video processing.

13. The image sensor of claim 12, wherein the accumulated charge on the floating diffusion for said one of the at least one bandpass pixel is digitized to provide a digitized signal corresponding to the intensity of the light received at said one of the at least one bandpass pixel to be combined with the digitized signal of each of said one or more of said at least one pixel to form the digital output signal for further video processing.

14. The image sensor of claim 13, wherein the accumulated charge on the floating diffusion for said one of the at least one bandpass pixel is digitized by an analog-to-digital converter.

15. The image sensor of claim 8, wherein photodiode for the at least one pixel and the bandpass photodiode are all configured to detect light in the same visible, infrared or ultraviolet spectrum.

16. An image sensor comprising: at least one pixel including a corresponding filter adapted to generate a signal in response to detecting a predetermined frequency of light incident on said pixel; at least one bandpass pixel distinct from said at least one pixel and including a corresponding bandpass filter adapted to generate a signal in response to detecting a predetermined frequency band of light incident on said control pixel, the frequency band calibrated so that the at least one bandpass pixel will accept more light than the each of the at least one pixel; and electrical circuitry for detecting the signal generated by the corresponding filter of each at least one pixel and each at least one bandpass pixel and for generating a digital output signal for further video processing, wherein the signal generated by at least one of said at least one pixel is digitized in relation to the signal generated by a corresponding one of the at least one bandpass pixel using a ratio-to-digital converter to produce a digitized signal corresponding to the relative intensity of at least one of the at least one pixel, said digitized signal for each of the at least one of said at least one pixel combined to form the digital output signal for further video processing.

Description

DESCRIPTION OF THE FIGURES

(1) FIG. 1 is a schematic of a prior art CMOS image sensor structure and pixel architecture.

(2) FIG. 2a is a diagram of pixel architecture according to one aspect of the present disclosure

(3) FIG. 2b is a schematic of a CMOS image sensor and pixel architecture according to one aspect of the present disclosure.

DETAILED DESCRIPTION

(4) For the purposes of promoting an understanding of the principles of the disclosure, reference will now be made to the embodiments illustrated in the drawings and described in the following written specification. It is understood that no limitation to the scope of the disclosure is thereby intended. It is further understood that the present disclosure includes any alterations and modifications to the illustrated embodiments and includes further applications of the principles disclosed herein as would normally occur to one skilled in the art to which this disclosure pertains.

(5) The present disclosure provides a different approach to address the problems of the prior art image sensors discussed above. In most cases when there is a bright pixel, it is not necessarily useful to know the precise brightness of the pixel. Instead, it is useful to identify it as a bright pixel and to measure the RGB content of the light reaching that pixel. In particular, the sensors and methods described herein determine the ratio between the R, G and B content of the light. The image sensors described herein address the dynamic range issue by capping the total integration time for the bright pixels based on the level of accumulated charge in those pixels. With this approach, the dynamic range of the front-end electronics and the data converters do not need to be large, yet all the necessary information can still be captured from the scene.

(6) An exemplary image sensor according to the present disclosure, with the corresponding pixel architecture, are shown in FIG. 2a, 2b. In addition to the traditional RGB filters, some of the pixels in this architecture are covered with a bandpass optical filter (or left blank), denoted by “I”, which stands for Intensity. The bandwidth of the bandpass optical filter for these intensity pixels I is calibrated so that the intensity pixels will always receive more photons and accumulate more charge than the other (RGB) pixels since the intensity pixels are sensitive to a larger optical spectrum than the other pixels. The arrangement of the pixels, shown in FIG. 2a, is chosen so that when looking at each column, the patterns “RIG” and “BIG” are alternated or advanced by one pixel per column. In other words, the filters for the pixels are provided in a pre-defined sequence, namely that in each column the pattern “RIG” (for red, intensity and green filters), and “BIG” (for blue intensity and green filters) are alternated along the column. The pre-defined sequence of the two patterns is shifted between adjacent columns so that no particular type of the RGBI filters is adjacent the same type of RGBI filter. With this pre-defined sequence of patterns, the density of the green G and intensity I pixels is twice as much as the red R and blue B pixels in the array. The extra green pixels provide a finer spatial resolution in the green spectrum for which the human eye is the most sensitive. The extra density of the intensity pixels with all-pass optical filters allows the collection of more photons in the overall imager to improve its low-light sensitivity. The given pattern for distribution of the filters provides a uniform density for all filter types and enables easier and more reliable interpolation algorithms in post processing steps.

(7) The circuit diagram for each of these three-pixel segments is similar. The circuit schematic of an “RIG” segment is shown in FIG. 2b as an example. Each pixel in the segment includes a photodiode PD.sub.x, a transfer gate TG.sub.x, and a floating diffusion C.sub.FD, x, where “x” corresponds to R, I or G in the present example or to B, I and G for the “BIG” three-pixel segment. The transfer gates TG.sub.R, TG.sub.I, TG.sub.G of all pixels are controlled by the same signal that can be activated at “OR” gate G either by a synchronous control signal “C” or by an asynchronous signal “S” locally generated at the comparator CP. The control signal “C” can be clocked to the image sensor system.

(8) In the beginning of a measurement time all pixels are reset by activating their corresponding transfer gates TG.sub.R, TG.sub.I, TG.sub.G. Then the integration begins and continues until the transfer gates are activated again on the occurrence of one of two conditions. In the first condition, the pixel “I” saturates to a predefined level “v.sub.0” generating the local flag or asynchronous signal “S” at comparator CP. In the second condition, if the intensity of pixel “I” is not high enough to lead to saturation, the integration continues until the end of the predefined integration period when the synchronous control signal “C” at gate G stops the integration. Once the integration is stopped, the charges on the floating diffusion C.sub.FD,R, C.sub.FD,G of the pixels “R” and “G”, respectively, are each digitized with reference to the charge in the “I” pixel by way of a ratio-to-digital converter (RDC) RDC. In other words, instead of using an external reference charge or voltage for analog-to-digital conversion, the charge from the “I” pixel can be used as a local reference. In particular, the ratios of the “R” and “G” charges to the charge of the intensity pixel “I” are provided as the digital values passed as the “R” and “G” values to the bus of the image sensor. As noted above, the signals R and G that are output from the circuitry of FIG. 2b are not absolute intensity values for the two colors red and green. Instead, the values are relative values or ratios by virtue of using the RDC for the digital conversion. The imaging device that receives these outputs R and G (as well as the outputs B and G from other patterns of sensors) can be configured and operable to use these color intensity ratios to generate a color image. After each conversion, the charge in the floating diffusion for each pixel from the previous measurement is reset to zero and another integration cycle begins until the voltage in the pixel “I” saturates and raises the local flag or the global asynchronous signal “S” stops the integration so that another digitization cycle follows.

(9) In general terms, an RDC can be considered as an analog-to-digital converter (ADC) in which the reference signal for the conversion is not provided externally, but rather is generated locally. Details of an RDC used in image sensors are disclosed in “Towards Always-On Mobile Object Detection: Energy vs. Performance Tradeoffs for Embedded HOG Feature Extraction,” A.Omid-Zohoor, Stanford University, available online at http://web.stanford.edu/group/murmann_group/cgi-bin/mediawiki/index.php/Alex_Omid-Zohoor, the entire disclosure of which is incorporated herein by reference. Further details of the structure and principles of operation of an RDC are found in “A Ratio-to-Digital Converter (RDC) with Time-Resolution Doubler for a Position Sensing System”, Rashid et al., KEK Proceedings 93-8 (June 1993), Radiation Detectors and Their Uses, pp. 60-65, the entire disclosure of which is incorporated herein by reference; and in “Realization of the concept of the ratio-to-digital converter (RDC)”, Matoba et al., Nuclear Instr. and Methods in Physics Res, Vol 224, Issue 1 (1984), p. 173-180, the entire disclosure of which is incorporated herein by reference.

(10) The charge from the pixel “I” can optionally be digitized using a separate analog-to-digital converter ADC. Digitization of the charge from the “I” pixel may not be necessary in all image sensors if the end application does not use the intensity information and instead uses only color data. In certain cases, the mechanism used to stop the integration time (either charge saturation or reaching maximum time) can also be recorded to tag the pixels as “bright” or “normal”. In particular, the asynchronous control signal “S” can be used as a flag that the intensity pixel “I” is bright. This can give extra information to the image or video processing software used for object or event recognition, or other image and video processing tasks.

(11) The particular implementation of the architecture described above in relation to FIG. 2b can be technology dependent. In a conventional planar circuit technology it is usually preferred to keep the digitization circuitry outside the pixel area to maximize the fill factor. In these types of imagers the data-converters and other sophisticated circuits with large foot-prints are shared among neighboring pixels or among pixels from an entire column. In more advanced technologies where backside illumination or 3D integration is available, more circuits can be placed underneath the detector area which can help improve the performance by local (pre-) processing of the information within the pixels. These technologies are particularly useful for the image sensors contemplated in the present disclosure, since the ratio-to-digital converters RDCs can have an architecture as simple as a successive approximation register (SAR) converter which easily fits into a pixel area in the advanced technology nodes. The ADC for the intensity pixel can also be implemented locally (since only one ADC is needed per three pixels), or it can be shared among the pixels in one column, similar to the traditional imagers.

(12) Other implementations of the image sensors disclosed herein can include different arrangements of the pixel segments and different circuit architectures. For instance, the RDC blocks can be time-multiplexed between “R” and “G” pixels (or between “B” and “G” for a BIG segment) to reduce the required circuit area. Other combinations of pixel filters, such as RGI or R I could be used as well. In a further alternative, the R, G and B outputs can be summed in the analog domain to create a composite “I” that is used to normalize the outputs (R, G and B).

(13) With the particular implementation of the present disclosure, it is necessary to directly access the photodetectors in “I” regions to dynamically compare their accumulated charge to the reference saturation level. Considering the fabrication process of the imagers with pinned photodiodes, this might add some complications to the design. One embodiment to address these complications is to modify the manner in which the signal “S” is generated. In one specific embodiment, the control signal “C” continues transferring the charge from photodiodes at small time steps, for example 100 μs. Then the comparator can be connected to the floating diffusion C.sub.FD,I of the “I” pixel to determine if at any of these steps the transferred charge exceeds a predefined saturation limit, in which case the integration and charge transfer process is stopped. As long as the transferred charge of the “I” pixel is within the saturation limit, the integration and transfer steps continue until the overall integration time reaches a predefined limit, for instance 10 ms. Another advantage of this technique is that the number of the short accumulation periods that it takes for a bright pixel to reach the predefined saturation limit can be used as a measure of its intensity. For instance, a bright pixel that reaches the saturation limit in 2×100 μs is 5 times brighter than a pixel that reaches this limit in 10×100 μs.

(14) In a further embodiment, rather than include the “I” all-pass optical filter in the filter pattern, the output of the “G” filter of an “RGB” pattern, similar to a conventional imager, can be used as the reference for digitizing the outputs of the “R” and “B” segments. It is understood that the amount of electric charge collected in a “G” segment may not necessarily be larger than the “R” and “B” segments, which can lead to a reduction in measurement precision. However, this embodiment may be well-suited for use with conventional CMOS imager technology, and therefore it could be attractive for applications where cost is of central importance

(15) The present disclosure should be considered as illustrative and not restrictive in character. It is understood that only certain embodiments have been presented and that all changes, modifications and further applications that come within the spirit of the disclosure are desired to be protected.

(16) For example, in the illustrated embodiment, a predetermined pattern of pixels with red, green and blue filters or photodetectors are provided along with pixel I with an all-pass filter or photodetector that controls the duration of the integration cycle and/or provides the internal reference signal for the RDCs. Alternatively, the one or more pixels with specific N wavelength filters can be provided together with a “control” pixel I having a wider wavelength band filter than the one or more other pixels. The wavelength band of the “control” pixel I is calibrated to ensure that the “control” pixel collects more photons from a particular scene than any of the one or more other pixels, thus ensuring that the transfer gate TG.sub.I for the “control” pixel I collects more charge than the transfer gates for the other pixels. The other non-“control” pixels can have wavelength band filters, rather than filters at a discrete wavelength, provided that the spectrum of the non-“control” pixels is less than the spectrum detected by the “control” pixel I.

(17) It is further understood that although the examples described herein refer to visible light spectrum for the filters, the filters can be calibrated to all light spectra, including infrared and ultraviolet. Thus, in certain applications all of the pixels and filters can be configured to detect wavelengths in the infrared or ultraviolet bands. In any case, the filter for the intensity pixel I must be in the same light spectrum as the other pixels (i.e., all in visible light, all in IR or all in UV), and the intensity pixel must configured to receive more light energy than any of the other filters. As discussed above, this can be accomplished by providing the intensity pixel filter with a wider bandwidth than the other filters.