METHODS AND APPARATUS FOR AUTOMATED NOISE AND TEXTURE OPTIMIZATION OF DIGITAL IMAGE SENSORS

Abstract

Systems and methods are disclosed for calibrating an image sensor using a source image taken from the image sensor and comparing it to a reference image. In one embodiment, the method may involve determining the luminance and chrominance values of portions of the image at successive frequency levels and calculating a standard deviation at each frequency level for both the source image and the reference image. The standard deviation values may be compared and a difference determined. Using unit vector search vectors, noise values may be calculated to determine sensor calibration values.

Claims

1. A method of tuning a noise level of an image sensor, the method comprising: determining a first set of frequency levels of an image produced by the image sensor using a set of low pass filters, the set of low pass filters comprising at least one bilateral low pass filter; determining a first set of chrominance noise values and luminance noise values corresponding to each frequency level of the first set of frequency levels; calculating a target set of noise values associated with each frequency level of the first set of frequency levels, the target set of noise values calculated based on comparing the first set of chrominance and luminance noise values to a second set of chrominance and luminance noise values of a reference image, the reference image captured using a calibrated image sensor; determining a first set of noise values associated with the image using a coarse search, the coarse search based on a first set of vectors, wherein each noise value of the first set of noise values corresponds to a frequency level of the first set of frequency levels; determining a second set of noise values associated with the image using a fine search, the fine search comprising a second set of vectors, wherein the second set of vectors are determined based on a noise value difference between the target set of noise values and the first set of noise values, wherein each noise value of the second set of noise values corresponds to a frequency level of the first set of frequency levels; updating a coefficient of the at least one bilateral low pass filter based on a difference between the second set of noise values and the first set of noise values; and adjusting the noise level of each pixel of the image sensor via applying the at least one bilateral low pass filter with the updated coefficient according to the target set of noise values.

2. The method of claim 1, wherein updating the at least one bilateral low pass filter is continued until the difference between the second set of noise values and the first set of noise values is within a matching range.

3. The method of claim 2, wherein the matching range is different for each frequency level.

4. The method of claim 1, wherein the first set of vectors and the second set of vectors each comprise a four-dimensional vector for each frequency level.

5. The method of claim 4, wherein the image and the reference image are captured at the same distance from a color calibration target, the color calibration target comprising an arrangement of colored regions.

6. The method of claim 5, wherein the coarse search is based on the following equation: ${.Math.}_{l = 1}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - {.Math.}_{l = 0}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, l, p} = 0$ where: n: a total number of colored regions in the color calibration target; m: a total number of frequency levels; l: a subject frequency level; k: a subject colored region; p: four-dimensional vector values.

7. The method of claim 6, wherein the fine search is based on the following equation: $MSE (\frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, pl}) .Math.$ where: MSE: Mean Square Error.

8. The method of claim 4, wherein the four-dimensional vector comprises four constants and each constant is a positive integer, and adjusting the noise level of each pixel of the image sensor comprises updating a noise reduction block of the image sensor.

9. An apparatus for tuning a noise level of an image sensor, the apparatus comprising: an image sensor configured to produce an image; a processor configured to: determine a first set of frequency levels of the image produced by the image sensor using a set of low pass filters, the set of low pass filters comprising at least one bilateral low pass filter; determine a first set of chrominance noise values and luminance noise values corresponding to each frequency level of the first set of frequency levels; calculate a target set of noise values associated with each frequency level of the first set of frequency levels, the target set of noise values calculated based on comparing the first set of chrominance and luminance noise values to a second set of chrominance and luminance noise values of a reference image, the reference image captured using a calibrated image sensor; determine a first set of noise values associated with the image produced by the image sensor using a coarse search, the coarse search based on a first set of vectors, wherein each noise value of the first set of noise values corresponds to a frequency level of the first set of frequency levels; determine a second set of noise values associated with the image produced by the image sensor using a fine search, the fine search comprising a second set of vectors, wherein the second set of vectors are determined based on a noise value difference between the target set of noise values and the first set of noise values, wherein each noise value of the second set of noise values corresponds to a frequency level of the first set of frequency levels; and update a coefficient of the at least one bilateral low pass filter based on a difference between the second set of noise values and the first set of noise values; and adjust the noise level of each pixel of the image sensor via applying the at least one bilateral low pass filter with the updated coefficient.

10. The apparatus of claim 9, wherein the processor is configured to update the at least one bilateral low pass filter until the difference between the second set of noise values and the first set of noise values is within a matching range.

11. The apparatus of claim 10, wherein the matching range is different for each frequency level.

12. The apparatus of claim 9, wherein the first set of vectors and the second set of vectors each comprise a four-dimensional vector for each frequency level.

13. The apparatus of claim 12, wherein the image and the reference image are captured at the same distance from a color calibration target, the color calibration target comprising an arrangement of colored regions.

14. The apparatus of claim 13, wherein the coarse search is based on the following equation: ${.Math.}_{l = 1}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - {.Math.}_{l = 0}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, l, p} = 0$ where: n: a total number of colored regions in the color calibration target; m: a total number of frequency levels; l: a subject frequency level; k: a subject colored region; p: four-dimensional vector values.

15. The apparatus of claim 14, wherein the fine search is based on the following equation: $MSE (\frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, pl}) .Math.$ where: MSE: Mean Square Error.

16. The apparatus of claim 15, wherein the four-dimensional vector comprises four constants and each constant is a positive integer, and adjusting the noise level of each pixel of the image sensor comprises updating a noise reduction block of the image sensor.

17. An apparatus, comprising: means for determining a first set of frequency levels of an image produced by an image sensor using a set of low pass filters, the set of low pass filters comprising at least one bilateral low pass filter; means for determining a first set of chrominance noise values and luminance noise values corresponding to each frequency level of the first set of frequency levels; means for calculating a target set of noise values associated with each frequency level of the first set of frequency levels, the target set of noise values calculated based on comparing the first set of chrominance and luminance noise values to a second set of chrominance and luminance noise values of a reference image, the reference image captured using a calibrated image sensor; means for determining a first set of noise values associated with the image produced by the image sensor using a coarse search, the coarse search based on a first set of vectors, wherein each noise value of the first set of noise values corresponds to a frequency level of the first set of frequency levels; means for determining a second set of noise values associated with the image produced by the image sensor using a fine search, the fine search comprising a second set of vectors, wherein the second set of vectors are determined based on a noise value difference between the target set of noise values and the first set of noise values, wherein each noise value of the second set of noise values corresponds to a frequency level of the first set of frequency levels; and means for updating a coefficient of the at least one bilateral low pass filter based on a difference between the second set of noise values and the first set of noise values; and means for adjusting a noise value of each pixel of the image sensor via applying the at least one bilateral low pass filter with the updated coefficient.

18. The apparatus of claim 17, wherein the means for updating the at least one bilateral low pass filter comprises means for updating the at least one bilateral low pass filter until the difference between the second set of noise values and the first set of noise values is within a matching range.

19. The apparatus of claim 18, wherein the matching range is different for each frequency level.

20. The apparatus of claim 17, wherein the first set of vectors and the second set of vectors each comprise a four-dimensional vector for each frequency level.

21. The apparatus of claim 20, wherein the image and the reference image are captured at the same distance from a color calibration target, the color calibration target comprising an arrangement of colored regions.

22. The apparatus of claim 21, wherein the coarse search is based on the following equation: ${.Math.}_{l = 1}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - {.Math.}_{l = 0}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, l, p} = 0$ where: n: a total number of colored regions in the color calibration target; m: a total number of frequency levels; l: a subject frequency level; k: a subject colored region; p: four-dimensional vector values.

23. The apparatus of claim 22, wherein the fine search is based on the following equation: $MSE (\frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, pl}) .Math.$ where: MSE: Mean Square Error.

24. The apparatus of claim 23, wherein the four-dimensional vector comprises four constants and each constant is a positive integer, and adjusting the noise level of each pixel of the image sensor comprises updating a noise reduction block of the image sensor.

25. The apparatus of claim 17, wherein: means for determining is a processor; means for calculating is the processor; means for updating is the processor; and means for storing is a memory.

26. A non-transitory, computer-readable medium comprising instructions executable by a processor of an apparatus, that causes the apparatus to: determine a first set of frequency levels of an image produced by an image sensor using a set of low pass filters, the set of low pass filters comprising at least one bilateral low pass filter; determine a first set of chrominance noise values and luminance noise values corresponding to each frequency level of the first set of frequency levels; calculate a target set of noise values associated with each frequency level of the first set of frequency levels, the target set of noise values calculated based on comparing the first set of chrominance and luminance noise values to a second set of chrominance and luminance noise values of a reference image, the reference image captured using a calibrated image sensor; determine a first set of noise values associated with the image produced by the image sensor using a coarse search, the coarse search based on a first set of vectors, wherein each noise value of the first set of noise values corresponds to a frequency level of the first set of frequency levels; determine a second set of noise values associated with the image produced by the image sensor using a fine search, the fine search comprising a second set of vectors, wherein the second set of vectors are determined based on a noise value difference between the target set of noise values and the first set of noise values, wherein each noise value of the second set of noise values corresponds to a frequency level of the first set of frequency levels; update a coefficient of the at least one bilateral low pass filter based on a difference between the second set of noise values and the first set of noise values; and adjust a noise value of each pixel of the image sensor via applying the at least one bilateral low pass filter with the updated coefficient.

27. The non-transitory computer-readable medium of claim 26, wherein the update of the at least one bilateral low pass filter is continued until the difference between the second set of noise values and the first set of noise values is within a matching range.

28. The non-transitory computer-readable medium of claim 27, wherein the matching range is different for each frequency level.

29. The non-transitory computer-readable medium of claim 26, wherein the first set of vectors and the second set of vectors each comprise a four-dimensional vector for each frequency level.

30. The non-transitory computer-readable medium of claim 29, wherein the image and the reference image are captured at the same distance from a color calibration target, the color calibration target comprising an arrangement of colored regions.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0018] FIG. 1 illustrates a series of filters for extracting levels of noise from an input image at each frequency level.

[0019] FIG. 2 illustrates an example of a region of high amplitude noise that is removed from the input image.

[0020] FIG. 3A illustrates a series of low pass filters for determining a high frequency component at each frequency level.

[0021] FIG. 3B illustrates an example block diagram of a system for comparing an image captured by an uncalibrated sensor with a reference image captured by a calibrated sensor, and updating the image signal processor.

[0022] FIG. 4 illustrates an example color calibration chart with sampling regions in each color square.

[0023] FIG. 5 illustrates the results of wavelet decomposition of an input figure at three different frequency levels.

[0024] FIG. 6 illustrates two example images, one representing an input image prior to denoise, and the other representing an image after denoise. Noise values for the luminance and chrominance values for each image are provided in tables.

[0025] FIG. 7 illustrates two charts, one representing the luminance noise values associated with the input image and the reference image, the other chart representing the chrominance noise values associated with the input image and the reference image.

[0026] FIG. 8 is a flow chart that illustrates an example process for calibrating an image sensor.

[0027] FIG. 9 is a flow chart that illustrates an example process for determining bilateral low pass filter values for image sensor calibration.

DETAILED DESCRIPTION

[0028] The following detailed description includes certain illustrative embodiments of the subject matter. However, the disclosed subject matter can be embodied in a multitude of different ways. It should be apparent that the aspects herein may be embodied in a wide variety of forms and that any specific structure, function, or both being disclosed herein is merely representative. Based on the teachings herein one skilled in the art should appreciate that an aspect disclosed herein may be implemented independently of any other aspects and that two or more of these aspects may be combined in various ways. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth herein. In addition, such an apparatus may be implemented or such a method may be practiced using other structure, functionality, or structure and functionality in addition to, or other than one or more of the aspects set forth herein Disclosed herein are automatic noise and texture optimization of image sensor.

[0029] The examples, systems, and methods described herein are described with respect to techniques for calibrating image sensors (e.g., for noise and texture). The systems and methods described herein may be implemented on a variety of platforms and operating systems.

Quantitative Metrics for Noise and Texture

[0030] This following solution involves performing automatic noise and texture optimization by using Wavelet decomposition high frequency component sigma (standard deviation) and a specialized search engine for Wavelet denoise. In one embodiment, the noise and texture of an image may be quantified by calculating different levels of frequency in an image, and determining the luminance (or “luma”) and chrominance (or “chroma”) noise values for each level of frequency in the image. FIGS. 1 and 3 illustrate examples of how the high frequency component sigma can be extracted from an image. The image may be captured using an image sensor before the sensor has been calibrated to adjust for noise levels of images the sensor produces. The image may be of a Macbeth ColorChecker chart 400, or any other color calibration medium containing spectral reflectance's. Using a calibrated sensor to capture an image of the Macbeth ColorChecker chart 400 may yield a reference image that may provide a target noise level. For example, the levels of frequency of the uncalibrated image may be determined by using each level produced by high frequency components as determined by a Wavelet decomposition process.

[0031] FIG. 1 illustrates a method for extracting 100 a noise value 105a-105n for each frequency level using a set of low pass filters, where level 1 is a highest frequency level, and level n is the lowest frequency level. In some embodiments, level n may be a sixth frequency level. Filters 1 through filter n present a method of wavelet decomposition, where the output of filter 1 is the input of filter 2, and so on. The output of filter 1 may be summed with the input image, resulting in noise and edge values at the first frequency level. The output of filter one may also be summed with the output of filter 2 in order to generate the noise and edge values at the second frequency level. In the same way, noise and edge values at each frequency level may be determined for a thresholding stage. The method for extracting 100 a noise value 105a-105n may be applied to the image as a whole in an RGB state, or it may be applied to each of the Y, Cb, and Cr components individually.

[0032] Still referring to FIG. 1, the thresholding blocks may remove a high amplitude area of pixel energy. This is done because the high amplitude region represents an edge, while the low amplitude region represents noise. The determination of the high amplitude area may be user configurable. FIG. 2 illustrates an example of amplitude removal 200. The y-axis of the Cartesian coordinate system represents a number of pixels, while the x-axis represent the amplitude of pixel energy. A threshold amplitude value 210 may be set in order to determine the portion of amplitude removed 205. The threshold amplitude value 210 may be user configurable or may be determined as a value that is normalized to a percentage based on the energy of the of the image sensor. For example, the threshold value may be set to a value that corresponds to a percentage of the average energy in the pixels of the sensor.

[0033] Still referring to FIG. 1, a first noise value 105a, or first noise extracted, may be calculated by subtracting the low amplitude signal that is output from the thresholding stage, from the input image. This first noise value 105a may correspond to the first frequency level. Similarly, the low amplitude signal that is output from the thresholding stages following filters 2 through n may be subtracted from the input image to generate the noise value 105b-105n for each corresponding frequency level 2 through n. The noise values 105a-105n are summed to generate the output image.

[0034] FIG. 3A illustrates in further detail how the standard deviation (high frequency component, “sigma”) may be determined 300 using the output image of FIG. 1. A series of low pass filters may be used to generate a high frequency component (σ) at each frequency level. For example, the level 1 low pass filter may pass a range of frequencies from the output image that are lower than a threshold value. The low frequency values that are passed from the level 1 low pass filter may be subtracted from the output image, and a standard deviation may be calculated for the remaining values. The standard deviation values may be the average of each pixel in a region of interest. The region of interest may be a region of a color square in a MacBeth ColorChecker chart 400. The standard deviation values may also be referred to as the high frequency component for a particular frequency level. The standard deviation values may be determined for each of the Y, Cb, and Cr components of the image output by the method for extracting 100 a noise value 105a-105n in FIG. 1.

[0035] FIG. 3B illustrates a calibrated camera 345 capturing an image of a ColorChecker chart 400, the captured image being a reference image. Also illustrated is an uncalibrated camera 350 capturing an image of a ColorChecker chart 400, the captured image being a source image of the uncalibrated image sensor 324. The calibrated camera 345 may include a lens 310. The lens 310 of a camera 345 captures the light from the subject and brings it to a focus on the image sensor. In general terms, the two main optical parameters of a photographic lens are maximum aperture and focal length. The focal length determines the angle of view, and the size of the image relative to that of the object (subject) for a given distance to the subject (subject-distance). The focal length may be adjusted using an actuator 312. The actuator 312 may be adjusted by the processor 307 based on an auto-focus algorithm such as a contrast auto-focus, or a manual focus. The maximum aperture (f-number, or f-stop) limits the brightness of the image and the fastest shutter speed usable for a given setting (focal length/effective aperture), with a smaller number indicating that more light is provided to the focal plane which typically can be thought of as the face of the image sensor in a simple digital camera. The actuator 312 may also adjust the aperture of the lens 310 using an auto focus algorithm or a manual setting. The calibrated camera may also include a working memory 370 for storing image capturing algorithms such as auto-focus algorithms. A second storage 375 may also be included for storing images captured by the camera, or for storing calibration parameters. The processor 307 coupled to the calibrated image sensor 314 may communicate processed image data to device 306. In an optional embodiment, the calibrated image sensor 314 may communicate unprocessed image data from the image sensor 324 to the device 306.

[0036] Still referring to FIG. 3B, the uncalibrated camera 350 may also include a lens 320. It is noted that the arrangement of FIG. 3B is for illustrative purposes, and that one skilled in the art would understand that both cameras 345, 350 are not necessarily required. For example, the calibrated image sensor 314 and the uncalibrated sensor 324 may share the same camera housing. For example, a single camera including the lens, actuator, processor, working memory, and second storage may be used for both the uncalibrated image sensor 324 and the calibrated image sensor 314. In another embodiment, two different camera system may be used to calibrate the image sensor, each camera capturing an image of the same ColorChecker chart 400. In such an example, the lens 320 of the uncalibrated camera 350 captures the light from the subject and brings it to a focus on the image sensor. The focal length may be adjusted using an actuator 322. The actuator 322 may be adjusted by the image sensor processor 327 based on an auto-focus algorithm such as a contrast auto-focus, or a manual focus. The actuator 322 may also adjust the aperture of the lens 320 using an auto focus algorithm or a manual setting. The calibrated camera may also include a working memory 380 for storing image capturing algorithms such as auto-focus algorithms. A second storage 385 may also be included for storing images captured by the camera, or for storing calibration parameters. The image sensor processor 327 coupled to the uncalibrated image sensor 324 may communicate processed image data to the device 306. In an optional embodiment, the uncalibrated image sensor 324 may communicate unprocessed image data to the device 306.

[0037] Still referring to FIG. 3B, the device 306 may transmit data to, and receive data from, the uncalibrated camera 350. For example, the image sensor processor 327 of the uncalibrated sensor may send image data to the device 306 for evaluation of the data with respect to data received from a calibrated image sensor. The intermediary processor 308 of the device 306 may then execute a program that modifies or modulates the noise reduction block of the uncalibrated image sensor 324 or the image sensor processor 327, or both. The intermediary processor may receive unprocessed or processed image data from the calibrated camera. This image data may be used as reference image data. The reference image data may be passed through a series of low pass filters in a low pass filter bank 301. The low pass filter bank 301 may be used to extract different noise components from the reference image data using the technique illustrated in FIG. 1. The reference image data may also be passed through a series of dynamic low pass filters, the dynamic low pass filters being part of a dynamic low pass filter bank 302. The intermediary processor may determine the low pass filter coefficient of each of the dynamic low pass filters. In another embodiment, the low pass filter coefficient may be user configurable. The dynamic low pass filter bank 302 may be used to extract different noise components from the reference image data using the technique illustrated in FIG. 2. The intermediary processor 306 may also include a chrominance/luminance extraction unit 303 for extracting the chrominance and luminance components from each of the frequency levels derived from the techniques illustrated in FIG. 1 and FIG. 3A. The chrominance/luminance extraction unit 303 may separate the chrominance and luminance components from each frequency level of the received image data, and store the components in the memory 305. The intermediary processor 306 may also include a comparator/search unit 304 for comparing the luminance and chrominance components of image data received from an uncalibrated image sensor 324 with image data received from a calibrated image sensor 314. The comparator/search unit 304 may also determine the search vectors for the coarse search and the fine search, discussed below. The intermediary processor 306 may also include a memory 305 for storing data received and generated by the intermediary processor 306.

[0038] Still referring to FIG. 3B, the cameras 345, 350 may connect to the intermediary processor 306 using an I/O interface 309. In some embodiments, the I/O interface 309 of the cameras 345, 350 may be configured to allow the connection of each camera to one or more external devices, such as a computer, a video monitor, or the intermediary processor 306. For example, the I/O interface 309 may include a USB connector, an HDMI connector, or the like. In some embodiments, the I/O interface 309 may be configured to transfer information between the cameras 345, 350 and the connected external device. In some embodiments, the I/O interface 309 may be configured to transfer information wirelessly (for example via infrared or Wi-Fi). In some embodiments, the intermediary processor 306 described above may be configured to control one or more aspects of the camera, including settings associated with the actuator, image sensor, processor, and working memory, navigating the options and menus of the camera, viewing and/or modifying captured images, or rendering a refocused image on a tilted image plane after capturing an image.

[0039] FIG. 4 illustrates a Macbeth ColorChecker chart 400 that may be used as a reference image or an input image for determining a target high frequency component value. The wavelet high frequency components of the source image may be measured or estimated based on the reference Macbeth ColorChecker chart 400 image. A pre-tuned adaptive sharpening filter (ASF) may also be used to determine the high frequency components. The output of the AIO module includes, but is not limited to, how much noise, and the character of the noise in the source image (noise_profile), the LPF characteristics of the wavlet decomposition (edge_softness), and the tuning parameters for removing the noise (denoise_scale). Denoise_weight is an additional parameter that may be used during optimization.

[0040] Still referring to FIG. 4, each square (405a-405x) of the Macbeth ColorChecker chart 400 may be of a different color. The reference image and the source image may be of the same chart for frequency noise comparison purposes. The source image may include regions for sampling (410a-410x) the noise levels.

[0041] FIG. 5 illustrates an example of noise and edge detection resulting from the low pass filtering found in FIG. 1. The first FIG. 505 represents the input image with low frequency values removed from it. In other words, the first FIG. 505 represents the results of the wavelet decomposition for frequency level one. The second FIG. 510 represents the wavelet decomposition results for frequency level two. The third FIG. 515 represents the wavelet decomposition results for frequency level five. Each of the three images are captured using an uncalibrated sensor and are of a Macbeth ColorChecker chart 400.

[0042] FIG. 6 illustrates an example set of images with corresponding Wavelet decomposition frequency levels and the luma and chroma values for each frequency level. The first image 605 a first table 605t corresponds to the first image 605. The first image 605 is a region of interest, or a section of a larger image captured of a MacBeth ColorChecker chart 400. The image is captured using an uncalibrated sensor. The image may be converted from an RGB to a YCbCr color space. The first table 605t shows six frequency levels, each frequency level having a row in a column labeled “Level.” The frequency levels descend from frequency level 1 being the highest frequency, to frequency level 6 being the lowest frequency. The luma values corresponding to each level are shown in a column labeled “Y.” The chroma values corresponding to each level are shown in two columns labeled “Cb” and “Cr.” Cb and Cr are representative of the two chroma channels associated with the first image 605, and Y representative of the luma channel. Note that the Y values of each frequency level have significant value deltas between other frequency levels. The Cb and Cr values also have significant value deltas.

[0043] Still referring to FIG. 6, a second image 610 and a second table 610t that corresponds with the second image 610 illustrates a result of the same image sensor after the de-noise operation. The table shows six frequency levels, each frequency level having a row in a column labeled “Level.” The luma values corresponding to each level are shown in a column labeled “Y.” The chroma values corresponding to each level are shown in two columns labeled “Cb” and “Cr.” Note that the texture of the first image 605 is more rough than the texture of the relatively smooth second image 610. Note that the luma and chroma values across the six frequency levels in the first table 605t have a greater variability relative to each other frequency level than the luma and chroma values in the second table 610t.

[0044] FIG. 7 illustrates the source and target noise profiles for an example image using the luma and chroma channel values. FIG. 7 includes two graphs. In both graphs, the y-axis represents a noise value, while the x-axis represents the six frequency levels. Both graphs illustrate the noise level differences between a source image and a target image. The first graph 700 has two lines. The first line is representative of the source image luma values for each of the six frequency values and is labeled “Y Source” 700a. The Y source 700a line represents the initial standard deviation values of the luma component of an image captured by an uncalibrated image sensor. The second line is representative of a target luma value for each frequency level, and is labeled “Y Target” 700b. The target value represents a target noise value for optimization of the source image, and may be generated from a reference image. Note the arrows indicating that the luma noise values of the source image need to be reduced to meet the target luma noise values. The search algorithm discussed further below may bring the Y source 700a values closer to the Y target 700b values. As noted above, frequency level 1 is the highest frequency level. As such, frequency level 1 will generally have the largest disparity between the source and target noise values.

[0045] With continued reference to FIG. 7, the second graph 705 has two lines. The first line is representative of first channel chroma values (Cr) for each of the six frequency values of the source image and is labeled “Cr Source” 705a. The second line is representative of a target chroma values for each frequency level, and is labeled “Y Target” 705b. The target values represent noise values for optimization of the source image at each frequency level. As noted above, frequency level 1 is the highest frequency level. As such, frequency level 1 may generally have the largest disparity between the source and target noise values. In the second graph 705, the difference between the source values and the target values are approximately even for frequency levels one and two.

[0046] With continued reference to FIG. 7, the two graphs 700, 705 provide a noise profile for a source image and a reference image. The noise profile is a standard deviation value of each wavelet high frequency component measured from a wavelet input buffer. The noise profile defines the overall strength of de-noise for each frequency level and for the luma and chroma channels. For example, the noise profile may be represented as equation (1) below (sigma represents standard deviation):

[00003] $\begin{matrix} \frac{σ_{.Math. source} - σ_{.Math. target}}{Ave .Math. .Math. σ_{target}} & (1) \end{matrix}$

Where:

[0047] σ.sub.source: standard deviation value of the source image at a particular frequency level, [0048] σ.sub.target: standard deviation value of the target image at a particular frequency level, [0049] Ave σ.sub.target: average of the standard deviation values across each of the frequency levels.

[0050] FIG. 8 is a flowchart 800 illustrating a series of steps associated with certain aspects of the present disclosure. At block 805, the apparatus receives image data captured by an uncalibrated digital image sensor. In some aspects, the image data may be of a scene that includes the MacBeth ColorChecker chart 400. In other aspects, the image data may be of any other scene that matches a reference image. The image data may be in any color space, for example RGB, CMYK, L*a*b*, etc. As such, the image sensor may include a Bayer color filter array color pattern, or other arrangement of RGB, RGBW, and CYGM color filters. In one embodiment, the color filters may be different sizes relative to the pixel size. For example, a single pixel may have multiple color filters that overlay the photosensitive region or regions of the pixel. A processor may determine a luma component (Y), and a blue-difference (Cb) and red-difference (Cr) chroma components at each of a series of frequency levels. The series of frequency levels may correspond to a set of recursive noise levels that are subtracted from the input image. For example, a first noise level may be generated with a first low pass filter and subtracted from the input image. A second noise level may be generated by taking the first noise level and passing it through a second low pass filter. In another example, the frequency levels may be determined using a non-recursive function. For example, each frequency level may be correspond to a set of noise levels determined independent of each other using low pass filters that output frequencies according to a step function at each successive low pass filter.

[0051] At block 810, the apparatus may determine the high frequency component or the sigma value (σ) at each frequency level. The high frequency component for each frequency level may be determined using a series of low pass filters to filter a series of frequencies from the output image of FIG. 1. FIG. 3A illustrates how the high frequency component for each frequency level is determined.

[0052] At block 815, the apparatus may determine low pass filter values based on noise values at each frequency level using an initial search vector. The low pass filters of FIG. 1 may be dynamic low pass filters, the quotients of which can be modulated or changed. The initial search vector may be any four value vector. For example, an initial search vector for the coarse search algorithm may be {0.1, 0.1, 0.1, 0.1}. The search algorithm may use a pre-defined noise weight parameter during optimization. This value may, for example, be the system's default value. Another parameter, may be decided based on the final edge softness and histogram of the high frequency component of each frequency level. Overall denoise parameters may be optimized based on pure four-dimension edge softness parameters or vectors. A problem with current methods is the local minimum problem, where the final noise profile often results in large errors because the first level denoise influences all subsequent frequency levels. For example, the second, third, and fourth frequency levels are affected by the first level frequency denoise parameters. The search algorithm disclosed herein overcomes the local minimum issue.

[0053] In one embodiment, the coarse search algorithm may include equation (2):

[00004] $\begin{matrix} {.Math.}_{l = 1}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - {.Math.}_{l = 0}^{m} .Math. .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, l, p} = 0 & (2) \end{matrix}$

Where:

[0054] n: number of patches in the color checker chart (e.g., number of color regions in a Macbeth ColorChecker chart); [0055] m: number of Wavelet decomposition frequency levels; [0056] l: subject wavelet level (e.g., Wavelet decomposition frequency level 4); [0057] k: subject patch number (e.g., Macbeth ColorChecker chart patch 4); [0058] p: values of the four constants in the four-dimensional vector (e.g., {1, 1, 1, 1}, can be any number between 0 and infinity.
The vector values representing initial vector values for a coarse search may be arbitrary and user selectable. The resulting curve may be monotonical, as there is only one root.

[0059] At block 820, a standard deviation value is calculated at each frequency level for each of Y, Cb, and Cr component of the output image based on the determined low pass filter values. FIG. 1 illustrates this process. At block 825, a processor may compare the calculated standard deviation values with the target standard deviation values. The target standard deviation values may be determined from the target image or reference image. A difference between the calculated standard deviation values and the target standard deviation values may be calculated at each frequency level.

[0060] At block 830, the bilateral filter values may be updated based on a new search vector. The new search vector may be a unit vector. The new search vector is based on the initial search vector used in the coarse search, and is part of a fine search. The updated bilateral filter values may be based on the difference between the calculated standard deviation values and the target standard deviation values in the coarse search at each frequency level. The bilateral filter values may be updated several times. In such a case, a new search vector is used. Table 1 below illustrates the search vectors used for each frequency level in the fine search.

TABLE-US-00001 Frequency Level Search Vector 1 1 0 0 0 2 0 1 0 0 3 0 0 1 0 4 0 0 0 1

[0061] Table 1 illustrates an example set of search vectors for each frequency level, using a fine search. In this example, there are four frequency levels, numbered 1-4. The search vector associated with frequency level 1 is {1, 0, 0, 0}, the search vector associated with frequency level 2 is {0, 1, 0, 0}, the search vector associated with frequency level 3 is {0, 0, 1, 0}, and the search vector associated with frequency level 4 is {0, 0, 0, 1}. The fine search is a specialized search method for Wavelet denoise architecture. Unlike current methods, the fine search uses a separate target for each search vector, eliminating a local minimum limitation. For each search vector, equation (3) below defines the cost function to find pl, which is edge softness for 1 level, where equation (3) is equal to zero or a minimum value.

[00005] $\begin{matrix} MSE (\frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, pl}) & (3) \end{matrix}$

Where:

[0062] l: subject wavelet level (e.g., Wavelet decomposition frequency level 4); [0063] k: subject patch number (e.g., Macbeth ColorChecker chart patch 4); [0064] MSE: Mean Square Error.
When a first round of all four search vectors is completed, the cost function equation (4) below may be used as a stop condition, using the delta of all four frequency levels. Equation (4) may represent a fine search.

[00006] $\begin{matrix} MSE ({.Math.}_{l = 0}^{m} .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{target, k, l} - {.Math.}_{l = 0}^{m} .Math. \frac{1}{n} .Math. {.Math.}_{k = 0}^{n} .Math. .Math. σ_{source, k, pl, p_{0 .Math. ∼ .Math. m}}) & (4) \end{matrix}$

Where:

[0065] MSE: Mean Square Error.
Due to the nature of the wavelet de-noise, the above meets convergence criteria.

[0066] At block 835, the processor may determine whether the updated bilateral filters produce a standard deviation value, or high frequency component, that matches the target values of the reference image. The match criteria may be modified based on the image sensor being used or based on user configuration. For example, the image sensor may differ from the sensor used to capture the reference image by way of the color filter array configuration, size of pixels, number of pixels, or any other difference in image sensor configuration. The user may also determine that a match exists when the standard deviation difference between the source image and the target image at each frequency level is within 10%. This is an example only, as other percentage values may be used. In another embodiment, the user may set a matching tolerance for each frequency level. For example, the user may set a matching tolerance of 30% at the first frequency level, a matching tolerance of 20% at the second frequency level, a matching tolerance of 10% at the third frequency level, etc. In this case, a match exists when the standard deviation difference between the source image and the target image at each frequency level is within the range prescribed to each frequency level. This is an example only, as other percentage values may be used. If the matching criteria is not met, the fine search vectors may be reused with the most recently used bilateral filters. In another embodiment, the search vectors may be updated to new values.

[0067] At block 840, if it is determined that the matching criteria is met for all frequency levels, the sensor may be calibrated based on the bilateral filter values. The image sensor being calibrated will generally have several resources or “blocks” for adjusting image parameters related to the image sensor. For example, the image sensor may include a color/contrast block, a brightness block, a noise block, an offset block, a gamma block, etc. Each block may be calibrated using a look-up table that correlates the bilateral filter coefficient to a parameter value in the block. For example, a weak bilateral filter coefficient may filter a low amplitude ratio (e.g., −100 to −70 dB) while a strong bilateral filter coefficient may filter a high amplitude ratio (e.g., 70 to 100 dB). The look-table may include parameter values for each block, where each parameter value, or range of parameter values, are associated with specific bilateral filter coefficient, or range of bilateral filter coefficients.

[0068] In one embodiment, the noise block of an image sensor may modulate a high-gain column or row amplifier for a column of pixels for increasing or decreasing the sensitivity of the pixels of that column or row. Another technique for adjusting noise levels of an image sensor may be by adjusting well capacity for each pixel of the image sensor, which modulates noise, leading to a modulated signal-to-noise ratio and/or a nonlinear sensor response. Still another technique for increasing

[0069] FIG. 9 illustrates an example method 900 for calibrating an image sensor. In step 905, a processor may determine a first set of frequency levels of a source image using a set of low pass filters. The low pass filters may be fixed value low pass filters or a bilateral, dynamic low pass filter. In step 910, the processor may determine a first set of chrominance noise values and luminance noise values for each frequency level. The luminance and chrominance values may be determined using a histogram approach. In one approach, the chrominance and luminance values may be determined based on the color filter array (CFA) pattern of the image sensor being calibrated. Equation 5 below enables expression of the CFA pattern y[n] in terms of the luminance component and chrominance components, n=[n.sub.1,n.sub.2] where n represents an address to a spatial coordinate on an two-dimensional square lattice having a horizontal position (n.sub.1) and a vertical position (n.sub.2). Using the Bayer CFA pattern as an example, a data value at point n can be represented by the following equation:

y[n]=l[n]+((−1).sup.n.sup.1−(−1).sup.n.sup.2)c.sub.i[n]+(−1).sup.n.sup.1.sup.+n.sup.2c.sub.2[n] (5)

Where:

[0070] y[n]: CFA data value at point n=[n.sub.1,n.sub.2], [0071] l[n]: luminance value at point n=[n.sub.1,n.sub.2], [0072] c.sub.1[n]: chrominance value at point n=[n.sub.1,n.sub.2], [0073] c.sub.2[n]: chrominance value at point n=[n.sub.1,n.sub.2].

[0074] In step 915, the processor calculates a target set of noise values associated with each frequency level. The target noise values may comprise a set of target chrominance and luminance noise values the sensor being calibrated is to be adjusted to match. The target noise values may be calculated based on the type of sensor being calibrated, the number of pixels, the CFA pattern, etc. The target set of noise values may also be determined by looking up known values in a memory component. The target set of noise values may also be determined by capturing an image using a calibrated image sensor and using the wavelet decomposition technique illustrated in FIG. 1 to calculate the target set of noise values.

[0075] In step 920, the processor may determine a first set of noise values associated with the image produced by the image sensor using a coarse search. The coarse search may be an arbitrary four dimensional vector of a region of an image captured by the uncalibrated sensor, or the sensor being calibrated. A search vector may be used for each of the luma and chroma channels. For example, the luma channel may use the following search vector: {0, 0.25, 0.5, 0.5} while the chroma channels may use the following vector: {0,0,0,0}. In another example, an initial search vector for the coarse search algorithm may be {0.1, 0.1, 0.1, 0.1} for each of the chroma and luma channels. The search algorithm may use a pre-defined noise weight parameter during optimization. This value may, for example, be the system's default value. In a situation that includes a bright light luma channel, using a search vector of {0,0,0,0} may provide more texture on a high frequency object. Similarly, for a low light luma channel, a search vector of {0, 0.25, 0.5, 0.5} because {0,0,0,0} may cause artifacts.

[0076] In step 925, the processor may determine a second set of noise values associate with the image produced by the image sensor using a fine search. The fine search may be used to bring the four vector values closer to matching the target noise values. As with the coarse search vectors, the fine search vectors may be used to generate a σ value that matches the target σ of each frequency level, and each channel.

[0077] In step 930, the processor may update the at least one bilateral low pass filter based on a difference between the first set of noise values and the second set of noise values. The bilateral low pass filters may be updated based on the σ value associated with either of the coarse or fine search vectors using a look-up table. The bilateral filters may also be updated based on each of the frequency levels associated with the vector values for the luma and chroma channels.

Other Considerations

[0078] Implementations disclosed herein provide systems, methods and apparatus for submicron wafer alignment. One skilled in the art will recognize that these embodiments may be implemented in hardware, software, firmware, or any combination thereof.

[0079] Information and signals disclosed herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0080] The various illustrative logical blocks, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0081] The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices such as general purposes computers, wireless communication device handsets, or integrated circuit devices having multiple uses including application in wireless communication device handsets and other devices. Any features described as devices or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, performs one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may comprise memory or data storage media, such as random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, magnetic or optical data storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer, such as propagated signals or waves.

[0082] Processor(s) in communication with (e.g., operating in collaboration with) the computer-readable medium (e.g., memory or other data storage device) may execute the instructions of the program code, and may include one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, an application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general purpose processor may be a microprocessor; but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Accordingly, the term “processor,” as used herein may refer to any of the foregoing structure, any combination of the foregoing structure, or any other structure or apparatus suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software or hardware configured for imaging, encoding, and/or decoding. Also, the techniques could be fully implemented in one or more circuits or logic elements.

[0083] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of inter-operative hardware units, including one or more processors as described above, in conjunction with suitable software and/or firmware.

[0084] As used herein, the term “plurality” denotes two or more. For example, a plurality of components indicates two or more components. Conditional language used herein, such as, among others, “can,” “might,” “may,” “e.g.,” and the like, unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or states. Thus, such conditional language is not generally intended to imply that features, elements and/or states are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without author input or prompting, whether these features, elements and/or states are included or are to be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively, in an open-ended fashion, and do not exclude additional elements, features, acts, operations, and so forth. Also, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. The term “determining” encompasses a wide variety of actions and, therefore, “determining” can include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” can include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” can include resolving, selecting, choosing, establishing and the like.

[0085] The phrase “based on” does not mean “based only on,” unless expressly specified otherwise. In other words, the phrase “based on” describes both “based only on” and “based at least on.”

[0086] Headings are included herein for reference and to aid in locating various sections. These headings are not intended to limit the scope of the concepts described with respect thereto. Such concepts may have applicability throughout the entire specification.

[0087] Although the foregoing has been described in connection with various different embodiments, features or elements from one embodiment may be combined with other embodiments without departing from the teachings of this disclosure. However, the combinations of features between the respective embodiments are not necessarily limited thereto. Various embodiments of the disclosure have been described. These and other embodiments are within the scope of the following claims.

METHODS AND APPARATUS FOR AUTOMATED NOISE AND TEXTURE OPTIMIZATION OF DIGITAL IMAGE SENSORS

Inventors

Cpc classification

Classification Explorer

G06T5/00

PHYSICS

Classification Explorer

G06T5/50

PHYSICS

Classification Explorer

H04N23/10

ELECTRICITY

Classification Explorer

G06T2207/20016

PHYSICS

Classification Explorer

G06T5/20

PHYSICS

Classification Explorer

G06T2207/10024

PHYSICS

Classification Explorer

H04N17/002

ELECTRICITY

Classification Explorer

G06T5/002

PHYSICS

Classification Explorer

G06T2207/20182

PHYSICS

Classification Explorer

H04N25/60

ELECTRICITY

Classification Explorer

G06T2207/20028

PHYSICS

Classification Explorer

G06F18/22

PHYSICS

International classification

Classification Explorer

H04N5/357

ELECTRICITY

Classification Explorer

G06K9/52

PHYSICS

Classification Explorer

H04N9/04

ELECTRICITY

Classification Explorer

G06K9/46

PHYSICS

Classification Explorer

G06K9/62

PHYSICS

Classification Explorer

G06T5/00

PHYSICS

Classification Explorer

H04N17/00

ELECTRICITY

Abstract

Claims

Description