Stray light compensation techniques for an infrared camera
09979903 ยท 2018-05-22
Assignee
Inventors
Cpc classification
International classification
Abstract
Various techniques are provided for a stray light compensation method for an infrared (IR) camera. For example, a stray light compensation method includes: capturing an IR image of a scene by an IR camera, generating a fixed pattern noise estimate FPNest.sub.t0 for time t0 using the captured IR image and a stray light model associated with the IR camera, and performing a fixed pattern noise (FPN) compensation of the captured IR image based on said FPNest.sub.t0 to obtain a stray light compensated IR image. The fixed pattern noise estimate may be generated through operations in a frequency domain representation of the captured IR image and the stray light model according to one or more embodiments.
Claims
1. A method comprising: capturing an infrared (IR) image of a scene; determining a first IR image frequency pattern associated with the captured IR image, wherein the first IR image frequency pattern comprises first frequency components and associated first frequency component amplitudes; generating a second IR image frequency pattern based at least on the first IR image frequency pattern and a frequency pattern associated with a stray light model, wherein the frequency pattern associated with the stray light model comprises second frequency components and associated second frequency component amplitudes, wherein the second frequency components comprises a set of principal components and a set of non-principal components, and wherein the generating the second IR image frequency pattern comprises adjusting the first frequency component amplitudes of the first IR image frequency pattern corresponding to the set of non-principal components to obtain the second IR image frequency pattern; generating a fixed pattern noise (FPN) estimate FPNest.sub.t0 for time t.sub.0 based at least on the second IR image frequency pattern; and performing an FPN compensation of the captured IR image based on said FPNest.sub.t0 to obtain a stray light compensated IR image.
2. A method comprising: capturing an infrared (IR) image of a scene; generating a fixed pattern noise (FPN) estimate FPNest.sub.t0 for time t.sub.0 using the captured IR image and a stray light model, wherein the generating the fixed pattern noise estimate-comprises: performing a transform on said captured IR image to a frequency domain thereby obtaining an IR image frequency pattern, wherein the IR image frequency pattern comprises frequency components with associated frequency component amplitudes and a center frequency, and wherein the stray light model and the transformed IR image have frequency patterns with the same number of and corresponding frequency components; generating a truncated IR image frequency pattern by setting the frequency component amplitudes of the IR image frequency pattern corresponding to non-principal components of the stray light model to zero; performing an inverse frequency transform on the truncated IR image frequency pattern to obtain a fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0; and determining the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 based on said fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0; and performing an FPN compensation of the captured IR image based on said FPNest.sub.t0 to obtain a stray light compensated IR image.
3. The method of claim 2, further comprising: determining whether the frequency components of the IR image frequency pattern fulfill a first condition based on the stray light model, wherein the first condition is that a correlation measure based on the frequency pattern of the transformed IR image and the stray light model exceeds a predetermined correlation threshold value, wherein the generating the truncated IR image frequency pattern, the performing the inverse frequency transform, and the determining the fixed pattern noise estimate are performed upon determination that the first condition is fulfilled.
4. The method of claim 1, further comprising: generating an adjusted version of the fixed pattern noise estimate FPNest.sub.t0, by adjusting the fixed pattern noise estimate FPNest.sub.t0 iteratively until a condition is fulfilled, wherein the performing the FPN compensation of the captured IR image is based on said adjusted version of the FPNest.sub.t0.
5. The method of claim 2, wherein determining the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 is further based on previously determined fixed pattern noise estimate deltas FPNest.sub.t-M, . . . , FPNest.sub.t-2, FPNest.sub.t-1.
6. The method of claim 1, wherein the first IR image frequency pattern and the frequency pattern of the stray light model have the same number of and corresponding frequency components.
7. The method of claim 1, wherein the determining the first IR image frequency pattern comprises determining a frequency domain representation of the captured IR image to obtain the first IR image frequency pattern.
8. The method of claim 7, wherein the determining the frequency domain representation comprises performing a transform on the captured IR image to a frequency domain to obtain the frequency domain representation.
9. The method of claim 1, wherein the adjusting comprises setting the first frequency component amplitudes of the first IR image frequency pattern corresponding to the set of non-principal components associated with the stray light model to zero to obtain the second IR image frequency pattern.
10. The method of claim 1, wherein each second frequency component of the stray light model is a principal component in the set of principal components or a non-principal component in the set of non-principal components based at least on the corresponding second frequency component amplitude of the second frequency component and one or more threshold values.
11. The method of claim 1, further comprising: determining a fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0 based at least on the second IR image frequency pattern, wherein the fixed pattern noise estimate FPNest.sub.t0 is based at least on the fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0.
12. The method of claim 11, wherein the determining the fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0 is based at least on an inverse frequency transform of the second IR image frequency pattern.
13. The method of claim 11, wherein the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 is further based on previously determined fixed pattern noise estimate deltas FPNest.sub.t-M, . . . , FPNest.sub.t-2, FPNest.sub.t-1.
14. The method of claim 1, further comprising: determining whether the first frequency components of the first IR image frequency pattern fulfill a first condition based on the stray light model, wherein the first condition is that a correlation measure based on the first IR image frequency pattern associated with the captured IR image and the frequency pattern associated with the stray light model exceeds a correlation threshold value, and wherein the generating the second IR image frequency pattern is performed upon determination that the first condition is fulfilled.
15. A system configured to perform the method of claim 1, the system comprising: at least one processor configured to perform the determining the first IR image frequency pattern, the generating the second IR image frequency pattern, the generating the FPN estimate FPNest.sub.t0 for time t.sub.0, and the performing the FPN compensation; an IR camera configured to perform the capturing the IR image of the scene; and a memory circuit configured to store the stray light model.
16. The system of claim 15, wherein the IR camera comprises a housing enclosing the at least one processor and the memory.
17. A system configured to perform the method of claim 2, the system comprising: at least one processor configured to perform the generating the FPNest.sub.t0 for time t.sub.0 and the performing the FPN compensation; an IR camera configured to perform the capturing the IR image of the scene; and a memory circuit configured to store the stray light model.
18. The system of claim 17, wherein the IR camera comprises a housing enclosing the at least one processor and the memory.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8) Embodiments of the invention and their advantages are best understood by referring to the detailed description that follows.
DETAILED DESCRIPTION
(9) Detectors incorporated in a detector matrix of an IR camera do not behave equally but have variations in amplification and offset. In order to manage these variations, so-called gain and offset maps are recorded and stored preferably already in production. With the aid of the gain map, corrections are made during operation for amplification variations of the individual detectors in a matrix. Correspondingly, the offset map is used, during operation, for the parallel displacement of the detector signals of integral detectors such that the amplification curves of the detectors substantially coincide. In order to further clarify the principles behind gain and offset mapping, reference is made to our published US patent application US 2011/0164139 A1.
(10) In
(11) With reference to a schematic flow diagram shown in
(12) In connection with the production of an IR camera 3, gain and offset maps 5, 6 are expediently recorded. The block 7 in
(13) In order to eliminate low-frequency contributions from the scene and from the optics in the defocused state, according to block 9 a high-pass filtering is carried out. An offset map which from the noise aspect can favourably manage the high-frequency noise is thereby produced.
(14) In order to manage stray light, stray light images are measured in production for different temperatures against flat radiators and stored in a memory, such as the memory 4 shown in
(15) A block 12 illustrates an image in the IR camera 3, which image is accessible during operation.
(16) In order to establish the magnitude of the stray light, a matching of stored stray light images takes place, block 11, with the particular direct image, block 12, according to the principles described below.
(17) Starting from the particular direct image, a selection is first made. The selection can be made such that one or more lines in the image, a few points or the whole of the image are processed. By limiting the selection to some lines or a few points, the calculation time can be kept down. It is especially proposed to choose the two diagonals in the image. Other suitable choices can be some parallel lines and some horizontal lines.
(18) For the sake of clarity, a selection which is constituted by a single horizontal line 13 shown in a block 14 in order to show the principle behind the stray light compensation is described below. Such a horizontal line, like other selection examples, also contains scene content which has nothing to do with the stray light. In order to separate the stray light information from the rest of the image, a low-pass filtering of the horizontal line, see block 15, is first carried out. After this, it can be checked whether the low-pass filtered line has a form which coincides with the form for stray light. Points which markedly deviate, for example square notches or steep peaks, are eliminated. A block 16 indicates a comparison between the low-pass filtered horizontal line and characteristics for a stray light form acquired from block 17. After low-pass filtering and elimination of any points, a line 19 is obtained, which line can have a curve shape schematically shown in the block 18.
(19) Stray light images which have been taken during the production phase have previously been stored. The block 11 makes such images available and a corresponding line 21 in the stray light images is produced and exemplified schematically in block 20.
(20) A line 19 representing part of the direct image of the IR camera and lines 21 representing known curves from stray light images now exist. In a block 22, the curve shape of the line 19 is matched against the curve shapes of the lines 21 by calculating the factor X which makes the most pixels from the curves coincide. The calculation can be conducted by means of the least squares method. Block 22 illustrates the equation set-up and supplies the calculated factor X. The block 22 can also include the conductance of a reasonability assessment of the calculated factor X.
(21) The produced factor X is next multiplied by the stray light image which best corresponds to the line 19, see block 23. A connection 24 here shows that the stray light image can be fetched from the block 11, and a connection 25 indicates the particular stray light image. As a result of the multiplication, a stray light map, marked by a block 26, is obtained. This stray light map is placed in a block 27 together with an offset map generated in block 8 and high-pass filtered in block 9, to form a map 28 which is applied in a block 29 to the direct image of the IR camera represented by block 12, resulting in a finished image, block 30, compensated for stray light and other, both low-frequency and high-frequency noise.
(22) A problem with IR cameras is stray light, wherein stray light is light in an optical system, which was not intended in the design. The light may be from the intended source, but follow paths other than intended, or it may be from a source other than the intended source. This light may set a working limit on the dynamic range of the IR imaging system; it limits the signal-to-noise ratio or contrast ratio. To compensate for this phenomenon, various techniques disclosed herein aim at providing stray light compensation.
(23) When the stray light changes in an infrared camera system, the aperture shading effect, Narcissus effects and optics may give rise to a low frequency fixed pattern noise (FPN) in infrared images. In a cooled IR camera, the frequency of the noise is related to the dimensions of the cold shield and the distance to the detector, such as a focal plane array, FPA, but also to the optics. The amplitude and sign of the noise depends on the amount of stray-light in the system. In conventional systems, this may compensated for by performing a Non-Uniformity Correction (NUC) with a shutter (e.g., a spade). In one or more embodiments of the present invention, stray light compensation is performed without using a shutter, by direct processing of the image data.
(24) With reference now to
(25)
(26) At block S400, capturing an IR image of a scene;
(27) At block S410, generating a fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 using the captured IR image and a stray light model; and
(28) At block S480, performing a FPN compensation of the captured IR image based on said FPNest.sub.t0 to obtain a stray light compensated IR image 490.
(29) In one non-limiting example, the FPN compensation may be performed by reducing data values of the captured IR image with data values of the FPNest.sub.t0. In one or more embodiments, the stray light model may be associated with an IR camera that captured the IR image, as further described below.
(30) Generating a Fixed Pattern Noise Estimate
(31) At the time of production of the IR camera, IR images with stray light present may be captured, which may then be transformed into a frequency domain (e.g., by a Discrete Cosine Transform (DCT), a Fast Fourier Transform (FFT), or other suitable transform) and stored in a memory (e.g., memory circuit 4) in the IR camera as stray light models.
(32) It should be noted that while the generation of stray light images according to block 11 of
(33)
(34) In one or more embodiments, the stray light model amplitude values are quantified to a value of one representing a principal component amplitude value and a value of zero representing a non-principal component amplitude value.
(35) In one or more embodiments, a shape coefficient mask may be generated in which principal components are represented as a value of one and non-principal components are presented as a value of zero. Such a shape coefficient mask be applied to a frequency domain representation of an image to truncate (e.g., reduce to zero or other insignificant value) those frequency components in the image that correspond to non-principal components, for example.
(36) Referring now to
(37) At block S605, optionally obtaining a measured IR image Y.sub.meas by subtracting a mean value (e.g., a mean irradiance value of all pixels in the captured IR image) from the captured IR image Y. For example, the measured IR image Y.sub.meas may be representative of a rough approximation of stray light present in the captured IR image Y. For embodiments in which block S605 is not performed, other operations of
(38) At block S610, optionally down sampling the captured IR image (or the measured IR image depending on embodiments) of the scene to a down-sampled IR image with a predetermined factor, wherein said IR image is down-sampled from a first original resolution or dimension to a second lower or reduced resolution or dimension, thereby reducing computational complexity for the stray light compensation. In one non-limiting example, the IR image is down-sampled with a factor of 80, but any suitable down-sampling factor may be used depending on circumstances. An advantage of down sampling the captured IR image is that the calculations made on a down sampled image have a very low computational complexity.
(39) At block S620, performing a transform on said captured IR image, said measured IR image, or said down-sampled IR image to the frequency domain, thereby obtaining an IR image frequency pattern 560, wherein the frequency pattern comprises frequency components with associated frequency component amplitudes and center frequency, wherein the stray light model and the transformed IR image have frequency patterns with the same number of and corresponding frequency components.
(40) In one non-limiting example, the stray light model and the transformed IR image frequency pattern 560 is represented in matrix form and have corresponding matrix positions, or coordinates, such as row and column or x and y coordinates. In one embodiment, the transform is a Discrete Cosine Transform (DCT) or a Fast Fourier Transform (FFT).
(41) At block S630, determining whether the frequency components of frequency pattern of the transformed IR image fulfill a first condition based on the stray light model. In one or more embodiments, the first condition is that a correlation measure based on the frequency pattern of the transformed IR image and the stray light model exceeds a predetermined threshold value C1.
(42) If it is determined at block S630 that said first condition is fulfilled, the following operations may be performed according to one or more embodiments:
(43) At block S640, generating a truncated IR image frequency pattern 570 by setting the frequency component amplitudes of the transformed IR image frequency pattern 560 corresponding to non-principal components of the stray light model to zero, e.g. by truncating the amplitude value. In one non-limiting example, the truncated IR image frequency pattern 570 may be obtained by applying the shape coefficients mask 580, which may be generated as discussed above for
(44) At block S650, performing an inverse transform (e.g., an inverse DCT, inverse FFT, or other operations that are inverse of the transform performed at block S620) on the truncated frequency pattern 570 to obtain a fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t0.
(45) At block S660, optionally up-sampling the fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t0 with a predetermined factor, not shown in the figure. In one non-limiting example the IR image is up-sampled with a factor of 80, but any suitable up-sampling factor may be used depending on circumstances.
(46) At block S670, determining the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0. In one or more embodiments, the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 is determined based on said fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0. In one non-limiting example, fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 is set equal to said fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0.
(47) In one or more embodiments, determining the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 is further based on one or more previously determined fixed pattern noise estimate delta (FPNest.sub.t-M, . . . , FPNest.sub.t-2, FPNest.sub.t-1). In one non-limiting example, determining the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0 comprises finite impulse response (FIR) filtering said fixed pattern noise estimate FPNest.sub.t0.sub._.sub.delta for time t.sub.0 and previously determined fixed pattern noise estimate delta (FPNest.sub.t-M, . . . , FPNest.sub.t-2, FPNest.sub.t-1) to obtain said fixed pattern noise estimate FPNest.sub.t0 for time t.sub.0.
(48) In one or more embodiments, determining a fixed pattern noise estimate to be used for FPN compensation of the capture IR image is performed recursively/iteratively. This is illustrated in
(49) In the embodiments described in connection with
(50) As illustrated in
(51) Referring again to block S630 of
(52) In one or more embodiments described in connection with
(53) At block S720, determining a fixed pattern noise estimate FPNest.sub.t0 for time t.sub.o. According to one or more embodiments, the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.o may be represented as a matrix having a resolution corresponding to the resolution of the captured IR image 710 and having corresponding matrix positions, or coordinates, such as row and column or x and y coordinates.
(54) In one embodiment, all values of the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.o are initially, at the beginning of the first iteration, set to 0. In the following iterations (the second iteration and onwards, at block S720, the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.o may be adjusted based on a fixed pattern noise estimate FPNest.sub.adjusted according to one or more embodiments. The fixed pattern noise estimate FPNest.sub.adjusted is the result of operations at S730 and is fed back to block S720 if the second condition, which is checked at block S740, is not fulfilled. This is further described below.
(55) At block S730, the captured IR image 710 and the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.o may be combined. In one or more embodiments, the fixed pattern noise estimate FPNest.sub.t0 for time t.sub.o is multiplied with a factor ?, and the captured IR image 710 is multiplied with a factor (1??), before combination, wherein 0???1. In one or more embodiments, the start value of ? may be preset in production or calibration of the IR imaging device, or it may be set by a user of the IR camera through input using an input device of the IR camera. The value of ? is after initialization, according to embodiments, adjusted empirically during the iterations of operations in blocks S720, S730 and S740. The output of the combination of block S730 is a fixed pattern noise estimate FPNest.sub.adjusted.
(56) At block S740, whether or not the fixed pattern noise estimate FPNest.sub.adjusted fulfills a second condition may be determined. In one or more embodiments, the second condition is based on the stray light model. If the fixed pattern noise estimate FPNest.sub.adjusted fulfills the second condition, this may mean that the fixed pattern noise estimate FPNest.sub.adjusted will provide a sufficient stray light compensation when applied onto the capture IR image 710. In this case, the method may proceeds to block S750. If the fixed pattern noise estimate FPNest.sub.adjusted does not fulfills the second condition, the method may proceed to block S720 for another iteration.
(57) In step S750: performing an FPN compensation of the captured IR image 710 based on said fixed pattern noise estimate FPNest.sub.adjusted to obtain a stray light compensated IR image. The compensation at block S750 corresponds in some embodiments to block S480 described in connection with
(58) In embodiments where the stray light model 700 was down-sampled by a preset factor and transformed into the frequency domain before it is input to the method of
(59) In one non-limiting example, the FPN compensation may be performed through additive methods, by reducing, or increasing, data values of the captured IR image with data values of the FPNest.sub.adjusted depending on how the FPNest.sub.adjusted is calculated.
(60) Where applicable, the ordering of various steps described herein can be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein. Embodiments described above illustrate but do not limit the invention. It should also be understood that numerous modifications and variations are possible in accordance with the principles of the invention. Accordingly, the scope of the invention is defined only by the following claims.