Apparatus for generating image and method thereof

Abstract

An apparatus includes an image conversion unit suitable for converting a short exposure block according to an exposure ratio; a motion and saturation detection unit suitable for receiving a long exposure block and an L short exposure block converted by the image conversion unit, and detecting motion and saturation; a radiance map generation unit suitable for fusing the long and L short exposure blocks using a guided filter, and generating a radiance map; a luminance acquirement unit suitable for acquiring luminance of the radiance map, and generating luminance using the L short exposure block or respectively generating luminance using the long and L short exposure blocks; and a dynamic range compression unit suitable for performing global tone mapping using the luminance acquired by the luminance acquirement unit, and compressing a dynamic range.

Claims

1. A method for generating an image, comprising: converting a short exposure block according to an exposure ratio; receiving a long exposure block and an L short exposure block converted by the converting of the short exposure block, and detecting motion and saturation; fusing the long L and short exposure blocks using a guided filter, and generating a radiance map; acquiring luminance of the generated radiance map, and generating luminance using the L short exposure block or respectively generating luminance using the long and L short exposure blocks; and performing global tone mapping using the acquired luminance, and compressing a dynamic range, wherein the long exposure block is used as a guidance image, and the short exposure block is used as a target image based on the guided filter.

2. The method of claim 1, wherein the fusing of the long and L short exposure blocks generates a 3×3 block according to a color of a center pixel of an input block in order to use a Bayer image as on input of the guided filter.

3. The method of claim 1, wherein the guided filter is implemented as expressed by equation A below, f.sub.mean of equation A is changed as expressed by equation B below, and q of equation A is changed as expressed by equation C below:
[Equation A]
1:Mean.sub.I=f.sub.mean(I)
Mean.sub.p=f.sub.mean(p)
Corr.sub.I=f.sub.mean(I.*I)
Corr.sub.Ip=f.sub.mean(I.*p)
2:Var.sub.I=Corr.sub.I−Mean.sub.I.*Mean.sub.I
Cov.sub.Ip=Corr.sub.Ip−Mean.sub.I.*Mean.sub.p
3:a=Cov.sub.Ip./(Var.sub.I+λ)
b=Mean.sub.p−a.*Mean.sub.I
4:Mean.sub.a=f.sub.mean(a)
Mean.sub.b=f.sub.mean(b)
5:q=Mean.sub.aI.*I+Mean.sub.b
[Equation B]
Out[0]=(x[0]*3+x[1]*3+x[2]*2+x[3]*3+x[4]*3+x[6]*2)>>4
Out[1]=(x[0]*3+x[1]*3+x[2]*3+x[3]*2+x[4]*3+x[5]*2)>>4
Out[2]=(x[1]*3+x[2]*3+x[0]*2+x[4]*3+x[5]*3+x[8]*2)>>4
Out[3]=(x[0]*3+x[1]*2+x[3]*3+x[4]*3+x[6]*3+x[7]*2)>>4
Out[4]=(x[0]+x[1]+x[2]+x[3]+x[5]+x[6]+x[7]+x[8])>>3
Out[5]=(x[1]*2+x[2]*3+x[4]*3+x[5]*3+x[7]*2+x[8]*3)>>4
Out[6]=(x[3]*3+x[4]*3+x[6]*3+x[7]*3+x[0]*2+x[8]*2)>>4
Out[7]=(x[3]*2+x[4]*3+x[5]*2+x[6]*3+x[7]*3+x[8]*3)>>4
Out[8]=(x[4]*3+x[5]*3+x[7]*3+x[8]*3+x[2]*2+x[6]*2)>>4
[Equation C]
q=Mean.sub.a×I.sub.center+Mean.sub.b, wherein, p and I indicate an input image and a guidance image, f.sub.mean indicates a mean filter having a windows radius of r, λ is a parameter for adjusting strength of the guided filter, and all operations except for f.sub.mean indicate matrix operations.

4. The method of claim 1, wherein, when detecting of the motion and saturation detects a motion and saturation area, the fusing of the long and L short exposure blocks replaces the long exposure block with the L short exposure block, and performs noise filtering for an L short exposure image, and wherein, when the detecting of the motion and saturation does not detect the motion and saturated area, the fusing of the long and L short exposure blocks performs a weighted sum for the long exposure block and a result of the guided filter, and eliminates noise.

5. The method of claim 1 wherein, when the detection of the motion and saturation detects a motion and saturation area, the acquiring of the luminance generates L short luminance using the L short exposure block similarly to the luminance of the radiance map, and wherein, when the detecting of the motion and saturation does not detect the motion and saturation area, the acquiring of the luminance respectively generates long luminous and L short luminance using the exposure block and the L short exposure block similarly to the luminance of the radiance map and a weighted sum is performed.

6. The method of claim 1, wherein the performing of the global tone mapping uses local weight.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is a block diagram illustrating a low complexity high dynamic range image generation apparatus in accordance with an embodiment of the present invention.

(2) FIG. 2 is a conceptual diagram illustrating generation of different 3×3 blocks according to a color of a center pixel of a block.

(3) FIG. 3 is a pixel diagram illustrating indexes of a 3×3 block.

(4) FIG. 4a to FIG. 4c are input images and resultant images,

(5) FIG. 5a to FIG. 5c are zoomed input images and resultant images shown in FIG. 4a to FIG. 4c.

(6) FIG. 6 is a flowchart illustrating a low complexity high dynamic range image generation method in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

(7) Various embodiments will be described below in more detail with reference to the accompanying drawings. The present invention may, however, be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present invention to those skilled in the art. Throughout the disclosure, like reference numerals refer to like parts throughout the various figures and embodiments of the present invention.

(8) FIG. 1 is a block diagram illustrating a low complexity high dynamic range image generation apparatus in accordance with an embodiment of the present invention.

(9) As illustrated in FIG. 1, the low complexity high dynamic range image generation apparatus includes an image conversion unit 11 for converting a short exposure block according to an exposure ratio, a motion and saturation detection unit 12 for receiving a long exposure block and an L short exposure block converted by the image conversion unit 11 and detecting motion and saturation, a radiance map generation unit 13 for fusing the long and L short exposure blocks from the motion and saturation detection unit 12 using a guided filter and generating a radiance map, a luminance acquirement unit 14 for acquiring luminance of the radiance map generated by the radiance map generation unit 13 and generating luminance using the L short exposure block or respectively generating luminance using the long and L short exposure blocks, and a dynamic range compression unit 15 for performing global tone mapping using the luminance acquired by the luminance acquirement unit 14 and compressing a dynamic range.

(10) In general, the structure of a low complexity high dynamic range (LC HDR) image generation apparatus is highly different from conventional high dynamic range algorithms in that it uses the guided filter in order to generate the radiance map.

(11) The LC HDR image generation apparatus receives long and short exposure Bayer 5×5 blocks from an external analog-to-digital converter (ADC, not illustrated). The long exposure block (“long exposure Bayer image” in FIG. 1) is used as a long exposure block in the motion and saturation detection unit 12. The short exposure block (“short exposure Bayer image” in FIG. 1) is multiplied with an exposure ratio, which is provided from an external control unit (not illustrated), and is converted to an L short exposure block by the image conversion unit 11. The image conversion unit 11, for example, may be implemented with a multiplier. The inputted long and short exposure Bayer blocks may have different sizes than 5×5, and the number of inputted images may vary.

(12) The motion and saturation detection unit 12 uses block matching of the conventional high dynamic range (HDR) algorithm for the motion detection.

(13) As expressed in equation 1 below, the motion and saturation detection unit 12 uses the 5×5 block, calculates a Sum of Absolute Difference (SAD) of the long exposure block and the L short exposure block, and detects a motion when the Block_dif is larger than a preset first threshold value TH.sub.1.

(14) $\begin{array}{cc}\mathrm{Block\_Dif}=\frac{\underset{y}{.Math.}\underset{x}{.Math.}\mathrm{abs}\left({\mathrm{Long}}_{x,y}-{\mathrm{LShort}}_{x,y}\right)}{25}\mathrm{if}\mathrm{Block\_Dif}>{\mathrm{TH}}_1\mathrm{then}\mathrm{Motion\_Flag}=1& \left[\mathrm{Equation}1\right]\end{array}$

(15) The motion and saturation detection unit 12 performs the saturation detection as expressed by equation 2 below. The motion and saturation detection unit 12 separately processes the long exposure block and the L short exposure block and detects saturation when Block_aver is larger than a preset second threshold value TH.sub.2. In this case, the motion and saturation detection unit 12 receives the long exposure block and the L short exposure bock converted by the image conversion unit 11, calculates an average value regardless of colors, and detects the saturation.

(16) $\begin{array}{cc}\mathrm{Block\_Aver}=\frac{\stackrel{3}{\underset{y=1}{.Math.}}\stackrel{3}{\underset{x=1}{.Math.}}{\mathrm{Block}}_{x,y}}{9}\mathrm{if}\mathrm{Block\_Aver}>{\mathrm{TH}}_2\mathrm{then}\mathrm{Sat\_Flag}=1& \left[\mathrm{Equation}2\right]\end{array}$

(17) Finally, as expressed by equation 3 below, when any one of the three flags Motion_Flag, Sat_Flag.sub.L, or Sat_Flag.sub.LS has been set, it is determined as a motion and saturation area (MS_Flag=1).
Equation 3
If Motion_Flag or Sat_Flag.sub.L or Sat_Flag.sub.LS then MS_Flag=1

(18) The radiance map generation unit 13 generates a 3×3 block from the 5×5 block, fuses the long exposure block with the L short exposure block using the guided filter, and generates the radiance map. FIG. 2 is a conceptual diagram illustrating generation of different 3×3 blocks according to a color of a center pixel of a block. As illustrated in FIG. 2, different 3×3 blocks are generated according to the color of the center pixel of the 5×5 input block in order to use the Bayer image as an input of the guided filter.

(19) The guided filter has noise elimination performance similar to that of the bilateral filter, is advantageous in that it is easily implemented because exponential operation is not required, and has superior structure-transferring filtering characteristics capable of allowing a target image to have a structure of a guidance image. By using such characteristics, the radiance map generation unit 13 uses the long exposure image as the guidance image, uses the short exposure image as the target image, and fuses them. The reason the short exposure image is used as the target image is because the short exposure image has a dynamic range wider than that of the long exposure image and has low motion blur. The guided filter may be expressed by equation 4 below.
[Equation 4]
1:Mean.sub.I=f.sub.mean(I)
Mean.sub.p=f.sub.mean(p)
Corr.sub.I=f.sub.mean(I.*I)
Corr.sub.Ip=f.sub.mean(I.*p)
2:Var.sub.I=Corr.sub.I−Mean.sub.I.*Mean.sub.I
Cov.sub.Ip=Corr.sub.Ip−Mean.sub.I.*Mean.sub.p
3:a=Cov.sub.Ip./(Var.sub.I+λ)
b=Mean.sub.p−a.*Mean.sub.I
4:Mean.sub.a=f.sub.mean(a)
Mean.sub.b=f.sub.mean(b)
5:q=Mean.sub.a.*I+Mean.sub.b

(20) In equation 4 above, p and I indicate the input image and the guidance image, and f.sub.mean indicates a mean filter having a windows radius of r. Furthermore, λ is a parameter for adjusting strength of the guided filter, but is not used in the low complexity high dynamic range image generation apparatus. In addition, all operations except for f.sub.mean indicate matrix operations.

(21) The guided filter operates as the following example. When the size of f.sub.mean is 3×3, Mean.sub.x should be a 3×3 matrix. Accordingly, f.sub.mean should be performed in a patch of minimum 5×5 or more. In short, for the operation of the guided filter having a diameter (r*2) of 3, five or more line memories are required. This represents that 9 or more line memories are required in a Bayer domain which is a target of the low complexity high dynamic range image generation apparatus. This deviates from the purpose of the low complexity high dynamic range image generation apparatus.

(22) In order to solve such a problem, in the LC HDR image generation apparatus, f.sub.mean is changed as expressed by the following equation 5 implementing a structure in which a small number of line memories may be used while maintaining maximum performance.
[Equation 5]
Out[0]=(x[0]*3+x[1]*3+x[2]*2+x[3]*3+x[4]*3+x[6]*2)>>4
Out[1]=(x[0]*3+x[1]*3+x[2]*3+x[3]*2+x[4]*3+x[5]*2)>>4
Out[2]=(x[1]*3+x[2]*3+x[0]*2+x[4]*3+x[5]*3+x[8]*2)>>4
Out[3]=(x[0]*3+x[1]*2+x[3]*3+x[4]*3+x[6]*3+x[7]*2)>>4
Out[4]=(x[0]+x[1]+x[2]+x[3]+x[5]+x[6]+x[7]+x[8])>>3
Out[5]=(x[1]*2+x[2]*3+x[4]*3+x[5]*3+x[7]*2+x[8]*3)>>4
Out[6]=(x[3]*3+x[4]*3+x[6]*3+x[7]*3+x[0]*2+x[8]*2)>>4
Out[7]=(x[3]*2+x[4]*3+x[5]*2+x[6]*3+x[7]*3+x[8]*3)>>4
Out[8]=(x[4]*3+x[5]*3+x[7]*3+x[8]*3+x[2]*2+x[6]*2)>>4

(23) In Equation 5 above, Out[index] and x[index] follow indexes of a 3×3 block as illustrated in FIG. 3.

(24) f.sub.mean of the fourth term of equation 4 above calculates a mean of “a” and “b”. That is, Mean.sub.a and Mean.sub.b have single values other than a matrix. The fifth term of equation 4 above is changed as expressed by equation 6 below.
[Equation 6]
q=Mean.sub.a×I.sub.center+Mean.sub.b

(25) The short exposure image, to which the guided filtering is performed, is fused with the long exposure image. During the fusion of the long and short exposure images, the characteristics of saturation and motion burr of the long exposure image are also applied to the short exposure image. In order to prevent this, when MS_Flag calculated in advance by the motion and saturation detection unit 12 is 1, the long exposure block is replaced with the L short exposure block and noise filtering is performed only to the L short image. Through such a process, the generation of the radiance map is completed, but it is not possible to sufficiently eliminate noise of the short exposure image due to the small filter of the size 3×3. In order to compensate for this, when MS_Flag is 0, a weighted sum is performed to the long exposure image and a result of the guided filter as expressed by equation 7 below, so that noise is further eliminated.
[Equation 7]
w.sub.1=Max(abs(I.sub.center−q)/TH.sub.3,1)
w.sub.2=1−w.sub.1
Radi_Map=(I.sub.center×w.sub.2)+(q×w.sub.1)

(26) Since the generated radiance map has a wide dynamic range it requires dynamic range compression. That is, tone mapping should be performed. The tone mapping is performed using only a luminance component of the radiance map. In order to calculate luminance in the Bayer domain, color interpolation is required and in order to perform the color interpolation, a line memory for the radiance map is additionally required.

(27) In order to solve this problem, the luminance acquirement unit 14 partly uses the luminance of the radiance map, and partly generates luminance maximally similar to the luminance of the radiance map using the L short exposure block (when MS_Flag is 1), or respectively generates luminance maximally similar to the luminance of the radiance map using the long exposure block and the L short exposure block (when MS_Flag is 0). Equation 8 below indicates conversion of the long or short exposure block to luminance. For the luminance conversion, any one of various cases of Equation 8 below may be used. For example, in case GB shown in equation 8 below, the luminance acquirement unit 14 partly uses the luminance of the radiance map as with G=Radi_Map, and partly generates luminance similar to the luminance of the radiance map as with R and B.
[Equation 8]
Case GB:
R=(x[5]+x[7]*6+x[9]+x[5]+x[17]*6+x[19])>>4
G=Radi_Map
B=(x[24]+x[3]+x[11]*6+x[13]*6+x[21]+x[23])>>4
Case GR:
R=(x[1]+x[3]+x[11]*6+x[13]*6+x[21]+x[23])>>4
G=Radi_Map
B=(x[5]+[7]*6+x[9]+x[15]+x[17]*6+x[19])>>4
Case B:
R=(x[6]+x[8]+x[16]+x[18])>>2
G=(x[7]+x[11]+x[13]+x[17])>>2
B=Radi_Map
Case R:
R=Radi_Map
G=(x[7]+x[11]+x[13]+x[17])>>2
B=(x[6]+x[16]+x[8]+x[18])>>2
Lumi=(B+(G×40)+(R×20))/61

(28) In Equation 8 above, x[index] conforms to the rule of the 5×5 block of the left side of FIG. 2. The luminance acquirement unit 14 calculates long luminance and L short luminance using equation 8 above, and performs a weighted sum to the long luminance and the L short luminance using the weight of the radiance map of equation 7 above again as expressed by Equation 9 below.
[Equation 9]
Fusion_Lumi=(Long_Lumi×w.sub.2)+(LShort_Lumi×w.sub.1)

(29) Similarly to the aforementioned radiance map, Equation 9 above is performed only when MS_Flag is 0 and only the L short luminance is calculated when MS_Flag is 1.

(30) Basically, the tone mapping is largely classified into two types of global tone mapping and local tone mapping. The global tone mapping has low complexity, but may damage local contrast of an image. The local tone mapping has high complexity, but does not seriously damage the local contrast of the image. However, since the local tone mapping causes a Halo effect, a lot of research has been put into removing the Halo effect. In tone mapping of the conventional high dynamic range algorithm, global tone mapping is used, but to maintain local contrast, only a low frequency component of the image is compressed. However, in such case, distortion such as a Halo effect occurs, and noise is also boosted as well as a high frequency component of the image. Furthermore, an additional line memory is required.

(31) Accordingly, the dynamic range compression unit 15 uses the global tone mapping and an adoptive tone mapping curve in consideration of the local contrast. In this case, a kernel of a basic adoptive tone mapping curve is expressed by equations 10 and 11 below.

(32) $\begin{array}{cc}\mathrm{TML}=\left(D_{\max }-D_{\min }\right)\times \frac{\log \left(I+\tau \right)-\log \left(I_{\min }+\tau \right)}{\log \left(I_{\max }+\tau \right)-\log \left(I_{\min }+\tau \right)}+D_{\min }& \left[\mathrm{Equation}10\right]\\ \tau =\alpha \left(I_{\max }-I_{\min }\right)& \left[\mathrm{Equation}11\right]\end{array}$

(33) In equations 10 and 11 above, D.sub.max and D.sub.min respectively indicate maximum and minimum of output, and I indicates the luminance. Furthermore, I.sub.max and I.sub.min respectively indicate maximum and minimum of the luminance of the radiance map. in order to calculate I.sub.max and I.sub.min, a histogram of a high dynamic range resultant image, which is previously acquired, is used. TML indicates tone mapping luminance. α is a compressive strength parameter. As α approaches 1, compressive strength weakens, and as α approaches 0, compressive strength strengthens.

(34) The dynamic range compression unit 15 defines α as expressed by equation 12 below and calculates τ such that the compressive strength may be adoptively applied according to brightness characteristics of an input image.

(35) $\begin{array}{cc}{C}_H=\frac{I_{\max }+I_{\min }}{2}+\frac{{\mathrm{Hist}}_H-\left(I_{\max }+I_{\min }\right)/2}{2}{C}_{\mathrm{HH}}=\frac{C_H+I_{\min }}{2}+\frac{{\mathrm{Hist}}_{\mathrm{HH}}-\left(C_H+I_{\min }\right)/4}{2}{\alpha }_{\mathrm{Global}}=\frac{C_{\mathrm{HH}}}{P\times I_{\max }}\times \left(I_{\max }+I_{\min }\right)& \left[\mathrm{Equation}12\right]\end{array}$

(36) In equation 12 above, Hist.sub.H indicates a luminance value corresponding to a half of a histogram accumulation value ((Width*Height)/2), and Hist.sub.HH indicates a luminance value corresponding to a quarter of the histogram accumulation value ((Width*Height)/4). P indicates a compressive strength parameter, and as this value is large, the entire brightness is increased. However, since τ calculated using α.sub.Global is a global parameter, local contrast may be reduced. In order to compensate for this, locality is applied in consideration of the local contrast as expressed by equation 13 below.

(37) $\begin{array}{cc}{W}_g=\frac{I_{\max }-I}{I_{\max }}{W}_l=1-W_g{\tau }_{\mathrm{Bright}}=\left({\tau }_{\mathrm{Global}}\times W_g\right)+\left(I\times W_l\right)& \left[\mathrm{Equation}13\right]\end{array}$

(38) In equation 13 above, τ.sub.Global is calculated using α.sub.Global. The calculated τ.sub.Bright is put into equation 10 above to calculate TML.sub.Bright. However, in TML.sub.Bright, since a dim color is also determined as a dark area and noise is boosted, it is probable that image quality will be degraded. In order to prevent the degradation of the image quality, TML.sub.Dark is used. Equation 14 below indicates τ.sub.Dark used in calculating TML.sub.Dark.
[Equation 14]
τ.sub.Dark=(I.sub.max−I.sub.min)

(39) Calculation of final tone mapping luminance TML using TML.sub.Dark and TML.sub.Bright is finally expressed by equation 15 below.
[Equation 15]
TML=(TML.sub.Bright×W.sub.g)±(TML.sub.Dark×W.sub.1)

(40) The generated tone mapping luminance TML is applied to the radiance map as expressed by equation 16 below, and is outputted as a resultant image (see FIG. 4c) of the LC HDR.
[Equation 16]
HDR=(Radi_Map/Lumi)×TML

(41) As described above, since the dynamic range compression unit 15 reflects locality in the global tone mapping using the local weight, it is advantageous in that it is easily implemented with hardware.

(42) FIG. 4a to FIG. 4c are input images and resultant images,

(43) FIG. 4a illustrates the inputted long exposure Bayer image, FIG. 4b illustrates the inputted short exposure Bayer image, and FIG. 4c illustrates the resultant image of the low complexity high dynamic range. FIG. 5a to FIG. 5c are zoomed input images and resultant images shown in FIG. 4a to FIG. 4c.

(44) The low complexity high dynamic range image generation apparatus described with reference to FIGS. 1 to 5c may be provided at positions of an interior, a front end and the like of an image signal processor (ISP).

(45) FIG. 6 is a flowchart illustrating a low complexity high dynamic range image generation method in accordance with an embodiment of the present invention. Since a detailed embodiment thereof has been described in the description of the low complexity high dynamic range image generation apparatus of FIG. 1, the operation process thereof will be simply described below.

(46) The image conversion unit 11 converts the short exposure block according to an exposure ratio (61).

(47) The motion and saturation detection unit 12 receives the long exposure block and the L short exposure block converted by the image conversion unit 11, and detects motion and saturation (62).

(48) The radiance map generation unit 13 fuses the long and L short exposure blocks from the motion and saturation detection unit 12 using the guided filter and generates the radiance map (63).

(49) The luminance acquirement unit 14 acquires the luminance of the radiance map generated by the radiance map generation unit 13, and generates luminance using the L short exposure block or respectively generates luminance using the long and L short exposure blocks (64).

(50) The dynamic range compression unit 15 performs global tone mapping using the luminance acquired by the luminance acquirement unit 14, and compresses the dynamic range (65).

(51) Although various embodiments have been described for illustrative purposes, it will be apparent to those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the following claims.