METHOD OF PERFORMING L0 SMOOTHING ON THE BASIS OF DEEP GRADIENT PRIOR INFORMATION TO IMPROVE SHARPNESS

Abstract

In an embodiment of the present inventive concept, there is provided an 0 smoothing method performed on the basis of deep gradient prior information to improve sharpness of an image by an image quality improving device, and the method comprises: a gradient-improved image generation step of generating a gradient-improved image by minimizing the gradients of pixels of an original image, by the image quality improving device; and a smoothing-improved image generation step of generating a smoothing-improved image smoothing-processed through one-step (.sub.0) estimation on the gradient-improved image, by the image quality improving device.

Claims

1. An 0 smoothing method performed on the basis of deep gradient prior information to improve sharpness of an image by an image quality improving device, the method comprising: a gradient-improved image generation step of generating a gradient-improved image by minimizing gradients of pixels of an original image, by the image quality improving device; and a smoothing-improved image generation step of generating a smoothing-improved image smoothing-processed through one-step () estimation on the gradient-improved image, by the image quality improving device.

2. The method according to claim 1, wherein the gradient-improved image generation step includes: a critical edge area exclusion step of excluding a critical edge area of the original image; and a gradient minimization step of minimizing a horizontal gradient and a vertical gradient using a GNet, which is a convolutional neural network, for an image excluding the critical edge area.

3. The method according to claim 2, wherein the GNet is a convolutional neural network configured of 33 convolution, 55 convolution, and Rectified Linear Unit (ReLU) filters.

4. The method according to claim 3, wherein when f denotes an image layer, i denotes a layer before passing through the GNet, k denotes a layer after passing through the GNet, b denotes a bias, * denotes a convolution operator, and h denotes a weight, the gradient minimization step is performed by processing a formula of f.sub.i.sup.k=ReLU(b.sub.i.sup.k+.sub.j.sup.k-1*h.sub.i,j.sup.k).

5. The method according to claim 2, wherein the smoothing-improved image generation step includes the steps of: calculating a loss function, which is a function showing a difference between an original image and a gradient-improved image; and generating a smoothing-improved image by reducing the loss function.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0024] FIG. 1 is a perspective view showing an image quality improving device that performs an .sub.0 smoothing process based on deep gradient prior information to improve sharpness according to an embodiment of the present inventive concept.

[0025] FIG. 2 is a flowchart illustrating an .sub.0 smoothing process performed on the basis of deep gradient prior information to improve sharpness according to an embodiment of the present inventive concept.

[0026] FIG. 3 is a view showing the GNet structure of the present inventive concept.

[0027] FIG. 4 is a view showing an example of smoothing process according to an embodiment of the present inventive concept.

[0028] FIG. 5 is a view showing smoothing-processed exemplary pictures from which text is removed according to an embodiment of the present inventive concept.

[0029] FIG. 6 is a view showing exemplary pictures with images of improved sharpness according to an embodiment of the present inventive concept.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0030] Hereinafter, the advantages and features of the present inventive concept, and a method for achieving them will become apparent with reference to the embodiments described below in detail, together with the accompanying drawings. However, the present inventive concept is not limited to the embodiments disclosed below, and may be implemented in various different forms. The present inventive concept is provided to fully inform the scope of the present inventive concept to those skilled in the art, and the present inventive concept is only defined by the scope of the claims. In addition, in describing the present inventive concept, when it is determined that related known techniques or the like may blur the gist of the present inventive concept, the detailed description thereof will be omitted.

[0031] Hereinafter, a picture quality improving filter used to improve picture quality as known will be briefly described, before describing the present inventive concept.

[0032] (1) Image Quality Improvement Based on Local-Based Filter

[0033] Local-based filters typically include weighted median, bilateral, and guided filters. These local-based filters are widely used in a simple and speedy way, and a basic image degradation model may be defined as follows.

I=x+n

[0034] Here, I denotes an image including textures, x denotes an ideal image, and n denotes noise. In addition, the local-based filter is applied in an image by the following equation.

S=F(I)

[0035] Here, S is an edge-preserving image, F is a filter such as a WMF, bilateral, and guided filter, and I is an input image. Although local-based filter methods may remove small textures and noises with a simple or repetitive filtering method, they are filtering methods using only local image statistical values, and therefore, there is a problem in that a phenomenon of edge area halo occurs, or a phenomenon of gradient reversal occurs in an area with no similar pixel values in the filtering area.

[0036] (2) Global-Based Image Quality Improvement

[0037] Global-based methods include methods such as ROF-TV, WLS, .sub.0-smoothing, and the like. Unlike the local-based methods, these methods remove noises and small textures by preserving edges using an optimization method. The WLS method provides a result of preserving the main edges by adjusting matrix affinities according to the gradient in an image. The .sub.0-smoothing method is a method of preserving edges using a term that counts the number of non-zero gradients in an image as a regularization term, and it shows excellent edge preservation performance, and is defined as follows.

C(S)=#{p|.sub.xS.sub.p.sup.q|+|.sub.yS.sub.p.sup.g|0|}

[0038] Here, I denotes an input image, S denotes a result image, and |.sub.xS.sub.p| and denote x and y direction gradients with respect to S. In the above equation, C(S).sub.0 is |.sub.xS.sub.p|+|.sub.yS.sub.p|0, and counts the number of components of which the sum of the x and y direction gradients is not 0. It can be redefined as a minimization problem as follows on the basis of this equation.

[00001]$\underset{S}{\min }\left\{{.Math.}_p{\left(S_p-I_p\right)}^2+C\left(S\right)\right\}$

[0039] However, compared to the local-based methods, the global-based methods have problems in that they cannot preserve locally important edges, or the amount of computation increases due to global operations.

C(S)=#{pS0|}

[0040] The problem of increasing the processing speed generated in the existing methods can be solved using a neural network that estimates h and v gradients used in an .sub.0-norm method, which is a global-based optimization method. As an existing global-based edge preservation method, there is .sub.0-norm smoothing defined by the following equation.

C(h.sub.p.sup.g,v.sub.p.sup.g)=#{p|.sub.xS.sub.p.sup.g|+|.sub.yS.sub.p.sup.g|0|}

[0041] Here, I denotes an input image, S denotes a result image, and |.sub.xS.sub.p| and |.sub.yS.sub.p| denote x and y direction gradients with respect to S. In the above equation, C(h.sub.p.sup.g,v.sub.p.sup.g) is |.sub.xS.sub.p|+|.sub.yS.sub.p|0, and counts the number of line segments of which the sum of the x and y direction gradients is not 0. It can be redefined as a minimization problem as follows on the basis of this equation.

[00002]$\underset{S,S^g}{\min }\left\{{.Math.}_p{\left(S_p-I_p\right)}^2+{\left(S_p^g-I_p^g\right)}^2+C\left(h_p^p,v_p^g\right)\right\}$

[0042] As the above equation is an .sub.0-norm regularization term and is non-linear, it is non-convex and non-differentiable. For this reason, the .sub.0-norm regularization term can be solved using a half-quadratic splitting .sub.0-norm minimization method, and a result S.sub.p of preserving edges can be obtained using the equation.

[0043] (3) Learning-Based Image Quality Improvement

[0044] Learning-based methods may provide a result of preserving edges through supervised and unsupervised learning of an artificial neural network configured of convolutions. This method has the advantage of overcoming the limitations of performance and computational complexity existing in the conventional local and global methods, and learns the neural network using the following loss function.

[00003]$\mathrm{Loss}=\frac{1}{N}{.Math.}_{n=1}^N{.Math.f\left(x,\right)-y.Math.}_2^2$

[0045] Here, x denotes an input image, y denotes an image preserving edge, f() denotes an artificial neural network, N denotes the number of learning data, and denotes weight and bias parameters constituting the artificial neural network.

[0046] The present inventive concept proposes a gradient network (GNet) for preserving main edges in an image. The proposed method is for minimizing the gradients of image pixels, and each derivative for horizontal and vertical gradients should be minimized except in an critical edge area. The GNet proposed to solve this problem solves the problem of increasing the amount of computation by estimating (h,v,, as a network using a convolutional neural network. Hereinafter, it will be described in detail.

[0047] FIG. 1 is a perspective view showing an image quality improving device that performs an .sub.0 smoothing process based on deep gradient prior information to improve sharpness according to an embodiment of the present inventive concept.

[0048] The image quality improving device of the present inventive concept is a device for performing .sub.0 smoothing image quality improvement on the basis of deep gradient prior information to improve sharpness. Here, .sub.0 means only one step (one-step), not several steps.

[0049] Existing image smooth filters have high computational complexity due to optimization through iterative operations such as a weighted least squares algorithm, or L0 or L1-based gradient descent, whereas in the present inventive concept, a one-step smoothing process is performed.

[0050] The image quality improving device of the present inventive concept is a device equipped with an arithmetic processing unit such as a CPU, and a display panel such as a monitor, and may be implemented, for example, as a computer equipped with a monitor, a server equipped with a monitor, or the like as shown in FIG. 1.

[0051] The computer or the server includes program modules implemented through various types of languages such as C, C++, Java, Visual Basic, and Visual C in terms of software to perform various functions. In addition, they may be implemented using programs diversely provided according to the operating system such as DOS, Windows, Linux, Unix, Macintosh, or the like on general server hardware, and representatively, a website used in a Windows environment, an Internet Information Server (IIS), and CERN, NCSA, APPACH, or the like used in a Unix environment may be used.

[0052] FIG. 2 is a flowchart illustrating an .sub.0 smoothing process performed on the basis of deep gradient prior information to improve sharpness according to an embodiment of the present inventive concept. FIG. 3 is a view showing the GNet structure of the present inventive concept. FIG. 4 is a view showing an example of smoothing process according to an embodiment of the present inventive concept. FIG. 5 is a view showing smoothing-processed exemplary pictures from which text is removed according to an embodiment of the present inventive concept. FIG. 6 is a view showing exemplary pictures with images of improved sharpness according to an embodiment of the present inventive concept.

[0053] The .sub.0 smoothing method performed on the basis of deep gradient prior information to improve sharpness of an image of the present inventive concept may include, as shown in FIG. 2, a gradient-improved image generation step (S210) of generating a gradient-improved image by minimizing the gradients of pixels of an original image, by the image quality improving device; and a smoothing-improved image generation step (S220) of generating a smoothing-improved image smoothing-processed through one-step (.sub.0) estimation on the gradient-improved image, by the image quality improving device.

[0054] The gradient-improved image generation step (S210) is a step of generating a gradient-improved image by minimizing the gradients of pixels of an original image, by the image quality improving device.

[0055] The gradient-improved image generation step may include a critical edge area exclusion step (S211), and a gradient minimization step (S212).

[0056] The critical edge area exclusion step (S211) is a step of excluding the critical edge area of an original image. Here, the critical edge area is a portion outside the boundary of a preset edge area of an image, and is preset as an area without the need of an image quality improving process.

[0057] The gradient minimization step (S212) is a step of minimizing a horizontal gradient and a vertical gradient using a GNet, which is a convolutional neural network, for an image excluding the critical edge area. Here, the GNet is a convolutional neural network configured of 33 convolution, 55 convolution, and Rectified Linear Unit (ReLU) filters. When f denotes an image layer, i denotes a layer before passing through the GNet, k denotes a layer after passing through the GNet, b denotes a bias, * denotes a convolution operator, and h denotes a weight, the gradient minimization step is performed by processing the formula of equation 1.

f.sub.i.sup.k=ReLU(b.sub.i.sup.k+.sub.jf.sub.j.sup.k-1*h.sub.i,j.sup.k)[Equation 1]

[0058] Describing in detail, the GNet structure is as shown in FIG. 3. Here, f is the feature of a hidden layer, and is a partial differentiation function for extracting horizontal and vertical gradients.

[0059] The i-th feature map of the k-th layer passing through the GNet uses 33 and 55 convolutions and the Rectified Linear Unit (ReLU) as shown in the following equation. b and h denote a bias, a convolution operator, and a weight filter, respectively. For reference, the feature map refers to input/output data of a convolution layer as is known.

[0060] In order to extract texture information with comparatively low computational complexity, the size of the weight filters (h) of the first layer and the second layer is defined as 33, and the size of other layers is defined as 55. The reconstruction layer combines the feature maps as shown in [Equation 2] through a 33 convolution operation.

Output=b.sub.i.sup.6+.sub.jf.sub.j.sup.5*h.sub.i,j.sup.6[Equation 2]

[0061] This network is configured of branch networks of a horizontal network and a vertical network. Horizontal and vertical gradients are involved in the extracted feature maps to analyze each network and input data. The loss function is defined as shown in [Equation 3].

[00004]$\begin{array}{cc}\mathrm{Loss}\left(\left\{h,v\right\};{}_{*}\right)=\frac{1}{N}{.Math.}_{n=1}^N{.Math.\mathrm{GNet}\left(h^i;{}_h\right)+\mathrm{GNet}\left(v^i;{}_v\right)-\left(h_{\mathrm{gt}}^i+v_{\mathrm{gt}}^i\right).Math.}_2^2& \left[\mathrm{Equation}3\right]\end{array}$

[0062] The GNet returns a horizontal gradient and a vertical gradient, and is a parameter. h.sub.gt and v.sub.gt are the gradients of the ground truth.

[0063] Meanwhile, the smoothing-improved image generation step (S220) is a step of generating a smoothing-improved image smoothing-processed through one-step (.sub.0) estimation on the gradient-improved image. As is known, smoothing is frequently used to remove noise, and is used to blur an image or remove noise existing in an image. Smoothing is a method of determining the current position value of a result image using an average of the pixel value at the current position and the neighboring pixel values. When smoothing is applied to an image, it removes noise or connects broken edges, instead of reducing the sharpness.

[0064] The present inventive concept processes smoothing through estimation of step (.sub.0) performed on the smoothing-improved image, and includes a step of calculating a loss function, which is a function showing the difference between an original image and a gradient-improved image (S221), and a step of generating a smoothing-improved image by reducing (e.g., minimizing) the loss function (S222).

[0065] Hereinafter, an example of processing a smoothing-improved image will be described in more detail with reference to equations.

[0066] Existing image smooth filters have high computational complexity due to optimization through iterative operations such as a weighted least squares algorithm, or L0 or L1-based gradient descent. The proposed GNet may acquire deep gradient prior information, and generally, when it is combined with a loss function, optimal smooth image estimation is possible only by one-step estimation. The loss function for one-step L0 minimization is defined as shown in [Equation 4].

[00005]$\begin{array}{cc}\underset{S,h_S,v_S}{\min }{.Math.S-I.Math.}_2^2+{.Math.h_G,v_G.Math.}_0^0+{.Math.{}_{y}S-h_S.Math.}_2^2+{.Math.{}_{y}S-v.Math.}_2^2+\mathrm{loss}\left(\left\{{}_{x}S,{}_{y}S\right\};{}_{*}\right)+\mathrm{loss}\left(\left\{{}_{x}G,{}_{y}G\right\};{}_{*}\right)& \left[\mathrm{Equation}4\right]\end{array}$

[0067] S denotes an optimal smooth image, h* and v* are horizontal and vertical gradients minimized for both the target image (S) and the guide image (G). loss is the loss function of GNet. To minimize the loss function, the cost function is divided to be reconstructed as shown in [Equation 5].

[00006]$\begin{array}{cc}\underset{h_{*},{}_{*}}{\min }\mathrm{loss}\left(\left\{{}_{x}S,{}_{y}S\right\};{}_{*}\right)+\mathrm{loss}\left(\left\{{}_{x}G,{}_{y}G\right\};{}_{*}\right)& \left[\mathrm{Equation}5\right]\end{array}$

[0068] Since the optimized loss function has already been acquired through a stochastic gradient descent method, the solution of the above objective function may be said to be the output of a previously trained GNet. That is, the gradient components .sub.xS, .sub.xG, .sub.yS, .sub.yG are expressed as shown in [Equation 6].

.sub.yS=GNet(.sub.yI;.sub.v),.sub.yG=GNet(.sub.yg;.sub.v)

.sub.yS=GNet(.sub.yI;.sub.v),.sub.xG=GNet(.sub.yg;.sub.v)[Equation 6]

[0069] Specifically, an image applied with Gaussian filtering is used as the guide image g. Since the minimized gradient components are estimated initially, the objective function of [Equation 4] may be more specifically formulated as shown in [Equation 7] using the L0 norm term while maintaining the gradient term.

[00007]$\begin{array}{cc}{.Math.}_p{\underset{h_{*},v_{*}}{\min }\left({\left({}_{x}S\left(p\right)-h_S\left(p\right)\right)}^2+{}_{y}S\left(p\right)-v_S\left(p\right)\right)}^2+\frac{}{}L\left(h_G\left(p\right),v_G\left(p\right)\right))& \left[\mathrm{Equation}7\right]\end{array}$

[0070] The role of the counting function is to count and aggregate the number of non-zero gradients in the guide image. When |h.sub.G+v.sub.G| is not 0, L( ) returns 1, and the following three conditions are satisfied.

[00008]$\begin{array}{cc}{}_{x}G\left(p\right)+{}_{y}G\left(p\right){}_{x}S\left(p\right)+{}_{y}S\left(p\right)+{}_{x}G\left(p\right)+{}_{y}G\left(p\right)\frac{}{}& \left[\mathrm{Equation}9\right]\end{array}$
.sub.xS(P)+.sub.yS(P)+.sub.xG(p)+.sub.yG(p)0[Equation 10]

L(h.sub.G,v.sub.G)0 or L(h.sub.G,v.sub.G)=0[Equation 11]

[0071] In the case of L(h.sub.G,v.sub.G)0, the gradient cost of objective function (8) is defined as shown in [Equation 12].

[00009]$\begin{array}{cc}{\left({}_{x}S\left(p\right)-h_S\left(p\right)\right)}^2+{\left({}_{y}S\left(p\right)-v_S\left(p\right)\right)}^2+\frac{}{}\frac{}{}& \left[\mathrm{Equation}12\right]\end{array}$

[0072] However, the above condition is not established by the condition of [Equation 9]. Therefore, when the conditions of [Equation 9] to [Equation 11] are summarized, it is as shown in [Equation 13].

[00010]$\begin{array}{cc}\left[h_S\left(p\right),v_S\left(p\right)\right]=\{\begin{array}{cc}0& \left(p\right)=1\\ ({}_{x}S\left(p\right),\left({}_{y}S\left(p\right)\right)& \mathrm{otherwise}\end{array}& \left[\mathrm{Equation}13\right]\end{array}$

[0073] Here, is a logical function and is defined as shown in [Equation 14].

[00011]$\begin{array}{cc}\left(p\right)=\{\begin{array}{cc}1& ({}_{x}G\left(p\right)+{\left({}_{y}G\left(p\right)\right)}^2\frac{}{}\\ 0& \mathrm{otherwise}\end{array}& \left[\mathrm{Equation}14\right]\end{array}$

[0074] To acquire a closed form solution based on partial differentiation, objective function [Equation 4] is finally defined as shown in [Equation 15].

[00012]$\begin{array}{cc}\underset{S}{\min }{.Math.}_p{\left(S\left(p\right)-I\left(p\right)\right)}^2+{\left({}_{x}S\left(p\right)-h_S\left(p\right)\right)}^2+{\left({}_{y}S\left(p\right)-v_S\left(p\right)\right)}^2& \left[\mathrm{Equation}15\right]\end{array}$

[0075] The result of the optimal smooth image S of the objective function [Equation 15] is as shown in [Equation 16].

[00013]$\begin{array}{cc}S\left(p\right)=\left(\frac{\left(I\left(p\right)\right)+(\left({}_x{h}_S\left(p\right)+{}_y{v}_S\left(p\right)\right)}{1+\left(\left({}_x^T\right)\left({}_x\right)+\left({}_y^T\right)\left({}_y\right)\right)}\right)& \left[\mathrm{Equation}16\right]\end{array}$

[0076] and are fast Fourier transform and inverse fast Fourier transform, respectively. In addition, initialization is performed with =100 and =0.01. FIG. 4 shows the proposed method in detail.

[0077] Meanwhile, FIG. 5 shows a result of smoothing performed using an NKS data set [6]. As shown in FIGS. 5(c) and (d), although the GUF and L0-smoothing methods successfully remove unnecessary textures, blurring occurs in the main edge area. FIG. 5(e) shows a result of removing textures, but color distortion is observed due to excessive removal. FIGS. 5(f) and (g) predict high-performance texture removal and fast-processing speed with a deep learning-based technique using FCN or VDCNN. However, there is a problem in that the speed is lowered as the image size increases. On the contrary, the proposed method removes textures without color distortion and maintains the fast-processing speed even when the image size increases.

[0078] As shown in FIG. 6(a), it is a method of showing an image of which the details are lost as the image is out of focus since it shows an image with loss of details. The processing method may be expressed as E=2IS. Here, I denotes an input image, and S denotes a smooth image. FIG. 6 shows results of improving sharpness. It can be seen that although the images shown in FIG. 6(b) to (d) are improved as a result, there is a limitation in performance, and the results are evaluated as being unnatural, and although the images shown in FIGS. 6(e) and (f) are improved as a result, the colors are distorted due to excessive improvement. On the contrary, the present inventive concept may provide more natural results while improving details.

[0079] The embodiments in the above description of the present inventive concept are the most preferred examples selected among various possible implementations and presented to help understanding of those skilled in the art, and the technical spirit of the present inventive concept is not necessarily limited or restricted only by the embodiments, but various changes and modifications and equivalent other embodiments will be possible without departing from the technical spirit of the present inventive concept.