IMAGE CONVOLUTION METHOD IN HYPERBOLIC SPACE

Abstract

Disclosed is a method for performing image convolution by considering a hierarchical relationship of hyperbolic feature vectors in a hyperbolic space. According to an embodiment of the present disclosure, an image convolution method in a hyperbolic space includes steps of embedding an image feature vector on a Euclidean space into a hyperbolic feature vector on a hyperbolic space, allocating a hierarchical weight on the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector, and convolutioning the hyperbolic feature vector by applying the hierarchical weight.

Claims

1. An image convolution method in a hyperbolic space performed by a convolution module comprising steps of: embedding an image feature vector on a Euclidean space into a hyperbolic feature vector on a hyperbolic space; setting a hierarchical weight on the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector; rearranging the hyperbolic feature vector in the size order of the hierarchical weight; generating a hierarchical weight vector arranged in the size order of the hierarchical weight; and convolutioning the hyperbolic feature vector after applying the hierarchical weight vector to the hyperbolic feature vector.

2. The image convolution method of claim 1, wherein the embedding step includes embedding each pixel constituting the image feature vector on the hyperbolic space through a mapping function.

3. The image convolution method of claim 1, wherein the embedding step includes embedding the image feature vector into the hyperbolic feature vector on the Poincaré ball through a one-to-one correspondence function connecting the Euclidean space and the Poincaré ball.

4. The image convolution method of claim 1, wherein the setting of the hierarchical weight includes setting the hierarchical weight based on a geodesic distance of the hyperbolic feature vector.

5. The image convolution method of claim 1, wherein the setting of the hierarchical weight includes setting a hierarchical weight inversely proportional to the geodesic distance of the hyperbolic feature vector.

6. The image convolution method of claim 1, wherein the setting of the hierarchical weight includes setting the hierarchical weight based on a distance between a reference pixel and remaining pixels constituting the hyperbolic feature vector.

7. The image convolution method of claim 1, wherein the convolutioning step includes multiplying the hyperbolic feature vector by the hierarchical weight vector consisting of the hierarchical weight, and convolutioning the hyperbolic feature vector multiplied by the hierarchical weight vector.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The patent or application file contains at least one drawing executed in color. Copies of this patent or patent application publication with color drawing(s) will be provided by the Office upon request and payment of the necessary fee.

[0023] The above and other aspects, features and other advantages of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0024] FIG. 1 is a flowchart schematically illustrating an image convolution method of the present disclosure;

[0025] FIG. 2 is a diagram illustrating a state in which two vectors on a Euclidean space are embedded in a hyperbolic space;

[0026] FIG. 3 is a diagram illustrating a state in which two vectors in a Euclidean space are embedded in a Poincaré ball;

[0027] FIG. 4 is a diagram for describing a geodesic distance on a hyperbolic space;

[0028] FIG. 5 is a flowchart illustrating an image convolution method in a hyperbolic space according to an embodiment of the present disclosure;

[0029] FIG. 6 is a flowchart illustrating an image convolution method in a hyperbolic space according to another embodiment of the present disclosure; and

[0030] FIG. 7 is a diagram for comparing image reproduction accuracy of the present disclosure and the related art.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0031] The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present disclosure pertains will be able to easily implement the technical idea of the present disclosure. In describing the present disclosure, a detailed description of related known technologies will be omitted if it is determined that they unnecessarily make the gist of the present disclosure unclear. Hereinafter, preferred exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, like reference numerals are used to indicate like or similar components.

[0032] Hereinafter, the fact that an arbitrary configuration is disposed at the “upper (or lower)” of the component or “on (or below)” the component may mean that not only that an arbitrary component is disposed in contact with an upper surface (or lower surface) of the component, but also that other components may be interposed between the component and any component disposed on (or below) the component.

[0033] Further, a singular form used in the present specification may include a plural form if there is no clearly opposite meaning in the context. In this specification, the term such as “comprising” or “including” should not be interpreted as necessarily including all various components or various steps disclosed in the specification, and it should be interpreted that some component or some steps among them may not be included or additional components or steps may be further included.

[0034] Throughout this specification, unless otherwise described to the contrary, “A and/or B” means A, B, or A and B, and unless otherwise described to the contrary, “C to D” means C or more and D or less.

[0035] The present disclosure relates to a method for performing image convolution by considering a hierarchical relationship between hyperbolic feature vectors in a hyperbolic space. More specifically, the present disclosure is a method applied to a neural network using spatial propagation, and may be used to spatially propagate a sparse input to generate a dense output.

[0036] Hereinafter, an image convolution method of the present disclosure will be described in detail with reference to FIGS. 1 to 7.

[0037] FIG. 1 is a flowchart schematically illustrating an image convolution method of the present disclosure.

[0038] FIG. 2 is a diagram illustrating a state in which two vectors on a Euclidean space are embedded in a hyperbolic space and FIG. 3 is a diagram illustrating a state in which two vectors in a Euclidean space are embedded in a Poincaré ball.

[0039] FIG. 4 is a diagram for describing a geodesic distance on a hyperbolic space.

[0040] FIG. 5 is a flowchart illustrating an image convolution method in a hyperbolic space according to an embodiment of the present disclosure and FIG. 6 is a flowchart illustrating an image convolution method in a hyperbolic space according to another embodiment of the present disclosure.

[0041] FIG. 7 is a diagram for comparing image reproduction accuracy of the present disclosure and the related art.

[0042] Referring to FIG. 1, an image convolution method in a hyperbolic space to be described below may include embedding an image feature vector on a Euclidean space into a hyperbolic feature vector on a hyperbolic space (S110).

[0043] In addition, the image convolution method may include allocating a hierarchical weight on the hyperbolic feature vector based on a hierarchical property of the hyperbolic feature vector (S120). In addition, the image convolution method may include convolutioning the hyperbolic feature vector by applying the previously allocated hierarchical weight (S130).

[0044] Each of steps S110 to S130 described above may be programmed and implemented through software, or may also be implemented through hardware including a processing unit such as a graphic processing unit (GPU) and a central processing unit (CPU). Hereinafter, for convenience of description, it is assumed that each of steps S110 to S130 of the present disclosure is performed by a convolution module. Hereinafter, each step illustrated in FIG. 1 will be described in detail.

[0045] The convolution module may receive the image feature vector on the Euclidean space and embed the received image feature vector on the hyperbolic space (S110). More specifically, the convolution module may convert the image feature vector located on the Euclidean space into a hyperbolic feature vector located on the hyperbolic space.

[0046] The hyperbolic space is a non-Euclidean space, and may be a homogeneous space having a uniform negative curvature at all points on the space. The hyperbolic space may include a multidimensional vector that is difficult to be shaped.

[0047] In the following drawings, it is assumed that the hyperbolic space is expressed in a three dimension in order to shape the hyperbolic space. However, as described above, the hyperbolic space may include a multidimensional vector of four or more dimensions.

[0048] Referring to FIG. 2, the convolution module may embed two image feature vectors F1 and F2 located in a two-dimensional Euclidean space to two hyperbolic feature vectors H1 and H2 located in the hyperbolic space. To this end, the convolution module may use a mapping function M.

[0049] The mapping function M may be a one-to-one correspondence function for mapping a Euclidean space to a hyperbolic space or the hyperbolic space to the Euclidean space. For example, the mapping function M may be defined as a linear function, and in this case, each coefficient constituting the linear function may be a learnable parameter.

[0050] Meanwhile, referring to FIG. 3, the aforementioned hyperbolic space may be modeled as a Poincaré ball. At this time, the convolution module may embed the two image feature vectors F1 and F2 into the hyperbolic feature vectors H1 and H2 on the Poincaré ball through a mapping function M connecting the Euclidean space and the Poincaré ball, specifically, a one-to-one correspondence function. Here, the mapping function M may be defined as a linear function, and in this case, each coefficient constituting the linear function may be a learnable parameter.

[0051] Meanwhile, since a multiple operation and a sum operation constituting the mapping function M are performed on the hyperbolic space, Möbius sum operation and multiple operation may be used. More specifically, the Möbius multiple operation may be expressed as in [Equation 1] below, and the Möbius sum operation may be expressed as in [Equation 2] below.

[00001]$\begin{array}{cc}M{.Math.}_{\kappa }u=\left(1/\sqrt{\kappa }\right)\tanh \left(\frac{.Math.\mathrm{Mu}.Math.}{.Math.u.Math.}{\tanh }^{-1}\left(\sqrt{\kappa }.Math.u.Math.\right)\right)\frac{\mathrm{Mu}}{.Math.\mathrm{Mu}.Math.}& \left[\mathrm{Equation}1\right]\end{array}$

[0052] (M represents a matrix, u represents a vector, k represents a curvature of a Poincaré ball, and ||.Math.|| represents a Euclidean norm)

[00002]$\begin{array}{cc}u{.Math.}_{\kappa }v=\frac{\left(1+2\kappa .Math.u,v.Math.+\kappa {.Math.v.Math.}^2\right)u+\left(1-\kappa {.Math.u.Math.}^2\right)v}{1+2\kappa .Math.u,v.Math.+{\kappa }^2{.Math.u.Math.}^2{.Math.v.Math.}^2}& \left[\mathrm{Equation}2\right]\end{array}$

[0053] (u and v represent vectors, k represents a curvature of a Poincaré ball, and .Math., .Math. represents a Euclidean inner product)

[0054] As described above, when the embedding is completed, the convolution module may allocate the hierarchical weight on the hyperbolic feature vector based on the hierarchical property of the hyperbolic feature vector (S120).

[0055] Here, the hierarchical property is a property of similarity and/or affinity between the hyperbolic feature vectors. More specifically, the hierarchical property may include any parameter indicating any value or degree regarding the similarity and/or affinity of the vectors on the hyperbolic space.

[0056] In the mapping function M described in step S110, the hyperbolic feature vectors are uniformly distributed on the hyperbolic space, such that the image feature vectors are arranged on a regular grid in the Euclidean space. However, since the hyperbolic feature vector is projected into a manifold space through dimensionality reduction for data visualization, the above-described premise may not be satisfied. Accordingly, when the hyperbolic feature vector is convolutioned without additional data processing, there is a problem in that non-Euclidean spatial characteristics are not faithfully reflected.

[0057] In order to alleviate this problem, the convolution module may allocate the hierarchical weight to each hyperbolic feature vector based on a geodesic distance of the hyperbolic feature vector.

[0058] Referring to FIG. 4, the geodesic distance in the hyperbolic space may be different from a Euclidean distance in the Euclidean space. The Euclidean distance may mean a linear distance between two pixels H(x1, y1) and H(x2, y2), whereas the geodesic distance may mean a curved distance passing through the surface of a hyperboloid.

[0059] The convolution module may calculate a hyperbolic feature vector embedded on the hyperbolic space, specifically, a geodesic distance between respective pixels constituting the hyperbolic feature vector, and allocated a hierarchical weight according to the calculated distance.

[0060] Referring to FIG. 5 as an example, the image feature vector in the Euclidean space may be embedded as a hyperbolic feature vector h on the Poincaré ball by the mapping function M. Accordingly, the image feature vector in which a distance from an adjacent pixel has been normalized may be mapped to a hyperbolic feature vector {acute over (h)} having different distances from the adjacent pixel.

[0061] The convolution module may calculate a geodesic distance between the respective pixels in the hyperbolic feature vector h, and allocate a hierarchical weight to each pixel based on the calculated distance.

[0062] The geodesic distance may be calculated according to

[0063] [Equation 3] below.

[00003]$\begin{array}{cc}{d}_{\kappa }\left(u,v\right)=\frac{2}{\sqrt{\kappa }}{\tanh }^{-1}\left(\sqrt{\kappa }.Math.\left(-u\right){.Math.}_{\kappa }v.Math.\right)& \left[\mathrm{Equation}3\right]\end{array}$

[0064] (dk represents a geodesic distance, u and v represent two points on the Poincaré ball, k represents a curvature of the Poincaré ball, and ||.Math.|| represents a Euclidean norm)

[0065] The closer the geodesic distance between hyperbolic feature vectors in the hyperbolic space, the higher the affinity between respective pixels. In the neural network using spatial propagation, neural network learning may be performed in consideration of such affinity.

[0066] The convolution module of the present disclosure may give higher importance to a pixel having high affinity in order to improve the learning efficiency of a spatial propagation neural network (SPN). To this end, the convolution module may allocate a hierarchical weight that is inversely proportional to the geodesic distance between the hyperbolic feature vectors h. In this case, the hierarchical weight may be normalized to be inversely proportional to the geodesic distance.

[0067] Hereinafter, step S120 of allocating the hierarchical weight to the hyperbolic feature vectors {acute over (h)} will be described in detail with reference to FIG. 5.

[0068] The convolution module may allocate a hierarchical weight based on a distance between a reference pixel rf and the remaining pixels of constituting the hyperbolic feature vector h. For example, the convolution module may set a dimgray pixel illustrated in FIG. 5 as the reference pixel rf, and may calculate a distance between the reference pixel rf and the remaining pixels nf.

[0069] Referring to the positions of pixels embedded in the Poincaré ball, a geodesic distance between a dimgray reference pixel rf and a darkgray pixel nf.sub.1 may be the shortest, and a geodesic distance between the dimgray reference pixel rf and an lightgray pixel nf.sub.2 may be the longest.

[0070] Since the geodesic distance from the dimgray reference pixel rf is 0, the convolution module may set the hierarchical weight for the dimgray reference pixel rf to a maximum value, and since the distance from the darkgray pixel nf.sub.1 is closer than the distance from the lightgray pixel nf.sub.2, the hierarchical weight for the darkgray pixel nf.sub.1 may be set higher than the hierarchical weight for the lightgray pixel nf.sub.2.

[0071] The convolution module may generate a hierarchical weight vector g consisting of a hierarchical weight for each pixel.

[0072] A method of generating the hierarchical weight vector g will be described with reference to FIG. 5 again. In FIG. 5, for convenience of description, the size of the hierarchical weight is expressed as a shade that becomes darker in proportion to the size of the hierarchical weight.

[0073] The convolution module may generate a hierarchical weight vector g having the same size (width and height) as the hyperbolic feature vector {acute over (h)}. For example, as illustrated in FIG. 5, when the hyperbolic feature vector h is in the form of a 3×3 matrix, the hierarchical weight vector may also have the form of a 3×3 matrix. In this case, the hierarchical weight vector g may consist of pixel values indicating the hierarchical weight of each pixel in the hyperbolic feature vector h.

[0074] As described above, since the hierarchical weight of the dimgray reference pixel rf (row 2 column 2) is the maximum, in FIG. 5, the pixel value of row 2 column 2 of the hierarchical weight vector ǵ may be set highest. In addition, since the hierarchical weight of the darkgray pixel nf.sub.1 (row 3 column 2) is the second highest, the pixel value of row 3 column 2 of the hierarchical weight vector g may be set second highest. Meanwhile, since the hierarchical weight of the lightgray pixel nf.sub.2 (row 1 column 3) is the minimum, the pixel value of row 1 column 3 of the hierarchical weight vector g may be set lowest.

[0075] As described above, the convolution module may generate a hierarchical weight vector g corresponding to each hyperbolic feature vector {acute over (h)} using the hierarchical weight allocated based on the geodesic distance.

[0076] According to the present disclosure, by allocating the hierarchical weight to the hyperbolic feature vector h by the above-described method, it is possible to set higher importance of a pixel having a close geodesic distance, that is, high affinity. Accordingly, according to the present disclosure, it is possible not only to increase the learning effect and the learning efficiency of the spatial propagation neural network, but also to greatly improve the learning accuracy for a learnable parameter in a filter and/or kernel used for convolution.

[0077] Meanwhile, according to the method of allocating the hierarchical weight described with reference to FIG. 5, respective pixel values in the hierarchical weight vector.

[0078] may be randomly disposed according to a hierarchical property of the hyperbolic feature vector {acute over (h)}. Specifically, unlike those illustrated in FIG. 5, if the geodesic distance between the dimgray reference pixel rf and the darkgray pixel nf.sub.1 is the longest, a pixel value in row 3 column 2 of the hierarchical weight vector g may be set lowest. Due to the randomness of the hierarchical weight vector g, there is a limit that the learning consistency of deep learning may be somewhat lowered.

[0079] In order to solve the above-mentioned potential problems, the convolution module may rearrange a hyperbolic feature vector h according to the hierarchical weight (S121), and may generate a hierarchical weight vector g by arranging hierarchical weights according to the rearrangement order (S122).

[0080] Hereinafter, another method of generating the hierarchical weight vector g will be described with reference to FIG. 6. Even in FIG. 6, for convenience of description, the size of the hierarchical weight is expressed as a shade that becomes darker in proportion to the size of the hierarchical weight.

[0081] As described with reference to FIG. 5, the convolution module may determine a hierarchical weight for each pixel based on the hierarchical property, for example, the geodesic distance between pixels constituting the hyperbolic feature vector {acute over (h)}.

[0082] Then, the convolution module may rearrange the hyperbolic feature vector h in the size order of the hierarchical weight. More specifically, the convolution module may rearrange the positions of pixels in the hyperbolic feature vector h in the increasing or decreasing order of the hierarchical weight.

[0083] Referring to FIG. 6, based on the geodesic distance on the Poincaré ball above, the weight for the dimgray reference pixel rf may be set largest, and the weight for the darkgray pixel nf.sub.1 may be set second largest. Meanwhile, the weight of the lightgray pixel nf.sub.2 may be set to the smallest.

[0084] At this time, the convolution module may arrange each pixel of the hyperbolic feature vectorh consisting of 3 rows and 3 columns in the increasing order of the hierarchical weight from row 1 column 1 to row 1 column 3, from row 2 column 1 to row 2 column 3, and from row 3 column 1 to row 3 column 3. Accordingly, as illustrated in FIG. 6, the dimgray reference pixel rf having the largest hierarchical weight may be disposed in row 1 column 1, the darkgray pixel nfi having the second largest hierarchical weight may be disposed in row 1 column 2, and the lightgray pixel nf.sub.2 having the smallest hierarchical weight may be disposed in row 3 column 3.

[0085] Unlike those described above, the convolution module may rearrange each pixel in the decreasing order of the hierarchical weight, and may also arrange the pixels in the order from row 1 column 1 to row 3 column 1, from row 1 column 2 to row 3 column 2, and from row 1 column 3 to row 3 column 3.

[0086] Then, the convolution module may generate a hierarchical weight vector g arranged in the increasing or decreasing order of the hierarchical weight. In other words, the convolution module may generate a hierarchical weight vector g, but each inner pixel value (hierarchical weight) may be arranged in the order of size. In this case, the arrangement direction of pixels in the hierarchical weight vector g may be the same as the rearrangement direction of the aforementioned hyperbolic feature vector h.

[0087] Referring back to FIG. 6, the convolution module may arrange each pixel of the hierarchical weight vector g consisting of 3 rows and 3 columns in the increasing order of the value (hierarchical weight). Accordingly, the hierarchical weight corresponding to the dimgray reference pixel rf may be arranged in row 1 column 1, the hierarchical weight corresponding to the darkgray pixel nfi may be arranged in row 1 column 2, and the hierarchical weight corresponding to the lightgray pixel nf.sub.2 may be arranged in row 3 column 3.

[0088] As described above, when the allocation of the hierarchical weight and/or the generation of the hierarchical weight vector g are completed, the convolution module may convolution the hyperbolic feature vector by applying the hierarchical weight (S130). More specifically, the convolution module may first apply a hierarchical weight to the hyperbolic feature vector h and then convolution the hyperbolic feature vector h applied with the hierarchical weight.

[0089] Referring to FIGS. 5 and 6, when the hierarchical weight vector g is generated as described above, the convolution module may multiply the hyperbolic feature vector h by the hierarchical weight vector g to apply the hierarchical weight to the hyperbolic feature vector h. Here, the Möbius multiple operation expressed by the above-mentioned [Equation 1] may be used for the multiple operation.

[0090] Then, the convolution module may convolution the hyperbolic feature vector h applied with the hierarchical weight. In this case, various techniques used in the corresponding technical field may be applied to convolution, and in particular, a technique used in a spatial propagation neural network (SPN) may be used.

[0091] According to the method of applying the hierarchical weight described with reference to FIG. 6, respective pixel values in the hyperbolic feature vector h and the hierarchical weight vector g may be arranged regularly according to the size. Accordingly, in an output obtained by multiplying the hyperbolic feature vector h by the hierarchical weight vector g, the pixels may be arranged in the order of importance.

[0092] The arrangement of output pixels may have a tendency according to importance, and when the tendency is applied to a convolutional neural network, learning consistency and learning accuracy are improved to establish a robust model. Referring to FIG. 7, according to the present disclosure, dense depth information may be generated using sparse depth information through the spatial propagation neural network (SPN). To this end, according to the present disclosure, a depth sample may be received as an image feature vector and embedded on the hyperbolic space.

[0093] Then, according to the present disclosure, it is possible to apply a weight to each pixel based on the geodesic distance between pixels in the embedded depth sample, and convolution the hyperbolic feature vector h applied with the weight.

[0094] In light of the convolution result of the present disclosure illustrated in FIG. 7, when comparing the present disclosure with a convolutional spatial propagation network (CSPN) introduced in “Xinjing Cheng, Peng Wang, and Ruigang Yang. Depth estimation via affinity learned with convolutional spatial propagation network. In Proceedings of European Conference on Computer Vision (ECCV), 2018.”, which is a recent remarkable research achievement in a spatial propagation neural network (SPN) field, it can be confirmed that the image reconstructed using the convolution method of the present disclosure is more similar to ground truth (GT).

[0095] As described above, the present disclosure has been described with reference to the illustrated drawings, but the present disclosure is not limited to the exemplary embodiments of the present disclosure and the drawings, and it will be apparent that various modifications can be made by those skilled in the art within the scope of the technical idea of the present disclosure. Further, it is natural that even through effects according to the configuration of the present disclosure are not explicitly described while describing the exemplary embodiments of the present disclosure above, expectable effects should be recognized by the configuration.