NEURAL NETWORK-BASED KEY POINT TRAINING APPARATUS AND METHOD

Abstract

A neural network-based key point training apparatus according to an embodiment disclosed includes a key point model trained to extract key points from an input image and an image reconstruction model trained to reconstruct the input image with the key points output by the key point model as the input.

Claims

1. A neural network-based key point training apparatus comprising: a key point model trained to extract key points from an input image; and an image reconstruction model trained to reconstruct the input image with the key points output by the key point model as the input.

2. The apparatus according to claim 1, wherein the key point model is trained to minimize differences between the extracted key points and key points labelled with respect to the input image; and the image reconstruction model is trained to minimize a difference between the reconstructed image and the input image.

3. The apparatus according to claim 2, wherein the key point model is primarily trained to minimize the differences between the extracted key points and the key points labelled with respect to the input image and is secondarily trained to extract the key point so that a difference between the reconstructed image and the input image is minimized.

4. The apparatus according to claim 1, wherein the key point model comprises: an extraction module trained to extract a feature vector from the input image; and a prediction module trained to predict key point coordinates of the input image based on the feature vector extracted by the extraction module; and the image reconstruction model is trained to reconstruct the input image based on the key point coordinates output by the key point model.

5. The apparatus according to claim 4, wherein optimized parameters of the key point model and the image reconstruction model are calculated through the following equation:
θ*,φ*=argmin.sub.θ,φ(αL.sub.prediction+βL.sub.reconstruction)   [Equation] where, θ*: optimized parameter of key point model; φ*: optimized parameter of image reconstruction model; L.sub.prediction: objective function of key point model, L.sub.prediction=∥K−G(I;θ)∥; K: labelled key point coordinates of input image; G: neural network constituting key point model; θ: parameter of the key point model; I: input image; G(I;θ): key point coordinates predicted from key point model; ∥K−G(I;θ)∥: function to find difference between key point coordinates predicted from key point model and labelled key point coordinates of input image; L.sub.reconstruction: objective function of image reconstruction model, L.sub.reconstruction=∥I−D(G(I;θ);φ)∥; D: neural network constituting image reconstruction model; φ: parameter of image reconstruction model; ∥I−D(G(I;θ); φ)∥: function to find difference between input image and image reconstructed by image reconstruction model; α: weight of objective function of key point model; and β: weight of objective function of image reconstruction model.

6. The apparatus according to claim 1, wherein the key point model comprises: an extraction module trained to extract a feature tensor from the input image; and a prediction module trained to predict a key point image based on the feature tensor extracted by the extraction module; and the image reconstruction model is trained to reconstruct the input image based on the key point image output by the key point model.

7. The apparatus according to claim 6, wherein the key point image is an image indicating whether or not each pixel corresponds to the key point in an image space corresponding to the input image as a probability value.

8. The apparatus according to claim 7, wherein an objective function L.sub.prediction of the key point model is expressed through the following equation:
L.sub.prediction=−Σ{p.sub.target(x.sub.i,y.sub.i)log(p(x.sub.i,y.sub.i))+(1−p.sub.target(x.sub.i,y.sub.i))log(1−p(x.sub.i,y.sub.i))}  [Equation] where, p(x.sub.i,y.sub.i): probability value of whether or not pixel p(x.sub.i,y.sub.i) is key point; p(x.sub.i,y.sub.i)=probability distribution (P(F(x.sub.i,y.sub.i); δ)); P: neural network constituting key point model; F(x.sub.i,y.sub.i): feature tensor of pixel (x.sub.i,y.sub.i); δ: parameter of key point model; and p.sub.target(x.sub.i,y.sub.i): value of whether or not pixel (x.sub.i,y.sub.i) of the input image is labelled key point.

9. The apparatus according to claim 8, wherein optimized parameters of the key point model and the image reconstruction model are calculated through the following equation:
δ*,η*=argmin.sub.δ,η(αL.sub.prediction+βL.sub.reconstruction)   [Equation] where, δ*: optimized parameter of key point model; η*: optimized parameter of image reconstruction model; α: weight of objective function of key point model; β: weight of objective function of image reconstruction model; L.sub.reconstruction: objective function of the image reconstruction model, L.sub.reconstruction=∥I−H(P(I;δ);η)∥; I: input image; H: neural network constituting image reconstruction model; and η: parameters of image reconstruction model.

10. A neural network-based training method performed in a computing device including one or more processors and a memory for storing one or more programs executed by the one or more processors, the method comprising: a step of learning to extract key points from an input image, in a key point model; and a step of learning to reconstruct the input image with the key points output by the key point model as the input, in an image reconstruction model.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0048] FIG. 1 is a block diagram schematically illustrating a configuration of a key point training apparatus according to an embodiment of the present invention.

[0049] FIGS. 2A and 2B are diagrams schematically illustrating another training method of the key point training apparatus according to an embodiment of the present invention.

[0050] FIG. 3 is a block diagram illustrating a configuration of a key point training apparatus according to a first embodiment of the present invention.

[0051] FIG. 4 is a block diagram illustrating a configuration of a key point training apparatus according to a second embodiment of the present invention.

[0052] FIG. 5 is a block diagram illustratively describing a computing environment including a computing device suitable for use in example embodiments.

DETAILED DESCRIPTION

[0053] Hereinafter, a specific embodiment of the present invention will be described with reference to the drawings. The following detailed description is provided to aid in a comprehensive understanding of the methods, apparatus and/or systems described herein. However, this is illustrative only, and the present invention is not limited thereto.

[0054] In describing the embodiments of the present disclosure, when it is determined that a detailed description of related known technologies related to the present invention may unnecessarily obscure the subject matter of the present invention, a detailed description thereof will be omitted. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to the intention or custom of users or operators. Therefore, the definition should be made based on the contents throughout this specification. The terms used in the detailed description are only for describing embodiments of the present invention, and should not be limiting. Unless explicitly used otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as “comprising” or “including” are intended to refer to certain features, numbers, steps, actions, elements, some or combination thereof, and it is not to be construed to exclude the presence or possibility of one or more other features, numbers, steps, actions, elements, some or combinations thereof, other than those described.

[0055] In the following description, the terms “transfer”, “communication”, “transmission”, “reception” and other terms with similar meanings include not only direct transmission of a signal or information from one component to another component, but also transmission through another component. In particular, “transferring” or “transmitting” a signal or information to a component indicates the final destination of the signal or information and does not mean a direct destination. The same is true for “reception” of the signal or information. In addition, in this specification, when two or more data or information are “related”, it means that when one data (or information) is acquired, at least a part of the other data (or information) can be acquired based thereon.

[0056] In addition, terms such as the first and second may be used to describe various components, but the components should not be limited by the terms. The above terms may be used for the purpose of distinguishing one component from another component. For example, without departing from the scope of the present invention, a first component may be referred to as a second component, and similarly, the second component may also be referred to as the first component.

[0057] FIG. 1 is a block diagram schematically illustrating a configuration of a key point training apparatus according to an embodiment of the present invention.

[0058] Referring to FIG. 1, a key point training apparatus 100 may include a key point model 102 and an image reconstruction model 104. In an exemplary embodiment, a case in which the key point training apparatus 100 extracts key points from a face image and reconstructs the face image based on the extracted key points is described as an example, but the present invention is not limited thereto and may be applied to various other images.

[0059] In addition, the key point training apparatus 100 may be implemented with a convolutional neural network (CNN)-based machine training technique, but the machine training technique is not limited the CNN-based machine training technique, and various other machine training techniques may be applied thereto. In addition, in an exemplary embodiment, the key point model 102 and the image reconstruction model 104 may be configured with an encoder-decoder type of neural network, but are not limited thereto.

[0060] The key point model 102 may be trained to extract key points from an image input (input image). In an exemplary embodiment, the input image may be an image including a face, but is not limited thereto. The key point model 102 may be trained to minimize differences between the key points extracted from the input image and key points labelled with respect to the input image (i.e., correct answer values).

[0061] In an exemplary embodiment, the key point model 102 may include one or more convolutional layers and one or more pooling layers. The convolution layer may extract feature values (i.e., key points) of pixels corresponding to a filter of a preset size (e.g., 3×3 pixel size) while moving the filter in the input image at regular intervals. The pooling layer may perform down sampling by receiving an output of the convolution layer as an input.

[0062] The image reconstruction model 104 may receive the key points output by the key point model 102. The image reconstruction model 104 may be trained to reconstruct the input image from the input key points. The image reconstruction model 104 may be trained to minimize a difference between an image reconstructed from the key points and the input image input to the key point model 102.

[0063] That is, the image reconstruction model 104 is trained to reconstruct the input image through the key points output by the key point model 102. In this case, the key point model 102 is primarily trained to minimize the differences between the key points extracted from the input image and the key points labelled for the input image, and is secondarily trained to extract the key points so that the difference between the input image and the reconstructed image reconstructed by the image reconstruction model 104 is minimized.

[0064] Here, since the input image itself does not have noise over time (i.e., annotation noise) unlike the key points, it is possible to extract key points that are continuous and stable over time while training is in progress. In addition, as the accuracy of key point extraction is increased and key points stable over time are extracted, quality of the reconstructed image reconstructed by the image reconstruction model 104 may also be improved.

[0065] Meanwhile, in FIG. 1, a case in which the key point model 102 and the image reconstruction model 104 are simultaneously trained in one neural network model has been described as an example, but the present invention is not limited thereto. As illustrated in FIGS. 2A and 2B, it is possible to train a first machine learning model M1 so that the key point model 102 is allowed to extract the key points from the input image in a first machine learning model M1, but the differences between the extracted key points and the key points labelled with respect to the input image is minimized (FIG. 2A). In addition, by adding the image reconstruction model 104 to an output end of the trained first machine training model M1, the image reconstruction model 104 may be trained to reconstruct the input image based on the key points output from the first machine training model M1 (FIG. 2B). That is, after training the first machine training model M1 including the key point model 102, the image reconstruction model 104 may be added to the first machine training model M1.

[0066] FIG. 3 is a block diagram illustrating a configuration of a key point training apparatus according to a first embodiment of the present invention.

[0067] Referring to FIG. 3, a key point training apparatus 200 may include a key point model 202 and an image reconstruction model 204.

[0068] The key point model 202 may include an extraction module 202a and a prediction module 202b. The extraction module 202a may be trained to extract a feature vector from an input image. The extraction module 202a may extract the feature vector from the input image through a plurality of convolutional neural network layers. In an exemplary embodiment, the extraction module 202a may be configured with an image encoder.

[0069] The prediction module 202b may be trained to predict key point coordinates of the input image based on the feature vector extracted by the extraction module 202a.

[0070] That is, the prediction module 202b may be trained to predict which coordinate values (x, y) correspond to a key point in a coordinate system of the input image based on the extracted feature vector

[0071] Here, the key point coordinates K of the input image may be expressed by Equation 1 below.

K=[(x.sub.1, y.sub.1), (x.sub.2, y.sub.2) , . . . , (x.sub.n, y.sub.n)]  (Equation 1)

[0072] x.sub.n: x-axis coordinate value of n-th key point

[0073] y.sub.n: y-axis coordinate value of n-th key point

[0074] Predicting and extracting key point coordinates from the input image in by the key point model 202 may be expressed through Equation 2 below.

K′=G(I;θ)   (Equation 2)

[0075] Here, K′ represents the key point coordinates predicted from the key point model 202, G represents the neural network constituting the key point model 202, I represents the input image, and θ represents the parameters of the neural network G.

[0076] The image reconstruction model 204 may be trained to reconstruct the input image based on key point coordinates output by the key point model 202. In an exemplary embodiment, the image reconstruction model 204 may be configured as a decoder.

[0077] Reconstructing the input image based on the key point coordinates by the image reconstruction model 204 may be expressed through Equation 3 below.

I′=D(K′;φ)   (Equation 3)

[0078] Here, I′ represents the image reconstructed by the image reconstruction model 204, D represents the neural network constituting the image reconstruction model 204, and φ represents the parameters of the neural network D

[0079] Meanwhile, the key point model 202 may be trained to minimize the difference between the key point coordinates extracted from the input image and the labelled key point coordinates (i.e., the correct answer value). In addition, the image reconstruction model 204 may be trained to minimize the difference between the reconstructed image and the input image based on the key point coordinates extracted by the key point model 202.

[0080] In this case, an objective function L.sub.prediction of the key point model 202 may be expressed through Equation 4 below.

L.sub.prediction=∥K−G(I;θ)∥  (Equation 4)

[0081] K represents the labelled key point coordinates of the input image, and the ∥A−B∥ function is a function to find the difference between A and B (e.g., a function to find the Euclidean distance (L2 distance) or Manhattan distance (L1 distance) between A and B), etc.).

[0082] In addition, an objective function L.sub.reconstruction of the image reconstruction model 204 may be expressed through Equation 5 below.

L.sub.reconstruction=∥I−D(G(I;θ)   (Equation 5)

[0083] In addition, the optimized parameters (θ*,φ*) of the key point model 202 and the image reconstruction model 204 may be expressed through Equation 6 below.

θ*,φ*=argmin.sub.θ,φ(αL.sub.prediction+βL.sub.reconstruction)   (Equation 6)

[0084] Here, argmin.sub.θ,φ(A+B) denotes a function to find θ and φ that minimize (A+B). α denotes the weight of the objective function of the key point model 202 and β denotes the weight of the objective function of the image reconstruction model 204.

[0085] FIG. 4 is a block diagram illustrating a configuration of a key point training apparatus according to a second embodiment of the present invention.

[0086] Referring to FIG. 4, a key point training apparatus 300 may include a key point model 302 and an image reconstruction model 304.

[0087] The key point model 302 may include an extraction module 302a and a prediction module 302b. The extraction module 302a may be trained to extract a feature tensor from an input image. The extraction module 302a may extract the feature tensor so that the key point is expressed as a point on an image space corresponding to the input image. By configuring the extraction module 302a with an encoder-decoder type neural network, it is possible to increase the efficiency of an image-to-image training scheme.

[0088] The prediction module 302b may be trained to predict a key point image based on the feature tensor extracted by the extraction module 302a. Here, the key point image indicates whether or not each pixel corresponds to the key point in the image space corresponding to the input image. The key point image may be an image in which, for each pixel, a pixel value is 1 if each pixel corresponds to the key point and a pixel value is 0 if each pixel does not correspond to the key point.

[0089] In an exemplary embodiment, the prediction module 302b may predict the key point image by outputting a probability value between 0 and 1 (i.e., a probability value of whether or not each pixel is a key point) for each pixel based on the extracted feature tensor. Outputting a probability value for each pixel by the prediction module 302b may be expressed through Equation 7 below.

p(x.sub.i,y.sub.i)=probability distribution (P(F(x.sub.i,y.sub.i);δ))   (Equation 7)

[0090] Here, p(x.sub.i,y.sub.i) denotes a probability value of whether or not a pixel (x.sub.i,y.sub.i) is a key point, P denotes a neural network constituting the key point model 302, F(x.sub.i,y.sub.i) denotes a feature tensor of the pixel (x.sub.i,y.sub.i), and δ denotes the parameter of the neural network P. In addition, as the probability distribution function, Sigmoid, Gaussian, etc. may be used, but is not limited thereto.

[0091] Meanwhile, the objective function L.sub.prediction of the key point model 302 may be expressed through Equation 8 below.

L.sub.prediction=−Σ{p.sub.target(x.sub.i,y.sub.i)log(p(x.sub.i,y.sub.i))+(1−p.sub.target(x.sub.i,y.sub.i))log(1−p(x.sub.i,y.sub.i))}   (Equation 8)

[0092] Here, p.sub.target(x.sub.i,y.sub.i) denotes a value of whether or not the pixel (x.sub.i,y.sub.i) of the input image is a labelled key point. That is, it may be labelled to have a value of 1 when the corresponding pixel is a key point and have a value of 0 when it is not the key point.

[0093] According to Equation 8, when the value of whether or not the pixel is the labelled key point is 1, it is trained to increase the probability value (i.e., p(x.sub.i,y.sub.i)) of whether or not the pixel is the key point, and when the value of whether or not the pixel is the labelled key point is 0, it is trained to reduce the probability value (i.e., p(x.sub.i,y.sub.i)) of whether or not the pixel is the key.

[0094] The image reconstruction model 304 may be trained to reconstruct the input image based on the key point image output by the key point model 302. In an exemplary embodiment, the image reconstruction model 304 may be configured with an encoder-decoder type of neural network. The objective function L.sub.reconstruction of the image reconstruction model 304 may be expressed through Equation 9 below.

L.sub.reconstruction=∥I−H(P(I;δ);η)∥  (Equation 9)

[0095] Here, I denotes an input image, H denotes a neural network constituting the image reconstruction model 304, and η denotes a parameter of the neural network H.

[0096] In addition, the optimized parameters (δ*/η*) of the key point model 302 and the image reconstruction model 304 may be expressed through Equation 10 below.

δ*,η*=argmin.sub.δ,η(αL.sub.prediction+βL.sub.reconstruction)   (Equation 10)

[0097] Meanwhile, in this specification, a module may mean a functional and structural combination of hardware for performing the technical idea of the present invention and software for driving the hardware. For example, the “module” may mean a logical unit of a predetermined code and a hardware resource for executing the predetermined code, and does not necessarily mean physically connected code or a single type of hardware.

[0098] FIG. 5 is a block diagram illustratively describing a computing environment 10 including a computing apparatus suitable for use in exemplary embodiments. In the illustrated embodiment, respective components may have different functions and capabilities other than those described below, and may include additional components in addition to those described below.

[0099] The illustrated computing environment 10 includes a computing apparatus 12. In an embodiment, the computing apparatus 12 may be the key point training apparatus 100, 200, or 300.

[0100] The computing apparatus 12 includes at least one processor 14, a computer-readable storage medium 16, and a communication bus 18. The processor 14 may cause the computing apparatus 12 to operate according to the exemplary embodiment described above. For example, the processor 14 may execute one or more programs stored on the computer-readable storage medium 16. The one or more programs may include one or more computer-executable instructions, which, when executed by the processor 14, may be configured such that the computing apparatus 12 performs operations according to the exemplary embodiment.

[0101] The computer-readable storage medium 16 is configured such that the computer-executable instruction or program code, program data, and/or other suitable forms of information are stored. A program 20 stored in the computer-readable storage medium 16 includes a set of instructions executable by the processor 14. In one embodiment, the computer-readable storage medium 16 may be a memory (volatile memory such as a random access memory, non-volatile memory, or any suitable combination thereof), one or more magnetic disk storage devices, optical disk storage devices, flash memory devices, other types of storage media that are accessible by the computing apparatus 12 and capable of storing desired information, or any suitable combination thereof.

[0102] The communication bus 18 interconnects various other components of the computing apparatus 12, including the processor 14 and the computer-readable storage medium 16.

[0103] The computing apparatus 12 may also include one or more input/output interfaces 22 that provide an interface for one or more input/output devices 24, and one or more network communication interfaces 26. The input/output interface 22 and the network communication interface 26 are connected to the communication bus 18. The input/output device 24 may be connected to other components of the computing apparatus 12 through the input/output interface 22. The exemplary input/output device 24 may include a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touch pad or touch screen), a voice or sound input device, input devices such as various types of sensor devices and/or photographing devices, and/or output devices such as a display device, a printer, a speaker, and/or a network card. The exemplary input/output device 24 may be included inside the computing apparatus 12 as a component constituting the computing apparatus 12, or may be connected to the computing apparatus 12 as a separate device distinct from the computing apparatus 12.

[0104] Although representative embodiments of the present invention have been described in detail, those skilled in the art to which the present invention pertains will understand that various modifications may be made thereto within the limits that do not depart from the scope of the present invention. Therefore, the scope of rights of the present invention should not be limited to the described embodiments, but should be defined not only by claims set forth below but also by equivalents to the claims.