METHOD AND APPARATUS FOR ESTIMATING HUMAN MOTION KINETICS INFORMATION

Abstract

A method and apparatus for estimating kinematic information of human motion are provided. The method and apparatus comprises, calculating a length change and angle of a leg spring from a trajectory of human motion by using a compliant walking model modeling a human leg as the spring, and estimating kinematic information including at least one of a ground reaction force, a lower limb angle, an ankle joint trajectory and an ankle joint torque by using the length change and the angle. The compliant walking model comprises an off-centered curvy foot modeling a human foot, a lower limb-ankle spring modeling ranging from the CoM to the ankle joint, and an ankle-foot spring modeling ranging from the ankle joint to the curvy foot.

Claims

1. A method of estimating kinematic information of human motion in an apparatus operated by at least one processor, the method comprising: generating a default compliant walking model including a leg spring modeling a leg of a human body as a spring and a curvy foot modeling a human foot in a curved shape; receiving a center of mass (CoM) trajectory of a moving user; generating a compliant walking model for the user by calculating variables related with a length change and an angle of the leg spring, which constitute the default compliant walking model, based on the CoM trajectory; and estimating kinematic information including at least one of a ground reaction force, a lower limb joint angle, an ankle joint trajectory, and an ankle joint torque, by using the compliant walking model for the user, wherein one end of the leg spring is connected to the CoM and the other end of the leg spring is connected to a position apart from a center of the curvy foot by a predefined distance, and wherein the leg spring is divided into a lower limb-ankle spring modeling ranging from the CoM to the ankle joint and an ankle-foot spring modeling ranging from the ankle joint to the curvy foot.

2. The method of claim 1, wherein generating the compliant walking model for the user comprises calculating the length change and the angle by using the CoM trajectory, a radius of curvature of the curvy foot, the predefined distance, and a whole length of the leg spring.

3. The method of claim 2, wherein estimating the kinematic information comprises calculating a constraint force of the ankle joint by using a constant of the leg spring, the length change, the radius of curvature, an offset value, the angle and an initial length of the leg spring, and calculating the ground reaction force by using the constant, the constraint force, the length change, and the angle

4. The method of claim 3, wherein estimating the kinematic information further comprises calculating an ankle joint torque and an ankle joint trajectory, by using the ground reaction force, the radius of curvature, the offset value, the initial length of the leg spring, the length change, the angle, a ratio between a length of the lower limb-ankle spring and a length of the ankle-foot spring, and a ratio between a stiffness of the lower limb-ankle spring and a stiffness of the ankle-foot spring.

5. The method of claim 4, wherein estimating the kinematic information further comprises calculating a length and an angle of a lower limb joint from the CoM trajectory and the ground reaction force, respectively, and wherein the length and the angle of the lower limb joint includes a length and an angle of a thigh segment ranging from the CoM to a knee, a length and an angle of a shank segment ranging from the knee to the ankle joint, and a length and an angle of a foot segment ranging from the ankle joint to the curvy foot.

6. The method of claim 1, further comprising, before calculating the variables, from a human motion image, obtaining the CoM trajectory including position data of each point at a predetermined time interval from a point where a movement of a specific part of human body starts to a point where the movement ends.

7. The method of claim 1, further comprising, after estimating the kinematic information, correcting the estimated kinematic information by using a machine learning model previously trained with the estimated kinematic information.

8. An estimation apparatus comprising: a memory that stores a default compliant walking model including a leg spring modeling a leg of a human body as a spring and a curvy foot modeling a human foot in a curved shape, and a program for estimating kinematic information from a trajectory of a center of mass (CoM) of the human body; and at least one processor that executes the program, wherein the program includes instructions to receive a trajectory of the CoM of a moving user, and to generate a compliant walking model for the user by calculating variables related with a length change and an angle of the leg spring, which constitute the default compliant walking model, based on the CoM trajectory, and to estimate kinematic information including at least one of a ground reaction force, a lower limb joint angle, an ankle joint trajectory, and an ankle joint torque, by using the compliant walking model for the user, and wherein one end of the leg spring is connected to the CoM and the other end of the leg spring is connected to a position apart from a center of the curvy foot by a predefined distance, and wherein the leg spring is divided into a lower limb-ankle spring modeling ranging from the CoM to the ankle joint and an ankle-foot spring modeling ranging from the ankle joint to the curvy foot.

9. The apparatus of claim 8, wherein the program includes instructions to calculate a length change of the leg spring, and an angle of the leg spring moved from a vertical line connecting the CoM and ground surface during human motion, by using a two-dimensional position data, a radius of curvature of the curvy foot, the predefined distance and a whole length of the leg spring.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] FIG. 1 is a block diagram showing a configuration of an apparatus for estimating kinematic information of human motion according to an embodiment of the present disclosure.

[0020] FIG. 2 is diagram showing a trajectory of a center of mass (CoM) according to an embodiment of the present disclosure.

[0021] FIG. 3 is a graph showing a trajectory of a CoM according to an embodiment of the present disclosure.

[0022] FIG. 4 is a diagram for describing a human walking model according to an embodiment of the present disclosure.

[0023] FIG. 5 is a diagram for describing parameters of a human walking model according to an embodiment of the present disclosure.

[0024] FIG. 6 is a diagram for describing parameters of a human walking model according to another embodiment of the present disclosure.

[0025] FIG. 7 is a diagram for describing parameters of a human walking model according to yet another embodiment of the present disclosure.

[0026] FIG. 8 is a flowchart showing a method for estimating kinematic information of a human motion according to an embodiment of the present disclosure.

[0027] FIG. 9 is a diagram for explaining a correction process of the kinematic information of human motion according to an embodiment of the present disclosure.

[0028] FIG. 10 is a block diagram showing hardware configuration of an apparatus for estimating kinematic information of human motion according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings so that the person of ordinary skill in the art may easily implement the present disclosure. The present disclosure may be modified in various ways and is not limited to the embodiment described herein. In the drawings, elements irrelevant to the description of the present disclosure are omitted for clarity of explanation, and like reference numerals designate like elements throughout the specification.

[0030] Throughout the specification, when a part is referred to include a certain element, it means that it may further include other elements rather than exclude other elements, unless specifically indicates otherwise.

[0031] In addition, the term such as . . . unit, . . . block, . . . module, or the like described in the specification mean a unit that processes at least one function or operation, which may be implemented with a hardware, a software or a combination thereof.

[0032] Hereinafter, a method and an apparatus for estimating kinematics information of human motion according to embodiments of the present disclosure are described with reference to the accompanying drawings.

[0033] FIG. 1 is a block diagram showing a configuration of an apparatus for estimating kinematic information of human motion according to an embodiment of the present disclosure. FIG. 2 is diagram showing a trajectory of a center of mass (CoM) according to an embodiment of the present disclosure. FIG. 3 is a graph showing a trajectory of a CoM according to an embodiment of the present disclosure. FIG. 4 is a diagram for describing a human walking model according to an embodiment of the present disclosure. FIG. 5 is a diagram for describing parameters of a human walking model according to an embodiment of the present disclosure. FIG. 6 is a diagram for describing parameters of a human walking model according to another embodiment of the present disclosure. And, FIG. 7 is a diagram for describing parameters of a human walking model according to yet another embodiment of the present disclosure.

[0034] Referring to FIG. 1, a kinematic information estimation apparatus 100 includes a motion trajectory generation unit 101, a kinematic information estimation unit 103, and a kinematic information correction unit 105.

[0035] The motion trajectory generation unit 101 generates a trajectory P11 of a CoM of a human body as shown in FIG. 2.

[0036] According to an embodiment, the motion trajectory generation unit 101 generates a motion trajectory of the human body from the human motion image taken by at least one image taking means (not shown). The human motion image may include a walking image, a running image, and the like. The image taking means may be a high speed camera designed for high speed photographing.

[0037] The motion trajectory generation unit 101 estimates a trajectory of a feature point of the human body from an RGB image and derives a trajectory of the CoM from the estimated trajectory of the feature point. Here, the feature point of the human body may be selected from body parts having oscillatory motion such as a head, a shoulder, an arm, a waist, a thigh, a knee, an ankle, and the like.

[0038] The motion trajectory generation unit 101 specifies a specific body part, for example, a waist, in the RGB motion image of the human body. The motion trajectory generation unit 101 tracks an image area including the waist in the RGB motion image of the human body. Here, for tracking the image area, various image analysis techniques may be used, such as an object area tracking algorithm, a histogram-based mean-shift tracking method, a difference image method where an image fragment of a specific area including a feature in an image is used to identify correlation with another image, a block matching method, and the like. However, the image analysis techniques are not limited thereto. The motion trajectory generation unit 101 extracts a waist position shown in pixel unit from the tracked image area. And, the motion trajectory generation unit 101 may obtain a change length represented by a unit pixel from a height of a subject in the motion image, and may convert the extracted waist position in unit of centimeter using the change length.

[0039] At this time, the extracted waist position reflects a movement of not the center of body but the surface of the body. Since the waist position of the body center should be obtained from that of the surface in order to extract the CoM, symmetry of the motion is considered.

[0040] In addition, since the CoM reflects mass distribution according to a shape change of a lower limb during movement, the CoM differs from the extracted waist position which is a fixed point on the body. Therefore, the motion trajectory generation unit 101 speculates the shape of the lower limb from the extracted waist position with a stride and a leg length, and derives the CoM in consideration of a mass ratio among each of the body parts. The graph shown in FIG. 3 represents the derived CoM.

[0041] The kinematic information estimation unit 103 estimates various kinematic information by using the CoM trajectory output from the motion trajectory generation unit 101 and a compliant working model. The estimated kinematic information includes a ground reaction force (GRF), an ankle joint torque, an ankle joint trajectory, a foot angle, a lower-limb joint angle, and the like.

[0042] The kinematic information estimation unit 103 applies a compliant walking model to the CoM trajectory (x.sub.com, y.sub.com) as shown in FIG. 4. The compliant walking model includes an off-centered curvy foot (hereinafter referred to as curvy foot) P1 and a leg spring P3 connecting the curvy foot P1 and the CoM. The leg spring P3 is composed of a lower-limb spring 1 (P7) supporting the CoM at the ankle P5 and a lower-limb spring 2 (P9) supporting the ankle at the curvy foot P1.

[0043] Kinematic information estimation unit 103 calculates parameters of the compliant walking model. The parameters of the compliant walking model will be described with reference to FIG. 5 to FIG. 7.

[0044] Hereinafter, since the parameters represented by the same symbol in equations are the same, descriptions on the same parameter will not be repeated.

[0045] Referring to FIG. 5, L.sub.0 is a length of the unstretched leg spring ranging from the CoM to the ground P11. L.sub.0 satisfying a boundary condition is defined by the following Equation 1.

(t.sup.)=(t.sup.+)=0Equation 1

[0046] is a change length of the leg spring. The kinematic information estimation unit 103 sets initial L.sub.0 as 1 m, and obtains from the time point t of the heel contact to the time point t.sup.+ of detach. Here, an obtained positive at the time of detach means that the initial L.sub.0 is long and an obtained negative at the time of detach means that the initial L.sub.0 is short. In consideration of this, the kinematic information estimation unit 103 calculates L.sub.0 that makes the at the time of detach be zero, as changing the length of L.sub.0.

[0047] L.sub.0 is divided into a length l.sub.0 of the lower limb spring 1 ranging from the CoM to an ankle and a length h.sub.0 of the lower limb spring 2 ranging from the ankle to the curvy foot. The length l.sub.0 of the lower limb spring 1 and the length h.sub.0 of the lower limb spring 2 satisfy the following equation 2.

h.sub.0=r@l.sub.0Equation 2

[0048] Here, r a proportional factor, and h.sub.0 is a fixed value.

[0049] The CoM trajectory (x.sub.com, y.sub.com) is as shown in the following Equation 3.

x.sub.com=R+(L.sub.0{square root over (R.sup.2d.sup.2)})sin d cos

y.sub.com=R+(L.sub.0{square root over (R.sup.2d.sup.2)})cos +d sin Equation 3

[0050] Here, state variables are and . is an angle between a leg spring and a line (shown by dotted line in FIG. 5) vertically projected from the CoM, that is, a leg angle. is a length change of the leg spring L. R is a radius of curvature of the curvy foot, and d is an offset amount of the curvy foot. R and d are fixed values.

[0051] The CoM trajectory (x.sub.com, y.sub.com) is obtained through the motion trajectory generation unit 101. Since L.sub.0 is calculated through Equation 1 and R and d are fixed values, and can be obtained through Equation 3.

[0052] FIG. 6 shows degree of freedom of a human body, Fx and Fy represent ground reaction forces (GRFs). The ground reaction force (GRF) is a reaction force applied to a ground contact point of the curvy foot and is headed for the CoM. The GRF is represented by the following equation 4.

[00001] $\begin{matrix} {\begin{matrix} F_{x} = k .Math. .Math. .Math. .Math. \sin .Math. .Math. - F_{ank} .Math. \cos .Math. .Math. \\ F_{y} = k .Math. .Math. .Math. .Math. \cos .Math. .Math. + F_{ank} .Math. \sin .Math. .Math. \end{matrix} & Equation .Math. .Math. 4 \end{matrix}$

[0053] Here, F.sub.ank is a constraint force at an ankle joint. F.sub.ank is defined by the following Equation 5.

[00002] $\begin{matrix} F_{ank} =_{n}^{2} .Math. .Math. .Math. \frac{R .Math. .Math. \sin .Math. .Math. + d}{L_{0} - \sqrt{R^{2} - d^{2}} - + R .Math. .Math. \cos .Math. .Math.} & Equation .Math. .Math. 5 \end{matrix}$

[0054] Equation 6

[0055] Here, G is gravity constant. l(t) is an average value of l(t) throughout the stance phase. m is a mass of the leg spring and k is a stiffness of a lower limb joint and is defined by a spring constant. Thus, k is calculated based on body weight.

[0056] T.sub.ank represents an ankle joint torque and is as shown in the following Equation 7. The T.sub.ank is estimated from a constraint torque that aligns ankle-foot segment with the leg throughout the stance phase

[00003] $\begin{matrix} T_{ank} = F_{x} .Math. {R - (\sqrt{R^{2} - d^{2}} - \frac{r}{r + 1} .Math. L_{0} + \frac{1}{+ 1} .Math.) .Math. \cos .Math. .Math. + d .Math. .Math. \sin .Math. .Math.} + F_{y} .Math. {(\sqrt{R^{2} - d^{2}} - \frac{r}{r + 1} .Math. L_{0} + \frac{1}{+ 1} .Math.) .Math. \sin .Math. .Math. + d .Math. .Math. \cos .Math. .Math.} & Equation .Math. .Math. 7 \end{matrix}$

[0057] Here, satisfies the following equation 8.

k.sub.foot=.Math.k.sub.legEquation 8

[0058] Here, k.sub.leg represents a stiffness of the lower limb spring 1 (P7 shown in FIG. 4) ranging from the CoM to the ankle, and k.sub.foot represents a stiffness of the lower limb spring 2 (P9 shown in FIG. 4) ranging from the ankle to the foot.

[0059] In addition, the ankle joint trajectory (x.sub.ank, y.sub.ank) is calculated through the following equation 9.

[00004] $\begin{matrix} x_{ank} = R .Math. .Math. - (\sqrt{R^{2} - d^{2}} - \frac{r}{r + 1} .Math. L_{0} + \frac{1}{+ 1} .Math.) .Math. \sin .Math. .Math. - d .Math. .Math. \cos .Math. .Math. & Equation .Math. .Math. 9 \\ y_{ank} = R - (\sqrt{R^{2} - d^{2}} - \frac{r}{r + 1} .Math. L_{0} + \frac{1}{+ 1} .Math.) .Math. \cos .Math. .Math. + d .Math. .Math. \sin .Math. .Math. \end{matrix}$

[0060] Further, the kinematics of the rest of the lower limb segments such as a foot, a knee and a thigh will be described with reference to FIG. 7.

[0061] Referring to FIG. 7, the lower limb segment described in FIG. 4 to FIG. 6 is divided at a knee joint A and an ankle joint B.

[0062] l.sub.thigh represents a length of a lower limb segment corresponding to the thigh and is a length from the center of mass C to the knee joint A. .sub.thigh represents a thigh segment angle, which means an angle between the thigh segment and a line (shown as a dotted line) perpendicular to the ground.

[0063] The l.sub.shank represents a length of a lower limb segment corresponding to a shank, that is, a length from the knee joint A and the ankle joint B. .sub.shank is a shank segment angle, which means an angle between the shank segment and a line (shown as a dotted line) perpendicular to the ground.

[0064] l.sub.foot is a length from the ankle joint B to a tip of the curvy foot, from the whole lower limb segment. The l.sub.foot is calculated as a length from the ankle joint B to the point of contact with the curved surface of the curved foot in the horizontal direction. .sub.foot is a foot angle and represented by an angle from a horizontal point of the ankle joint B to a point considered as the tip of the foot.

[0065] l.sub.thigh, .sub.thigh, l.sub.shank, .sub.shank are calculated through the following Equation 10 which uses Equation 3 and Equation 9.

x.sub.ankx.sub.com=l.sub.shank sin .sub.sh+l.sub.thigh sin .sub.th

y.sub.comy.sub.ank=l.sub.shank cos .sub.sh+l.sub.thigh cos .sub.thEquation 10

[0066] l.sub.foot nd .sub.foot are calculated through the following Equation 11.

[00005] $\begin{matrix} _{foot} = - \tan .Math. .Math. (\frac{}{(+ 1) .Math. l_{foot}}) & Equation .Math. .Math. 11 \end{matrix}$

[0067] As described above, the kinematic information estimation unit 105 may estimate the ankle joint trajectory (x.sub.ank, y.sub.ank), the ground reaction force (F.sub.r, F.sub.y), the ankle joint torque (T.sub.ank), the lower limb joint angles .sub.thigh, .sub.shank, and .sub.foot, and the like, through the curvy foot compliant walking model. Therefore, more information for the lower limb segment may be estimated, while the previous compliant walking model may estimate only the ground reaction force and the joint angle.

[0068] Referring back to FIG. 1, the kinematic information correction unit 105 corrects the kinematic information estimated by the kinematic information estimation unit 103 using a machine learning algorithm. Here, the machine learning algorithm may include a deep learning model and the like.

[0069] The kinematic information correction unit 105 may establish a machine learning model by training the kinematic information estimated from various human motion images in advance. Further, the kinematic information estimated by the kinematic information estimation unit 103 may be corrected by using such machine learning model.

[0070] FIG. 8 is a flowchart showing a method for estimating kinematic information of a human motion according to an embodiment of the present disclosure. FIG. 9 is a diagram for explaining a correction process of the kinematic information of human motion according to an embodiment of the present disclosure.

[0071] Referring to FIG. 8 and FIG. 9, the motion trajectory generation unit 101 obtains a CoM trajectory (x.sub.com, y.sub.com), as described above with reference to FIG. 1 to FIG. 7 (S101). The kinematic information estimation unit 103 calculates a change length of the leg spring and a leg angle from the obtained CoM trajectory, by using the curvy foot compliant walking model (S103).

[0072] Here, the curvy foot compliant walking model is divided into a default compliant walking model and a user compliant walking model. Based on the CoM trajectory (x.sub.com, y.sub.com) received from the motion trajectory generation unit 101, the kinematic information estimation unit 103 establish a user compliant walking model by calculating variables composing the default compliant walking model, that is, and . Then, by using the user compliant walking model, that is, by applying and , the kinematic information of a user is estimated. At this time, the kinematic information estimation unit 103 calculates a spring constant k representing stiffness of the leg spring by using weight (S105). Based on the calculated values in steps S103 and S105, the kinematic information estimation unit 103 estimates lower limb joint angles .sub.high, .sub.shank, and .sub.foot, an ankle joint trajectory (x.sub.ank, y.sub.ank), ground reaction forces (F.sub.x, F.sub.y), and an ankle joint torque (T.sub.ank) (S107).

[0073] The kinematic information correction unit 105 corrects the kinematic information estimated by the kinematic information estimation unit 103 using a machine learning model (S109).

[0074] FIG. 10 is a block diagram showing hardware configuration of an apparatus for estimating kinematic information of human motion according to an embodiment of the present disclosure.

[0075] Referring to FIG. 10, the apparatus 100 for estimating kinematic information of human motion shown in FIG. 1 may be a computing device 200. The computing device 200 may include an input device 201, a memory 203, an output device 205, and at least one processor 207. The input device 201 and the output device 205 are connected to the at least one processor 207. The input device 201 receives and an image, a sensor signal, and the like and transmits them to the at least one processor 207. The output device 205 outputs estimated and corrected kinematic information of human motion on a display. The memory 203 is connected to the at least one processor 207 and stores a program. The program includes configuration and/or method according to above-described embodiments with reference to FIG. 1 to FIG. 9, that is, instructions to execute operations of the motion trajectory generation unit 101, kinematic information estimation unit 103, and the kinematic information correction unit 105. The program implements the present disclosure in combination with hardware such as the memory 203, the at least one processor 207 and the like.

[0076] Embodiments of the present disclosure described above are not implemented only through the apparatus and the method, but may be implemented through a program that realizes functions corresponding to the configuration of the embodiments of the present disclosure or a recording medium on which the program is recorded.

[0077] While the present disclosure has been illustrated and descried with reference to embodiments thereof, the right scope of the present disclosure is not limited thereto. Further, it will be understood by a person of ordinary skill in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the present disclosure as defined by the following claims.

METHOD AND APPARATUS FOR ESTIMATING HUMAN MOTION KINETICS INFORMATION

Inventors

Cpc classification

Classification Explorer

G06N7/01

PHYSICS

Classification Explorer

G16H50/50

PHYSICS

Classification Explorer

A61B5/1122

HUMAN NECESSITIES

Classification Explorer

G06N20/00

PHYSICS

Classification Explorer

A61B5/4595

HUMAN NECESSITIES

Classification Explorer

A61B2503/10

HUMAN NECESSITIES

International classification

Classification Explorer

G16H50/50

PHYSICS

Classification Explorer

A61B5/00

HUMAN NECESSITIES

Classification Explorer

G06N20/00

PHYSICS

Classification Explorer

A61B5/11

HUMAN NECESSITIES

Classification Explorer

G06N7/00

PHYSICS

Abstract

Claims

Description