Method and apparatus for sensing moving ball
11582426 · 2023-02-14
Assignee
Inventors
Cpc classification
G06T7/246
PHYSICS
H04N13/239
ELECTRICITY
H04N7/181
ELECTRICITY
G06V10/42
PHYSICS
International classification
H04N7/18
ELECTRICITY
G06T7/246
PHYSICS
G06V10/42
PHYSICS
Abstract
Provided are an apparatus and method for sensing a moving ball, which extract a feature portion such as a trademark, a logo, etc. indicated on a ball from consecutive images of a moving ball, acquired by an image acquisition unit embodied by a predetermined camera device, and calculate a spin axis and spin amount of rotation the moving ball based on the feature portion and thus spin of the ball is simply, rapidly, and accurately calculated with low computational load, thereby achieving rapid and stable calculation of the ball in a relatively low performance system. The sensing apparatus includes an image acquisition unit for acquiring consecutive images, an image processing unit for extracting a feature portion from the acquired image, and a spin calculation unit for calculating spin using the extracted feature portion.
Claims
1. A method of sensing a moving ball, for acquiring and analyzing an image of the moving ball to calculate spin of the moving ball, the method comprising: acquiring consecutive images by an image acquisition unit according to movement of a ball with a surface having a predetermined feature portion indicated thereon, wherein the feature portion is an unspecified indication including a ball's trademark or a logo and a scratch on the ball; preparing a first image and a second image consecutively acquired by the image acquisition unit; finding a ball portion on the first image and extracting the ball portion having a predetermined size from the first image as a first ball image; finding a ball portion on the second image and extracting the ball portion having the predetermined size from the second image as a second ball image; extracting the feature portion of the ball from the first ball image to prepare a first feature portion image; extracting the feature portion of the ball from the second ball image to prepare a second feature portion image; converting position information of each pixel on the first feature portion image into 3D position information, applying a trial spin axis and a trial spin amount to the converted 3D position information, and re-converting the 3D position information to which the trial spin axis and the trial spin amount have been applied into 2D position information, wherein the re-converted 2D position information is prepared as a trial spin feature portion; and comparing information of the trial spin feature portion with information of the second feature portion image to calculate a similarity, and determining the trial spin axis and the trial spin amount corresponding to the trial spin feature portion as a final spin axis and spin amount according to the similarity between information of the trial spin feature portion and information of the second feature portion image, wherein the method further comprises calculating position correction information for correcting a position and direction of a camera as if respective balls on consecutively acquired images are viewed at the same position and direction with respect to the camera, wherein the position correction information is applied to the converted 3D position information to apply the trial spin information and to calculate spin information.
2. A method of sensing a moving ball, for acquiring and analyzing an image of the moving ball to calculate spin of the moving ball, the method comprising: acquiring consecutive images by an image acquisition unit according to movement of a ball with a surface having a predetermined feature portion indicated thereon, wherein the feature portion is an unspecified indication including a ball's trademark or a logo and a scratch on the ball; preparing a first image and a second image consecutively acquired by the image acquisition unit; finding a ball portion on the first image and extracting the ball portion having a predetermined size from the first image as a first ball image; finding a ball portion on the second image and extracting the ball portion having the predetermined size from the second image as a second ball image; extracting the feature portion of the ball from the first ball image to prepare a first feature portion image; extracting the feature portion of the ball from the second ball image to prepare a second feature portion image; searching for a trial axis and a trial spin amount in 3D space, for allowing the first feature portion image into the second feature portion image; converting position information of each pixel on the first feature portion image into 3D position information, applying the searched trial spin axis and the searched trial spin amount to the converted 3D position information, and re-converting the 3D position information to which the trial spin axis and the trial spin amount have been applied into 2D position information, wherein the re-converted 2D position information is prepared as a trial spin feature portion; and comparing information of the trial spin feature portion with information of the second feature portion image to calculate the number of corresponding pixels between information of the trial spin feature portion and information of the second feature portion image; and determining the trial spin axis and the trial spin amount corresponding to the trial spin feature portion as a final spin axis and spin amount information according to the number of corresponding pixels between information of the trial spin feature portion and information of the second feature portion image, wherein the method further comprises calculating position correction information for correcting a position and direction of a camera as if respective balls on consecutively acquired images are viewed at the same position and direction with respect to the camera, wherein the position correction information is applied to the converted 3D position information to apply the trial spin information and to calculate spin information.
3. The method according to claim 2, wherein the trial spin axis information applied to the converted three-dimensional position information comprises spin amount information arbitrarily selected from a preset range according to maximum spin limitation for rotation by user hit and a frame rate of a camera for capturing an image.
4. The method according to claim 2, wherein the searching for of the spin axis and the spin amount comprises collecting based on a predetermined function a plurality of spin information calculated by performing the step of applying trial spin information, the step of comparing position information, and the step of determining the spin axis and spin amount of the trial spin information as final spin axis and spin amount for each of a plurality of pairs, each pair of images comprising a preceding one and a later one of two consecutive images to calculate the spin information of the moving ball.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
(1) The above and other objects, features and other advantages of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
DETAILED DESCRIPTION
(13) Hereinafter, exemplary embodiments of a method and apparatus for sensing a moving ball according to the present invention will be described in detail with reference to the accompanying drawings.
(14) The present invention may be basically configured to photograph a golf ball (hereinafter, referred to as a “ball”), which is hit by a golf club of a user, via a predetermined camera, to analyze the captured image, and to calculate spin of the ball in flight. Here, the camera may be a three-dimensional camera or a stereo camera configured by a plurality of cameras in a stereo manner and may be configured to convert coordinates of a two-dimensional image of the ball into three-dimensional coordinates or vice versa.
(15) In addition, the method and apparatus for sensing a moving ball according to the present invention may be applied to various fields such as analysis of a ball in flight according to golf swing of a user, a virtual golf using a virtual reality-based simulation, and so on.
(16) First, an apparatus for sensing a moving ball according to an embodiment of the present invention will be described with reference to
(17) As illustrated in
(18) The image acquisition unit 100 may be a camera device and may be embodied through a three-dimensional camera device or a stereo camera including a plurality of cameras, as described in the background art.
(19) The image processing unit 200 is a component that extracts a ball image that is an image corresponding to a ball portion from each image acquired by the image acquisition unit 100 and removes a dimple portion and various noise portions from the ball image to extract a feature portion of the ball image, that is, an unspecified indication such as a trademark or a logo indicated on the ball, a scratch, etc.
(20) The image processing unit 200 may include a ball image extraction part 210 and a ball feature extraction part 230.
(21) The ball image extraction part 210 is configured to extract a ball image that is an image corresponding to a ball portion from a source image that is an image acquired by the image acquisition unit 100 and to extract a central coordinate of the ball portion, which will be described below in detail.
(22) The ball feature extraction part 230 is a component that normalizes size, brightness, and so on of each ball image extracted by the ball image extraction part 210 and extracts the feature portion (i.e., an unspecified indication such as a trademark or a logo indicated on the ball, a scratch, etc.) indicated on the image, which will be described below in detail.
(23) The spin calculation unit 300 is configured to analyze a position change in feature portions that are respectively extracted from two consecutive ball images, that is, to analyze a spin axis and spin amount in a predetermined three-dimensional space, by which a feature portion of a preceding image is converted into a feature portion on a subsequent image to calculate final spin axis and spin amount information of the ball. As illustrated in
(24) The position correction part 310 facilitates spin calculation when balls in two consecutive images are viewed at corresponding positions and the same angle for spin calculation of a ball to accurately calculate spin, which will be described below in detail.
(25) According to the present invention, analysis is basically performed in units of images among images of a consecutively photographed moving ball. When a preceding image is referred to as a first image and a subsequent image is referred to as a second image, spin when a ball state of a point for acquisition of the first image is converted into a ball state of a point for acquisition of the second image may be calculated and, in detail, may be calculated using a method of calculating a spin axis and spin amount for allowing a position of a feature portion in the first image to be converted into a position of a feature portion of the second image.
(26) The spin applying part 320 extracts and applies trial spin axis and spin amount information in order to calculate the spin axis and spin amount for allowing the position of the feature portion on the first image to be converted into the position of the feature portion of the second image. Thus, the spin determination part 330 may determine whether the trial spin axis and spin amount information applied by the spin applying part 320 are close to target spin axis and spin amount information to calculate final spin information.
(27) That is, until the final spin information is calculated, the spin applying part 320 extracts and applies a trial spin axis and a trial spin amount every time, and the spin determination part 330 checks the application result and determines the most appropriate trial spin information as final spin information, which will be described below in detail.
(28) As illustrated in
(29) That is, the first camera 110 and the second camera 120 may each be an image acquisition device having a stereoscopic configuration, and thus, may extract three-dimensional coordinate information of an object from images acquired by photographing the same object via two cameras, i.e., the first camera 110 and the second camera 120. In
(30) In this state, some of consecutively acquired images of any one camera are illustrated in
(31) That is,
(32) As seen from
(33) Here, the images illustrated in
(34) With regard to the ball portions 21, 22, and 23 on the source image, positions of feature portions F1, F2, and F3 indicated on a surface of the ball change as the ball rotates.
(35) Here, interest objects are the feature portions F1, F2, and F3 on the ball. In this regard, the feature portions F1, F2, and F3 may be accurately extracted and a position change of the feature portions F1, F2, and F3 may be analyzed to calculate spin of the ball.
(36) To this end, it is necessary to effectively extract only images of the ball portions 21, 22, and 23, that is, only a ball image from the source image, as illustrated in
(37)
(38) Images obtained by extracting only the ball portions 21, 22, and 23 from the respective source images, that is, ball images 211, 212, and 213 have sizes corresponding to ball portions on the respective source images, and thus, have different sizes. The ball images 211, 212, and 213 indicated in the respective source images are differently affected by illumination according to a distance from a camera, and thus, brightness also varies for each respective ball image.
(39) In order to accurately extract a ball feature portion, it is necessary to equalize the sizes of the ball images 211, 212, and 213 and normalize the brightness of the ball images 211, 212, and 213.
(40)
(41) That is, the ball images 211, 212, and 213 may be enlarged or reduced according to respective preset sizes or remaining ball images may be enlarged or reduced based on any one of the ball images 211, 212, and 213, that is, a normalization process may be performed on each ball image, thereby equalizing the sizes of ball images 211, 212, and 213.
(42) In addition, a normalization process may be performed on each ball image using an average value of all pixels constituting a ball portion to equalize overall brightnesses of ball images.
(43) As described above, after normalization is completed on each ball image, extraction of a ball feature portion using a ball feature extraction part is performed on each normalized ball image, as illustrated in
(44) That is, when a ball image of
(45) Here, the feature portion may be extracted using various image processing schemes such as a differential image scheme and so on.
(46) When feature portion images having the extracted feature portions FC1 and FC2 are prepared, a procedure of calculating spin information of a moving ball is performed using a spin calculation unit.
(47) Here, the spin of the ball may be calculated by calculating coordinate information about a spin axis in a three-dimensional space based on an i-axis, j-axis, and k-axis coordinate system and an angle for rotation with respect to the spin axis, that is, information about a spin amount, as illustrated in
(48) As illustrated in
{right arrow over (ω)}=θi+λj+ρk [Equation 1]
(49) Based on the spin vector (ω), the spin axis information and the spin amount information may be calculated according to Equations 2 and 3 below, respectively. Here, α is the spin amount information.
(50)
(51) Accordingly, the spin axis and spin amount information may be obtained by calculating θ as a yaw rotational component of spin of a moving ball, λ as a roll rotational component, and ρ as a pitch rotational component.
(52) The spin axis and spin amount information may be obtained from a feature portion extracted from the ball image shown in
(53) That is, as illustrated in
(54) To this end, first, it is necessary to correct position information of each pixel constituting the feature portion.
(55) That is, with regard to positions of a ball on an image acquired by a camera, a spin axis and spin amount are differently viewed according to a direction in which the camera is directed, and thus, it is necessary to accurately establish a reference and to calculate an absolute spin axis and spin amount based on the reference. To this end, position and direction information of the camera may be corrected as if respective balls on consecutively acquired images are viewed at the same position and direction with respect to the camera, thereby accurately calculating spin information of the ball.
(56)
(57) The sensing apparatus and method according to the present invention calculates spin information from position change of a feature portion of two consecutive images. In order to accurately calculate the spin information, a pair of consecutive images needs to be viewed at the same position and direction. That is, it is necessary to correct consecutive acquired images as if respective balls on the images are viewed at the same position and direction with respect to the camera while the camera moves together with the ball (a passing ball is always photographed by a fixed camera).
(58) As an example for establishing references of a position and direction of the camera, as illustrated in
(59) That is, as illustrated in
(60) The position correction may be achieved by correcting an angle at which a ball is viewed with respect to a camera, using coordinate information of a fixed camera and central coordinate information of the ball, which are already known. The position correction may be achieved by applying information about a portion to be corrected to position coordinates of pixels each extracted feature portion instead of actually correcting an image.
(61) Position information of each pixel constituting the first feature portion FC1 (refer to
(62) Comparing the converted two-dimensional position information with the position information of pixels constituting the second feature portion, when the converted two-dimensional position information corresponds to the position information of pixels constituting the second feature portion, the applied trial spin axis and spin amount information may be accurate target spin axis and spin amount information.
(63) Accordingly, as described above, the sensing apparatus and method according to the present invention may repeatedly apply trial spin axis and spin amount information to obtain position information and may find spin axis and spin amount information that correspond to the obtained position information at a predetermined level or more to calculate final spin information.
(64) This is represented according to the following equations.
(65) This may be represented by converting position information of pixels constituting each of the first and second feature portions into three-dimensional position information according to Equations 4 and 5 below.
PC1set_3D=C1*P1set_3D [Equation 4]
PC2set_3D=C2*P2set_3D [Equation 5]
(66) In Equations 4 and 5, P1set_3D is a matrix obtained by converting P1 set into three-dimensional coordinate information, where P1set is a matrix of coordinates (two-dimensional coordinates) of each pixel on the first feature portion FC1 (refer to
(67) P2set_3D is a matrix obtained by converting P2set into three-dimensional coordinate information, where P2set is a matrix of coordinates (two-dimensional coordinates) of each pixel on the second feature portion FC2 (refer to
(68) C1 and C2 are rotation matrices calculated as position correction information according to the principle illustrated in
(69) PC1set_3D is a matrix obtained by correcting positions of three-dimensional coordinates of the pixels of the first feature portion and PC2set_3D is a matrix obtained by correcting positions of three-dimensional coordinates of the pixels of the second feature portion.
(70) When a spin axis vector as target final spin information is ω and a spin amount is α, a rotation matrix R(ω, α) may be calculated using ω and α. A correlation of R(ω, α), and PC1set_3D and PC2set_3D that are obtained by converting positions of pixels of the respective feature portions into three-dimensional coordinate information satisfies Equation 6 below.
R(ω,α)*PC1set_3D=PC2set_3D [Equation 6]
(71) Equations 4 and 5 above may be substituted into Equation 6 above to obtain a correlation represented in Equation 7 below.
PC2set_3D=C2T*R(ω,α)*C1*P1set_3D [Equation 7]
(72) Here, C2T is transposition of matrix C2.
(73) According to Equation 7 above, position information of the pixels of the first feature portion may be converted into three-dimensional position information, and a resulting value may be obtained by applying position correction information and applying trial spin information to the three-dimensional position information. That is, trial spin information may be substituted into R(ω,α) as target final spin information in Equation 7 above.
(74) When Tset_3D is a result obtained by substituting a rotation matrix R(ω′,α′) calculated using a trial spin axis vector ω′ and a trial spin amount α′ as trial spin information into Equation 7 above, Equation 8 below is satisfied.
Tset_3D=C2T*R(ω′,α′)*C1*P1set_3D [Equation 8]
(75) When Tset is obtained by converting Tset_3D calculated according to Equation 8 above into a two-dimensional coordinate, if Tset and P2set correspond to each other, R(ω′,α′) and R(ω,α) have the same value, and thus, R(ω′,α′) as the trial spin information R(ω′,α′) may be determined as final spin information.
(76) That is, trial spin information applied to Tset closest to P2set, that is, having highest similarity among a plurality of Tset to which different trial spin information is applied may be determined as the final spin information.
(77) That is, if Tset and P2set are compared and the number of pixels having corresponding position coordinates is calculated, when the number of the corresponding pixels is equal to or greater than a predetermined level, or if similarity between pixels of Tset and pixels of P2set is calculated according to a predetermined function is equal to or greater than a predetermined level, trial spin information applied to the corresponding Tset is final spin information.
(78) Here, the similarity may be determined, for example, based on a ratio of the number of the corresponding pixels to all pixels.
(79) The aforementioned procedure of calculating spin of a moving ball will be described with reference to a flowchart of
(80) First, consecutive images of a moving ball are acquired (S10) and a background portion and so on are removed from each of the acquired images to extract respective source images (S12).
(81) In addition, a ball portion is found from each source image to extract ball images (S20) and a ball feature portion is extracted from each of the ball images to prepare respective feature portion images (S30).
(82) When a preceding image of two images among the consecutive feature portion images is referred to as a first feature portion image and a subsequent image of the two images is referred to as a second feature portion image, spin may be calculated using position information of pixels constituting the first feature portion on the first feature portion image and position information of pixels constituting the second feature portion on the second feature portion image.
(83) That is, the position information of pixels constituting the first feature portion is converted into three-dimensional position information (S42) and position correction information is applied to the three-dimensional position information (S44).
(84) In addition, a trial spin axis and a trial spin amount are extracted and applied to the three-dimensional position information (S52) and the three-dimensional position information is re-converted into two-dimensional position information (S54).
(85) The two-dimensional position information converted in operation S54 is compared with the position information of the pixels constituting the second feature portion (S62). When the number of pixels having corresponding position coordinates is calculated, if the number is equal to or greater than a preset number (S64), the applied trial spin axis and spin amount is determined as a final spin axis and spin amount (S72).
(86) When the calculated “number of corresponding pixels” does not reach a preset number, other trial spin axis and spin amount are applied to repeat operations S52, S54, S62, S64, and so on, a trial spin axis and a trial spin amount is determined as the final spin axis and spin amount when the “number of corresponding pixels” is highest.
(87) Here, the flowchart of
(88) In addition, instead of selecting final spin information through the number of pixels, similarity may be calculated according to a function and the final spin information may be selected according to the similarity.
(89) Consideration of all of a pitch rotational component, a yaw rotational component, and a roll rotational component for applying a trial spin axis, as illustrated in
(90) That is, limited conditions for extraction a trial spin axis and a trial spin amount may be set in consideration of the characteristics of ball movement and a frame rate of a camera, thereby rapidly calculating spin.
(91) When a golf ball is hit by a golf club, a pitch rotational component and a yaw rotational component are strongly represented among three rotational component of spin and a roll rotational component is not present or is almost negligible, as illustrated in
(92) That is, when the golf ball is hit, right and left side spin and forward and reverse spin are realized and a roll rotational component based on a proceeding direction of a ball is not present or is almost negligible.
(93) Accordingly, a trial spin axis may be selected in consideration of only a pitch rotational component and a yaw rotational component while disregarding a roll rotational component to some extent, and thus, the number of cases to be considered may be significantly reduced.
(94) When the golf ball is hit by the golf club, there is a limit in a rotation degree from a ball state of a captured image of one frame to a ball sate of a captured image of a subsequent frame.
(95) That is, since there is a limit in maximum spin for rotation according to user hit and there is a limit in a frame rate of a camera for capturing an image, the number of cases to be considered for a trial spin amount may be largely limited.
(96) Accordingly, as described above, limited ranges for a trial spin axis and a trial spin amount may be preset to reduce computation load, thereby rapidly calculating spin.
(97) Thus far, the procedure of calculating spin information about only two images among consecutively acquired images has been described. Alternatively, several tens of frames to several thousands of frames may be acquired per second according to a camera, and the aforementioned spin calculation procedure may be performed on all of a plurality of consecutive images or some of the consecutive images only.
(98) That is, the aforementioned spin calculation procedure may be performed on a plurality of pairs, each of which includes a preceding image and a subsequently acquired image of two consecutive images to obtain a plurality of spin information. The plural spin information may be slightly different and may be collected according to a preset function (for example, extraction of an average value or extraction of an average value of only several values with high similarity) to calculate final spin information, or spin information with highest similarity among the calculated plural spin information may be selected as the final spin information.
(99) Various embodiments have been described in the best mode for carrying out the invention.
(100) A method and apparatus for sensing a moving ball according to the present invention is applicable to an industrial field related to golf training including analyzing a ball in flight according to golf swing, a so-called screen golf industrial field providing a virtual reality-based simulation to allow a user to play a virtual golf game, and so on.