DETERMINATION OF SPIN RATE AND SPIN AXIS OF A BALL IN FLIGHT

Abstract

Embodiments are disclosed for determining the spin rate and the spin axis of a ball in flight. In some embodiments, a method comprises: capturing, with an image sensor, a time series of images of a ball in flight and corresponding image capture times; inputting, with at least one processor, the image frames and the image capture times into a machine learning model; and predicting, with the at least one processor, a spin axis and a spin rate of the ball based on the machine learning model.

Claims

1. A method comprising: capturing, with an image sensor, a time series of images of a ball in flight and corresponding image capture times; inputting, with at least one processor, the image frames and the image capture times into a machine learning model; and predicting, with the at least one processor, a spin axis and a spin rate of the ball based on the machine learning model.

2. The method of claim 1, wherein the machine learning model includes a first neural network trained to predict seam norms of the ball in 3D space based on a training set of ball images, a second neural network trained to predict the spin rate of the ball based on the predicted seam norms and time differences obtained from the training set of ball images, and a third neural network trained to predict the spin axis of the ball based on the predicted seam norms and ratios of the time differences.

3. The method of claim 2, wherein the training set of ball images include synthetic ball images of different types of balls, and wherein the ball images have non-uniform lighting, complex backgrounds, occlusions, or artifacts.

4. The method of claim 2, wherein the training set of ball images include augmented ball images of different types of balls, and wherein the ball images have at least one of random color, random contrast, random brightness, random sharpness, random shift or random noise level.

5. The method of claim 2, wherein randomly generated synthetic seam norms are used to train the first neural network.

6. The method of claim 2, wherein the first, second and third neural networks are trained using batch processing, stochastic gradient descent, Nesterov momentum without dampening, a weight decay or a cosine-shaped learning rate.

7. The method of claim 2, wherein the first, second and third neural networks are regression neural networks.

8. The method of claim 6, wherein the first neural network comprises M blocks, where each block includes a transition unit to downsample a feature size, followed by N residual units, where N and M are integer values greater than zero.

9. The method of claim 6, wherein the second and third neural networks each include M blocks, each block having a fully connected layer to increase a number of inputs, followed by N residual units, where each residual unit includes a first fully connected layer, followed by a first activation function, followed by a second fully connected layer, followed by a second activation function, where N and M are integer values greater than zero.

10. The method of claim 1, wherein the ball is a cricket ball.

11. A system comprising: at least one camera; at least one processor; memory storing instructions that when executed by the at least one processor, cause the at least one processor to perform operations comprising: capturing, with the at least one camera, a time series of images of a ball in-flight and corresponding image capture times; inputting the image frames and image capture times into a machine learning model; and predicting, with the machine learning model, a spin axis and a spin rate of the ball.

12. The system of claim 11, wherein the machine learning model includes a first neural network trained to predict seam norms of the ball in 3D space based on a training set of ball images, a second neural network trained to predict the spin rate of the ball based on the predicted seam norms and time differences obtained from the training set of ball images, and a third neural network trained to predict the spin axis of the ball based on the predicted seam norms and ratios of the time differences.

13. The system of claim 12, wherein the training set of ball images include synthetic ball images of different types of balls, and wherein the ball images have non-uniform lighting, complex backgrounds, occlusions, or artifacts.

14. The system of claim 12, wherein the training set of ball images include augmented ball images of different types of balls, and wherein the ball images have at least one of random color, random contrast, random brightness, random sharpness, random shift or random noise level.

15. The system of claim 12, wherein randomly generated synthetic seam norms are used to train the first neural network.

16. The system of claim 12, wherein the first, second and third neural networks are trained using batch processing, stochastic gradient descent, Nesterov momentum without dampening, a weight decay or a cosine-shaped learning rate.

17. The system of claim 12, wherein the first, second and third neural networks are regression neural networks.

18. The system of claim 17, wherein the first neural network comprises M blocks, where each block includes a transition unit to down sample a feature size, followed by N residual units, where M and N are integer values greater than zero.

19. The system of claim 17, wherein the second and third neural networks each include M blocks, each block having one fully connected layer to increase a number of inputs, followed by N residual units, where each residual unit includes a first fully connected layer, followed by a first activation function, followed by a second fully connected layer, followed by a second activation function, where M and N are integer values greater than zero.

20. The system of claim 11, wherein the ball is a cricket ball.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG. 1A is a perspective view of a cricket ball.

[0017] FIG. 1B illustrates a seam norm for the cricket ball of FIG. 1A.

[0018] FIG. 2 is a three-dimensional (3D) diagram illustrating how the seam norm moves in 3D space over time for a ball in flight or in motion.

[0019] FIG. 3 is a block diagram of a deep learning network for predicting the spin rate and spin axis of a ball in flight or in motion, according to one or more embodiments.

[0020] FIG. 4 illustrates a seam norm regression neural network, according to one or more embodiments.

[0021] FIG. 5 illustrates a spin rate regression neural network, according to one or more embodiments.

[0022] FIG. 6 illustrates a spin axis regression neural network, according to one or more embodiments.

[0023] FIGS. 7A and 7B illustrate synthetic seam norm generation, according to one or more embodiments.

[0024] FIG. 8 is a flow diagram of a process of determining the spin rate and the spin axis of a ball in flight, according to one or more embodiments.

[0025] FIG. 9 is a block diagram of a ball launch monitoring system, which includes a measuring device that utilizes the processes described in reference to FIGS. 3-8, according to one or more embodiments.

[0026] FIG. 10 is a block diagram of the measuring device of FIG. 9, according to one or more embodiments.

DETAILED DESCRIPTION

[0027] Bowling in cricket is the action of propelling the ball toward the wicket defended by a batter. Understanding proper bowling mechanics is crucial for bowlers to master various bowling tactics, such as delivering fast or spin bowling. Distinguishing between different bowling techniques relies on factors like the ball's trajectory and speed, determined by the angular velocity and initial velocity of the ball at the point of release.

[0028] Additionally, factors like drift and dip also play a significant role in the ball's direction during its flight. When a ball drifts, it veers sideways during its flight, creating a horizontal deviation. On the other hand, dip causes the ball to descend closer to the ground than expected, resulting in a non-linear alteration in its vertical movement. The spinning motion of the ball during its flight counteracts the airflow, leading the airflow to move upward, generating a downward force that causes the ball to dip. This phenomenon, known as the Magnus effect, is closely associated with the spin. While a radar gun is capable of measuring the speed of a cricket ball, it falls short in providing insights into the ball's spin properties. Therefore, the ability to measure the spin properties of a ball becomes insightful. Such measurements can equip coaches with a quantitative assessment of key spin bowling performance variables, enabling them to offer objective feedback to players on their performance.

[0029] In baseball, understanding the spin axis and the spin rate can provide information to the pitcher in letting the pitcher know whether their pitch is a fastball, knuckleball, curveball, etc. This information helps the pitcher in improving and developing the perfect pitch.

[0030] The present disclosure provides the use of seam norm from ball images with a trained deep learning neural network to estimate the spin properties including spin axis and the spin rate. This information can be used to measure the spin properties of the ball, for example a cricket ball. The seam norm vector model described herein is advantageously more efficient and effective as it does not segment out seam lines and thus feature matching is not required. It is acknowledged that segmenting the whole seam line is challenging due to non-uniform illumination artifact, logo, occlusion, and different types of balls. Further, the segmentation approach is time consuming to do feature matching. The methods disclosed herein advantageously measure seam norm directly from ball image with a neural network. Further, spin rate and axis regression modes are advantageously robust to outliers. For balls that do not have a seam that circumvents the entire ball and thus defines a seam plane (e.g., baseballs, golf balls, soccer balls, footballs, basketballs), a sticker can be wrapped around the ball that will act as a proxy for a stitched seam, and the disclosed embodiments can be applied to any ball regardless of whether it has a stitched seam.

Physics of Spin Rate and Spin Axis

[0031] FIG. 1A is a perspective view of cricket ball 100. Cricket ball 100 has a stable and constant seam line 101. Two feature points used for training neural networks are the seam norm vectors 105a(n) and 105b(n). These vectors are normal to seam plane 104 as shown in FIG. 1B (hereinafter also referred to as seam norms). As described below, the seam norms can be used to measure the spin rate and spin axis 103 of cricket ball 100 in flight or in motion. Although the discussion that follows is with respect to a cricket ball, the processes described below are applicable to any ball that has a seam (e.g., a golf ball with a marker, a softball or a baseball).

[0032] FIG. 2 is a 3D diagram illustrating how seam norms move in 3D space over time. As shown in FIG. 2, the seam norms of spinning ball 100 form two circles along and normal to spin axis 103 over time. If there are N sequential camera images captured of ball 100, the i-th ball is captured at time t.sub.i and its seam norms in 3D space are n.sub.i=(x.sub.i, y.sub.i, z.sub.i) and n.sub.i=(x.sub., y.sub.i, z.sub.i), where x.sub.i.sup.2+y.sub.i.sup.2+z.sub.i.sup.2=1 and z.sub.i>0. Spin axis 103 (V=(a, b, c) of ball 100 is obtained by fitting a plane to the 3D n.sub.i points according to Equation [1]:

[00001]$\begin{array}{cc}a*x_i+b*y+c*z_i+d=0,i=1,2,.Math.N,& \left[1\right]\end{array}$

where d is the distance of the fitted plane from the origin.

[0033] The direction of spin axis 103 can be determined with the right hand rule from point n.sub.1 to point n.sub.2 in the circle as depicted in FIG. 2. Once spin axis 103 is determined, the spin rate can be obtained by determining the ratio of the angle 201 of n.sub.i to n.sub.i+1 relative to spin axis 103 and the corresponding time difference of t.sub.i+1 to t.sub.i as shown in Equation [2]:

[00002]$\begin{array}{cc}\mathrm{spinRate}={.Math.}_{i=1}^{N-1}{}_i/{.Math.}_{i=1}^{N-1}{T}_i={.Math.}_{i=1}^{N-1}{}_i/\left(t_N-t_1\right),& \left[2\right]\end{array}$

where .sub.i is the angle 201 of point n.sub.i to point n.sub.i+1 relative to spin axis 103, and T.sub.i=t.sub.i+1t.sub.i is the time difference of t.sub.i+1 to t.sub.i.

Example System

[0034] FIG. 3 is a block diagram of deep learning network 300 for predicting spin rate and spin axis of a ball in flight or in motion, according to one or more embodiments. In this example embodiment, input 301 to network 300 (e.g., a deep learning network) includes 8 ball images and corresponding image capture times, and predicts the spin rate and spin axis of the ball, where each of the ball images contains spins about a different axis. In other embodiments, 3 or more ball images can be used. At least 3 points (seam norm) can determine one circle in 3D, and the spin axis is vertical to the circle. Once the spin axis is determined, any 2 points on the circle can be used to determine the spin rate as in Equation [2].

[0035] The example network 300 shown in FIG. 3 includes three deep learning neural networks (nodes): seam norm regression neural network 302, spin rate regression neural network 304 and spin axis regression neural network 306. In some embodiments, seam norm regression neural network 302 regresses 8 seam norm vectors 303 from input 301 with batch processing. Spin rate and axis regression neural networks 304, 306, regress spin rate 307 and spin axis 308, respectively, from the input of seam norm vectors 303 and corresponding image capture times 305, respectively.

[0036] Note that although deep learning networks are shown in FIG. 3, other machine learning models are also possible.

[0037] In some embodiments, a specific method for baseball can be used without the use of stickers or other seam proxies. A cricket ball is 2 symmetrical and a baseball is 4 symmetrical. Thus, two seam planes can be used to make sure every baseball image captured by the camera is unique. In some embodiments, the same seam norm regression, spin axis and rate regression neural networks for the cricket ball can be used for the baseball, with the difference as follows. For the seam norm regression neural network, the input is a baseball image, and the output is two seam norm vectors. One seam norm vector is (x, y, z) and its vertical is (x, y, z). For the spin axis and rate regression neural network, the input is a combination of all the seam norm vectors and the time difference. The output is the spin axis and spin rate, respectively.

[0038] FIG. 4. illustrates a seam norm regression neural network 400, according to one or more embodiments. Instead of seam line segmentation, seam norm regression neural network 400 is directly trained. In this example embodiment, input 401 is a 6464 sized ball image and output 402 is its seam norm vector. In this example configuration, seam norm regression neural network 400 includes 5 blocks, each block with one transition unit to downsample the feature size, followed by 2 residual units. Each block includes three sub-blocks oplus two sub-blocks, where the oplus may denote an exclusive or, dilation or direct sum. Each sub-block includes a convolution filter, a batch normlization function and a rectified linear unit. The convolution filter may be of size 11 or 33. Output 402 includes three nodes, representing the three vector component magnitudes (x, y, z) in 3D space. In this example, seam norm regression neural network 400 is trained for 300 epochs with a batch size of 128 using stochastic gradient descent with Nesterov momentum of 0.9 without dampening, and a weight decay of 10.sup.4. The learning rate is cosine-shaped reducing from 0.1 to 0. The learning rate is a scalar value to tune a neural network to achieve better performance. Learning rate determines the step size at each training iteration while moving toward an optimum of a loss function. The direction towards an optimum of a loss function can be found by calculating a gradient of the loss function. The learning rate parameter specifies how large the step size is considered in the direction towards the optimum.

[0039] In some embodiments, the blocks of neural network 400 may be arranged in series, where the blocks are daisy chained, and where the last block is connected to a fully connected layer. In some embodiments, the blocks of neural network 400 may be arranged in parallel manner, where the blocks are daisy chained, and where the last block is connected to a fully connected layer. In some embodiments, real images of the ball are used with ground truth data to train neural network 400. In other embodiments, synthetic data is used together with, or in lieu of, the real image data for training neural network 400. For example, synthetic ball images can be generated with non-uniform lighting, complex backgrounds occlusions, artifacts and different types of balls, resulting in additional synthetic ball images with their seam norms uniformly allocated on the unit sphere.

[0040] In some embodiments, during training, the real and/or synthetic ball images can be augmented by modifying the color, contrast, brightness, sharpness, shift, and noise level of the ball images. The augmentation levels of color, contrast, brightness, and sharpness can be random ranging from about 0.8 to about 1.2, for example. In this example embodiment, the maximum shift is 4 pixels, which is the maximum error caused by ball finding. For noise, Gaussian blur or other suitable image processing technique can be applied, with a sigma value that varies from about 0.1 to about 9.0, for example.

[0041] In some embodiments, an augmentation level of 1.0 corresponds to the original image, i.e., not applying any data augmentation. Data augmentation is used to minimize the gap between training and testing data. For example, if the training images have a brightness level of 10, and the testing images have a brightness level of 8, there is 2=108 difference in brightness levels. This data shift causes lower testing accuracy of the pretrained model. By applying data augmentation with, for e.g., a random ratio from 0.8 to 1.2, the pretrained model can have good accuracy with brightness from 8(0.8*10) to 12(1.2*10). In some embodiments, the range can be fine-tuned to have a better accuracy on the set of test images. Thus, the brightness range can be wider but it cannot be lower than 0, which corresponds to a completely black image. Note that the brightness range cannot be too high either, since the maximal intensity is 255. Thus, in this example, the top ratio for brightness is given by 255/(current training brightness).

[0042] FIG. 5 illustrates a spin rate regression neural network 500, according to one or more embodiments, and FIG. 6 illustrates a spin axis regression neural network 600, according to one or more embodiments. In this example, both spin rate and axis regression neural networks 500, 600 share the same architecture. It is to be understood that a different architecture of regression neural networks may also be used. Inputs 501, 601 are 1D arrays that include 16 seam norms and corresponding time differences. Output 502 for neural network 500 includes one node to regress spin rate, and output 602 for neural network 600 includes three nodes to regress spin axis in 3D space, respectively.

[0043] Both neural networks 500 and 600 each have 5 blocks, where each block includes one fully connected layer to increase the number of inputs and 4 residual units. Each residual unit includes a first fully connected layer coupled to a first activation function (e.g., a rectified linear unit (ReLU) activation function), followed by a second fully connected layer, followed by a second activation function (e.g., a ReLU activation function). Similar to seam norm neural network 400, in this example embodiment, neural networks 500, 600 are trained for 1000 epochs with batch processing using a batch size of 2560, stochastic gradient descent with Nesterov momentum of 0.9 without dampening, and a weight decay of 10.sup.4. The learning rate is cosine-shaped reducing from 0.1 to 0.

[0044] FIGS. 7A and 7B illustrate synthetic seam norm vector generation, according to one or more embodiments. Referring to FIG. 7A, inputs 501, 601 for spin rate regression neural network 500 and spin axis regression neural network 600, respectively, are shown. In this example embodiment, inputs 501, 601 are 1D arrays that each include 16 seam norms, including n.sub.i=(x.sub.i, y.sub.i, z.sub.i) and n.sub.i=(x.sub.i, y.sub.i, z.sub.i), where i=1,2, . . . 8. Additionally, the 1D array for input 501 includes time differences T.sub.j and the 1D array for input 601 includes time difference ratios T.sub.j/T.sub.j+1, where j=1,2, 3,4 . . . 7.

[0045] The spin rate of a cricket ball is typically in a range of about 400 to about 3000 revolutions per minute (abbreviated as rpm), and the spin axis can be any unit vector in 3D space. The spin axis can be uniformly allocated along the whole unit sphere, where each spin axis has a random spin rate in a range of about 300 to about 3600 rpm. As shown in FIG. 7B, in some embodiments, one spin axis is picked from the unit sphere and another point n.sub.1 is chosen as the initial seam norm. In some embodiments, a random number from 300 to 3600 is chosen as the spin rate, and n.sub.1 is rotated with different time differences to generate 7 seam norms. In some embodiments where ball images are captured by two cameras (a spin camera and a speed camera), there are two different timings. For the speed camera, the time difference is constant at, e.g., 1.0/220 s. For a spin camera, the time difference is, e.g., 8, 9, 10, 11, and 12 ms.

[0046] To make both the spin rate and spin axis regressions more robust, in some embodiments, 3 out of 8 seam norms are allowed as outliers with a 2 degree difference from groundtruth for each seam norm. During training, 3 seam norms are randomly selected and initialized with random values, and each seam norm is rotated with a random degree from 0 to 2 degrees in a random direction, for example.

Example Process

[0047] FIG. 8 is a flow diagram of a process of determining the spin rate and the spin axis of a ball in flight, according to one or more embodiments. Process 800 can be implemented in, for example, a ball launch monitoring system 900, as described in reference to FIGS. 9 and 10.

[0048] Process 800 includes: capturing, with an image sensor, a time series of images of a ball in-flight and corresponding image capture times (801), inputting the image frames and the image capture times into a machine learning model (802), and predicting a spin axis and a spin rate of the ball based on the machine learning model (803). Each of these steps were previously described above in reference to FIGS. 3-7.

Example Ball Launch Monitoring System

[0049] FIG. 9 is a block diagram of a ball launch monitoring system 900, according to one or more embodiments. Measurement or measuring device 902 is positioned on the floor or ground behind user 905 (e.g., behind cricket ball 904), who uses bat 903 to strike cricket ball 904. Processing of data can be further done through a cloud service. Viewing device 901 (e.g., computer or mobile device) is provided for user 905 or their coach to visualize the measurement results on a display screen (e.g., display screen of a computer or mobile device). Alternatively, or additionally, measurement device 902 may be positioned to a side of user 905.

[0050] FIG. 10 is a block diagram of measuring device 902 of FIG. 9, according to one or more embodiments. In some embodiments, measurement device 902 may include two sets cameras. The first set of cameras includes two high-speed, low resolution cameras 1004A, 1004B that form a first stereo system for measuring the movement of the bat and cricket ball during a swing of bat 903 by user 905. The second set of cameras includes two high-resolution cameras 1006A, 1006B that form a second stereo system for measurement of the ball trajectory. In some embodiments, the second set of cameras may have a lower speed than the first set of cameras. In some embodiments, lighting system 1005 ensures properly lit images for 3D scanning of bat 903, radar unit 1003 provides timing and sequencing of image captured by cameras 1004A, 1004B, 1006A, 1006B, computing and controlling system 1002 performs real-time processing of the captured images and radar signals, and wireless communication system 1001 sends data to viewing device 901 shown in FIG. 9.

[0051] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as descriptions of features specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub combination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub combination or variation of a sub combination.

[0052] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.