Systems and methods for illustrating the flight of a projectile

Abstract

A system for illustrating the flight of a sports ball includes a radar, an imager, and a controller. The imager is configured to image a moving sports ball. The controller is configured to (i) receive, from the imager, an image including the moving sports ball, (ii) receive, from the radar, radar data associated with the moving sports ball, (iii) determine, from the radar data, a portion of a trajectory of the moving sports ball, (iv) alter the image to illustrate the portion of the trajectory relative to the moving sports ball, and (v) output the altered image.

Claims

1. A method for tracking a moving object comprising the steps of: calibrating data from an imaging device to data from a radar device, where a field of view of the imaging device at least partially overlaps with a field of view of the radar device in a common field of view; and tracking the moving object as it moves through the common field of view simultaneously using both the imaging device and the radar device where the moving object is one of a sports ball and a sports ball striking implement.

2. The method of claim 1, wherein tracking includes determining one of at least a portion of a trajectory of a sports ball and at least a portion of a path of movement of the striking implement.

3. The method of claim 1, further comprising calibrating data from the radar device to a world based coordinate system relating to locations within the common field of view.

4. The method of claim 3, further comprising defining an imaging device coordinate system relating data from the imaging device to physical locations within the field of view of the imaging device and defining a radar coordinate system relating data from the radar device to physical locations within the field of view of the radar device, and correlating the imaging device coordinate system to the radar device coordinate system.

5. The method of claim 3, further comprising correlating data from the imaging device to the world based coordinate system.

6. The method of claim 1, further comprising altering images from the imaging device to illustrate one of a position, trajectory and velocity of the one of the sports ball and the sports ball striking implement and an orientation of the sports ball striking implement.

7. The method of claim 6, wherein the images from the imaging device are altered to include at correct positions in the images a graphic representation of one of the position, trajectory and velocity of the one of the sports ball and the sports ball striking implement.

8. The method of claim 1, wherein positions of the one of the sports ball and the sports ball striking implement as it is tracked are determined based on data from the radar device and the imaging device.

9. The method of claim 8, further comprising determining a launch location of the sports ball based on data from the imaging device before the sports ball is launched.

10. The method of claim 8, further comprising determining a launch location of the sports ball based on data from the radar device after the sports ball has been launched.

11. The method of claim 8, further comprising identifying a target location in the image at which the sports ball is to be launched.

12. The method of claim 11, wherein the target location is identified by pattern recognition.

13. The method of claim 1, further comprising identifying an impact location at which a striking implement impacts the sports ball at launch.

14. The method of claim 13, wherein the identification of the impact location is based on data from both the radar device and the imaging device.

15. The method of claim 13, further comprising determining an impact time at which the sports ball striking implement impacts the sports ball, further comprising determining one of a position on the sports ball striking implement at which the sports ball impacts the sports ball striking implement and an angle at which the sports ball striking implement impacts the sports ball by interpolating between positions of the sports ball striking implement in images before and after the impact time.

16. The method of claim 15, further comprising determining an orientation of the sports ball striking implement at the impact time by interpolating between positions of the sports ball striking implement in images before and after the impact time.

17. The method of claim 13, wherein the striking implement is a golf club and the sports ball is a golf ball.

18. The method of claim 1, wherein the sports ball is tracked using angular measurements based on data from the imaging device, and wherein distance and velocity measurements of the sports ball are based on data from the radar device.

19. The method of claim 1, further comprising predicting a landing location of the sports ball at which the sports ball will contact the ground based.

20. The method of claim 19, further comprising altering an image from the imaging device to illustrate the landing location.

21. The method of claim 20, further comprising reducing a size of the predicted landing location as the sports ball approaches the ground.

22. A device for tracking a moving object comprising: an imaging device having an imaging device field of view; and a radar device having a radar field of view at least partially overlapping with the imaging field of view in a common field of view; and a control device coordinating data from the imaging device and data from the radar device in space and time to track the moving object as it moves through the common field of view, wherein the moving object is one of a sports ball and a sports ball striking implement.

23. The device of claim 22, wherein the control device correlates data from the radar device to a world based coordinate system relating to locations within the common field of view.

24. The device of claim 22, wherein the control device defines an imaging device coordinate system relating data from the imaging device to physical locations within the field of view of the imaging device and defines a radar coordinate system relating data from the radar device to physical locations within the field of view of the radar device, and correlating the imaging device coordinate system to the radar device coordinate system.

25. The device of claim 23, wherein the control device correlates data from the imaging device to the world based coordinate system.

26. The device of claim 22, further comprising a data storage device storing image data and wherein the control device alters one of data from the imaging device and image data from the data storage device to match a predetermined reference perspective.

27. The device of claim 26, wherein the control device scales one of data from the imaging device and image data from the data storage device so that a pre-determined element of known dimensions represented in the one of the data from the imaging device and the image data from the data storage device has a predetermined extent.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) In the following, the invention will be described with reference to the drawing wherein:

(2) FIG. 1 illustrates an assembly according to the invention imaging a golfer,

(3) FIG. 2 illustrates an image or video provided of the golfer with radar data provided,

(4) FIG. 3 illustrates a close-up image or video of a club at the time impact, and

(5) FIG. 4 illustrates a rotatable assembly.

DETAILED DESCRIPTION OF EMBODIMENTS

(6) In FIG. 1, an assembly 10 is illustrated having a radar 20 and a camera 30 both mounted within a housing 12. The radar 28 and camera 30 are calibrated to each other so that position data obtained from a moving object and using the radar 20 may be converted into a position in an image or video provided by the camera 30. This calibration may be provided using one or more elements being visible, so that the camera 30 may image it, while being adapted to reflect radar radiation so that its position may be determined by the radar 20.

(7) The radar 20 and camera 30 are provided so as to have at least partly overlapping fields of view, so that movements of one or more objects in these fields of views may be both imaged and determined/quantified by the radar.

(8) In FIG. 1, the golfer 50 swings a club 52, so as to swing it along a club trajectory 54, and hits a ball 56 which subsequently flies along a ball trajectory 58.

(9) The movements of the club 52 and ball 56 are determined by the radar 20 while the camera 30 images the movements in one or more images, such as a video sequence with a predetermined number of frames per second.

(10) From the data from the radar 20, it is possible to determine positions, velocities, angles and the like from the different moving elements (club 52 and ball 56) at different points in time, and this data may be correlated with images provided by the camera 30 at the same points in time. In this manner, both image data and movement/position/velocity/acceleration data may be provided.

(11) In addition, from the image(s), see FIG. 2, additional data may be provided, such as distances or relative positions of elements in an image. Interesting parameters for a golfer 50 is a distance between the golfer 50 and the ball 56 before striking the ball 56, such as from the golfer's shoes to the ball 56. Another interesting relative position is the position of the golfers head in relation to the ball 56, along the horizontal plane.

(12) In order to determine such data, it is desired to know the distance from the camera 30 to the golfer 50 or ball 56. This distance may be determined in a number of manners, such as from the apparent size of the ball 56. As all golf balls have a standard size, other dimensions of the image may be derived.

(13) Another manner is to determine the distance from ball 56 to radar 20 from the tee position 60 as determined subsequent to launching the ball 56. This is described in WO 2006/002640.

(14) From the radar data, the trajectory of the club 52 head may be determined and may be provided in an image of the Roller 50 taken during swinging of the club 52. In addition, the trajectory of the ball 56, such as a 3D ball launch sector and ball spire may be provided in the image data, and the direction/angle 64 of movement of the club 52 head, such as 3D club impact vector 66, at impact may be compared to the direction of the ball trajectory 58 after impact (see FIG. 3). From these directions, a face angle of the club 52, i.e. the angle of the club surface actually impacting the ball, may be determined, which is of interest to the golfer 50.

(15) Actually, the trajectories of the club and ball may alternatively or additionally be determined from the video, if desired.

(16) Another interesting parameter is the so-called lie angle which relates to the relation between the ground and the axis of the club head (whether the club has the heel or the toe closest to the wound). This may be determined front the video or image.

(17) Actually, it may be desired to provide a grid or the like at the ground or from the ground up at the golfer in order for the golfer to know and compare (see below) the relative positions of the feet, tee 60, head and the like.

(18) Also, the camera 30 may be programmed or set to provide a close-up image or video of the ball 56 and club 52 at launch (see FIG. 3). A close-up video may be taken with an increased number of frames per second in order to gain a better insight of the parameters at launch. In addition the video or images may be provided with the determined trajectories 54/58 and/or directions etc. in order to illustrate the relative movements at or around launch.

(19) From such images, the actual point of impact of the club head may be determined which is also of into to the golfer.

(20) In FIG. 3, a club 52 about to strike a ball 56 is illustrated along with data relating to the club trajectory 54 and the ball trajectory 58. From this data, the impact location of the ball on the club face can be determined. This is done by determining the location of the ball in the image and then subsequently comparing the location of the club head in an image taken at point of impact. If an image is not available at exactly the time of impact, the club trajectory data determined by the radar can be used for determining where the club head will be moving from the location in the image just prior to impact or prior to an image just after impact.

(21) It is noted, as described above, that a velocity, for example of the ball, determined using the images or the radar, may be subsequently controlling the frame rate when replaying the video. Thus, the frame rate may be reduced when the velocity increases in order to better illustrate the video in spite of the increased velocity.

(22) Reverting to FIG. 2, and as is mentioned in WO 2006/002639, a golfer may, in the image, identify a target 62, which is aimed at. Then, as the tee or point of launch 60 and the target 62 are identified, a coordinate system may be set up with origo in the tee 60 and an axis along the direction from the tee 60 to the target 62. This is the main coordinate system of the golfer 50.

(23) However, the positioning of the assembly 10 in relation to the golfers coordinate system will depend on a number of parameters, whereby it is far from certain that image data obtained at different points in time (whereby the assembly may be positioned at different relative positions, distances, angles) may easily be compared, in order to provide comparable data, the coordinate system is determined in the originally obtained image/video/radar data, where after the image(s) is/are rotated, using simple image manipulation such as perspective transformation, so as to have the axis from tee 60 to target 62 be along a predetermined direction in the altered image, in addition, a scaling (enlargement/reduction) or translation of the image may be desired in order to have the distance from the tee 60 to the assembly 10 be as desired, or other image manipulation such as to obtain a desired angular section or perspective.

(24) This rotation may also be performed of the radar data, so that the same trajectories/positions or the like may be illustrated correctly also in the rotated/scaled image. This is a simple operation, as the rotation/scaling is known.

(25) After this altering of the image(s), comparison is possible. Then, the same golfer 50 may, independently of the relative position of the assembly, store a number of swings for later comparison, or he/she may compare a swing to that of another golfer, such as a trainer, a pro or the like. Also, the trajectories, planes, angles and the like may be compared rather easily.

(26) Naturally, if more than one camera 30 is provided, the rotated image may be generated on the basis of multiple images (preferably provided or taken at least substantially the same point in time), in any of a number of manners.

(27) In one situation, the generation of the image from multiple images may be performed by identifying, for each part or pixel in the final image, corresponding parts of the initial images and weighting the information in the two initial images on the basis of the distance from the pertaining imaging device to the object. In this manner, the imaging device with the smallest distance will be given the largest weight for that part of the final image.

(28) If a second camera 32 is provided, this should be calibrated to the first camera 30 and/or the radar 20 so that a correlation of elements in the image of the second camera 32 with elements imaged by the first camera 30 and/or as detected by the radar 20 is possible.

(29) This calibration may also be a correspondence between generation of images with the image generation of the first camera 30 and/or the obtaining of data by the radar 20. A number of manners of obtaining this are known such as the forwarding of instructions to the second camera 32 as to when to generate an image, the providing with the images from the second camera 32 of timing information describing a point in time of deriving the image, or the like, so that images or image and radar information corresponding in time (such as derived within a predetermined maximum time period) are selected and used.

(30) In this manner, it may be possible to actually provide, in the image/video of the second camera 32, data relating to the radar data in the same manner as described above for the image/video of the first camera 30.

(31) Naturally, the information derived may be presented in a number of manners, such as an image having therein trajectory data (ball and/or club), target/tee positions and wherein images of the golfer at different points in time during the swing may be overlapped.

(32) FIG. 4 illustrates another manner of using the assembly 10, which is in connection with one or more radar reflecting elements 70 positioned in the surroundings of the assembly 10 and within (at least at times) the field of view of the radar 20.

(33) A radar reflecting element 70 may be an element reflecting the radio waves either by providing a moving element (such as a fan or flag) or by receiving the wave and outputting a corresponding wave (so-called transponder).

(34) When the assembly 10 is rotatable and rotating, the position of a reflecting element 70 will move within the field of view of the radar 20, which will then determine this movement or position. Knowing the relative positions of the elements 70 will, even during movement/rotation of the assembly 10, make the assembly 10 able to determine its position and/or rotation/rotational position, in relation to the surroundings from the positions determined by the radar 20.

(35) In this situation, the images provided by the camera 30 of the assembly may be adapted to this movement/rotation/position/rotational position.

(36) In one situation, an element in the surroundings has a known position in relation to the assembly 10 or the element(s) 70 even if not imaged by the camera 30 and/or determined/detected by the radar 20. This position may be known from measurements of another assembly 10 or those illustrated in the above-mentioned WO-references.

(37) Nevertheless, it may be known or estimated that this element will enter the field of view of the camera 30, whereby the image of the camera 30 may be altered to reflect this. One manner of reflecting this may be to illustrate where or when in the image the element will enter.

(38) However, it is clear that when the assembly 10 moves, the point of entry of the element into the image of the camera 30 will change, but due to the constant monitoring of the movement/rotation/etc. of the assembly 10 by the radar 20 and potentially a constant or intermittent updating of the position/uncertainty, this change may be determined and the image thereafter be made to reflect the new position upon movement of the assembly.

(39) In a particularly interesting situation, the camera 30 may image an expected or calculated/estimated touch down spot or landing spot of a struck golf ball or baseball. This landing spot may be determined in an desired manner such as from a radar as that described in WO 2005/116678.

(40) From e.g. a determined trajectory of the flying golf ball or baseball, the actual position and landing spot may be constantly determined in order to provide an estimate of the landing spot with greater and greater accuracy.

(41) In the image of the assembly 10 imaging the estimated landing spot, the estimated landing zone may be illustrated by a circle having a diameter reflecting the uncertainty of the landing spot determination. As the ball approaches the landing spot, the certainty will increase and the circle illustrated in the image of the camera 30 may be made correspondingly smaller. Also, the landing spot or center of the circle may move.

(42) Naturally, other types of data may be illustrated, such as an estimated trajectory of the ball, an expected rolling of the ball or the like.

(43) This illustrating, due to the operation of the radar 20 and the elements 70, is no longer dependent on the camera 30 and the assembly 10 being fixed in relation to the surroundings. If the assembly 10 rotates, for example, the rotation will be detected by the radar 20, as the position(s) of the elements) 70 will change. From this change, the rotation may be quantified, and the landing spot or other position or the surroundings may be correspondingly altered.

(44) Then, the camera 30 may be rotated to follow the path of the ball 56 in flight while illustrating, when the estimated landing spot enters the field of view of the camera 30, the landing spot estimate or other desired data.

(45) Naturally, a zooming in or out of the camera 30 will alter the relative positions of the element(s) 70 in the image. As the zooming is known, however, this is easily determined so that zooming in may be performed with the corresponding amendment of the added data, such as landing spot position and uncertainty.

(46) Even though the figures and pertaining description have focused on the use of the present assembly for use in gulf the same functionality may be obtained in any other situation in which imaging and motion detection/determination is desired, such as in other sports (baseball, tennis, table tennis, cricket, soccer, football, handball, volley ball, basketball, or, the like), as well as in other situations when a movement is desired imaged and quantified/illustrated at the same time.