Method for synchronizing data collected from a camera device and a motion sensing device and data processing device generating the synchronized data

Abstract

Provided is a method for synchronizing data between different types of devices for collecting and processing data with respect to a moving object, comprising: sampling first type data with respect to a motion of the object from one device; sampling second type data with respect to a motion of the object from another device having a larger sampling interval than the device; and generating data corresponding to a sampling time of the first type data by combining second type data respectively sampled before and after the sampling time of the first type data in a sampling interval of the second type data, to generate data synchronized with the sampling time of the first type data with respect to the second type data.

Claims

1. A method for synchronizing data between a camera device for collecting image data with respect to a moving object and a motion sensing device for collecting motion sensing data with respect to the moving object, comprising: sampling the image data with respect to the moving object from the camera device; sampling the motion sensing data with respect to the moving object from the motion sensing device; and generating synchronized image data corresponding to a sampling time of the motion sensing data by composing the sampled image data respectively sampled before and after the sampling time of the motion sensing data in a sampling interval of the image data, wherein the synchronized image data is synchronized with the sampling time of the motion sensing data, the synchronized image data generated by: calculating weights for the sampled image data, the weights increasing as the sampled image data approaches the sampling time of the motion sensing data; respectively applying the calculated weights to the sampled image data respectively sampled before and after the sampling time of the motion sensing data; and composing the weights applied image data to generate the synchronized image data corresponding to the sampling time of the motion sensing data.

2. The method according to claim 1, wherein the generating of the synchronized image data comprises generating image data synchronized with the sampling time of the motion sensing data by applying weights, calculated on the basis of sampling times of two image data close to the sampling time of the motion sensing data and a sampling interval of the motion sensing data, to the image data and composing the image data.

3. The method according to claim 1, wherein the generating of the synchronized image data comprises: calculating a ratio of an interval between the sampling time of the motion sensing data and a sampling time of first image data corresponding to image data before the motion sensing data sampling time to the sampling interval of the image data as a second weight; calculating a ratio of an interval between the sampling time of the motion sensing data and a sampling time of second image data corresponding to image data after the motion sensing data sampling time to the sampling interval of the image data as a first weight; and applying the first weight to the first image data, applying the second weight to the second image data and composing the first image data and the second image data to which the first and second weights have been respectively applied.

4. The method according to claim 3, wherein the composing of the image data comprising: applying the first weight to values of all pixels of the first image data and applying the second weight to values of all pixels of the second image data; and summing pixel values to which the first weight has been applied and pixel values respectively corresponding to the pixel values and weighted by the second weight, to generate composite image data having the summed pixel value.

5. The method according to claim 1, further comprising generating images based on the image data synchronized with the motion sensing data sampling time and including an image generated based on the motion sensing data and displayed thereon.

6. A data processing device for generating synchronized data between a camera device for collecting image data with respect to a moving object and a motion sensing device for collecting motion sensing data with respect to the moving object, the data processing device generating the synchronized data according to the method of claim 1.

7. The data processing device according to claim 6, wherein the data processing device is further configured to calculate a ratio of an interval between the sampling time of the motion sensing data and a sampling time of first image data corresponding to image data before the motion sensing data sampling time to the sampling interval of the image data as a second weight and to calculate a ratio of an interval between the sampling time of the motion sensing data and a sampling time of second image data corresponding to image data after the motion sensing data sampling time to the sampling interval of the image data as a first weight.

8. The data processing device according to claim 7, wherein the data processing device is further configured to apply the first weight to values of all pixels of the first image data, to apply the second weight to values of all pixels of the second image data and to sum pixel values to which the first weight has been applied and pixel values respectively corresponding to the pixel values and weighted by the second weight, to generate composite image data having the summed pixel value.

9. The data processing device according to claim 6, wherein the data processing device is further configured to generate images based on the image data synchronized with the motion sensing data sampling time and including an image generated based on the motion sensing data and displayed thereon.

Description

DESCRIPTION OF DRAWINGS

(1) The above and other objects, features and other advantages of the present invention will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings, in which:

(2) FIG. 1 is a view showing a configuration of an apparatus for generating a result using image data regarding a golf swing motion of a user and motion sensing data obtained by sensing a golf club motion during golf swing;

(3) FIG. 2 is a view showing sampling time of the image data and motion sensing data acquired with respect to the golf swing motion of the user through the apparatus shown in FIG. 1;

(4) FIG. 3 is a view showing consecutive frames of an image created in such a manner that an object based on motion sensing data is displayed on image data, as a result generated according to the apparatus and the data sampling time shown in FIGS. 1 and 2;

(5) FIG. 4 is a view illustrating a method for synchronizing data between different types of device according to an embodiment of the present invention;

(6) FIG. 5 is a view illustrating a configuration of a data processing device for generating synchronized data according to an embodiment of the present invention;

(7) FIG. 6 is a view illustrating an example of information with respect to sampling time of data acquired through the configuration shown in FIG. 5;

(8) FIG. 7 is a view illustrating a principle of calculating weights for generating synchronized images corresponding to motion sensing data sampling time on the basis of the motion sensing data sampling time; and

(9) FIGS. 8 to 15 are views illustrating examples of image data with respect to a golf swing motion of a user, which includes synchronized images according to an embodiment of the present invention.

BEST MODE

(10) A method for synchronizing data between different types of devices and a data processing device for generating synchronized data according to the present invention will be described in detail with reference to the attached drawings.

(11) The method for synchronizing data between different types of devices according to an embodiment of the present invention provides a method of generating an integrated result using different types of data acquired by the devices with respect to a moving object.

(12) FIG. 4 is a view illustrating the method for synchronizing data between different types of devices according to an embodiment of the present invention and shows sampling time of first type data regarding motion of an object, acquired by one device, and sampling time of second type data regarding the motion of the object, acquired by the other device.

(13) Here, the first type data and the second type data are different types of data with respect to the object and, for example, may be image data and motion sensing data or different types of motion sensing data.

(14) Referring to FIG. 4, since the sampling interval t of the first type data is shorter than the sampling interval T of the second type data, the number of pieces of first type data sampled per unit time is greater than the number of pieces of second type data sampled per unit time. That is, a sampling rate of the first type data is higher than that of the second type data.

(15) For example, even if time t1 of the first type data is synchronized with type T1 of the second type data, as shown in FIG. 4, synchronized data is barely present after the time t1 and the time T1 since the first type data and the second type data have different sampling rates. Accordingly, it is different to generate a result using the two types of data.

(16) With respect to data having different sampling rates as illustrated in FIG. 4, the method for synchronizing data between different types of devices according to an embodiment of the present invention generates data, which are synchronized with the sampling time of the first type data on the basis of the first type data, for the second type data.

(17) That is, when the second type data includes data D1 sampled at time T1 and data D2 sampled at time T2, as shown in FIG. 4, data Dg1, data Dg2 and data Db3 respectively corresponding to sampling times t2, t3 and t4 of the first type data are generated through synchronization so as to generate data synchronized with the first type data with respect to the second type data.

(18) More specifically, the data Dg1, Dg2 and Dg3 shown in FIG. 4 are respectively synchronized with sampling times t2, t3 and t4 of the first type data. The data Dg1, Dg2 and Dg3 may be generated by combining data D1 and D2 respectively acquired at sampling times T1 and T2 of the second type data, which are closest to the sampling times t2, t3 and t4 of the first type data, according to predetermined conditions.

(19) Here, combination may depend on data type. For example, when the first type data are motion sensing data regarding a motion of an object and the second type data are image data regarding the object, combination may mean composing the image data according to predetermined conditions, summing the data D1 corresponding to T1 and the data D2 corresponding to T2 according to predetermined conditions, or deriving a value obtained by applying D1 and D2 to a predetermined function.

(20) Here, it may be possible to combine D1 and D2 by respectively applying weights to D1 and D2 depending on time. For example, the data Dg1 corresponding to the time t2 may be generated by assigning weights, which increase as D1 and D2 approach the time 2, to D1 and D2 and then combining the weighted D1 and D2.

(21) For example, the data Dg1 can be generated by calculating a first weight W1 to be applied to D1 as (T2t2)/T, calculating a second weight to be applied to D2 as (t2T1)/T and combining W1*D1 and W2*D2.

(22) Similarly, the data Dg2 can be generated by combining {(T2t3)/T}*D1 and {(t3T1)/T}*D2 and the data Dg3 can be generated by combining {(T2t4)/T}*D1 and {(t4T1)/T}*D2.

(23) A data processing device for generating synchronized data according to an embodiment of the present invention will be described with reference to FIG. 5.

(24) FIG. 5 is a block diagram of a data processing device for generating a result using image data with respect to a golf swing motion of a user U with a golf club GC and motion sensing data regarding the golf club. Referring to FIG. 5, the data processing device 200 according to an embodiment of the present invention may be connected with a camera device 100 and a display device 400, a motion sensing device MSD is attached to the golf club GC held by the user U and the data processing device 200 wirelessly receives data acquired by the motion sensing device MSD and processes the received data.

(25) The data processing device 200 according to an embodiment of the present invention may include a first sampler 210, a second sampler 220, a synchronizer 240 and an image generator 260.

(26) The first sampler 210 samples image data about the user who is performing golf swing, acquired from the camera device 100, and the second sampler samples motion sensing data about the golf club GC moving according to golf swing of the user U, acquired by the motion sensing device MSD.

(27) The synchronizer 240 generates image data synchronized with motion sensing data having a high sampling rate. The synchronizer 240 generates synchronized image data by applying predetermined weights to image data sampled before and after a motion sensing data sampling time and combining the weighted image data.

(28) More specifically, the synchronizer 240 includes a weight calculator 242 and a composing unit 244.

(29) The weight calculator 242 is configured to calculate a weight, which increases as image data corresponding thereto is closer to a motion sensing data sampling time, for the image data. The weight calculator 242 respectively calculates weights for image data sampled before and after the motion sensing data sampling time.

(30) The composing unit 244 respectively applies the calculated weights to the image data sampled before and after the motion sensing data sampling time and composes the weighted image data to generate image data corresponding to the motion sensing data sampling time.

(31) The image generator 260 generates an image by overlapping an image corresponding to the generated image data synchronized with the motion sensing data sampling time and an image generated based on the motion sensing data.

(32) The synchronized image generated by the image generator 260 is reproduced through the display device 400 and provided to the user.

(33) An example of information with respect to sampling times of data acquired by the aforementioned configuration is shown in FIG. 6 and examples of images generated corresponding to the sampling times are shown in FIGS. 8 to 15. FIG. 7 is a view for explaining a principle of weight calculation by the weight calculator.

(34) Referring to FIG. 6, image data acquired through the camera device 100 and motion sensing data acquired through the motion sensing device MSD in the configuration shown in FIG. 5 are sampled at different sampling intervals. Here, 8 motion sensing data are sampled while one frame of image data is sampled.

(35) It is assumed that time T1 is synchronized with time t1 and time T2 is synchronized with time t8, as shown in FIG. 6, for convenience of description.

(36) An object of the data processing device according to an embodiment of the present invention is to generate image data synchronized with times t2 to t7 between the time T1 and time T2.

(37) The image data synchronized with t2 to t7 may be generated by combining, that is, composing, image data acquired at T1 and image data acquired at T2. To compose the image data, weights are calculated and applied to the image data.

(38) A description will be given of a process of calculating weights when image data corresponding to a motion sensing data sampling time ti between a sampling time Ta and a sampling time Tb of the image data (the interval between Ta and Tb is referred to as T) with reference to FIG. 7.

(39) When the interval between Ta and ti is Ta and the interval between Tb and ti is Tb, Ta and Tb may be calculated as follows.
Ta=tiTa
Tb=Tbti

(40) Image data acquired at Ta is denoted by fa, a pixel value of the image data fa is denoted by Pa[Ra,Ga,Ba], image data acquired at Tb is denoted by fb and a pixel value of the image data fb is denoted by Pb[Rb,Gb,Bb]. When a weight to be applied to the image data fa is Wa and a weight to be applied to the image data fb is Wb, Wa and Wb may be calculated as follows.
Wa=Tb/T
Wb=Ta/T

(41) When synchronized image data corresponding to ti is fg and a pixel value thereof is Pg[Rg,Gg,Bg], Pg[Rg,Gg,Bg] may be calculated as follows.
Pg[Rg,Gg,Bg]=Pa[Wa*Ra,Wa*Ga,Wa*Ba]+Pb[Wb*Rb,Wb*Gb,Wb*Bb]

(42) That is, the weight Wa is applied to values of all pixels of the image data fa, the weight Wb is applied to values of all pixels of the image data fb, and pixel values to which the weight Wa has been applied and pixel values respectively corresponding to the pixel values and weighted by Wb are summed to generate composite image data having the summed pixel values, thereby generating image data corresponding to the time ti.

(43) In this manner, images shown in FIGS. 8 to 15 may be obtained by representing images based on image data generated corresponding to t2 to t7 using image data acquired at T1 and T2, as shown in FIG. 6, along with an object SO generated based on motion sensing data acquired at t1 to t8 and overlapping the images (SO denoting the object is shown only in FIG. 8 and omitted in FIGS. 9 to 15).

(44) FIG. 8 shows an image corresponding to the image data acquired at T1 and having an object generated based on motion sensing data acquired at t1 and displayed thereon.

(45) FIG. 15 shows an image corresponding to the image data acquired at T2 and having an object generated based on motion sensing data acquired at t8 and displayed thereon.

(46) FIG. 9 shows an image generated to be synchronized with t2, FIG. 10 shows an image generated to be synchronized with t3, FIG. 11 shows an image generated to be synchronized with t4, FIG. 12 shows an image generated to be synchronized with t5, FIG. 13 shows an image generated to be synchronized with t6 and FIG. 14 shows an image generated to be synchronized with t7.

(47) The images shown in FIGS. 9 to 14 are generated by applying weights calculated according to the principle illustrated in FIG. 7 to the image data corresponding to the images shown in FIGS. 8 and 15 and composing the image data.

(48) Referring to FIG. 9, the image data (referred to as image data f1) corresponding to T1 and the image data (referred to as image data f2) corresponding to T2 are composed to generate the image. Here, it can be confirmed that the image data f1 is represented more clearly and the image data f2 is represented less clearly.

(49) It can be confirmed that the image data f1 becomes less clear and the image data f2 becomes clearer in the images shown in FIGS. 9 to 14.

(50) FIGS. 8 to 15 show still images and thus it is difficult to recognize effects of synchronized images therefrom. However, if the still images are continuously reproduced to be displayed as a moving image, the images are viewed as if the image data respectively acquired at t1 to t8 are continuously reproduced due to persistence of vision (the optical illusion whereby multiple discrete images blend into a single image in the human mind).

(51) That is, when synchronized images are generated and reproduced according to the present invention, even if the images are acquired using a middle speed camera, for example, it is possible to obtain the same effect as reproduction of images acquired using a super-high speed camera (that is, even when a camera which acquires an image of 50 frames per second is used, effects of acquiring and reproducing an image of hundreds of frames per second are achieved by generating images synchronized with motion sensing data sampling times according to the present invention).

(52) Generation of synchronized data according to the present invention is not limited to image data and motion sensing data as described above and is applied in various manners.

(53) For example, when a sports motion of a user, such as a golf swing motion is analyzed, data acquired by a motion tracking device attached to the body of the user may be synchronized with data acquired by a motion sensing device attached to a golf club gripped by the user. If a sampling rate of the motion sensing device is higher than that of the motion tracking device, data, obtained by calculating weights according to the principle illustrated in FIG. 7, respectively applying the weights to sampled motion tracking data, and summing the weighted data or applying the data to a predetermined function, may be generated as data synchronized with motion sensing data. A result such as creation of accurate analysis information with respect to the golf swing motion of the user can be generated using the motion tracking data and motion sensing data generated as above.

(54) As described above, according to the method for synchronizing data between different types of devices and the data processing device for generating synchronized data according to the present invention, synchronized data can be generated with respect to data acquired by the devices. Accordingly, when a result is generated, data acquired by the different types of devices can be used.

(55) Therefore, it is possible to generate a result having a greater value than that obtained when only data acquired by each device are used.

MODE FOR INVENTION

(56) Various embodiments for carrying out the invention have been described in the best mode for carrying out the invention.

INDUSTRIAL APPLICABILITY

(57) The method for synchronizing data between different types of devices and the data processing device for generating synchronized data according to the present invention can generate synchronized data with respect to data acquired by the different types of devices, that is, heterogeneous devices to solve a problem that data acquired by different types of devices with respect to the same object are not synchronized due to a data acquisition time difference between the devices and thus can be available in industrial fields associated with information provision through processing and analysis of digital information.