Method and apparatus for position and motion instruction
09852652 · 2017-12-26
Assignee
Inventors
- Allen Yang Yang (Richmond, CA, US)
- Mohamed Nabil Hajj Chehade (Los Angeles, CA, US)
- Sina Fateh (Mountain View, CA, US)
- Sleiman Itani (East Palo Alto, CA, US)
Cpc classification
International classification
Abstract
World data is established, including real-world position and/or real-world motion of an entity. Target data is established, including planned or ideal position and/or motion for the entity. Guide data is established, including information for guiding a person or other subject in bringing world data into match with target data. The guide data is outputted to the subject as virtual and/or augmented reality data. Evaluation data may be established, including a comparison of world data with target data. World data, target data, guide data, and/or evaluation data may be dynamically updated. Subjects may be instructed in positions and motions by using guide data to bring world data into match with target data, and by receiving evaluation data. Instruction includes physical therapy, sports, recreation, medical treatment, fabrication, diagnostics, repair of mechanical systems, etc.
Claims
1. A machine-implemented method, comprising: sensing world data via a sensor of a motion instruction apparatus, said world data comprising a world position of a body part of a user; determining target data via a processor of the motion instruction apparatus, said target data comprising a target position of the body part of the user; producing visual guide data via the processor, said visual guide data being based on the target data and the world data; displaying said visual guide data overlaid onto the world data to the user via a pass-through display of the motion instruction apparatus; wherein said visual guide data comprises at least one of a group consisting of visual virtual reality data and visual augmented reality data; and repeating, in response to a movement of the body part, the sensing, determining, producing and displaying steps until the world position of the body part matches the target position of the body part to within a predetermined margin; signaling, via the pass-through display, to the user that the world position matches the target position.
2. The method of claim 1, wherein: said world data comprises a world point of contact with said entity.
3. The method of claim 1, wherein: said target data comprises a target point of contact with said entity.
4. The method of claim 1, wherein: said guide data comprises a guide point of contact with said entity.
5. The method of claim 1, wherein: said world data comprises a world path for said entity.
6. The method of claim 1, wherein: said target data comprises a target path for said entity.
7. The method of claim 1, wherein: said guide data comprises a guide path for said entity.
8. The method of claim 1 wherein: said world data comprises a world model for at least a portion of said entity.
9. The method of claim 1, wherein: said target data comprises a target model for at least a portion of said entity.
10. The method of claim 1, wherein: said guide data comprises a guide model for at least a portion of said entity.
11. The method of claim 1, wherein: said guide data comprises a visual representation of at least a portion of said entity.
12. The method of claim 1, wherein: said guide data comprises a visual representation of at least a portion of said subject.
13. The method of claim 12, wherein: said visual representation is at least substantially anthropomorphic.
14. The method of claim 1, comprising: outputting said guide data such that at least a portion of said world data is visible therewith.
15. The method of claim 1, comprising: passing at least a portion of said world data to said subject.
16. The method of claim 1, comprising: outputting at least a portion of said world data to said subject.
17. The method of claim 1, comprising: outputting said guide data at least substantially aligned with said entity.
18. The method of claim 1, comprising: outputting said guide data at least substantially aligned with said world data.
19. The method of claim 1, wherein: said target data is representative of at least one of a group consisting of physical therapy, sports activity, recreational activity, physical training, medical training, medical treatment, medical diagnostics, first aid/cpr, fabrication, repair, and device/system operation.
20. The method of claim 1, wherein: said world entity comprises a mechanical system.
21. The method of claim 1, wherein: said world entity comprises at least one of a virtual reality entity and an augmented reality entity.
22. The method of claim 1, comprising: dynamically updating at least a portion of said world data until a condition is satisfied.
23. The method of claim 1, comprising: dynamically updating at least a portion of said target data responsive to said world data until a condition is satisfied.
24. The method of claim 1, comprising: dynamically updating at least a portion of said guide data responsive to said target data until a condition is satisfied.
25. The method of claim 1, comprising: establishing evaluation data substantially representing an evaluation of said world data against said target data.
26. The method of claim 25, comprising: dynamically updating at least a portion of said evaluation data until a condition is satisfied.
27. The method of claim 25, comprising: outputting at least a portion of said evaluation data to said subject.
28. The method of claim 27, wherein: said evaluation data comprises at least one of a group consisting of text instructions, audio instructions, a visual image, an animation, and an icon.
29. A machine-implemented method, comprising: sensing visual world data with a sensor, said visual world data comprising a world motion of a body part of a user; communicating said visual world data to a processor; determining visual target data in said processor with respect to said visual world data, said visual target data comprising a target motion of said body part; generating visual guide data in said processor based on said visual world data and said visual target data; displaying said visual guide data overlaid onto the world data from said processor to said user via a pass-through display; wherein said visual guide data comprises visual augmented reality data displayed in conjunction with said visual world data on said pass-through display; and repeating, in response to a movement of the body part, the sensing communicating, determining, generating, and displaying steps until the world motion of the body part matches the target motion of the body part to within a predetermined margin; signaling, via the pass-through display, to the user that the world position matches the target position.
30. The method of claim 1, wherein: said visual guide data comprises at least one of a group consisting of text instructions, a visual image, an animation, and an icon.
Description
BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS
(1) Like reference numbers generally indicate corresponding elements in the figures.
(2)
(3)
(4)
(5)
(6)
(7)
(8)
(9)
(10)
(11)
(12)
(13)
(14)
(15)
(16)
(17)
(18)
(19)
(20)
DETAILED DESCRIPTION OF THE INVENTION
(21) It is noted that for simplicity, some of the following description is written to refer to position/motion instruction directed to a subject's kinesiology, that is, to the position and/or motion of the body of the subject himself or herself. However, this is an example only, and the present invention is not limited only to kinesiology instruction.
(22) In addition, the terms “position” and “motion” as used herein are to be understood broadly. A position, for example considering a hand position, is not necessarily limited to only a set of coordinates in space, in the sense of an x, y, z position. Rather, a hand position may include specific arrangements of fingers, angles at certain joints, an overall orientation, etc. Likewise, a motion is not necessarily limited to a change from one set of coordinates to another, but may encompass substantially any dynamic transition. The terms position and motion might also be considered as “posture” and “gesture”, or individual positions, motions, etc. could be defined very specifically. However, for purposes of simplicity, an arrangement wherein the goal is a substantially static configuration is typically referred to herein as a position, while an arrangement wherein the goal is a substantially dynamically changing configuration is typically referred to herein as a motion.
(23) Referring now to
(24) With regard to the terms “entity” and “subject”, an entity is an object, location, creature, phenomenon, etc. to which a position and/or a motion is to be applied. A subject is a person to be instructed or otherwise informed in the execution of a position and/or motion. In certain instances, a subject may be considered an actor, with an entity being a thing that is acted upon. However, it is emphasized that an entity and a subject may be one and the same, or at least closely related, in at least some embodiments. For example, for physical therapy motions of a hand, the subject (executing the motions) may be a patient performing exercises, while the entity (what is being moved) may be the patient's hand. In such an example, the patient is arguably both subject and entity.
(25) Returning to a description of world data, typically though not necessarily world data is established 108 in a data processor, such as a processor in a personal computer, smart phone, head mounted display, etc.
(26) The present invention is not particularly limited with regard to what entities may be considered. In particular, the terms “world” and/or “world data” should not be understood to imply that the present invention is limited only to considering objects within the physical world as entities. For example, in certain embodiments, virtual reality entities and/or augmented reality entities may be utilized; in such instances, world data will not necessarily reflect the position and/or motion of a physical entity, but may reflect the position and/or motion of a data construct (i.e. a virtual or augmented reality entity).
(27) The present invention is not particularly limited with regard to subjects. Typically, though not necessarily, the subject may be a person who is executing some position and/or motion. However, subjects may also include (but are not limited to) others such as instructors, evaluators, medical personnel, etc.
(28) The present invention also is not particularly limited with regard to how world data is established 108. For example, world data may be established 108 by being sensed using one or more sensors. As a more concrete example, an imaging sensor, depth sensor, etc. may be used to determine the position of the entity in question, the motion of the entity, etc., with that real-world position and motion information then being used as or being used to inform the world data. As an alternative, world data may be received from some external system, e.g. via wired or wireless communication. As another alternative, world data may be read from data storage, for example memory in a processor itself or some distinct data store such as a hard drive, solid state drive, etc. As yet another alternative, world data may be generated, e.g. within a processor. An example of the latter arrangement might include a situation wherein the entity in question is a virtual or augmented reality entity, and the processor in question is generating and/or updating that entity (thus, the processor might be presumed “know” the position and/or motion of the entity that the processor is generating).
(29) The present invention furthermore is not particularly limited with regard to the form or content of the world data. World data may describe the entity and/or the environment of the entity in substantially any fashion. World data may be static and/or dynamic, and may include, but is not limited to, mathematical information, coordinates, one-, two-, three-, and/or higher-dimensional geometric constructs, images, video, audio, text, and models such as a model of the entity in question. World data may vary considerably within such examples (which are not themselves exhaustive). For example, considering geometric constructs, world data may include points such as points of contact, e.g. between points on an entity, points on a subject, points of contact between an entity and a subject (as a more particular example, between a hand and some held device), etc. Other geometric constructs might include vectors or paths of motion, start marks, midway marks, and/or stop marks, boundary marks (e.g. “do not move past this point”), etc.
(30) World data is not limited only to information that directly describes position and/or motion of an entity. For example, world data may include information such as the mass of an entity, the color of an entity, the air resistance of an entity (which may in turn include or reflect information regarding ambient environmental conditions such as air temperature), etc. Other data also may be equally suitable.
(31) Continuing in
(32) Typically, though not necessarily, the target data is established 110 in a data processor, such as a processor in a personal computer, smart phone, head mounted display, etc.
(33) As already described with regard to world data, the present invention is not particularly limited regarding what subjects and/or entities may be considered for target data, how target data may be established, or the form and content of target data. For example, target data may be static and/or dynamic, and may include but is not limited to mathematical information, coordinates, one-, two-, three-, and/or higher-dimensional geometric constructs, points of contact, vectors or directional arrows, images, video, audio, text, and models such as a model of the entity in question.
(34) However, it is noted that typically though not necessarily, target data may be established 110 in advance of a desired position and/or motion that is to be represented thereby. Considering physical therapy of a subject's hand as an example, positions and/or motions for the subject's hand may be partially or entirely pre-determined, e.g. being specified by knowledgeable medical personnel in order to achieve some therapeutic end. However, such an arrangement is an example only, and other arrangements may be equally suitable.
(35) Moving on in
(36) As has been described with regard to world data and target data, the present invention is not particularly limited with regard to what subjects and/or entities may be considered, how guide data may be established, or the form and content of guide data. Guide data may be static and/or dynamic, and may include, but is not limited to, mathematical information, coordinates, one-, two-, three-, and/or higher-dimensional geometric constructs, images, video, audio, text, and models such as a model of the entity in question. However, it is noted that typically though not necessarily, guide data may be situation-specific, that is, guide data may consider and/or depend on either or both of the world data and the target data. To continue the example of physical therapy for a subject's hand, the guide data may consider the actual position and/or motion of the hand (world data) in conjunction with the desired position and/or motion of the hand (target data) in establishing the guide data 112. Put another way, guide data is not necessarily limited to static, predetermined guidance for executing a position or motion, but may be tailored specifically to a particular subject, time, place, etc., so as to provide tailored guidance to a subject in executing the relevant position or motion. As a more concrete example, guide data may consider world data such as the position of the subject's hand, the current level of mobility of individual joints thereof, etc., and/or target data such as motions designed for a particular user's medical status (e.g. recovering from carpal tunnel surgery). World data and/or target data likewise may be tailored to subjects and/or circumstances.
(37) With regard to form and content of guide data, it is noted that guide data typically though not necessarily will have a form dictated at least in part by the manner by which the guide data is to be outputted (below). For example, visual guide data might include points of contact, path information such as vectors or directional arrows, outlines of subject and/or entity and/or portions thereof, models (including but not limited to 2D and 3D models) that enclose and/or are coincident some or all of the subject and/or entity, etc.
(38) It is noted that, although different data groups are referred to herein—e.g. world data, target data, guide data, evaluation data (described subsequently herein)—this does not imply that such data groups are or necessarily must be different in content. For example, for certain embodiments of the present invention target data and guide data might be identical. That is, the guide data that is to be outputted to a subject to guide a position or motion might be the same as the target data considered as the ideal for that position or motion, etc. Other similarities or equivalencies between data groups also may exist, and though not required are not prohibited.
(39) Continuing in
(40) Although
(41) For example, it will be understood that the arrangement shown in
(42) However, the present invention is not limited only to static arrangements for position and/or motion instruction. Moving on to
(43) In the method shown in
(44) As already described with regard to world data, target data, and guide data, the present invention is not particularly limited with regard to how the termination condition is established 206. Similarly, the present invention is not particularly limited insofar as what may serve as a termination condition.
(45) For example, for certain embodiments time may be used as a termination condition. That is, when some period of time has elapsed the termination condition is satisfied (and thus the method of
(46) As an alternative, some number of cycles of the method may be utilized, for example as a counter. Thus, the method might operate for 1,000 cycles (possibly though not necessarily relating to the clock cycles of a controlling processor) before terminating.
(47) As another alternative, the termination condition might be based on whether the position and/or motion in question is executed. This might be accomplished for example by considering whether the world data matches the target data. That is, considering a physical therapy hand position as target data, when the world data for the subject's hand matches the target data (perhaps to within some margin), it may be considered that the position has been executed, at which point the termination condition may be satisfied and the method then terminates.
(48) The preceding are examples only, and other arrangements for termination conditions may be equally suitable.
(49) Continuing in
(50) However, in the arrangement of
(51) Although certain steps of the method shown in
(52) However, although world data, target data, guide data, etc. is not required to change when established 208, 210, 212, etc. in multiple cycles, neither is such change prohibited. Indeed, changes in one or more of world data, target data, and guide data (and also evaluation data, described below) may enable useful features of the present invention.
(53) Notably, in establishing world data 208 in each cycle, world data may be dynamically updated in each cycle. For example, for a subject manipulating some entity toward a position or in a motion, the actual position, orientation, speed, path, etc. for the entity—world data—may be updated over time. Using as an example hand motions, as the subject moves his or her hand over time, the world data describing hand position and hand motion is updated over time. World data thus is not necessarily static or fixed, but rather may be dynamically updated.
(54) Similarly, target data may be dynamically updated. While for at least certain embodiments a target position or motion may be essentially fixed for the execution of that position or motion, this is not required. To continue the hand motion example above, for at least certain embodiments the targets for where and how a subject is to move his or her hand may change dynamically depending on the position and motion of the subject's hand from one moment to another. For example, if actual positions and/or motions (world data) for a subject's hand indicate that the subject has limited range of motion, experiences pain from certain positions and/or motions, has higher-than-average flexibility, etc., then the target data might be adjusted in response, e.g. a target position might be made less complex, a target motion might be made slower, etc. Put another way, if the subject has difficulty performing a motion, the motion might be modified so as to be less difficult for the subject. Examples wherein such dynamic updating of target data might be useful include, but are not limited to, physical therapy, athletic training, instruction in playing a musical instrument, etc.
(55) Likewise, guide data may be dynamically updated. Even if target data remains unchanged, as a subject proceeds to execute a position and/or motion, if the world data is changing—for example, if the subject has begun to move toward the position and/or initiated the motion—then the degree and direction of change in position, motion, etc. that is required of the subject may change. More concretely, if a subject is to execute a golf swing, when the subject is halfway through the swing the amount and direction of motion that is yet to be executed is different than when the subject is just beginning the swing. Guide data may be dynamically updated for example to reflect the current difference between world data and target data, so that the guide data presented to the subject is up-to-date. Furthermore, if the subject deviates from a planned position or motion, guide data may be dynamically updated to show the subject how to correct the position or motion.
(56) It is noted that even in embodiments that include dynamic updating of world data, target data, and/or guide data, it is not necessary to update each of world data, target data, and guide data. That is, target data might be updated without updating world data, guide data might be updated without updating target data, etc. Additionally, it is not necessary for all embodiments to update all of any or all such types of data, e.g. none, some, or all of guide data might be dynamically updated, some, none, or all of target data may be dynamically updated, some, none, or all of world data may be updated, etc.
(57) Turning now to
(58) In the arrangement of
(59) Continuing in
(60) The present invention is not particularly limited with regard to how evaluation data is established 314. Typically, though not necessarily, evaluation data is established through comparison of data already available, e.g. world data against target data and/or guide data as described above, but this is an example only and other arrangements may be equally suitable.
(61) The present invention also is not particularly limited with regard to form and/or content of evaluation data. Evaluation data may include but is not limited to numerical data (e.g. between a world data position and a target data position, between a world data velocity for a motion and a target data velocity for that motion, etc.), tables, charts, images, etc. In addition, evaluation data may include messages, warnings, etc. As more particular examples, evaluation data might include information for a subject regarding common errors in the position or motion, warnings about dangers associated with the position or motion, an instruction to cease executing the position or motion if the user makes some serious or dangerous error, an overall judgment regarding the execution of one or more positions and/or motions (e.g. “complete”, “successful”, “excellent”, “improved over previous set”, etc.).
(62) Still with reference to
(63) Evaluation data also may be outputted 322. As previously noted, evaluation data at least substantially represents a degree to which the position and/or motion being instructed is successfully conveyed to the subject who is to execute the position and/or motion, and may include comparisons of an actual position/motion (world data) against a planned position/motion (target data), etc. In step 322, some or all of the evaluation data established in step 314 may be outputted, e.g. to the subject executing the position/motion.
(64) Outputting evaluation data 322 is an optional step, as may be understood for example by noting that no analog to outputting evaluation data 322 appears in
(65) In addition, as has been noted with regard to world data, target data, and guide data, evaluation data may be dynamically updated, e.g. from one repetition to another. Thus, as a subject executes a position and/or a motion, an evaluation therefor may be established and/or outputted, reflecting changes in position and/or motion dynamically as those changes occur.
(66) Moving on in
(67) Turning now to
(68) In the example method of
(69) As noted above with regard to world data, target data, etc., the present invention is not particularly limited with regard to how an evaluation condition may be established, the form or content of an evaluation condition, etc.
(70) Continuing in
(71) Again as already noted with regard to world data, target data, etc., the present invention is not particularly limited with regard to how an evaluation response may be established, the form or content of an evaluation response, etc.
(72) Still with reference to
(73) A determination is made 424 as to whether the termination condition is satisfied. If the termination condition is not satisfied, the method returns to step 408. If the termination condition is satisfied, the method is complete.
(74) As has been noted with regard to
(75) Turning now to
(76) In
(77) Nevertheless, for clarity in
(78) As may be seen in
(79) Moving on to
(80) Continuing with
(81) Now with reference to
(82) With reference to
(83) While individually each of
(84) It is noted that the particular sequence of positions represented by
(85) Now with reference to
(86) Although for clarity the world data 1030 is shown to be in the desired state as established by the target data 1032, it will be understood that in practice world data 1030 will not necessarily match target data 1032 at all times (nor will world data necessarily ever match target data in at least certain instances). It may be desirable for a subject to manipulate an entity—in the example of
(87) It is emphasized that target data 1032 is not required to be visible to a subject, and for many embodiments may not be visible. Furthermore, even if target data 1032 is visible, target data 1032 will not necessarily be overlaid onto world data 1030, nor will target data 1032 necessarily be of the form shown in
(88) In
(89) Moving on to
(90) Continuing with
(91) Now with reference to
(92) With reference to
(93) While individually each of
(94) Similarly to the arrangement described with regard to
(95) Now with reference to
(96) Moving on to
(97) Continuing with
(98) Now with reference to
(99) With reference to
(100) In
(101)
(102) With reference to
(103) Moving on to
(104) Continuing with
(105) Now with reference to
(106) With reference to
(107) Collectively,
(108) Similarly, collectively
(109) Also similarly, collectively
(110) It will be understood that as static illustrations,
(111) With reference now to
(112) With reference now to
(113) Although not shown herein, similar intermediate images may be understood to be possible (and/or implied) with regard to other sequences of images in
(114) Thus, as shown, and as previously described, world data, target data, and/or guide data may update dynamically, so as to reflect changes to world data, target data, and/or guide data appropriate to (and/or in response to) changes within the system (e.g. changes of the entity, changes of the subject, etc.)
(115) Collectively,
(116) At this point it may be useful to expand somewhat on the range of possible forms that world data, target data, and/or guide data may take.
(117) As previously indicated, world data at least substantially represents a state of “the world” (whether that world is real, augmented, virtual, etc.) as may include one or more entities. That is, world data may be considered to represent the world “as it is”. In
(118) However, world data may represent and/or reflect the status of entities without necessarily either showing the entities themselves or realistically simulating those entities. In particular, it is noted that world data is not necessarily required to be displayed to a subject for all embodiments of the present invention. An arrangement—such as may be seen in
(119) Still with reference to a hand as an entity, world data therefor might include a simple outline of the hand, a wireframe for the hand, a simple segment-and-joint illustration or “stick drawing”, etc. In addition, world data is not required to be anatomic at all, even when the entity in question is anatomical. For example, world data for a hand might include points, circles, x-marks, etc. representing the position of fingertips, joints, edges, and/or other features.
(120) In addition, world data is not necessarily limited only to reflecting an entity. Consider an example (illustrated and described subsequently herein) wherein a subject is to be instructed in position/motion for a golf club. Even if the golf club is considered the entity under consideration, this would not exclude the possibility that world data may include the hands of the subject manipulating the club, a golf ball, a golf tee, etc. Nor is world data limited to only a single entity at any given time. (Thus, it also is not excluded to consider both the club and the subjects hands to be entities.)
(121) Furthermore, world data is not required to be visible in nature, nor is world data required to be entirely visible even if some portion thereof is visible. For example, world data might include mathematical descriptions of position and/or motion that are not readily expressed visually, and/or that are not visually represented to the subject. World data likewise might include such information as an identity of an entity and/or a subject, for example world data could include information on a specific model of golf club as an entity, the identity of a subject manipulating that golf club, etc.
(122) Typically, though not necessarily, world data may include realistic imagery of one or more entities, and/or some portion of a subject. However, as has been described, this is an example only, and other arrangements may be equally suitable.
(123) With regard to target data, as has been previously indicated target data at least substantially represents a desired state for an arrangement that may include one or more entities. That is, target data may be considered to represent the world “as it should be”. In
(124) However, target data may represent target positions and/or motions for entities in different forms. In particular, it is noted that target data is not required to be displayed to a subject. An arrangement—such as may be seen in
(125) Regardless of whether target data is displayed, the form of target data may vary considerably. Although target data as shown in
(126) As with world data, target data might include a simple outline of the hand, a wireframe for the hand, etc. In addition, target data may be non-anatomical, even when the entity in question is anatomical. For example, target data for a hand might include points, circles, x-marks, etc. representing the position of fingertips, joints, edges, and/or other features. Target data also may include ranges, limits, and/or or other non-unique features, including but not limited to range-of-motion limits such as marks showing an acceptable or preferred range of positions or motions for a hand or finger, a range of suitable angles for bending a finger knuckle or other joint, etc.
(127) Furthermore, target data might include negative information, i.e. information representing something that a subject is not to do, as opposed to what a subject is to do. As more concrete examples, target data might include information indicating that fingers should not be touching in an exercise, that certain joints should not be bent (or should not be straightened), that a hand should not extend past some point or move outside of some range, etc.
(128) In addition, target data is not necessarily limited only to reflecting an entity. Continuing the example wherein a subject is to be instructed in position/motion for a golf club, even if the golf club is considered the entity under consideration, this would not exclude the possibility that target data may include the hands of the subject manipulating the club, a golf ball, a golf tee, etc. Nor is target data limited to only a single entity at any given time. (Thus, it also is not excluded to consider both the club and the subjects hands to be entities.)
(129) Typically, though not necessarily, target data may be non-visible to a subject, including models and/or other data for consideration by a processor. However, as has been described, this is an example only, and other arrangements may be equally suitable.
(130) With regard to guide data, as has been previously indicated guide data at least substantially represents information indicating to a subject how to bring an entity into line with a target position and/or motion. That is, guide data may be considered to represent a bridge for linking world data for an entity “as it is” and target data for an entity “as it should be”, so as to guide the subject in bringing the world into the configuration that the world is desired to have.
(131) In
(132) However, guide data may represent target positions and/or motions for entities in different forms. Although target data as shown in
(133) As with world data and target data, guide data may include a line-and-joint model of the hand, a wireframe for the hand, a filled/colored area for the hand, etc. In addition, guide data may be non-anatomical, even when the entity in question is anatomical. For example, guide data for a hand might include points, circles, x-marks, etc. representing the position of fingertips, joints, edges, and/or other features. Guide data also may include ranges, limits, and/or or other non-unique features, including but not limited to range-of-motion limits such as marks showing an acceptable or preferred range of positions or motions for a hand or finger, a range of suitable angles for bending a finger knuckle or other joint, etc.
(134) Furthermore, guide data may include negative information, i.e. information representing something that a subject is not to do, as opposed to what a subject is to do. As more concrete examples, guide data might include information indicating that fingers should not be touching in an exercise, that certain joints should not be bent (or should not be straightened), that a hand should not extend past some point or move outside of some range, etc.
(135) In addition, guide data is not necessarily limited only to reflecting an entity. Continuing the example wherein a subject is to be instructed in position/motion for a golf club, even if the golf club is considered the entity under consideration, this would not exclude the possibility that guide data may include the hands of the subject manipulating the club, a golf ball, a golf tee, etc. Nor is guide data limited to only a single entity at any given time. (Thus, it also is not excluded to consider both the club and the subjects hands to be entities.)
(136) Moving on now to
(137)
(138) It will be understood that bandage compression as shown may represent position and/or motion instruction associated with medical training, medical treatment, first aid/cpr, etc.
(139) Now with reference to
(140) Referring now to
(141) In
(142) Now with reference to
(143) With reference particularly to
(144) With reference to
(145) In
(146) Considering
(147) Similarly, although examples are not shown in
(148) Similarly, although examples are not shown in
(149) As has been stated, not all world data, target data, and/or guide data regarding an entity or a subject must be either established or output to a subject. Now with reference to
(150) In
(151) Turning to
(152) Moving on to
(153) Similarly, although not shown, target data likewise is not required to encompass an entirety of an entity, and/or an entirety of a subject.
(154) It will be understood that gripping a golf club as illustrated in
(155) As noted previously the present invention may include evaluation data. For example, in
(156) In addition,
(157) The evaluation data 3536 as shown in
(158) For example, turning to
(159)
(160) These are examples only, and evaluation data may take other forms, including but not limited to text, visual images, animations, icons, gauges, models, data analyses, etc.
(161) In addition, evaluation data may be dynamically updated similarly to world data, target data, and/or guide data. For example, with reference again to
(162) Now with reference to
(163) Moving on to
(164) Turning to
(165) Collectively,
(166) As has been described, guide data in particular may take many forms. World data at least substantially represents an entity and/or subject as they are, and target data at least substantially represents the entity and/or subject as they should be, and both may vary considerably. However, guide data does not necessarily have to represent either what is or what should be, but rather is adapted to provide guidance in transitioning between what is and what should be. Such guidance may vary greatly, depending for example on individual entities, individual subjects, cultural norms, instruction conventions, etc.
(167) In particular, it is noted that guide data according to the present invention is not required to be either complete or explicit. That is, at least certain information may be suggested, implied, or even omitted, so long as the intent insofar as position or motion may be understood.
(168) For example in
(169)
(170)
(171) Collectively,
(172) In addition, it will be understood that manipulating a hex bolt as illustrated in
(173) Now with reference to
(174)
(175) Collectively
(176) Turning now
(177)
(178) However, although as shown in several preceding examples the present invention does not necessarily require complete, explicit and/or comprehensive guide data, such complete, explicit and/or comprehensive guide data also is not excluded from the present invention.
(179) Turning now to
(180) One or both of the processor 4742 and the sensor 4744 are adapted to establish world data, the world data including a world position of one or more entities and/or a world motion of one or more entities. It is noted that, for an arrangement wherein the processor 4742 is adapted to establish world data absent a sensor, the sensor 4744 may not be required. However, typically although not necessarily, the sensor 4744 either establishes the world data or cooperates with the processor 4742 in establishing the world data.
(181) The world data may include world positions and/or world motions for non-entities, including but not limited to a person or other subject (although as noted elsewhere herein in some arrangements the subject may be the entity in question), environments, phenomena and conditions, etc. The world data also may include information other than position and/or motion, including but not limited to other properties of the entity, properties of the subject, properties of the environment, etc.
(182) Where the processor 4742 is adapted to establish world data, the processor 4742 may be adapted to dynamically update world data, for example until some condition is met.
(183) The processor 4742 is also adapted to establish target data, the target data including a target position of one or more entities and/or a target motion of one or more entities. As with world data, target data may include target positions and/or motions for non-entities, and/or may include information other than position and/or motion.
(184) The processor 4742 may be adapted to dynamically update target data, for example until some condition is met.
(185) The processor 4742 is further adapted to establish guide data, the guide data being adapted to provide guidance (e.g. to a subject) for the entity toward the target position and/or target motion. Guide data may include guidance other than that for guiding the entity toward the target position and/or target motion, and/or information other than such guidance.
(186) The processor 4742 may be adapted to dynamically update guide data, for example until some condition is met.
(187) The processor 4742 may be adapted to establish evaluation data, the evaluation data being at least substantially representative of an evaluation and/or comparison of world data against target data. Evaluation data may include information evaluating and/or comparing data other than world data against target data, and/or may include information other than evaluation and/or comparison data.
(188) These are examples only, and other arrangements for world data, target data, guide data, and evaluation data may be equally suitable.
(189) The processor 4742 may be adapted to record any or all of the world data, target data, guide data, and/or evaluation data. Other information generated or obtained associated with the use of the apparatus (or execution of a method according to the present invention) likewise may be stored.
(190) A range of general-purpose, special-purpose, and embedded systems may be suitable for use as the processor 4742. It may also be suitable for the processor 4742 to consist of two or more physical or logical processor components.
(191) As noted, the sensor 4744 may not be required (and thus may not be present) for some embodiments wherein the processor 4742 is adapted to establish world data absent a sensor. Where present, the sensor 4744 may be adapted to establish world data as described above.
(192) For embodiments including a sensor 4744, a variety of sensors may be suitable. As illustrated in
(193) As noted, world data may include data other than position and/or motion data, and a sensor 4744 may be adapted to establish data other than position and/or motion data. For example, such sensors may include but are not limited to biological sensors such as biomedical and/or biometric sensors. In addition, the use of multiple sensors, including but not limited to sensors such as imagers arranged in stereo pairs, also may be suitable.
(194) The display 4746 is adapted to output the guide data, e.g. so as to be viewable or otherwise perceivable by a person or other subject. The display 4746 may be adapted to output other data besides the guide data, including but not limited to world data and target data. The display 4746 may be, but is not required to be, a visual display (i.e. adapted to output visual guide data), but other arrangements including but not limited to audio and tactile displays may be equally suitable. The display 4746 also may be, but again is not required to be, a three dimensional display such as a stereo display or holographic display.
(195) A range of devices may be suitable for use as the display 4746, including but not limited to light emitting diodes (LED), organic light emitting diodes (OLED), plasma screen panels (PDP), liquid crystal displays (LCD), etc. Likewise, the use of projected or transmitted displays, wherein the viewed surface (if any) is essentially a passive screen for an image projected or otherwise transmitted after being generated elsewhere, may also be suitable. Other arrangements including but not limited to systems that display images directly onto a viewer's eyes also may also be suitable. Either digital or analog display technologies may be suitable. In addition, the use of multiple displays, including but not limited to displays arranged in stereo pairs, also may be suitable.
(196) In particular, the display 4746 may be transparent or semi-transparent, for example so as to passively transmit some or all world data therethrough (that is, a subject might see world data through a transparent display 4746, rather than the display actively outputting world data). However, this is an example only, and a display 4746 that actively outputs some or all world data is not excluded.
(197) A display 4746 may be adapted to output guide data proximate world data, to output guide data associated with world data, and/or to output guide data aligned with world data (e.g. so that guide data representing an outline or model of a hand substantially matches at least some portion of a world-data hand). The display 4746 may output world data and guide data that are proximate, associated, and/or aligned even if the display 4746 is partially or fully transparent as noted above; while for such instance the display 4746 may passively transmit world data, and thus may have limited or no control over where and/or how world data is positioned, oriented, etc., nevertheless the display 4746 may actively output guide data (and thus control position, orientation, etc. of guide data) so that the guide data corresponds with the world data.
(198) More broadly, a display 4746 may be adapted to output any two or more groups of data (world data, target data, guide data, evaluation data, etc.) so as to be proximate, associated, and/or aligned. (It is noted that terms such as “proximate” should be understood to apply to the appearance of the data as output to and/or perceived by a subject. That is, world data for an entity that is several meters distant might be considered to be itself likewise several meters distant. It is not suggested that presenting, for example, guide data proximate world data necessarily requires that guide data be presented at a distance of several meters as well; rather, guide data may be considered proximate guide data—even if world data is several meters away through a transparent display, and guide data is on the surface of that display—so long as a subject would perceive the guide data as being proximate the world data, regardless of the actual distance between guide data and world data.)
(199) Such arrangement of data to be proximate, associated, and/or aligned may facilitate convenient user comparison of different data, e.g. intended or optimal position and/or motion (target data) against actual position and/or motion as executed (world visible). Proximate or adjacent output of data may facilitate side-by-side comparison of data, for example world data against target data and/or guide data. Aligned and/or overlapping data might be transparent or opaque, with areas of overlap and/or difference potentially made readily visible thereby (and/or potentially highlighted, colored, shaded, etc.). However, these are examples, and other arrangements may be equally suitable.
(200) Where groups of data (world data, target data, guide data, evaluation data, etc.) are arranged together, the manner of arrangement will depend to at least some degree upon the nature of the groups of data. Also, as previously noted, each group of data may themselves take a range of forms.
(201) For example, in some embodiments guide data and/or world data may be outputted as models of entities or elements thereof that are to be positioned and/or moved. More concretely, for a position and/or motion related to a human arm, the guide data may be displayed as an image or model of a human arm, and the world data likewise may be displayed as an image of the subject's arm or a model based on the subject's arm. Such an arrangement may be advantageous in that it enables convenient comparison of more-or-less life-like imagery representing an example or target against more-or-less life-like imagery representing the user's real world arm and the motions thereof. However, this is an example only.
(202) In other embodiments, guide data and/or world data may be outputted as indicators or guidelines for the positioning and/or motion of entities. To continue the above example, for a position and/or motion related to a human arm, the guide data could be displayed as initial and final position markers for the subject's arm, as angular scales for the joints of the subject's arm, as pathways indicating example motions, etc. Likewise world data could be displayed as position markers showing actual initial and final locations of the subject's arm, the actual angular range of motion of the joints of the subject's arm, the pathways followed by the subject's arm when moving, etc. Again, this is an example only.
(203) Guide data and world data may be presented as individual still images, as a series of images, as full-motion video, and/or in other modes. Guide data and/or world data may be displayed multiple times. For example, guide data and/or world visual data may be displayed from multiple angles. Alternately, guide data might be presented once to illustrate a position/motion to a subject, then displayed again while the subject executes the position/motion, and/or displayed yet again as a comparison against world visual data (potentially being incorporated at least partly into evaluation data). However, such arrangements are examples only, and other arrangements may be equally suitable.
(204) For some embodiments, the association of guide data and world data may take place in real time, for example being substantially simultaneous. However, for other embodiments guide data and world data might be displayed alternatingly, i.e. first the guide data, then the world data (possibly repeating thereafter).
(205) Guide data and the world data need not take the same form. Thus, guide data might be presented as indicators and guidelines for position and/or motion, while world data might be presented as a realistic image or model of the entity that is to be positioned and/or moved. Likewise, models and/or images may be combined with indicators and guidelines for either or both of the guide data and the world data.
(206) It will be understood that for an arrangement wherein the display is transparent to enable viewing of world data as passively transmitted through the display, at least a portion of the world data necessarily will be realistic imagery of the entity in question (since the entity in question is in fact what is being viewed through the transparent display).
(207) Where used, models and images presented may be two dimensional “flat” images and models, three dimensional “solid” images and models, wireframe models, stick figures, etc. Likewise, where used indicators and guidelines may be two dimensional and/or three dimensional, and/or may include text, non-text symbols, graphics, etc.
(208) The arrangements described for the content and arrangement of guide data and world data are examples only, and other arrangements may be equally suitable. In particular, it is noted that although examples are provided above with regard to association of world data with guide data, similar associations of other groups of data (whether two at a time or more) are equally suitable, and may behave similarly and/or with comparable flexibility. However, guide data and world data are presented as examples since a comparison thereof may have readily understandable results, e.g. by comparing the guide data—that is, the position/motion as instructed—against the world visual data—the position/motion as actually executed—a subject may improve the likelihood of correctly executing a given position and/or motion. The subject may also see, if the position/motion is executed incorrectly, what errors were made, and where and when those errors were made. This can provide guidance for future improvement.
(209) Still with reference to
(210) Turning now to
(211) The communicator 4848, if present, may be adapted to communicate with some entity external to the apparatus. For example, a communicator 4848 might communicate through wired or wireless communication with some external data system, so as to record data therein, receive data therefrom, send or receive instructions, activate external devices, etc.
(212) A range of devices may be suitable for use as the communicator 4848, including but not limited to wired systems such as wired modems, wireless systems such as wifi transmitters, receivers, and/or transceivers. Other arrangements may be equally suitable.
(213) For embodiments including a communicator 4848, information available to the processor 4842 may be communicated to an external entity, and information from the external entity may be communicated with the processor 4842. For example, feedback regarding world data, target data, guide data, evaluation data, etc. could be so communicated.
(214) In such manner the world signal or other data could be transmitted to some other location for storage, expert analysis, collation into broader research, etc. For example, useful data (potentially part of world data, target data, guide data, and/or evaluation data, though also potentially separate therefrom) might include times and places of positions and motions, repetitions of positions and motions, speeds and performances of positions and motions, etc. Such data may be considered to correspond with the time and place that a subject executed a particular position and/or motion, and the number of times the subject repeated that position and/or motion; such information might be useful, for example, when instructing a subject in physical therapy, wherein the subject should perform certain exercises on certain schedules for best results. However, this is an example only, and other arrangements may be suitable.
(215) For embodiments having a communicator 4848, a subject or subjects may view and/or otherwise consider world data, target data, guide data, evaluation data, etc. without participating in executing positions or motions, or without even being present at the execution of positions or motions.
(216) For example, a subject may utilize an apparatus according to the present invention for physical therapy, while data therefrom is viewed by a trained physical therapist, who may be some considerable distance away. Similarly, a subject may utilize an apparatus according to the present invention to gather world data representing the motion of an industrial machine, while an engineer views the world data (and potentially target data, evaluation data, etc.) at a distant location. (It is noted that such an arrangement might potentially exclude the use of guide data; if world data is collected and compared with target data, without necessarily controlling the world data directly, guide data may not be necessary. To extend the example above, the subject observing the industrial machine may not directly control the machine's motions or other operations, and thus guide data might not be necessary since altering the machine's actual motions—the world data—may not be feasible. Thus, an arrangement of the present invention absent guide data may for certain embodiments be possible and useful. Typically, though not necessarily, such embodiments may utilize evaluation data, so as to compare world data against target data without necessarily requiring or utilizing guide data.) In addition, it may be possible for multiple persons to view world data, target data, guide data, evaluation data, etc. at one time, and/or to view such data at a time other than that when the position and/or motion was executed. However, such arrangements are examples only, and other arrangements may be equally suitable.
(217)
(218) A processor 4942 is disposed on the body 4950.
(219) First and second sensors 4944A and 4944B are disposed on the body 4950, the sensors 4944A and 4944B being arranged in a stereo pair. Moreover, as shown sensors 4944A and 4944B are arranged such that when a subject wears the body 4950, the field of view of sensors 4944A and 4944B may be substantially aligned with the line of sight of the subject, and/or the field of view of the sensors 4944A and 4944B may substantially match that of the subject (depending on the fields of view of the sensors 4944A and 4944B and/or the subject).
(220) First and second displays 4946A and 4946B are disposed on the body 4950, the displays 4946A and 4946B being arranged in a stereo pair. In addition, in the arrangement shown the displays 4946A and 4946B are arranged such that when a subject wears the body 4950 the displays 4946A and 4946B will be proximate to, substantially in front of, and disposed substantially aligned with the eyes of the subject.
(221) Without reference to a specific illustration, and speaking broadly with regard to the present invention, it is emphasized that methods and apparatuses according to the present invention may exhibit utility in a wide range of functions.
(222) For example, the present invention may instruct users regarding positions and/or motions related to kinesiology, that is, position and motion of the human body. Details such as the nature of guide data, world data, etc. and other features may depend to at least some extent on the particular application, and may vary considerably. Potential applications of the present invention are described below for illustrative purposes. They should not be taken as the only possible applications, however, and the present invention is not limited thereto.
(223) Physical therapy is in at least some instances dependent on proper positioning and motion of the human body in order to achieve proper results while avoiding injury. In some cases, the necessary positions and motions may be non-intuitive, perhaps difficult and/or painful. The present invention enables instruction wherein a human subject may match his or her body positions and motions to an example that can be readily compared with his or her own performance so as to efficiently achieve the benefits of proper physical therapy. Non-restorative physical training (e.g. strength training) likewise may be supported by the present invention.
(224) Sports performance may be to at least some degree a function of proper physical positioning and motion. For example, how and where a tennis racket or golf club is gripped, moved, etc. can have significant effects on how the ball moves. In addition, how and where the racket or club is gripped and moved can have an impact on the likelihood that a subject will sustain injury, whether single-event (e.g. sprains) or repetitive (e.g. “tennis elbow”). Through the present invention a subject may match actual position and motion to examples so as to promote improved performance and/or decreased rates of injury.
(225) Even for non-competitive recreational activities wherein performance is not considered critical, and/or wherein injury is unlikely, the present invention may promote proper form. For example, although a waltz may not be “scored”, nevertheless there may be “right” and “wrong” ways for a subject to move and/or position his or her body. Distinctions between proper and improper form can be observed by comparing the subject's world body position and motion with guide and/or target position and motion data according to the present invention.
(226) Similarly, non-recreational activities wherein proper form is of importance may be instructed through the present invention. For example, many actions performed by medical professionals for diagnosis, treatment, etc., can be extremely sensitive in terms of both where those actions are performed and how those actions are performed. When performing reduction of a dislocated shoulder, for example, where to grip and how to move the patient's body are of definite importance. Similarly, for certain surgical procedures even such subtle details as the precise position of fingers on a scalpel or other instrument may be important. The present invention may offer instruction regarding such actions.
(227) In addition, it is noted that for medical training, real-life training opportunities may be limited. Moreover, practicing certain medical procedures on a live person as a training exercise may be undesirable. Thus, non-patient training such as that provided by the present invention may be of particular importance.
(228) Likewise, non-training medical applications may also be facilitated by instruction with the present invention. Certain injuries, conditions, etc. may be diagnosed and/or evaluated at least in part by comparing a patient's world body position or body motion to a target and/or guide position or motion therefor. One such example might be monitoring of Huntington's disease, wherein a subject might attempt to move his or her hand to trace a line or other simple shape (i.e. the line or shape might serve as target data and/or guide data). The degree to which uncontrolled motions cause the subject's hand to deviate from that line may be used to determine the status and progress of the disease.
(229) While certain examples presented herein refer to instruction in terms of training or education, it is noted that “instruction” in the sense that the term is used with regard to the present invention is not necessarily limited only to training or educational environments, and that the present invention likewise is not limited to training or educational applications. Thus the present invention may be used to support proper execution of tasks “for real”, i.e. outside of a training environment. For example, a subject attempting to properly execute first aid or CPR might rely on the instruction provided by this invention in a real emergency, regardless of whether that subject had previous training or education in first aid/CPR. Similarly, a soldier attempting to disarm an improvised explosive device might benefit from real-time instruction as to proper procedure and known hazards, even if (and perhaps particularly if) he or she has no previous training in working with such munitions.
(230) Even in matters wherein proper body form is of lesser or no significance, instruction according to the present invention may be beneficial. For example, for a subject servicing a motor vehicle, how that subject positions his or her body to grip a particular mechanical part may be of little consequence, but how that part must be moved in order to be removed, attached or adjusted may be very significant, and may also be non-intuitive (especially for complex mechanisms such as engine components). In many cases, the essence of how a device is built, adjusted, and/or repaired may be related to how pieces are positioned and/or moved with respect to one another. The present invention may be beneficial in such instances.
(231) As previously stated, the present invention is not limited only to kinesiology, nor to the position/motion of a subject's body. Evaluating actual (world) position and/motion compared with example (target and/or guide) position and/or motion may be illuminating for other tasks. For example, returning to the example of the engine above, how different components in an engine move can provide guidance as to whether the engine is operating properly, and/or what problems the engine may have if the engine is not operating properly (e.g. what components are broken, missing, mis-adjusted, etc.).
(232) The range of positions and/or motions instructed through the present invention is extremely broad, and is not limited to those described. Additional tasks include, but are not limited to, physical fitness instruction, posture training, performance training, combat training, time and motion studies, and task-related physical performance evaluations, playing of musical instruments, martial arts, typing, writing, and assorted craft, fabrication, and repair skills.
(233) The above specification, examples, and data provide a complete description of the manufacture and use of the composition of the invention. Since many embodiments of the invention can be made without departing from the spirit and scope of the invention, the invention resides in the claims hereinafter appended.