HOLOGRAPHIC AUGMENTED REALITY ULTRASOUND NEEDLE GUIDE FOR INSERTION FOR PERCUTANEOUS SURGICAL PROCEDURES

Abstract

A holographic augmented reality ultrasound needle guide system and method includes an augmented reality display such as a headset wearable by a user. The augmented reality display is configured to depict a virtual ultrasound image. The augmented reality display is further configured to allow a user to select a desired reference point on the virtual ultrasound image. The system is configured to depict a holographic needle guide based on the selection of the desired reference point. The system is also configured to adjust a trajectory of the holographic needle guide to avoid intersecting undesired anatomical structures. The augmented reality display is further configured to stamp the holographic needle guide into a selectively locked trajectory and position.

Claims

1. A system for guiding percutaneous insertion of a needle by a user into a patient, comprising: an augmented reality display configured to depict a virtual ultrasound image of a portion of the patient and configured to depict a holographic needle guide on the patient based upon selection of a reference point in the virtual ultrasound image.

2. The system of claim 1, wherein the augmented reality display is configured to depict the virtual ultrasound image of the portion of the patient as a member selected from a group consisting of a virtual window, a virtual ultrasound projection on the patient, and a combination thereof.

3. The system of claim 1, further comprising a computer having a processor and a memory, the memory including non-transitory processor-executable instructions directing the augmented reality display to display the holographic needle guide on the patient based upon selection of the reference point in the virtual ultrasound image of the portion of the patient.

4. The system of claim 3, wherein the augmented reality display includes a headset display wearable by the user.

5. The system of claim 4, wherein the computer is integrated into the headset display wearable by the user.

6. The system of claim 1, wherein the virtual ultrasound image is selectable to identify the reference point by at least one of the user and a remote user located at a different site from the user and guiding the percutaneous insertion of the needle into the patient.

7. The system of claim 6, wherein a trajectory of the holographic needle guide is adjustable by at least one of the user and the remote user.

8. The system of claim 1, further comprising an ultrasound probe.

9. The system of claim 8, wherein the system is configured to obtain the virtual ultrasound image of the portion of the patient from the ultrasound probe in one of real time and as a prerecorded image.

10. The system of claim 8, further comprising a needle insertion guide configured to indicate at least one predetermined insertion point on the patient to the user, wherein the needle insertion guide is a physical template configured to be placed on the patient.

11. The system of claim 10, wherein the needle insertion guide is disposed on the ultrasound probe, and the needle insertion guide is configured to provide a virtual needle insertion point as the at least one predetermined insertion point on the patient.

12. The system of claim 1, further comprising tracking means configured to provide enhanced visualization of at least one of an anatomy of the patient and the needle, the tracking means selected from a group consisting of an infrared marker, an electro-magnetic tracker, an optical tracker, and combinations thereof.

13. The system of claim 1, wherein the augmented reality display is configured to depict a holographic error bar adjacent the holographic needle guide, the holographic error bar configured to alert the user of a deviation from a predetermined threshold of variance of a position of the needle in relation to a trajectory of the holographic needle guide.

14. A method for performing a surgical procedure, which includes percutaneous insertion of a needle by a user into a patient, the method comprising the steps of: providing an augmented reality display configured to depict a virtual ultrasound image of a portion of the patient and configured to depict a holographic needle guide on the patient based upon selection of a reference point in the virtual ultrasound image; selecting the reference point in the virtual ultrasound image of the portion of the patient; displaying the holographic needle guide on the patient based upon the selection of the reference point in the virtual ultrasound image of the portion of the patient; and percutaneously inserting the needle along a trajectory of the holographic needle guide.

15. The method of claim 14, further comprising a step of providing the virtual ultrasound image of a portion of the patient using an ultrasound probe in one of real time and as a prerecorded image.

16. The method of claim 14, further comprising a step of adjusting an angle of trajectory associated with the holographic needle guide after displaying the holographic needle guide on the patient.

17. The method of claim 16 further comprising a step of stamping the holographic needle guide in a desired position after the adjusting the angle of trajectory associated with the holographic needle guide.

18. The method of claim 14, further comprising a step of tracking a position of the needle in comparison to the trajectory of the holographic needle guide after inserting the needle along the holographic needle guide.

19. The method of claim 14, further comprising alerting the user through the augmented reality display of a deviation from a predetermined threshold of variance of a position of the needle in relation to the trajectory of the holographic needle guide after inserting the needle along the holographic needle guide.

20. The method of claim 14, further comprising a step of providing a physical device for displaying a visual cue on the patient to indicate a needle insertion point after displaying the holographic needle guide.

Description

DRAWINGS

[0023] The drawings described herein are for illustrative purposes only of selected embodiments and not all possible implementations, and are not intended to limit the scope of the present disclosure.

[0024] FIG. 1 is a perspective view of the holographic augmented reality ultrasound needle guide system, in operation, depicting an augmented virtual window and an augmented virtual ultrasound projection showing a virtual ultrasound scan visible through a headset display, according to one embodiment of the present disclosure;

[0025] FIG. 2 is a perspective view of the system shown in FIG. 1, and depicting the user selecting a reference point on the augmented virtual window, according to one embodiment of the present disclosure;

[0026] FIG. 3 is a perspective view of the system shown in FIGS. 1 and 2, and depicting the holographic needle guide being displayed and the user adjusting the trajectory of the holographic needle guide, according to one embodiment of the present disclosure;

[0027] FIG. 4 is a perspective view of the system shown in FIGS. 1-3, depicting the user inserting a needle along the holographic needle guide, according to one embodiment of the present disclosure;

[0028] FIG. 5 is an enlarged perspective view of the user of the system taken at callout 5 in FIG. 4, shown from the point of view of the user wearing the headset display, and inserting the needle along the holographic needle guide, according to one embodiment of the present disclosure;

[0029] FIG. 6 is a schematic view of the system shown in FIGS. 1-5, according to another embodiment of the present disclosure;

[0030] FIG. 7 is a front elevational view of the system shown in FIGS. 1-6, depicting the system disposed on a movable cart, according to a further embodiment of the present disclosure;

[0031] FIG. 8 is a schematic view of an ultrasound scan, and depicting the ultrasound scan with the reference point selected and displaying a virtual needle insertion point, according to one embodiment of the present disclosure;

[0032] FIG. 9A is a top plan view of a needle insertion guide for determining a desired needle insertion point, and depicting a physical template with holes disposed adjacent to an ultrasound probe for use with the needle insertion guide, according to one embodiment of the present disclosure;

[0033] FIG. 9B is a side elevational view of a needle insertion guide for determining the desired needle insertion point, and depicting a physical template with measurement marks disposed adjacent to the ultrasound probe for use with the needle insertion guide, according to another embodiment of the present disclosure;

[0034] FIGS. 10A and 10B are schematic views illustrating a method for generating and moving the needle insertion guide according to the present disclosure;

[0035] FIGS. 11A, 11B, and 11C are fragmentary perspective views illustrating a stepwise flow for needle guide targeting using a heads-up display method;

[0036] FIGS. 12A, 12B, and 12C are fragmentary perspective views illustrating a stepwise flow for needle guide targeting using a flashlight display method; and

[0037] FIG. 13 is a flowchart illustrating a method for using the system in FIGS. 1-12C, according to various embodiments of the present disclosure.

DETAILED DESCRIPTION

[0038] The following description of technology is merely exemplary in nature of the subject matter, manufacture and use of one or more inventions, and is not intended to limit the scope, application, or uses of any specific invention claimed in this application or in such other applications as may be filed claiming priority to this application, or patents issuing therefrom. Regarding methods disclosed, the order of the steps presented is exemplary in nature, and thus, the order of the steps can be different in various embodiments, including where certain steps can be simultaneously performed. “A” and “an” as used herein indicate “at least one” of the item is present; a plurality of such items may be present, when possible. Except where otherwise expressly indicated, all numerical quantities in this description are to be understood as modified by the word “about” and all geometric and spatial descriptors are to be understood as modified by the word “substantially” in describing the broadest scope of the technology. “About” when applied to numerical values indicates that the calculation or the measurement allows some slight imprecision in the value (with some approach to exactness in the value; approximately or reasonably close to the value; nearly). If, for some reason, the imprecision provided by “about” and/or “substantially” is not otherwise understood in the art with this ordinary meaning, then “about” and/or “substantially” as used herein indicates at least variations that may arise from ordinary methods of measuring or using such parameters.

[0039] Although the open-ended term “comprising,” as a synonym of non-restrictive terms such as including, containing, or having, is used herein to describe and claim embodiments of the present technology, embodiments may alternatively be described using more limiting terms such as “consisting of” or “consisting essentially of.” Thus, for any given embodiment reciting materials, components, or process steps, the present technology also specifically includes embodiments consisting of, or consisting essentially of, such materials, components, or process steps excluding additional materials, components or processes (for consisting of) and excluding additional materials, components or processes affecting the significant properties of the embodiment (for consisting essentially of), even though such additional materials, components or processes are not explicitly recited in this application. For example, recitation of a composition or process reciting elements A, B and C specifically envisions embodiments consisting of, and consisting essentially of, A, B and C, excluding an element D that may be recited in the art, even though element D is not explicitly described as being excluded herein.

[0040] As referred to herein, disclosures of ranges are, unless specified otherwise, inclusive of endpoints and include all distinct values and further divided ranges within the entire range. Thus, for example, a range of “from A to B” or “from about A to about B” is inclusive of A and of B. Disclosure of values and ranges of values for specific parameters (such as amounts, weight percentages, etc.) are not exclusive of other values and ranges of values useful herein. It is envisioned that two or more specific exemplified values for a given parameter may define endpoints for a range of values that may be claimed for the parameter. For example, if Parameter X is exemplified herein to have value A and also exemplified to have value Z, it is envisioned that Parameter X may have a range of values from about A to about Z. Similarly, it is envisioned that disclosure of two or more ranges of values for a parameter (whether such ranges are nested, overlapping or distinct) subsume all possible combination of ranges for the value that might be claimed using endpoints of the disclosed ranges. For example, if Parameter X is exemplified herein to have values in the range of 1-10, or 2-9, or 3-8, it is also envisioned that Parameter X may have other ranges of values including 1-9, 1-8, 1-3, 1-2, 2-10, 2-8, 2-3, 3-10, 3-9, and so on.

[0041] When an element or layer is referred to as being “on,” “engaged to,” “connected to,” or “coupled to” another element or layer, it may be directly on, engaged, connected or coupled to the other element or layer, or intervening elements or layers may be present. In contrast, when an element is referred to as being “directly on,” “directly engaged to,” “directly connected to” or “directly coupled to” another element or layer, there may be no intervening elements or layers present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., “between” versus “directly between,” “adjacent” versus “directly adjacent,” etc.). As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items.

[0042] Although the terms first, second, third, etc. may be used herein to describe various elements, components, regions, layers and/or sections, these elements, components, regions, layers and/or sections should not be limited by these terms. These terms may be only used to distinguish one element, component, region, layer or section from another region, layer or section. Terms such as “first,” “second,” and other numerical terms when used herein do not imply a sequence or order unless clearly indicated by the context. Thus, a first element, component, region, layer or section discussed below could be termed a second element, component, region, layer or section without departing from the teachings of the example embodiments.

[0043] Spatially relative terms, such as “inner,” “outer,” “beneath,” “below,” “lower,” “above,” “upper,” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. Spatially relative terms may be intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. For example, if the device in the FIGS. is turned over, elements described as “below” or “beneath” other elements or features would then be oriented “above” the other elements or features. Thus, the example term “below” can encompass both an orientation of above and below. The device may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein interpreted accordingly.

[0044] FIGS. 1-7 illustrate a system 100 for guiding percutaneous insertion of a needle 102 by a user 104 into a patient 106 for a medical procedure. The system 100 includes an augmented reality display 108. The augmented reality display 108 is configured to depict a virtual ultrasound image 110 of a portion of the patient 106 in a mode known as heads-up display or “HUD” mode. The augmented reality display 108 is also configured to depict a holographic needle guide 112 on the patient 106 based upon the selection of a reference point 114 in the virtual ultrasound image 110, in a mode known as “Flashlight” mode. The holographic needle guide 112 may be virtually depicted in the form of an elongate axis, tube, or cylinder, as non-limiting examples, each illustrating a preferred angle of trajectory for the physical instrument. Advantageously, the system 100 is cost-effective, minimizes medical waste by elimination of the need for a physical need guide and bracket from the medical procedure, and provides the practitioner with a full and unrestricted range of angle guidance for percutaneous surgical procedures.

[0045] In one example, the system 100 may further include a computer 116 having a processor (not shown) and a memory (not shown). The memory (not shown) may have non-transitory processor-executable instructions directing the augmented reality display 108 to generate and display or depict the holographic needle guide 112 on the patient 106 based upon selection of the reference point 114 in the virtual ultrasound image 110 of the portion of the patient 106. In particular, the processor-executable instructions may permit the computer 116 to be operated in accordance with the method 200 as shown in FIG. 13.

[0046] As shown in FIGS. 1-2, the augmented reality display 108 may be configured to depict the virtual ultrasound image 110 of the portion of the patient 106 as part of a virtual window 118 and/or as part of a virtual ultrasound projection 120 on the patient 106. In a more specific example, the augmented reality display 108 may include a headset display 122 wearable by the user 104 that is in communication with the computer 116. In an even more specific example, the computer 116 may be integrated into the headset display 122 wearable by the user 104. Even more specifically, the headset display 122 may be a Microsoft HoloLens® having a tracking system (e.g., inertial measurement unit), integrated CPU and holographic processing unit, camera, and holographic projection lenses, for example, as described in U.S. Patent Application Publication No. 2018/0303563 to West et al., the entire disclosure of which including definitions is hereby incorporated herein by reference. One skilled in the art may select other suitable displays within the scope of the present disclosure.

[0047] The virtual ultrasound projection 120 that is generated by the computer 116 and depicted on the patient 106 may be further defined as a virtual display of the virtual ultrasound image 110 disposed adjacent to an ultrasound probe 124. In operation, the virtual ultrasound projection 120 may be linked to the ultrasound probe 124 so that a position of the virtual ultrasound projection 120 follows a position of the ultrasound probe 124. For example, the ultrasound probe 124 may be provided with tracking means 125 (shown in FIGS. 6, 11A-11C, and 12A-12C) such as an optical tracking cube. Alternatively, the virtual ultrasound projection 120 may be stamped in a virtually locked position (not shown). More particularly, where the virtual ultrasound projection 120 is stamped in a virtually locked position (not shown), the computer 116 will not recognize the movements of the user 104 as instructions to adjust the position of the virtual ultrasound projection 120.

[0048] In a specific, non-limiting example, the virtual ultrasound projection 120 may be displayed directly above the ultrasound probe 124 in operation, for example, as shown in FIG. 2. Advantageously, the virtual ultrasound projection 120 on the patient 106 may allow the user 104 to continue viewing the patient 106 while monitoring the virtual ultrasound image 110.

[0049] With continued reference to FIG. 2, the virtual ultrasound image 110 may be selectable by the user 104 to identify the reference point 114. In a more particular example, for example, as shown in FIG. 6, the virtual ultrasound image 110 may also be selectable by a remote user 126 to identify the reference point 114. In an even more particular example, the remote user 126 may be located at a different site (not shown) relative to the site of the user 104 guiding the percutaneous insertion of the needle 102 into the patient 106.

[0050] As shown in FIG. 3, an angle or trajectory 128 of the holographic needle guide 112 is adjustable by the user 104. In a specific example, the trajectory 128 of the holographic needle guide 112 may also be adjustable by the remote user 126. In an even more specific example, the remote user 126 may be located at a different site (not shown) from the user 104 guiding the percutaneous insertion of the needle into the patient 106. Advantageously, the user 104 and/or the remote user 126 may adjust the trajectory 128 of the holographic needle guide 112 to provide a less invasive path for the needle 102. Desirably, other nontargeted anatomical structures 130 may be more accurately and efficiently avoided by adjusting the trajectory 128 of the holographic needle guide 112.

[0051] In a particular instance, the computer 116 may be configured to define a modality or setting 132, 134 for selecting the trajectory 128 of the holographic needle guide 112, either automatically or manually within the scope of the present disclosure. In a more particular instance, the setting 132, 134 may be selected from a group consisting of an in-plane modality 132 (shown as a substantially vertical orientation in FIG. 3), an out-of-plane modality 134 (shown as the orientation set apart from the substantially vertical orientation by the angle of trajectory 128 in FIG. 3), a free hand modality (not shown), and combinations thereof.

[0052] The settings 132, 134 are based on the angle of trajectory 128 of the holographic needle guide 112 in comparison to a plane 136 associated with the patient 106. In one non-limiting example, as shown in FIGS. 3, 10A, and 10B, the plane 136 of the patient 106 may be substantially horizontal where the patient 106 is lying on an operating table, i.e., parallel with the surface of the table. The in-plane modality 132 may be described as orienting the trajectory 128 of the holographic needle guide 112 to be substantially perpendicular to the plane 136 of the patient 106. The out-of-plane modality 134 may be described as automatically orienting the trajectory 128 of the holographic needle guide 112 to a predetermined or desired angle other than the substantially perpendicular angle of the in-plane modality 132. The free-hand modality (not shown) may be used where the computer 116 does not automatically orient the trajectory 128 of the holographic needle guide 112 to a desired angle. Instead, the free-hand modality (not shown) relies on the user 104 to freely select a desired orientation for the trajectory 128 of the holographic needle guide 112.

[0053] In an even more particular instance, the holographic needle guide 112 may be depicted as a cylinder- or rod-shaped structure. The holographic needle guide 112 may depend from the selected reference point 114 and extend outwardly from the patient 106 to or through an external point 115 (shown in FIGS. 10A and 10B). The external point 115 may be a point on the plane 136, for example, such as a point on a periphery of a circle or ellipse on the plane 136 that is approximately centered on a location of the ultrasound probe 124. The external point 115 may be selected by any suitable method, including automatic selection based on algorithms configured to generate an optimum approach angle for the needle 102, or by manual selection by the user 104.

[0054] In operation, the user 104 may select the holographic needle guide 112 by grasping, pinching, tapping, and/or holding the holographic needle guide 112. While grasping, pinching, and/or holding the holographic needle guide 112, the user 104 may adjust the trajectory 128 of the holographic needle guide 112 by moving their hand with the holographic needle guide 112 to a desired position. The movement of the holographic needle guide 112 may be displayed as an arc, depending from the selected reference point 114. As shown in FIGS. 10A-10B, the movement of the holographic needle guide 112 may be correlated with a position of the ultrasound probe 124. The free-hand modality (not shown) may be further adjustable on a three dimensional setting, allowing arcs to be made in a globular pattern around the selected reference point 114. Further, the holographic needle guide 112 may be stamped in a virtually locked position (not shown). More particularly, where the holographic needle guide 112 is stamped in a virtually locked position (not shown), the computer 116 will not recognize the movements of the user 104 as instructions to adjust the position of the holographic needle guide 112. In a most particular instance, the computer 116 may have the in-plane modality 132 as a default setting. Advantageously, the user 104 may select a desirable setting 132, 134 based on the type of operation being conducted and the anatomical structures 130 of the patient 106 to more efficiently set the trajectory 128 of the holographic needle guide 112. A skilled artisan may select other suitable modalities for setting the trajectory 128 of the holographic needle guide 112, within the scope of the present disclosure.

[0055] As shown in FIGS. 1-4 and 6, the system 100 may further include the ultrasound probe 124. In a particular example, the system 100 may be configured to obtain the virtual ultrasound image 110 of the portion of the patient 106 from the ultrasound probe 124. In a more particular example, the system 100 may be configured to obtain the virtual ultrasound image 110 of the portion of the patient 106 from the ultrasound probe 124 in real time. In an alternative particular example, the virtual ultrasound image 110 of the portion of the patient 106 may be prerecorded.

[0056] In a specific example, the system 100 may also include a robotic arm (not shown). The robotic arm (not shown) may be configured to hold each of the ultrasound probe 124 and the needle 102. In a more specific example, the remote user 126 may be able to move the robotic arm (not shown) by using the computer 116. In an even more specific example, the remote user 126 may be located at a different site (not shown) from the user 104 moving the robotic arm (not shown) to perform the percutaneous insertion of the needle 102 into the patient 106. One skilled in the art may select other suitable methods of remotely performing the percutaneous insertion of the needle 102 into the patient 106, within the scope of the present disclosure.

[0057] In a specific example, the system 100 may include a tracking means (shown in FIGS. 11A-11C as 135). The tracking means may be configured to provide enhanced visualization of the anatomical structures 130 of the patient 106 and the needle 102. The tracking means may be placed on the patient 106, or on the needle 102, or both. The tracking means (not shown) may be an infrared marker (not shown), an electro-magnetic tracker (not shown), an image or model tracker (not shown), and/or an RFID tracker (not shown). As a non-limiting example, the electromagnetic tracking means (not shown) may be provided by the Aurora® tracking system, commercially available from Northern Digital Inc. As another non-limiting example, the infrared marker tracking means (not shown) may be employed with the Stylus XR® tracking system commercially available from Holo-Light GmbH. A non-limiting example of the image or model tracking means (not shown) may include the VisionLib™ tracking system commercially available from Visometry GmbH. Additionally, a non-limiting example of the RFID tracking means (not shown) may be employed with autoclavable RFID tags such as Xerafy® tags, commercially available from Xerafy Singapore Pte Ltd.

[0058] With reference to FIGS. 11A-11C, the tracking means 135 of the system 100 may include at least one optical tracking marker disposed on the patient. The optical tracking marker is configured to track the position of the body of the patient. In addition, the optical tracking marker may be further configured to spatially anchor operating information for view by the user through the headset display 108, 122. For example, where the operation projection is an ultrasound plane, the ultrasound plane may be spatially anchored to the body of the patient via the optical tracking marker. Desirably, this permits the practitioner to put the untracked instrument or needle 102 aside, while allowing the ultrasound plane to remain anchored to the body of the patient.

[0059] Nonlimiting examples of the optical tracking marker include passive markers and active markers. Passive markers may consist of retro-reflective material, which reflects incoming infrared light. Active markers may consist of infrared light emitting diodes. However, it should be appreciated that a skilled artisan may employ other types of optical tracking markers within the scope of this disclosure.

[0060] Referring now to FIG. 5, where the system 100 is provided with the tracking means, the augmented reality display 108 may also be configured to depict a holographic error bar 137. The holographic error bar 137 may be further configured to alert the user 104 of a deviation from a predetermined threshold of variance of a position of the needle 102 in relation to the trajectory 128 of the holographic needle guide 112. Non-limiting examples of the alert may include a visual color change, an auditory sound, a visual signal, and/or a vibration. In a particular example, the display of the holographic needle guide 112 on the patient 106 may include a minimum range (not shown) and a maximum range (not shown) dependent upon a physical characteristic of the needle 102. As a non-limiting example, the holographic needle guide 112 may be adjustable and configured to depict a physical length and a diameter of the needle 102 being inserted into the patient 106. Advantageously, by providing the holographic error bars 137 and the physical limitations of the needle 102 in the display of the holographic needle guide 112, the user 104 may more accurately, expeditiously, and confidently perform the surgery. A skilled artisan may use other methods of identifying and alerting a user 104 of a deviation from a predetermined threshold of variance of a position of the needle 102 in relation to the trajectory 128 of the holographic needle guide 112, within the scope of the present disclosure.

[0061] As shown in FIGS. 8, 9A, and 9B, the system 100 may include a needle insertion guide 138 for use in conjunction with the holographic needed guide 112 as described herein. The needle insertion guide 138 may be a physical device providing a visual cue on the patient 106. The needle insertion guide 138 may be configured to indicate a desirable or a predetermined needle insertion point 140, 148. In a specific example, as shown in FIGS. 9A and 9B, the needle insertion guide 138 may be a physical template 142 placed on the patient 106 near a location of the holographic needed guide 112. In a more specific example, the physical template 142 may include a bar 144 with reference markings (shown in FIG. 9B) that define adjacent to the reference markings the possible insertion points 140, or a bar 145 with a plurality of holes 146 arranged in a linear row (shown in FIG. 9A) inside of which define the possible insertion points 148, in order to give the user a further visual cue of the desirable needle insertion points 140, 148 to achieve a desired trajectory (not shown) of the needle 102 in conjunction with the holographic needed guide 112.

[0062] In an alternative example, as shown in FIG. 8, the needle insertion guide 138 may be disposed on or attached to the ultrasound probe 124. Where the needle insertion guide 138 is disposed on the ultrasound probe 124, the needle insertion guide 138 may be configured to provide a visual cue of the intended needle insertion point on the patient 106 and corresponding with the holographic needle guide 112.

[0063] As shown in FIG. 6, various components of the system 100 may be interconnected in various ways. Each of the ultrasound probe 124, the needle 102, and the needle insertion guide 138 may be in communication with the computer 116. The computer 116 may also be in communication with the augmented reality display 108. The computer 116 may also deliver and receive data from a remote user 126, for example, thorough a terminal or external computer of the remote user 126 in communication with the computer 116 over a wide area network such as the Internet. The communication may be provided through suitable wired or wireless technology means. Internet connected devices (not shown) may also be utilized to further provide the aforementioned communications. A skilled artisan may select other suitable methods to interconnect and provide communications within the system 100, within the scope of the present disclosure.

[0064] As shown in FIG. 7, the various components of the system 100 may be provided together on a movable cart 150. The cart 150 may include each of the computer 116, the augmented reality display 108, and the ultrasound probe 124. The cart 150 may store the augmented reality display 108 in a case 152 for enhanced protection from damage. Where the computer 116 is provided on the cart 150, the computer 116 may also include an external server 154 and an external wireless router 156 with which the computer 16 is in communication. Advantageously, the cart 150 may more easily permit a user 104 to move the system 100 to a location (not shown) of the patient 106 for the medical procedure.

[0065] As shown in FIG. 13, the present technology includes methods 200 of using the holographic augmented reality ultrasound needle guide systems 100 described herein. The method 200 may include a step 202 of providing an augmented reality display 108. The augmented reality display 108 may be configured to depict a virtual ultrasound image 110 of a portion of the patient 106. The augmented reality display 108 may also be configured to depict a holographic needle guide 112 on the patient 106 based upon the selection of a reference point 114 on the virtual ultrasound image 110. The method 200 may also include a step 204 of providing the virtual ultrasound image 110 of a portion of the patient 106 using an ultrasound probe 124 in real time. Desirably, the real time imaging provided by the ultrasound probe 124 may enable more accurate visualization of the anatomical structures 130 of the patient 106. Alternatively, the method 200 may include a step 206 of providing a prerecorded virtual ultrasound image 110. Advantageously, where the virtual ultrasound image 110 is prerecorded, the user 104 may not be required to hold the ultrasound probe 124 while also adjusting the trajectory 128 of the holographic needle guide 112.

[0066] The method 200 may include a step 208 of selecting the reference point 114 in the virtual ultrasound image 110 of the portion of the patient 106. Subsequently, the method 200 may include a step 210 of displaying the holographic needle guide 112 on the patient 106 based upon the selection of the reference point 114 on the virtual ultrasound image 110 of the portion of the patient 106.

[0067] With continued reference to FIG. 13, the method 200 may further include a step 212 of adjusting the angle of trajectory 128 associated with the holographic needle guide 112 after displaying the holographic needle guide 112 on the patient 106. In a particular example, the computer 116 may include a setting 132, 134 for automatically selecting the trajectory 128 of the holographic needle guide 112. The setting 132, 134 may be selected from an in-plane modality 132, an out-of-plane modality 134, and/or a free hand modality (not shown).

[0068] As shown in FIG. 13, the method 200 may include a step 214 of stamping the holographic needle guide 112 in a desired position after adjusting the angle of trajectory 128 associated with the holographic needle guide 112. Where the user 104 may need both of their hands when performing ultrasound guided needle procedures, the holographic needle guide 112 may be stamped in a virtually locked position (not shown). More particularly, where the holographic needle guide 112 is stamped in a virtually locked position (not shown), the computer 116 will not recognize the movements of the user 104 as instructions to adjust the trajectory 128 of the holographic needle guide 112. In an even more particular example, the computer 116 may allow each of the reference point 114, the holographic needle guide 112, the virtual ultrasound projection 120, and combinations thereof to be stamped in virtually locked positions (not shown). Advantageously, the stamping feature affords the user 104 to put the needle 102 or the ultrasound probe 124 down and then come back and know where they wanted to insert the needle 102 based on holographic needle guide 112 and the reference point 114.

[0069] As further shown in FIG. 13, the method 200 may include a step 216 of displaying a needle insertion guide 138 on the patient 106 to indicate a needle insertion point 140, 148 after displaying the holographic needle guide 112. Then, the method 200 may include a step 218 of tracking a position of the needle 102 in comparison to the trajectory 128 of the holographic needle guide 112 after inserting the needle 102 along the holographic needle guide 112. As shown in FIG. 5, where the position of the needle 102 is tracked, the method 200 may have another step 220 of alerting the user 104 through the augmented reality display 108 of a deviation from a predetermined threshold of variance of a position of the needle 102 in relation to the trajectory 128 of the holographic needle guide 112 after inserting the needle 102 along the holographic needle guide 112. Afterwards, the method 200 may include a step 222 of percutaneously inserting the needle along a trajectory 128 of the holographic needle guide 112.

Examples

[0070] The system 100 and method 200 of the present disclosure can further be described as a unique combination of ultrasound technology with holography, which can be further illustrated in view of the following non-limiting examples described with general reference to FIGS. 10A-10B, 11A-11C, and 12A-12C.

[0071] Needle Guide Object:

[0072] In a particular example, the needle guide is defined as being a line between two points. The visual target is located at the distal tip of the line. There is also a ghosted geometry at the proximal end of the line, resembling the fixture to connect the physical needle to the physical guide.

[0073] Interactions:

[0074] In certain examples, the target may not be an interactable. The fixture geometry may be interactable and may perform translation-only (i.e., no rotation or scaling). The fixture may support both far and near interactions as defined by the Microsoft Mixed Reality Toolkit (MRTK). For “far interaction,” the user may select the fixture by pointing a hand ray at the handle and then performing a pinch-and-hold. While pinched, the fixture may be transformed through hand gestures. For “near interaction,” the system may support direct manipulation of the fixture via existing patterns in the MRTK. Interaction states of the fixture may follow the MRTK patterns, for example, as described in the MRTK Interactable Feature Overview (found at https://microsoft.gihub.io/MixedRealityToolkit-Unity/Documentation/README_Interactable.html, last accessed Nov. 27, 2020). The interact states include: i) Default (normal, observation); ii) Focused (targeted, hover). e.g. the fixture may light up; and iii) Active (pressed) (grabbed vs. ungrabbed)—fixture changes color to blue.

[0075] Visual Specification of Needle Guide:

[0076] In particular examples, the line may be rendered as a cylinder without end caps. This could also include a native line, if issues arise with rendering a cylinder. The cylinder's material may be a whiteish x-ray shader. There may be front-facing rendering-only on the cylinder. The target may be a billboard shape. The fixture may be a translucent, green, x-ray shader, resembling the real-world plastic object. The scale of each of the needle guide features can be as follows: i) target, one and half (1.5) cm diameter; ii) cylinder, a quarter (0.25) cm diameter; and iii) fixture, about three (3) cm in length.

[0077] Needle Guide Targeting:

[0078] In yet other examples, a near cursor on the user's index finger may be the default MRTK cursor. Targeting may be possible on both the heads-up display (HUD), as shown in FIGS. 11A-11C, and the flashlight ultrasound planes, as shown in FIGS. 12A-12C. Targeting may support eye-gaze, hand-ray (far) and direct manipulation (near). When the HUD or flashlight plane is targeted/hovered, the cursor may visually change to targeting/hover state. Flashlight mode can be toggled on and off. The HUD can be toggled on and off.

[0079] The user may then commit a target location by one of the following: i) far interaction, where the user extends arm, aims hand-ray, and either AirTaps or says the “Set target” voice command; or ii) near interaction, where the user presses the HUD or flashlight ultrasound plane directly with their index finger.

[0080] Upon committing the target location, the following may happen. First, a visual indication may occur on the ultrasound plane (either HUD or flashlight or both simultaneously) making it clear that an action was received (e.g. shockwave). Second, the committed target visual on the HUD or flashlight plane may fade away as soon as it's committed. Third, the target location on the HUD or flashlight plane may be transformed to world space. Fourth, the needle guide may appear in the scene. Fifth, the distal tip of the needle guide may be initialized at the target location. Sixth, the same visual indicator (e.g. shockwave) may occur when the target is initialized in 3D space. Seventh, the direction of the needle guide may be determined by the ultrasound plane and may be “in-plane” when initialized. Eighth, the proximal fixture on the needle guide's cylindrical shaft may default to about 4 cm “above” the transducer probe (i.e., outside the patient) so the user can easily grab it.

[0081] The user may be permitted to “re-target” at any time on the HUD or flashlight planes. If a target is committed on the HUD it can automatically be populated on the flashlight mode and vice versa. The user can lock and stamp in a target and holographic needle guide in physical space and reference that relative to the optical tracking marker on the patient (shown in FIGS. 11A-11C and 12A-12C) to account for movement between the patient and ultrasound probe. The user can lock and stamp the position of an ultrasound probe in physical space and reference that relative to the optical tracking and targeting and holographic needle guide by referencing an optical maker on the patient

[0082] Advantageously, the ultrasound needle guide system 100 and method 200 are cost-effective, minimizes medical waste, and provides the practitioner with a full and unrestricted range of angle guidance for percutaneous surgical procedures. Importantly, the system 100 and related method 200 involves holographic augmented reality and can be used with any type of ultrasound transducer.

[0083] Example embodiments are provided so that this disclosure will be thorough, and will fully convey the scope to those who are skilled in the art. Numerous specific details are set forth such as examples of specific components, devices, and methods, to provide a thorough understanding of embodiments of the present disclosure. It will be apparent to those skilled in the art that specific details need not be employed, that example embodiments may be embodied in many different forms, and that neither should be construed to limit the scope of the disclosure. In some example embodiments, well-known processes, well-known device structures, and well-known technologies are not described in detail. Equivalent changes, modifications and variations of some embodiments, materials, compositions and methods can be made within the scope of the present technology, with substantially similar results.