System and method for location determination using a mixed reality device and multiple imaging cameras

Abstract

A system and method for determining a location for a surgical jig in a surgical procedure includes providing a mixed reality headset, a 3D spatial mapping camera, an infrared or stereotactic camera, and a computer system configured to transfer data to and from the mixed reality headset and the 3D spatial mapping camera. The system and method also include attaching a jig to a bone, mapping the bone and jig using the 3D spatial mapping camera, and then identifying a location for the surgical procedure using the computer system. Then the system and method use the mixed reality headset to provide a visualization of the location for the surgical procedure.

Claims

1. A method for determining a location for a surgical jig in a surgical procedure, comprising: providing a mixed reality headset; providing a 3D spatial mapping camera; providing an imaging camera comprising one of an infrared camera or a stereotactic camera; providing a computer system connected to each of the mixed reality headset, 3D spatial mapping camera and imaging camera, the computer system configured to send display data to the mixed reality headset, and receive imaging data from the 3D spatial mapping camera and the imaging camera; creating a first digital map of a bone of a patient on which the surgical procedure is to be performed using first imaging data from the 3D spatial mapping camera combined with first coordinates data from the imaging camera; creating a second digital map of a physical jig to be used in the surgical procedure using second imaging data from the 3D spatial mapping camera combined with second coordinates data from the imaging camera; using the first digital map and the second digital map to determine a location for the surgical procedure on the bone and a position of the physical jig relative to the bone and generating a digital image of a virtual representation of the bone with a virtual representation of the jig superimposed over the virtual representation of the bone; displaying the virtual representation of the bone and jig on the mixed reality headset to provide a virtual visualization for indicating a location for placement of the physical jig relative to the bone; and attaching the physical jig to the bone at the location wherein the visualization of the location for the surgical procedure is indicated.

2. The method of claim 1, further comprising: manipulating the location of the virtual representation of the jig as viewed in the mixed reality headset, with respect to the virtual representation of the bone.

3. The method of claim 2, further comprising: fixing the location of the virtual representation of the jig as viewed in the mixed reality headset, with respect to the visual representation bone; and manipulating the location of the jig with respect to the bone, to substantially match the location of the visual representation jig with respect to the visual representation bone.

4. The method of claim 1, wherein the jig includes a marker configured to be scanned by the mixed reality headset to provide data relating to the location of the jig.

5. The method of claim 4, further comprising: wherein the imaging camera is an infrared camera; scanning the marker of the jig using the infrared camera; identifying a position of the jig with respect to the bone based, at least in part, on the data provided by scanning the marker.

6. The method of claim 5, wherein the marker is a fiducial marker.

7. The method of claim 6, wherein the infrared camera and the spatial mapping camera are placed in location separate from one another.

8. The method of claim 4, further comprising: wherein the imaging camera is a stereotactic camera; scanning the marker of the jig using the stereotactic camera; identifying a position of the jig with respect to the bone based, at least in part, on the data provided by scanning the marker.

9. The method of claim 8, wherein the stereotactic camera and the spatial mapping camera are placed in location separate from one another.

10. The method of claim 1, wherein the imaging camera and the spatial mapping camera are formed as a combined unitary structure.

11. The method of claim 1, wherein at least one of the spatial mapping camera and the imaging camera is positioned within a cavity of a patient having the surgical procedure, and wherein the spatial mapping camera and the imaging camera provide imaging information for the surgical procedure from within the patient.

12. The method of claim 1, wherein the imaging camera is an infrared camera.

13. The method of claim 1, wherein the imaging camera is a stereotactic camera.

14. The method of claim 1, wherein the step of identifying the location of the surgical procedure further comprises: identifying a location on the bone for the surgical procedure.

15. A method for determining a location for a surgical procedure, comprising: providing a mixed reality headset; providing a 3D spatial mapping camera; providing an imaging camera comprising one of an infrared camera or a stereotactic camera; providing a computer system connected to each of the mixed reality headset, 3D spatial mapping camera and imaging camera, the computer system configured to send display data to the mixed reality headset, and receive imaging data from the 3D spatial mapping camera and the imaging camera; creating a first digital map of a body part of a patient on which the surgical procedure is to be performed using first imaging data from the 3D spatial mapping camera combined with first coordinates data from the imaging camera; creating a second digital map representing a plurality of locations where surgical procedures are to be performed on the body part using second imaging data from the 3D spatial mapping camera combined with second coordinates data from the imaging camera; using the first digital map and the second digital map to determine positions of the plurality of locations relative to the body part and generating a digital image of a virtual representation of the body part with a virtual representation of the plurality of locations superimposed over the virtual representation of the body part; displaying the virtual representation of the body part and plurality of locations on the mixed reality headset to provide a virtual visualization for indicating a location for surgical procedure to be performed on the body part.

16. The method of claim 15, further comprising: manipulating the location of the virtual representation of the plurality of locations as viewed in the mixed reality headset, with respect to the visual representation bone.

17. The method of claim 16, further comprising: fixing the location of the virtual representation of the of the plurality of locations as viewed in the mixed reality headset, with respect to the virtual representation of the body part; and manipulating the location of the plurality of locations with respect to the body part, to substantially match the location of the virtual representation of the plurality of locations with respect to the virtual representation of the body part.

18. The method of claim 15, wherein the jig includes a marker configured to be scanned by the mixed reality headset to provide data relating to the location of the jig.

19. The method of claim 18, further comprising: wherein the imaging camera is a stereotactic camera; scanning the marker of the jig using the stereotactic camera; identifying a position of the jig with respect to the bone based, at least in part, on the data provided by scanning the marker.

20. The method of claim 18, further comprising: wherein the imaging camera is an infrared camera; scanning the marker of the jig using the infrared camera; identifying a position of the jig with respect to the body part based, at least in part, on the data provided by scanning the marker.

21. The method of claim 20, wherein the infrared camera and the spatial mapping camera are placed in location separate from one another.

22. The method of claim 15, further comprising: continuously generating the digital image, throughout the surgical procedure, using a continuously generated digital map generated by the 3D spatial mapping camera and a continuously generated digital map generated by the imaging camera.

23. The method of claim 15, wherein the imaging camera and the spatial mapping camera are placed in location separate from one another.

24. The method of claim 15, wherein the imaging camera and the spatial mapping camera are formed as a combined unitary structure.

25. The method of claim 15, wherein at least one of the spatial mapping camera and the imaging camera is positioned within a cavity of a patient having the surgical procedure, and wherein the spatial mapping camera and the imaging camera provide imaging information for the surgical procedure from within the patient.

26. The method of claim 15, wherein the step of identifying the location of the surgical procedure further comprises: identifying a location on the body part for the surgical procedure.

27. A method for determining a location for a surgical jig in a surgical procedure, comprising: providing a mixed reality headset; providing a 3D spatial mapping camera; providing a stereotactic camera; providing a computer system configured to transfer data to and from the mixed reality headset and the 3D spatial mapping camera; attaching a jig to a bone; mapping the bone and jig using the 3D spatial mapping camera; identifying coordinates of the location of the jig using the stereotactic camera; identifying a location for the surgical procedure using the computer system, which superimposes a mapping image from the 3D spatial mapping camera and a coordinates based image from the stereotactic camera; using the mixed reality headset to provide a visualization of the location for the surgical procedure; using the mixed reality headset to provide a virtual jig and a virtual bone, which are representations of the jig and bone; manipulating the location of the virtual jig as viewed in the mixed reality headset, with respect to the virtual bone; fixing the location of the virtual jig as viewed in the mixed reality headset, with respect to the virtual bone; wherein the jig includes a fiducial ball marker, wherein the fiducial ball is configured to be scanned by the stereotactic camera to provide data relating to the desired location of the jig; identifying a position of the jig with respect to the bone based, at least in part, on the data provided by scanning the fiducial ball marker; manipulating the location of the jig with respect to the bone, to substantially match the location of the virtual jig with respect to the virtual bone; and wherein the mixed reality headset is separate and not mechanically connected to the 3D spatial mapping camera and the stereotactic camera.

28. A method for determining a location for a surgical jig, comprising: providing a 3D spatial mapping camera; providing an infrared camera; providing a computer system configured to transfer data to and from the mixed reality headset, the 3D spatial mapping and the infrared camera; mapping a bone and a jig using the 3D spatial mapping camera; identifying coordinates of the location of the jig using the infrared camera; creating a digital map of the bone and the jig using the mapping of the jig and bone from the 3D spatial mapping camera combined with the coordinates of the location of the jig from the infrared camera; and attaching the jig to the bone.

29. The method of claim 28, further comprising: identifying a location for the surgical jig using the computer system, which superimposes a mapping image from the 3D spatial mapping camera and a coordinates based image from the infrared camera.

30. The method of claim 29, as part of a surgical procedure, further comprising: providing a mixed reality headset; and using the mixed reality headset to provide a visualization of the location for the surgical procedure.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) The features and advantages of the disclosure will become apparent from consideration of the subsequent detailed description presented in connection with the accompanying drawings in which:

(2) FIG. 1 is a schematic rendering of a mixed reality system of the present disclosure;

(3) FIG. 2 is a flow diagram illustrating a method of using the mixed reality system.

(4) FIG. 3 is a flow diagram illustrating another method of using the mixed reality system.

(5) FIG. 4 is a perspective view of another embodiment of a mixed reality system of the present disclosure;

(6) FIG. 5 is a schematic rendering of a view through a mixed reality display of a further embodiment of the present disclosure;

(7) FIG. 6 is a schematic view of a combined unitary camera structure;

(8) FIG. 7 is a front view of a user perspective of a mixed reality headset viewer of the present disclosure; and

(9) FIG. 8 is schematic rendering of another mixed reality system of the present disclosure.

DETAILED DESCRIPTION

(10) For the purposes of promoting an understanding of the principles in accordance with the disclosure, reference will now be made to the embodiments illustrated in the drawings and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of the disclosure is thereby intended. Any alterations and further modifications of the inventive features illustrated herein, and any additional applications of the principles of the disclosure as illustrated herein, which would normally occur to one skilled in the relevant art and having possession of this disclosure, are to be considered within the scope of the disclosure claimed.

(11) It must be noted that, as used in this specification and the appended claims, the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise.

(12) In describing and claiming the present disclosure, the following terminology will be used in accordance with the definitions set out below.

(13) As used herein, the terms “comprising,” “including,” “containing,” “characterized by,” and grammatical equivalents thereof are inclusive or open-ended terms that do not exclude additional, unrecited elements or method steps.

(14) As used herein, the terms “virtual,” and “hologram” are used interchangeably, and grammatical equivalents thereof are inclusive or open-ended terms that do not exclude additional, unrecited elements or method steps. These terms are used to describe visual representations of an actual physical device or element, having all, or mostly all, of the same visual characteristics of the physical device, including size and shape.

(15) Applicant has discovered a novel system and method for generating and using a virtual jig, or virtual instrument, in a surgical procedure, for example, in a knee or tibial implant procedure, or other desired surgical procedure.

(16) The phrase “virtual system,” as used herein, shall refer broadly to any system capable of generating or creating a simulated or virtual rendering or projection of physical or structural features identical or substantially identical to an actual physical device, instrument or other physical structure, as known in the art. A virtual system may also include a device, mechanism, or instrument capable of projecting or displaying the desired simulated or virtual rendering or projection of physical or structural features identical or substantially identical to an actual physical device. A virtual system may also enable a user to manipulate, move and/or modify the simulated or virtual rendering or projection.

(17) The phrase “mixed or augmented reality system,” as used herein, shall refer broadly to any system capable of generating or creating a simulated or virtual rendering or projection of physical or structural features identical or substantially identical to an actual physical device, instrument or other physical structure, as known in the art. A mixed or augmented reality system may also include a device, mechanism, or instrument capable of projecting or displaying the desired a simulated or virtual rendering or projection of physical or structural features identical or substantially identical to an actual physical device overlaid or concurrently with actual physical structures, mechanism or devices in reality, thus incorporating the virtual rendering or projection in real world settings with actual physical element. A mixed or augmented reality system may also enable a user to manipulate, move and/or modify the simulated or virtual rendering or projection.

(18) The phrase “mixed or augmented reality instrument,” as used herein, shall refer broadly to any device, mechanism or instrument used in a mixed or augmented reality system, including a device capable of generating or creating a simulated or virtual rendering or projection of physical or structural features identical or substantially identical to an actual physical device, instrument or other physical structure, as known in the art. A mixed or augmented reality instrument may also be capable of projecting or displaying the desired a simulated or virtual rendering or projection of physical or structural features identical or substantially identical to an actual physical device overlaid or displayed concurrently with actual physical structures, mechanism or devices in reality, thus incorporating the virtual rendering or projection in real world settings with actual physical elements. A mixed or augmented reality instrument may also enable a user to manipulate, move and/or modify the simulated or virtual rendering or projection.

(19) The phrase “holographic representation,” as used herein, shall refer broadly to a visualization or visual rendering or projection representing an actual physical device or element, having all, or mostly all, of the same visual characteristics of the corresponding physical device or element, including size and shape, as known in the art.

(20) Referring now to the drawings, FIG. 1, in a disclosed embodiment, illustrates a mixed or augmented system 100, which can be used to produce, or display, a desired mixed or augmented reality instrument, such as a virtual jig or cutting guide in a display to a surgeon or user, or stated another way, that is visible and manipulatable by a surgeon or user. The mixed or augmented reality system 100 may also enable a user to activate or deactivate, in full or in part, the virtual instrument or instruments, making a virtual instrument appear or disappear, as, for example, may be desired in a mixed reality assisted surgery.

(21) The mixed or augmented reality system 100 may include a mixed or augmented reality headset 102 which may include a transparent or mostly transparent viewer 104 which can be suspended or positioned in front of a user's eyes. The headset 102 may include a headband 106 attached to the viewer 104, which may be used to secure the headset 102 to a user's head 108, thereby securing the viewer 104 in place in front of the user's eyes.

(22) The transparent viewer 104 may be configured to project, or otherwise make viewable, on an interior surface of the viewer 104, a holographic image or images, such as a virtual device, for example, a virtual cutting guide, which may be positionally manipulated by the user, surgeon, third party or remote system, such as a remote computer system. The headset 102 may be configured to view holographic images or, alternatively, the holographic images may be turned off and the user wearing the headset 102 may be able to view the surrounding environment through the transparent viewer 104 without obstruction. As such, a user, such as a surgeon for example, can wear the mixed or augmented reality headset 102 and then can choose to activate a holographic image to aide in facilitating a surgical procedure and then shut off the holographic image in order to perform the surgical procedure in a visually un-obscured manner.

(23) One embodiment of the disclosed headset 102 utilizes a Microsoft Corporation product, known as the HoloLens® mixed or augmented reality system. Other suitable mixed or augmented reality systems for generating virtual images viewable by a user or surgeon may also be employed. Thus, the headset 102 may be a conventional “off the shelf” product with a built-in platform that enables all of the features described herein with respect to the headset 102. Furthermore, the headset 102, such as Microsoft's HoloLens® product, can be loaded or preloaded with all desired or required virtual instruments, including virtual jigs or surgical cutting guides, virtual drill bits, and/or a virtual target which can identify relative locations of a plurality of holes to be drilled by a surgeon to facilitate the fastening of a jig or other device onto a desired bone at the proper desired location, and any other desired virtual instruments or holograms. The Microsoft HoloLens® product and its capabilities and features, or any suitable mixed or augmented reality system, such as is described herein with respect to the headset 102, are known to those skilled in the art.

(24) The mixed reality system 100 may also include a computer or computer system 200 having enabling software to communicate with the headset 102, by both receiving information from the headset 102 and transmitting data and images to the headset 102. It is therefore to be understood, by way of the circuit diagram and dashed lines shown in FIG. 1, that headset 102 is electronically connected to the computer system 200 and a 3D spatial mapping camera 300 and another imaging camera or infrared camera 350. In alternative embodiments, the imaging camera 350 may be a stereotactic or stereotaxic device or camera, which may also be represented by reference numeral 350, as the included figures and drawings are schematic in nature and do not disclose the mechanical or electrical details of the infrared camera or the stereotactic devices, as those details are known by those skilled in the art. The 3D spatial mapping camera 300 and infrared or stereotactic camera 350 are electronically connected to the headset 102 and the computer system 200, as shown in the circuit diagram and dashed lines shown in FIG. 1. While the 3D spatial mapping camera 300 and infrared or stereotactic camera 350 may be electronically connected to the headset 102, the 3D spatial mapping camera 300 and infrared or stereo tactic camera 350 may be separate from and not mechanically connected to the headset 102.

(25) The mixed reality system 100 may also include a 3D spatial mapping camera 300 and an infrared or stereotactic camera 350. One embodiment of the disclosed spatial mapping camera 300 may be a product created and manufactured by Microsoft Corporation, known as the Azure Kinect®, or any suitable 3D spatial mapping camera capable of continuous 3D mapping and transition corresponding 3D images, such as bones, anatomy, or other desired 3D objects. The spatial mapping camera 300 may be a conventional “off the shelf” product with a built-in platform that enables all of the features described herein with respect to the spatial mapping camera 200. Furthermore, the spatial mapping camera 200, such as Microsoft's Azure Kinect® product, can be loaded or preloaded with all necessary software to enable wireless communication between the spatial mapping camera 300 and the computer system 200 and/or the headset 102. Microsoft's Azure Kinect® product and its capabilities and features, or any suitable 3D spatial mapping camera such as is described herein with respect to the spatial mapping camera 300, are known to those skilled in the art.

(26) The spatial mapping camera 300 may include sensor software development kits for low-level sensor and device access, body tracking software development kits for tracking bodies or objects in 3D, and speech cognitive services software development kits for enabling microphone access and cloud-based or wireless speech services.

(27) Additionally, the spatial mapping camera 300 may include the following features: depth camera access and mode control (a passive IR mode, plus wide and narrow field-of-view depth modes); RGB camera access and control (for example, exposure and white balance); a motion sensor (gyroscope and accelerometer) access; synchronized depth-RGB camera streaming with configurable delay between cameras; external device synchronization control with configurable delay offset between devices; camera frame meta-data access for image resolution, timestamp, etc.; and device calibration data access.

(28) The spatial mapping camera 300 may also include: software development kits that enable a viewer tool to monitor device data streams and configure different modes; a sensor recording tool and playback reader API that uses the Matroska container format, as known to those of ordinary skill in the field; a library and runtime to track bodies in 3D when used with the spatial mapping camera 300; an anatomically correct skeleton for each partial or full body; a unique identity for each body; ability to track bodies over time; and speech recognition and translation capabilities, such as, speech-to-text, speech translation and text-to-speech.

(29) In one disclosed embodiment, the imaging camera 350 comprises an infrared camera 350, which may be any suitable infrared camera capable of continuous infrared imaging, such as bones, anatomy, or other desired 3D objects. The infrared imaging camera 350 may be a conventional “off the shelf” product with a built-in platform that enables all of the features described herein with respect to the infrared camera 200. Furthermore, the infrared camera 350 can be loaded or preloaded with all necessary software to enable wireless communication between the infrared imaging camera 350 and the computer system 200 and/or the headset 102. The infrared camera 350 may be any suitable infrared camera such as is described herein with respect to the infrared imaging camera 350, are known to those skilled in the art. The infrared imaging camera 350, as known in the art, may provide or identify desired coordinates in a 3D space, thus providing accurate and detailed location determination in a desired 3D space.

(30) In another disclosed embodiment the imaging camera 350 comprises a stereotactic camera 350, which may be any suitable stereotactic camera capable of continuous stereotaxic imaging, such as bones, anatomy, or other desired 3D objects. The stereotactic imaging camera 350 may be a conventional “off the shelf” product with a built-in platform that enables all of the features described herein with respect to the stereotactic camera 200. Furthermore, the stereotactic imaging camera 350 can be loaded or preloaded with all necessary software to enable wireless communication between the stereotactic imaging camera 350 and the computer system 200 and/or the headset 102. The stereotactic imaging camera 350 may be any suitable stereotactic camera such as is described herein with respect to the stereotactic camera 350, are known to those skilled in the art. The stereotactic imaging camera 350, as known in the art, may provide or identify desired coordinates in a 3D space, thus providing accurate and detailed location determination in a desired 3D space.

(31) The headset 102, computer system 200, spatial mapping camera 300, and infrared or stereotactic imaging camera 350, may be programmed and configured to enable a surgeon 107 to see and manipulate a virtual, or holographic target or jig, or visual representation of a jig, with respect a patient's bone 400, anatomical, or any other desired location, which may receive a surgical implant. The headset 102, computer system 200 and spatial mapping camera 300 may communicate with one another via a local network connection, Wi-Fi, Bluetooth®, or any other known wireless communication signal.

(32) Specifically, the spatial mapping camera 300, which may be programed to communicate with the computer system 200 having enabling software, may utilize such enabling software to combine coordinate data or information from the infrared camera or sterotactic camera 350 with the imaging data from the spatial mapping camera 300 to digitally map the bone 400 and/or jig 500, or other desired anatomy, producing a first digital map, to help identify the proper location for fastening a jig, or other device, to the bone 400, prior to cutting the knee.

(33) The spatial mapping camera 300 differs from traditional imaging, such as an MRI, CT scan, x-ray or the like, in many ways. For example, the spatial mapping camera 300 may be mounted, fixedly or moveably, in an operating room, thus giving the most up-to-date mapping information possible. The spatial mapping camera 300 may also continuously map the surface and 3D contours of the bone 400 to provide the surgeon 107 with real-time feedback, imaging data, and information to make the proper drill holes, cuts, or other preparations on the bone 400, before, during, and/or after any given procedure.

(34) The computer system 200 with enabling software may utilize and overlay or superimpose the respective images, coordinates, or imaging information or display data from both the spatial mapping camera 300 and the infrared or stereotactic camera, to identify coordinates or key features of the bone 400 and or jig 500, or other desired anatomy, or may identify key features of a desired location for a surgical procedure, to help identify the proper location for fastening a jig, or other device, to the bone 400, prior to cutting the knee. The computer system 200 may then identify coordinates of the location of the jig 500 using the stereotactic camera 350 and identify a location for the surgical procedure by using the computer system 200 to superimpose a mapping image from the 3D spatial mapping camera 300 and a coordinates based image from the infrared or stereotactic camera 350, to produce a second digital map.

(35) The imaging data or information from the spatial mapping camera 300 may provide contour mapping of the surrounding bone 400 or other desired 3D space, while the infrared or stereotactic camera 350 can provide coordinates that more accurately identify exact or substantially exact locations in three dimensions. Thus, the combination of the superimposed images from the spatial mapping camera 300 and the infrared or stereotactic camera 350 can provide more accurate location information than either camera 300 or 350 alone.

(36) Additionally, the use of the combination of the spatial mapping camera 300 and the infrared or stereotactic camera 350 can enable the cameras 300 and 350 to be placed or fixed at a location that is further from the surgical procedure while still maintaining accurate imaging. For example, a spatial mapping camera, when operating alone, may need to be placed or fixed at a location that is about 60-80 cm from the surgical procedure to maintain accuracy. However, the combination of the spatial mapping camera 300 and infrared or stereotactic camera 350 may be placed or fixed at a location that is about 1 meter away from the surgical procedure, thus providing a less cluttered surgical workspace that enables a surgeon more working freedom during the procedure and reduces the potential for contamination, with the cameras 300 and 350 a greater distance from the surgical procedure.

(37) The mixed reality system 100 may also include an alignment jig 500 that can be secured to the exposed bone 400, or other desired anatomy. The jig 500 may include a first marker 502 which may be attached to the jig 500 at a fixed location on the bone 400. The first marker 502 may be a fiducial ball or other small identifier. Due to the high accuracy and resolution of the infrared or stereotactic camera 350, a small fiducial-type marker may be all that is necessary to clearly and accurately identify a desired location for the jig 500 on the bone 400, or a desired location of the surgical procedure on a bone 400 or other part of the anatomy. The jig 500 may also include a second marker 510 that may be moveable with respect to the jig 500 and the bone 400. The second marker 510 may also include a fiducial-type marker. The size and features of a fiducial ball or fiducial marker are well known in the art, and are particularly advantageous during surgical procedures due to the size, for example 3.5 cm in diameter, or less, or more, if desired.

(38) FIG. 2 illustrates a flow diagram, used to show the method of using the mixed reality system 100 according to an exemplary embodiment. The system 100 in a first step S10, provides the computer system 200 connected to each of the mixed reality headset 102, 3D spatial mapping camera 300 and imaging camera 350, the computer system 200 configured to send display data to the mixed reality headset 102, and receive imaging data from the 3D spatial mapping camera 300 and the imaging camera 350.

(39) The next step S11 includes creating a first digital map of the bone 400 of a patient on which the surgical procedure is to be performed using first imaging data from the 3D spatial mapping camera 300 combined with first coordinates data from the imaging camera 350. Then in the next step S12, the system 100 creates a second digital map of a physical jig 500 to be used in the surgical procedure using second imaging data from the 3D spatial mapping camera 300 combined with second coordinates data from the imaging camera 350.

(40) Then in the next step S13 the system 100, uses the first digital map and the second digital map to determine a location for the surgical procedure on the bone 400 and a position of the physical jig 400 relative to the bone 500 and generates a digital image of a virtual representation of the bone 400 with a virtual representation of the jig superimposed over the virtual representation of the bone.

(41) The system 100 in the next step S14 displays the virtual representation of the bone and jig on the mixed reality headset to provide a virtual visualization for indicating a location for placement of the physical jig relative to the bone, and then in the final step S15, the surgeon can attach the physical jig 500 to the bone 400 at the location wherein the visualization of the location for the surgical procedure is indicated.

(42) FIG. 3 illustrates a flow diagram, used to show the method of using the mixed reality system 100 according to another exemplary embodiment. The system 100 in a first step S20 creates a first digital map of a body part of a patient on which the surgical procedure is to be performed using first imaging data from the 3D spatial mapping camera 300 combined with first coordinates data from the imaging camera 350. The in the next step S21, the system 100 creates a second digital map representing a plurality of locations where surgical procedures are to be performed on the body part using second imaging data from the 3D spatial mapping camera 300 combined with second coordinates data from the imaging camera 350.

(43) In a next step S22 the system 100 uses the first digital map and the second digital map to determine positions of the plurality of locations relative to the body part and generating a digital image of a virtual representation of the body part with a virtual representation of the plurality of locations superimposed over the virtual representation of the body part.

(44) And then the system in a final step S23, can display the virtual representation of the body part and plurality of locations on the mixed reality headset to provide a virtual visualization for indicating a location for surgical procedure to be performed on the body part.

(45) FIG. 3 illustrates a schematic view of a perspective through the viewer 104 of the mixed reality headset 102, representing the view of a wearer of the mixed reality headset 100 during a procedure. As shown, the view can display a visualization of a jig or a virtual jig 550 overlaid on the actual physical bone 400, enabling the wearer to see or view a virtual jig 550 simultaneously with the bone 400, after utilizing the system methodology of FIGS. 2 or 3.

(46) In another embodiment, as shown in FIG. 4, the headset 102, the spatial mapping camera 300 and the infrared or stereotactic camera 350 may utilize a physical calibration tool 700 to calibrate and align 3D coordinates with the 3D contours of the exposed bone 400 or anatomy. The calibration tool 700 may include a specially designed calibration pattern 702 that may be scanned by the headset 102 and/or the spatial mapping camera 300, and/or the infrared or stereotactic camera 350, which can then send the corresponding calibration data to the computer system 200 which can then use the calibration data to provide accurate coordinates to the headset 102 to identify where to drill, cut, or otherwise prepare the bone 400.

(47) Using the mixed reality system 100, a surgeon or user may perform a surgical procedure by first exposing the bone 400, or other desired anatomy. The spatial mapping camera 300 may then continuously 3D spatially map the exposed bone 400, and send the corresponding image data or mapping information to the computer system 200. In alternative embodiments, multiple spatial mapping cameras 300 may be used in unison, to provide more accurate and expansive imaging.

(48) A surgeon or user may then substantially simultaneously, or before, or after, the 3D mapping, identify coordinates on the exposed bone 400 identifying a desired placement of the jig 500 or surgical procedure by using the infrared or stereotactic camera 350, and send the corresponding coordinate information to the computer system 200. In alternative embodiments, multiple infrared or stereotactic cameras 350 may be used in unison to provide more accurate and expansive imaging.

(49) The surgeon may then attach the jig 500 to the exposed bone 400 at a predetermined or desired location. The spatial mapping camera 300 may spatially map, and the infrared or stereotactic camera may identify coordinates of, the jig 500 and the exposed bone 400 to map the surface of the exposed bone and relative location of the jig 500.

(50) As shown in FIG. 6, the 3D spatial mapping camera 300 and the infrared or stereotactic camera 350 may be formed as a combined unitary structure 375. This unitary structure 375 may permanently fix cameras 300 and 350 relative to one another, or enable cameras 300 and 350 to be removably connected to one another. This combined structure or device may increase the portability of the cameras 300 and 350, which in some circumstances may increase efficiency of an operation or surgical procedure. As shown in FIG. 1, however, cameras 300 and 350 may also be placed or fixed at separate locations in an operating room or other desired space.

(51) As shown in FIG. 7, the headset 102 may generate a holographic image 900, or visual representation, viewable by the surgeon on the viewer 104, such that the surgeon can maintain a real-world view of the bone 400 and simultaneously view the holographic image 900. The holographic image 900 may include a real-time rendering or visual representation of the exposed bone 400 and a real-time rendering or visual representation of the jig 500. The headset 102 can then scan one or both of the markers 502 or 504 and identify the proper or required position of the jig 500, relative to the bone 400. The surgeon can then manipulate the jig 500, until the proper position is set and determined.

(52) Once the proper position of the jig 500 is set in the holographic image 900, the surgeon can use the headset 102 to lock the holographic image 900 in place. Then the surgeon can manipulate the actual physical jig 500 to substantially match the positioning of the set rendering of the jig 500.

(53) In another embodiment, as shown in FIG. 8, the mixed reality system 100 may be used such that one of the cameras 300 or 350 may be utilized outside of a patient's body 450, while another one of the cameras 300 and 350 may be inserted within the patient's body to spatially map or identify coordinates for a surgical procedure from within the body. Alternatively, the system 100 may use multiple 3D spatial mapping cameras 300, e.g., one or more spatial mapping cameras on the exterior of the patient and one or more additional spatial mapping cameras on the interior of the patient. The combined images of the camera 300 or 350 outside of the patient's body, and the camera 300 or 350 inside the patient's body may provide more accurate data for identifying the location of a surgical procedure. For example, camera 300 or 350 may be inserted into a patient's lungs, throat, stomach, colon, vagina, rectum, or any other desired cavity or interior space within a patient's body. The image information of the inside and outside cameras 300 and 350 can then be superimposed as described above to provide additional detail and location orientation that are not otherwise available from imaging only outside the body. As an additional example, a surgeon can use the imaging information from the two or more cameras 300 or 350 to then identify a surgical procedure location within the patient, and such location could be marked or “tattooed” and then a surgeon could then more easily locate the marked location during the surgical procedure, e.g., using a laparoscope or other similar surgical device.

(54) A surgical jig is conventionally, and may be, a surgical tool that may be used to help a surgeon make predetermined and accurate cuts of a desired bone to facilitate attachment of a surgical implant. A jig may have one or a series of slots located at specific predetermined locations and at specific predetermined angles, with respect to a body of the jig, such that when the jig is attached to a bone surface. The surgeon can make precise and accurate cuts using the jig as a guide without the need of additional measurements. Once the desired cuts of the bone have been made using the jig as a guide, the jig may be removed from the bone and the surgical implant may be properly secured to the bone, which has now been accurately cut to receive the implant.

(55) Due to the accuracy of the disclosed method and system, the jig 500 may be made of plastic, metal, polyamide, or any other desired material. Manufacturing the jig 500 out of a plastic or polyamide material, or other relatively inexpensive material, may allow the jig 500 to be disposable, while still maintaining the precision and accuracy of traditional metal jigs. The jig 500 may also be manufactured using a 3D printer, which can further reduce the cost of manufacturing and storage of jigs, since 3D printed jigs could be made on demand and customized to the size and shape required by individual patients and users. The physical jig 500 may also be manufactured using any other known technique for forming a physical jig.

(56) In the foregoing Detailed Description, various features of the present disclosure are grouped together in a single embodiment for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed disclosure requires more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive aspects lie in less than all features of a single foregoing disclosed embodiment. Thus, the following claims are hereby incorporated into this Detailed Description of the Disclosure by this reference, with each claim standing on its own as a separate embodiment of the present disclosure.

(57) It is to be understood that the above-described arrangements are only illustrative of the application of the principles of the present disclosure. Numerous modifications and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of the present disclosure are intended to cover such modifications and arrangements. Thus, while the present disclosure has been shown in the drawings and described above with particularity and detail, it will be apparent to those of ordinary skill in the art that numerous modifications, including, but not limited to, variations in size, materials, shape, form, function and manner of operation, assembly and use may be made without departing from the principles and concepts set forth herein.