AUGMENTED REALITY SOFT TISSUE BIOPSY AND SURGERY SYSTEM

Abstract

Apparatus and methods are described for use with a region of interest (ROI) within breast tissue of a breast of a subject and one or more reference markers configured to be placed at respective reference points on skin of the breast. A processing unit determines a relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast, and, subsequently, determines a location of the ROI within the breast tissue by identifying the one or more reference markers to thereby localize the ROI within the breast tissue. The processing unit outputs augmented-reality data at a location corresponding to the ROI. Other applications are also described.

Claims

1. An apparatus for use with a region of interest (ROI) within breast tissue of a breast of a subject and one or more reference markers configured to be placed at respective reference points on skin of the breast, the apparatus comprising: a processing unit configured to: determine a relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast; subsequently, determine a location of the ROI within the breast tissue by identifying the one or more reference markers to thereby localize the ROI within the breast tissue; and output augmented-reality data at a location corresponding to the ROI.

2. The apparatus according to claim 1, wherein the one or more reference markers include one or more markings on the skin of the breast and/or one or more objects placed on the skin of the breast, and wherein the processing unit is configured to identify the one or more reference markers by identifying the one or more markings on the skin of the breast and/or the one or more objects placed on the skin of the breast.

3. The apparatus according to claim 1, wherein the apparatus is configured for use with one or more auxiliary reference markers placed on skin and/or tissue of the breast, and wherein, in the event that one or more of the reference points on the skin of the breast lies in an area requiring skin removal or surgical entry, the processing unit is configured to: determine a relationship between the position of the ROI within the breast tissue and the one or more auxiliary reference markers; subsequently, determine a location of the ROI within the breast tissue by identifying the one or more auxiliary reference markers to thereby localize the ROI within the breast tissue; and output augmented reality data at a location corresponding to the ROI.

4. The apparatus according to claim 1, wherein the processing unit is configured to receive an input from a user indicating that the relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast has been determined and in response thereto to associate the position of the ROI within the breast tissue and the one or more reference markers on the skin of the breast via a 3D connectivity mesh.

5. The apparatus according to claim 1, wherein the processing unit is configured to output the augmented reality data at a location on the breast that corresponds to the ROI.

6. The apparatus according to claim 1, wherein the one or more reference markers comprise three or more reference markers and wherein the processing unit is configured to determine the relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast by determining respective relationships between the position of the ROI within the breast tissue and positions of each of the reference markers on the skin of the breast.

7. The apparatus according to claim 1, wherein the processing unit is configured to determine the relationship between the position of the ROI within the breast tissue and positions of each of the reference markers on the skin of the breast, by defining one or more wired 3D geometry figures with nodes that connect the ROI within the breast tissue and positions of the one or more reference markers on the skin of the breast.

8. The apparatus according to claim 1, wherein the one or more reference markers comprise one or more visible reference markers.

9. The apparatus according to claim 1, wherein the one or more reference markers are configured to be repositioned to new positions during a procedure that is performed on the subject's breast, and wherein the processing unit is configured to determine an updated relationship between positions of the ROI within the breast tissue and the new positions of the reference markers.

10. The apparatus according to claim 1, wherein the processing unit is configured to localize the ROI within the breast tissue by translating movement of the one or more reference markers to movement of the ROI within the breast tissue.

11. The apparatus according to claim 10, wherein the processing unit is configured to translate movement of the one or more reference markers to movement of the ROI within the breast tissue by assuming that movement of the breast tissue will not significantly vary a distance between the ROI within the breast tissue and the one or more reference markers on the skin of the breast.

12. The apparatus according to claim 1, wherein the apparatus is for use with at least one breast marker that is placed within the breast tissue such as to designate the ROI and wherein the processing unit is configured to determine the relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast by determining a relationship between positions of the at least one breast marker within the breast tissue and the one or more reference markers on the skin of the breast.

13. The apparatus according to claim 12, wherein the apparatus is for use a plurality of breast markers that are placed within the breast tissue such as to designate the ROI and wherein the processing unit is configured to define wired 3D geometry figures with nodes that connect the plurality of breast markers within the breast tissue and positions of respective reference markers on the skin of the breast in order to define the ROI volume.

14. The apparatus according to claim 1, wherein the apparatus is configured for use with augmented reality glasses and wherein the processing unit is configured to output augmented reality data at a location corresponding the ROI on the augmented reality glasses.

15. The apparatus according to claim 14, wherein the processing unit is configured to output an augmented reality image of the ROI volume at the location corresponding to the ROI on the augmented reality glasses.

16. The apparatus according to claim 14, wherein the apparatus is configured to use with a tool having a working edge and the processing unit is configured to output an augmented reality image of a real-time position of the working edge of a tool relative to the ROI on the augmented reality glasses.

17. The apparatus according to claim 1, wherein the ROI includes a tumor and wherein the processing unit is configured to: determine a relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast; subsequently, determine a location of the tumor within the breast tissue by identifying the one or more reference markers to thereby localize the tumor within the breast tissue; and output augmented-reality data at a location corresponding to the tumor.

18. The apparatus according to claim 17, wherein the processing unit is configured to determine the position of the tumor within the breast tissue based on an ultrasound image of the tumor and to determine the relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast by determining a relationship between the positions of the one or more reference markers on the skin of the breast and the position of the tumor within the breast tissue, as determined based on an ultrasound image of the tumor.

19. The apparatus according to claim 17, wherein the processing unit is configured to determine the position of the tumor within the breast tissue by localizing one or more passive markers that are placed in a vicinity of the tumor, and to determine the relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast by determining a relationship between positions of the one or more reference markers on the skin of the breast and the position of the tumor within the breast tissue, as determined by localizing the one or more passive markers.

20. The apparatus according to claim 17, wherein the processing unit is configured to determine the position of the tumor within the breast tissue using guide wire localization, and to determine the relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast by determining a relationship between positions of the one or more reference markers on the skin of the breast and the position of the tumor within the breast tissue, as determined using guide wire localization.

21. A method for use with a region of interest (ROI) within breast tissue of a breast of a subject and one or more reference markers configured to be placed at respective reference points on skin of the breast, the method comprising: determining a relationship between position of the ROI within the breast tissue and the one or more reference markers on the skin of the breast; subsequently, determining a location of the ROI within the breast tissue by identifying the one or more reference markers to thereby localize the ROI within the breast tissue; and outputting augmented reality data at a location corresponding to the ROI.

22. The method according to claim 21, further comprising receiving an input from a user indicating that the relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast has been determined and in response thereto associating the position of the ROI within the breast tissue and the one or more reference markers on the skin of the breast via a 3D connectivity mesh.

23. The method according to claim 21, wherein the one or more reference markers include three or more reference markers and wherein determining the relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast comprises determining respective relationships between the position of the ROI within the breast tissue and positions of each of the reference markers on the skin of the breast.

24. The apparatus according to claim 21, wherein determining the relationship between the position of the ROI within the breast tissue and positions of each of the reference markers on the skin of the breast comprises defining one or more wired 3D geometry figures with nodes that connect the ROI within the breast tissue and positions of the one or more reference markers on the skin of the breast.

25. The method according to claim 21, wherein the ROI includes a tumor and wherein: determining the relationship between positions of the ROI within the breast tissue and the one or more reference markers on the skin of the breast comprises determining a relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast; determining the locations of the ROI within the breast tissue by identifying the one or more reference markers to thereby localize the ROI within the breast tissue comprises determining a location of the tumor within the breast tissue by identifying the one or more reference markers to thereby localizing the tumor within the breast tissue; and outputting augmented reality data at the location corresponding to the ROI comprises outputting augmented-reality data at a location corresponding to the tumor.

26. The method according to claim 25, further comprising determining the position of the tumor within the breast tissue based on an ultrasound image of the tumor, wherein determining the relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast comprises determining a relationship between positions of the one or more reference markers on the skin of the breast and the position of the tumor within the breast tissue, as determined based on the ultrasound image.

27. The method according to claim 25, further comprising determining the position of the tumor within the breast tissue by localizing one or more passive markers that are placed in a vicinity of the tumor, wherein determining the relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast comprises determining a relationship between positions of the one or more reference markers on the skin of the breast and the position of the tumor within the breast tissue, as determined by localizing the one or more passive markers that are placed in the vicinity of the tumor.

28. The method according to claim 25, further comprising determining the position of the tumor within the breast tissue by using guide wire localization, wherein determining the relationship between positions of the tumor within the breast tissue and the one or more reference markers on the skin of the breast comprises determining a relationship between positions of the one or more reference markers on the skin of the breast and the position of the tumor within the breast tissue, as determined by using guide wire localization.

Description

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

[0049] FIG. 1 is a conceptual diagram of a binocular system incorporating augmented reality with the main components of an exemplary AR system illustrated within a coordinate system.

[0050] FIG. 2 is a diagram illustrating the binocular concept of remote measurements with simplified (2D) remote point localization.

[0051] FIG. 3 is a conceptual diagram illustrating the estimation of the position of an obscured locator's tip, showing a method of determining the reference points and geometrically determining the markers' group (ROI) 3D object.

[0052] FIG. 4 is a conceptual diagram illustrating a virtual markers' group (ROI) determination, using the reference points alone rather than by using a locator for marker group tracking.

[0053] FIG. 5A is a conceptual diagram illustrating a group of markers establishing the shape of a virtual object defining a relatively large ROI volume to be removed.

[0054] FIG. 5B is a conceptual diagram illustrating a single marker centrally located and establishing the shape of a virtual object defining a small ROI volume to be removed.

[0055] FIG. 5C is a conceptual diagram illustrating a single marker proximately located and establishing the shape of a virtual object defining a small ROI volume to be removed.

[0056] FIG. 6 is a conceptual diagram illustrating a virtual operational ROI virtual object created by the combination of smart marker volume and light defined surgical instrument (6-2) whose position is defined by at least one reference point (6-11).

[0057] FIG. 7 is a diagram illustrating a procedure for calibrating a surgical instrument's length for use in AR surgery by touching the calibration LED (7-1) while being observed by the AR glasses.

DETAILED DESCRIPTION OF VARIOUS EMBODIMENTS

[0058] The present invention, as well as features and aspects thereof, is directed towards providing a marker localization system with binocular augmented reality (AR) glasses that are communicatively coupled to a server that directs a procedure.

[0059] The disclosed AR system, which may be utilized in surgical settings, biopsy settings or other procedures, comprises a marker localization system with binocular AR glasses wire/wirelessly interconnected with at least one master computer or processing unit (server, tablet, laptop, etc.). The computer can be the computer of the smart-marker localization system which can run additional appropriate SW for control of the AR processes, a computer independent to the AR system (a sole AR computer) or combined AR computer and marker localization system computers with appropriate interfaces for automatically transferring the data of markers depth in real-time. The term AR computer can also be referred to as the AR master computer. Throughout the disclosure, the general term computer will be understood to refer to either or a combination of the above computer systems.

[0060] The AR glasses can comprise an assembly of VR glasses and high-resolution binocular video cameras wherein both the binocular system and VR glasses have appropriate interfaces with the computer(s) allowing real-time operations. The system also includes optically contrasted objects/signs/icons. The computer can provide: the distance measurements from the currently selected marker to a locator (which may be handheld); the marker's electronic ID if available; and inter-marker distance as well as to perform all required AR calculations. Simultaneously, the AR glasses provide the visual information used for geometry measurements, geometry transformation and composite image creation. The reader is referred to reference U.S. Provisional Patent Application No. 63/218,973 Filed on Jul. 7, 2021 now U.S. patent application Ser. No. 17/841,663 filed on Jun. 16, 2022 (incorporated herein above by reference) for a detailed description of an exemplary marker system that is suitable for various embodiments of the AR surgical system. It should be appreciated by those skilled in the relevant art that markers and locators of design other than what is described in U.S. Ser. No. 17/841,663 may also be used to construct a virtual rendering of the ROI. Generally, the markers and locator design should be such that they provide marker depth/distance from locator tip to marker measurements. For example, Cianna Medical Savi Scout, Health Beacons LOCalizer, Magseed, Elucent, and Molli markers and their appropriate locators can be used to generate the AR rendering of the ROI by, for example, attaching appropriate LED's to their locators (the locator in the case of Elucent being a surgical instrument). Grid/chess board references can also be used for this purpose.

[0061] Embodiments of the AR system utilize markers that are transponders, although in some embodiments markers that are signal reflectors or signal activated may also be used. Further, the transponder embodiment of the AR system operate to process the phase shift between the signals that are received from the markers and the reference signal rather than just the wave amplitude. In some embodiments, smart markers may be utilized that have collective ID's. Some embodiments of the AR surgical system utilize a pair of commonly available glasses and smart markers of various companies each covered by patents, combined with a control system to provide a unique AR surgical system. The various embodiments provide a solution for use in soft tissue surgery or procedures, especially in mobile tissue environments, such as breast surgery, where standard augmented reality image fusion is difficult. The various embodiments likewise provide a solution for image guided soft tissue biopsy, especially using ultrasound, in cases where the tumor has completely or almost completely disappeared after the use of some systemic therapy given prior to planned surgery. In such instances, there is a need to biopsy a tissue area previously occupied by the tumor, which is no longer visible, in order to see if microscopic remnants of tumor are still present.

[0062] FIG. 1 is a conceptual diagram of a binocular system incorporating augmented reality with the main components of an exemplary AR system illustrated within a coordinate system. In the illustrated binocular AR system (1-14), the left and right electro-optical subsystems (1-10) and (1-11) respectively, are identical, comprised of the lens (1-1) (left) and (1-2) (right) located at their focal distance (1-3) from the video sensors (1-4) (left) and (1-5) (right). The left video sensor (1-4) being viewed from the left eye (1-12) and the right video sensor (1-5) being viewed from the right eye (1-13) from a particular distance. The coordinate system XYZ of the AR system has its origin O at the plane of the video sensors (1-4) and (1-5), placed at the mid-point between these sensors. The distance D between these sensors establishes the base of the binocular system. The rays from some point P0 (X0,Y0,Z0) (1-6) in soft tissue (1-8) housing a group of smart markers (1-7) create two different images at the left sensor (mL,nL) pixels and right sensor (mR,nR) pixels. Using the (known) sensor dimensions of locator (1-9) and pixel resolutions (which can be different in the x-axis and y-axis) we can calculate (reconstruct) the coordinates of point P0. In some embodiments a processing unit (1-20), which may be a desktop computer, laptop computer, mobile device, server, etc., is communicatively couple to the AR binocular glasses (1-14) and or the locator (1-9). The coupling may be wired or wireless in various embodiments. The processing unit (1-20) may be a computer or processing unit integrated with the smart-marker localization system and running additional appropriate SW for control of the AR processes. Similarly, the processing unit (1-20) maybe a computer or processing unit that is independent to the AR system (a sole AR computer). Further, the processing unit (1-20) may be a combination of an independent AR computer and a marker localization system computer(s) with appropriate interfaces for automatically transferring the data of markers depth in real-time. Even further, the processing unit may be fully integrated within the AR glasses (1-14) or distributed between the AR glasses (1-14) localization system and/or an independent AR computer.

[0063] FIG. 2 is a diagram illustrating the binocular concept of remote measurements with simplified (2D) remote point localization. The following algorithm can be demonstrated on the simplified 2D projection presented in FIG. 2:

[0064] The coordinate system XOZ has an origin at the middle between left sensor matrix (2-1) and right sensor matrix (2-2). The rays from the point P (X,Y,Z) pass through left lens (2-3) and right lens (2-4) to create images at the left matrix (n.sub.L,m.sub.L) pixels and at the right matrix (n.sub.R,m.sub.R) pixels. Both matrices are identical and have vertical number of pixels M and horizontal number of pixels N. In linear units of measure, the matrix resolutions are .sub.X and .sub.y respectively. The local origins of each matrix are at their centers, while the axis of local coordinate system of each one is:

[00001]$\mathrm{Vertical}:m=-M/2,.Math.0,.Math.M/2,\mathrm{and}$$\mathrm{Horizontal}:n=-N/2,0.Math.0,N/2.$

[0065] The coordinates of point P can be calculated according to the following formulas:

[00002]$\begin{array}{cc}X=\frac{D}{2}.Math.\frac{n_R+n_L}{n_R-n_L}& \left[\mathrm{F1}\right]\end{array}$$\begin{array}{cc}Z=\frac{D.Math.F}{\left(n_R-n_L\right).Math.{}_X}+F& \left[\mathrm{F2}\right]\end{array}$$\begin{array}{cc}Y=\frac{Z}{F}.Math.\frac{m_R+m_L}{2}.Math.{}_Y& \left[\mathrm{F3}\right]\end{array}$

[0066] This binocular system, for example, has a base D=70 mm, focal lengths F=10 mm, sensor resolution of 20 MP (mega-pixels) and pixel size of 1.5 m and provides an accuracy of X,Y,Z coordinate estimation of about 0.1 mm, at a distance range of Z=[350 to 500] mm. This range satisfies the distance from the eyes of the surgeon to the operation field for most surgeries including breast surgeries.

[0067] As described above, the AR process requires the surgeon to utilize the locator in the creation of the virtual image. For example, the surgeon can hold the locator and glide its tip across the tissue surface, for example, the patient's breast. The computer system provides accurate measurement of the distance between the tip of the locator and the marker-under-test. The tip of the locater can be (visually) obscured by the hands of the surgeon resulting in its coordinates not being optically measured directly by the AR system. This situation can require an estimation of the coordinates of the locator's tip.

[0068] Estimation of the coordinates of the locator's tip is illustrated by FIG. 3. FIG. 3 is a conceptual diagram illustrating the estimation of the position of an obscured locator's tip, showing a method of determining the reference points and geometrically determining the markers' group (ROI) 3D object. Two LEDs can be used in the current disclosure to assign visible points on the Locator (3-1). One LED (3-2) can be located on the upper (visible) tip of the locator at the point P1=(X1,Y1,Z1). The other LED (3-3) can be located also on the visible part of the locator at P2=(X2, Y2,Z2). This point resides at a position along the locator length L. The algorithm disclosed allows reconstruction of both points P1 and P2. In this case, the required point P=P (X,Y,Z) (3-4) of the locator's tip can be recalculated according to the following formulas for 2D solution:

[00003]$\begin{array}{cc}k=\frac{Z_2-Z_1}{X_2-X_1}& \left[\mathrm{F4}\right]\end{array}$$\begin{array}{cc}X=X_1\frac{L}{\sqrt{k^2+1}}& \left[\mathrm{F5}\right]\end{array}$$\begin{array}{cc}Z=Z_1\frac{L.Math.K}{\sqrt{k^2+1}}& \left[\mathrm{F6}\right]\end{array}$

[0069] The solution is a point with maximum value of Z (most remote point from the AR origin). Due to the outside diameter of the locator body, a number of LEDs can be used around the body at point P2 to ensure an optically accessible (unobscured) visual point and allow for correct calculation(s).

[0070] The 3D estimation of the locator's tip position is performed according to the following algorithm:

[00004]$\begin{array}{cc}\mathrm{Calculate}\mathrm{the}\mathrm{vector}V=P_2-P_1;& \left[\mathrm{F7}\right]\end{array}$$\begin{array}{cc}\mathrm{Calculate}\mathrm{the}\mathrm{Norm}\mathrm{of}\mathrm{the}\mathrm{vector}V:\mathrm{Norm}\left(V\right)=\sqrt{{\left(X_2-X_1\right)}^2+{\left(Y_2-Y_1\right)}^2+{\left(Z_2-Z_1\right)}^2};& \left[\mathrm{F8}\right]\end{array}$$\begin{array}{cc}\mathrm{Calculate}\mathrm{unit}\mathrm{vector}{V}_{\mathrm{UNIT}}=V/\mathrm{Norm}\left(V\right);& \left[\mathrm{F9}\right]\end{array}$$\begin{array}{cc}\mathrm{Find}\mathrm{required}\mathrm{point}{P}_3\left(X,Y,Z\right)=P_1+L.Math.V_{\mathrm{UNT}}.& \left[\mathrm{F10}\right]\end{array}$

[0071] The result of this algorithm can be verified:

[00005]$\begin{array}{cc}\sqrt{{.Math.}_i^3{\left(P_3\left({}_i\right)-P_1\left({}_i\right)\right)}^2}=L,\mathrm{where}X={}_1,Y={}_2,Z={}_3& \left[\mathrm{F11}\right]\end{array}$

[0072] This algorithm provides an accuracy of 3D estimation of locator tip position to better than +/0.2 mm.

[0073] The accumulation of marker localization data permits determination of the distances between the implanted markers and between the last localized marker and the tip of the locator. The combination of all these distances in the computer memory can be interpreted as a (3-5) 3D pattern, for example a pyramid as depicted in FIG. 3, extending from the marker group 3-10. This pattern within the computer memory is defined with respect to the last location of the locator's tip. This last location creates a refence point. The refence point can be designated on the relevant tissue, for example, the skin of the breast with an appropriate ink or marker. The pattern within the computer memory can be reinforced by choosing multiple reference points.

[0074] FIG. 4 is a conceptual diagram illustrating a virtual markers' group (ROI) determination, using the reference points alone rather than by using a locator for marker group tracking. Thus, the diagram of FIG. 4 illustrates the AR system measurements after removing the locator. Three reference points (4-1), (4-2) and (4-3) ensure localization of the pyramid 3D pattern (4-4) in the absence of the locator. Significantly, these three points should not lie on a straight line. Any movement or rotation of the AR glasses will cause a corresponding movement and rotation of the pyramid pattern, providing a spatially accurate (virtual) 3D visualization of the implanted marker group.

[0075] By performing the above-described process, the surgeon, after having set up a first marker pyramid, is free to operate on the markers and the ROI which they define with free hands and real-time vision. During the course of a surgery, for example with tissue mobilized and moved, the surgeon can at any time reestablish a new pyramid by relocating the markers and establishing a new ROI refence point. In the event of 3 or more markers being placed, the markers defining the base of the resultant 3D volume will determine the configuration/shape of that volume and the position of its reference tip.

[0076] FIGS. 5A, 5B and 5C, collectively referred to as FIG. 5, illustrate the creation of differing examples of marker-assigned ROI's. FIG. 5A is a conceptual diagram illustrating a group of markers establishing the shape of a virtual object defining a relatively large ROI volume to be removed. FIG. 5B is a conceptual diagram illustrating a single marker centrally located and establishing the shape of a virtual object defining a small ROI volume to be removed. FIG. 5C is a conceptual diagram illustrating a single marker proximately located and establishing the shape of a virtual object defining a small ROI volume to be removed.

[0077] Looking at FIG. 5A, in the case of a relatively large volume of suspicious tissue (5-1), the markers can be implanted adjacent to the tumor edges (5-2). This is referred to as bracketing. If desired, a virtual object (5-3) of varying geometry can be fitted as an outer envelope surrounding the marker wired object. The fitted geometry can be parallelepiped, ellipsoid, spherical, etc. The position of this virtual object is defined by the XYZ-coordinates of each marker determined during the localization process described above. After localization of all markers of the virtual object, their position can be tracked optically (using the binocular sub-system of the AR system) relative to the reference point (5-4) also determined during the localization procedure.

[0078] Turning to FIG. 5B, in the case of a relatively small volume (for example diameter 1-2 cm) of suspicious tissue (5-5) or (5-9 in FIG. 5C), a single marker can be used for establishing the virtual object for surgery of the ROI. The elements (5-5)-(5-8) in FIG. 5B illustrate the case where the marker is implanted in the center of the suspicious volume (5-5), while element (5-9)-(5-12) in FIG. 5C depicts the marker implanted at the edge of the suspicious volume (5-9). In both cases, the single-marker can be used for defining the lesion. The chosen virtual object can be of a plurality of shapes, for example a sphere. The dimensions, for example the diameter of a virtual spherical object, are defined by the operator by inspection of the medical images of the implanted markers pre-surgery. The diameter of a spherical virtual object can be determined by measurements derived from post marker implant, pre-surgical imaging studies such as Ultrasound, Mammography, CT, MRI, etc. For example, the position of the center of a spherical virtual object is defined by the position (XYZ-coordinates) of the marker. The desired appropriate measurement can be entered by the operator into a computer program which defines the size of the sphere as seen virtually at surgery or for example at image guided needle biopsy. The program can assign a plurality of shapes. The shapes may be irregular and assigned by an artificial intelligence (AI) interface based on tumor outline data from preceding diagnostic imaging such as MRI, CT and Ultrasound. The operator can also elect to place the chosen virtual object shape eccentrically depending on presurgical imaging.

[0079] FIG. 6 is a conceptual diagram illustrating a virtual operational ROI virtual object created by the combination of smart marker volume and light defined surgical instrument (6-2) whose position is defined by at least one reference point (6-11). Thus, the illustration depicts the creation of the Virtual operational ROI. FIG. 6 illustrates the creation of the combination of a virtual object composed of smart markers volume (6-10) and a light defined surgical instrument (6-2), whose locations are defined relative to at least one reference point (6-11). The surgical instrument (6-1) may be a biopsy needle device, ablation device needle or other interventional device, such as a local radiation device, as non-limiting examples. Similar to the estimation of the locator tip described above, the tip of surgical instrument (6-2) or other relevant device as described can be determined by the binocular system (6-6), (6-7), (6-8), (6-9), according to the equations [F4]-[F6] and algorithm [F7]-[F11]. The only difference is an unknown arbitrary length of the surgical instrument L. This value can be determined by the calibration procedure illustrated in FIG. 7.

[0080] FIG. 7 is a diagram illustrating a procedure for calibrating a surgical instrument's length for use in AR surgery by touching the calibration LED (7-1) while being observed by the AR glasses. The calibration of the length of the surgical instruments (7-1) can be used on instruments such as biopsy needle devices or surgical instruments. Biopsy needle devices/surgical instruments used in biopsy procedures for example, ultrasound guided biopsies and/or AR surgery can have attached two clips, for example, each composed of a sterile sealed mechanical assembly comprising a point-LED optical source and a small battery. One LED can be located at the upper tip of the needle/instrument (p1), while second one can be attached at the mid part of the needle/instrument (p2). The lower tip (p) of the needle/instrument can be free for biopsy/operating.

[0081] The calibration procedure can be performed prebiopsy/presurgery for biopsy needles and surgical instruments such as an electrocautery device, diathermy knife and scalpels etc., planned to be used in the course of the biopsy/surgical procedure. The calibration process uses a calibration point (calibration LED (7-5)) located on an appropriate calibration surface. The operator can touch the calibration LED and assign this event by sending a signal to a computer system designating the start of the appropriate calibration program. The binocular sub-system of the AR system performs an estimation of the 3D locations of 3 LEDs: P.sub.1, P.sub.2 and P. The length of the needle/instrument is determined by the following algorithm:

[00006]$\begin{array}{cc}\mathrm{Calculate}\mathrm{the}\mathrm{vector}V=P_2-P_1;& \left(\mathrm{F12}\right)\end{array}$$\begin{array}{cc}\mathrm{Calculate}\mathrm{the}\mathrm{Norm}\mathrm{of}\mathrm{the}\mathrm{vector}V:\mathrm{Norm}\left(V\right)=\sqrt{{\left(X_2-X_1\right)}^2+{\left(Y_2-Y_1\right)}^2+{\left(Z_2-Z_1\right)}^2};& \left(\mathrm{F13}\right)\end{array}$$\begin{array}{cc}\mathrm{Calculate}\mathrm{unit}\mathrm{vector}{V}_{\mathrm{UNIT}}=V/\mathrm{Norm}\left(V\right);& \left(\mathrm{F14}\right)\end{array}$$\begin{array}{cc}\mathrm{Find}\mathrm{required}\mathrm{length}L=\left(P\left(X,Y,Z\right)-P_1\right)/V_{\mathrm{UNT}}.& \left(\mathrm{F15}\right)\end{array}$

[0082] A method other than utilizing LED'S attached to the locator and surgical instruments can be used. For example, grid/chess board references can also be used for calibration.

[0083] The combination of the system of marker coordinates together with reference points and the coordinates of the tip of a surgical instrument/needle, for example of a biopsy device, can create the virtual object of the dynamic field for surgery/needle biopsy of the tumor bed. This object combined (by appropriate SW) with the binocular AR glasses worn by for example a surgeon/breast imager creates an AR dynamic field allowing the operator full visual control of the entire ROI as defined by the marker(s).

[0084] The described method can allow the surgeon to see if the tip of his instrument(s), for example the cutting diathermy or scalpel, has entered the virtual volume of tissue as defined by the markers and hence, transgressed the desired surgical margin. This transgression can also be indicated by a signal, such as for example, by a warning sound or a red (transgression) or green (non-transgression) light. By including the surgical instrument as described above in the AR scene, the surgeon is able to operate in real-time, seamlessly, without the need to periodically revert to using the locator. The surgical instrument can be seen in 3D with respect to the AR defined ROI. For example, the instrument can be seen or designated as lying anterior or posterior to the 3D wired virtual marker defined object.

[0085] In cases where tumors have responded fully or almost completely to preoperative systemic therapy, a biopsy of the suspected residual tumor bed can be done under imaging, for example with ultrasound guidance. The physician/operator can see, for example at real-time ultrasound, if the tip of the biopsy needle has entered the virtual volume of tissue as defined by the markers or transgressed the desired originally defined tumor volume. This transgression can also be indicated by a signal such as for example by a warning sound or a red (transgression) or green (non-transgression) light. The biopsy needle can be seen in 3D with respect to the AR defined ROI. For example, the needle/instrument can be seen or designated as lying anterior or posterior to the 3D wired virtual marker defined object. The AR field can be image fused with the image of the imaging modality being used for the biopsy for example the ultrasound or CT or MRI screen image.

[0086] The node points established by the markers and the designated reference points can be integrated with an AI program that can be combined with the above-described method to enhance the integration of the reference points and 3D virtual defined volume. AI can be used to collect data from each procedure for the establishment of a historical data set. This will allow for suggestions and alerts to surgeons, providing real time visual data for intelligent surgery.

[0087] The AR system described and the associated computer program may be used to generate a simulation model of the ROI/ROI's requiring biopsy and or excision. The markers and associated ROI/ROI'S in the simulation model can duplicate the in vivo scenario and allow for teaching/review and practice of the interventional procedures prior to the actual in vivo event/s.

[0088] In the description and claims of the present application, each of the verbs, comprise, include and have, and conjugates thereof, are used to indicate that the object or objects of the verb are not necessarily a complete listing of members, components, elements, or parts of the subject or subjects of the verb.

[0089] The present invention has been described using detailed descriptions of embodiments thereof that are provided by way of example and are not intended to limit the scope of the invention. The described embodiments comprise different features, not all of which are required in all embodiments of the invention. Some embodiments of the present invention utilize only some of the features or possible combinations of the features. Variations of embodiments of the present invention that are described and embodiments of the present invention comprising different combinations of features noted in the described embodiments will occur to persons of the art.

[0090] It will be appreciated by persons skilled in the art that the present invention is not limited by what has been particularly shown and described herein above. Rather the scope of the invention is defined by the claims that follow.