Automatic multimodal real-time tracking of a moving marker for image plane alignment inside a MRI scanner

Abstract

A system for an automatic multimodal real-time tracking of moving instruments for image plane alignment inside an MRI scanner includes: an MRI scanner, an MRI multi-plane pulse sequence generating unit allowing to interactively modify the position and orientation of one or several image planes in real-time, one or several external optical sensors with high frame rate, preferably a RGB-D sensor or other similar camera system like a stereovision systems, a multimodal marker including at least one MR visible feature and one visual feature able to be tracked by both the MRI scanner and the at least one external optical sensor, a computer for processing in real-time images from both MRI and optical sensor to fuse the detected marker position and orientation or pose from both modalities, and predict the next image plane position and orientation based on the estimated motion of the moving marker.

Claims

1. A method for tracking a moving multimodal marker, stand-alone or fixed on an instrument, on a holding system, or on a subject's body, wherein said method comprises an automatic image plane alignment process using a system (1) comprising: an MRI scanner (2), an MRI multi-plane pulse sequence generating means configured to interactively modify the position and orientation of one or more image planes in real-time, at least one external optical sensor (3) with high frame rate, a multimodal marker (4) comprising at least one MR visible feature and one visual feature configured to be tracked by both the MRI scanner (2) and the at least one external optical sensor (3), a computer (5) configured to process in real-time images from both the MRI scanner and the at least one external optical sensor in order to fuse a detected marker position and an orientation or a pose from both modalities, and predict or estimate a position and an orientation of a next image plane based on estimated motion of the multimodal marker (4), in order to automatically track the multimodal marker (4) and automatically align the image planes with respect to the multimodal marker (4), or to the holding system or body location the multimodal marker (4) is fixed on, said method comprising the steps of: processing in real-time the images from both the MRI scanner and the at least one external optical sensor in order to fuse the detected position of the multimodal marker (4) and orientation or pose from both modalities, and of predicting or estimating the next image plane position and orientation based on the estimated motion of the moving multimodal marker (4), in order to automatically track said multimodal marker (4) and automatically align the image planes with respect to the multimodal marker (4), or to the instrument or body location the multimodal marker (4) is fixed on, wherein with the multimodal marker being fixed on the instrument, performing a process for automatic multimodal real-time tracking of the multimodal marker (4) for image plane alignment in magnetic resonance imaging comprises the steps of: providing the multimodal marker (4) comprising at least one MR visible feature and at least one visual feature; providing a real-time interactive MRI pulse sequence configured to update of the image plane(s) position and orientation; providing an MRI sensor (2) for marker pose measurement in the MRI frame; providing the at least one external optical sensor (3) for marker pose measurement inside the MRI scanner; defining the system time period as the period of the at least one external optical sensor with the highest acquisition frame rate; (a) measuring the marker pose in the MRI frame and/or in the at least one external optical sensor frame, depending on their respective frame rates; (b) when a marker pose measurement is available from the at least one external optical sensor, translating the marker pose measured in the at least one external optical sensor frame to the MRI frame using the current rigid transformation between the MRI frame and the the at least one external optical sensor frame; (c) filtering and estimating the optimal pose of the multimodal marker considering a model of the tracking process and the multimodal marker's (4) noise, using a data fusion algorithm; (d) predicting the marker pose based on the optimal estimation, using a data fusion algorithm; (e) calculating the corresponding image plane position and orientation encompassing the multimodal marker based on the prediction of the marker pose from the data fusion algorithm; (f) providing the predicted image plane position and orientation to an MRI console; (g) updating the real-time interactive MRI pulse sequence image plane position and orientation based on the predicted image plane position and orientation; (h) in parallel to steps (b) to (g), following step (a), providing a new pair of matched points when matching marker poses are available from both the MRI sensor and the at least one external optical sensor over the time period; (i) in parallel to steps (b) to (g), following step (h), adding to the current list of matched points the new pair of matched points when adding the new pair of matched points improves the estimation of the rigid transformation between the MRI frame and the the at least one external optical sensor frame; and (j) in parallel to steps (b) to (g), following step (i) updating the estimation of the rigid transformation between the two frames using an optimization algorithm based on the updated list of matched points.

2. The method according to claim 1, wherein a hybrid method is used for the automatic alignment of the image planes to the multimodal marker (4) whose 3D pose can be measured by at least the MRI sensor (2) and the at least one external optical sensor (3) in the MRI scanner (2).

3. The method according to claim 1, wherein simultaneous acquisitions of the marker pose in both modalities are used in order to perform an online and automatic registration of the system during the method, including the estimation of the rigid transformation between the MRI frame and the at least one external optical sensor frame.

4. The method according to claim 3, wherein a new registration is performed automatically when the location of the at least one external optical sensor in an MRI room is changed during the method.

5. A method for scanning the anatomy of a subject, wherein said method incorporates tracking of a stand-alone hybrid marker according to the method of claim 1, the scanning being performed with one or several imaging planes aligned with respect to the multimodal marker.

6. A method for tracking subject motion during MR imaging, wherein said method incorporates tracking of a stand-alone hybrid marker, mounted on the body of the subject or maintained in contact with the body of the subject, according to the method of claim 1, in order to stabilize MR image acquisition with regards to a given anatomical feature of the subject.

7. The method according to claim 1, wherein the at least one external optical sensor (3) is a RGB-D sensor.

8. The method according to claim 1, wherein the multimodal marker (4) is a stand-alone hybrid marker configured to i) interactively locate an entry point on the skin of the subject (6) and angulation towards a targeted area of the instrument or ii) interactively scan the subject anatomy.

9. The method according to claim 1, wherein the multimodal marker (4) is a hybrid marker which is fixed on a surgical instrument (7) in order to automatically track the instrument and automatically align the image planes with respect to the instrument actual path, including the path of the tip or of an end portion of the instrument inside the subject.

10. The method according to claim 1, wherein the data fusion algorithm is an information filter that estimates the multimodal marker pose from the fusion of measurements of both tracking modalities, or from the measurement of a single or remaining tracking modalities when the pose measurement from other tracking modalities is unavailable at a current sampling time.

11. The method according to claim 1, wherein simultaneous acquisitions of the marker pose in both modalities enable online and automatic registration of the system during the method, including the estimation of the rigid transformation between the MRI frame and the at least one external optical sensor frame.

12. The method according to claim 11, wherein a new registration is automatic when the location of the at least one external optical sensor in an MRI room is changed during the method.

13. The method according to claim 1, wherein the multimodal marker contains a skin marking means in order to mark the entry point on the subject's skin.

14. The method according to claim 1, wherein the multimodal marker is mounted on the holding system.

15. The method according to claim 1, wherein the multimodal marker is mounted on the holding system configured to change, and then to maintain a chosen position and angulation of the multimodal marker inside the MRI scanner.

16. The method according to claim 1, wherein the MRI visible feature is a stereotactic marker.

17. The method according to claim 1, wherein the visual feature is a stereotactic target.

Description

(1) In the accompanying drawings:

(2) FIG. 1 is a flow chart of the multimodal automatic tracking and online registration process in a preferred embodiment with RGB-D sensor as external optical sensor;

(3) FIG. 2 is a flow chart of the real-time automatic multimodal tracking, showing typical MR images with the preferred embodiment;

(4) FIG. 3 is an image view illustrating the rigid transformation, i.e. translation and rotation, .sup.MRIM.sub.RGB-D between the MRI spatial coordinate system and the optical sensor spatial coordinate system;

(5) FIG. 4 is a sectional view of an example for a multimodal marker compatible with the invention. The central and cylindrical body is filled with an MRI visible material and the optical sensor detects the 2 spherical and colored features located at the distal ends of the marker;

(6) FIG. 5 is a chronogram for the fusion of MRI and RGB-D pose measurements of the marker in the preferred embodiment. If the high frame rate measurement of the RGB-D sensor is temporarily unavailable due to loss of line-of-sight, update of image planes is based only on MR images, and inversely;

(7) FIGS. 6a to 6c are MR images of the marker during its pose tracking. FIGS. 6b and 6c are respectively sagittal and transversal plane images with an overlay depicting results of the marker position and orientation detection;

(8) FIG. 7 is a schematic representation of the hardware components of the preferred embodiment of the invention: a RGB-D sensor as optical tracker and an external computer. The external computer receives directly the RGB-D images and the MR images through the MRI console. After processing, the predicted image plane orientation and position are sent to the MRI console for acquisition of the following MR image;

(9) FIG. 8 is a more detailed view of the inventive system shown on FIG. 7, also illustrating the signal and data transmissions and treatment and the implemented method steps (as shown on FIGS. 1 and 2).

(10) FIGS. 2, 7 and 8 show a system 1 according to the invention, comprising: an MRI scanner 2, an MRI multi-plane pulse sequence generating means allowing to interactively modify the position and orientation of one or several image planes in real-time, one or several external optical sensors 3 with high frame rate, preferably a RGB-D sensor or an other similar camera system like a stereovision systems, a multimodal marker 4 comprising at least one MR visible feature and one visual feature able to be tracked by both the MRI scanner 2 and the at least one external optical sensor 3, a computer 5 able to process in real-time the images from both MRI and optical sensor in order to fuse the detected marker position and orientation or pose from both modalities, and predict or estimate the next image plane position and orientation based on the estimated motion of the moving marker 4, in order to automatically track said marker 4 and automatically align the image planes with respect to the marker 4, or to the holding system or body location it is fixed on.

(11) The clinical MR imaging sequence that is used during the procedure (scanning, medical intervention, motion tracking) allows to sequentially acquire MR image planes and to change interactively their individual position and orientation. In the preferred embodiment, the MRI marker is directly detected in the clinically used MR images, thus there is not need for additional MRI acquisitions specifically for tracking the marker.

(12) In the context of medical interventions, the instrument, such as a surgical needle, has normally an elongated body, rigid or not, and the marker can be fastened at any specific known location along said body, but preferably near the tip or the end portion of said body, and in any case on a known part of or position along the instrument which is to be located inside the patient during the procedure.

(13) For MR-guided percutaneous interventions, the presented invention can be used mounted on an interventional instrument, in order to automatically align at least two image planes to the main axis of the medical instrument. Typically, two image planes are acquired orthogonal to each other, in order to monitor in real-time the needle advancement inside the patient's body. This set up allows depicting the needle path and the surrounding tissues in a 3D-like manner. This setup also allows to obtain the full 3D pose of the marker with precise spatial information in all directions, one slice providing the missing information on the marker pose in the direction perpendicular to the other image plane.

(14) Alternatively, the image planes of the real-time pulse sequence are also sequentially displaced in order to scan the surrounding anatomy during needle advancement.

(15) As mentioned before, the system and method of the presented invention can be implemented with a stand-alone hybrid marker, in a similar fashion as an ultra sound imaging probe, in order to scan the anatomy with one or several imaging planes aligned with respect to the marker.

(16) Usually, at the beginning of the MR-guided percutaneous procedure, the medical instrument path, from the entry point on the patient skin to the targeted area, is defined by the physician. Typically, this entry point and the adequate path angulation are defined by interactively aligning one or several image planes to the main axis of an MR-visible object.

(17) In this preliminary preparatory phase, the present invention can be implemented with a stand-alone hybrid marker, containing an ink pen or other skin marking device. After an interactive search for the adequate instrument path (entry point and orientation), the entry point could be traced on the patient's skin.

(18) Alternatively in interventional or conventional diagnostic MRI, as well as for patient scanning, the hybrid marker could be used stand-alone, mounted on the patient body, either glued or maintained in contact with the patient body with another holding system (for instance a compliant arm). The presented invention can be used to track physiological (e.g. respiration), voluntary (e.g. limb motion) or unvoluntary (e.g. tremor or patient table motion) patient motion, for instance to stabilize the image acquisition with regards to a given anatomical feature.

(19) The hybrid marker used for tracking has to be detected in both the MR images and in the RGB-D sensor images for the hybrid tracking.

(20) In one embodiment, a cylindrical marker (FIG. 4) filled with an MR contrast agent/water solution (Gd-DTPA 5 mM) and with two colored spherical features attached at its distal ends is used. The MR contrast agent solution and the colored spherical features are well detectable in MR images and RGB images from the RGB-D sensor, respectively. The MR and optical features have a joint main axis, referred in the following as the hybrid marker main axis.

(21) For tracking of the marker pose (orientation and position) in the MR images alone, clinically usable real-time image planes are used during the real-time tracking. In the preferred embodiment, two simple oblique image planes, orthogonal to each other, are aligned to the marker main axis. The tracking procedure typically starts with the acquisition of a simple oblique sagittal plane (Image 1, FIG. 6a). The passive marker is then detected in Image 1, its 3D pose (Pose 1) is computed and the corresponding pose of a new simple oblique transversal image plane (Image 2, FIG. 6b) aligned with the marker is calculated. After acquisition of Image 2, the current pose (Pose 2) of the passive marker is again detected in the image in order to update position and orientation of a new sagittal oblique image plane (Image 3).

(22) From then on, the acquisition alternates between the two orthogonal image planes: typically a sagittal (odd image numbers) and a transversal plane (even image numbers). The detected marker main axis in one of the two image planes is used to update the position and orientation of the following image plane according to the MR feature detection (FIG. 2 with a disabled optical sensor).

(23) These images typically depict the marker held by the physician over the patient's body (FIG. 6). The detection algorithm has thus to be robust against potentially segmented anatomical structures of the patient's body or of the physician's body holding the marker. For that matter, the first step of the detection algorithm is the image segmentation.

(24) A threshold is determined in order to separate image foreground and background. Foreground of the image consists in the patient's body, the marker and the anatomical structures of the person holding the marker.

(25) After labelling of the segmented objects, their geometrical properties, such as position, intensity weighted position and mean intensity, are determined. Further geometrical properties are determined using an ellipse-fitting algorithm calculating orientation, axes length and eccentricity of the ellipses fitted to the segmented objects (cf. A label geometry image filter for multiple object measurement. D. Padfield and J. Miller. The Insight Journal, http://hdl.handle.net/1926/1493, 2008).

(26) The classification of objects is then achieved with a score system based on the previously determined geometrical features. According to their global score, objects are classified as marker (only one is expected) or non-marker objects. The global score is calculated for every segmented object based on its sub-scores for size (A), intensity (B) and eccentricity (C). The global score is the weighted sum of the previously calculated scores: S(i)=0.6.A(i)+0.3 B(i)+0.6 C(i).

(27) According to its global score, an object is classified as marker if its score is above a defined threshold. If more than one object is classified as marker, only the highest score is retained as the marker.

(28) The 3D pose of the marker in the MRI frame (defined in FIG. 3) is then defined by x.sub.pose=(x, y, z, , , ).sup.T with (x, y, z).sup.T the position of the marker center and (, , ).sup.T the roll-pitch-yaw orientation angles of the marker main axis.

(29) From this pose, the pose of the next orthogonal image plane is computed and send to the MRI scanner.

(30) However, pure tracking in the MRI scanner suffers from a slow temporal acquisition rate, around 1 image per seconds, of thin image slices, (slice thickness typically on the order of 5 mm), leading to frequent loss of the passive marker in the image with fast motion during fast marker motion.

(31) In accordance with the invention, the MRI based passive tracking is augmented with an optical tracking, providing a perspective projection measurement and a high temporal frequency acquisition rate to compensate for pure MRI tracking drawbacks. As an external tracker requires registration with the MRI frame, an automatic and online registration is proposed based on their simultaneous respective marker pose measurements.

(32) In the preferred embodiment of the invention, a RGB-D (Red Green Blue-Depth) sensor is used as optical tracker. An RGB-D sensor delivers RGB images and a depth image of an observed scene. It combines an RGB camera with an infrared (IR) projector and an IR camera. The IR projector and camera work together as a depth sensor: an IR speckle dot pattern is projected on a scene and the IR camera captures the reflected speckles. The projected dot pattern and the camera observed dot pattern are compared. Due to a position shift of the observed dot pattern that is a function of the distance of the object to the RGB-D sensor, it is possible to determine the depth of the object.

(33) In order to measure the 3D pose of the marker with the RGB-D sensor, positions of the two colored features are detected in the images and their 3D positions are calculated in the RGB-D camera frame. The RGB and depth images are temporally synchronized and acquired with the higher frame rate of 30 frames per second.

(34) Detection of the two colored features starts with the transformation of the RGB image to Hue Saturation Value (HSV) color space and the application of an experimentally determined threshold. The HSV space revealed as the most intuitive color representation.

(35) The orientation of the marker is computed by defining a vector from the 3D position of the upper feature pointing towards the position of the lower feature. The 3D marker position is defined as the middle position between the two detected features. However, expressed in the RGB-D sensor, this marker pose requires to be transformed in the MRI spatial coordinate system in order to align the image planes on the marker.

(36) Registration consists in the determination of a geometrical relationship between two spatial coordinate systems using a transformation.

(37) An online registration consists in finding and updating the transformation .sup.MRIM.sub.RGB-D between the MRI frame and the RGB-D sensor frame online during the intervention (FIG. 1). The simultaneous acquisitions of the marker pose in both modalities enable the online and automatic registration of the system during the procedure, i.e. the estimation of the rigid transformation between the MRI frame and the vision sensor frame, enabling data fusion and preventing a manual registration procedure.

(38) In the beginning of the tracking procedure, the registration between the MRI frame and the optical sensor spatial coordinate system is unknown. During the automatic online registration step, the instrument tracking is performed using MRI measurements only (directly in the MRI frame). At the beginning of the intervention, a number of matched points (at least three) is acquired in both the MRI frame and the RGB-D sensor frame in order to determine the 3D rigid transformation between the point sets. In our case, a pair of matched points is composed of the marker center coordinates in both modalities and detected during the same RGB-D acquisition period. After successful automatic online registration, the instrument tracking is performed using combined data from both modalities.

(39) An advantage of the chosen approach is that the point sets in both frames are extended and updated during the entire intervention allowing to improve and refine the registration accuracy over time. In addition, the continuous online registration allows to change the position of the RGB-D sensor at any time during a procedure, for instance in order to achieve an optimal line-of-sight between the optical sensor and optical marker inside the MRI scanner. The position and orientation of the optical sensor with respect to the MRI scanner could also unwillingly change during the image-guided procedure, or it could be altered for practical access reasons to the patient or to a given area in the room.

(40) In such cases, change in the relative position and orientation of one sensor to the other is detected by monitoring of the distance between corresponding points. If the euclidien distance between the last set of acquired corresponding points goes above a threshold with the current registration, i.e. current rigid transformation between the 2 frames of reference, a change in the pose of one sensor is assumed. Consequently, the online registration is automatically reset, and started over.

(41) In Vision par ordinateur: outils fondamentaux. R. Horaud and O. Monga. Herms, 2nde dition, May 2011, the registration problem is presented as the search for an optimal correspondence transformation (rotation .sup.MRIR.sub.RGB-D and translation .sup.MRIT.sub.RGB-D) between two 3D point clouds describing the same observations but acquired in different frames. This method for determination of the optimal rotation is based on a work for robot hand/eye registration, presented in A new technique for fully autonomous and efficient 3Drobotics hand/eye calibration. R. Y. Tsai and R. K. Lenz. IEEE Transactions on Robotics and Automation, 5(3):345-358, 1989. They underline that this method has a singular case for a rotation of around one rotation axis.

(42) For this reason and in order to use this approach regardless of its singular case, an avoidance mechanism is implemented in the embodiment of the invention. If the approximate rotation angle is near , the marker positions in frame F.sub.RGB-D are represented in an intermediate frame F.sub.I allowing to avoid the singularity. The rotation according to the method here above is then calculated between F.sub.MRI and F.sub.I. The final rotation between F.sub.MRI and F.sub.RGB-D can then be derived.

(43) The approximate solution to the registration problem is based on the definition of a frame Feig that is linked to the 2 point clouds in MRI and RGB-D frames. Three non-aligned matched points are used for definition of this frame with the vectors

(44) $u_{\mathrm{RGB}-D}=\frac{O_1{O}_2}{.Math.O_1{O}_2.Math.},w_{\mathrm{RGB}-D}=\frac{O_1{O}_3}{.Math.O_1{O}_3.Math.},v_{\mathrm{RGB}-D}=w_{\mathrm{RGB}-D}{u}_{\mathrm{RGB}-D}$

(45) $u_{\mathrm{MRI}}=\frac{O_1{O}_2}{.Math.O_1{O}_2.Math.},w_{\mathrm{MRI}}=\frac{O_1{O}_3}{.Math.O_1{O}_3.Math.},v_{\mathrm{MRI}}=w_{\mathrm{MRI}}{u}_{\mathrm{MRI}}$

(46) The rotation matrices .sup.RGB-DR.sub.eig=[u.sub.RGB-D, v.sub.RGB-D, w.sub.RGB-D] and .sup.MRIR.sub.eig=[u.sub.MRI, v.sub.MRI, w.sub.MRI] represent the rotation between Feig and the frames F.sub.RGB-D and F.sub.MRI, respectively. The approximate rotation matrix between the frames F.sub.MRI and F.sub.RGB-D can thus be calculated with
.sup.MRIR.sub.RGB-D_approx=.sup.MRIR.sub.eig(.sup.RGB-DR.sub.eig).sup.T

(47) One can notice that this method is referred to as approximate as it relies only on three matched points instead of all the available points and is thus highly sensitive to measurement errors on the chosen points. If the rotation angle in the axis-angle representation of the rotation matrix is .sup.MRIR.sub.RGB-D_approx within 0.2 radian near the aforementioned method is near its exceptional case. In order to avoid the singularity, the point cloud of RGB-D marker position is represented in an intermediate frame F.sub.I, rotated by around the approximate rotation axis. For this purpose the approximate rotation axis is multiplied by and then converted to the rotation matrix .sup.IR.sub.RGB-D. The marker position clouds O in RGB-D frame are then represented in the frame FI with
.sup.IO=.sup.IR.sub.RGB-DO.

(48) The rotation matrix .sup.MRIR.sub.I is then searched between the marker positions clouds C and .sup.IO according to the previously quoted Horaud and Monga publication. Eventually, the final rotation matrix between the point clouds C and O can be calculated by accounting for the intermediate frame in the following manner:
.sup.MRIR.sub.RGB-D=.sup.MRIR.sub.I .sup.IR.sub.RGB-D

(49) The translation .sup.MRIT.sub.RGB-D between the two frames is then obtained from the previously quoted Horaud and Monga publication with the previously estimated rotation.

(50) Continuous update of the registration is then achieved by updating the point list of matched points between the 2 frames used in the estimation of the transformation .sup.MRIM.sub.RGB-D described above.

(51) New matching points are verified with respect to their distance to the MRI isocentre and the distance to the other points in the list. If the list is already full, it is tested if the new point would improve spatial distribution of the point list. If so, an existing point in the list will be replaced by the arriving point.

(52) As soon as enough matching points are available and a frame transformation is estimated, pose measurements of the RGB-D sensor can be expressed in the MRI frame and used for tracking.

(53) During the online registration (beginning of the procedure or motion of the optical sensor relative to the MRI scanner), the MR feature tracking measurement alone is available and used for marker tracking. In such case, if there is no current knowledge on the marker position and orientation (beginning of the procedure or marker loss), an MR tracking initialization step is necessary. This step consists in acquiring a dedicated, typically transversal, projection image (FIG. 6a) at the MRI scanner isocentre, or centered on the last known marker position. This projection image has a large slice thickness (typically 100 to 600 mm) compared to clinical real-time MR-guidance images (typically 5 mm). The image dimension and its slice thickness provide an initial search volume within the MRI-scanner. The MR feature and all anatomical structures of the patient lying within this volume are thus superimposed in a single projection image in said volume. As a consequence, these projection acquisitions are typically not clinically usable, and therefore referred to as dedicated image acquisition. The MR feature is detected in the projection image in a similar fashion as in the real-time clinical images, and its position and orientation are calculated.

(54) Prediction of the marker pose for image plane alignment according to available measurements is then efficiently achieved with data fusion in an information filter (FIG. 1).

(55) The information filter is a formulation of the Kalman filter that uses information measurements to represent the states of the system. The implementation of the Information filter proposed in A formally verified modular decentralized robot control system. A. G. O. Mutambara and H. F. Durrant-Whyte. In Proceedings of the 1993 IEEE/RSJ International Conference on Intelligent Robots and Systems '93, IROS '93, volume 3, pages 2023-2030 vol. 3, 1993, and in Decentralized Estimation and Control for Multisensor Systems. A. G. O. Mutambara. CRC Press Inc, January 1998 is used in the preferred embodiment of the invention.

(56) One common model used with tracking is the constant velocity assumption. The motion model is then given by
x(k)=A.Math.x(k1)+n(k1)
with

(57) $x\left(k\right)=\left[\begin{array}{c}{x}_{\mathrm{pose}}\left(k\right)\\ {\stackrel{.}{x}}_{\mathrm{pose}}\left(k\right)\end{array}\right],$
the marker pose (position and orientation) and its derivative states
and

(58) $A=\left[\begin{array}{cc}I& t.Math.I\\ 0& I\end{array}\right],$
the transition matrix with t the sampling time of the optical sensor and n(k) a white noise.

(59) As the marker pose is directly measured in both the MR and RGB-D images, their observation matrices in the measurement equation are equal:
H=H.sub.MRI=H.sub.RGB-D=[I 0]
z(k)=H.Math.x(k)+m(k)

(60) The white noises on measurements m.sub.RGB-D(k) and m.sub.MRI(k) have respectively a noise covariance matrix denoted Q.sub.MRI and Q.sub.RGB-D.

(61) Therefore, if is the information state of the filter and is defined by =Y{circumflex over (x)} with Y the information matrix, then its estimate at current time sample k is given by:
Y(k|k)=Y(k|k1)+I.sub.RGB-D(k)+I.sub.MRI(k)
Y(k|k)=Y(k|k1)+H.sup.TQ.sub.MRI.sup.1H+H.sup.TQ.sub.RGB-DH
(k|k)={circumflex over (y)}(k|k1)+i.sub.MRI+i.sub.RGB-D
(k|k)=(k|k1)+H.sup.TQ.sub.MRI.sup.1z.sub.MRI+H.sup.TQ.sub.RGB-D.sup.1z.sub.RGB-D
with the local information state contribution i.sub.MRI, i.sub.RGB-D and z.sub.MRI, z.sub.RGB-D the marker pose measurement respectively by the MRI and RGB-D sensors.

(62) If one modality measurement is not available at the current time step, its local contribution state is simply removed from the sum used for the estimation (FIG. 5). The prediction step associated with the estimation step is given by:
(k+1|k)=L(k+1|k)(k|k)
Y(k+1|k)=[A Y.sup.1(k|k)A.sup.T+Q(k+1)].sup.1
with
L(k+1|k)=Y(k+1|k)A Y.sup.1(k|k)

(63) Prediction of the marker pose is retrieved from the information filter prediction
{circumflex over (x)}(k+1|k)=Y.sup.1(k+1|k)

(64) based on the constant velocity model and current available measurement (MRI/RGB-D). It is then used to compute the alignment of the image planes for the next acquisition.

(65) The system runs periodically, with a period equal to the highest sensor acquisition framerate, i.e. optical sensor in the preferred embodiment.

(66) More precisely, in accordance with a preferred embodiment of the invention, the method for multi-sensor tracking and online registration runs periodically the following steps 1 to 8:

(67) Step 1 (Acquisition and image processing)

(68) Marker pose measured respectively in the MRI frame and in the optical frame, depending on their respective frame rates.

(69) If over the time period, matching marker poses are available from both modalities, a new pair of matched points between the two frames is recorded.

(70) Step 2 (Online registration) If a new pair of matched points

(71) If the pair of matched points improves the estimation of the rigid transformation between the 2 frames, it is added to the current list of matched points. If so, an optimization algorithm provides an updated estimation of the rigid transformation between the two frames based on the updated list of matched points.

(72) Step 3 (Frame transformation) The current rigid transformation is used to translate to the MRI frame the marker pose in the optical frame, so that measured marker poses in both modalities are now expressed in the MRI frame.

(73) Step 4 (Fusion)

(74) A data fusion algorithm, for instance an information filter, is used to filter the available pose(s) and estimate the optimal pose of the marker considering a model of the tracking process and its noise.

(75) Step 5 Pose Prediction using the data fusion algorithm

(76) Step 6 Calculation of the corresponding image plane position and orientation encompassing the marker

(77) Step 7 Predicted image plane position and orientation sent to the MRI console to update the image plane acquisition.

(78) Thus, as described before, the invention aims at automatic image plane alignment to a moving instrument in order to accelerate and improve the accuracy of medical interventions under MRI-guidance.

(79) The presented system and methods of the invention intends to suppress the requirement for time-consuming manual alignment of the image plane to the marker or instrument pose. The presented invention also accelerates the procedure thanks to its automatic online registration, therefore not requiring a prior registration step before the procedure starts (FIGS. 1 and 8).

(80) A hybrid marker composed of one feature visible in the MR images and visual features detected by an external optical sensor, preferably a RGB-Depth sensor, is tracked in real-time. The current pose (position + orientation) of the moving marker is measured using both MRI and optical sensors, and the MR image planes are automatically aligned to the predicted pose at the time of the following MRI acquisition. The marker can be used stand-alone for dynamic scanning of the anatomy, and can also be used in order to localize the optimal entry point and path angulation for the interventional instrument prior to the intervention. Placed on the surgical instrument, the marker enables an automatic alignment of the image planes to the instrument axis in order to obtain a 3D-like view of the instrument path inside the patient's body towards the targeted area, and of the surrounding anatomy.

(81) By fusion, as with an information filter, of both MRI scanner and optical sensor measurements, the pose of the marker is estimated even if one of these two modalities is not available. Pose estimation is retrieved by image processing on an external computer that accordingly drives the position and orientation of acquired MRI image planes based on the prediction.

(82) Simultaneous tracking in both modalities enables the continuous online registration of the system, i.e. estimation of the rigid transformation between the MRI frame and the optical sensor frame. It thus automatically allows for new registration if the location of the optical sensor is changed relative to the MRI scanner during an image-guided procedure. Therefore it becomes possible to change the optical sensor position in order to achieve the best line-of-sight between the marker and the optical sensor with automatic new registration of the system.

(83) As mentioned before, the inventive method comprises a priori image alignment with no update of the magnetic field gradients during the image acquisition time, nor adaptation of the image reconstruction to motion during the image acquisition.

(84) The present invention is of course not limited to the preferred embodiments described and represented herein, changes can be made or equivalents used without departing from the scope of the invention.