Method of computing a boundary

Abstract

The disclosure relates to a method for determining a boundary about an area of interest in an image set. The includes obtaining the image set from an imaging modality and processing the image set in a convolutional neural network. The convolutional neural network is trained to perform the acts of predicting an inverse distance map for the actual boundary in the image set; and deriving the boundary from the inverse distance map. The disclosure also relates to a method of training a convolutional neural network for use in such a method, and a medical imaging arrangement.

Claims

1. A method of determining a boundary about an area of interest in an image set, the method comprising: obtaining the image set from an imaging modality; and processing the image set in a convolutional neural network, wherein the convolutional neural network has been trained through: predicting an inverse distance map for an actual boundary in the image set by blurring the actual boundary by a Gaussian function; and deriving the boundary from the inverse distance map.

2. The method of claim 1, wherein the deriving of the boundary from the predicted inverse distance map comprises thresholding the inverse distance map to obtain a binary boundary and subsequently extracting a medial axis of the binary boundary.

3. The method of claim 1, wherein the boundary computed by the convolutional neural network is an open boundary or a closed surface boundary.

4. The method of claim 1, wherein the convolutional neural network is configured to perform semantic image segmentation.

5. The method of claim 1, wherein the convolutional neural network implements a U-Net architecture comprising four layers comprising encoder blocks and decoder blocks, wherein each encoder block comprises a plurality of convolution filters, and wherein each decoder block comprises a plurality of deconvolution filters.

6. The method of claim 5, wherein the image set comprises a plurality of two-dimensional (2D) images, wherein the inverse distance map is predicted for a two-dimensional surface boundary about an area of interest in a 2D image of the plurality of 2D images, and wherein a two-dimensional boundary is inferred from the inverse distance map.

7. The method of claim 5, wherein the image set comprises a three-dimensional (3D) image volume, wherein the inverse distance map is predicted for a three-dimensional surface boundary about an area of interest in the 3D image volume, and wherein the three-dimensional surface boundary is inferred from the inverse distance map.

8. The method of claim 7, further comprising, prior to predicting the inverse distance map: generating the 3D image volume from a plurality of image slices obtained from a 2D imaging modality.

9. The method of claim 1, wherein the image set comprises a plurality of two-dimensional (2D) images, wherein the inverse distance map is predicted for a two-dimensional surface boundary about an area of interest in a 2D image of the plurality of 2D images, and wherein a two-dimensional boundary is inferred from the inverse distance map.

10. The method of claim 1, wherein the image set comprises a three-dimensional (3D) image volume, wherein the inverse distance map is predicted for a three-dimensional surface boundary about an area of interest in the 3D image volume, and wherein the three-dimensional surface boundary is inferred from the inverse distance map.

11. The method of claim 10, further comprising, prior to predicting the inverse distance map: generating the 3D image volume from a plurality of image slices obtained from a 2D imaging modality.

12. The method of claim 1, wherein the imaging modality is intracardiac echocardiography.

13. A method of training a convolutional neural network for use in determining a boundary about an area of interest in an image set, the method comprising: annotating the image set to identify the boundary about the area of interest in the image set; replicating an inverse distance map of the boundary for use as a ground truth by the convolutional neural network, wherein the replicating of the inverse distance map of the boundary is performed by blurring the boundary by a Gaussian function; applying the convolutional neural network to the image set and the ground truth to predict the inverse distance map approximating the ground truth; and repeating the annotating, the replicating, and the applying until a desired level of accuracy has been achieved.

14. The method of claim 13, wherein the convolutional neural network is configured to minimize a mean squared error between the ground truth and the predicted inverse distance map.

15. A medical imaging arrangement comprising: a processor comprising a convolutional neural network loaded into a memory of the processor, wherein the processor is configured to: obtain an image set and process the image set in the convolutional neural network, and wherein the convolutional neural network has been trained by predicting an inverse distance map for an actual boundary in the image set by blurring the actual boundary by a Gaussian function and deriving a computed boundary from the inverse distance map; and a user interface configured to display the computed boundary in a context of the image set.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Other objects and features of the present disclosure will become apparent from the following detailed descriptions considered in conjunction with the accompanying drawings. It is to be understood, however, that the drawings are designed solely for the purposes of illustration and not as a definition of the limits of the disclosure.

(2) FIG. 1 illustrates an example of a training stage of the method.

(3) FIG. 2 shows a simplified block diagram of an example of a convolutional neural network used in the method.

(4) FIG. 3 is an example of a 2D representation of the development of a replication of an inverse distance map.

(5) FIG. 4 illustrates an example of a workflow when applying the method to determine a 3D surface boundary.

(6) FIG. 5 is an example of a 2D representation of stages in the method.

(7) FIG. 6 is an example of a 2D projection of an exemplary outcome of the method.

(8) FIG. 7 compares an exemplary outcome of the method with a corresponding manual annotation.

(9) FIG. 8 shows an embodiment of the medical imaging arrangement.

(10) In the diagrams, like numbers refer to like objects throughout. Objects in the diagrams are not necessarily drawn to scale.

DETAILED DESCRIPTION

(11) The diagrams illustrate exemplary embodiments using images of a 3D Intracardiac Echocardiography (ICE) image volume, but it shall be understood that the method may be deployed to infer a boundary about a region of interest in a 2D image. Furthermore, it shall be understood that the method may be deployed for other imaging modalities. For example, a set of images from a 2D imaging modality may be assembled into a sparse 3D volume for use as an input to the training network. A 3D volume is considered sparse in the case of incomplete sampling of all voxel locations. While the following diagrams may only show images in the plane of the page, it shall be understood that the method is performed on a 3D volume and that any image shown in a diagram is only one “slice” of a 3D volume.

(12) FIG. 1 illustrates stages in a training stage of the method. In a first stage 30, a 3D volume M is obtained for an imaging modality such as ICE. A boundary mesh B.sub.manual is identified for a surface boundary, for example an interior or exterior surface of an organ, the interior surface of a blood vessel, etc. This act may be done by a trained expert. In act 31, the surface boundary B.sub.manual thus identified is then subject to Gaussian filtering as explained above, with a suitable choice of standard deviation a. In act 32, the blurred boundary mesh M.sub.gauss and the 3D volume M are then fed into a suitable convolutional neural network 1, which learns to infer an inverse distance map M.sub.infer while minimizing the difference between the input ground truth M.sub.gauss and the inferred distance map M.sub.infer. Training is repeated for a plurality of ground truths M.sub.gauss until a desired accuracy is obtained.

(13) FIG. 2 illustrates an exemplary embodiment of a U-Net architecture used to train a model to identify a 3D boundary. The convolutional neural network 1 includes four encoder blocks 10 and four decoder blocks 11. Each encoder block 10 includes several convolution filters 10F, and each decoder block 11 includes several deconvolution filters 11F. The filters 10F, 11F may vary in number. For example, the first, second, third and fourth blocks may implement 32, 64, 128, and 196 filters, respectively. Each filter 10F, 11F may be 3×3×3 in size. Each encoder block 10 is followed by a 2×2×2 average pooling layer 10P, while each decoder block 11 is preceded by a bilinear upsampling 11U layer.

(14) As explained above, this convolutional neural network 1 is trained on a non-binary ground truth representation, (e.g., non-binary representations of boundaries are used to train the convolutional neural network).

(15) In an exemplary embodiment, the ground truth is an inverse distance map. This stage of the training is illustrated with FIG. 3. A 3D image volume is presented to a user (e.g. in the display of a graphical user interface or GUI). The user identifies a structure, (e.g., a heart cavity or cardiac chamber), and annotates the 3D volume with a boundary mesh B.sub.manual of the structure. The annotated boundary mesh B.sub.manual may be regarded as a set of points in Cartesian space, (e.g., a distance map as expressed by equation (1)). The boundary mesh is then blurred by passing it through a Gaussian filter as explained above with the aid of equation (2). The purpose of the Gaussian filtering is to mimic or replicate an inverse distance map for the boundary. A replication of the inverse distance map is used as a ground truth for training.

(16) The drawing indicates an image 30 shown in the display 21, and any such image may be assumed to be similar to the 3D ICE images shown in FIG. 5 and FIG. 7. The drawing shows a slice through the manually annotated boundary mesh B.sub.manual; a slice through a filtered boundary mesh M.sub.gauss after blurring by a Gaussian filter with σ=1; and a slice through a filtered boundary mesh M.sub.gauss after blurring by a Gaussian filter with σ=2. The value of σ determines the smoothness of the filtered boundary.

(17) The task of the convolutional neural network 1 of FIG. 2 is to learn how to identify a surface boundary without first computing the enclosed volume. The forward pass of the convolutional neural network 1 estimates the 3D boundary through a 3D distance map. The network training aims to minimize a loss function between the predicted distance map M.sub.infer and the ground truth distance map M.sub.gauss. As explained above, a suitable choice of loss function may be the mean squared error of equation (3) above. The convolutional neural network 1 is trained using multiple ground truths. A high degree of accuracy of the model may be obtained by training with several hundred ground truths. This compares favorably with the conventional methods, which may require several thousand annotated segmentation masks in order to train a neural network to a comparable level of accuracy.

(18) After training is complete, the model 1 may be applied to detect or infer boundary surfaces in a 3D image volume. In inference mode, the model 1 predicts a distance map M.sub.infer from the surface boundary about an area of interest, after which thresholding and skeletonization acts are performed to arrive at the predicted surface boundary.

(19) FIG. 4 illustrates a workflow when applying the method to determine a 3D surface boundary. Here, an image volume M is obtained, for example, an image volume of an ICE procedure. In act D, the clinician identifies an area of interest, for example, a chamber or a blood vessel of the heart. It may be sufficient for the clinician to position a cursor over the area of interest, (e.g., “inside” the cavity shown on the monitor), so that the coordinates of the cursor relative to the image may be noted. Of course, the area of interest may be identified by the clinician or user in any suitable manner. In act E, the 3D volume M and any information identifying the area of interest are then fed to the model 1 which has been trained as explained above and illustrated in FIG. 2. The model 1 infers the boundary M.sub.infer of the surface enclosing the area of interest and this inferred boundary is then refined in act F by thresholding and skeletonization to obtain a 3D surface boundary B.sub.infer which may then be shown to the user, for example, by overlaying the inferred boundary onto the original image data.

(20) FIG. 5 is a 2D representation of stages in the method and shows a 2D “slice” 30 of a 3D ICE volume to illustrate some of the main method acts. The area of interest is the large cavity which may be seen in the upper region of the image 30, in this case, the left atrium of the heart. It is the surface of the 3D cavity which is of interest to the clinician.

(21) The trained convolutional neural network 1 described in FIG. 2 above predicts a 3D inverse distance map M.sub.infer for the cavity surface. The predicted 3D inverse distance map M.sub.infer is visualized here by projecting it onto the 2D image slice 30. The thickness of the 3D inverse distance map M.sub.infer may be controlled by choosing a suitable value of standard deviation σ in equation (2).

(22) The inverse distance map M.sub.infer is then thresholded to obtain a binary boundary B.sub.binary, (e.g., a set of values that either belong to the boundary or do not belong to the boundary). An exemplary result of thresholding the distance map M.sub.infer is superimposed on the image 30 and shows a band B.sub.binary representing the values that are deemed to belong to the boundary. The “thickness” of the binary boundary B.sub.binary depends on the choice of threshold, and a threshold value may be chosen that will provide a non-fragmented boundary.

(23) In a final stage, the surface boundary B.sub.infer is refined by performing a skeletonization act (or “thinning”) on the binary boundary B.sub.binary to extract the medial axis. The 3D surface boundary B.sub.infer may then be presented to the user by a suitable graphics program. Alternatively, as indicated here, a slice through the surface boundary B.sub.infer may be superimposed on the corresponding slice 30 of the 3D volume and shown to the user. Once the 3D surface boundary B.sub.infer is established for the area of interest, the user may interact with the imaging modality to alter the viewing angle, and the image presented on the monitor is continually updated to show the correct slice through the surface boundary B.sub.infer.

(24) FIG. 6 is a 2D projection of an exemplary outcome of the method. The diagram indicates various planes 40, . . . , 40n, each of which corresponds to a 2D image from which a sparse 3D volume is compiled. Even though the 3D volume contains relatively little information, the method is able to identify a surface boundary B.sub.infer for a cavity in an organ imaged using a modality such as ICE. To provide such a 3D volume when only 2D images are available, a suitable form template (e.g., from a library of shapes) may be chosen which best fits a boundary shown in the images. A rough mesh may be derived from the template, and this rough mesh may then be used in a training act for the neural network.

(25) FIG. 7 illustrates the effectiveness of the method. The diagram shows a 2D slice 30 of a 3D ICE volume. The upper part of the diagram shows a 2D projection of a 3D ground truth boundary mesh B.sub.manual for the left atrium, manually annotated in the image 30 by an expert. This ground truth B.sub.manual is used in a training act to train the convolutional neural network 1 of FIG. 2. The lower part of the diagram shows a 2D projection of the 3D left atrium boundary surface B.sub.infer as predicted or inferred by the convolutional neural network 1.

(26) FIG. 8 shows an exemplary medical imaging arrangement 2 with an imaging module 20, in this case a 3D echocardiography device configured to provide a 3D ICE volume M from ultrasound data 200 received from an ultrasound probe. After insertion of a catheter carrying an intracardiac ultrasound probe, a clinician 80 may view the interior regions of the heart on a display of a user interface 21, for example, in order to arrive at a diagnosis for the patient 81. This exemplary embodiment of the medical imaging arrangement 2 includes a processing unit 22 that is configured to infer a 3D surface boundary B.sub.infer for any area of interest in the 3D ICE volume M. The processing unit 22 includes a convolutional neural network 1 trained as explained above. The user interface 21 may display the computed or inferred three-dimensional surface boundary B.sub.infer in the context of the 3D ICE volume. Because the CNN 1 may very quickly infer a 3D surface boundary for any cardiac structure, the clinician 80 may view the inferred surface “live” during the ICE imaging procedure. The various components are shown separately in this exemplary schematic, but it shall be understood that the imaging module 20, user interface 21 and processing unit 22 may be realized collectively as a stand-alone unit.

(27) Although the present disclosure has been discussed in the form of certain embodiments and variations thereon, it will be understood that numerous additional modifications and variations may be made thereto without departing from the scope of the disclosure. For example, when inferring a boundary from a 2D image, the architecture of the CNN is based on two-dimensional filters, and the ground truths used to train the CNN may be simple contours (e.g., open or closed) obtained by manual annotation of 2D images.

(28) For the sake of clarity, it is to be understood that the use of “a” or “an” throughout this application does not exclude a plurality, and “comprising” does not exclude other acts or elements. The mention of a “unit” or a “module” does not preclude the use of more than one unit or module.

(29) It is to be understood that the elements and features recited in the appended claims may be combined in different ways to produce new claims that likewise fall within the scope of the present disclosure. Thus, whereas the dependent claims appended below depend from only a single independent or dependent claim, it is to be understood that these dependent claims may, alternatively, be made to depend in the alternative from any preceding or following claim, whether independent or dependent, and that such new combinations are to be understood as forming a part of the present specification.

(30) While the disclosure has been illustrated and described in detail with the help of the disclosed embodiments, the disclosure is not limited to the disclosed examples. Other variations may be deducted by those skilled in the art without leaving the scope of protection of the claimed disclosure.