MULTI-TASK DEEP LEARNING METHOD FOR A NEURAL NETWORK FOR AUTOMATIC PATHOLOGY DETECTION

Abstract

Multi-task deep learning method for a neural network for automatic pathology detection, comprising the steps: receiving first image data (I) for a first image recognition task; receiving (S2) second image data (V) for a second image recognition task; wherein the first image data (I) is of a first datatype and the second image data (V) is of a second datatype, different from the first datatype; determining (S3) first labeled image data (I.sub.L) by labeling the first image data (I) and determining second synthesized labeled image data (I.sub.SL) by synthesizing and labeling the second image data (V); training (S4) the neural network based on the received first image data (I), the received second image data (V), the determined first labeled image data (I.sub.L) and the determined second labeled synthesized image data (ISL); wherein the first image recognition task and the second image recognition task relate to a same anatomic region where the respective image data is taken from and/or relate to a same pathology to be recognized in the respective image data.

Claims

1. A multi-task deep learning method for a neural network for automatic pathology detection, comprising: receiving first image data for a first image recognition task; receiving second image data for a second image recognition task; wherein the first image data is of a first datatype and the second image data is of a second datatype, different from the first datatype; determining first labeled image data by labeling the first image data and determining second labeled synthesized image data by synthesizing and labeling the second image data; training the neural network based on the received first image data, the received second image data, the determined first labeled image data and the determined second labeled synthesized image data; wherein the first image recognition task and the second image recognition task relate to a same anatomic region where the respective image data is taken from and/or relate to a same pathology to be recognized in the respective image data.

2. The method of claim 1, further comprising: determining synthesized second image data by synthesizing the second image data into the first datatype and/or first dimension; and determining the second labeled synthesized image data by labeling the synthesized second image data.

3. The method of claim 1, further comprising: determining second labeled image data by labeling the second image data; and determining the second labeled synthesized image data by synthesizing the second labeled image data into the first datatype and/or dimension of the first image data.

4. The method of claim 1, wherein the first image recognition task and/or the second image recognition task comprises at least one of classification, localization, object detection and/or segmentation.

5. The method of claim 4, wherein the classification, the localization, the object detection and/or the segmentation relates to at least one of a pathology, a foreign object and/or an anatomical region.

6. The method of claim 1, wherein the first datatype comprises 2D image data, and wherein the second datatype comprises 3D image data.

7. The method of claim 1, wherein the first image data is of a first dimension and the second image data is of a second dimension, different from the first dimension.

8. The method of claim 1, wherein the first labeled image data and the second labeled synthesized image data comprise at least one of an identification and/or a location of a pathology, a foreign object and/or an anatomical region.

9. The method of claim 1, wherein the first image data comprises X-ray image data, and wherein the second image data comprises computer tomography image data.

10. The method of claim 9, wherein the computer tomography image data is synthesized into 2D image data in form of a digitally reconstructed radiograph.

11. The method of claim 1, wherein the pathology comprises at least one of Cardiomegaly, Emphysema, Edema, Hernia, Pneumothorax, Effusion, Masses, Fibrosis, Atelectasis, Consolidation, Pleural Thickening, Nodules, and Pneumonia.

12-14. (canceled)

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Exemplary embodiments of the invention will now be described with reference to the following drawings, which are not to scale, wherein:

[0061] FIG. 1 shows a schematic block diagram of the device executing the multi-task deep learning method;

[0062] FIG. 2 shows a schematic block diagram of an application of the trained deep learning model; and

[0063] FIG. 3 shows a flow chart of the multi-task deep learning method for a neural network for automatic pathology detection.

DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] The device 10, described in FIG. 1 comprises a synthesis unit 20, a labeling unit 30 and a model training unit 40. The device 10 is provided with first image data in the form of 2D image data I and second image data in form of 3D volume data V. The 3D volume data V is the result of a first image recognition task, in this case from detection of a pneumothorax in X-ray imaging. The 2D image data I is the result of the second image recognition task, in this case from detection of a pneumothorax in computer tomography imaging. The first image recognition task and the second image recognition task are related to each other, in particular in view of relating to a same anatomic region where the respective image data is taken from and/or relating to a same pathology to be recognized in the respective image data.

[0065] In this case, the 3D volume data V and the 2D image data I are not of the same image datatype. In order to be mutually processed, the image datatype of the 3D volume data V and the 2D image data I have to be of the same image datatype, or at least a similar image datatype. Preferably the respective image datatype are similar in view of pixel resolution and/or synthesis geometry. Thus, the 3D volume data V had to be synthesized in to an image datatype that is similar to the image datatype of the 2D image data I. Therefore, the 3D volume data V is provided to the synthesis unit 20. The synthesis unit 20 synthesizes the 3D volume data V into an image datatype that is close to the image datatype of the 2D image data I in view of pixel resolution and synthesis geometry by synthesizing the 3D volume data V into a plurality of 2D image data. In other words, the 3D volume of the 3D volume data V is sliced into a stack of 2D image data. In this case, the synthesis of the 3D volume data V from the computer tomography image is called digitally reconstructed radiograph, DDR. Thus, the 3D volume data V is synthesized in to synthesized 2D image data I.sub.S. The synthesis 2D image data I.sub.PS is then provided to the labeling unit 30.

[0066] In image recognition basically two image recognition tasks are performed, namely segmentation and/or classification. Segmentation relates to an annotation of labels to pixels, wherein classification relates to an annotation of labels to images. Thus, in any case the provided image data has to be labeled. Therefore the labeling unit 30 is not only provided with the synthesized 2D image data I.sub.S, but although with the 2D image data I. The labeling unit 30 then labels the provided image data, in particular by performing the first image recognition task on the 2D image data I and by performing the second image recognition task on the synthesized image data I.sub.S, determining labeled 2D image data IL and labeled synthesized 2D image data I.sub.SL. Thus, different architectures are possible. A first architecture comprises multi-task-learning for X-ray classification and computer tomography classification. The second architecture comprises multi-task-learning for x-ray segmentation and computer tomography segmentation. A third architecture comprises multi-task-learning for x-ray segmentation and computer tomography classification. A fourth architecture comprises multi-task-learning for x-ray classification and computer tomography segmentation

[0067] The model training unit 40 is then provided with the determined labeled 2D image data I.sub.L, the labeled synthesized 2D image data I.sub.SL, the 2D image data I and the synthesized 2D image data I. The model training unit 40 then trains a model M with a neural network in particular by multi-task deep learning methods, based on the provided determined labeled 2D image data I.sub.L, the labeled synthesized 2D image data I.sub.SL, the 2D image data I and the synthesized 2D image data I.sub.S. Thus, the x-ray images and the synthesized computer tomography images are used to train a network architecture with shared layers. In this case, such an approach highly increases the available training data for the neural network. Since the training data, which in this case is based on the provided 3D volume data V and the provided 2D image data I, was generated on related image recognition tasks, in this case related in view of the anatomic region and the task of detecting the same pathology, increased amount of training data is available which also enhances the accuracy of the annotation of the image data.

[0068] In general, in order to leverage the advantages of multi-task learning by means of combined training with computer tomography data and x-ray data, a hard parameter shearing approach can be used, wherein the first layers of the neural network are reused for multiple tasks. For such an application, the computer tomography data has to be adapted. This can be achieved by selecting single slices out of the computer tomography volume and resize them by interpolation or by an artificial forward synthesizes to a 2D image with the correct dimensions. Alternatively, using a soft sharing approach, every network has its own parameters, which allows also for different datatypes, while a regularization method is employed in order to encourage joined learning.

[0069] As illustrated in FIG. 2, the trained model M can then be used in further applications. Therefore, model application unit 50 is provided with the trained model M. For example, the image recognition task that should be performed by the model application unit 50 comprises detection of a pneumothorax in 2D image data I generated from X-ray imaging. Thus, the 2D image data I is provided to the model application unit 50. The trained model M was trained on a relatively high number of training data, which in addition have an improved accuracy, since the model was not only trained on training data of pathology detection in 2D X-ray-imaging but also on training data of pathology detection in a similar area of the body in 3D computer tomography imaging. Thus, the model application unit 50 is able to determine labeled 2D image data I.sub.L by annotating the provided 2D image data I in an improved way.

[0070] FIG. 3 shows a flow chart of the multi-task deep learning method for a neural network for automatic pathology detection. In a first step S1, first image data I for a first image recognition task is received. In a second step S2, second image data V for a second image recognition task is received wherein the first image data I is of a first datatype and the second image data V is of a second datatype, different from the first datatype. The first image recognition task and the second image recognition task relate to a same anatomic region where the respective image data is taken from and/or relate to a same pathology to be recognized in the respective image data. In a third step S3 first labeled image data I.sub.L is determined by labeling the first image data I and second labeled synthesized image data I.sub.SL is determined by synthesizing and labeling the second image data V. In a fourth step S4, the neural network is trained based on the received first image data I, the received second image data V, the determined first labeled image data I.sub.L and the determined second labeled synthesized image data I.sub.SL.

LIST OF REFERENCE SIGNS:

[0071] 10 device [0072] 20 synthesis unit [0073] 30 labeling unit [0074] 40 model training unit [0075] 50 model application unit [0076] V 3D volume data [0077] I 2D image data [0078] I.sub.S synthesized 2D image data [0079] I.sub.L labeled 2D image data (first labeled image data) [0080] I.sub.SL labeled synthesized 2D image data (second labeled synthesized image data) [0081] M model [0082] S1 receiving first image data [0083] S2 receiving second image data [0084] S3 determining first labeled image data [0085] S4 training the neural network