Method of Augmenting the Number of Labeled Images for Training a Neural Network

Abstract

A method of augmenting the number of labeled images for training a neural network comprising the steps of—Starting from a dataset of labeled images with corresponding segmentation masks and a dataset of unlabeled images, gathering for a given image i in a data set of labeled images a number of images with similar metadata in said dataset of unlabeled images so as to form data sub-set Sim i,—Training a multiclass segmentation neural network on said labeled images thereby generating segmentation masks for the images in subset Sim i,—On the basis of these segmentation masks judging similarity between images of Sim i and image i and finding the most similar image(s) in Sim i by computing and comparing histograms of segmentation masks of image i and images in Sim i—Transferring the histogram of the most similar images in Sim i to given image i.

Claims

1-3. (canceled)

4. A method of augmenting the number of labeled images for training a neural network, the method comprising: starting from a dataset of labeled images with corresponding segmentation masks and a dataset of unlabeled images, gathering for a given image i in a data set of labeled images a number of images with metadata that have at least one item with the same value in said dataset of unlabeled images so as to form a data sub-set Sim i, training a multiclass segmentation neural network on said labeled images thereby generating segmentation masks for the images in sub-set Sim i, on the basis of these segmentation masks judging similarity between images of Sim i and image i and finding most similar image(s) in Sim i by computing histograms of segmentation masks of image i and images in Sim i and by comparing them, and transferring the histogram of the most similar images in Sim i to given image i.

5. The method of claim 4, wherein said image i and images in said sub-set Sim i are registered on top of each other before said histograms are compared.

6. The method of claim 5, wherein as a postprocessing step an image is sorted out when use of such an image would render an overexposed or underexposed result.

7. The method of claim 4, wherein as a postprocessing step an image is sorted out when use of such an image would render an overexposed or underexposed result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0026] Although the invention will be explained with reference to segmentation task of medical images, it is not limited to this application nor to this type of images.

[0027] The method is applicable to augment the number of labeled data for training neural networks for all types of tasks and all types of images.

[0028] A digital signal representation of a medical image to which a neural network is applied can be acquired in several image acquisition ways among which are X-ray imaging, MRI, CT scanning . . . .

[0029] The digital image representation can be acquired directly or can be acquired via the intermediary of an image recording medium such as a photographic film or a photostimulable phosphor screen etc. In the latter situation the recording material is read out and the read out signal is digitized before a neural network is applied to it.

[0030] The image is identified by its metadata among which are data regarding the patient identification, the body part identification and the acquisition.

[0031] The method of the present invention is thus applied to digital signal representations of an image and generally comprises the following steps: [0032] The method starts with a small dataset of images and their segmentation masks (further on referred to as labeled images) and a larger dataset of unlabeled images. The segmentation masks should provide information about the composition of the image. For example for X-Ray data these segmentation masks can consist of five classes: background, bone area, collimation area, soft tissue, foreign object. It will be clear that these classes are only mentioned as an example and that other class types and class definitions may be used, depending in particular on the type of acquisition means. [0033] For a given image i in the labeled dataset, all images which are considered similar in terms of metadata in the unlabeled dataset are gathered and e.g. listed. In this context images are considered similar when e.g. they relate to the same body part, they have the same orientation, the relate to the same gender, they relate to persons with the same age or to persons with the same weight etc. Other metadata types may be envisaged. [0034] With these images a dataset SIM i as a subset of the unlabeled dataset is generated. [0035] Next a neural network for multiclass segmentation is trained on the labeled images and segmentation masks are generated for the images in SIM i. [0036] Based on these segmentation masks similarity between images of SIM i and image i is judged. [0037] Optionally images are first matched on top of each other by means of a predefined registration framework. [0038] For example, images of SIM i onto the given image i are warped by a pre-defined rigid or non-rigid method, [0039] Then the next step is applied which consists of computing and comparing histograms of segmentation masks of given image and images in SIM i, e.g. by KL divergence (other methods may be envisaged). [0040] The most similar image(s) in SIM_i have a similar composition to the image i according to this method. [0041] Next the histogram of the newly found most similar image(s) in SIM_i is transferred to given image i.

[0042] For training neural networks, the original few labeled images are used together with their histogram augmented versions.

[0043] The histogram augmentation step can be done in two ways: [0044] For every labeled image i search through the database for the n closest images in SIM i according to the criteria above. Transfer their histograms onto the labeled image and train with the original image (and its expert labeled annotation) and the histogram augmented versions (with the same labels). [0045] For every image in the unlabeled dataset, transfer its histogram onto the closest image in the labeled dataset according to the criteria above.

[0046] The histogram is preferably transferred based on a quantile transformation.

[0047] A post-processing step can be applied which manually sorts out unrealistic looking images. Unrealistic images may occur for example, when the most similar image according to the criteria defined above is not similar enough and transferring of the histogram results in an image that appears overexposed or underexposed.

[0048] With the method of this invention, even though two images are probably different in terms of what they depict and where they depict it (for example, image 1 showing a hand in the upper left corner, while image 2 showing a different hand in the lower right corner), they can still be similar enough in terms of image composition (image 1 and image 2 can e.g. both consist of approximately 10% hand, 70% background, 20% collimation) such that the histogram can be transferred.

[0049] This method is advantageous over the conventional prior approach in which one image would be registered on top of another and then transferring the histogram since registering for example a random hand onto another random hand is complicated and sometimes not realistically possible. One of them might be pictured from the top, while the other might be pictured from the side and usually this is not known without looking at every image individually, which would be time consuming.