PROGRAM FOR INDICATING HUNNER LESION, LEARNED MODEL, AND METHOD FOR GENERATING SAME

Abstract

The present disclosure relates to a method for generating a learning model, a learned model, a program, and a controller of a bladder endoscope in which the program or the model is recorded. The method including: acquiring, as teaching data, endoscope image data on a Hunner lesion in a bladder; and generating the learning model by using the teaching data such that a bladder endoscope image serves as an input and the position indication of a Hunner lesion in the bladder endoscope image serves as an output. The program causing a computer to perform acquiring endoscope image data on a Hunner lesion in a bladder, inputting a target bladder endoscope image to a learning model in which a bladder endoscope image serves as an input and position-indication data on a Hunner lesion in an endoscope image serves as an output, and outputting the position indication of the Hunner lesion.

Claims

1. A method for generating a learning model, the method comprising: acquiring, as teaching data, endoscope image data on a Hunner lesion in a bladder, and generating the learning model by using the teaching data such that a bladder endoscope image serves as an input and position indication of a Hunner lesion in the bladder endoscope image serves as an output.

2. The method for generating a learning model according to claim 1, wherein endoscope image data on air in a bladder is further included as teaching data.

3. The method for generating a learning model according to claim 1, wherein endoscope image data on a normal bladder and endoscope image data on an abnormal bladder free of Hunner lesions are further included as teaching data.

4. The method for generating a learning model according to claim 1, wherein the endoscope image data in the teaching data includes both of an image of narrow band imaging and an image of white light imaging.

5. The method for generating a learning model according to claim 4, wherein determination on whether a bladder in an inputted bladder endoscope is a normal bladder is further included as an output.

6. A program that causes a computer to perform processing for acquiring endoscope image data on a Hunner lesion in a bladder, inputting a target bladder endoscope image to a learning model in which a bladder endoscope image serves as an input and position-indication data on a Hunner lesion in an endoscope image serves as an output, and outputting position indication of the Hunner lesion.

7. The program according to claim 6, wherein the endoscope image data in teaching data includes both of an image of narrow band imaging and an image of white light imaging.

8. The program according to claim 6, wherein the learning model further includes, as teaching data, endoscope image data on a normal bladder free of Hunner lesions and endoscope image data on an abnormal bladder free of Hunner lesions.

9. The program according to claim 6, wherein endoscope image data on air in a bladder is further included as teaching data.

10. The program according to claim 7, further comprising processing for determining whether a bladder is normal or not and outputting the determination.

11. A controller of a bladder endoscope, wherein the program according to claim 6 is recorded.

12. A learned model that uses a bladder endoscope image acquired by using a bladder endoscope system, the learned model comprising: an input layer that receives an input of the bladder endoscope image; an output layer in which position indication data on a Hunner lesion in the endoscope image serves as an output, and an intermediate layer having a parameter learned by using teaching data in which endoscope image data on a Hunner lesion in a bladder serves as an input and position indication data on the Hunner lesion in the bladder image serves as an output, the learned model causing a computer to function to input a target bladder endoscope image to the input layer, perform an operation in the intermediate layer, and output the position indication data on the Hunner lesion in the image.

13. The learned model according to claim 12, wherein endoscope image data on air in a bladder is further included as teaching data.

14. The learned model according to claim 12, further comprising, as teaching data, endoscope image data on a normal bladder and endoscope image data on an abnormal bladder free of Hunner lesions.

15. The learned model according to claim 12, wherein the endoscope image data in the teaching data includes both of an image of narrow band imaging and an image of white light imaging.

16. The learned model according to claim 14, further comprising, as an output, whether a bladder is a normal bladder or an abnormal bladder.

17. A controller of a bladder endoscope, wherein the learned model according to claim 12 is recorded.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0021] FIG. 1 shows examples of WLI and NBI images of a normal condition, bubbles, and a Hunner lesion;

[0022] FIG. 2 shows a score example of an IoU;

[0023] FIG. 3 is a conceptual diagram of TP, FP, and FN;

[0024] FIG. 4 shows an example of the NBI image prediction result of a detection model (Cascade R-CNN);

[0025] FIG. 5 shows an example of the NBI image prediction results of the detection model and a segmentation model;

[0026] FIG. 6 shows an example of the WLI image prediction results of the detection model and the segmentation model; and

[0027] FIG. 7 shows a detection result example of NBI normal bladder images in the detection model and the segmentation model.

DESCRIPTION OF EMBODIMENTS

[0028] Dataset

[0029] Based on moving images captured by a bladder endoscope, correct information is added (annotation) to the moving images in the following steps: [0030] 1. A candidate image is extracted every ten frames from all frames, and a Hunner-lesion candidate image is selected; [0031] 2. Correct information is added to the selected image; and [0032] 3. In frames around the image to which the correct information has been added, the correct information is added to similar images.

[0033] Data to which the correct information has been added was divided into NBI and WLI to produce a database of still images. In an extracted image, a black area outside an endoscope image was erased by trimming. This processing obtains a pixel size of about 1000×900 pixels. Moreover, an annotation at the position of a Hunner lesion was made by software called Label me.

[0034] The number of moving images

TABLE-US-00001 TABLE 1 Dataset The number of moving images Year 2016 1940 Year 2017 1952 Year 2018 2005 Year 2019 2259 Year 2020 64

[0035] Still-Image Database Configuration

TABLE-US-00002 TABLE 2 NBI WLI The number The number of still images of still images Normal image 672 726 Hunner lesion 2758 1591 Bubbles 573 17 *Normal images include images of normal bladders and abnormal bladders free of Hunner lesions but do not include images having Hunner lesions.

[0036] Bubbles include images of normal bladders and images having Hunner lesions.

[0037] FIG. 1 shows examples of WLI and NBI images of a normal condition, bubbles, and a Hunner lesion.

[0038] The image examples of bubbles in FIG. 1 are image examples of abnormal bladders free of Hunner lesions. Original images are all color images that clarify a difference in visibility between WLI and NBI, facilitating understanding of an advantage of NBI.

[0039] Model

[0040] An experiment was conducted by using a detection model and a segmentation model. The detection model is a model for estimating a rectangular area including a Hunner lesion area. The position and size of the rectangular area and the lesion confidence factor of the rectangle. The segmentation model is a model for outputting a lesion confidence factor for each pixel. A Hunner lesion area is estimated as well as the shape of the area.

[0041] In the experiment, models of high performance for a dataset of general images (COCO, CITYSCOPES) were used as follows: [0042] Detection model: Cascade R-CNN; [0043] Segmentation model: Cascade Mask R-CNN; and [0044] Segmentation model: OCNet.

[0045] The three models were learned with data of NBI and WLI, and six models were created in total.

[0046] Experiment Setting and Result

[0047] In the experiment, the dataset was randomly divided into learning data of 85% and test data of 15% five times, and learning and evaluation were performed five times. The data was divided for each case. Tables 3 and 4 indicate the number of images and the number of cases in NBI and WLI datasets.

[0048] The number of images and the number of cases in NBI dataset

TABLE-US-00003 TABLE 3 Learning data Test data Divided The The The The dataset number of number of number of number of number images images images images 1 3478 122 514 20 2 3478 122 514 20 3 3365 122 627 20 4 3361 122 631 20 5 3439 122 553 20

[0049] The number of images and the number of cases in WLI dataset

TABLE-US-00004 TABLE 4 Learning data Test data Divided The The The The dataset number of number of number of number of number images images images images 1 1909 95 329 15 2 1969 95 269 15 3 1952 95 286 15 4 1773 95 465 15 5 1875 95 363 15

[0050] Three models (Cascade R-CNN, Cascade Mask R-CNN, OCNET) were learned for each image type and each divided dataset, and the performance of each model was evaluated based on the sensitivity and the positive prediction value of each Hunner lesion area. The sensitivity and the positive prediction value (PPV) are expressed as follows: Sensitivity=#TP/(#TP+#FN) PPV=#TP/(#TP+#FP) where #TP is the number of true positives, #FN is the number of false negatives, and #FP is the number of false negatives.

[0051] A method for determining a true positive (TP), a false negative (FN), and a false positive for each Hunner lesion area will be described below. First, the degree of overlapping of a predicted area and a correct area is calculated by an IoU (Intersection over Union). The IoU is the ratio of the number of duplicate pixels in the predicted area and the correct area relative to the number of pixels in the union area of the predicted area and the correct area. If the two areas do not overlap each other at all, the IoU is 0, whereas if the two areas completely agree with each other, the IoU is 1. FIG. 2 shows a score example of the IoU for rectangular areas. Dotted lines indicate correct areas, and solid lines indicate predicted areas.

[0052] In the present example, TP, FN, and FP were defined as follows: [0053] TP≡a correct area where the IoU exceeds 0.3 relative to all predicted areas, each including at least one duplicate pixel; [0054] FN≡a correct area where the IoU is not larger than 0.3 relative to all predicted areas, each including at least one duplicate pixel; and [0055] FP≡a predicted area where the IoU is not larger than 0.1 relative to the correct area.

[0056] As described above, Cascade R-CNN, a detection model, is evaluated with rectangular regions. Cascade Mask R-CNN and OCNET, segmentation models, are evaluated in consideration of area shapes. The correct areas and the predicted areas are provided. As shown in FIG. 3, the upper correct area (left) and the upper predicted area hardly overlap each other and thus are denoted as FN and FP. The lower correct area (center) sufficiently overlapping multiple predicted areas is denoted as TP.

[0057] FIG. 3 shows a conceptual diagram of the TP, FP, and FN.

[0058] Tables 5 to 8 below show the evaluation results of each model. FIGS. 4 to 7 show the NBI image prediction results of each model. FIG. 6 shows a detection result example of NBI normal bladder images in the detection model and the segmentation model. The original images of FIGS. 4 to 7 are all color photos.

[0059] Table 5: the performance of the detection model (Cascade R-CNN) evaluated in an NBI image

[0060] Table 6: the performance of the segmentation model evaluated in an NBI image

[0061] Table 7: the performance of the detection model (Cascade R-CNN) evaluated in a WLI image

[0062] Table 8: the performance of the segmentation model evaluated in a WLI image

[0063] The performance of the detection model (Cascade R-CNN) evaluated in an NBI image

TABLE-US-00005 TABLE 5 Evaluation in predicted Divided Evaluation in areas having a confidence number all predicted areas factor of 50% or more Sensi- Sensi- dataset TP FP FN PPV tivity TP FP FN PPV tivity 1 359 350 145 0.51 0.71 324 136 180 0.70 0.64 2 308 608 208 0.34 0.60 256 228 260 0.53 0.5 3 524 689 327 0.43 0.61 424 271 427 0.61 0.5 4 464 598 205 0.44 0.69 405 250 264 0.62 0.6 5 486 844 157 0.36 0.76 432 432 211 0.5 0.67 Average 0.41 0.67 0.59 0.58 + ± + ± 0.068 0.068 0.079 0.078

[0064] The performance of the segmentation model evaluated in an NBI image

TABLE-US-00006 TABLE 6 Divided dataset Cascade Mask R-CNN OCNET number PPV Sensitivity PPV Sensitivity 1 0.67 0.71 0.84 0.57 2 0.49 0.64 0.62 0.38 3 0.63 0.68 0.59 0.40 4 0.5 0.65 0.81 0.66 5 0.38 0.66 0.55 0.44 Average 0.53 ± 0.116 0.67 ± 0.027 0.68 ± 0.133 0.49 ± 0.120 *Evaluation with all prediction pixels (pixels with a confidence factor other than 0%) having reactions from the model

[0065] The performance of the detection model (Cascade R-CNN) evaluated in a WLI image

TABLE-US-00007 TABLE 7 Evaluation in predicted Evaluation in areas having confidence Divided all predicted areas fact of 50% or more dataset Sensi- Sensi- number TP FP FN PPV tivity TP FP FN PPV tivity 1 190 133 36 0.59 0.84 179 59 47 0.75 0.79 2 241 150 26 0.62 0.90 225 74 42 0.75 0.84 3 251 195 90 0.56 0.74 218 109 123 0.66 0.63 4 470 245 61 0.66 0.88 418 83 113 0.83 0.79 5 337 241 100 0.58 0.77 312 130 125 0.71 0.71 Average 0.60 0.83 0.74 0.75 ± ± ± ± 0.039 0.069 0.062 0.083

[0066] The performance of the segmentation model evaluated in a WLI image

TABLE-US-00008 TABLE 8 Divided dataset Cascade Mask R-CNN OCNET number PPV Sensitivity PPV Sensitivity 1 0.45 0.91 0.78 0.54 2 0.56 0.98 0.87 0.64 3 0.45 0.90 0.76 0.47 4 0.56 0.94 0.80 0.79 5 0.50 0.91 0.63 0.38 Average 0.50 ± 0.055 0.93 ± 0.032 0.76 ± 0.087 0.56 ± 0.158 *Evaluation with all prediction pixels (pixels with a confidence factor other than 0%) having reactions from the model

[0067] FIG. 4 shows an example of the NBI image prediction result of the detection model (Cascade R-CNN).

[0068] All predicted areas with a confidence factor other than 0% are displayed. Dotted lines (green in the original images) indicate correct areas, and solid lines (red in the original images) indicate predicted areas, which sufficiently indicates a Hunner lesion.

[0069] FIG. 5 shows an example of the NBI image prediction results of the detection model and the segmentation model.

[0070] Rectangular outputs are displayed with confidence factors in addition to area specifications. An upper left image, an upper central image, and an upper right image sequentially indicate Hunner lesions with confidence factors of 0.78 (left), 0.9 (center), and 0.97 (right). A lower left image, a lower central image, and a lower right image sequentially indicate a Hunner lesion with a confidence factor of 0.33 and bubbles with a confidence factor of 0.79 (left), bubbles with a confidence factor of 0.97 (center), and a Hunner lesion with a confidence factor of 0.98 and a confidence factor of 0.4. In the present example, bubbles are indicated as bubbles in output display, so that the indication is successfully distinct from that of a Hunner lesion. In the original image, the correct area is indicated in green.

[0071] FIG. 6 shows an example of the WLI image prediction results of the detection model and the segmentation model.

[0072] Rectangular outputs are displayed with confidence factors in addition to area specifications. From the left to the right, a Hunner lesion is indicated with confidence factors of 0.96 and 0.99 (left), 0.97 (center), and 0.997 (right). This successfully provides a sufficient indication of a Hunner lesion. In the original image, the correct area is indicated in green.

[0073] FIG. 7 shows a detection result example of NBI normal bladder images in the detection model and the segmentation model. In the normal bladder, any Hunner lesions have not been indicated. Determination of a false positive has been avoided even in a bladder having a different image appearance.

[0074] As described above, a learned model applicable to images of narrow band imaging and white light imaging can be formed. Moreover, bubbles can be indicated. According to the present invention, an opportunity can be provided to accurately and quickly identify a Hunner lesion in images of narrow band imaging and white light imaging regardless of the knowledge and skill of a doctor. Furthermore, determination of a false positive is avoided even if a height difference is made or a red phase is caused by a shadow or the like in a normal bladder. Consequently, the present invention contributes to the progress of an accurate diagnosis of interstitial cystitis, the rescue of interstitial cystitis patients, and international consensus building.