Method and Apparatus of Image Adjustment for Gastrointestinal Tract Images
20230143451 · 2023-05-11
Inventors
Cpc classification
International classification
Abstract
A method of processing gastrointestinal images is disclosed. According to embodiments of the present invention, a plurality of images from an endoscope, wherein the plurality of images is captured by the endoscope when the endoscope travels through a GI (gastrointestinal) tract are received. Corresponding anatomical parts associated with the plurality of images are then determined. A selected image process is applied to a target image from the plurality of images, where one or more parameters of the selected image process are determined according to a corresponding anatomical part determined for the target image.
Claims
1. A method of processing gastrointestinal images, the method comprising: receiving a plurality of images from an endoscope, wherein the plurality of images is captured by the endoscope when the endoscope travels through a GI (gastrointestinal) tract; determining corresponding anatomical parts associated with the plurality of images; and applying a selected image process to a target image from the plurality of images, wherein one or more parameters of the selected image process are determined according to a corresponding anatomical part determined for the target image.
2. The method of claim 1, wherein the corresponding anatomical parts comprise two or more parts selected from a group comprising a small intestine and a colon.
3. The method of claim 1, wherein the selected image process corresponds to a color correction process or a color enhancement process.
4. The method of claim 3, wherein the color correction process or the color enhancement process is applied to the target image in a color space selected from a color space group comprising RGB, YUV, HSV (hue, saturation, value) and CIELAB.
5. The method of claim 3, wherein the color correction process or the color enhancement process accentuates redness only if the target image is from a colon.
6. The method of claim 3, wherein the color correction process or the color enhancement process accentuates redness more for the target image from a colon than the target image from a small intestine.
7. The method of claim 3, wherein the color correction process or the color enhancement process is implemented in a form of matrix multiplication or a nonlinear function, wherein a color corrected or enhanced output pixel is generated by multiplying an input pixel with a matrix and then adding an offset or applying the nonlinear function.
8. The method of claim 3, wherein each color component of a color corrected or enhanced output pixel is generated by modifying a corresponding color component of an input pixel.
9. The method of claim 1, wherein if a set of images are captured when the endoscope enters a new anatomical part, one or more new parameters of the selected image process are used for the set of images.
10. The method of claim 1, wherein the endoscope corresponds to a tethered endoscope or a capsule endoscope.
11. A non-transitory computer-readable medium having stored thereon a computer-readable code executable by a processor to cause the processor to: receive a plurality of images from a GI (gastrointestinal) tract; determine corresponding anatomical parts associated with the plurality of images; and apply a selected image process to a target image from the plurality of images, wherein one or more parameters of the selected image process are determined according to a corresponding anatomical part determined for the target image.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0012]
DETAILED DESCRIPTION OF THE INVENTION
[0013] It will be readily understood that the components of the present invention, as generally described and illustrated in the figure herein, may be arranged and designed in a wide variety of different configurations. Thus, the following more detailed description of the embodiments of the systems and methods of the present invention, as represented in the figures, is not intended to limit the scope of the invention, as claimed, but is merely representative of selected embodiments of the invention. References throughout this specification to “one embodiment,” “an embodiment,” or similar language mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present invention. Thus, appearances of the phrases “in one embodiment” or “in an embodiment” in various places throughout this specification are not necessarily all referring to the same embodiment.
[0014] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. One skilled in the relevant art will recognize, however, that the invention can be practiced without one or more of the specific details, or with other methods, components, etc. In other instances, well-known structures, or operations are not shown or described in detail to avoid obscuring aspects of the invention. The illustrated embodiments of the invention will be best understood by reference to the drawing, wherein like parts are designated by like numerals throughout. The following description is intended only by way of example, and simply illustrates certain selected embodiments of apparatus and methods that are consistent with the invention as claimed herein.
[0015] Endoscopes are normally inserted into the human body through a natural opening such as the mouth or anus. Therefore, endoscopes are preferred to be small sizes so as to be minimally invasive. As mentioned before, endoscopes can be used for diagnosis of human gastrointestinal (GI) tract. The captured image sequence can be viewed to identify any possible anomaly. If any anomaly is found, it is of interest to identify the characteristics of the anomaly as well as its location. Beside diagnosis by a human professional, automated endoscopic disease activity assessment based on artificial intelligence has gained popularity in recent years. In particular, advances in machine learning methods have greatly improved the performance of artificial intelligence (AI). The accuracy of AI based diagnosis often can outperform human based diagnosis. Gastroenterology has been a field to see rapid adoption of AI methods for automated assessment of various GI conditions and diseases. For example, the AI based methods can replicate expert endoscopic interpretation, polyp recognition in the colon. This situation applies to tethered endoscopes and capsule endoscopes as well.
[0016] Assessment of GI conditions and diseases using images captured by an endoscope relies heavily on the picture quality of the images regardless by a human expert or an AI method. In some cases, the color of an image may reveal important diagnostic information. For example, a doctor may look for bright red color as an indication of possible bleeding in the GI wall. This situation applies to tethered endoscopes and capsule endoscopes as well.
[0017] For conventional image processing, a selected image processing technique is often applied to images that are presumably subject to certain artifacts or some impairments such as under- or over-exposure, improper color balance, or blurry. Even if the image quality may appear to be normal, sometimes, it may be desirable to apply different image processing techniques to images from different anatomic GI parts (e.g., small intestine and colons). For example, it may be desirable to accentuate the redness of the colons so the pathologies (e.g., bleeding or polyps) may look more prominent for doctors to see easily. However, the small bowel already has much more redness and we would not want to accentuate the redness further to avoid saturation.
[0018] Accordingly, embodiments of the present invention select a target image processing method or select image processing parameters depending on the anatomic part associated with the underlying images. In one embodiment, the image process may correspond to a color correction process or a color enhancement process. The color correction process or the color enhancement process by itself is well known in the image processing field. Various color correction or enhancement algorithms have been disclosed. Usually, a suitable color space is used to represent pixels of a color image. The RGB (red, green and blue) has been a very popular color space being used for image sensor. The RGB corresponds to three primary colors that can be mixed to reproduce various colors. The YUV color space is another popular color space being used, where Y corresponds to the luminance signal while U and V represent chrominance signals. On the other hand, CIELAB is another popular color space that is designed to approximate human vision. The CIELAB color space is also referred as L*a*b* model since it uses L*a*b* color coordinates representing the lightness of the color (L*), its position between red and green (a*) and its position between yellow and blue (b*). The L* component closely matches human perception of lightness, where L*=0 yields black and L*=100 indicates diffuse white. In addition, HSV (hue, saturation, value) is yet another popular color space. The present invention is not limited to any particular color space.
[0019] The parameters for color correction can be designed using reference patterns with known colors. For example, color chart with known color components can be used as ground truth. The captured color pixels can be compared with the colors of the ground truth to derive the color correction matrix or nonlinear function. In this case, the corrected color pixel can be derived by multiplying a captured color pixel by a color correction matrix or applying the nonlinear function. An offset value can be added to the multiplication result. Color enhancement can also be performed in a similar fashion using matrix multiplication. However, color enhancement can also be done by simply modifying individual color component such as boosting the red component in case that the RGB format is used.
[0020] The images associated with different anatomical parts may have different characteristics. A selected image process may help improve image quality or help ease diagnosis by viewing the processed images associated with one particular anatomical part. However, the selected image process or the particular parameters of the selected image process may not be proper for images from other anatomical part. For example, by boosting the red color may help visually spot anomaly in the colon. However, boosting the red color may cause color saturation since the images from the small intestine often look more reddish than the images from the colon. Accordingly, we may like to accentuate the redness for images from the colon only and we do not accentuate the redness for images from other anatomic parts. In another example, we can accentuate the redness for images from the colon more than other anatomic parts. The same method can be applied to all anatomical parts of GI tracts, such as esophagus, stomach, small bowel and colon, and the associated sub-parts. For a capsule endoscope, in one embodiment, the parameters for esophagus can be applied in the beginning knowing that it is at the beginning of the GI tract after the mouth. For a capsule endoscope, in another embodiment, the parameters for the stomach can be applied after a certain time, for example, knowing at that time the capsule is in the stomach.
[0021] The images from the GI tract can be divided into groups according to their associated anatomic parts such as small bowel and colons. The segmentation can be done manually by scanning through the image sequence by a medical professional to spot particular noticeable features for transition from one anatomic part to another. The segmentation can also be done using an automated process to identify or detect certain characteristics of the images associated with the anatomic parts. Upon segmentation of the GI anatomic parts, image processing can be selectively applied a selected GI anatomic part. Location determining devices such as a gyroscope, accelerometer, or electronic compass may be embedded or attached to the capsule device to record the capsule locations in the GI tract.
[0022] The method mentioned above can be implemented using various programmable devices such as micro-controller, central processing unit (CPU), field programmable gate array (FPGA), digital signal processor (DSP) or any programmable processor with or without firmware or software. A display can be used for presenting the visual information related to travelled distance and/or image information. Furthermore, mobile devices such as tablets or smart phones can be used to implement the method mentioned above since the mobile devices usually have a display and sufficient computational capability to handle the needed processing mentioned above. A notebook or a computer can also serve as the system to support the method mention above.
[0023]
[0024] The above description is presented to enable a person of ordinary skill in the art to practice the present invention as provided in the context of a particular application and its requirements. Various modifications to the described embodiments will be apparent to those with skill in the art, and the general principles defined herein may be applied to other embodiments. Therefore, the present invention is not intended to be limited to the particular embodiments shown and described, but is to be accorded the widest scope consistent with the principles and novel features herein disclosed. In the above detailed description, various specific details are illustrated in order to provide a thorough understanding of the present invention. Nevertheless, it will be understood by those skilled in the art that the present invention may be practiced.
[0025] The invention may be embodied in other specific forms without departing from its spirit or essential characteristics. The described examples are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.