Endoscopic image enhancement system and method

Abstract

The invention provides an image color enhancement system, method, storage medium and endoscope. Since the human skin color and the basic tone of endoscopic image are similar, a data set of Macbeth color card is utilized to generate a color correction matrix for skin color enhancement. The color correction matrix is used by an endoscope for real time image capture, resulting in a vivid expression of the basic tone of an endoscopic image without introducing artifacts in other tonal image parts.

Claims

1. An endoscopic image color enhancement system, comprising: an endoscope configured to: acquire a first image, and use a color correction matrix generated based on the first image for skin tone enhancement to enhance color of endoscopic images; a data processing module configured to: acquire a set of color enhancement target data, the set of color enhancement target data including coordinates (H0, S0, V0) of a point P0 in HSV space; obtain first data of color patches in first row and first column of Macbeth color card, perform a first interpolation of the set of color enhancement target data and the first data, and replace the first data with result of the first interpolation; obtain second data of color patches in first row and second column of Macbeth color card, and perform a second interpolation of the set of color enhancement target data and the second data, and replace the second data with result of the second interpolation; merge data of rest patches of Macbeth color card with the replaced first and the replaced second data resulting in a data set of Macbeth color card for skin tone enhancement; generate the CCM color correction matrix for skin color enhancement based on the data set of the Macbeth color card for skin tone enhancement and the first image through a color correction matrix generation algorithm.

2. An endoscopic image color enhancement method, comprising the steps of: acquiring a first endoscopic image; acquiring a set of color enhancement target data, the set of color enhancement target data including coordinates (H0, S0, V0) of a point P0 in HSV space; obtaining first data of color patches in first row and first column of Macbeth color card, performing a first interpolation of the set of color enhancement target data and the first data, and replacing the first data with result of the first interpolation; obtaining second data of color patches in first row and second column of Macbeth color card, performing a second interpolation of the set of color enhancement target data and the second data, and replacing the second data with result of the second interpolation; merging data of rest patches of Macbeth color card with the replaced first and the replaced second resulting in a data set of Macbeth color card for skin tone enhancement; generating a CCM color correction matrix for skin color enhancement based on the data set of the Macbeth color card for skin tone enhancement and the first endoscopic image through a color correction matrix generation algorithm; enhancing color of endoscopic images using the color correction matrix.

Description

A BRIEF DISCUSSION OF THE DRAWINGS

(1) FIG. 1: Module diagram of the endoscopic image color enhancement system.

(2) FIG. 2: Flow chart of the process for color enhancement of endoscopic images.

(3) FIG. 3: Flow chart of generating the color enhancement target data;

(4) FIG. 4: An embodiment of an endoscope incorporating the color enhancement functionality

DETAILED DESCRIPTION

(5) An endoscope is preferably equipped with LED light source for illumination, preferably with a color temperature between 3000 and 7000 k. The image color of the endoscope mainly depends on the illumination, the spectral characteristics of the objects in the scene, which is the digestive tract of a patient as relating to the current invention, and the image sensor and processor used to capture the image. The mucous membrane is the main structure of the normal human digestive tract. It is usually reddish and constitutes the base tissue for the basic tone of the endoscopic image. The basic tone or the predominant tone of an endoscopic image can be achieved preferably by calculating a weighted average of image pixels preferably in a sRGB color space, and preferably converting the weighted average from the sRGB space to HSV space. Other tones visible under the endoscope include bluish submucosal veins, food residues, digestive fluids, abnormal diseased tissues, and image noise.

(6) A color correction matrix is a 2D data matrix comprising 3*3 elements. A color correction matrix operates on a pixel of an image typically in a linear R, G, B color space by the following formula:
[R′,G′,B′]=A*[R,G,B].sup.T.   [1]

(7) Most imaging devices including the camera module in an endoscopy incorporate a CCM module in the ISP pipeline that operates in real time on every pixel of an image acquired by the image sensor. Data of color correction matrix are loaded to ISP hardware at the initialization and dynamically refreshable. Software based implementation of a color processing of CCM operation is also applicable on various platforms. To acquire a standard color correction matrix, a camera turns off its CCM module, takes a first image of a standard color card preferably the Macbeth color card under a standard illumination, wherein an color correction matrix generation algorithm runs the image and a data set of Macbeth color card to find out a set of optimal data which can performs formula [1] on the first image such that the output image data of the first image match a set of target data of the standard color card patch by patch within an acceptable range of error. There are many ways to implement a color correction matrix generation algorithm in the prior art, and since the objective is not for inventing a novel color correction matrix generation algorithm, there is no need to describe it in detail.

(8) The color patches in the first row, first column and first row and second column of the Macbeth color card are the closest to and represent the human skin tone and are preferably referred to as the skin tone patches hereby. It is a known technique in prior art to generate a color correction matrix for enhancing the skin color by altering the values of the skin tone patches in a color correction matrix generation process. Due to a large amount of data showing that the color pick of the human digestive tract under a preferred endoscope lighting color temperature appears close to skin tone, it suggests enhancing the endoscopic image just like enhancing the skin color in a beauty camera.

(9) As in FIG. 1, the endoscope 101, after turning off the working lighting of the endoscope and CCM function of its ISP, takes a first image of the Macbeth color card under a standard illumination of a color temperature matching the working lighting color temperature of endoscope 101 or another endoscope of the same model including having the same type of image sensor as 101. Preferably, the color temperature of the working lighting of endoscope 101 may be 5000K; further, endoscope 101 adopts a color correction matrix, which is generated by module 102 based on the first image, in the CCM module of the image processor for real-time enhancement of the image captured by endoscope 101; A data processing module 102 is configured to: acquire a set of color enhancement target data, the set of color enhancement target data preferably including coordinates (H0, S0, V0) of a point P0 in the HSV space; Obtain standard data of the color patches in the first row and the first column of the Macbeth color card, including preferably coordinates (H1, S1, V1) of a point P1 in the HSV space; Obtain standard data of the color patches in the first row and the second column of the Macbeth color card, including preferably coordinates (H2, S2, V2) of a point P2 in the HSV space; perform a first interpolation between the coordinates of P0 and the coordinates of P1, then replaces the coordinates of P1 with the result of the first interpolation; performs a second interpolation between the coordinates of P0 and the coordinates of P2, then replaces the coordinates of P2 with the result of the first interpolation; merge data of the rest patches of standard data of the Macbeth color card color with the replaced P1 coordinates and the replaced P2 coordinates to obtain a data set of the Macbeth color card for skin color enhancement; generate the color correction matrix for skin color enhancement through a color correction matrix generation algorithm using the data set of the Macbeth color card for skin color enhancement and the first image which is adopted preferably by endoscope 101 or another endoscope of the same model as endoscope 101 for endoscopic image enhancement in real time or by a software based color processing platform.

(10) The first interpolation preferably comprises:
H10=a1*H0+a2*H1;  [2]
S10=b1*S0+b2*S1;  [3]
V10=c1*V0+c2*V1  [4],
wherein, the value of a1, a2, b1, b2, c1, and c2 are in [0, 1].
The second interpolation preferably comprises:
H20=a1*H0+a2*H2  [5];
S20=b1*S0+b2*S2  [6];
V20=c1*V0+c2*V2,  [7]
wherein, the value of a1, a2, b1, b2, c1, and c2 are in [0, 1].

(11) FIG. 3 is a flow chart for obtaining a set of color enhancement target data from an endoscopic image. Step 301 obtains an image by an endoscope in a first subject in a preview mode. Step 302 obtain multiple images in preview mode by adjusting the hue and saturation parameters of the camera of the endoscope. Step 303 captures an image with the best visual effect as a first target image. Step 304 calculate a weighted average value of pixels of the first target image with wij being a weight of a pixel

(12) with coordinates of i and j, wherein, wij is determined by:

(13) converting the first target image to HSV space resulting in a temporary image;

(14) generating a first data set of a two-dimensional histogram of H and S from the temporary image;

(15) performing a threshold filtering on the first data set resulting in a second data set of the two-dimensional histogram;

(16) acquiring a centroid PCenter of the second data set;

(17) calculating the distance dij from each pixel in the second data set to PCenter, wherein

(18) wij and dij are inversely proportional and wij equals 0 for pixels not in the second data set. Step 305 generates the set of color enhancement target data by converting the weighted average to the HSV color space.

(19) FIG. 4 is a schematic diagram of an endoscope incorporating the system or method of the endoscopic image color enhancement.

(20) The embodiments described above are for the purpose of illustration, any obvious changes derivative of this disclosure is claimed within the protection scope of the present invention.