Self-diagnostic method for jaundice and an APP product thereby

Abstract

A self-diagnostic method for momentarily generating a jaundice index, clinically equivalent to the serum bilirubin level, for newborns including Step S1: obtaining a newborn's image of face based on color markers; Step S2: color correction of the image based on the color markers; Step S3: calculating jaundice index based on color information of pixels of the facial image, excluding undesired interfering. The method can be executed by an APP (Application) product.

Claims

1. A non-transitory computer-readable storage medium encoded with executable instructions for execution by a processor to detect jaundice by a self-diagnostic method for jaundice, including: Step S1: placing a plurality of color markers aside a newborn to obtain a self-diagnostic image including an image of the newborn's face and an image of the plurality of color markers, wherein the self-diagnostic image is taken by a smart phone, tablet, or personal computer with a camera equipped with the executable instructions, and the color markers include a pure red color marker, a pure green color marker and a pure blue color marker; Step S2: color correction of the image of a newborn's face based on the image of the plurality of color markers, wherein step S2 includes; Step S21 automatically grabbing the color markers using a RGB (Red, Green, and Blue) color model so as to obtain a pixel array of the grabbed color markers, the pixel array having three array elements including a red color element, a green color element and a blue color element, and obtaining an image pixel number of each obtained array element such that an image pixel number of the pure red color marker, an image pixel number of the pure green color marker and an image pixel number of the pure blue color marker are obtained, and then when the image pixel number of the pure red color marker, the image pixel number of the pure green color marker and the image pixel number of the pure blue color marker are all larger than a threshold value, obtaining an average color value of the pure red color marker, an average color value of the pure green color marker and an average color value of the pure blue color marker, Step S22 color normalization of the image of the newborn's face by using the average color value of the pure red color marker, the average color value of the pure green color marker and the average color value of the pure blue color marker, and Step S3: obtaining a jaundice index of a color normalized image of the newborn's face based on color information of the color normalized image of the newborn's face, wherein step S3 includes: Step S31 grabbing facial skin pixels based on ordinary hues of skin and obtaining the total number of the grabbed facial skin pixels, wherein if the facial skin pixel number is not larger than a quality threshold value, then return to grab the image again, and wherein if the facial skin pixel number is larger than the quality threshold value, then continues to step S32, Step S32 grabbing jaundice pixels among the facial skin pixels based on a statistic range of hues of jaundice, Step S33 obtaining an average saturation of the jaundice pixels from the grabbed jaundice pixels such that the number of jaundice pixels is obtained, and obtaining a Jaundice Skin Ratio (JSR) which is a ratio of a number of jaundice pixels to a number of facial skin pixels; wherein when JSR is less than a preset value, then return to grab the image again, and when the JSR is equal to or larger than the preset value, the method then continues to step S34, Step S34 converting the average saturation value of jaundice pixels to a jaundice index, wherein when the jaundice index is larger than a preset value of a normal jaundice index, an indication that a further diagnostics is required is issued.

2. A self-diagnostic method for jaundice as claimed in claim 1, wherein objects around the newborn which contain colors of red, green, blue or yellow are avoided.

3. A self-diagnostic method for jaundice as claimed in claim 1, wherein a preview of the image includes a facial outline and one or more color marker contour, with Step S21 includes automatic grabbing the color markers and allocating the color markers within the color marker contour, for the color marker of red: grabbing and connecting the facial skin pixels within the one or more contour of a distance to a pure red pixel greater than a red threshold as the largest continuous area thereof, and defining a pure red marker (11) therein; for the color marker of green: grabbing and connecting the facial skin pixels within the one or more contour of a distance to a pure green pixel greater than a green threshold as the largest continuous area thereof, and defining a pure green marker (12) therein; and for the color marker of blue: grabbing and connecting the facial skin pixels within the one or more marker contour of a distance to a pure blue pixel greater than a blue threshold as the largest continuous area thereof, and defining a pure blue marker (13) therein; the pure red pixel (r.sub.r, r.sub.g, r.sub.b)=(255, 0, 0), the pure green pixel (g.sub.r, g.sub.g, g.sub.b)=(0, 255, 0), the pure blue (b.sub.r, b.sub.g, b.sub.b)=(0, 0, 255) Step S22 includes color normalization to each of the pixels in the image in reference to the red channel, the green channel and the blue channel, respectively, such that the value of the red channel, the green channel and the blue channel is inversely proportionally to an average value of a red channel in the red color marker, an average value of a green channel in the green marker and an average of a blue channel in a blue marker, and adjusted to a number.

4. A self-diagnostic method for jaundice as claimed in claim 3, wherein in the Step S31, RGB color information is converted to HSV (Hue, Saturation and Value) color information, in reference to a statistical range of hues in facial skin of newborns, facial skin pixels within the ellipitical facial contour are denoted as S.sub.k, with a total number of S.sub.k denoted as N.sub.sk; Step S33 includes obtaining jaundice pixels J among the facial skin pixels S.sub.k, with the total number of J denoted as N.sub.J, with the total number of S.sub.k denoted as N.sub.sk, wherein the image is evaluated as poor when the ratio between N.sub.J and N.sub.sk is less than th.sub.JSR (50%), and retaking of the image is recommended; Step S34 includes first obtaining a saturation average l.sub.s of jaundice pixels, then converting by a function T(x) to a jaundice index, clinically equivalent to the serum bilirubin level lv from blood tests, wherein the function T(x) is a regressional statistic function of Js and Jv.

5. A self-diagnostic method for jaundice as claimed in claim 4, wherein the Step S3 further includes calculating the total number of image pixels N.sub.E within the elliptical facial outline along with its average saturation S.sub.E, and a ratio SER, Skin Ellipse ratio of N.sub.sk to N.sub.E obtained by SER=N.sub.sk/N.sub.E when SER<th.sub.JSR, the quality of the image taken is evaluated as poor and recommended to retake the image; when SER>th.sub.JSR, and S.sub.E>th.sub.SE, proceed to Step S32.

6. A self-diagnostic method for jaundice as claimed in claim 1, wherein the birth time of the newborn is taken into account, together with the calculation of the jaundice index, to evaluate the need for a doctor visit or photo therapy.

7. A self-diagnostic method for jaundice as claimed in claim 1, executable in an Application software downloadable to electronic means.

8. A self-diagnostic method for jaundice as claimed in claim 2, executable in an Application software downloadable to electronic means.

9. A self-diagnostic method for jaundice as claimed in claim 3, executable in an Application software downloadable to electronic means.

10. A self-diagnostic method for jaundice as claimed in claim 4, executable in an Application software downloadable to electronic means.

11. A self-diagnostic method for jaundice as claimed in claim 5, executable in an Application software downloadable to electronic means.

12. A self-diagnostic method for jaundice as claimed in claim 6, executable in an Application software downloadable to electronic means.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 shows the steps of the method of the instant invention

(2) FIG. 2 shows the flowchart of the method of the instant invention.

(3) FIG. 3 shows image taken as well as the color markers in the image.

(4) FIG. 4: A regression statistic chart of average values of saturation vs. bilirubin levels of the instant invention;

(5) FIG. 5: A chart of STB vs. postnatal age published by American Academy of Pediatrics

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

(6) Please refer first to FIG. 1-3. FIG. 1 shows the procedural steps of the present application. FIG. 2 is the flow chart of the method disclosed in the present application. FIG. 3 shows the color markers applied in the image taken in the instant application. In step S1, newborns face and color markers are photographed; in step S2, color correction; in step S3, calculation is made to obtain the index for jaundice based on color information of the color normalized image of the newborn's face obtained in step S2 in the newborn facial image taken in the step S1, excluding undesired image pixels.

(7) Wherein in step S1, smart phones, tablets or PC's with a camera or photographing function are used for taking a newborn's facial image, including the color markers thereof as defined in the instant method. The color markers contain the pure red marker No. 11, the pure green marker No. 12, and the pure blue marker No. 13, each marker of size greater than 1 cm.sup.2. As shown in FIG. 3, in the preview for taking an image using an electronic mean (such as smart phones, tablets or personal computers, etc.), the elliptical facial outline (2) of a newborn is pre-set in the shape of an ellipse, and the two rectangles are allocated as the two different preferred locations for color markers of pure red (11), pure green (12) and pure blue (13). In the preferred embodiment of the instant invention, the preferred location for color markers 11, 12, and 13 is in the upper location of the preview for an image. Also, the birth date and time of the newborn is taken into account as a factor of postnatal age to improve accuracy in the reading of the index of the newborn's jaundice, either before or after taking the image photo. The input data of birth date and time of the newborn is applied in reference to FIG. 5, which shows serum total bilirubin level, or STB, (in mg/dL) vs. postnatal age (in hours). Background colors within the preview are, preferrably, so arranged to avoid objects surrounding the newborn, such as wrapping clothes, mattresses or pillows, to contain the colors of red, green, blue, and even yellow to reduce interference with the later color analysis of the image taken.

(8) In step S2, color correction is performed on the image taken, based on the conventional RGB color model. Normalization of color pixels of the image taken are performed according to the average color values of color markers. To be more specific, at an initial stage of step S2, i.e. S21, color markers are first automatically grabbed, followed by obtaining the image pixel number Nx within the area of each of the three color markers, with x{r, g, b}, namely, x representing red, green or blue color, and therefore Nx representing an image pixel number of the pure red color marker, an image pixel number of the pure green color marker and an image pixel number of the pure blue within the geometric area (for instance, a rectangle) defined by color markers. When the image pixel number Nx within the area of color markers are all larger than a threshold value, an average color value of the pure red color marker, an average color value of the pure green color marker and an average color value of the pure blue color marker are obtained. After the average color value of the pure red color marker, the average color value of the pure green color marker and the average color value of the pure blue color marker are obtained, step S22 follows, which performs (color) normalization to the photo image taken, based on the average color value of the pure red color marker, the average color value of the pure green color marker and the average color value of the pure blue color marker. In step S21, after the photo image of a newborn is taken, areas containing colors closer to pure red, pure green, and pure blue are respectively grabbed within the (for instance, rectangular) area of the two color marker contours (3). Afterwards, the pure red color marker (11), pure green color marker (12), and pure blue color marker (13) are then defined to be the largest continuous area connecting the areas of colors closer to pure red, pure green, and pure blue, respectively.

(9) In mathematical terms, to automatically grab color markers, mxRri, mxRgi, mxRbi are defined to be the largest continuous area for Rri, Rgi, Rbi, respectively. Whereas, maxi {mxRri, mxRgi, mxRbi} is defined to be a color area within the ith rectangular area that contains the maximum mxRri, mxRgi, mxRbi, corresponding to pure red marker (11), pure green marker (12) and pure blue marker (13), respectively.

(10) In an example embodiment of the instant application, color markers of pure red (11), pure green (12) and pure blue (13) can be grabbed within the color marker contour (3) above the newborn's face. The total number of color pixels within each of the mxRri, mxRgi, mxRbi is defined as Nx, with x{r,g,b}. When Nx is less than a preset threshold value of th.sub.Nx, it implies that either color markers are not correctly grabbed within the specified color marker contour (3), or the light is too dim for a clear photo image to be taken. Nx is preset to correspond to the image resolution of the camera: a higher value of Nx corresponds to a higher image resolution. For instance, for an image resolution of 47523168, th.sub.Nx is recommended to be 10,000. When th.sub.Nx is less than 10,000, it is recommended that the locations for color markers be readjusted for retaking a better image.

(11) {CP.sub.i} is set to be the collection of the image pixels within the specified color marker contour (3), wherein i{0,1}, and i does not belong to a specific area. Let Rr.sub.i, Rg.sub.i, Rb.sub.i be the area of a color closet to pure red, pure green and pure blue, respectively, within the ith rectangular area, x{r, g, b},
Rx.sub.i={P},P{CP.sub.i},
and
|(P.sub.r,P.sub.g,P.sub.b)(x.sub.r,x.sub.g,x.sub.b)|<th_.sub.p, wherein

(12) x{r, g, b},

(13) (r.sub.r,r.sub.g,r.sub.b)=(255,0,0),

(14) (g.sub.r,g.sub.g,g.sub.b)=(0,255,0),

(15) (b.sub.r,b.sub.g,b.sub.b)=(0,0,255),

(16) |ab| is the distance between a and b {square root over ((a.sub.rb.sub.r).sup.2+(a.sub.gb.sub.g).sup.2+(a.sub.bb.sub.b).sup.2)}

(17) th_cp represents the preset threshold value for the color markers, preferably set as 10.

(18) P.sub.r is defined to be the average value of the red channel in the red marker (11). In the same manner, P.sub.g is defined to be the average value of the green channel in the green marker (12), and P.sub.b is defined to be the average value of the blue channel in the blue marker (11).

(19) There are two rectangular color marker contours (3) shown in FIG. 5. In actual practice, one single color marker contour would do. If color markers for some unavoidable reasons cannot be physically grabbed within the specified color marker contour (3) when the photo image is being taken, color markers can temporarily randomly be placed outside of the facial image within the view of the image taken. After the image is taken, color markers can be manually regrabbed by choosing a central point of a color marker followed by automatic regrab as previously described for the instant application.

(20) In step S22, color normalization is performed to the whole image based on the average color value of the pure red color marker, the average color value of the pure green color marker and the average color value of the pure blue color marker. In particular, each image pixel is readjusted corresponding to the red, green and blue channel, respectively such that the red channel of the red marker (11) is closer to 255, which is also true of the green channel of the green marker (12), as well as the blue channel of the blue marker (13).

(21) Let normalization factor

(22) $f_r=\frac{255}{P_r},f_g=\frac{255}{P_g},f_b=\frac{255}{P_b},$

(23) such that the red channel C.sub.r of each image pixel adjusted in normalization as C.sub.r=f.sub.rC.sub.r,

(24) in the same manner, the green channel C.sub.g of each image pixel adjusted n normalization as
C.sub.g=f.sub.gC.sub.g, and

(25) the blue channel C.sub.b of each image pixel adjusted n normalization as
C.sub.b=f.sub.bC.sub.b

(26) Error in RGB color analysis of the image can be reduced by calculation of pixel value of each image pixel corresponding to the red, green and blue marker (11, 12, and 13).

(27) Refer to FIG. 4, which shows the regression chart of average saturation vs. serum bilirubin level of the present invention. In step S3, the color information of the facial image pixels obtained for calculating the jaundice index is based on an HSV (Hue Saturation Value) color model. Color information of facial skin pixels are automatically obtained, and further screened for areas of jaundice. The saturation average of pixels in the areas of jaundice is first obtained, and then converted by a specific function to a jaundice index, equivalent in medical terms to the serum bilirubin level obtained in blood tests. The evaluation of the sampling quality of the image taken is based on the ratio between the jaundice pixels and the facial skin pixels, or Jaundice Skin Ratio, (JSR). The sampling quality of the image taken is less desired when JSR is less than the preset threshold value th.sub.JSR, implying a larger error in the calculation of the jaundice index. th.sub.JSR is a preset threshold value statistically obtained by the instant invention.

(28) In the initial step of S3, S31, facial skin pixels are screened according to the range of skin hues for jaundice, and the total number of which are denoted as N.sub.Sk. In step S32, jaundice pixels are then grabbed among these facial skin pixels based on the range of hues of jaundice. In step S33, the average value of saturation of jaundice pixels, and the Jaundice Skin Ratio (JSR) are calculated. In step S34, the average value of saturation of jaundice pixels is converted to a jaundice index.

(29) C.sub.r, C.sub.g, C.sub.b (RGB) is conventionally converted to HSV by

(30) $\begin{array}{cc}M=\max \left(R,G,B\right)m=\min \left(R,G,B\right)C=M-m{H}^{}=\{\begin{array}{cc}\mathrm{underfined},& \mathrm{if}C=0\\ \frac{G-B}{C},& \mathrm{if}M=R\\ \frac{B-R}{C}+2,& \mathrm{if}M=G\\ \frac{R-G}{C}+4,& \mathrm{if}M=B\end{array}H=60H^{}& \left(1\right)\\ V=\frac{1}{3}\left(R+G+B\right)& \left(2\right)\\ S=\{\begin{array}{cc}0,& \mathrm{if}C=0\\ \frac{c}{v},& \mathrm{otherwise}\end{array}& \left(3\right)\end{array}$

(31) In step S31, the color information of RGB is converted to that of HSV, the range of hues in facial skin of a newborn is set as a parameter (about 42.518.5 in the preferred embodiment of the invention). The collection of the facial skin pixels inside the elliptical facial outline (2) of the image taken are denoted as Sk.

(32) In step S31, if the facial skin is unable to fit completely into the elliptical facial outline (2), after an image is taken, a user can manually adjust the elliptical facial outline (2) in size, location, and angles, such that the facial skin can be recaptured in the manually adjusted elliptical facial outline (2).

(33) To improve the quality in sampling of the image pixels, in step S311, the total number of image pixels N.sub.E within the elliptical facial outline (2), its average saturation S.sub.E, and the ratio SER, i.e. the ratio between N.sub.Sk (i.e. the total number of facial skin pixels) and N.sub.E (i.e. the total number of pixels within the elliptical facial outline 2) are all obtained

(34) $\mathrm{SER}=\frac{N_{\mathrm{Sk}}}{N_E}.$

(35) When SER<th.sub.SER, the quality of the image taken is evaluated as poor and recommended to retake the image.

(36) In the instant invention, when SER<th.sub.SER (70%), the sampling quality of the image taken is evaluated as poor and recommended to have the image retaken.

(37) The average value of saturation of all image pixels within the elliptical facial outline (2) is denoted as S.sub.E. In the instant invention, when S.sub.E<th.sub.SER (0.5), the light source is deemed insufficient to produce a fair sampling and is recommended to have the image retaken.

(38) In step S32, the range of hues for newborns is statistically set as a parameter for jaundice pixels, which is about 40-60 in the instant invention. The collection of jaundice pixels obtained among the facial pixels of the image is denoted as J.

(39) The collection of jaundice pixels (or pixels in a jaundice area) is denoted as J, with the total number of J denoted as N.sub.J. The collection of facial skin pixels in the image taken is denoted as Sk, with the total number of Sk denoted as N.sub.Sk. JSR is defined as the ratio between N.sub.J and N.sub.Sk.

(40) To avoid misjudgment due to poor quality of the image, the image is evaluated as poor when JSR is less than th.sub.JSR (50%), and hence retaking of the image is recommended to avoid an error in J.sub.s.

(41) Average saturation of J is denoted as J.sub.s, which is converted by a specific function T(x) to a jaundice index, clinically equivalent to the serum bilirubin level obtained from blood tests. As shown in FIG. 4, the function T(x) is a regressional statistic function of J.sub.s and J.sub.v, both of which are positively correlated. A higher value of J.sub.s corresponds to a higher value of J.sub.v, and vice versa. FIG. 5 shows a chart of STB vs. postnatal age published by American Academy of Pediatrics for as a guideline for treatment.

(42) The procedural steps in the method can be implemented by APP (Application software), downloaded from the internet to smart phones, tablets, etc.

(43) The color analysis is conveniently made in accordance with color markers in real-time. When the facial skin pixel number is not larger than a quality threshold value (due to dim lights, or unfavorable locations of color markers), the user would be requested for retaking an image to avoid errors in producing a biased jaundice index.