HYPERSPECTRAL IMAGE CONSTRUCTION OF BIOLOGICAL TISSUE FOR BLOOD HEMOGLOBIN ANALYSIS USING A SMARTPHONE
20230023592 · 2023-01-26
Assignee
Inventors
- Young L. Kim (West Lafayette, IN, US)
- Md Munirul Haque (Fishers, IN, US)
- Michelle Amaris Visbal-Onufrak (Las Piedras, PR, US)
- Sang Mok Park (West Lafayette, IN, US)
Cpc classification
G01N21/31
PHYSICS
A61B5/14546
HUMAN NECESSITIES
A61B5/0075
HUMAN NECESSITIES
A61B5/1455
HUMAN NECESSITIES
International classification
A61B5/145
HUMAN NECESSITIES
A61B5/00
HUMAN NECESSITIES
Abstract
A bloodless system for numerically generated hyperspectral imaging data for measuring biochemical compositions is disclosed which includes an optical imaging device adapted to acquire an RGB image from an area of interest, a processor adapted to receive a hyperspectral dataset representing an a priori hyperspectral data of the area of interest of a population to which the subject belongs, receive RGB response for each one of RGB channels of the optical imaging device, pair the corresponding RGB data with the hyperspectral data, obtain a transformation matrix adapted to convert a subject-specific RGB image dataset into a subject-specific hyperspectral dataset for the optical imaging device, receive a subject- specific RGB dataset, generate a subject- specific hyperspectral dataset using the transformation matrix, and compute a blood hemoglobin level of the subject from the generated subject-specific hyperspectral dataset.
Claims
1. A bloodless system for numerically generated hyperspectral imaging data for measuring biochemical compositions, comprising: an optical imaging device adapted to acquire an RGB image from an area of interest, thereby generating a subject-specific RGB dataset representing the area; and a processor adapted to: receive a hyperspectral dataset representing an a priori hyperspectral data of the area of interest of a population to which the subject belongs, receive RGB response for each one of RGB channels of the optical imaging device, pair the corresponding RGB data with the hyperspectral data, obtain a transformation matrix adapted to convert the subject-specific RGB image dataset into a subject-specific hyperspectral dataset for the optical imaging device, receive a subject-specific RGB dataset, generate a subject-specific hyperspectral dataset using the transformation matrix, and compute a blood hemoglobin level of the subject from the generated subject-specific hyperspectral dataset.
2. The system of claim 1, wherein the paired pixels from the RGB image is associated with a 3×1 RGB value matrix.
3. The system of claim 2, wherein the paired pixels from the hyperspectral dataset is associated with an N×1 spectrum, where N represents discretized spectra between a lower bound and an upper bound.
4. The system of claim 3, wherein the lower and upper bounds are 400 nm and 800 nm, respectively.
5. The system of claim 1, wherein the transformation matrix is a form of inverse of the RGB response matrix of the RGB sensor that converts an RGB to a spectrum.
6. The system of claim 5, wherein the inverse of the transformation matrix is determined numerically by using the paired RGB and hyperspectral data of the population.
7. The system of claim 1, wherein the biochemical compositions include blood hemoglobin.
8. The system of claim 1, wherein the area of interest includes the inner surface of a subject's inner eyelid.
9. The system of claim 1, wherein the biochemical compositions are determined using spectral analysis.
10. The system of claim 9, wherein the spectral analysis includes a partial least square regression statistical modeling technique to first build a model from a training set of a first hyperspectral dataset vs. the biochemical composition and then apply the model to a second dataset from the generated hyperspectral image dataset.
11. A method for a bloodless numerically generated hyperspectral imaging data for measuring biochemical compositions, comprising: obtaining an RGB image from an area of interest, thereby generating a subject-specific RGB dataset representing the area of interest; receiving a hyperspectral dataset representing an a priori hyperspectral data of the area of interest for a population to which the subject belongs; receiving an RGB response for each one of RGB channels of the optical imaging device, pairing the corresponding RGB data with the hyperspectral data; obtaining a transformation matrix adapted to convert the subject-specific RGB image dataset into a subject-specific hyperspectral dataset for the optical imaging device; generating a subject-specific hyperspectral dataset using the transformation matrix; and computing a blood hemoglobin level of the subject from the generated subject-specific hyperspectral dataset.
12. The method of claim 1, wherein the paired pixels from the RGB image is associated with a 3×1 RGB value matrix.
13. The method of claim 12, wherein the paired pixels from the hyperspectral dataset is associated with an N×1 spectrum, where N represents discretized spectra between a lower bound and an upper bound.
14. The method of claim 13, wherein the lower and upper bounds are 400 nm and 800 nm, respectively.
15. The method of claim 11, wherein the transformation matrix is a form of inverse of the RGB response matrix of the RGB sensor that converts an RGB to a spectrum.
16. The method of claim 15, wherein the inverse of the transformation matrix is determined numerically by using the paired RGB and hyperspectral data of the population.
17. The method of claim 11, wherein the biochemical compositions include blood hemoglobin.
18. The method of claim 11, wherein the area of interest includes the inner surface of a subject's inner eyelid.
19. The method of claim 11, wherein the biochemical compositions are determined using spectral analysis.
20. The method of claim 19, wherein the spectral analysis includes a partial least square regression statistical modeling technique to first build a model from a training set of a first hyperspectral dataset vs. the biochemical composition and then apply the model to a second dataset from the generated hyperspectral image dataset.
21. A method for a bloodless numerically generated hyperspectral imaging data for measuring biochemical compositions, comprising: obtaining an RGB image from an area of interest, thereby generating a subject-specific RGB dataset representing the area of interest; receiving a hyperspectral dataset representing an a priori hyperspectral data of the area of interest for a population to which the subject belongs; receiving an RGB response for each one of RGB channels of the optical imaging device, pairing the corresponding RGB data with the hyperspectral data; converting the subject-specific RGB image dataset into a subject-specific hyperspectral dataset for the optical imaging device; and computing a blood hemoglobin level of the subject from the generated subject-specific hyperspectral dataset.
Description
BRIEF DESCRIPTION OF DRAWINGS
[0028]
[0029]
[0030]
[0031]
[0032]
[0033]
[0034]
[0035]
DETAILED DESCRIPTION
[0036] For the purposes of promoting an understanding of the principles of the present disclosure, reference will now be made to the embodiments illustrated in the drawings, and specific language will be used to describe the same. It will nevertheless be understood that no limitation of the scope of this disclosure is thereby intended.
[0037] In the present disclosure, the term "about" can allow for a degree of variability in a value or range, for example, within 10%, within 5%, or within 1% of a stated value or of a stated limit of a range.
[0038] In the present disclosure, the term "substantially" can allow for a degree of variability in a value or range, for example, within 90%, within 95%, or within 99% of a stated value or of a stated limit of a range.
[0039] For noninvasive blood hemoglobin (Hgb) measurements, it is important to rely on an appropriate anatomical sensing site where the underlying microvasculature is exposed on the skin surface without being affected by confounding factors of skin pigmentation and light absorption of molecules (e.g. melanin) in tissue. Commonly used clinical examination sites of pallor or microcirculation, such as the conjunctiva, the nailbed, the palm, and the sublingual region, provide a clue for an examination site selection. Specially, the palpebral conjunctiva (i.e. inner eyelid) can serve as an ideal site for peripheral access, because the microvasculature is easily visible and melanocytes are absent. The easy accessibility of the inner eyelid allows for reflectance spectroscopy and digital photography to be tested for anemia assessments.
[0040] While for blood Hgb quantifications, spectroscopic analyses of light absorption of Hgb in reflection spectra can be used to measure Hgb content in tissue, such methods rely heavily on complex and costly optical instrumentation such as spectrometers, imaging spectrographs, mechanical filter wheels, or liquid crystal tunable filters which can also result in significantly slow data acquisition, hampering clinical translation. While in the sister patent application referred to in the CROSS-REFERENCE TO RELATED APPLICATIONS section of the present disclosure to which the present disclosure claims priority, a method and system is disclosed whereby a simplified hyperspectral imaging device capable of generating a linescan dataset, such imaging apparatus remains unattainable in certain areas of the world, particularly in developing countries. Towards this end, the method and system of the present disclosure, provides a mathematical solution to construct hyperspectral with high spectral resolution or multispectral with several spectral measurements data from RGB images taken using a conventional camera (i.e. three-color sensors). This data-driven approach lays the groundwork for computational spectroscopy that overcomes the aforementioned hardware limitations.
[0041] Therefore, according to the present disclosure, we introduce a system and method referred to herein as the Virtual Hyperspectral Image Construction (VHIC) for a noninvasive blood Hgb measurements, with results that are relatively comparable to clinical laboratory blood Hgb tests.
[0042] Referring to
[0043] Referring to
[0044] A beta version of mHematology application is developed for data acquisition in low-end smartphones of SAMSUNG GALAXY J3 to build a robust mobile platform for all smartphones regardless of RGB image qualities. On the main application screen, the mHematology application displays a circle and arc to serve as guidance for locating the eyeball and the inner eyelid at a consistent distance and position within the image. To remove the background room light, the application automatically acquires two RGB photographs by controlling the built-in flashlight (i.e. white-light LED) to turn on and off. To compensate for the system response, two RGB images of a reflectance standard are taken. Similarly, the application automatically takes two RGB images with flash on and flash off for the individual's exposed eyelid.
[0045] Our data-driven computational spectroscopy provides the following advantages: the inner eyelid is used as a peripheral sensing site (vs. fingertip or fingernail) with the aforementioned advantages, hyperspectral image data construction vs. mere RGB images, spectroscopic analysis of Hgb (vs. empirical approach), and built-in camera in a smartphone (vs. costly accessory attachment). The two-step algorithm for blood Hgb estimation according to the present disclosure includes a first step which is to apply VHIC to the eyelid portion of the RGB image. The methodology then uses fixed design linear regression with polynomial features to build a construction matrix for the hyperspectral data from the RGB image.
[0046] Referring to
[0047] In order to make comparison with clinical data, reference is made to
[0048] Our spectroscopic and VHIC blood Hgb measurements are not affected by variations in the illumination and detection of the imaging systems as well as the background ambient room light as follows: The spectral intensity I.sub.m(λ) reflected from the inner eyelid in a given location of (x, y) is expressed as a function of the wavelength λ:
where L(λ) is the spectral shape of the illumination light source, C(λ) is the spectral response of all optical components in the imaging system (e.g. lenses and diffraction grating), D(λ) is the spectral response of the detector (e.g. mono sensor in the image-guided hyperspectral line-scanning system or three-color RGB sensor embedded in the smartphone), and r(λ) is the true spectral intensity reflected from the inner eyelid. First, to compensate for the system response (i.e. L(λ)C(λ)D(λ)), we use the reference reflectance standards that have a reflectivity of 99% in the visible range. I.sub.m(λ) is normalized by the reflectance I.sub.reference(λ) of the diffuse reflectance standard in which r.sub.reference(λ) = 0.99 in the visible range
Second, to remove the ambient stray and background light I.sub.background(λ), two measurements are acquired with the external light source (i.e. built-in flashlight LED of the smartphones) on and off. The measurements are repeated without the sample while the illumination is kept on. Finally, r(λ) is calculated by subtracting I.sub.background(λ) from each measurement such that:
This systematic and rigorous data acquisition procedure serves as the foundation for developing a reliable and universal blood Hgb computation algorithm without being affected by the ambient light and the different systems (e.g. smartphones). It should be noted that the built-in data acquisition step to factor out the contributions of room light conditions and different smartphone models provide a unique advantage to generate this reliable blood Hgb calculation.
[0049] VHIC in mHematology is the key concept to achieve spectrometer-free, yet hyperspectral, quantification of blood Hgb content. VHIC allows for the mathematical reconstruction of the full spectral information from an RGB image taken by a conventional camera (i.e. three-color information from R, G, and B channels). The mathematical relationship between the full spectrum and the RGB intensity is described as
where x is a vector corresponding to the reflection intensity in each R, G, and B channel, S is a matrix of the RGB spectral responses of the three-color sensor, r is a vector of the spectral intensity reflected from the inner eyelid, and e is a vector of the system noise. In our case, the hyperspectral construction from the RGB signal is an inverse problem such that the number of actual measurements (i.e. three-color information) is less than the dimensionality of the full spectrum with λ = λ.sub.1, λ.sub.2, ..., λ.sub.N. We took advantage of fixed-design linear regression with polynomial features to reliably construct the full spectral information r(λ.sub.1, λ.sub.2, ..., λ.sub.N) from the RGB signals x(R, G, B) of the three-color RGB sensor embedded in the smartphone, as shown in
where x is a 3 × 1 vector corresponding to the reflection intensity in each R, G, and B channel, S is a 3 × N matrix of the RGB spectral responses of the 3-color sensor (i.e. built-in camera of SAMSUNG GALAXY J3), r is an N × 1 vector that has the spectral reflection intensity, and e is a 3 × 1 vector of the system noise with zero mean. In our case, r(λ = λ.sub.1, λ.sub.2, ..., λ.sub.N) is discretized from 450 nm to 679 nm with a spectral interval of 1 nm. We take advantage of fixed-design linear regression to reconstruct hyperspectral data from RGB images. We paired the hyperspectral reflection dataset (acquired by the image-guided hyperspectral line-scanning system) and the RGB dataset (acquired by the RGB camera). It should be noted that the RGB dataset can also be generated by applying the RGB spectral responses to the hyperspectral dataset. X.sub.3×m and R.sub.N×m, are formed by adding x.sub.3×1 and r.sub.N×1 from m different measurements. The relationship in Equation (5-1) is described as:
which can be expressed as:
where the transformation (or extrapolation) matrix T.sub.N×3 = [S.sub.3×N].sup.-1. If Equation (5-3) is solved for the unknown T.sub.N×3, then T.sub.N×3 can be used to transform the RGB dataset into the hyperspectral reflection dataset. Each three-color sensor model in different cameras has unique RGB spectral responses with spectral overlaps among the R, G, and B channels (also known as the sensitivity function of the camera of SAMSUNG GALAXY J3). To effectively incorporate the RGB spectral response of the camera, we expanded
to
for maximizing the accuracy of the hyperspectral reconstruction such that:
here
can be expressed explicitly such that:
where the exact powers of i and j of the single and cross terms are uniquely determined for a specific three-color sensor model, by checking the error between the reconstructed hyperspectral data and the original data.
[0050] The inverse of the expanded transformation matrix
Equation (5-4) can be considered to be the minimum-norm-residual solution to
.Typically, this inverse problem is to solve a least-squares problem. We used QR decomposition, in particular the QR solver. After QR factorization is applied
,
to is estimated by minimizing the sum of the squares of the elements of
and is selected such that the number of nonzero entries in
is minimized. Overall, the computation of the transformation (extrapolation) matrix establishes VHIC, eliminating a need of bulky dispersion hardware components (e.g. spectrometer, spectrograph, mechanical filter wheel, or liquid crystal tunable filter).
[0051] We now describe the partial least square regression (PLSR). We built a model for computing blood Hgb content from the hyperspectral reflection data of the inner eyelid. Analytical model-based Hgb prediction methods are often used, because Hgb has distinct spectral signatures (e.g. Soret and Q bands) in the visible range. However, such model-based approaches often require a priori information on all possible light absorbers in tissue for reliable Hgb quantification. Thus, we made use of PLSR, which has been extensively used to model relationships among measured variables (i.e. predictors) and response variables (i.e. outcomes) in a variety of biological and medical applications. Because PLSR transforms high-dimensional measured variables onto a reduced space of latent variables, it is highly beneficial to examine the significance of individual measured variables by eliminating insignificant variables. While PLSR is based on the extraction of principal components, it incorporates variations of both predictor and outcome variables simultaneously, enhancing the prediction performance. Similar to principal component analysis, it is critical to determine an optimal number of components in PLSR. The determination of an optimal number of principal components in ten-fold cross-validation of PLSR is performed. In particular, as the number of partial least squares (PLS) components increases, the percentage variance in the true Hgb values (outcome variable) increases, while the mean squared prediction error has minimal values for 18 components. These numbers of PLS components contribute to appropriate representation of variations in the spectroscopic and laboratory blood Hgb values simultaneously, thus making its prediction errors lower. As a result, 18 PLS components are selected and used in the Hgb prediction model. Although the use of PLSR often avoids overfitting when the number of predictors is larger than the sample size, it is also important to evaluate the ability for predicting Hgb levels from a completely new dataset after the model is established properly. As described above, we defined the two datasets for training and testing the blood Hgb model without reutilization of data from the same individuals.
[0052] Based on the aforementioned information, a hyperspectral/imaging data processing and statistical analysis is now provided. For data processing and algorithm development, we computed the hyperspectral and RGB data and developed the blood Hgb prediction model and the VHIC algorithm using MATLAB (MATLAB R2018b, The MathWorks, Inc.). For statistical analyses, we evaluated multiple linear regression, linear correlations, and intra-class correlations using STATA (STATA 14.2, STATACORP LLC). We conducted Bland-Altman analyses to compare the blood Hgb measurements as non-parametric methods. The bias is defined by the mean of the differences between the hyperspectral (or VHIC) and central laboratory blood Hgb measurements (d = y.sup.VHI - y.sup.central):
The 95% limits of agreement (LOA) is defined by a 95% prediction interval of the standard deviation:
TABLE-US-00001 Patient characteristics Disorder Number of patients Cancer 45 HIV 46 Tuberculosis 8 Sickle cell disease 19 Acute kidney failure 1 Heart failure 4 Malaria 1 Anemia 3 Immune thrombocytopenic purpura 1 No major disease 25 Dataset Average Hgb (g dL.sup.-1) Training dataset (n = 138) 12.65 (SD = 3.11) Testing dataset (n = 15) 11.06 (SD = 3.62)
[0053] The hyperspectral data reconstructed from the smartphone RGB images of the inner eyelids reliably estimate the actual blood Hgb levels. We evaluated SAMSUNG GALAXY J3 (see
[0054] Although mHematology is not limited to anemia assessments, when anemia is defined as Hgb < 12 g dL.sup.-1 for females and Hgb < 13 g dL.sup.-1 for males, the receiver operating characteristic (ROC) curves of SAMSUNG GALAXY J3 report the comparable performance with the image-guided hyperspectral line-scanning system (see
TABLE-US-00002 Multiple linear regression of simple RGB information without VHI, acquired using the camera of SAMSUNG GALAXY J3 Source SS df MS Number of obs = 153 F(3, > F = 40.38 Model 690.829718 3 230.276573 Residual 849.758666 149 5.7030783 R-squared Adj R-squared Root MSE = 0.4484 = 0.4373 = 2.3881 Total 1540.58838 152 10.1354499 hgb Coef. Std. Err. t P>|t| [95% Conf. Interval] r 22.31038 3.008851 7.41 0.000 16.36485 28.25591 g -38.94604 4.708287 -8.27 0.000 -48.24967 -29.6424 b 18.88886 5.01539 3.77 0.000 8.978383 28.79933 .sub._cons 3.007299 2.860896 1.05 0.295 -2.645869 8.660467
[0055] Using all of 153 individuals, the multiple linear regression analyses of the actual blood Hgb levels (i.e. outcome variable) against the eyelid RGB signals (i.e. predictor variables) return underperforming R.sup.2 values of 0.45 in Galaxy J3 (see Tables 2). In other words, simple conjunctival redness scoring or pallor examination using mere RGB data may not provide sufficient information to reliably assess blood Hgb levels. Theoretically, mHematology can be used by an individual user as an ‘eyelid selfie.’ Overall, the reported results validate the potential of VHIC for translating RGB images into computational spectroscopy in a smartphone - mHematology that can be used for noninvasive, continuous and real-time blood Hgb measurements, which are comparable to clinical laboratory blood Hgb tests.
[0056] Those having ordinary skill in the art will recognize that numerous modifications can be made to the specific implementations described above. The implementations should not be limited to the particular limitations described. Other implementations may be possible.