DETECTION OF FLU USING THERMAL IMAGING

Abstract

Methods for determining change in physical condition or illness of a mammal by obtaining a thermal image of the subject and determining the body temperature of the mammal based on the thermal image are described. An example method is implemented on a first electronic device having a first display and a thermal imaging hardware. The method includes obtaining the thermal image of a mammal using the first electronic device; comparing said thermal image to a reference thermal image of the subject at healthy state with no symptoms or characteristics of the physical condition or illness such as by way of example, flu, fever, hypothermia, ovulation, heat stress, cardiac condition; comparing the intensity of said thermal image to the reference thermal image; and in response to such comparison, determining whether the subject shows symptoms or hallmarks of a certain illness or condition marked by a change in body temperature and associated intensity of the thermal image; and displaying information regarding said determination on the first display of the first electronic device. The method includes first obtaining a reference thermal image of a mammal in a normal and/or healthy condition, then later obtaining additional image(s) for comparison to the reference image; and in response to such comparison, determining whether the subject shows symptoms of a certain illness or condition marked by a change in body temperature and associated intensity of the thermal image.

Claims

1. A method of detecting a change in body condition in a mammal comprising obtaining a first reference thermal image of the body or a part of the body of said mammal using a thermal imaging device when the mammal is within its normal body temperature range and does not have any symptoms or characteristics of illness or physical condition associated with body temperature different from its normal body temperature range; taking a second thermal image of the body or a part of the body of said mammal at a later time; comparing the intensity of the second thermal image to the first reference thermal image; and in response to such comparison, determining whether said mammal shows symptoms of a certain illness or physical condition associated with a change in body temperature and intensity of the thermal image; and displaying information regarding said determination on the first display of the first electronic device.

2. The method of claim 1 wherein the thermal imaging device is a smart phone or tablet having a camera with the capability to take thermal images.

3. The method of claim 1 wherein said mammal is a human.

4. The method of claim 3 wherein the part of the body for which the first reference thermal image and the second thermal image are obtained is of the face of said human.

5. The method of claim 3 wherein the illness or physical condition associated with a change in said human's body temperature is flu, stress, heatstroke, ovulation, hypothermia, or cardiac dysfunction.

6. The method of claim 1 wherein said mammal is a domesticated animal.

7. The method of claim 4 wherein the illness or physical condition associated with a change in said human's body temperature is flu, stress, heatstroke, hypothermia, or cardiac dysfunction.

8. A method of determining the body temperature of a mammal comprising obtaining thousands of infrared datapoints of said mammal's body or part of the body using thermal imaging, and using a deep learning engine to correlate these datapoints and thermal imaging to determine a single surrogate temperature value.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Reference will now be made, by way of example, to the accompanying drawings which show example embodiments of the present application and in which:

[0018] FIG. 1 is a diagram of a human face with numbers corresponding to universal mammalian facial regions;

[0019] FIG. 2 is a schematic of the invention's workflow as related to the flow of data.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0020] Example embodiments of the present disclosure will be apparent to those of ordinary skill in the art from a review of the following detailed descriptions in conjunction with the drawings.

[0021] Example embodiments of the present disclosure are not limited to any particular operating system, electronic device architecture, server architecture or computer programming language.

Regions of a Mammalian Face

[0022] By reference to FIG. 1, a mammalian face can be categorized into regions associated with unique thermal profiles, with the following representative examples of approximate facial regions: [0023] 1) Forehead [0024] 2) Eyebrows [0025] 3) Bridge of nose [0026] 4) Nose [0027] 5) Temples [0028] 6) Eyes [0029] 7) Cheeks [0030] 8) Chin [0031] 9) Mouth [0032] 10) Upper lip,

[0033] Where the numbered facial regions above correspond to the numbered areas in FIG. 1.

Schematic of Invention Workflow

[0034] By reference to FIG. 2, a graphical schematic is provided by which the bidirectional flow of data occurs. The 5 distinct but interrelated steps, corresponding to numbers 1-5 in the graphical schematic, are identified below: [0035] 1) Infrared camera records multiple thermal images from a mammal, e.g. a human face. [0036] 2) By transforming multiple readings into a composite body temperature, the related app provides a rapid, personalized alert if body temperature deviates from normal. [0037] 3) The system transforms multiple readings into a composite body temperature, then uploads information to a cloud platform, where the cloud refers to on-demand delivery of compute power, database storage, applications, and other IT resources via the internet. [0038] 4) The cloud platform integrates temperature measurements to derive actual patterns of infection and spread of contagion. [0039] 5) The cloud platform uses this real-world data to add rich context to the personalized alerts referenced in step 2.

SUMMARY OF INVENTION

Individual-Level Temperature Monitoring

[0040] Electronic devices such as smartphones, laptops, tablet computers, smart televisions, etc. are often adapted to include sensors including cameras. Such cameras may be adapted to take thermal images including the thermal image of mammals such as by way of example, humans and cattle.

[0041] Normal human body temperature is known to fall approximately within the range of 36.5-37.5 C. (97.7-99.5 F.). Currently, body temperature can be measured rapidly by infrared cameras or digital thermometers (which may also use infrared sensors). These are usually stand-alone devices that, in some cases, can be connected to a smartphone (either by physical cable or via wireless communication protocol) which then records and uploads the temperature measurement. However, the requirements for (1) a standalone measurement device, and (2) a step of connecting it to a smartphone, are not conducive to measuring temperature rapidly and frequently throughout a given day or night. Therefore, they have limited utility toward identifying the specific timeline of fever onset. Conversely, our approach leverages the existing integration of infrared imaging hardware within the device, especially a smartphone, which facilitates frequent interaction between the user and the temperature-measuring application. For example, each time the user deploys infrared facial recognition to unlock the smartphone, body temperature readings would be collected automatically and simultaneously. Because consumers interact with their smartphone devices on average 80 or more times a day (New York Post, 2017), this drives the collection of sufficient daily data points to capture not only the user's baseline healthy temperature but also the early onset of fever. Taking by way of example the Apple iPhone's Face ID, this system projects 30,000 infrared points onto the user's face, then constructs an infrared image from these data. Each of these infrared points may be mapped against specific regions of the user's face [SEE FIG. 1], because specific regions of the face are known to reflect the user's body temperature in distinct ways. For example, extremities like the nose and the ears are known to have lower temperature profiles compared to capillary-rich facial regions like the forehead and cheeks. This approach is far more robust than the aforementioned single-point infrared measurements taken at the forehead.

[0042] Ultimately, each device would run this unsupervised deep learning exercise to construct an accurate, data-driven temperature profile that serves as a personalized baseline of its owner, against which future infrared measurements of that owner would be compared. To increase the clinical relevance of this deep learning exercise, these datasets at first could also be compared and/or trained against measurements obtained with sensitive, traditional thermometers (e.g. rectal, inguinal, axillary). Ultimately, however, the accuracy of the deep learning model should circumvent the need for training against traditional thermometer benchmarks, and the elimination of traditional thermometers further increases the likelihood of an owner engaging frequently with his/her temperature measurement device.

[0043] Thermal images of an individual mammal such as a human may vary depending on the body temperature of the individual at the time the thermal image is taken; therefore, a thermal image can be correlated with the temperature of the individual. An individual with an elevated body temperature above 99.5 C. will have a different thermal image than when the individual has a normal body temperature in the range of approximately 97.7 C.-99.5 C. The fact that baseline normal temperature exhibits natural variation among humans (U.S. National Library of Medicine, 2018) underscores the value of having each device trained specifically against the unique temperature profile of its owner. Elevated body temperature may be associated with fever, ovulation, heat stress due to exertion, certain cardiac conditions, and other conditions described below. The change in thermal image can be used to correlate to elevated temperature, and therefore predict and detect fever or other physical condition. As a supervised machine learning exercise, the range of biologically normal human body temperature would be contrasted against the range of temperatures that correspond to fever. Thermal images also show variation if an individual's body temperature is below the normal range such as when the individual is suffering from hypothermia, congestive heart failure, or other conditions. At the individual level, the invention provides a person with real-time information on changes in temperature in the context of possible harmful conditions such as fever, heat stress, etc.

Community-Level Temperature Integration and Modeling

[0044] It is appreciated that electronic devices that contain sensors and cameras can also contain GPS elements. Therefore, thermal images that are captured by an electronic device can also be geotagged (the process of adding geographical identification metadata to an existing piece of data). Because thermal images can be correlated to temperature, and can also be geotagged, they have unique value as real-time, individual-level temperature data points that can be integrated into a large cloud-computing framework that tracks and analyzes actual incidences and patterns of disease outbreak. It is appreciated that electronic devices, especially smartphones, can also metatag thermal images with real-time, location-specific indicators of environmental factors such as temperature, humidity, elevation, climate, etc. The integration of these multiple real-time variables via deep learning potentially offers a far richer and more accurate assessment of influenza risk as compared to existing mathematical modeling approaches that rely on combinations of historical data, algorithm-generated estimates, or inferences based on online search engine results or location-based news reporting.

[0045] At the community level, the invention therefore provides a novel way to integrate individual-level data into a large, population-level computational model that can accurately track and predict migration patterns of infection-causing pathogens based on actual data points.

Description of the Community-Level Temperature Integration Invention [See FIG. 2]

[0046] The integration of individual-level datapoints into a large dataset uncovers several potential applications, as outlined below. [0047] 1) Body temperatures will be recorded passively as outlined in the Individual-Level temperature monitoring section. Because normal or reference temperatures will have been obtained for each individual, personalized temperature thresholds can be established. Temperatures are uploaded to a community-level computational framework and are tagged as normal or abnormal along with geocoding and other metadata such as temperature, humidity, elevation, climate, etc. [0048] 2) The computational framework, using deep learning approaches (for example, unsupervised approaches), integrates these individually-generated temperature measurements into a dynamic map that traces and predicts severity, location, kinetics, and migration of fever. As individuals commute or travel great distances, the computational framework accounts for these changes and integrates them into potential infection transmission patterns which are of clear value to national and global health organizations. This is a novel, optimal method for epidemiological monitoring that incorporates actual fever, environmental, and travel-related data points. [0049] 3) The framework would also deliver safety notifications to vulnerable populations. For example, if epidemiological monitoring indicates that a city is highly vulnerable to an outbreak, resources such as flu vaccines, flu treatments, and extra nursing staff in that city should be deployed towards hospitals, nursing homes, daycare, etc. A digital diagnostic product that helps identify and stratify individuals according to their likely benefit from a flu vaccine and/or flu medication is an example of a companion diagnostic, which is a popular tool by which pharmaceutical companies determine whether a therapeutic drug is suitable for a specific person and/or population. A companion diagnostic is a potential use case for this invention. Whereas current companion diagnostics make recommendations based only on test results from an individual, the proposed invention would integrate individual-level fever data along with societal and environmental inputs to generate a more holistic and comprehensive score related to, by way of example, the likely benefit of receiving flu medication early in the course of flu infection. [0050] 4) Trends and risk patterns generated by this computational framework can also be transmitted as actionable recommendations back to individuals. Examples of possible transmissions from computational framework to individual: [0051] a) The individual's fever is increasing rapidly, and the framework recommends that he/she should visit a doctor. The invention, via a permissions-based data disclosure hierarchy, could also be authorized to transmit the individual's fever history to the doctor's office in advance of the visit. [0052] b) The individual lives or works in a region that has been designated as a possible hotspot for an outbreak of one or more pathogens, and the framework recommends that he/she take extra precaution at home and/or work. [0053] c) The individual has scheduled travel to a region that has been designated as a possible infection hotspot, and the framework recommends postponing or adjusting his/her travel. Digital virtual assistants, for example Apple's Siri, can be leveraged to convey personalized risk recommendations back to users in an actionable and intuitive way since virtual assistants are commonly synchronized to a user's travel and/or work schedule. [0054] d) Senior citizens are expected to be highly susceptible to a particular season's predominant influenza strains, and the framework recommends that they receive the flu shot early in the season.

Additional Fever-Measurement Applications for Human Conditions

[0055] Hypothermia: this is the condition of the body dissipating heat and reaching temperatures below 35.0 C. (95.0 F.). Hypothermia has various causes including, but not limited to, drowning, skin disorders, burns, drugs, environmental conditions, medical interventions, and metabolic or neurological conditions. It is a potential cause of cardiac arrest, confusion, lethargy, loss of consciousness, coma, and death. Therefore, our rapid method of detecting abnormal temperature may provide critical real-time guidance to healthcare staff such as emergency medical technicians (EMTs), nurses, doctors, etc. when confronted with a possible case of hypothermia, especially early-stage hypothermia. In this use case, a device would not necessarily have to refer to a patient's personal baseline, because detection of the hypothermia threshold of 95.0 F. or below is of clinical value for any human regardless of their baseline temperature. A single device therefore could be used for multiple patients, because it would not be restricted to hypothermia evaluation in just one person. [0056] Stress: this is a complex condition involving hormones, altered heart rate, activation of the nervous system, etc. Fever is a known outcome of certain kinds of stress. Our fever-monitoring invention can therefore serve as a useful adjunct to digital apps and services that are already attempting to track and mitigate stress as a component of mental and emotional wellness. For example, an emotional wellness app may encourage an individual to face toward the device (smartphone) camera, which can track and record the individual's temperature while also projecting words, images, haptic feedback, and/or sounds that promote wellness and happiness during times of stress. [0057] Fertility: the fertility industry supports people that want to improve their success rate with either natural conception methods or in vitro fertilization procedures. Robust monitoring of ovulation cycles, which helps identify the precise window of ovulation, is a critical component of improving the likelihood of achieving pregnancy. Because temperature is correlated with ovulation, our invention would provide a real-time benefit for the synchronizing of ovulation with fertilization in the example of natural conception, and would also help with scheduling of egg retrieval in the case of in vitro fertilization. [0058] Heatstroke: This condition is caused by the body overheating, usually as a result of prolonged heat exposure or physical exertion in high temperatures (for example, strenuous exercise in hot climates, or firefighters returning from a fire). Heatstroke requires emergency treatment and if left untreated can quickly damage the brain, heart, kidneys and muscles (Mayo Clinic, 2018). The damage is exacerbated by delays in treatment, increasing risk of serious complications or death, and emphasizes the value of rapid, real-time determination of heatstroke. A single device would not be restricted to evaluation of heatstroke in just one person since the clinical threshold is relatively universal; a single device could therefore be used for multiple people.

Additional Nonhuman Applications

[0059] Ovulation in animals: The successful breeding of domesticated mammals is a critical component of food security for humans. For mammalian species including, for example, cattle, pigs, and sheep, ovulation detection/prediction enables the optimal scheduling of breeding techniques including (but not limited to) Artificial Insemination, Superovulation, In vitro Fertilization, and Embryo Transfer. In this application the infrared camera sensor would be directed toward the female's vaginal area to measure cyclical changes in vulvar temperature associated with estrus, the period of sexual receptivity and fertility in many female mammals (Sakatani, Takahashi, & Takenouchi, 2016). This would enable both veterinarians and non-veterinarians to rapidly and objectively categorize female animals by stage of estrus. Compared to previous applications of infrared thermography for animal estrus (Scolari, Clark, & Knox, 2011), the described invention offers a much greater level of insight because (1) it can rapidly integrate vulvar temperature along with environmental factors, (2) it can complete calculations within the on-site device itself instead of transmitting data to a separate off-site computer, and (3) it can simultaneously generate predictions and recommendations regarding animal infections with fever components. Furthermore, the prevalence of low-cost transmitter tags in the livestock industry (for example RFID ear tags) enables the automatic synchronizing of each infrared measurement to a specific animal, thereby expediting the collection and organization of animal temperature data.

REFERENCES

[0060] Apple. (2018). About Face ID advanced technology. Retrieved from Apple Support: [0061] https://support.apple.com/en-ca/HT208108

[0062] Berksoy, E., Ba, O., Yazici, S., & elik, T. (2018, February). Use of noncontact infrared thermography to measure temperature in children in a triage room. Medicine (Baltimore), e9737.

[0063] Ioannou, S., Morris, P., Mercer, H., Baker, M., Gallese, V., & Reddy, V. (2014 Aug. 4). [0064] Proximity and gaze influences facial temperature: a thermal infrared imaging study. Front. Psychol.

[0065] Mayo Clinic. (2018). Heatstroke. Retrieved from Mayo Clinic: [0066] https://www.mayoclinic.org/diseases-conditions/heat-stroke/symptoms-causes/syc-20353581

[0067] New York Post. (2017 Nov 8). Americans check their phones 80 times a day.

[0068] Priest, P., Duncan, A., Jennings, L., & Baker, M. (2011). Thermal Image Scanning for Influenza Border Screening: Results of an Airport Screening Study. PLoS One, 6(1), e14490.

[0069] Sakatani, M., Takahashi, M., & Takenouchi, N. (2016). The efficiency of vaginal temperature measurement for detection of estrus in Japanese Black cows. J Reprod Dev, 62(2), 201-207.

[0070] Scolari, S., Clark, S., & Knox, R. (2011). Vulvar skin temperature changes significantly during estrus in swine as determined by digital infrared thermography. J Swine Health Prod., 19(3), 151-155.

[0071] Sun, L. (2017 Oct. 25). Drop in adult flu vaccinations may be factor in last season's record-breaking deaths, illnesses. The Washington Post. Retrieved from [0072] https://www.washingtonpost.com/health/2018/10/25/drop-adult-flu-vaccinations-may-be-factor-last-seasons-record-breaking-deaths-illnesses/noredirect=on&utm_term=.ccf1ce80a9fd

[0073] U.S. National Library of Medicine. (2018, October). Body temperature norms. Retrieved from Medline Plus: https://medlineplus.gov/ency/article/001982.htm

[0074] Wagner, M. (2013, Oct. 17). Emerging Biometric Technology: Infrared Facial Matching. Retrieved from Dev Technology Group: http://devtechnology.com/2013/10/emerging-biometric-technology-infrared-facial-matching/