Health Monitoring System Including Privacy-Ensuring Obfuscated Camera Images
20200358925 ยท 2020-11-12
Inventors
- David R. Hall (Provo, UT, US)
- Daniel Allen (Provo, UT, US)
- Conrad Rosenbrock (Provo, UT, US)
- Ben Swenson (Lehi, UT, US)
- Daniel Hendricks (Provo, UT, US)
- Andrew Nguyen (Provo, UT, US)
Cpc classification
A61B5/0077
HUMAN NECESSITIES
A61B1/313
HUMAN NECESSITIES
G06F21/83
PHYSICS
A61B5/0816
HUMAN NECESSITIES
H04N7/188
ELECTRICITY
A61B5/1032
HUMAN NECESSITIES
A61B3/12
HUMAN NECESSITIES
G06F21/6254
PHYSICS
H04N7/181
ELECTRICITY
G06V20/52
PHYSICS
A61B5/444
HUMAN NECESSITIES
A61B1/307
HUMAN NECESSITIES
A61B1/267
HUMAN NECESSITIES
A61B3/14
HUMAN NECESSITIES
A61B5/02055
HUMAN NECESSITIES
A61B5/448
HUMAN NECESSITIES
A61B1/247
HUMAN NECESSITIES
A61B1/24
HUMAN NECESSITIES
A61B5/4561
HUMAN NECESSITIES
International classification
A61B1/24
HUMAN NECESSITIES
A61B1/247
HUMAN NECESSITIES
A61B1/267
HUMAN NECESSITIES
A61B1/307
HUMAN NECESSITIES
A61B1/313
HUMAN NECESSITIES
A61B3/117
HUMAN NECESSITIES
A61B3/12
HUMAN NECESSITIES
A61B3/14
HUMAN NECESSITIES
A61B5/00
HUMAN NECESSITIES
A61B5/01
HUMAN NECESSITIES
A61B5/0205
HUMAN NECESSITIES
A61B5/103
HUMAN NECESSITIES
A61B5/107
HUMAN NECESSITIES
A61B5/11
HUMAN NECESSITIES
G06F21/62
PHYSICS
H04L9/06
ELECTRICITY
H04N1/32
ELECTRICITY
Abstract
The unobtrusive health-monitoring apparatus includes a camera, a controller, a data transmission port, and, optionally, a graphics processing unit (GPU) which executes nonlinear transformation algorithms. The camera may be inconspicuously disposed within a fixture in the room. The camera collects graphic data which may include still images or video of the user, or both. The GPU may execute the nonlinear transformation algorithms which may transform the graphic data into formats which cannot be recognized by humans. However, these formats preserve features that can be parsed by machine learning methods and used for health tracking purposes. The formats cannot be parsed by humans or be converted back to the original image or video by mathematical inversion or available computational methods. User privacy is thus preserved. The graphic data may be transmitted to a remote processor. The remote processor may perform algorithms which create a health status analyses.
Claims
1. A health-monitoring system comprising: at least one camera positioned to capture graphic data from a user; a controller; the controller comprising a memory, wherein the controller is in electronic communication with the at least one camera, wherein the controller receives graphic data from the at least one camera; a communication port, wherein the communication port is in electronic communication with the controller, and wherein the controller comprises instructions to transmit the graphic data through the communication port; and non-transitory computer-readable media comprising instructions for applying an algorithm to transform the graphic data into obfuscated graphic data, to create an analysis of the obfuscated graphic data, and to create an assessment of a user's health status.
2. The health-monitoring system of claim 1, further comprising a light source, wherein the light source directs light toward an angle of view of the at least one camera.
3. The health-monitoring system of claim 2, wherein the light source provides light which consists of one or more of the following types of light: visible light, high color temperature light, infrared light, structured light, and modulated light.
4. The health-monitoring system of claim 1, wherein the at least one camera comprises of one or more of a 3D time of flight camera, a stereoscopic camera, an infrared thermal imaging camera, a video camera, a structured light 3D scanner, and a still image camera.
5. The health-monitoring system of claim 1, wherein the at least one cameras is disposed within a diagnostic tool, wherein the diagnostic tool consists of one or the following: an otoscope, an ophthalmoscope, an endoscope, a laparoscope, a laryngoscope, a colposcope, a hysteroscope, a bronchoscope, a pharyngoscope, a laparoscope, and a dental tool.
6. The health-monitoring system of claim 1, wherein the at least one camera is disposed within a fixture.
7. The health-monitoring system of claim 6, wherein the fixture comprises a partially silvered mirror, wherein the mirror is disposed in front of a lens of the at least one camera.
8. The health-monitoring system of claim 1, further comprising one or more auxiliary sensors, wherein the one or more auxiliary sensor is in electronic communication with the controller, and wherein the controller comprises instructions to actuate the health-monitoring apparatus in response to a signal the controller receives from the one or more auxiliary sensors.
9. The health-monitoring a system of claim 1, wherein the assessment of the user's health status comprises an analysis of at least one of the following list of clinical observations: speed of user movement, user movement patterns, user posture, swelling beneath the user's eyes, swollen lymphatic glands, visual photoplethysmography, heart rate, moles, skin growths, body shape, skin coloration, sclera coloration, degree of transparency of a lens in the user's eye, hair loss, breathing rate, time in front of the at least one camera, and body shape.
10. The health-monitoring system of claim 1, wherein the controller further comprises a graphics processing unit, wherein the graphics processing unit comprises instructions for performing a plurality of nonlinear transformations on the graphic data thereby converting the graphic data to obfuscated graphic data which is unrecognizable by a human viewer and which retains a feature recognizable by the algorithm, and wherein the non-transitory computer-readable media comprises instructions for applying a algorithm to create an analysis of the obfuscated graphic data and an assessment of a user's health status.
11. The health-monitoring system of claim 10, wherein the obfuscated graphic data is created using one or more of the following techniques: deep convolution; compressed sensing obfuscation using sparse basis expansion and discarding basis functions; and block-chain based obfuscation, wherein difference hashing between frames is secured via blockchain.
12. The health-monitoring system of claim 10, further comprising a light source, wherein the light source directs light toward an angle of view of the at least one camera.
13. The health-monitoring system of claim 12, wherein the light source provides light which consists of one or more of the following types of light: visible light, high color temperature light, infrared light, structured light, and modulated light.
14. The health-monitoring system of claim 10, wherein the at least one camera comprises at least one of a 3D time of flight camera, a stereoscopic camera, an infrared thermal imaging camera, a video camera, a structured light 3D scanner, and a still image camera.
15. The health-monitoring system of claim 10, wherein the one or more cameras is disposed within a diagnostic tool, wherein the diagnostic tool comprises of one or more of an otoscope, an ophthalmoscope, an endoscope, a laparoscope, a laryngoscope, a colposcope, a hysteroscope, a bronchoscope, a pharyngoscope, a laparoscope, and a dental tool.
16. The health-monitoring system of claim 10, wherein the at least one camera is disposed within a fixture.
17. The health-monitoring system of claim 16, wherein the fixture comprises a partially silvered mirror, wherein the mirror is disposed in front of a lens of the at least one camera.
18. The health-monitoring system of claim 10, further comprising one or more auxiliary sensors.
19. The health-monitoring system of claim 18, wherein the one or more auxiliary sensor is in electronic communication with the controller, and wherein the controller comprises instructions to actuate the health-monitoring apparatus in response to a signal the controller receives from the one or more auxiliary sensors.
20. The health-monitoring system of claim 10, wherein the assessment of the user's health status comprises an analysis of at least one of the following list of clinical observations: speed of user movement, user movement patterns, user posture, swelling beneath the user's eyes, swollen lymphatic glands, visual photoplethysmography, heart rate, moles, skin growths, body shape, skin coloration, sclera coloration, hair loss, breathing rate, time in front of the camera, and body shape.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Non-limiting and non-exhaustive implementations of the disclosure are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. Advantages of the disclosure will become better understood regarding the following description and accompanying drawings where:
[0015]
[0016]
[0017]
[0018]
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
[0026]
DETAILED DESCRIPTION
[0027] The following description recites various aspects and embodiments of the inventions disclosed herein. No particular embodiment is intended to define the scope of the invention. Rather, the embodiments provide non-limiting examples of various compositions, and methods that are included within the scope of the claimed inventions. The description is to be read from the perspective of one of ordinary skill in the art. Therefore, information that is well known to the ordinarily skilled artisan is not necessarily included.
Definitions
[0028] The following terms and phrases have the meanings indicated below, unless otherwise provided herein. This disclosure may employ other terms and phrases not expressly defined herein. Such other terms and phrases shall have the meanings that they would possess within the context of this disclosure to those of ordinary skill in the art. In some instances, a term or phrase may be defined in the singular or plural. In such instances, it is understood that any term in the singular may include its plural counterpart and vice versa, unless expressly indicated to the contrary.
[0029] As used herein, the singular forms a, an, and the include plural referents unless the context clearly dictates otherwise. For example, reference to a substituent encompasses a single substituent as well as two or more substituents, and the like.
[0030] As used herein, for example, for instance, such as, or including are meant to introduce examples that further clarify more general subject matter. Unless otherwise expressly indicated, such examples are provided only as an aid for understanding embodiments illustrated in the present disclosure, and are not meant to be limiting in any fashion. Nor do these phrases indicate any kind of preference for the disclosed embodiment.
[0031] As used herein, safe means that data which could reasonably be exploited by actors with malicious intent is prevented from being accessed by such actors or their agents.
[0032] As used herein, secure means only authorized parties receive data.
[0033] As used herein, private means the data is kept exclusively within the health monitoring system to which the user has subscribed.
[0034] As used herein, user means the individual from whom the disclosed system is collecting graphical data.
[0035] As used herein, electronic means either wired or wireless. For example, the phase in electronic communication means either a wired communication between two devices or could mean a wireless communication between devices, such as WiFi.
Exemplary Embodiments
[0036] While this invention is susceptible of embodiment in many different forms, there are shown in the drawings, which will herein be described in detail, several specific embodiments with the understanding that the present disclosure is to be considered as an exemplification of the principals of the invention and is not intended to limit the invention to the illustrated embodiments.
[0037] We disclose a system for discrete and unobtrusive health monitoring using video, still photos, or both. The system may be used in the home, care facility, or clinical setting. The system may include one or more cameras and a controller. The controller may include a memory and be in electronic communication with the camera. The electronic communication may be either wired or wireless. The one or more cameras may collect graphic data and transmit the graphic data to the controller.
[0038] The controller may include a memory which may house non-transitory computer-readable medium. The non-transitory computer-readable medium may include instructions for applying an algorithm, which in some embodiments is a machine-learning algorithm, to create an analysis of the graphic data. The algorithm may further convert the analysis of the graphical data to an assessment of the user's health status. Alternatively, separate algorithms may perform the tasks of directly analyzing the graphic data and creating an assessment of the user's health. An algorithm may also create a report of the assessment of the user's health and transmit the report to the user or the user's healthcare provider.
[0039] The controller may also include a communication port which is in electronic communication with the controller, either wired or wirelessly. The controller may include non-transitory computer-readable medium which includes instructions to transmit the graphic data through the communications port. The communication port may transmit the graphic data to a remote server, for example, a cloud computing resource. Graphic data may be stored on the remote server. The remote server may store and implement non-transitory computer-readable medium which includes instructions for applying one or more machine-learning algorithms to create an analysis of the graphic data and to create an assessment of a user's health status. In some embodiments, the remote server is secure, safe, private, or a combination thereof.
[0040] Some embodiments may include a light source and a light source modulator. The light source modulator may direct light toward an angle of view of the camera. The light provided by the light source may enable the camera to collect better quality still or video images which are more readily analyzable. The modulator may select the wavelength of light or other characteristics of the light provided by the light source. For example, the light source may provide one or more of the following types of light: visible light, high color temperature light, infrared light, structured light, and modulated light. In an example, visible light may be useful to illuminate the subject to record movement and behavior. Controlled wavelength illumination may allow a degree of depth profiling. Infrared illumination may reveal vascular structure deeper in the skin. High color temperature (blue-tint) light may be useful to detect features on the skin surface. Infrared illumination may be used to record the user's motion in an otherwise unlit environment. Stereoscopy, time of flight, or structured lighting may be used to collect 3D information about the user.
[0041] The camera may be a single camera. Alternatively, the system may include a plurality of cameras. Each camera may include, but is not limited to, one or more of the following types of cameras: a 3D time of flight camera, a stereoscopic camera, an infrared thermal imaging camera, a video camera, a structured light 3D scanner, and a still image camera.
[0042] In an example, a camera is disposed within an otoscope, an ophthalmoscope, an endoscope, a laparoscope, a laryngoscope, a colposcope, a hysteroscope, a bronchoscope, a pharyngoscope, a laparoscope, or a dental tool. In some examples, the camera in these devices is a perspective camera.
[0043] In some embodiments, the camera may be disposed within a fixture within the room. By disposing the camera within a fixture, the camera is unobtrusive to the user and therefore, unlikely to cause the user to modify his or her behavior due to self-consciousness when the camera is collecting graphic data. In an example, the fixture may be placed in a bathroom, a kitchen, a hallway, an entryway, an office, or a living room.
[0044] In an example, the fixture includes a partially silvered mirror. The mirror may be disposed between the user and the lens of the camera (in front of the lens of the camera). The user may view himself or herself in the mirror in a traditional manner while the camera collects graphic data from the user. The mirror may appear no differently to a user than a traditional mirror. This characteristic may prevent the user from behaving differently because the user is focusing on the fact that the camera is collecting graphical data. In another example, a small window is disposed in the fixture and over the camera lens so that the camera may collect graphic data on the other side of the fixture.
[0045] In addition to the one or more cameras, the system may include one or more auxiliary sensors, each which may be in electronic communication with the controller. The non-transitory computer-readable medium on the controller may include instructions to actuate the health-monitoring apparatus when the controller receives a signal from an auxiliary sensor. In an example, an auxiliary sensor may be a pressure sensor. In a more specific example, the pressure sensor may be placed on or within flooring in the environment surrounding the one or more cameras. In some embodiments, the pressure sensor may be a floor scale or pressure sensitive floor mat. When a user enters the environment and crosses over the pressure sensor, the pressure sensor may detect the user's mass and send a signal to the controller. In another example, the auxiliary sensor may be a motion detector which sends a signal to the controller when it detects the presence of a user in the environment. In the embodiments which include an auxiliary sensor, the user does not need to consciously actuate the system. The system's presence may, therefore, be less noticeable. Consequently, the user may be less likely to modify his or her behavior due to self-consciousness about being recorded by the camera.
[0046] In some embodiments, the system may include an interactive display for the user. In the embodiment which includes a mirror, the display may appear on the mirror. The display may provide health data to the user, as measured in real time (such as the measured weight from the floor scale), as well as health data which is the result of the analysis performed by the system. The display may also include indicators of when images are being captured and when the process has completed. The display may also be connected to other information sources, such as a smart home system, and may be used to display non-health related data, such as weather, schedule or news.
[0047] The one or more cameras may record a plurality of clinically relevant observations. These may include behaviors and movements, for example, the speed of the user's movement, the patterns of the user's movements, the user's posture, the time of day that the user appears in front of the camera, uncharacteristic use of the non-dominant hand, and the amount time the user spends in front of the camera. Observations relating to behavior and movement may assist in the diagnosis of psychological, neurological, skeletal, and motor aberrations.
[0048] The one or more cameras may also record physiological changes in a user including, but not limited to, edema including swelling beneath the user's eyes, swollen lymphatic glands, visual photoplethysmography, heart rate, moles, skin growths, skin coloration, sclera coloration, mucous membrane coloration, coloration of the ear canal, hair loss, breathing rate, and body shape.
[0049] In an example, the camera may be disposed within an otoscope and collect graphic data which may be used to identify inflammation within the external auditory canal.
[0050] In another example, the camera may collect graphic data which may be used to assess the transparency of the lens of the user's eye. If the lens is relatively opaque, the assessment of the user's health status may indicate a possible cataract. Scratches and abrasions on the user's eye may also be detected. In such embodiments, the camera may be disposed within an ophthalmoscope or behind a mirror. In the example in which the camera is disposed behind a mirror, the mirror may be a bathroom mirror which the user spends time standing directly in front of and facing while performing toiletry tasks, for example, hair and teeth brushing. Consequently, the camera may be able to collect images of the user's eyes as the user peers directly into the mirror.
[0051] In an example in which the camera is disposed within a fixture, for example, behind a mirror, the camera may detect coloration in the user's skin. A yellow coloration may indicate jaundice while a blue coloration may indicate cyanosis. A pale coloration may indicate pallor which is a symptom of emotional stress, anemia, and other pathologies. The camera may measure the color and shape of a mole or skin growth leading to a dermatological diagnosis. The camera may record movement in the user's chest which may be used to calculate the user's breathing rate.
[0052] In some embodiments, the controller further includes a graphic processing unit (hereinafter, GPU). The GPU may include non-transitory computer-readable medium which includes instructions for performing algorithms which transform and obfuscate the graphic data the camera collects. The algorithms may perform multiple convolutions or nonlinear transformations on the graphic data which convert the graphic data to a form which is unrecognizable by a human viewer, but which also retains a feature recognizable by the machine-learning algorithm. The graphic data may be transformed into several kinds of transformed data by several different transformations, each transformation preserving a feature recognizable or useful for a machine-learning or computer classification algorithm. The graphic data is now obfuscated graphic data. In addition to being unrecognizable by humans, the obfuscated graphic data may not be converted back to the original graphic data by mathematical inversion or available computational methods. Examples of techniques which may be used to transform the graphic data include deep convolution, compressed sensing obfuscation using sparse basis expansion and discarding basis functions; and block-chain based obfuscation. In the block-chain obfuscation technique, difference hashing between frames may be secured via blockchain.
[0053] Obfuscation may be used to preserve safety, security, and user privacy. After the graphic data has been transformed, it may be transmitted to a remote processor which may apply algorithms to assess the status of the user's health. Because the graphic data has been transformed as described herein, the system maintains the user's privacy and the safety and security of the data is enhanced relative to graphic data which has not been transformed.
[0054] An example of an obfuscating, non-linear transformation is a multi-layer convolution network with non-linear pooling layers. A series of 2D kernels may be applied to each 2D frame of a video or still image collection using convolution, possible varying the padding and step size. This may produce a set of convolved images for each image (one for each kernel). At this point, it is still possible for an inverse convolution to reconstruct the original image. However, a pooling layer is then applied whereby the size of each convolved image is reduced by taking the average or maximum absolute value of all pixels within a neighborhood; for example, a group of 4 pixels may be replaced by a single pixel that has the maximum value of any of the original 4 pixels. This step is non-linear and cannot be exactly inverted. Typical implementations may have an alternating succession of 2D convolution and pooling layers to produce a final set of convolved images that are flattened to a single vector. This vector may encode information that may be useful to a machine-learning algorithm, but will appear to be random noise to a human viewer.
[0055] Another approach to obfuscating the video stream involves expanding each video frame using a sparse representation (for example, wavelet or discrete cosine) using an algorithm which may include compressed sensing techniques. These sparse regression algorithms penalize solutions to have a minimal l_1 norm, which preferentially selects sparse solutions. Then, machine-learning algorithms may be given access to data in the transformed space to train on. Only those basis functions that are relevant to the machine learning model are selected, allowing a large number of basis functions to be discarded. This loss of basis function renders the images unrecognizable.
[0056] Another way to obfuscate graphic data collected by the camera within the disclosed system is using a blockchain-based system. In this example, the user and server exchange public keys. By giving the server the user's public key, the user is granting permission for the server to access the obfuscated video stream. A difference hash is computed for each frame in the video relative to its neighboring frames. This hash is then combined with the user's private key and the server public key to generate a transaction in a blockchain ledger. Each frame captured and sent to the server is traceable to a transaction that was approved by the user for use by the server. Only the user or the server possess the keys to decrypt the difference hash in the video frames.
[0057] Referring now to the drawings,
[0058]
[0059]
[0060]
[0061]
[0062]
[0063] Should the image in
[0064]
[0065] While specific embodiments have been illustrated and described above, it is to be understood that the disclosure provided is not limited to the precise configuration, steps, and components disclosed. Various modifications, changes, and variations apparent to those of skill in the art may be made in the arrangement, operation, and details of the methods and systems disclosed, with the aid of the present disclosure.
[0066] Without further elaboration, it is believed that one skilled in the art can use the preceding description to utilize the present disclosure to its fullest extent. The examples and embodiments disclosed herein are to be construed as merely illustrative and exemplary and not a limitation of the scope of the present disclosure in any way. It will be apparent to those having skill in the art that changes may be made to the details of the above-described embodiments without departing from the underlying principles of the disclosure herein.