MOBILE SYSTEM AND AUXILIARY METHOD FOR EVALUATING THERMOGRAPHIC BREAST IMAGES

Abstract

An architecture for a mobile system and process for evaluating breast thermographic images having a system capable of analyzing and evaluating breast thermal images of a patient captured by a mobile device connected to a thermal imager, wherein the analysis returns to an auxiliary index, which is evaluated through an artificial intelligence tool, so that a health professional can make a decision, being able to show a diagnosis or lead the patient to more specific exams.

Claims

1. An auxiliary mobile system for evaluating breast thermographic images, comprising: a) a mobile electronic device connected with at least one thermal imager, the mobile electronic device being provided with at least one electronic application (3) embedded provided with an interface; b) an intermediate module (2) communicating with the electronic application (3), the intermediate module (2) being provided with a data driver, in which the intermediate module (2) receives thermal image data from the electronic application (3); and c) a server (1), communicating with the intermediate module (2), provided with an artificial intelligence tool and a data repository; wherein, the intermediate module (2) sends the captured thermal image data to the server (1); and the artificial intelligence tool receives the thermal image data and returns an index of suspicion, which is received by the intermediate module (2) and directed to the electronic application (3) embedded in the mobile electronic device; being the artificial intelligence tool previously trained and fed with breast thermographic images.

2. The auxiliary mobile system according to claim 1, wherein the artificial intelligence tool comprises a convolutional neural network.

3. The auxiliary mobile system according to claim 1, wherein the intermediate module (2) stores the index of suspicion received in the data repository of the server (1).

4. The auxiliary mobile system according to claim 3, wherein the repository comprises a data history of at least one patient, wherein the history is updated with patient information and the index of suspicion.

5. The auxiliary mobile system according to claim 1, wherein the electronic application interface (3) comprises at least one framework support mask in capturing the breast thermographic image.

6. An auxiliary process for evaluation of breast thermographic images, comprising the steps of: a) collecting at least one breast thermographic image by an electronic application (3) embedded in a mobile electronic device; b) sending the breast thermographic image to a server (1), which is provided with an artificial intelligence tool evaluating the breast thermographic image and returning an index of suspicion; and c) receiving the index of suspicion by the electronic application (3) and making available the index of suspicion in the interface of the electronic application (3).

7. The auxiliary process according to claim 6, wherein the process is implemented in an auxiliary mobile system, as defined in claim 1.

8. The auxiliary process according to claim 6, further comprising a previous step of capturing the breast thermographic image by a thermal imager connected to the mobile electronic device, wherein the step of capturing the breast thermographic image comprises a sub-step of image framework from a support mask implemented in the interface of the electronic application (3).

9. The auxiliary process according to claim 6, wherein the step of sending the breast thermographic image to a server (1) comprises the sub-steps of: a) converting the breast thermographic image into a data string on the mobile electronic device; and b) receiving evaluation request by an intermediate module (2), communicating with the electronic application (3) and with the server (1), wherein the intermediate module (2) receives the data from the breast thermographic image and directs it to the server (1); wherein, the intermediate module (2) is responsible for receiving the index of suspicion evaluated by the artificial intelligence tool from the server (1) and sending it to the electronic application (3).

10. The auxiliary process according to claim 6, further comprising a step of sending and updating patient history data, wherein the intermediate module (2) sends patient information and/or index of suspicion to the server repository (1).

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The following figures are showed:

[0020] FIG. 1 shows an embodiment of the action flowchart.

[0021] FIG. 2 shows an embodiment of the system class diagram.

[0022] FIG. 3 shows an embodiment of the system architecture.

[0023] FIG. 4 shows an embodiment of the system architecture.

[0024] FIG. 5 shows an embodiment of the home screen of the mobile device platform.

[0025] FIG. 6 shows an embodiment of the application's pre-exam screen.

[0026] FIG. 7 shows an embodiment of the front image capture screen.

[0027] FIG. 8 shows an embodiment of the left side image capture screen.

[0028] FIG. 9 shows an embodiment of the right-side image capture screen.

[0029] FIG. 10 shows an embodiment of the image review screen.

[0030] FIG. 11 shows an embodiment of the result screen.

[0031] FIG. 12 shows an embodiment of the profile screen.

[0032] FIG. 13 shows an embodiment of the exam history screen.

[0033] FIG. 14 shows an embodiment of the exam detail screen.

[0034] FIG. 15 shows an embodiment of the thermal imager.

[0035] FIG. 16 shows an embodiment of the thermal imager connected with a mobile electronic device through physical connection.

[0036] FIG. 17 shows an embodiment of the thermal imager.

[0037] FIG. 18 shows an embodiment of the thermal imager connected with a mobile electronic device through physical connection.

[0038] FIG. 19 shows an embodiment of the thermal imager.

[0039] FIG. 20 shows an embodiment of the login screen, which shows the connection of the thermal imager to the mobile electronic device.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0040] The descriptions that follow are showed by way of example and do not limit the scope of the invention and will make the object of the present patent application more clearly understood.

[0041] In a first object, the present invention shows an auxiliary mobile system for evaluating breast thermographic images that comprises: a mobile electronic device connected with at least one thermal imager, the mobile electronic device being provided with at least one electronic application (3) embedded provided with an interface; an intermediate module (2) communicating with the electronic application (3), the intermediate module (2) being provided with a data driver, in which the intermediate module (2) receives thermal image data from the electronic application (3); and a server (1), communicating with the intermediate module (2), provided with an artificial intelligence tool and a data repository, wherein the intermediate module (2) sends the captured thermal image data to the server (1); and the artificial intelligence tool receives the thermal image data and returns an index of suspicion, which is received by the intermediate module (2) and directed to the electronic application (3) embedded in the mobile electronic device; being the artificial intelligence tool previously trained and fed with breast thermographic images.

[0042] In one embodiment, the mobile electronic device is a smartphone, cell phone, notebook, etc., which may be intrinsically provided with a thermal image sensor. In one embodiment, the thermal imager is a thermal image sensor module connectable to the mobile electronic device. For purposes of exemplification, the imager is commonly known as a thermal camera, and this term may be used/repeated throughout the present application, without limiting the scope of the invention.

[0043] In one embodiment, the thermal camera is connected to the mobile electronic device via wireless connection, e.g., Wi-Fi, Bluetooth, etc. In another embodiment, the thermal camera is connected to the mobile electronic device via physical connection (cables, connectors, etc.).

[0044] For the purposes of the present invention, the electronic application (3) is any software (or firmware) capable of being implemented in the electronic device, generating a user access interface, in such a way that the operation is performed by the user. In one embodiment, the electronic application (3) comprises an interface provided with a form to obtain statistical data from the patient (not necessarily personal data), in addition to having a tool that makes it possible to view the image captured by the thermal camera.

[0045] In a further embodiment, the electronic application (3) comprises a mask to assist in framing the thermal image captured by the thermal camera. Furthermore, in this embodiment, the electronic application (3) indicates the required or preferred positions for obtaining the thermal images.

[0046] The intermediate module (2) can be understood as any software structure/component mediating and targeting information between the electronic application (3) and the server (1), which may be implemented in the mobile electronic device, in the electronic application (3), on the server (1), on a web page, etc. In one embodiment, the intermediate module (2) is an API implemented locally on the mobile electronic device. In another embodiment, the intermediate module (2) is an API that runs in a Web environment. In another embodiment, the intermediate module (2) is an API that runs on the server (1) itself.

[0047] Thus, the intermediate module (2) is responsible for directing the data from the breast thermographic image to the server (1) and, soon after receiving the response from the server with the index of suspicion calculated, the intermediate module (1) directs the index both for the electronic application and for the repository of the server (1).

[0048] In one embodiment, the breast thermographic image data results from a conversion of the breast thermographic image into a data string (character chain). This conversion can be performed by the mobile electronic device, by the electronic application (3) or by the intermediate module (2).

[0049] In one embodiment, upon receiving the index of suspicion directed by the intermediate module (2), the repository updates a patient history with the index of suspicion obtained and with patient information (not necessarily patient's personal information).

[0050] As for the server (1) used in the system architecture, it can be physical or remote, capable of communicating with the electronic application (3). In one embodiment, the server (1) is a cloud operable remote server.

[0051] In one embodiment, said artificial intelligence tool is a convolutional neural network. This convolutional neural network is previously trained and fed with breast thermography images obtained, for example, in exams performed by people skilled in the subject. Based on this, the thermographic images were input into the neural network with the respective diagnostic results linked, for example, a sequence of thermal mammographic images wherein the cancer was diagnosed by a physician.

[0052] From this, the convolutional neural network can analyze captured thermographic images and show a probability that the patient will develop a tumor related to breast cancer.

[0053] In a second object, the present invention shows an auxiliary process for evaluating breast thermographic images comprising the steps of: collecting at least one breast thermographic image by an electronic application (3) embedded in a mobile electronic device; sending the breast thermographic image to a server (1), which is provided with an artificial intelligence tool that evaluates the breast thermographic image and returns an index of suspicion; and receiving the index of suspicion by the electronic application (3) and availability of the index of suspicion in the interface of the electronic application (3).

[0054] The step of collecting a breast thermographic image by the electronic application (3) can be performed: i) upon loading a breast thermographic image previously acquired and/or stored eventually in a database; or ii) by means of a thermal camera which, when capturing the image, forwards it to the electronic application (3).

[0055] In one embodiment, a previous step of capturing the breast thermographic image is performed by a thermal imager connected to the mobile electronic device, wherein the step of capturing the breast thermographic image comprises a sub-step of framing the image from a support mask implemented in the electronic application interface (3).

[0056] In one embodiment, the auxiliary process of the present invention is implemented in the system herein described.

[0057] Thus, starting from the previous embodiment—referring to the implementation in the system, the step of sending the breast thermographic image to a server (1) comprises the sub-steps: conversion of the breast thermographic image into a data string on the mobile electronic device; and receiving an evaluation request by an intermediate module (2), communicating with the electronic application (3) and with the server (1), wherein the intermediate module (2) receives the data from the breast thermographic image and forwards it to the server (1). Said intermediate module (2) is responsible for receiving the index of suspicion evaluated by the artificial intelligence tool of the server (1) and sending it to the electronic application (3).

[0058] Additionally, a step of sending and updating patient history data is performed, wherein the intermediate module (2) sends patient information and/or index of suspicion to the repository of the server (1).

[0059] In one embodiment, the artificial intelligence tool is a convolutional neural network previously trained and fed with breast thermography images and the respective diagnostic results previously identified. Based on this, the convolutional neural network performs the analysis step of the thermal image captured by the thermal camera and probabilistically evaluates the chances of the patient developing a breast cancer-related tumor.

[0060] In one embodiment, prior to the evaluation performed by the artificial intelligence tool, a pre-processing is performed on the breast thermographic image. Said pre-processing is performed by dividing the image into two halves, such as a horizontal flip technique is performed on the first half and the resulting is compared with the second half, to analyze the pattern of colors of the thermographic image, so that to identify similarities and/or disparities in patterns.

[0061] In one embodiment, the result of the analysis follows, in addition to the index of suspicion, a recommendation to seek or not to seek a specialized physician in the field.

[0062] In one embodiment, the evaluated thermal image is fed back into the convolutional neural network, together with a diagnosis/opinion of a specialized physician, for training and improving the analysis and evaluation performed by the convolutional neural network.

[0063] Thus, in view of the objects herein showed that materialize the proposed concept, it is possible to state that what is exposed in this document is advantageous compared to current technologies performing mammograms, since current mammography equipment is severely large and incapable of becoming mobile, in addition to there is an extreme demand. With this technology, on the other hand, it is possible to provide an index to assist a qualified health professional/specialist in the diagnosis of breast cancer, so that he/she can make the decision to refer the patient for more specific exams or even take the decision regarding the diagnosis of the existence or not of the breast tumor. The technology proposed in the present invention makes it possible to take this type of exam to places where there is no mammography device or there are long waiting queues for exams, which can work, for example, as a way of executing a triage or prioritizing care.

Example 1—System Architecture

[0064] The examples shown herein are intended only to exemplify one of the numerous ways of performing the invention, however without limiting its scope.

[0065] FIG. 1 shows an action flowchart the system performs to request analysis of a patient photograph. Initially, the patient fills in personal data, such as name, age, zip code, etc. Then, the image photographed by the thermal imaging sensor is selected and sent for analysis. Subsequently, the intermediate module (2), in this case an API, performs an image analysis request to the server (1) and then, the thermal image is analyzed by an artificial intelligence tool, which is a convolutional neural network. After the analysis, the API (2) receives the feedback from the server (1) and, from this saves the image analysis data in an analysis repository. Finally, the API receives the analysis status feedback and sends the index of suspicion to the electronic application (3).

[0066] The index of suspicion, in this case, is a probability that the patient develops a breast cancer-related tumor based on the analysis of the breast thermographic image.

[0067] For this analysis, a pre-processing is performed on the breast thermographic image before being submitted to the convolutional neural network. In this example, a pre-processing algorithm was implemented dividing the image into two halves, starting to work with image A and image B, then the algorithm applies a horizontal flip technique on image A so that it is compared with image B, analyzing the color pattern to identify similarity in colors. If a pattern disparity is detected, it is an indication of an anomaly that should be checked, because according to thermology studies, hyper-radiated areas in the human body indicate a high metabolic activity, which in turn, can indicate an area with an active inflammatory process.

[0068] The same process is applied to image B. Then image A is submitted to the process of identifying colors per pixel, assigned a scale from 0 to 1 that considers the body thermal variation from 25 to 36 degrees, wherein 0 is black/blue representing hypo-radiant areas and 1 is red/white representing hyper-radiant areas. In this way, the algorithm extracts the region of interest of the image (ROI).

[0069] This decimal scale represents each pixel of the image as a thermal factor, for example, if a pixel is red/white, it can have a value between 0.9 and 1. In this way, it is possible to identify areas with higher concentration of hyper-radiation compared to the same area as image B. The same process is applied to image B.

[0070] Therefore, it is possible to identify thermal anomalies that may indicate high metabolic activity where it normally would not be. Then, the image ROI is submitted to the convolutional neural network, making it possible to learn about suspicious or non-suspicious patterns.

[0071] FIG. 2 shows a system class diagram. Within the class called “Person”, the system receives the patient's name in a string, the age in integer character, the first exam in Boolean logic, the zip code in integer and the photo in image. Also, this class gets photo, cancels and requests analysis. The “Image” class receives name and image in string, as well as it opens the gallery and opens the thermal camera. In the “Analysis” class, it receives personal data, exam date and percentage, and performs the function of: analysis request, returning analysis, saving analysis and saving analysis return. Finally, in the “Diagnostic Interface” class, it performs the analysis.

[0072] FIG. 3 shows another view of the system architecture. The server (1) comprises image recognition through artificial intelligence and a database that receives data in real time. The intermediary module (2) comprises the programming and development of the functions performed in the system. The mobile device platform (3)—electronic application—comprises the system platform available in the smartphone application store. Said platform (3) is accessible to the patient, where the patient receives the result of the evaluation and analysis of the thermal image.

[0073] FIG. 4 also shows another representation of the system architecture. As shown, the intermediate module (2) receives a request from the mobile device platform (3), requests and receives a response from the main module (1), after artificial intelligence analysis, and sends data to the database for storage, as well as shows the analysis feedback to the platform (3).

Example 2—App Linda

[0074] Thus, in a performance of the invention presented herein, it is shown a sequence of steps that a patient and/or user performs when using the electronic application in the system, and process proposed herein. It is worth noting that both a patient and a healthcare professional can operate these steps in the system.

[0075] FIG. 5 shows the home screen of platform of the mobile device (3) corresponding to the login screen on the platform. Said platform (3) is a smartphone application.

[0076] On the login screen, the previously registered user can log in using their username (e-mail) and password. In case he/she has forgotten his/her access data, he/she can click on “forgot password” where he/she must fill his/her e-mail in a modal box to receive a temporary password. When accessing the application with the temporary password, the user is redirected to the profile screen where he/she can modify the temporary password. The temporary password lasts for 24 hours.

[0077] Also, the login screen has “login” integrations—Correct Login, Wrong Login, Temporary Password Login; “forgot password” integrations—Email found password sent, Email not found; and “Password Reset” integrations—Reset error, password changed successfully.

[0078] FIG. 6 shows the application pre-exam screen. In the pre-exam screen, the user includes the patient's information before the exam is performed. The user is asked for the following information: Date of Birth, Number of Health Care System, Date of the last period, if she uses hormones, if she has already had a mammogram, and city. Also, the application collects data indirectly to complement patient data, such as: GPS coordinates, temperature of the day and time of the exam. The app saves the information at memory to send the data at the end of the image capture. If there is an error in capturing the image or the app closes, when returning to the exams screen, the data is automatically filled in, making it possible to resume the exam. In addition, the exam screen is the main screen of the app, so the menu is on this screen, containing the items of “History”, which directs the user to the performed exam history screen, and the item “Profile” directs the user to the profile screen where the user can view her registration information.

[0079] Also, the pre-exam screen has integrations of “Save data offline”—the exam data is saved in the device memory in case an error occurs during the process, making it possible to resume the exam; “Save exam dates”—saves the date and time the exam was taken; and “Save GPS coordinates”—saves the device coordinates for checking, if the device is in the covered area.

[0080] In order to perform the exam, in this example, a protocol was proposed to be followed prior to capturing the thermal images. The requirements are described below:

[0081] 1. Patient must remove the upper part of the clothing covering her breasts (including bra);

[0082] 2. Patient needs to tie her hair if it is loose;

[0083] 3. Patient must remove any and all accessories that are around her neck, as well as necklaces, chokers, etc.;

[0084] 4. Patient must be positioned close to the wall, and the operator must stand at a distance of 40 cm from the tip of her feet to capture the image;

[0085] 5. Patient should raise her arms, interlacing her fingers behind her head;

[0086] 6. The room temperature must be between 22° and 23° C.;

[0087] 7. Patient must undergo thermal harmonization, waiting about 15 minutes in the air-conditioned environment;

[0088] 8. Patient must not have made any physical effort in the last 24 hours;

[0089] 9. Patient must not be breastfeeding;

[0090] 10. Patient must not have the flu;

[0091] 11. Patient must not have a fever or be feverishly;

[0092] 12. Patient must not be in menstrual period;

[0093] By checking these requirements, image capture can be performed.

[0094] FIGS. 7, 8, and 9 show the image capture screens (exam). FIG. 7 shows the frontal image, which is captured to be forwarded to the artificial intelligence tool. Optionally, the system can indicate that side photos of the patient breast are captured. FIG. 8 shows the left side image and FIG. 9 shows the right-side image.

[0095] After the user fills in the pre-exam screen with the necessary data and clicks on “start exam”, the user is directed to the image capture screen. The screen displays the thermal image captured by the camera and, under the view, there is a mask to help frame the breasts for the capture. Under the mask appear instructions for the user to obtain a better image. Clicking on the screen, the instructions disappear showing the thermal view along with the mask. The mask contains, in addition to the image framing guides, a button to capture the image and the thermal camera battery percentage. If the thermal camera is not connected to the device, a black view appears on the screen with the phrase “Connect thermal camera”.

[0096] After capturing the frontal image, as illustrated in FIG. 7, the user reviews the captured image, judging if the image is good enough to be sent for evaluation in the artificial intelligence model. If the image is believed unsatisfactory, the user has the option to click on “Redo Image” to repeat the capture. If the image is believed satisfactory, the user clicks on “send image”, submitting it for analysis by the artificial intelligence model. Optionally, the user can capture, in addition to the frontal image, the left and right-side images, as shown in FIGS. 8 and 9, and with this the system can compose a single image formed by the three views, which is illustrated by FIG. 10. After the final composition of the image, the user judges the image according to the same process indicated above. In this case, the user has the option to choose whether to redo one of the three photos needed for the exam or redo all the images.

[0097] Also, the image review screen has integration with “Submission to the artificial intelligence model”—it sends image and patient data and returns a percentage of chance of having a pathological pattern.

[0098] FIG. 11 shows the result screen. On the result screen, the app displays a summary with patient information and displays a percentage of the analysis performed by the artificial intelligence model. In addition, this percentage shows a text suggesting a next step to be followed by the user, such as, for example, that the user looks for a specialized physician if the percentage is high to positive. Also, clicking on “Redo Exam”, the user is directed to the “Image capture (exam)” screen to redo the patient image. If the user clicks on “End Exam”, they are directed to the “Pre-Exam” screen, which is the main screen of the application.

[0099] Also, the result screen has integration with “Save exam”—saves the exam as well as the image in the database and if there is a connection error, the user is warned and instructed to connect to a network. The app checks if there is an active internet connection.

[0100] FIG. 12 shows the profile screen. On this screen, the user views her registration data as well as edits her profile picture and changes the password to access the application. Also, the “back” function returns to the home screen.

[0101] Also, the profile screen has integrations with “Get user data”—the method brings the logged user data; and “Update User Data”—the user can change her photo or access password.

[0102] FIG. 13 shows the exam history screen. On this screen, the user views the list of exams performed by her. The history list is sorted by date, from newest to oldest. By scrolling through the history list, the application can continue loading the oldest historical exams, like an infinite list layout. Each line in the list displays the exam date and the patient number of the health care system. Clicking on a line in the list, the user is directed to the details screen of the clicked exam. At the top of the screen, the “back” button directs the user to the home screen.

[0103] Also, the exam history screen has integrations with “Get exam history”—method returns a list with the last 15 exams; and “Get More histories”—layout method that is called when the app reaches the end of the loaded list (15 exams).

[0104] FIG. 14 shows the exam detail screen. On this screen, it is possible to review the patient information as well as the result of the analysis of the artificial intelligence model. At the top of the screen, the “back” button directs the user to the home screen.

[0105] Also, the exam detail screen has integration with “Get exam detail”—it returns the details of the selected exam, giving the patient number of health care system.

[0106] FIG. 15 shows an embodiment of the thermal camera and FIG. 16 shows said camera connected with a smartphone through physical connection.

[0107] FIG. 17 shows another embodiment of the thermal camera and FIG. 18 shows said camera connected with a smartphone through physical connection.

[0108] FIG. 19 shows another embodiment of the thermal camera and FIG. 20 shows an app login screen, which shows that said thermal camera is connected to the smartphone via Wi-Fi.

[0109] Those skilled in the art will appreciate the knowledge presented herein and will be able to reproduce the invention in the modalities presented and in other variants and alternatives, covered by the scope of the claims below.