METHOD FOR OBTAINING IMAGES FOR ASSESSING ORAL HEALTH USING A MOBILE DEVICE CAMERA AND A MIRROR

Abstract

A method for obtaining images for assessing oral health using a mobile device (103) having a screen (205) on one side of a housing of the mobile device (103) and a camera (207) on the opposite side of the housing (103), comprising: —on the mobile device (103) performing a dialogue with a user (101) of the mobile device (103), —determining an area of the user's oral cavity with symptoms based on the dialogue, —instructing the user (101) to position him-/herself in front of a minor (102) for capturing an image of the area of the user's oral cavity with symptoms, —capturing at least one image of the area of the user's oral cavity with symptoms with the camera (207) as instructed, and—either processing the at least one image locally on the mobile device (103), or—transmitting the at least one image or a representation thereof to a remote server (409).

Claims

1-8. (canceled)

9. A computer program product encoded on a machine-readable non-transitory storage medium and operable to cause a processor of a mobile device (103) having a screen (205) on one side of a housing of the mobile device (103) and a camera (207) provided on the opposite side of the housing of the mobile device (103) to operate the mobile device to perform the steps of: on the mobile device (103) performing, a dialogue with a user (101) of the mobile device (103), the dialogue comprising prompting one or more questions to the user (101) via a user interface of the mobile device (103) and recording answers to the questions from the user (101) via the user interface of the mobile device (103), determining an area of the user's oral cavity with symptoms based on the user's answers to the questions; instructing the user (101) to position him-/herself in front of a mirror (102) for capturing an image of the area with symptoms, instructing the user (101) to rotate the mobile device (103) such that the screen (205) faces the mirror (102) and the camera (207) faces the user (101), with the camera (207) being directed towards the area with symptoms for capturing an image of the area with symptoms, capturing at least one image of the area with symptoms with the camera (207) of the mobile device (103) as instructed, and either processing the at least one captured image locally on the mobile device (103) for assessment on the mobile device (103) of the user's oral health based on the at least one captured image, or transmitting the at least one captured image or a representation thereof from the mobile device (103) to a remote server (409) for remote assessment of the user's oral health based on the at least one captured image.

10. (canceled)

11. The computer program product according to claim 9, comprising instructions for: instructing the user (101) to rotate the camera (207) to landscape position and capturing at least one image in landscape mode of the area of the user's oral cavity with the camera (207) of the mobile device (103).

12. The computer program product according to claim 9, comprising instructions for instructing the user (101) to position him-/herself such that the mirror (102) is at a given distance from the user's oral cavity.

13. The computer program product according to claim 9, comprising instructions for capturing with the camera (207) of the mobile device (103) an image of at least a portion of at least one of a tooth, a lip, or gums.

14. The computer program product according to claim 9, wherein the mobile device (103) is a smartphone or a tablet.

15. The computer program product according to claim 9, comprising instructions for displaying the image captured by the main camera (207) in real-time on the main screen (205) of the mobile device (103).

16. The computer program product according to claim 9, comprising instructions for instructing the user (101) to view the reflection of the area of the user's oral cavity displayed on the screen (205) of the mobile device (103) in the mirror (102) before/when capturing the image of the area of the user's oral cavity.

17. The computer program product according to claim 9, comprising instructions for guiding the user (101) to capture an image of the area of the user's oral cavity in a predefined view.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In the following detailed portion of the present disclosure, the aspects, embodiments, and implementations will be explained in more detail with reference to the example embodiments shown in the drawings, in which:

[0049] FIG. 1 is a diagrammatic representation according to an embodiment, showing a user, a mirror, the user's reflection in the mirror, a mobile device, and an image of an area of the user's oral cavity, which image is displayed on the main screen of the mobile device;

[0050] FIG. 2a is a front view of a mobile device according to an embodiment showing a main screen and a front camera;

[0051] FIG. 2b is a rear view of the mobile device of FIG. 2a, showing a main camera and a flashlight;

[0052] FIG. 3 is a diagrammatic representation of the user, mirror, and mobile device of FIG. 1, the reflections of the user and the mobile device in the mirror, and the obtaining of an image of an area of the user's oral cavity;

[0053] FIG. 4 is a diagrammatic representation of a wireless network according to an embodiment comprising a user's mobile device, a remote server, and a dental professional's computer;

[0054] FIG. 5 is a flowchart of an embodiment of a method for obtaining images for assessing a user's oral health; and

[0055] FIGS. 6 to 28 are graphical user interfaces (GUIs) displayed to the user on the main screen of the mobile device according to an embodiment, exemplifying the user's interaction with an application software executed on the mobile device.

DETAILED DESCRIPTION

[0056] FIG. 1 is a diagrammatic representation according to an embodiment showing a user 101, a mirror 102, the user's reflection 101′ in the mirror, a mobile device 103, and an image 104 of an area of the user's oral cavity, which image 104 is displayed on the main screen of the mobile device 103.

[0057] In an embodiment, the user 101 may wish to obtain one or more images of an area of his/her oral cavity for assessment of his/her oral health. Here, “assessment” preferably refers to an assessment performed in order to determine the user's oral health condition. For example, the assessment may comprise at least one of monitoring the user's oral health condition, identifying any oral health problems, identifying any oral health risks, or determining that no symptoms of any oral health problems or diseases are present. In an embodiment, the assessment is performed automatically or semi-automatically using artificial intelligence, e.g. by using the one or more images obtained as input to statistical algorithms and machine learning methods. In an embodiment, depending on the circumstances, the assessment is performed by a human expert, e.g. a dental professional (e.g. dentist, dental hygienist, dental therapist, dental nurse, dental technician), a medical practitioner, or another health professional. The user 101 may have observed symptoms of an oral health problem or disease and may therefore wish to obtain images of an area of his/her oral cavity to have his/her oral health assessed based on the images. Or, the user 101 may not have observed any symptoms but may, nevertheless, wish to obtain images of an area of his/her oral cavity to have his/her oral health assessed. In an embodiment, the assessment may be part of a dental health treatment or follow-up plan prescribed by a dental professional.

[0058] Herein, “oral cavity” may refer to e.g. lips, hard palate, soft palate, retromolar trigone (area behind the wisdom teeth), tongue, gingiva (gums), buccal mucosa, the floor of the mouth under the tongue, and/or teeth.

[0059] Further, herein, “oral health problem” or “oral disease” may refer to at least one of Abnormal taste in mouth, Abrasion of tooth, Acid reflux, Acute necrotizing ulcerative gingivitis, Addison's disease, Alveolitis of jaw, Amalgam tattoo, Amelogenesis imperfecta, Anemia, Aphthous ulcer of mouth, Atrophy of tongue papillae, Black hairy tongue, Bleeding gums, Broken tooth injury, Bruxism (teeth grinding), Burning mouth syndrome, Cancer, Candidiasis, Cheek biting, Cheilosis, Chemical burn (mouth), Chicken pox (Varicella), Cold sore, Complete avulsion of tooth, Contusion, Crazing of enamel, Cyst, Dental caries, Dental filling lost, Dental peri-implant mucositis, Dental plaque, Dental restoration present, Dentinogenesis imperfecta, Denture stomatitis, Diastema of teeth, Early tooth exfoliation, Electrical burn, Enamel hypoplasia, Enlarged labial frenulum, Erosion of teeth, Eruption cyst, Erythema, Erythroleukoplakia of internal part of mouth, Excessive salivation, Fibroma, Fistula, Fluorosis, Fordyce's disease, Fracture of tooth, Fractured dental restoration, Geographic tongue, Gingival recession, Gingivitis, Glossopyrosis, Hemangioma, Herpes labialis, Herpes zoster, Herpes infection, Hyperplasia of gingiva, Infectious mononucleosis, Leukemia, Leukoedema, Leukoplakia, Lichen planus (mouth), Linea alba of oral mucosa, Lip biting, Lipoma, Lymphadenopathy, Malignant melanoma, Malignant tumor of major salivary gland, Malocclusion of teeth, Measles, Melanin pigmentation (mouth), Melanocytic nevus (mouth), Melanoma in situ (mouth), Mucocele of mouth, Mumps, Necrosis of the pulp, Osteoporosis, Pain in throat, Papillary hyperplasia, Papilloma, Parulis, Pemphigoid, Pemphigus, Pericoronitis, Periodontal abscess, Periodontitis, Pseudomembranous thrush, Pulpitis, Pyogenic granuloma, Rubella, Sexually transmitted infectious disease, Sialolithiasis, Sinusitis, Smokeless tobacco keratoses, Smoker's melanosis, Staining of tooth, Stomatitis, Subluxation of tooth, Syphilis, Teething syndrome, Temporomandibular joint disorder, Thermal burn, Tongue biting, Tonsillitis, Tooth absent, Tooth sensitivity to brush or floss, Tooth sensitivity to cold, Tooth sensitivity to palpation, Traumatic granuloma, Traumatic injury, Trigeminal neuralgia, Turner's Tooth, Ulcer, Verrucous carcinoma, Vitamin deficiency, Wegener's granulomatosis, White sponge nevus of mucosa, or Xerostomia.

[0060] In an embodiment, the person whose oral cavity is captured on image is the same person as the person operating the mobile device 103 for obtaining the one or more images. In another embodiment, the person operating the mobile device 103 is not the same person as the person whose oral cavity is captured on image. This may, for example, be the case when the person whose oral cavity is captured on image is a child or other person requiring help, e.g. from a parent or other family member, for capturing the one or more images of an area of his/her oral cavity.

[0061] The mobile device is a portable computing device comprising a camera and a screen. In the embodiment of FIG. 1, the mobile device 103 is a smartphone. FIG. 2a is a front view of a mobile device 103 according to an embodiment showing a main screen 205 configured to display images to the user 101 and a front camera 206. FIG. 2b is a rear view of the mobile device 103 of FIG. 2a showing a main camera 207 and a flashlight 208. In another embodiment, the mobile device 103 may be e.g. a tablet computer (tablet).

[0062] In an embodiment, the mobile device 103 is configured to execute an application software (“app”) and comprises a main camera 207 for capturing images and a main screen 205 for e.g. displaying the images, wherein the main screen 205 and the main camera 207 are provided on opposite sides of the housing of the mobile device 103. In another embodiment, the camera for capturing images may be a secondary camera and/or the screen for displaying the images may be a secondary screen provided on opposite sides of the housing of the mobile device.

[0063] According to an embodiment, for capturing the one or more images for the oral health assessment, the user 101 is instructed by the app to position him-/herself in front of a mirror 102, as shown in FIGS. 1 and 3. FIG. 3 is a diagrammatic representation of the user 101, mirror 102, and mobile device 103 of FIG. 1, the reflections of the user 101′ and the mobile device 103′ in the mirror, and the obtaining of an image of an area of the user's oral cavity. In an embodiment, the mirror 102 is at a given distance from the user's face, preferably at a given distance from the user's oral cavity, and the app instructs the user to position him/herself accordingly. The mirror may be e.g. a wall-mounted mirror, a floor mirror, a tabletop mirror, or a handheld mirror. In the embodiment of FIG. 3, the user 101 has, upon instructions received from the app, rotated the mobile device 103 such that the main screen 205 faces the mirror 102 and the main camera 207 faces the user 101. Thus, the main screen 205, and anything displayed thereon, may be reflected in the mirror 102 and may be viewed via the mirror by the user 101.

[0064] In an embodiment, upon instruction from the app, the user 101 rotates the main camera 207 and the main screen 205 into a horizontal/landscape position for capturing an image in landscape mode, i.e. an image having more width than height, of an area of his/her oral cavity, as shown in FIG. 3.

[0065] In an embodiment, real-time images detected with the main camera 207 are displayed in real-time on the main screen 205, wherein real-time refers to the concurrent, or substantially concurrent, display of images detected by the camera 207 on the main screen 205. Thus, the user 101 may view real-time images detected with the camera 207 and displayed on the main screen 205 reflected in the mirror 102. That is, he/she may view e.g. his/her own oral cavity in real-time via the mirror 102, the main screen 205, and the main camera 207 and may use this as information input for positioning/guiding the main camera 207 in order to capture an image of an area of his/her oral cavity for oral health assessment. By being able to view images detected by the main camera 207 and represented in real-time on the main screen 205, the user may easily check and adjust e.g. the focus, lighting, level of detail, zoom factor, etc. of the images prior to capturing and saving the images on the mobile device 103 and/or transferring the captured images to a remote server 409.

[0066] The mobile device 103 is communicatively connected to a network of computer nodes, e.g. for transmitting image data and/or non-visual data from the mobile device 103 to a computer node and/or for receiving data from a computer node. FIG. 4 is a diagrammatic representation of a wireless network, such as a cellular telephone network, the internet, or a cloud-based network, comprising the mobile device 103, a remote server 409, and a dental professional's workstation 410. Image data and/or non-visual data may be transmitted from the mobile device 103 to the remote server 409 and/or the dental professional's workstation for remote assessment, processing, and/or storage of the data. The transmission of data may be synchronous, asynchronous, or a mixture of synchronous and asynchronous data transmission.

[0067] The method described herein for obtaining images for assessing oral health may at least partly be implemented in the form of an application software (“app”) which may be executed on a mobile device 103. The user 101 may interact with the application software on the mobile device 103, as described in the following with reference to the flowchart of FIG. 5 and to FIGS. 6 to 28 which are examples of graphical user interfaces (GUIs) displayed to the user 101 on the main screen 205 of the mobile device 103 during the user's interaction with the application software.

[0068] Initially, at step 511, the user 101 launches the app on the mobile device 103. The app is configured to perform a dialogue with the user 101, which dialogue comprises prompting the user 101 to indicate, for example, whether he/she wishes to continue an existing assessment, to view the results of a previous assessment, or to add a new assessment, as shown in FIG. 6. In another embodiment, the app is configured to require a login for use. The user 101 indicates “add new assessment” (FIG. 6) and may be prompted to indicate his oral health problem, e.g. toothache (FIG. 7). The app presents a list of selectable persons and/or may allow persons to be added, allowing the user 101 to select or identify the person whose oral cavity is to be captured on image, i.e. the person whom the oral health assessment is for (FIG. 8).

[0069] At steps 512 and 513, as part of the dialogue with the user, the app poses questions to the user 101 relating to the indicated oral health problem (if an oral health problem has been indicated), e.g. questions relating to pain, history of occurrence of symptoms, symptom triggers, and/or urgency, and receives the user's answers to the questions, which answers are recorded and processed locally on the mobile device and/or transmitted to a remote server 409 for processing. The questions are posed to the user 101 in the form of e.g. free text on the main screen 205, spoken text through a loudspeaker of the mobile device 103, a checklist on the main screen 205, a slider bar on the main screen 205, or a visual representation on the main screen 205. In an embodiment, a 3D representation of an oral cavity is presented to the user 101 for the user 101 to indicate in the representation an area corresponding to the area of the user's own oral cavity associated with the oral health problem. The area may e.g. be a specific tooth. The user's answers may be finite answers. For example, the user 101 may select one or more suggestions from a checklist (FIG. 9).

[0070] At step 512, the user 101 is prompted, for example, to localize a region/area associated with the oral health problem (FIG. 9), to describe symptoms (FIG. 10), to categorize the pain level (FIG. 11), to describe the pain (FIG. 12), the occurrence of the oral health problem (FIG. 13), symptom triggers (FIG. 14), the urgency (FIG. 15), to indicate whether he/she would like to be contacted by a service provider (e.g. application software hotline, insurance company, dental professional) (FIG. 16), and to indicate whether he/she has experienced any other health issues or symptoms lately (FIG. 17) (apart from the indicated oral health problem (FIG. 7)).

[0071] In an embodiment, the questions are preferably structured so as to lead to a result based on which an area of the oral cavity with symptoms can be determined with a finite number of questions and answers.

[0072] At step 514, the app prompts the user 101 to capture one or more images of an area of the user's oral cavity (FIG. 18), e.g. of an area with symptoms/an area associated with the oral health problem, which symptoms may, for example, be symptoms of at least one of gingivitis, periodontitis, dental caries, abrasion of tooth, bruxism, cold sore, erosion of teeth, fluorosis, herpes labialis, herpes zoster, or herpes infection. In an embodiment, based on the user's answers in step 512, an area of the user's oral cavity with symptoms is determined either by the software running on a processor of the mobile device 103 or remotely on a remote server, and the user is instructed by the app to capture an image of the determined area with symptoms.

[0073] The user 101 may capture the images with the main camera 207 of the mobile device 103 and/or may add existing images from a gallery available e.g. on or via the mobile device 103 (FIG. 23). For capturing the images (or for adding existing images from a gallery), the user 101 is prompted to select a particular view/pose annotation (FIG. 19) describing the view/pose of the images, e.g. closed mouth view, bite view, bottom/lower arch view, upper arch view, bottom/lower lip view, upper lip view, closed bite anterior view, open bite anterior view, closed bite buccal view, open bite buccal view, roof of mouth view, floor of mouth view, side of mouth view, or frontal view, and is guided to capture an image in the selected particular view/pose (FIGS. 20, 21, 22). The captured images are annotated according to the selected view/pose (FIG. 27).

[0074] At steps 515 and 516, the user 101 is instructed to position him-/herself in front of a mirror 102 for capturing the one or more images of an area of his/her oral cavity and to rotate/flip the mobile device 103 such that the main screen 205 faces the mirror 102 and the main camera 207 faces the user 101 when capturing the images, with the main camera (207) being directed towards the area of the user's oral cavity. At step 517, the main camera 207 of the mobile device 103 captures one or more images of an area of the user's oral cavity (FIGS. 24, 25), e.g. of a portion of at least one of a tooth, a lip, or gums. For this, the app may be configured to control the main camera 207 of the mobile device 103. In an embodiment, prior to capturing the one or more images (step 517), the user 101 is instructed by the app to rotate the main camera (207). For example, the user may be instructed by the app to rotate the main camera (207) to landscape/horizontal position and to capture the one or more images of an area of his/her oral cavity in landscape mode.

[0075] At step 518, the captured one or more image are saved on the mobile device 103 and processed locally on the mobile device 103 for assessment of the user's oral health based on the images. The images may be processed manually, automatically, or semi-automatically. In an embodiment, the processing and assessment is performed using artificial intelligence. Alternatively, or additionally, the captured images or representations thereof are transmitted (FIG. 26) to a remote server 409 for remote assessment of the user's oral health based on the captured images. The remote assessment is in an embodiment performed by a human expert, such as e.g. a dentist, and/or by software running on the remote server 409. The images may be used as input for statistical algorithms, machine learning, and/or deep learning technologies in order to assess the user's oral health. In an embodiment the software running on the remote server 409 uses artificial intelligence to assess the user's oral health based on the one or more images.

[0076] The assessment, whether performed locally or remotely, may comprise a referral to a dental professional, an estimate of the urgency of the oral health problem, a recommendation for self-treatment, etc. Particularly, the oral health assessment, performed locally and/or remotely, may comprise determining whether the user 101 is a candidate for an oral treatment, e.g. dental treatment or treatment of cold sores on the lips, based on the images and informing the user 101 on the mobile device 103, preferably via the app, whether he/she is a candidate for an oral treatment.

[0077] In an embodiment, as part of the processing of the one or more captured images, locally on the mobile device 103 or remotely on a remote server 409, the image quality of the one or more images or of a representation thereof is checked, preferably automatically. Particularly, it is checked whether the one or more images are of sufficient quality to allow for an assessment of the user's oral health based on the images, preferably whether the images have clinical image quality. The image quality check may comprise an assessment of e.g. focus, sharpness, lighting, resolution, etc. of the images. The user 101 is instructed by the app to capture another image of his/her oral cavity, if the image quality is below a threshold for image quality, i.e. if the images do not have clinical image quality.

[0078] Finally, at step 519, the user 101 receives a preliminary or final report comprising a preliminary assessment or final assessment of the user's oral health based on the one or more images of an area of the user's oral cavity.

[0079] The various aspects and implementations have been described in conjunction with various embodiments herein. However, other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed subject-matter, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word “comprising” does not exclude other elements or steps, and the indefinite article “a” or “an” does not exclude a plurality. A single processor or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measured cannot be used to advantage. A computer program may be stored/distributed on a suitable medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0080] The reference signs used in the claims shall not be construed as limiting the scope.