DEVICE AND METHOD FOR SKIN GLOSS DETECTION

Abstract

The present invention relates to a device and method for skin gloss detection. To enable skin gloss detection in a simple and efficient manner, preferably with an available user device, the device comprises an illumination unit (2) for illuminating a skin area at a flashing rate, an imaging unit (3) for acquiring images of the skin area at an imaging rate which is different than said flashing rate, and a processing unit (4) for processing acquired images and detecting the amount of gloss in the skin area from at least one partial image of the skin area while being illuminated by said illumination unit and at least one partial image of the skin area while not being illuminated by said illumination unit.

Claims

1. Device for skin gloss detection, said device comprising: an illumination unit for illuminating a skin area at a flashing rate, an imaging unit for acquiring images of the skin area at an imaging rate, which is different than said flashing rate, wherein images of the skin area while the illumination by said illumination unit is on and images of the skin area while the illumination by said illumination unit is off are acquired, and a processing unit for processing acquired images by detecting the amount of gloss in the skin area from a specular reflection component and a diffuse component in at least one first partial image of the skin area representing a portion of an image acquired by the imaging unit while the skin area is illuminated by said illumination unit and at least one partial image of the skin area representing a portion of an image acquired by the imaging unit while the skin area is not being illuminated by said illumination unit.

2. Device as claimed in claim 1, wherein said imaging unit is configured to acquire images of the illuminated skin area at an imaging rate which is higher than said flashing rate.

3. Device as claimed in claim 1, wherein said illumination unit is configured to illuminate the skin area at a flashing rate which is a multiple higher than said imaging rate.

4. Device as claimed in claim 3, wherein said illumination unit is configured to illuminate the skin area at a flashing rate, which is a non-integer multiple higher than two times said imaging rate.

5. Device as claimed in claim 1, further comprising a control unit for controlling the flashing rate and/or the imaging rate.

6. Device as claimed in claim 5, wherein said control unit is configured to generate configurable temporal light patterns with modifiable duty cycles.

7. Device as claimed in claim 1, wherein said processing unit is configured to determine the specular reflection component in one or more sub-areas of said skin area and to determine the amount of skin gloss in said sub-areas by determining the ratio of specular reflection component in said one or more sub-areas to a diffuse component in said one or more sub-areas.

8. Device as claimed in claim 1, wherein said processing unit is configured to determine skin sub-areas of the illuminated skin area which are substantially perpendicular to the optical axis of the imaging unit and to determine the amount of skin gloss from the determined skin sub-areas.

9. Device as claimed in claim 8, wherein said processing unit is configured to determine said skin sub-areas by use of a deformable model evaluating skin landmarks and/or pose estimation or by detecting flash components and diffuse components.

10. Device as claimed in claim 1, wherein said processing unit is configured to compensate one or more images of the skin area while not being illuminated by said illumination unit with a contribution of the illumination to a diffuse component estimation per color channel histogram shifting.

11. Device as claimed in claim 1, wherein said processing unit is configured to determine a numerical skin gloss value per pixel, in particular by dividing a specular component value per pixel by a diffuse component value per pixel or by dividing the sum of a specular component value per pixel and a diffuse component value per pixel by the diffuse component value per pixel.

12. Device as claimed in claim 1, wherein said processing unit is configured to determine a skin gloss map of the skin area indicating the amount of skin gloss per pixel or per group of pixels.

13. Device as claimed in claim 1, wherein said device is a mobile user device, in particular a smartphone, camera, laptop, or tablet.

14. Method for skin gloss detection, said method comprising: illuminating a skin area at a flashing rate, acquiring images of the skin area at an imaging rate which is different than said flashing rate, wherein images of the skin area while the illumination is on and images of the skin area while the illumination is off are acquired, and processing acquired images by detecting the amount of gloss in the skin area from a specular reflection component and a diffuse component in at least one first partial image of the skin area representing a portion of an image acquired while the skin area is illuminated and at least one second partial image of the skin area representing a portion of an image acquired while the skin are is not illuminated.

15. Computer program comprising program code means for causing a device as claimed in claim 1 to carry out the steps of the method when said computer program is carried out on the device.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0027] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiment(s) described hereinafter. In the following drawings

[0028] FIG. 1 shows a schematic diagram of an embodiment of a device according to the present invention,

[0029] FIG. 2 shows a diagram of the average luminance value per image frame F and two adjacent flash ON and flash OFF images,

[0030] FIG. 3 shows two example images with low gloss skin and high gloss skin,

[0031] FIG. 4 shows various examples of surface light reflection,

[0032] FIG. 5 shows face images to illustrate automatic face area tracking based on face landmark tracking,

[0033] FIG. 6 shows two example forehead regions with their color histograms,

[0034] FIG. 7 shows various difference images of the forehead region,

[0035] FIG. 8 shows an example binary mask indicating valid gloss values in the forehead region,

[0036] FIG. 9 shows an example output gloss map,

[0037] FIG. 10 shows three example frames while switching the flash at a flashing rate of 4.25 times the imaging rate, and

[0038] FIG. 11 shows a flow chart of a method for skin gloss detection according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0039] FIG. 1 shows a schematic diagram of an embodiment of a device 1 for skin gloss detection according to the present invention. The device may be a mobile user device, for instance a smartphone, camera, laptop, or tablet, which is available to many users for everyday use and which is adapted for the desired purpose of detecting skin gloss, e.g. by use of a software application (app) that makes use of existing hardware components and evaluates data that are obtained by existing hardware components. Alternatively, the device may also be a dedicated device made particularly for the purpose of skin gloss detection (and optionally other purposes). In the example shown here and discussed in the following, the device 1 may be implemented as smartphone.

[0040] The device 1 comprises an illumination unit 2 for illuminating a skin area at a flashing rate. The illumination unit 2 may e.g. be the flash of a user device. In the following, the illumination unit is often simply referred to as flash, which shall, however, be generally understood as illumination unit.

[0041] The device 1 further comprises an imaging unit 3 for acquiring images of the skin area at an imaging rate which is different than said flashing rate. The imaging unit 3 may e.g. be the camera of a user device. In the following, the imaging unit is often simply referred to as camera, which shall, however, be generally understood as imaging unit.

[0042] The device 1 further comprises a processing unit 4 for processing acquired images and detecting the amount of gloss in the skin area from at least one partial (or complete) image of the skin area while being illuminated by said illumination unit 2 and at least one partial (or complete) image of the skin area while not being illuminated by said illumination unit 2. The processing unit 4 may e.g. be the processor of a user device. Generally, the processing unit 4 may also be an external processor, e.g. of an external computer, to which the acquired images are transmitted for processing to obtain information on the skin gloss. It is, however, preferred that the processing unit forms an integrated device with the illumination unit 2 and the imaging unit 3.

[0043] The device 1 may comprise further optional elements, such as a control unit 5 for controlling the illumination unit 2 and/or the imaging unit 3, in particular to control the flashing rate and/or the imaging rate. The task of the control unit 5 may, however, also be performed by the processor 4, or the flashing rate and/or the imaging rate may be predetermined and fixed so that an active control unit 5 may not be required.

[0044] The device 1 may further comprise a user interface 6, e.g. a display, keypad, touchscreen, etc., allowing the user to enter information, e.g. to start and stop skin gloss detection, change settings, enter personal information, etc., and enabling output of information, e.g. the detected skin gloss information or user instructions.

[0045] According to the present invention the illumination unit 2 is controlled in such a way that it is exactly known which image line is exposed with flash (i.e. illumination) on or off. Additionally, face feature tracking (or other means) may be used to accurately track the region, shape and pose of the face, in which the skin gloss of a skin area (e.g. of the forehead, the cheeks, the chin, etc.) shall be determined. Additionally, averaging (e.g. horizontal averaging) of the image content may be performed resulting in a strong 1D signal indicating the exact switching times per image line even more accurately.

[0046] In the following various embodiments will be explained in more detail. A first main embodiment extracts skin gloss values from alternating images. A second main embodiment creates and combines alternating illuminations per image line.

[0047] In the first main embodiment, for calculation of the amount of gloss (using a zero order approximation) the skin sub-areas, which are perpendicular to the imaging unit's optical axis (since at a smartphone the flash is mounted very close to the camera). These perpendicular sub-areas may e.g. be determined in one of the following two combined ways.

[0048] One way is model based. It may particularly use accurate face landmark detection and 3D face pose estimation. The fit of the 3D deformable model indicates the skin-sub-areas which are perpendicular substantially perpendicular to the imaging unit's optical axis. There are many different methods known for face landmark detection and 3D face pose estimation (e.g. as currently be found at http://blog.mashape.com/list-of-10-face-detection-recognition-apis/ or https://facedetection.com/software/). Further methods are listed by Tim Cootes at http://personalpages.manchester.ac.uk/staff/timothy.f.cootes/tfc_publications.html. These include a computer algorithm to determine the visual location of the generic face features in the image and next fit a deformable 3D shape model to these generic face features in order to robustly estimate pose and surface angles. According to an embodiment spatial image locations are derived from the detected landmarks and the face pose, which are perpendicular to the camera. The 3D shape model registered to the generic face features provides an estimation of the skin normal vectors for each part of the skin.

[0049] According to another way the areas where the flash component is much larger than the diffuse component are used. The flash component is the illumination raise everywhere in the scene due to the flash (as later indicated by S+D). The specular component (S) of the flash illumination is much higher and only present at the location where the surface is close to perpendicular. On top of this, it is assumed that the ambient light contributes mostly to the baseline of D. The value of the diffuse component is either taken from the median of the delta in luminance in the face, since the majority of the face is not perpendicular to the optical axis of the camera. This works even more accurately by taking the areas from the above mentioned first way to determine these non-perpendicular areas.

[0050] FIG. 2 shows the average luminance value L (on the vertical axis) per image frame F (on the horizontal axis). Straight forward crossing detection with running means may be applied to find two adjacent flash ON and flash OFF images 10, 11.

[0051] FIG. 3 shows two example images 12, 13 of a sequence of images with low gloss skin (image 12) and high gloss skin (image 13). Each image is the difference image of two adjacent flash ON and flash OFF images.

[0052] FIG. 4 shows various examples of surface light reflection.

[0053] In short, the ratio of the specular component and the diffuse component value may be used as an indicator for the amount of gloss.

[0054] More refinement is achieved in another embodiment by compensating a non-flash image (e.g. image 11) with the contribution of the flash to the diffuse estimation per color channel histogram shifting. This histogram matching is preferably only applied in the tracked face area, derived from the face landmarks as shown in FIG. 5 depicting example images 14, 15 for illustrating automatic face area tracking based on face landmark tracking. In both images 14, 15 face landmarks are indicated by numbers. In image 15 the forehead region 16 and the cheek region 17 are indicated. Alternatively, the forehead area may include the area between the eyebrows (Tzone).

[0055] FIG. 6 shows two example forehead areas 20, 30 with their red 21, 31, green 22, 32 and blue 23, 33 color histograms. The forehead area 20 is obtained in an image without flash, the forehead area 30 is obtained in an image with flash. The histograms 31, 32, 33 are on average shifted up (right), plus an additional peak for the specular reflection components in the green and blue channels compared to the histograms 21, 22, 23. Histogram matching may thus be used to determine the flash contribution S+D more accurately per color channel. The flash does not simply contribute to the luminance equally across the light spectrum. In short, the superficial skin reflection is higher for green and blue. Red penetrates deeper into the skin.

[0056] FIG. 7 shows various difference images 40, 41. Image 40 represents the difference between the registered flashed and non-flashed images. After first pore precise alignment and warping of the image patches. This holds the specular reflection component plus the diffuse reflection component. Image 41 represents the difference between the histogram matched non-flashed image and the flashed image resulting only in the specular reflection component.

[0057] The face landmarks already give a robust registration/alignment of the face region per consecutive image frame. Additional accuracy may optionally be achieved by dense registration of image feature points inside this area.

[0058] The gloss estimation is preferably only computed for the skin sub-areas which are close to perpendicular to the camera axis and which show a significant specular component (e.g. having a value above a predetermined threshold). As mentioned, the perpendicularity estimation for each pixel in the image can be coarsely determined by using robust 3D face model based face pose estimation. This combined sub-area can e.g. be illustrated by a binary mask as illustrated as an example in FIG. 8.

[0059] The numerical gloss value may be computed per pixel by dividing the specular component value by the diffuse component value and can be assigned to the skin area where the binary mask is valid. Alternatively, the ratio between the specular+diffuse component divided by the diffuse component may also be used as a coarse indicator for the amount of gloss.

[0060] In another embodiment, multiple camera view pairs from a video sequence (angled views on the face) are combined (e.g. by taking the maximum measured gloss) into a single gloss map, which is then assigned to the image with the most frontal face present in the sequence of view pairs. This can yield a complete gloss map for each area of the face or the whole face. An example output gloss map 50 (with heat map color coding, i.e. areas 51 for low values, areas 52 for medium values and areas 53 for high values) for a forehead that was partly greased with oil to illustrate the difference between low gloss (left part of the image) and medium/high gloss (right part of the image).

[0061] Further, in an embodiment the exposure time is automatically fixed by setting auto exposure during the first exposure while holding the flash ON and then locking the exposure. This results in a recording which is hardly ever clipped by over-exposure.

[0062] In the second main embodiment, the frame rate of the flash (i.e. the flashing rate) is a non-integer times higher than the frame rate of the camera (i.e. the imaging rate). This results in exposed images where the duty cycle of the flash is clearly visible in each single image. By deliberately choosing a flashing rate not starting in sync with each frame exposure, the phase of the illuminated image lines is shifting for every image frame. Although this scheme is more complex, it has the benefit of being able to run the flash at a modulation rate which is not perceivable by the human eye, which may be a requirement for the application of gloss measurement by a smartphone.

[0063] The timing of the flash could be exactly controlled in order to know which lines are exposed with flash on and which ones with flash off. For some existing base hardware (e.g. certain smartphones or cameras) the flash cannot be easily controlled in this much detail. When this is the case, as illustrated in this embodiment, the image line, at which the switching of the flash happens, can be extracted from the image robustly, e.g. by adding up the rows of the image and detecting the switching from this 1D modulation signal. FIG. 10 shows a few examples of this effect.

[0064] FIG. 10 particularly shows three example frames 60, 61, 62, which are acquired while switching the camera flash at 4.25 times the camera frame rate (i.e. the imaging rate), while recording a white piece of paper, in order to illustrate how each image line is differently illuminated. The overlaid line 63 indicates the average illumination value for each image line, indicating the duty cycle and phase of the flash.

[0065] The subsequent processing is rather similar to the processing explained above for the first main embodiment, while taking care of assigning each set of image lines to either flash ON or flash OFF state.

[0066] FIG. 11 shows a flow chart of a method for skin gloss detection according to the present invention. In a first step S10 a skin area is illuminates at a flashing rate. In a second step S11 images of the skin area are acquired at an imaging rate which is different than said flashing rate. In a third step acquired images processed and the amount of gloss in the skin area is detected from at least one partial (or complete) image of the skin area while being illuminated by said illumination unit and at least one partial (or complete) image of the skin area while not being illuminated by said illumination unit. In summary, the present invention exploits the rolling shutter effect and adds some processing to measure skin gloss. Gloss can be detected at low flashing rates using complete flashed and non-flashed images. This can be achieved even when the image is teared by the flash going much faster with a flashing rate much higher than the imaging rate. With the proposed solution it is not a problem that the flash switches multiple times during the exposure of a frame.

[0067] In one embodiment a smartphone is set to record at a high frame rate (imaging rate) like e.g. 120 frames per second (fps), while (synchronously) switching the flashlight on and off at 60 fps. The recorded images now alternately consist of an ambient illuminated scene (consisting e.g. of a face) and a scene which is ambient illuminated plus the flash illumination. Using accurate face feature and pose tracking it has been shown to isolate and measure the specular reflection component with respect to the ambient component, which is a strong indication of the amount of glossiness of the skin. Additionally, in a second embodiment the camera runs a lower framerate while the flash operates at specific non-integer higher multiple of this frame rate, resulting, due to the rolling shutter effect, in different illuminations per set of image lines. Computer vision technology may be used to align the face areas despite the rolling shutter effect and measure different exposures for the same area resulting in similar gloss measurements.

[0068] While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive; the invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0069] In the claims, the word comprising does not exclude other elements or steps, and the indefinite article a or an does not exclude a plurality. A single element or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0070] A computer program may be stored/distributed on a suitable non-transitory medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0071] Any reference signs in the claims should not be construed as limiting the scope.