REFLECTION REMOVAL FROM AN IMAGE

Abstract

The technology of this application relates to a method for removing reflections from an image. The method detects one or more reflection areas in the image, wherein each reflection area includes a reflection. Further, the method extracts the one or more reflection areas from the image, and removes the reflection from each of the extracted reflection areas.

Claims

1. A method for removing reflections from an image, the method comprising: detecting one or more reflection areas in the image, wherein each reflection area, from the one or more reflection areas, includes a reflection; extracting the one or more reflection areas from the image; and removing the reflection from each of the extracted reflection areas.

2. The method of claim 1, further comprising: after removing the reflection from each of the extracted reflection areas, reinserting the extracted reflection areas without the reflection into the image to replace, respectively, the reflection areas with the reflection.

3. The method of claim 1, further comprising: detecting the one or more reflection areas using a first trained model; and/or removing the reflection from each of the extracted reflection areas using a second trained model.

4. The method of claim 3, wherein the first trained model includes a first convolutional neural network (CNN).

5. The method of claim 4, wherein the first CNN includes a semantic segmentation CNN configured to perform a semantic segmentation of the image.

6. The method of claim 5, wherein the one or more reflection areas are detected using a semantic mask.

7. The method of claim 3, wherein the second trained model includes a generative adversarial network (GAN).

8. The method of claim 7, wherein the GAN includes a conditional GAN.

9. The method of claim 3, wherein the second trained model comprises includes a second convolutional neural network.

10. The method of claim 1, wherein the image is a photograph of a person wearing eyeglasses, and detecting the one or more reflection areas comprises: detecting a face of the person in the image; detecting the eyeglasses in the image based on the detected face; and detecting the one or more reflection areas located within the eyeglasses detected in the image.

11. The method of claim 5, wherein eyeglasses are detected in the image using the semantic segmentation CNN.

12. The method of claim 11, further comprising: segmenting the eyeglasses detected in the image using the semantic segmentation CNN, into eyeglass segments; extracting the eyeglass segments from the image; detecting the one or more reflection areas located within the eyeglasses by removing the eyeglass segments without reflection from the extracted eyeglasses segments; and removing the reflection from each of the extracted eyeglasses segments with reflection.

13. A device configured to remove reflections from an image, the device comprising: a processor; and a memory configured to store computer readable instructions that, when executed by the processor, cause the device to: detect one or more reflection areas in the image, wherein each reflection area, from the one or more reflection areas, includes at least one reflection, extract the one or more reflection areas from the image, and remove the at least one reflection from each of the extracted reflection areas.

14. A computer program comprising program code for performing the method according to claim 1.

15. A non-transitory computer readable storage medium configured to store computer readable instructions that, when executed by a processor, cause the processor to provide execution comprising: detecting one or more reflection areas in the image; extracting the one or more reflection areas from the image; and removing the reflection from each of the extracted reflection areas.

16. The non-transitory computer readable storage medium of claim 15, wherein the processor is further caused to provide execution comprising: reinserting the extracted reflection areas without the reflection into the image.

17. The non-transitory computer readable storage medium of claim 15, wherein the processor is further caused to provide execution comprising: detecting the one or more reflection areas using a first trained model; and/or removing the reflection from each of the extracted reflection areas using a second trained model.

18. The non-transitory computer readable storage medium of claim 17, wherein the first trained model includes a first convolutional neural network (CNN).

19. The non-transitory computer readable storage medium of claim 18, wherein the first CNN includes a semantic segmentation CNN configured to perform a semantic segmentation of the image.

20. The non-transitory computer readable storage medium of claim 19, wherein the one or more reflection areas are detected using a semantic mask.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0034] The above described aspects and implementation forms of this disclosure will be explained in the following description of specific embodiments in relation to the enclosed drawings, in which:

[0035] FIG. 1 shows an example schematic representation of a conventional pipeline for reflection removal from an image;

[0036] FIG. 2 shows an example schematic representation of a method for reflection removal from an image according to an embodiment;

[0037] FIG. 3 shows an example schematic representation of a pipeline for reflection removal from an image according to an embodiment;

[0038] FIG. 4 shows an example schematic representation of a pipeline for reflection removal from an image according to an embodiment;

[0039] FIG. 5 shows an example schematic representation of a pipeline for reflection removal from an image according to an embodiment; and

[0040] FIG. 6 shows an example device for reflection removal from an image according to an embodiment.

DETAILED DESCRIPTION OF EMBODIMENTS

[0041] FIG. 2 shows a schematic representation of a method 200 for reflection removal from an image, according to an embodiment. The method 200 may be performed by a device 600 (see e.g. FIG. 6), for instance by a processor of an electronic consumer device like a smartphone, tablet or digital camera. The method 200 may be performed by a device 600 that is also capable of taking the image 500.

[0042] The method 200 comprises a step 201 of detecting one or more reflection areas in the image 500. Each reflection area includes at least one reflection. That is, the method 200 may detect specifically where the reflections are located in the image 500, and may determine accordingly image areas comprising or consisting of these reflections.

[0043] Further, the method 200 comprises a step 202 of extracting the one or more reflection areas from the image 500. For instance, the one or more detected reflection areas may be cropped from the image 500, or may be copied from the image 500. Further processing does then not have to be applied to the whole image 500, but may be applied only to the extracted reflection areas. Thus, the method 200, particularly the reflection removal, may work faster.

[0044] The method 200 further comprises a step 203 of removing the reflection from each of the extracted reflection areas. To this end, a conventional reflection removal algorithm may be used. Also, a trained model, trained for this purpose, may be used. This trained model may be specifically trained to operate on reflection areas, i.e., specific image segments, and not on entire images.

[0045] Moreover, the method 200 can also comprise a reinsertion step. In particular, after the removal 202 of the reflections from each of the extracted reflection areas, the method 200 may comprise a step of reinserting the extracted reflection areas without the reflection into the image 500, in order to replace, respectively, the reflection areas with the reflection. Thus, a processed image without reflections, or at least with a significantly reduced amount of reflections, may be obtained.

[0046] FIG. 3 shows a schematic representation of a pipeline for reflection removal from an image 500, which may be used for performing the method 200 according to an embodiment. The pipeline may be part of a device 600 that performs the method 200. The pipeline may comprise a reflection detection module 302 and a reflection removal module 304.

[0047] In the exemplary embodiment shown in FIG. 2, the reflection removal from an input image 500 (see also FIG. 5) may consist of the two modules 302 and 304. In the first reflection detection module 302, the areas of the input image 500 with reflections (i.e., the reflection areas) are detected. That is, the reflection detection module 302 can implement the step 201 of the method 200. Afterwards, the detected areas with reflection are extracted or cropped from the image 500 (i.e. the method step 202 is performed, e.g., also by the reflection detection module 302), and the extracted image segments or crops 303 with reflections are provided to the reflection removal module 304. That is, the method step 203 can be performed by the reflection removal module 304. The second reflection removal module 304 may comprise a CNN, which can be applied on the image segments or crops 303, in order to remove the reflections therein. The results, i.e. the reflection areas with reflections removed, may then be pasted back into the original image 500, in order to obtain a processed image 500 with removed reflections.

[0048] In an embodiment, a first trained model may be used to detect 201 the one or more reflection areas, i.e., it may be used in the reflection detection module 302. Further, a second trained model may be used to remove 202 the reflections from each of the extracted reflection areas, i.e., it may be used in the reflection removal module 304. The first trained model can comprise a first CNN. As described above, also the second trained module may comprise a (second) CNN. Moreover, the first CNN can comprise a semantic segmentation CNN, which is configured to perform a semantic segmentation of the image 500.

[0049] In particular, the detection step 201 can be performed by a semantic segmentation CNN, which detects the reflection areas. For example, a semantic mask may be used to find connected components (identified as reflections). Then, their minimum circumscribing rectangle can be determined to determine the reflection areas. Multiple such rectangles may define multiple reflection areas. The reflection areas may further be used to crop (as example) the image 500 as described above. Afterwards, the reflection removal step 202 can be applied to each crop (i.e., to each reflection area), and the result (i.e., the reflection area with reflections removed) can be pasted back to the image 500 as described above.

[0050] The embodiments of this disclosure provide several advantages: first, an improvement of the execution time for the reflection removal from the image 500 is achieved, due to the fact that the reflection removal is applied only to reflection areas. Second, the removal of undesirable artefacts by processing only crops or extractions (reflection areas) in the reflection removal step (e.g., implemented by a CNN), prevents areas without reflections from being changed. Third, a de-reflection quality may be improved by performing the reflection removal only in the reflection areas, and not requiring the reflection removal step 202 to detect reflections in the entire image 500.

[0051] Moreover, in order to train the first and/or second trainable (or trained) model (e.g., CNN or GAN), the method 200 can further take an initial image without a reflection from a pool of images without reflections, and can generate a synthetic reflection on the initial image with a reflection generator to obtain a training image. Then, the method 200 may process the training image with a generator of the first and/or second trainable model to obtain a synthetic image. Finally, the method 200 may calculate a loss function of the first and/or second trainable model on the basis of the initial image and the synthetic image.

[0052] FIG. 4 shows schematic representation of a pipeline for reflection removal from an image 500 that may be used for performing the method 200 according to an embodiment. The pipeline builds on the pipeline shown in FIG. 3. The pipeline of FIG. 4 may again be part of a device 600 that performs the method 200. The pipeline of FIG. 4 may again comprise the reflection detection module 302 and the reflection removal module 304.

[0053] In the embodiment of FIG. 4, analogously to the embodiment of FIG. 3, an input image 500 comprising reflections is given as input to module 302. The module 302 is configured to detect 201 the areas comprising reflection. In this embodiment, the reflection detection 201 is performed by means of a CNN. Moreover, the output of the module 302 comprises crops with reflections (extracted reflection areas), which are given as input to the module 304. The module 304 is configured to perform a reflection removal 202 from the extracted reflection areas. This can be done either by means of a CNN or by means of a GAN. In an embodiment, the GAN can be a conditional GAN.

[0054] FIG. 5 shows a schematic representation of another pipeline for removing reflections from an image 500 that may be used to perform the method 200 according to an embodiment. The pipeline may be part of a device that performs the method 200.

[0055] For the pipeline shown in FIG. 5, the image 500 is exemplarily a photograph of a person wearing eyeglasses. The pipeline includes a first stage 501 for detecting a face of the person in the image 500. Further, the pipeline may include a second stage 502 for detecting the eyeglasses in the image 500 based on the detected face. For instance, the eyeglasses can be detected in the image 500 using a segmentation CNN. A third stage 503 of the pipeline may then detect the one or more reflection areas located within the eyeglasses detected in the image 500.

[0056] Furthermore, the pipeline of FIG. 5 can perform the following steps. The second stage 502 may further segment the eyeglasses detected in the image 500, for instance, by using the segmentation CNN. The third stage 503 may further extract or crop the obtained eyeglass segments from the image 500, and may detect 201 the one or more reflection areas located within the eyeglasses by removing eyeglasses segments without reflection from the extracted eyeglasses segments. Further, a fourth stage 504 may remove the reflection from each of the extracted eyeglasses segments with reflection and may paste back 504 the areas with removed reflections to obtain the processed image 500.

[0057] Accordingly, in this embodiment, the reflections are removed only from eyeglasses area, not from any other part of image 500. That is, the first stage 501 may detect the eyeglasses, for example, using face detection and face parsing tool. Then, a CNN can be applied to the eyeglasses crops in the fourth stage 504 and the result can be pasted back.

[0058] FIG. 6 shows a device 600 according to an embodiment. The device 600 may be configured to perform the method 200, including the steps 201, 202 and 203, in order to remove reflections from an image 500. The output may be the processed image 500 with less reflections or no reflections. The device 600 may comprise a camera to obtain the image 500.

[0059] Further, the device 600 may comprise a processor or processing circuitry (not shown) configured to perform, conduct or initiate the various steps of the method 200 described herein. The processing circuitry may comprise hardware and/or the processing circuitry may be controlled by software. The hardware may comprise analog circuitry or digital circuitry, or both analog and digital circuitry. The digital circuitry may comprise components such as application-specific integrated circuits (ASICs), field-programmable arrays (FPGAs), digital signal processors (DSPs), or multi-purpose processors.

[0060] The device 600 may further comprise memory circuitry, which stores one or more instruction(s) that can be executed by the processor or by the processing circuitry, in particular under control of the software. For instance, the memory circuitry may comprise a non-transitory storage medium storing executable software code which, when executed by the processor or the processing circuitry, causes the method 200 to be performed.

[0061] In one embodiment, the processing circuitry comprises one or more processors and a non-transitory memory connected to the one or more processors. The non-transitory memory may carry executable program code which, when executed by the one or more processors, causes the device 600 to perform, conduct or initiate the method 200 described herein.

[0062] In particular, the processor or processing circuitry of the device 600 is configured to detect 201 one or more reflection areas in the image 500, wherein each reflection area includes a reflection. Further, it is configured to extract 202 the one or more reflection areas from the image 500, and to remove 203 the reflection from each of the extracted reflection areas.

[0063] In summary, compared to a conventional one-stage approach (see, e.g., FIG. 1), embodiments of the present disclosure perform a two-stage reflection removal procedure (see FIGS. 3-6). First, the areas with the reflections are detected 201, and, second, the reflection removal 202 is applied only to found reflection areas that have been extracted 203 from the image 500.

[0064] The present disclosure has been described in conjunction with various embodiments as examples as well as implementations. However, other variations can be understood and effected by those persons skilled in the art and practicing the claimed matter, from the studies of the drawings, this disclosure and the independent claims. In the claims as well as in the description the word comprising does not exclude other elements or steps and the indefinite article a or an does not exclude a plurality. A single element or other unit may fulfil the functions of several entities or items recited in the claims. The mere fact that certain measures are recited in the mutual different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.