METHOD, DEVICE AND SYSTEM FOR A DEGREE OF BLURRING TO BE APPLIED TO IMAGE DATA IN A PRIVACY AREA OF AN IMAGE

Abstract

Methods and devices for protecting personal privacy in captured image data by controlling privacy masking of an image, where the degree of blurring to be applied to a privacy area of the image depends on a threshold distance from the image capturing device, and the spatial resolution of content of the scene at this distance in the image.

Claims

1. A computer-implemented method for determining a degree of blurring to be applied to image data in a privacy area of an image, the image depicting a scene and being captured by an image capturing device, the method comprising the steps of: receiving input relating to coordinates of the privacy area of the image; receiving input relating to a threshold spatial resolution of image data corresponding to content of the scene located beyond a threshold distance from the image capturing device in the scene, said image data being within the privacy area of the image; calculating a maximum spatial resolution of the image data of the image corresponding to content of the scene located on or beyond the threshold distance from the image capturing device; calculating a difference between the maximum spatial resolution and the threshold spatial resolution; determining a degree of blurring based on the calculated difference, such that for a first difference being larger than a second difference, the degree of blurring determined based on the first difference is larger than the degree of blurring determined for the second difference; applying the degree of blurring to the image data in the privacy area of the image.

2. The method according to claim 1, wherein if the calculated difference is zero or negative, the degree of blurring is determined to be zero.

3. The method according to claim 2, further comprising when the difference is above a threshold difference, determining the degree of blurring to a predetermined value, and when the difference is between zero and the threshold difference, determining the degree of blurring to a value between zero and the predetermined value, wherein the value is related to the difference.

4. The method of claim 1, wherein the step of calculating the maximum spatial resolution is performed based on a focal length of an optical system of the image capturing device, the F-number of the optical system of the image capturing device, and a pixel pitch of an image sensor of the image capturing device.

5. The method of claim 1, wherein the step of applying the degree of blurring to the image comprises one from the list of: applying a gaussian filter, applying an average filter, applying a pixelization filter; wherein the degree of blurring determines the size of the applied filter.

6. The method of claim 1, further comprising the step of: determining that the privacy area covers at least a threshold portion of the image, setting a focus limit of the image capturing device to the threshold distance or less.

7. The method of claim 6, wherein the step of setting the focus limit comprises setting the focus limit of an autofocus algorithm of the image capturing device to the threshold distance or less.

8. The method of claim 6, wherein the step of setting a focus limit comprises: setting the focus limit to the threshold distance minus a determined value, wherein the value is determined based on a current focal length of an optical system of the image capturing device, the F-number of the optical system of the image capturing device and the threshold spatial resolution.

9. The method of claim 6, wherein the step of calculating a maximum spatial resolution comprises calculating the spatial resolution of image data of the image corresponding to content of the scene located on the threshold distance from the image capturing device.

10. A computer program product comprising a computer-readable storage medium with instructions adapted to carry out the method of claim 1 when executed by a device having processing capability.

11. A device adapted for determining a degree of blurring to be applied to image data in a privacy area of an image, the image depicting a scene and being captured by an image capturing device, the device comprising a processor configured to: receive input relating to coordinates of the privacy area of the image; receive input relating to a threshold spatial resolution of image data corresponding to content of the scene located beyond a threshold distance from the image capturing device in the scene, said image data being within the privacy area of the image; calculate a maximum spatial resolution of the image data of the image corresponding to content of the scene located on or beyond the threshold distance from the image capturing device; calculate a difference between the maximum spatial resolution and the threshold spatial resolution; determine a degree of blurring based on the calculated difference, such that for a first difference being larger than a second difference, the degree of blurring determined based on the first difference is larger than the degree of blurring determined for the second difference; apply the degree of blurring to the image data in the privacy area of the image.

12. A system comprising a first device for continuously capturing data of a scene, and producing a video stream comprising a sequence of images based on the captured data; a second device according to claim 11, adapted to continuously receiving the sequence of images from the first device.

13. The system of claim 12, wherein the first device and the second device are implemented in an image capturing device.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above, as well as additional objects, features and advantages, will be better understood through the following illustrative and non-limiting detailed description of preferred embodiments of the present invention, with reference to the appended drawings, where the same reference numerals will be used for similar elements, wherein:

[0050] FIG. 1 shows an image depicting scene and comprising a privacy area,

[0051] FIG. 2 shows the scene of FIG. 1 from above, including an image capturing device, the privacy area and a privacy threshold distance,

[0052] FIGS. 3-4 show example focus curves, and calculation of differences between maximum spatial resolution and a threshold spatial resolution according to embodiments,

[0053] FIGS. 5a-d each shows an embodiment of a mapping function between calculated differences according to FIGS. 3-4 and a degree of blurring to be applied to the privacy area of the image,

[0054] FIG. 6 shows a method for setting a focus limit of an image capturing device according to embodiments,

[0055] FIG. 7 shows a method for determining a degree of blurring to be applied to image data in a privacy area of an image according to embodiments,

[0056] FIG. 8 shows a method for setting a focus limit of an image capturing device according to embodiments.

DETAILED DESCRIPTION OF EMBODIMENTS

[0057] The aspects will now be described more fully hereinafter with reference to the accompanying drawings, in which embodiments are shown. The systems and devices disclosed herein will be described during operation.

[0058] FIG. 1 shows an image depicting a scene 100. The scene 100 comprises several objects 102, 104, 106, 108. In this example, the scene comprises a building 108, a tree 106, and two persons 102, 104. The building 108 has privacy restrictions, and privacy masking (e.g. blurring) is thus required for image data showing the building 108, e.g. the image data showing the building must be blurred to an extent such that details (according to the privacy restrictions) is not visible in the image showing the scene 100. For example, the privacy restrictions may define that the identity of persons entering the building 108 cannot be detected, or that the identity of persons that are visible through the windows of the building 108 is kept private for the viewer of the image. For this reason, a privacy area 110 is defined, surrounding the building 108. As visible in FIG. 1, two persons 102, 104 are depicted at least partly inside the privacy area 110, but clearly positioned in front of the building 108. This means that blurring applied to remove details (lower the spatial resolution) of image data corresponding to the building 108 will also affect the image data corresponding to the two persons 102, 104, since the blurring is applied to all pixels within the privacy area 110.

[0059] Using the concepts described herein, a threshold distance may be defined, which can be used for determining if, and how much, blurring should be applied to the privacy area of the image. FIG. 2 shows this concept. FIG. 2 shows the same scene 100 depicted in the image of FIG. 1 but seen from above. In addition to the objects 102-108 included in FIG. 1, FIG. 2 also shows an image capturing device 200, and the angle of view (AOV) 202 of the image capturing device 200. The image capturing device 200 may for example be a video camera a still image camera. In FIG. 2, the privacy area 110 is defined as an angular range originating from the image capturing device for ease of explaining. FIG. 2 further includes the threshold distance 204, which may be thought of as defining the privacy area 110 further in terms of a distance from the image capturing device 200. However, as understood by the skilled person, when applying the later determined blurring factor, this will affect all content of the scene within the privacy area 110 e.g. as shown in FIG. 1. FIG. 2 shows that the rightmost person 104 in FIG. 1 is positioned quite far from the threshold distance 204, while the leftmost person 102 is positioned closer to the threshold distance 204, but still on the side of the image capturing device 200 with respect to the threshold distance 204. From a privacy perspective, it is important that the spatial resolution of the image data showing the building 108 is low enough (depending on the use case) to meet the privacy requirements. However, from a surveillance perspective, it may be important that objects (e.g. persons 102, 104) between the building 108 and the video capturing device 200 can be shown in as much details as possible. The inventive concept described herein provides a solution to this problem.

[0060] FIG. 3 shows a schematic plot of two different focus curves 308a-b for the video capturing device 200 of FIGS. 1 and 2, depending on e.g. a focus point and a zoom value (focal length) of the image capturing device 200. The plot of FIG. 3 shows spatial resolution of image data based on a distance from the image capturing device at which the content of the scene which the image data corresponds to is located. In other words, the plot shows the spatial resolution as a function of a distance from the camera, or depth in the image. In the example of FIG. 3, the first curve 308a shows a focus curve when the leftmost person 102 is in focus for the image capturing device 200, at a first zoom level. The second curve 308b shows a focus curve when the rightmost person 104 is in focus for the image capturing device 200, at a second zoom level being lower than the first zoom level.

[0061] In the plot of FIG. 3, a threshold spatial resolution 302 is shown. The value of the threshold spatial resolution 302 sets a limit for how large the spatial resolution of image data corresponding to content of the privacy area 110 may be. In other words, the value of the threshold spatial resolution 302 sets a limit for the allowed spatial resolution of image data corresponding to content of the scene located beyond the threshold distance 204 from the image capturing device in the scene (the limit is set for image data being within the privacy area 110 of the image), such that the privacy requirements for such image data is met.

[0062] The plots of FIG. 3 also show the threshold distance 204. The focus curves 308a-b thus show, for different settings of the image capturing device, what spatial resolution different content of the image will have, depending on the depth in the image (i.e. distance from the image capturing device). For the first focus curve 308a, the spatial resolution for content at the threshold distance 204 differs from the threshold spatial resolution 302 with a positive value 306a. This means that the privacy requirements for the privacy area is not met. For example, this may mean that the faces of people in the windows of the building 108 in FIGS. 1-2 are identifiable. Based on this difference 306a between the maximum spatial resolution and the threshold spatial resolution, a degree of blurring will be determined (as will be further discussed below in conjunction with FIGS. 5a-d) and applied to the image data in the privacy area 110 of the image.

[0063] For the second focus curve 308b, the spatial resolution for content at the threshold distance 204 differs from the threshold spatial resolution 302 with a negative value 306b. This means that the privacy requirements for the privacy area are indeed met. For example, this may mean that the faces of people in the windows of the building 108 are not identifiable (e.g. the spatial resolution is less than 1 pixel per 0.1 meter on and beyond the threshold distance, or less than 2, 5, etc. pixels depending on the use case). Consequently, no blurring needs to be applied to the privacy area 110, which may mean that the image data corresponding to the objects 102, 104 can be better used for surveillance/monitoring purposes.

[0064] In FIG. 3, the plot shows that peaks 310a-b of the focus curves 308a-b are closer to the image capturing device 200 than the threshold distance 204. This means that the focal point 310a-b of the camera is between the threshold distance and the image capturing device 200, e.g. at threshold distance/1.5 or threshold distance/2. In this case, the maximum spatial resolution of the image data of the image corresponding to content of the scene located on or beyond the threshold distance 204 from the image capturing device 200 will always be the spatial resolution at the threshold distance 204. However, this is not always the case. FIG. 4 shows an example where the focus point 310c is beyond the threshold distance 204. In this case, the difference between the maximum spatial resolution and the threshold spatial resolution 302 will be the difference 306c between the spatial resolution at the focus point 310c distance from the image capturing device 200, i.e. the focal length of the optical system of the image capturing device 200 when capturing the image. This difference 306c will then be used to determine the degree of blurring to be applied to the image data of the privacy area 110 of the image.

[0065] The maximum spatial resolution for objects in a scene located on or beyond a threshold distance from the image capturing device may be calculated using algorithms defined for calculation of the term circle of confusion, e.g. as defined in the Wikipedia article on the subject at the date of filing of the present application (https.//en.wikipedia.org/wiki/Circle_of_confusion).

[0066] Specifically, to calculate the diameter of the circle of confusion in the image plane for an out-of-focus subject, one method is to first calculate the diameter of the blur circle in a virtual image in the object plane and then multiply by the magnification of the system, which is calculated with the help of the lens equation. The circle of confusion (CoC) diameter, i.e. the blur circle, of diameter C, may be calculated using the formula:

[00001]$\begin{array}{cc}C=\frac{\mathrm{abs}\left(S.Math..Math.2-S.Math..Math.1\right)}{S.Math..Math.2}*\frac{f^2}{N*\left(S.Math..Math.1-f\right)}& \mathrm{Equation}.Math..Math.1\end{array}$

where C=diameter of CoC, S1=the focal point of the image capturing device (i.e. the distance where the focus curve peaks, e.g. reference 310a-b in FIG. 3), S2=the threshold distance (e.g. reference 204 in FIG. 3), N=the F-number of a lens of the image capturing device and f=the focal length of the optical system of the image capturing device.

[0067] Given that

C=f*tan(th)Equation 2

where th=the angular resolution, and by the approximation that tan(th)=th for small angles, it is given that:

[00002]$\begin{array}{cc}\mathrm{th}=\frac{\mathrm{abs}\left(S.Math..Math.2-S.Math..Math.1\right)}{S.Math..Math.2}*\frac{f}{N*\left(S.Math..Math.1-f\right)}& \mathrm{Equation}.Math..Math.3\end{array}$

[0068] If Equation 3 is evaluated at the threshold distance (S2), and by subtracting the threshold spatial resolution (th), the degree of blurring (B) (in pixels) may be calculated by:

[00003]$\begin{array}{cc}B=f*\left(\mathrm{abs}\left(\frac{\mathrm{abs}\left(S.Math..Math.2-S.Math..Math.1\right)}{S.Math..Math.2}*\frac{f}{\left(N*S.Math..Math.1-f\right)}\right)-{\mathrm{th}}^{}\right)/p& \mathrm{Equation}.Math..Math.4\end{array}$

where p=the pixel pitch of an image sensor of the image capturing device, and where S1 is between the image capturing device and the threshold distance S2 (as described in FIG. 3). For the case where S1 is beyond S2 (as described in FIG. 4), B may be calculated by:

[00004]$\begin{array}{cc}B=\frac{f*{\mathrm{th}}^{}}{p}& \mathrm{Equation}.Math..Math.5\end{array}$

[0069] The equation 1-5 specifies an exact way of calculating the degree of blurring to be applied. This is schematically shown in FIG. 5a, where a continuous mapping function 502 between the degree of blurring and the calculated difference between the maximum spatial resolution and the threshold spatial resolution is shown. However, the mapping function 502 may comprise a predetermined maximum degree of blurring as shown in FIG. 5b. In this case, when the difference is above a threshold difference 506, the degree of blurring is set to a predetermined value 504 (for example a kernel size of 128*128 pixels for the used low-pass filter). When the difference is between zero and the threshold difference 506, the degree of blurring is determined to a value between zero and the predetermined value 504, wherein the value is related to the difference. In FIG. 5b, between zero and the threshold difference 506, the mapping function 502 is continuous as in FIG. 5a. Just as for the example shown in FIG. 5a, for differences below zero, the degree of blurring is determined to be zero.

[0070] In FIG. 5c, the max blurring degree 504 still applies as in FIG. 5b. However, in FIG. 5c, between zero and the threshold difference 506, the mapping function 502 is a step function being step wise correlated to the distance.

[0071] In the simplest form, the mapping function between the calculated difference and the determined blurring factor (degree of blurring) is as shown in FIG. 5d. For differences above zero, a predetermined degree of blurring 504 is applied to the privacy area of the image. For differences below zero, no degree of blurring is applied.

[0072] In all examples shown in FIGS. 5a-d, for a first difference being larger than a second difference, the degree of blurring determined based on the first difference is larger than the degree of blurring determined for the second difference.

[0073] FIG. 7 shows by way of example a method for determining a degree of blurring to be applied to image data in a privacy area of an image, for example for the scenario shown in FIGS. 1-2.

[0074] According to embodiments, the method comprises the step of receiving, S702, input relating to coordinates of the privacy area of the image.

[0075] The method further comprises receiving, S704, input relating to a threshold spatial resolution of image data corresponding to content of the scene located beyond a threshold distance from the image capturing device in the scene.

[0076] The threshold spatial resolution may be received e.g. in the form of an angular resolution for each pixel in the image (i.e. 1 or 0.5), or a value describing how many pixels an object of a certain size should be represented by at the threshold distance (e.g. 15*15 pixels for a 1*1 meter large object) or any other suitable measure of spatial resolution.

[0077] Further, the method comprises calculating S706 a maximum spatial resolution of the image data of the image corresponding to content of the scene located on or beyond the threshold distance from the image capturing device, e.g. using equation 3 above, or adaptations of equation 3 based on the format of the received threshold spatial resolution.

[0078] The two measures, the threshold spatial resolution and the maximum spatial resolution is then used for calculating S708 a difference between the maximum spatial resolution and the threshold spatial resolution, and determining S710 a degree of blurring based on the calculated difference, such that for a first difference being larger than a second difference, the degree of blurring determined based on the first difference is larger than the degree of blurring determined based on the second difference. The determined degree of blurring is then applied to the pixels of the privacy area, for example using a gaussian filter, an average filter or a pixelization filter, wherein the degree of blurring determines the size of the applied filter.

[0079] The determined degree of blurring is then applied S712 to the privacy area of the image, i.e. by applying the low pass filter with a kernel size according to the determined degree of blurring.

[0080] As described in FIGS. 3 and 4, the focus point of the image capturing device can be both closer to the image capturing device compared to the threshold distance, as well as beyond the threshold distance. The presence of a threshold distance can be taken advantage of in the context of focus limits of the image capturing device, in that the focus limit can be set to the threshold distance or less. Since, from a privacy perspective, it may not be advantageous if the focus point is on or beyond the threshold distance (which will increase the spatial resolution of the image data of the image corresponding to content of the scene located on or beyond the threshold distance from the image capturing device as described above), the focus limit can advantageously be set to the threshold distance or less. This in turn will reduce the spatial resolution of the image data of the image corresponding to content of the scene located on or beyond the threshold distance from the image capturing device, and thus reduce the required additional blurring that needs to be applied to the privacy area. Moreover, by setting a focus limit, autofocus, AF, of the image capturing device may be achieved both faster and requiring less processing power, since the possible range for AF is reduced.

[0081] According to some embodiments, the focus limit is set to the threshold distance minus a determined value. This is shown in FIG. 6, where the focus limit 602 is set at a distance 604 from the threshold distance 204. As described above, also content at a certain distance beyond the actual focus point will appear acceptably sharp (e.g. having a spatial resolution above the threshold spatial resolution, also known as Depth of Field, DoF) in the captured image data, which can be taken advantage of when setting the focus limit. The determined value 604 is calculated e.g. as described above, using the focal length, the F-number, and the threshold spatial resolution.

[0082] It should be noted that the above strategy of setting of a focus limit may be performed independently of the method for determining a degree of blurring to be applied to image data in a privacy area of an image as described above. Such independently performed method is shown in FIG. 8.

[0083] In its simplest form, the method for setting a focus limit of an image capturing device comprises the steps of receiving S802 input relating to a privacy threshold distance relating to a privacy requirement for images captured by the image capturing device. The privacy threshold distance may be defined similar to what is described above, i.e. a distance from the camera beyond which privacy requirements applies.

[0084] The method further comprises the step of setting S808 a focus limit of the image capturing device to the threshold distance or less.

[0085] This method can, according to embodiments, further take advantage of the DoF-concept as described above in conjunction with FIG. 6. In this embodiment, the method further comprises receiving S804 input relating to a threshold spatial resolution of image data corresponding to content of the scene located beyond the threshold distance 204 from the image capturing device in a scene captured by the image capturing device.

[0086] The threshold spatial resolution can then be used for calculating the DoF for the image capturing device (with its current settings, e.g. F-number) for focal points (focus points) at some distances from the privacy threshold distance. As described above, also content at a certain distance beyond the actual focus point will appear acceptably sharp (e.g. having a spatial resolution above the threshold spatial resolution, also known as Depth of Field, DoF) in the captured image data, which can be taken advantage of when setting the focus limit. By taking the DoF into account, the focus limit of the image capturing device can be set at a distance from the privacy threshold, such that the spatial resolution on and beyond the privacy threshold distance always will be lower than the threshold spatial resolution. Determining at what distance from the privacy threshold distance the focus limit should be may be an iterative process, where a plurality of focus points are used for calculation of DoF for the camera (where the threshold spatial resolution thus is used for defining the circle of confusion criteria), and where the focus point where the acceptably sharp area ends closest to the threshold distance is set S808 as the focus limit for the image capturing device.

[0087] In other embodiments, depending on the threshold spatial resolution and the settings of the image capturing device, a focus limit is set S808 based on e.g. a predetermined table of focus limits.

[0088] In the above the inventive concept has mainly been described with reference to a limited number of examples. However, as is readily appreciated by a person skilled in the art, other examples than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended claims.