CLASSIFICATION OF PIXEL WITHIN IMAGES CAPTURED FROM THE SKY

Abstract

Pixels are classified within a time series of first and second images for the first image, a first probability map is provided with a first probability for a cloud for each first pixel and, for the second image, a second probability map with a second probability for a cloud for each second pixel; first and second mean intensity values are calculated for the pixels; local zero mean images are calculated by subtracting the mean intensity value from the intensity value of the respective pixel; a maximum difference map is generated by calculating, for spatially corresponding pixels, an absolute difference value between a first and second zero mean value; a weighting map is produced by multiplying each absolute difference value with a non-linear function; and a classifying map is computed based on the first probability map, the second probability map, and the weighting map.

Claims

1-13. (canceled)

14. A method for classifying pixels within a time series of a previously captured first image and a currently captured second image of the sky, each image having a plurality of pixels each with a given intensity value, the method comprising: providing, for the first image, a first probability map that includes, for each pixel, a first probability value that the pixel represents a cloud in the sky and providing, for the second image, a second probability map that includes, for each pixel, a second probability value that the pixel represents a cloud in the sky; calculating a first mean intensity value for first pixels of the first image and a second mean intensity value for second pixels of the second image; determining a first local zero mean image by subtracting, for each first pixel, the first mean intensity value from the intensity value of the respective first pixel and a second local zero mean image by subtracting, for each second pixel, the second mean intensity value from the intensity value of the respective second pixel; generating a maximum difference map by calculating, for each first pixel and for a spatially corresponding second pixel, an absolute difference value between a respective first zero mean value of the first local zero mean image and a respective second zero mean value of the second local zero mean image; producing a weighting map by multiplying each absolute difference value of the maximum difference map with a function value of a non-linear function specifying the function value as a non-linear function of the absolute difference value; and computing a pixel classifying map based on the first probability map, the second probability map, and the weighting map.

15. The method according to claim 14, wherein: the first image is a first color image having at least three first spectral intensity values for each first pixel; and the second image is a second color image having at least three second spectral intensity values for each second pixel; for determining the two local zero mean images: the first mean intensity value is given by the mean intensity of all first spectral intensity values; the second mean intensity value is given by the mean intensity of all second spectral intensity values; the first local zero mean image comprises at least three first spectral zero mean images each being determined by subtracting, for each first pixel, the first mean intensity value from the first spectral intensity value of the respective first pixel; and the second local zero mean image comprises at least three second spectral zero mean images each being determined by subtracting, for each second pixel, the second mean intensity value from the second spectral intensity value of the respective second pixel; for generating the maximum difference map the absolute difference value is a maximum absolute difference value which is given by the biggest absolute difference of at least three spectral absolute difference values, wherein each one of the at least three spectral absolute difference values is calculated by, for each first pixel and for a spatially corresponding second pixel, the absolute difference value between one of the three first spectral intensity values and a spectrally corresponding one of the three second spectral intensity values.

16. The method according to claim 14, further comprising, after generating the maximum difference map and before producing the weighting map, modifying each absolute difference value by applying a threshold operation.

17. The method according to claim 16, wherein applying the threshold operation comprises: applying an upper threshold value, if the absolute difference value is geater than the upper threshold value; and/or applying a lower threshold value, if the absolute difference value is smaller than the lower threshold value.

18. The method according to claim 16, which comprises, after modifying each absolute difference value by applying the threshold operation and before producing the weighting map, further modifying each absolute difference value by applying a normalization operation.

19. The method according to claim 14, further comprising, after multiplying each absolute difference value of the maximum difference map with a function value of a non linear function, and before computing the pixel classifying map; modifying the weighting map to a modified weighting map by applying a filtering operation, and using the modified weighting map for computing the pixel classifying map.

20. The method according to claim 19, wherein the step of computing the pixel classifying map comprises applying the following expression:
Pn=PC*Wp+Pp×(1−Wp) where: Pn is the pixel classifying map; Pc is the second probability map; Pp is the first probability map; and Wp is the modified weighting map.

21. The method according to claim 14, wherein the step of computing the pixel classifying map comprises applying the following expression:
Pn=PC*Wp+Pp×(1−Wp) where: Pn is the pixel classifying map; Pc is the second probability map; Pp is the first probability map; and Wp is the weighting map.

22. The method according to claim 14, which comprises obtaining the function value of the non-linear function from a lookup table.

23. The method according to claim 14, wherein a codomain for the function value of the non-linear function lies between a lower saturation value, wherein the lower saturation value is between zero and unity.

24. The method according to claim 14, wherein, with increasing absolute difference value or with increasing modified absolute difference value, in a first region the non-linear function has a constant function value of unity; in a following second region the non-linear function decreases towards the lower saturation value; and in a further following third region the non-linear function has a constant function value of the lower saturation value.

25. A data processing unit for classifying pixels within a time series of a previously captured first image and a currently captured second image of the sky, wherein each image comprises a plurality of pixels each having a given intensity value, and wherein the data processing unit is configured for carrying out the method according to claim 14.

26. A non-transitory computer program for classifying pixels within a time series of at least one previously captured first image and a currently captured second image of the sky, wherein each image comprises a plurality of pixels each having a given intensity value, the computer program, when being executed by a data processing unit, being configured for carrying out the method according to claim 14.

27. An electric power system, comprising: a power network; a photovoltaic power plant for supplying electric power to said power network; at least one further power plant for supplying electric power to said power network and/or at least one electric consumer for receiving electric power from said power network; a control device for controlling an electric power flow between said at least one further power plant and said power network and/or between said power network and said at least one electric consumer; and a prediction device for producing a prediction signal being indicative of an intensity of a sun radiation to be captured by said photovoltaic power plant in the future; wherein said prediction device having a data processing unit according to claim 25; said prediction device is communicatively connected to said control device; and said control device being configured to control, based on the prediction signal, the electric power flow in the future.

Description

BRIEF DESCRIPTION OF THE DRAWING

[0051] FIG. 1 shows a flow diagram for a method for classifying pixel within a time series of input images.

[0052] FIG. 2 shows an exemplary non-linear function for an intensity difference mapping.

[0053] FIG. 3 shows two input images I1 and I2.

[0054] FIG. 4 shows an estimated perceptual structural difference image for the first input image I1 and second input image I2.

[0055] FIG. 5 shows two cloud classification images, the left one obtained with an algorithm according to prior art, the right one obtained with a method according to an embodiment of the invention.

[0056] FIG. 6 shows an electric power system with a data processing unit in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

[0057] The illustration in the drawing is schematic. It is noted that in different figures, similar or identical elements or features are provided with the same reference signs or with reference signs, which are different from the corresponding reference signs only within the first digit. In order to avoid unnecessary repetitions elements or features which have already been elucidated with respect to a previously described embodiment are not elucidated again at a later position of the description.

[0058] FIG. 1 shows a flow diagram for a method for classifying pixel within a time series of input images. The method starts with a step S1, wherein two images of the sky, a first image I1 and a second image I2, are input in a data processing unit with which the method is carried out. The images I1 and I2 have been captured by a camera viewing the sky. The camera is positioned close to a photovoltaic power station capturing sun light intensity and converting the sun light intensity into electric power.

[0059] In a next following step S2.1 there is calculated a first mean intensity value for first pixels of the first image I1 and a second mean intensity value for second pixels of the second image I2. Thereby, an (arithmetic) average intensity value for all pixels within one image I1, I2 is calculated.

[0060] In a next step S2.2 there is determined a first local zero mean image Z1 and a second local zero mean image Z2. Thereby, for each first pixel of the first image I1 the before calculated first mean intensity value is subtracted from the individual intensity value of the respective first pixel. Correspondingly, for each second pixel of the second image I2 the before calculated second mean intensity value is subtracted from the individual intensity value of the respective second pixel.

[0061] In a next step S2.3 there is generated a maximum difference map |Z1-Z2|, which in this document is denominated as M0.

[0062] Thereby, it is calculated for each first pixel of the first image I1 and for a spatially corresponding second pixel of the second image I2, an absolute difference value between a respective first zero mean value of the first local zero mean image Z1 and a respective second zero mean value of the second local zero mean image Z2.

[0063] In a next step S2.4 each absolute difference value is modified by applying a threshold operation. Thereby, those absolute difference values, which do not comply with a respective threshold condition, are transformed into modified absolute difference values which differ from the corresponding absolute difference values. Further, those absolute difference values, which do comply with the threshold condition, are transformed into modified absolute difference values which are the same as the corresponding absolute difference values. The result of this threshold operation is a threshold map M1.

[0064] According to the exemplary embodiment described here the threshold operation comprises (i) applying an upper threshold value Tu, if the absolute difference value is larger than (or equal to) the upper threshold value Tu and/or (ii) applying a lower threshold value Tl, if the absolute difference value is smaller than (or equal to) the lower threshold value Tl.

[0065] In a next step S2.5 a normalization operation is applied to each absolute difference value. According to the exemplary embodiment described here this normalization transforms all values into the range between 0 and 1. The result is a normalized map M2.

[0066] A perception of a processed image depends on the intensity of at least some pixels of the processed image. Therefore, in a next step S2.6 each value of the normalized map M2 is multiplied with a function value of a non-linear function L(g). The non-linear function L(g) being used in the embodiment described here is depicted in FIG. 2, which will be elucidated further below. With this multiplication operation, which yields a multiplied map M3, the perception of the respective image being specified by M3 is improved.

[0067] In a next step S2.7 a filtering operation is applied to M3 in order to generate a weighing map Wp. According to the exemplary embodiment described here the filtering operation is a convolution with which the number range of the values of the multiplied map M3 is expanded.

[0068] Parallel to at least one of the steps 2.1 to 2.7 there is carried out a step (or procedure) S3 with which there is generated a first probability map Pp and a second probability map Pc. Both probability maps Pp and Pc describe, for each pixel, the probability for representing a portion of a cloud. This is also denominated a cloud segmentation. The first probability map Pp is the cloud segmentation at a first (previous) point in time and the second probability map Pc is the cloud segmentation at a second (current) point in time. It is pointed out that both probability maps Pp and Pc are obtained by means of well-known image classification or processing procedures for a segmentation of cloud/sky images, which procedures also use the captured images I1 and I2 as an input.

[0069] With a step S4 the two generated probability maps Pp and Pc are provided as further inputs to the described method.

[0070] In a next step S5 the so far generated respectively provided maps Pc, Wp, and Pp are combined by means of a weighing operation. According to the exemplary embodiment described here the formula for weighing is Pn=PC*Wp+Pp×(1−Wp). Thereby, Pn is the new probability map which in this document is denominated a pixel classifying map.

[0071] Descriptively speaking, the weighing map Wp is used as a perceptual weighting between the current segmentation map Pc and the previous segmentation map Pp in order to arrive at the new segmentation map Pn (=Pc*Wp+Pp*(1−Wp)). Thereby, this equation is understood as an operation for each pixel value.

[0072] FIG. 2 shows an exemplary non-linear function L(g) for an intensity difference mapping from the normalized map M2 to the multiplied map M3. At the abscissa of the shown graph there is depicted the (modified) absolute difference value g. At the ordinate there is depicted the function value f of the non-linear function W(g).

[0073] As can be seen from FIG. 2, the non-linear function L(g) comprises a comparatively smooth transition between an upper value 1 and a lower saturation value f(n). The lower saturation value f(n) is between 0 and 1. According to the exemplary embodiment described here the lower saturation value f(n) is smaller than 0,5.

[0074] It is mentioned that a non-linear function L(g) with a smooth transition may be preferred. However, the algorithm described here is robust with respect to such fine tuning such that also non-linear functions can be used which exhibit a more sharp transition.

[0075] FIG. 3 shows two input images I1 and I2, which are processed with the described method. FIG. 4 shows an estimated perceptual structural difference image 460 illustrating the difference between the first input image I1 and second input image I2 to illustrate the effectiveness of the described method.

[0076] The optical conditions when capturing the two input images I1 and I2 are clear-sky. The two images I1 and I2 have been captured with a time difference of 10 seconds. The hardly visible clouds are indicated in each image I1, I2 by two circles. From the left image I1 to the right image I2 the clouds move a few pixels away and nothing else is perceivable.

[0077] The Perceptual Structural Difference (PSD) image which has been obtained with the algorithm of the described method is shown in FIG. 4. It is consistent with a human visual perception of the two images I1 and I2. In this Figure referred to as the PSD map, white zones are indicative for image regions which are the same in both images I1 and I2. Black zones are indicative for a clearly different cloud probability and visualize a cloud movement. In this PSD map continuous values are illustrated which are indicative of the probability of difference between the two images I1 and I2. Therefore, any value between the white and black (i.e., gray zone) indicates the strength of difference between I1 and I2. In other words: the darker an image region in the PSD map, the more likely is a difference between I1 and I2 (in this image region).

[0078] FIG. 5 shows two cloud classification or segmentation images 572 and 574. The cloud classification image 572 shows an image which has been obtained with an algorithm according to the prior art. The cloud classification image 574 has been obtained with a method according to an embodiment of the invention.

[0079] The grey scale code bar depicted vertically top right of each image is a scale for the cloud probability. Generally, dark zones indicate a low probability of the respective pixels for representing a cloud and bright zones indicate a high probability of being a cloud. One can clearly see that the image 574 includes some significant corrections of artefacts which are included in the image 572, in particular in the zone or in the region near the sun (approximately in the middle of each image). In the image 574 the clear sky condition is correctly identified without producing a false alarm for a cloud coverage.

[0080] FIG. 6 shows an electric power system 600 in accordance with an embodiment of the invention. The electric power system 600 comprises a power network 610 which receives electric power from three exemplary depicted power plants, a photovoltaic power plant 620, a coal-fired power plant 642, and a hydroelectric power plant 644. It is pointed out that the power plants 642 and 644 are just given as an example and other and/or different numbers of such plants can be used. Further, the electric power system 600 comprises two electric power consumers receiving electric power from the power network 610. In FIG. 6 there are depicted, by way of example, an industrial complex 646 and a household 648. The power flows from the power plants 620, 642, and 644 to the power network 610 as well as the power flows from the power network 610 to the electric consumers 646 and 648 are indicated in FIG. 6 with double arrows.

[0081] The photovoltaic power plant 620 is driven by the sun 698 irradiating on non-depicted solar panels of the photovoltaic power plant 620. In order to predict the electric power which can be generated by the photovoltaic power plant 620 in the near future there is provided a prediction device 630. The prediction device 630 comprises a camera 632 for capturing a series of images of the sky over the photovoltaic power plant 620. The captured images, two of which are the input images I1 and I2 as described above, are forwarded to a data processing and control device 634. A data processing section or data processing unit of the data processing and control device 634 is configured for carrying out the method as described above for classifying pixels within the captured images whether they represent cloud or sky. A control section of the data processing and control device 634 is communicatively connected with (at least some of) the power plants 642 and 644 and with (at least some of) the electric consumers 646 and 648. The corresponding wired or wireless data connections are indicated in FIG. 6 with dashed lines.

[0082] With (the data processing unit of) the data processing and control device 634 carrying out the described method a cloud occlusion prediction within the near future can be made. This cloud occlusion prediction directly corresponds to a prediction of the power, which can be supplied from the photovoltaic power plant 620 to the power network 610 in the near future. This allows to control, by means of (the control section of) the data processing and control device 634, the operation of the power plants 642, 644 and the electric consumers 646, 648 can be controlled in such a manner that the power flow to and the power flow from the power network 610 are balanced at least approximately. Hence, the stability of the power network 610 and, as a consequence, also the stability of the entire electric power system 600 can be increased.

[0083] It is pointed out that in the embodiment described here the data processing unit and the control section are realized by one and the same device, namely the data processing and control device 634. However, it should be clear that the data processing unit and the control section can also be realized by different devices which are communicatively connected in order to forward the prediction signal from the data processing unit to the control section.

[0084] In order to descriptively recapitulate embodiments of the invention disclosed in this document one can state: The method described in this document calculates a weighted sum between a current segmentation map and a previous segmentation map. This relies on the assumption that the previous segmentation is not optimal but still satisfactory. Although this is generally true, additional steps are described to further mitigate false alarms of cloud coverage prediction as follows.

[0085] (1) If the number of cloud pixels within a designated sun neighborhood is less than a predefined threshold, the near sun area is classified as clear sky and void all cloud pixels.

[0086] (2) If the cloud speed is not within certain limits of the (global) average speed, these clouds are removed. This temporally smooths out the cloud segmentation because sudden appearance of clouds would have a very large speed resulting from a most likely wrong segmentation.

[0087] 3. For those cloud pixels which are not moving towards the sun and which are not within a range of the average direction, they are likely to be noisy pixels and should be eliminated.

[0088] It should be noted that the term “comprising” does not exclude other elements or steps and the use of articles “a” or “an” does not exclude a plurality. Also elements described in association with different embodiments may be combined. It should also be noted that reference signs in the claims should not be construed as limiting the scope of the claims.

LIST OF REFERENCE SIGNS

[0089] S1 Start: Input two input images [0090] S2.1 Calculating: first mean intensity value & second mean intensity value [0091] S2.2 Determining: first local zero mean image Z1 & second local zero mean image Z2 [0092] S2.3 Generating: Maximum difference map M0=|Z1-Z2| [0093] S2.4 Thresholding/saturating:->M1 [0094] S2.5 Normalizing:->M2 [0095] S2.6 Multiplying: M3=M2×L(g) [0096] S2.7 Filtering:->Wp [0097] S3 conventional cloud segmentation [0098] S4 Providing: first probability map Pp & second probability map Pc [0099] S5 Weighing: Pn=Pc*Wp+Pp*(1−Wp) [0100] S6 Output: New pixel classifying map Pn [0101] I1, I2 first/second input image [0102] Z1, Z2 second/second local zero mean image [0103] M0 maximum difference map [0104] M1 threshold map [0105] M2 normalized map [0106] M3 multiplied map [0107] Wp (modified) weighing map [0108] Pp first (previous) probability map [0109] Pc second (current) probability map [0110] Pn (new) pixel classifying map [0111] g absolute difference value [0112] f function value [0113] L(g) non-linear function [0114] f(n) lower saturation value [0115] 460 estimated perceptual structural difference image for first output image I1 and second output image I2 [0116] 572 cloud classification image obtained with a prior art algorithm [0117] 574 cloud classification image obtained with a method according to an embodiment of the invention [0118] 600 electric power system [0119] 610 power network [0120] 620 photovoltaic power plant [0121] 630 prediction device [0122] 632 camera [0123] 634 data processing and control device [0124] 642 coal-fired power plant/gas-fired power plant [0125] 644 hydroelectric power plant [0126] 646 industrial complex/factory [0127] 648 household(s)/domestic home(s) [0128] 698 sun