CLOUD DETECTION IN AERIAL IMAGERY

Abstract

A method of detecting clouds in an acquired aerial image includes determining a region of a reference aerial image corresponding to a region of an acquired aerial image. For each of a plurality of locations over the region of the acquired aerial image and corresponding to a plurality of locations over the region of the reference aerial image, the mutual information of one or more variables associated with the location in the acquired aerial image and one or more variables associated with the corresponding location in the reference aerial image is calculated. Using the mutual information calculated for each of the plurality of locations over the region of the acquired aerial image, it is determined when the acquired aerial image displays a cloud at the location in the region of the acquired aerial image.

Claims

1. A method of detecting clouds in an acquired aerial image, the method comprising: determining a region of a reference aerial image corresponding to a region of an acquired aerial image; calculating, for each of a plurality of locations over the region of the acquired aerial image and corresponding to a plurality of locations over the region of the reference aerial image, the mutual information of one or more variables associated with the location in the acquired aerial image and one or more variables associated with the corresponding location in the reference aerial image; and determining, using the mutual information calculated for each of the plurality of locations over the region of the acquired aerial image, when the acquired aerial image displays a cloud at the location in the region of the acquired aerial image.

2. The method as claimed in claim 1, wherein the method further comprises: aligning the region of the acquired aerial image to the region of the reference aerial image.

3. The method as claimed in claim 2, wherein the region of the acquired aerial image is aligned to the region of the reference aerial image using meta-data associated with the acquired aerial image and meta-data associated with the reference aerial image.

4. The method as claimed in claim 1, wherein the method further comprises: warping the region of the acquired aerial image to the region of the reference aerial image.

5. The method as claimed in claim 1, wherein the method further comprises scaling the region of the acquired aerial image to the region of the reference aerial image

6. The method as claimed in claim 1, wherein the mutual information is calculated using one or more image data variables associated with the location in the acquired aerial image and one or more image data variables associated with the location in the reference aerial image.

7. The method as claimed in claim 6, wherein the acquired aerial image comprises a grayscale image, the reference aerial image comprises a colour image, and wherein the mutual information is calculated using the grayscale luminance data from the region of the acquired aerial image and luminance of the colour image data from the reference aerial image.

8. The method as claimed in claim 1, wherein the mutual information is calculated using the one or more variables over a patch encompassing the location in the acquired aerial image and the one or more variables over a patch encompassing the corresponding location in the reference aerial image.

9. The method as claimed in claim 1, wherein the step of determining when the acquired aerial image displays a cloud at the location in the region of the acquired aerial image comprises applying a threshold to the mutual information, and when calculated mutual information at the location is greater than the threshold, determining that the acquired aerial image is displaying a cloud at the location.

10. The method as claimed in claim 1, wherein the method further comprises: determining, using the determination of when the acquired aerial image displays a cloud at the plurality of locations in the region of the acquired aerial image, the proportion of the region of the acquired aerial image at which cloud is displayed

11. The method as claimed in claim 10, further comprising storing the proportion of the region of the acquired aerial image at which cloud is displayed in meta-data associated with the acquired aerial image.

12. The method as claimed in claim 1, wherein the method further comprises: for each of the plurality of locations over the region of the acquired aerial image at which it has been determined that the acquired aerial image displays a cloud; and superimposing at least some of the image data from the corresponding location in the reference aerial image onto the acquired aerial image at the location in the acquired aerial image

13. A computer readable storage medium storing computer software code which when executing on a data processing system performs a method as claimed in claim 1.

14. A data processing system detecting clouds in an acquired aerial image, the data processing system comprising: processing circuitry configured to: determine a region of a reference aerial image corresponding to a region of an acquired aerial image; calculate, for each of a plurality of locations over the region of the acquired aerial image and corresponding to a plurality of locations over the region of the reference aerial image, the mutual information of one or more variables associated with the location in the acquired aerial image and one or more variables associated with the location in the reference aerial image; and determine, using the mutual information calculated for each of the plurality of locations over the region of the acquired aerial image, when the acquired aerial image displays a cloud at the location in the region of the acquired aerial image.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0050] One or more non-limiting examples will now be described, by way of example only, and with reference to the accompanying figures in which:

[0051] FIGS. 1a and 1b show corresponding acquired and reference images to be compared;

[0052] FIG. 2 shows a data processing system for analysing the acquired and reference images shown in FIGS. 1a and 1b; and

[0053] FIG. 3 shows a flow chart of the steps of analysing the acquired and reference images shown in FIGS. 1a and 1b using the data processing system shown in FIG. 2.

DETAILED DESCRIPTION OF DRAWINGS

[0054] When analysing an acquired aerial image of an area of the earth's surface, it is helpful to be able to assess the cloud coverage in the image, such that this information may be used when performing further analysis of the acquired aerial image. An example of how this may be performed will now be described.

[0055] FIGS. 1a and 1b show an acquired aerial image 1 and a reference aerial image 2 respectively. The acquired aerial image 1 to be analysed is taken from an airborne platform (e.g. a drone) of an area of the earth's surface and, for example, comprises a grayscale image. The acquired aerial image 1 shows features such as roads 3 on the earth's surface and may also display clouds 4 that obscure the features (e.g. part of the roads 3) on the earth's surface.

[0056] For the purposes of analysing the acquired aerial image 1 to determine if the image 1 contains clouds 4, the image 1 is split up into square patches 5 centred on the pixels 6 of the acquired image 1. (For the purposes of clarity, not all of the pixels and patches in the acquired aerial image 1 are shown. It will be appreciated that if all the pixels in the acquired aerial image 1 are used, with patches of the size shown in FIG. 1a, these patches would overlap with at least the patches centred on the adjacent pixels, and in at least some examples such overlapping patches are used.)

[0057] The reference aerial image 2 to be compared to the acquired aerial image 1 is shown in FIG. 1b. The reference aerial image 2 (e.g. a colour image), showing the roads 3 on the earth's surface, is free from clouds. FIG. 1b also shows the acquired aerial image 1, divided into the square patches 5 centred on the pixels 6 of the acquired image 1, for the purposes of the comparison to be made. The resolution of the area of the reference aerial image 2 is the same as the resolution of the corresponding acquired aerial image 1 such that the pixels 6 in the acquired aerial image 1 correspond to pixels in the reference aerial image 2. Thus the region of the reference aerial image 2 that corresponds to the acquired aerial image 1 is also divided up into the same square patches.

[0058] The reference aerial image 2 is retrieved from a library of reference images (e.g. Google Maps). The retrieved reference aerial image 2 may cover the same geographic extent as the acquired aerial image 1 or, as shown in FIG. 1b, may cover a larger area, with a region of the reference aerial image 2 being used when comparing with the acquired aerial image 1.

[0059] FIG. 2 shows a data processing system 11 on which the acquired and reference images 1, 2 may be processed. The system 11 includes a client computer 12 that has a memory 13 in which the acquired aerial images are stored (this memory may be distributed over multiple computers connected to the client computer 12). The client computer 12 also includes a processor 14 that is arranged to analyse the acquired and reference aerial images 1, 2. The (e.g. processor 14 of the) client computer 12 may include a cache for storing regions of the acquired and reference aerial images 1, 2 locally while processing them.

[0060] The reference aerial images 2 are stored on and retrieved from a memory 15 on a remote server 16 (though, for example, could equally be stored on the client computer 12 or both the acquired and reference aerial images 1, 2 could be stored on the same computer). As with the memory 13 of the client computer 12, the memory 15 of the remote server 16 may be distributed across multiple connected remote servers.

[0061] Operation of the example shown in FIGS. 1a, 1b and 2 will now be described with reference to FIG. 3. FIG. 3 shows a flow chart detailing the steps of processing the acquired and reference aerial images 1, 2 to detect clouds 4 in the acquired aerial image 1.

[0062] First, an acquired aerial image 1 to be processed is selected by the processor 14 of the client computer 12 from its memory 13. Using meta-data (e.g. geographical coordinates) associated with the acquired aerial image 1, the corresponding region from a reference aerial image 2 is requested by the processor 14 from the memory 15 of the remote server 16 (step 21, FIG. 3).

[0063] Before the acquired aerial image 1 may be compared directly with the region of the reference aerial image 2, the acquired aerial image 1 is orthorectified to warp it onto the same perspective as the reference aerial image 2 (which will generally be in the Web Mercator format), so that the acquired and reference aerial images 1, 2 are aligned with each other (however, any shared perspective and/or projection between the acquired and reference aerial images 1, 2 may be used). The acquired aerial image 1 is also scaled to the same resolution as the corresponding region of the reference aerial image 2 (step 22, FIG. 3). This creates a 1-1 relationship between the pixels in the acquired aerial image 1 and the corresponding pixels in the reference aerial image 2.

[0064] Once the acquired aerial image 1 has been warped and scaled to the reference aerial image 2, the mutual information

[00001]$\left(I\left(X;Y\right)={}_{Y,X}.Math.p\left(x,y\right).Math.\log \left(\frac{p\left(x,y\right)}{p\left(x\right).Math.p\left(y\right)}\right).Math.\mathrm{dxdy}\right)$

for the image data in the acquired aerial image 1 and the image data in the reference aerial image 2 is calculated, for a patch 5 centred on each of the pixels 6 in the acquired aerial image 1 and the corresponding patch (centred on the corresponding pixels) in the reference aerial image 2 (step 23, FIG. 3).

[0065] The mutual information calculated for each patch 5 is compared to a threshold (step 24, FIG. 3). When the mutual information is greater than the threshold, this indicates that there is a good correlation between the image data in the acquired and reference aerial images 1, 2, indicating that it is likely that this patch 5 of the acquired aerial image 1 is free from cloud. When the mutual information is less than the threshold, this indicates that there is a poor correlation between the image data in the acquired and reference aerial images 1, 2, indicating that it is likely that this patch 5 of the acquired aerial image 1 is covered by cloud.

[0066] Comparing the calculated mutual information to the threshold for each of the patches 5 of the acquired aerial image 1 enables the number of patches 5 of the acquired aerial image 1 to be determined, simply by summing the number of patches 5 for which the mutual information is less than the threshold (step 25, FIG. 3). This then enables the proportion of the acquired aerial image 1 which is covered by cloud to be determined, by calculating the ratio of the number of patches 5 determined to contain cloud to the total number of patches 5.

[0067] The proportion is stored (i.e. written back to the memory 13) in the meta-data associated with the acquired aerial image 1, so that future users may use the proportion as a search key when filtering acquired aerial images 1 in future (e.g. they may only be interested in viewing acquired aerial images 1 that have less than a certain proportion of their area covered in cloud). When the proportion is very high, e.g. over 90%, the acquired aerial image 1 may simply be discarded, owing to it being of little value for future analysis of this area of the earth's surface.

[0068] To aid the user analyse the acquired aerial image 1 when some of the image is covered in cloud, the features from the reference aerial image 2 (corresponding to the patches 5 of the acquired aerial image 1 that are covered in cloud) can be blended onto the acquired aerial image 1 (step 26, FIG. 3).