IMAGE-BASED DISASTER DETECTION METHOD AND APPARATUS

Abstract

Disclosed herein are a method and apparatus for detecting a disaster based on images. The apparatus includes an image capture unit for capturing video using at least one camera and controlling the camera based on a camera control signal received from the outside; a disaster detection unit for generating a disaster log based on the video captured using the camera; a disaster analysis unit for calculating a disaster occurrence probability value based on the disaster log and determining whether to enter a camera control mode based on the disaster occurrence probability value; and a disaster alert unit for warning of a disaster based on a disaster alert request signal.

Claims

1. An apparatus for detecting a disaster, comprising: an image capture unit for capturing video using at least one camera and controlling the camera based on a camera control signal received from an outside; a disaster detection unit for generating a disaster log based on the video captured using the camera; a disaster analysis unit for calculating a disaster occurrence probability value based on the disaster log and determining whether to enter a camera control mode based on the disaster occurrence probability value; and a disaster alert unit for warning of a disaster based on a disaster alert request signal.

2. The apparatus of claim 1, wherein: the camera control signal is capable of including a disaster alert signal and information about a position at which it is suspected that a disaster occurs, and when the camera control signal includes the disaster alert signal and the information about the position at which it is suspected that a disaster occurs, the image capture unit rotates the camera to be directed at the position at which it is suspected that a disaster occurs and controls a lens of the camera so as to zoom in on the corresponding position.

3. The apparatus of claim 1, wherein the disaster log includes information about whether a disaster occurs on a time basis and information about a place at which a disaster occurs.

4. The apparatus of claim 1, wherein the disaster detection unit detects a disaster based on an image classification method using a convolutional neural network (CNN) model trained by classifying images into general images and disaster images and thereby generates the disaster log.

5. The apparatus of claim 4, wherein: the disaster detection unit performs disaster detection for a video section formed of n images (image sequence) from (t+1*s)-th to (t+n*s)-th frames on a time basis in the video captured using the camera and then performs disaster detection for a video section formed of n images (image sequence) from (t+d+1*s)-th to (t+d+n*s)-th frames, where t denotes a start position of the video, s denotes an interval between frames selected for disaster detection, d denotes an interval between video sections selected for disaster detection, and n denotes the number of images.

6. The apparatus of claim 1, wherein the disaster detection unit acquires a video section in which movement of the camera is minimized by calculating the movement of the camera using optical flow or Structure From Motion (SFM) technology and by performing inverse calculation, and generates the disaster log based on the video section.

7. The apparatus of claim 1, wherein: the image capture unit transmits the captured video to a server, and the disaster detection unit receives image data for the captured video from the server and generates the disaster log based on the image data.

8. An apparatus for detecting a disaster, comprising: a processor for generating a disaster log based on video captured using at least one camera, calculating a disaster occurrence probability value based on the disaster log, determining whether to enter a camera control mode based on the disaster occurrence probability value, and generating a camera control signal for controlling the camera depending on whether entry into the camera control mode is performed; and memory for storing one or more of the captured video, the camera control signal, and the disaster log.

9. The apparatus of claim 8, wherein: the camera control signal is capable of including a disaster alert signal and information about a position at which it is suspected that a disaster occurs, and when the camera control signal includes the disaster alert signal and the information about the position at which it is suspected that a disaster occurs, the processor rotates the camera to be directed at the position at which it is suspected that a disaster occurs and controls a lens of the camera so as to zoom in on the corresponding position.

10. The apparatus of claim 8, wherein the disaster log includes information about whether a disaster occurs on a time basis and information about a place at which a disaster occurs.

11. The apparatus of claim 8, wherein the processor detects a disaster based on an image classification method using a convolutional neural network (CNN) model trained by classifying images into general images and disaster images and thereby generates the disaster log.

12. The apparatus of claim 8, wherein: the processor performs disaster detection for a video section formed of n images (image sequence) from (t+1*s)-th to (t+n*s)-th frames on a time basis in the video captured using the camera and then performs disaster detection for a video section formed of n images (image sequence) from (t+d+1*s)-th to (t+d+n*s)-th frames, where t denotes a start position of the video, s denotes an interval between frames selected for disaster detection, d denotes an interval between video sections selected for disaster detection, and n denotes the number of images.

13. The apparatus of claim 8, wherein the processor acquires a video section in which movement of the camera is minimized by calculating the movement of the camera using optical flow or Structure From Motion (SFM) technology and by performing inverse calculation, and generates the disaster log based on the video section.

14. A method for detecting a disaster, comprising: capturing video using at least one camera; generating a disaster log based on the captured video; calculating a disaster occurrence probability value based on the disaster log; determining whether to enter a camera control mode based on the disaster occurrence probability value; generating a camera control signal for controlling the camera depending on whether entry into the camera control mode is performed; determining whether a disaster occurs based on the disaster occurrence probability value and generating a disaster alert request signal; and warning of the disaster based on the disaster alert request signal,

15. The method of claim 14, wherein, When the camera control signal is generated, the camera control signal is capable of including a disaster alert signal and information about a position at which it is suspected that a disaster occurs, and when the camera control signal includes the disaster alert signal and the information about the position at which it is suspected that a disaster occurs, the camera is rotated to be directed at the position at which it is suspected that a disaster occurs, and a lens of the camera is controlled to zoom in on the corresponding position.

16. The method of claim 14, wherein the disaster log includes information about whether a disaster occurs on a time basis and information about a place at which a disaster occurs.

17. The method of claim 14, wherein generating the disaster log is configured to detect a disaster based on an image classification method using a convolutional neural network (CNN) model trained by classifying images into general images and disaster images and to thereby generate the disaster log.

18. The method of claim 17, wherein: generating the disaster log is configured to perform disaster detection for a video section formed of n images (image sequence) from (t+1*s)-th to (t+n*s)-th frames on a time basis in the video captured using the camera and to then perform disaster detection for a video section formed of n images (image sequence) from (t+d+1*s)-th to (t+d+n*s)-th frames, where t denotes a start position of the video, s denotes a time interval between frames selected for disaster detection, d denotes an interval between video sections, and n denotes the number of image.

19. The method of claim 14, wherein generating the disaster log is configured to acquire a video section in which movement of the camera is minimized by calculating the movement of the camera using optical flow or Structure From Motion (SFM) technology and by performing inverse calculation and to thereby generate the disaster log.

20. The method of claim 14, wherein: capturing the video is configured to transmit the captured video to a server, and generating the disaster log is configured to receive image data for the captured video from the server and to thereby generate the disaster log based on the image data.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The above and other objects, features and advantages of the present invention will be more clearly understood from the following detailed description, taken in conjunction with the accompanying drawings, in which:

[0039] FIG. 1 is a block diagram illustrating an example of an apparatus for detecting a disaster according to an embodiment of the present invention;

[0040] FIG. 2 is a flowchart illustrating the disaster analysis procedure of a disaster analysis unit according to an embodiment of the present invention;

[0041] FIG. 3 illustrates an example in which a wildfire image is overlaid with the result of image classification for the wildfire image calculated as a probability map;

[0042] FIG. 4 illustrates an example in which a wildfire image is overlaid with a motion map;

[0043] FIGS. 5A and 5B illustrate an example of conversion of a 2D array into a 1D array and an example of conversion of N sequences formed of 1D arrays into a 2D array, respectively;

[0044] FIG. 6 is a flowchart illustrating the operation of detecting a disaster by converting the input image sequence according to an embodiment of the present invention; and

[0045] FIG. 7 is a block diagram illustrating an example of a computer system according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0046] The present invention will be described in detail below with reference to the accompanying drawings. Repeated descriptions and descriptions of known functions and configurations that have been deemed to unnecessarily obscure the gist of the present invention will be omitted below. The embodiments of the present invention are intended to fully describe the present invention to a person having ordinary knowledge in the art to which the present invention pertains. Accordingly, the shapes, sizes, etc. of components in the drawings may be exaggerated in order to make the description clearer.

[0047] Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

[0048] FIG. 1 is a block diagram illustrating an example of an apparatus for detecting a disaster according to an embodiment of the present invention.

[0049] Referring to FIG. 1, the apparatus for detecting a disaster according to an embodiment of the present invention includes an image capture unit 110, a disaster detection unit 130, a disaster analysis unit 150, and a disaster alert unit 170.

[0050] The image capture unit 110 captures video using at least one camera. The camera may be installed in a CCTV or a movable drone. The video captured by the image capture unit 110 may he transmitted to the disaster detection unit 130 or a server at a remote site. Meanwhile, the image capture unit 110 usually monitors a traffic accident or a disaster by sequentially changing the orientation of the camera from a short distance to a long distance according to a predetermined order. However, when it receives a disaster alert signal and a camera control signal including information about a suspected disaster spot, at which it is suspected that a disaster occurs, from the server at the remote site or the disaster analysis unit 150, the image capture unit 110 adjusts the orientation of the camera to be directed at the suspected disaster spot, zooms in and captures an image thereof, and transmits the same to the server at the remote site or the disaster detection unit 130.

[0051] The disaster detection unit 130 detects the occurrence of a disaster by analyzing video data captured by the image capture unit 110 and various kinds of feature data extracted from the video data and records the result, thereby periodically generating a disaster log. The disaster detection unit 130 may directly receive the video captured by the image capture unit 110, or may receive the same via the server.

[0052] The disaster analysis unit 150 analyzes the disaster log, thereby determining whether a disaster occurs. The disaster occurrence information detected by the disaster detection unit 130 is not accurate. Occasionally, cases in which the occurrence of a disaster is erroneously detected infrequently arise due to various reasons, such as clouds, falls, waves, birds, and the like. In this case, the disaster analysis unit 150 serves to exclude such misidentified information from the disaster log and to confirm the actual occurrence of a wildfire. To this end, the disaster analysis unit 150 calculates a disaster occurrence probability value based on the disaster log.

[0053] FIG. 2 is a flowchart illustrating the disaster analysis procedure of a disaster analysis unit 150 according to an embodiment of the present invention.

[0054] Referring to FIG. 2, the disaster analysis procedure of the disaster analysis unit 150 according to the present embodiment receives a disaster log from the disaster detection unit 130 at step S210.

[0055] Also, the disaster analysis unit 150 calculates a disaster occurrence probability value based on the disaster log received from the disaster detection unit 130 at step S220.

[0056] Also, the disaster analysis unit 150 determines at step S230 whether the disaster occurrence probability value is greater than a first threshold. The disaster occurrence probability value has a value close to 0 at normal times, that is, when no disaster occurs. However, when the incidence of disasters is equal to or greater than a certain frequency, the disaster occurrence probability value exceeds the first threshold.

[0057] Also, when it is determined at step S230 that the disaster occurrence probability value is greater than the first threshold, the disaster analysis unit 150 generates a camera control signal by entering a camera control mode, and requests the image capture unit 110 to adjust the camera at step S240.

[0058] The camera control signal may include a disaster alert signal and information about the position at which it is suspected that a disaster occurs. When the camera.

[0059] control signal includes the disaster alert signal and the information about the position at which it is suspected that a disaster occurs, the image capture unit 110 rotates the camera to be directed at the position at which it is suspected that a disaster occurs, and may control the lens of the camera to zoom in on the corresponding position.

[0060] Also, the disaster analysis unit 150 determines at step S250 whether the disaster occurrence probability value is greater than a second threshold. When disaster occurrence information is continuously generated at a specific spot, the disaster occurrence probability value exceeds the second threshold.

[0061] Also, when it is determined at step S250 that the disaster occurrence probability value is greater than the second threshold, the disaster analysis unit 150 confirms the occurrence of a disaster and requests the disaster alert unit 170 to issue a disaster alert at step S260. Here, the disaster analysis unit 150 may request the disaster alert unit 170 to issue a disaster alert by generating a disaster alert request signal and transmitting the same to the disaster alert unit 170.

[0062] Referring again to FIG. 1, the disaster alert unit 170 warns of the disaster based on the disaster alert request received from the disaster analysis unit 150.

[0063] Hereinafter, a method in which the disaster detection unit 130 detects a disaster from the video captured by the image capture unit 110 will be described in more detail.

[0064] The image data captured by the image capture unit 110 may be video information formed of multiple consecutive image frames.

[0065] The disaster detection unit 130 is capable of detecting a disaster from a single image frame captured by the image capture unit 110. Here, a disaster may he detected through an image classification method using a convolutional neural network (CNN) model that is trained by classifying images into general images and disaster images. According to an embodiment, a Residual Network (ResNet) may be used as a more specific CNN model, but the CNN model is not limited thereto.

[0066] FIG. 3 illustrates an example in which a wildfire image is overlaid with the result of image classification for the wildfire image calculated as a probability map.

[0067] Referring to FIG. 3, it can be seen that the roughly estimated spot at which it is suspected that a wildfire has occurred may be checked using a single image frame based on a CNN model.

[0068] The method using classification applied to a single image may not always ensure the correct result. It is likely that an incorrect result may be reached due to various situations that are similar to and can be mistaken for a disaster. For example, in the case of a wildfire, because a captured image of the clouds in the sky (especially clouds hanging low over the mountain) looks very similar to a captured image of wildfire smoke, it is not easy to differentiate the two images from each other, and waves and waterfalls may be mistaken for white smoke due to the white foam thereof when viewed from a long distance. A single image of a snowdrift may be erroneously detected by being mistaken for white smoke. If classification is adjusted so as to enable such similar situations to also be minutely classified in a training process for image classification, the accuracy of detection of a disaster may be improved. That is, when the image classification method is changed from a method of classifying images into two types, including general images and wildfire smoke images, to a method of classifying images into various types of images, including a general image, a smoke image, a cloud image, a wave image, a waterfall image, a snow image, and the like, the accuracy of detection of a disaster may be improved.

[0069] Also, provision of video in place of a single image may be helpful to check the spread of objects related to a disaster in the video and to thereby determine whether a disaster occurs. For example, in the event of a wildfire, wildfire smoke fans out and looks like a rising 3D smoke shape. In contrast, snowdrifts are motionless, a waterfall moves downwards, and waves move so as to form a wavefront. Also, because the overall movement of clouds is linear, they spread differently from wildfire smoke.

[0070] In the case of sequential data such as video, it is difficult to apply an image-based neural network model, such as a convolutional neural network (CNN), thereto, and it is known that it is necessary to additionally use a recurrent neural network (RNN) model along therewith. However, cases in which, when a CNN has good performance, specific information is successfully detected with high accuracy by converting sequential data, such as voice or sound, into an image and applying a CNN model thereto, have been proposed. The present invention provides technology for detecting wildfire smoke using an image classification method by converting an image sequence of a certain section of transmitted video into a single image and by applying a CNN model to the single image.

[0071] The disaster detection unit 130 may perform disaster detection for a video section formed of n images (image sequence) from the (t+1*s)-th to (t+n*s)-th frames on a time basis in the video captured by the image capture unit 110, and may then perform disaster detection for a video section formed of n images (image sequence) from the (t+d+1*s)-th to (t+d+n*s)-th frames. Here, t denotes the start point of the video, s denotes the interval between the frames selected for disaster detection, d denotes the interval between the video sections selected for disaster detection, and n denotes the number of images. Here, because the target object (for example, a wildfire or flood) of a disaster progresses slowly, there may be little change between adjacent image frames in the transmitted video. Therefore, when the first image frame for forming a video section is selected, it may be necessary to select an image frame at an interval of s frames, rather than the image frame immediately following the first image frame.

[0072] The video section data formed of n consecutive images stored in the server may be difficult to directly use for detection of a disaster. There is no problem if a camera used for capturing is fixed, but because the image capture unit 110 may use a Pan-Tilt-Zoom (PTZ) CCTV camera or a camera installed in a drone, the camera may capture video while rotating, moving, or zooming in. In order to observe how a. disaster-related object (e.g., wildfire smoke, a river, or the like) progresses in each image within the video section, it is desirable that there be little difference in the position and size of the object. However, when the camera rotates, moves, or zooms in, the position of the object is not fixed in the video. In order to solve this problem, the movement of the camera used for capturing the video section is calculated by tracking the same, and inverse calculation is performed, whereby a video section in which the movement of the camera is minimized may be acquired. For example, the movement of the camera may be calculated using optical flow or Structure From Motion (SFM) technology, In the present specification, a video section in which the movement of the camera is minimized may be defined as a ‘fixed video section’, and because the edges of the images within the fixed video section are capable of being cut from the images of the original video section, the images may have a small size.

[0073] The disaster detection unit 130 may generate additional data based on the fixed video section received from the image capture unit 110 or the server, and using this additional data, the performance of disaster detection may be improved.

[0074] The disaster detection unit 130 may generate a motion map sequence by applying optical flow technology to a sequence of images in the fixed video section stored in the server. The motion map sequence indicates a sequence of motion maps. The motion map may be an image acquired by mapping a 2D motion vector field, which is the result of calculating a motion in all of the pixels between consecutive images by applying optical flow technology, onto the color space of Hue, Saturation, and Value (HSV). This motion map sequence enables (1) identifying only objects exhibiting a. distinct motion by removing a stationary background or a background moving at constant speed from a video section and (2) detecting a change in the internal structure its of each of the objects.

[0075] FIG. 4 illustrates an example in which a wildfire image is overlaid with a motion map.

[0076] Referring to FIG. 4, it can be seen that the motion vector of a background part in the image has little change, but that the motion vector inside the smoke greatly changes, whereby it may be observed that the smoke is spreading.

[0077] Referring again to FIG. 1, the disaster detection unit 130 may generate a feature map sequence by applying a convolutional neural network (CNN) for disaster image classification to a sequence of images within the video section received from the image capture unit 110 or the server. The feature map sequence indicates a sequence of feature maps. The feature map may be a value acquired by inputting images to the network of a CNN that is trained in advance using different existing datasets, such as an ImageNet.

[0078] The disaster detection unit 130 may use any one of the image sequence of the fixed video section, the motion map sequence, and the feature map sequence or a combination thereof as an input image sequence, that is, input data. This input image sequence may be represented using a 3D matrix of (N, K, M). In the 3D array, N denotes the size of the sequence, that is, the number of images, and each of the images may be represented as a 2D array of (K, M), in which case a single pixel or a pixel block (formed of P horizontal pixels and Q vertical pixels) may correspond to an element of the array.

[0079] The disaster detection unit 130 may use all of the images forming the input image sequence after enlarging or reducing the same to a certain size. According to an embodiment, when the size of each of the images forming the input image sequence is very large, it takes a lot of time to calculate disaster detection. Therefore, in order to reduce the calculation time or to reduce the amount of noise included in the image, the sizes of all of the images may be reduced to less than half.

[0080] Also, the disaster detection unit 130 may convert the 3D array of (N, K, M), which is an input image sequence, into a 2D array of (S, T), which is a single image. The 2D image converted from the 3D array may be referred to as an input flow map image. The method of convening a 3D array into a 2D array is not limited to a single method.

[0081] FIGS. 5A and 5B illustrate an example of image sequence conversion according to an embodiment of the present invention. FIG. 5A illustrates an example in which a 2D array of (K, M) is converted into a 1D array of (M×K), and FIG. 5B illustrates the example in which N sequences of ID arrays configured with (1, M×K) are converted into a 2D array configured with (N, M×K=P).

[0082] Referring to FIGS. 5A and 5B, when an input image sequence is configured with images having a size of (K, M), each of the images is converted into an image having a size of (1, M×K). and a sequence of N images, each having a size of (1, M×K), may be converted into an image having a size of (N, M×K). Here, when each image having a size of (K, M) is converted into an image having a size of (1, M×K), conversion may be performed using any of various methods, for example, by placing the following column below the first column or by placing the following row on the right side of the first row. Referring to FIG. 5B, the image sequence may be finally converted so as to have a size of (N, P). Here, N and P may vary depending on the input sequence.

[0083] FIG. 6 is a flowchart illustrating an operation of detecting a disaster in such a way that the disaster detection unit 130 converts an input image sequence according to an embodiment of the present invention. The following operations may be performed in the disaster detection unit 130 of the apparatus for detecting a disaster.

[0084] Referring to FIG. 6, the disaster detection unit 130 generates an input image sequence at step S610.

[0085] Here, the input image sequence may be generated using any one of the image sequence of the fixed video section captured by the image capture unit 110, a motion map sequence, and a feature map sequence, or a combination thereof. For example, the input image sequence may be represented using a 3D matrix of (N, K, M). In this 3D array, N denotes the size of the sequence, that is, the number of images, and each of the images may be represented as a 2D array of (K, M).

[0086] Also, the disaster detection unit 130 converts the array of the input image sequence at step S630. For example, the disaster detection unit 130 may convert each of the images having a size of (K, M), which forms the input image sequence represented as a 3D matrix of (N, K, M), into an image having a size of (1, M×K), and may convert the sequence of N images having a size of (1, M×K) into an image having a size of (N, M×K). Here, when each of the images having a size of (K, M) is converted into an image having a size of (1, M×K), conversion may be performed using any of various methods, for example, by placing the following column below the first column or by placing the following row on the right side of the first row.

[0087] FIG. 7 is a block diagram illustrating an example of a computer system according to an embodiment of the present invention.

[0088] Referring to FIG. 7, an embodiment of the present invention may be implemented in a computer system including a computer-readable recording medium. As shown in FIG. 7, the computer system 700 includes a processor 710, an input/output unit 730, and memory 750, and the input/output unit 730 communicates with an external server 770.

[0089] The processor 710 implements the process and/or method of detecting and analyzing a disaster in the disaster detection apparatus proposed in the present specification. Specifically, the processor 710 implements all of the operations of the disaster detection apparatus described in the embodiment disclosed in the present specification and performs all of the operations of the disaster detection method according to FIGS. 2 to 6.

[0090] For example, the processor 710 may generate a disaster log based on video captured by at least one camera, calculate a disaster occurrence probability value based on the disaster log, determine whether to enter a camera control mode based on the disaster occurrence probability value, and generate a camera control signal for controlling the camera depending on whether entry into the camera control mode is performed.

[0091] Here, the camera control signal may include a disaster alert signal and information about the position at which it is suspected that a disaster occurs, and when the camera control signal includes the disaster alert signal and the information about the position at which it is suspected that a disaster occurs, the processor 710 may rotate the camera to be directed at the position at which it is suspected that a disaster occurs and control the lens of the camera to zoom in on the corresponding position.

[0092] Here, the disaster log may include information about whether a disaster occurs on a time basis and information about the place at which a disaster has occurred.

[0093] Here, the processor 710 may generate the disaster log by detecting a disaster based on an image classification method using a convolutional neural network (CNN) model that is trained by classifying images into general images and disaster images,

[0094] Here, the processor 710 may perform disaster detection for a video section formed of n images (image sequence) from the (t+1*s)-th to (t+n*s)-th frames on a time basis in the video captured by the camera, and may then perform disaster detection for a video section formed of n images (image sequence) from the (t+d+1*s)-th to (t+d+n*s)-th frames, where t denotes the start position of the video, s denotes the interval between the frames selected for disaster detection, d denotes the interval between the video sections selected for disaster detection, and n denotes the number of images.

[0095] Here, the processor 710 may acquire a video section in which the movement of the camera is minimized by calculating the movement of the camera using optical flow or Structure From Motion (SFM) technology and performing inverse calculation, and may generate the disaster log based on the video section.

[0096] The input/output unit 730 is connected with the processor 710, and transmits and/or receives information to/from the server 770. For example, the input/output unit 730 may receive image data for detecting a disaster and/or various kinds of feature data extracted from the image data from the server 770. Conversely, the input/output unit 730 may transmit the captured image to the server 770.

[0097] The memory 750 may he any of various types of volatile or nonvolatile storage media. Here, the memory 750 may store at least one of the captured image, the camera control signal, and the disaster log,

[0098] According to the present invention, a disaster may he detected with high accuracy and at a low malfunction rate by converting sequential data provided in the form of video captured by a camera in real time into a single image and by applying an image classification method using a learning model of a neural network, such as a convolutional neural network (CNN), thereto.

[0099] Also, information is compressed by compressing image sequence information to into a single image, and a method through which time-series data can be processed using only a CNN, as in a recurrent neural network, is proposed, whereby it may be possible to detect a disaster by measuring information using a small number of variables.

[0100] Also, a disaster may he detected by processing a sequence having a different length, regardless of the length of the image sequence.

[0101] As described above, the method and apparatus for detecting a disaster based on images according to the present invention are not limitedly applied to the configurations and operations of the above-described embodiments, but all or some of the embodiments may be selectively combined and configured, so the embodiments may be modified in various ways.