METHOD OF MONITORING A BUILDING ELEMENT, VISUAL FEATURE, CONNECTION ELEMENT, MONITORING SYSTEM, AND COMPUTER PROGRAM PRODUCT

Abstract

A method of monitoring a building element includes using an image capturing device and a computing device, in attributing a visual feature to each one of a first and a second region of the element; taking a first plurality of images of the visual features by the capturing device at a first point of time; taking a second plurality of images of the visual features by the capturing device at a second point of time; causing the computing device to process the images of the first and the second plurality of images to derive at least one of distances between the two visual features and relative orientations of the two visual features within the first and the second plurality of images; and causing the computing device to deduce a state of the element from at least one of the difference between the distances and the difference between the relative orientations.

Claims

1-15. (canceled)

16. A monitoring system for monitoring a building element, wherein the building element includes a first region and a second region each having a visual feature attributed thereto such that a connection element comprises at least one of the attributed visual features, the monitoring system comprising: an image capturing device including an image sensor configured to generate images each comprising image data; a non-transitory computer-readable medium bearing a building element monitoring program; and a computing device the computing device in operable arrangement with the image capturing device such that image data captured by the image capturing device is communicated to the computing device, the processor in operable arrangement with the non-transitory computer-readable medium such that the processor is configured to execute the building element monitoring program; wherein, when the building element monitoring program is executed, the computing device is configured to: cause the image capturing device to take a first plurality of images of the visual features at a first point of time and to transmit the image data from the first plurality of images to the computing device, cause the image capturing device to take a second plurality of images of the visual features at a second point of time and to transmit the image data from the second plurality of images to the computing device, process the image data of the images of the first and the second plurality of images to derive at least one of distances between the visual features of the first and second regions and relative orientations of the visual features of the first and second regions within the first plurality of images and within the second plurality of images, determine a state of the building element from at least one of the difference between the distances between the visual features and the difference between the relative orientations of the visual features.

17. The monitoring system according to claim 16, further comprising a rod for allowing the image capturing device to approach visual features, the image capturing device mounted to the rod.

18. The monitoring system according to claim 17, wherein the rod includes a telescopic portion having a tip such that a length of the rod is adjustable, the image capturing device being mounted to the telescopic portion a predetermined distance from the tip.

19. The monitoring system according to claim 16, further comprising a vehicle which transports at least the image capturing device.

20. The monitoring system according to claim 19, wherein the vehicle is a land vehicle, a water vehicle, or an aerial vehicle.

21. The monitoring system according to claim 16, wherein the image capturing device comprises at least two image sensors spaced apart from each other.

22. The monitoring system according to claim 16, further comprising a distance sensor.

23. The monitoring system according to claim 22, wherein the distance sensor comprises at least one of a 3D-image capturing sensor, a time-of-flight sensor, and a LIDAR sensor.

24. The monitoring system according to claim 16, further comprising the connection element having at least one of the visual features.

25. The monitoring system according to claim 24, wherein the connection element comprises a unique identifier.

26. The monitoring system according to claim 24, wherein the connection element comprises a sensor.

27. The monitoring system according to claim 26, wherein the sensor comprises a sensor material.

28. The monitoring system according to claim 26, wherein the sensor is responsive to a tension, to a temperature, to a humidity, to an abrasion and/or to an oxidation.

29. The monitoring system according to claim 16, wherein, when the building element monitoring program is executed, the computing device is configured to trigger an alarm when the determined state of the building element meets a predetermined threshold.

30. The monitoring system according to claim 16, wherein the computing device comprises a distributed computing device including a computing cloud environment having at least one of a storage device for storing images generated by the image capturing device and a processor for executing the building element monitoring program.

Description

[0066] In the drawings:

[0067] FIG. 1 shows a monitoring system and a building element;

[0068] FIG. 2 shows another monitoring system comprising a rod and another building element;

[0069] FIG. 3 shows another monitoring system comprising an UAV and another building element, and

[0070] FIG. 4 shows a flow chart of the method.

[0071] Same reference signs are used for functionally equivalent elements in all figures.

[0072] FIG. 1 schematically shows a monitoring system 10, comprising an image capturing device 12 and a computing device 14. The computing device 14 comprises at least a microprocessing unit, a graphics processor, a readable storage device, and a display. A computer program product 15 is stored in the readable storage device and can be executed on the microprocessor. If executed on the computing device 14, step b to step e of the method described further below in regard to FIG. 4 will be executed.

[0073] The image capturing device 12 comprises an image sensor, for example a CCD or a CMOS image sensor. It is a part of the computing device 14. Therefore, the image capturing device 12, the computing device 14, and hence the monitoring system 10 form a single unit. The monitoring system 10 may be part of a portable computer, e. g. a tablet computer.

[0074] FIG. 1 further schematically shows a building element 16. It comprises a baseplate 18. The baseplate 18 is mounted on an underground 20. The underground 20 may be a part of a wall or the like. The baseplate 18 is fixedly connected to the underground 20 by two connection elements 22, 23. The connection elements 22, 23 are anchors.

[0075] The building element 16 is to be monitored for alterations. In particular, any movement of the baseplate 18 relative to one of the connection elements 22, 23, for example to the connection element 23, is to be detected.

[0076] For this, the connection element 23 is defined as a first region 24. The baseplate 18 is defined as a second region 26.

[0077] To each one of the first and the second regions 24, 26, a visual feature 28, 30 is attributed. In particular, the connection element 23 and, hence, the first region 24, has been premanufactured of the visual feature 28. The visual feature 30 attributed to the baseplate 18 and, hence, to the second region 26, is glued onto the baseplate 18.

[0078] The visual features 28, 30 comprise visual patterns. In particular, they comprise different two-dimensional ArUco patterns.

[0079] FIG. 2 shows another monitoring system 10. Unless stated otherwise, it equals the previously described monitoring system 10.

[0080] As one of the differences to the foregoing embodiment, the monitoring system 10 comprises two separate parts. In particular, the image capturing device 12 is separated from the computing device 14. Both elements 12, 14 are connected electronically to each other, so that image data captured by the image capturing device 12 may be communicated to the computing device 14. Also, the computing device 14 is enabled to control the image capturing device 12.

[0081] Both parts are mounted on a rod 32. The rod 32 is telescopic. At one end of the rod 32, it comprises a tip 34. The image capturing device 12 is mounted on the rod 32 distanced by a distance I from the end of the rod 32 and, respectively, of an end of the tip 34.

[0082] In the situation shown in FIG. 2 the tip 34 contacts the building element 16. Two visual features 28, 30 are attached to, in particular screwed onto, the building element 16. Alternatively, it is also possible to have the visual features 28, 30 attached to the building element 16 by way of any other suitable fixing method, e. g. by direct fastening, gluing, printing, engraving, or the like.

[0083] The image capturing device 12 according to this embodiment of the invention is a stereoscopic image capturing device. To this end, it comprises two image sensors 36, which are distanced from each other.

[0084] FIG. 3 shows another monitoring system 10. Unless stated otherwise, it also equals the previously described monitoring system 10 according to FIG. 1.

[0085] The monitoring system 10 is used for monitoring a building element 16, formed in this example as a bridge. The building element 16 is expected to get, for example, fissures or other alterations over time. Pairs of first and second regions 24, 26 and, respectively, 40, 42 are defined as regions on the building element 16, wherein each one of the first regions 24, 40 is on a left-hand side and each one of the second regions 26, 42 is on a right-hand side of an area supposedly prone to such alterations.

[0086] Attached to the building element 16 and within each one of the first and second regions 24, 26, 40, 42 are two pairs of visual features 28, 30 and 44, 46. Each one of the visual features 28, 30, 44, 46 is formed as self-adhesive sticker and comprises a unique pattern. The patterns are position markers and unique identifiers. All patterns of the visual features are two-dimensional ArUCo patterns.

[0087] As one of the differences, the monitoring system 10 according to this embodiment comprises an unmanned aerial vehicle, in particular a drone 38. The drone 38 is adapted to move autonomously. In particular, it is adapted to sequentially fly to a series of positions close to the visual features.

[0088] By consecutively and, preferably, fully-autonomously flying from one pair of visual features to the other and applying the method according to the invention and described in more detail in regard to FIG. 4 the building element 16, i. e. the bridge, is monitored for alterations, e. g. fissures, by the monitoring system 10.

[0089] FIG. 4 schematically shows a flow chart of a method 100, which will now be described in more detail. Wherever possible, reference is made to the reference signs used hitherto, in particular to the reference signs and the embodiment of the invention according to FIG. 3.

[0090] In order to execute the method, a monitoring system 10, in particular comprising an image capturing device 12 and a computing device 14, according to one of the previous embodiments of the invention may be used.

[0091] In a first step 100.a the visual features 28, 30, 44, 46, are attributed to each one of first regions 24, 40 and to the second regions 26, 42 of the building element 16. For this, the visual features 28, 30, 44, 46 being self-adhesive stickers are sticked onto the building element 16 within the respective first and second regions 24, 26, 40, 42.

[0092] During a first flight of the drone 38 and according to a second step 100.b of the method two first pluralities of images of the visual features 28, 30 and, respectively, of the visual features 44, 46 are taken by the image capturing device 12 at a first point of time. Each one of the first pluralities comprises several, for example 10, images. As the drone 38 flies along the building element 16, each one of these images is taken from another position of the monitoring system 10 and, in particular, of the image capturing device 12.

[0093] The first pluralities of images are then stored in a long-term storage; for example they may be uploaded to a cloud-based storage.

[0094] The following steps may be repeated, for example in regular time intervals, until the monitoring is to end or until an expected end of life of the building element 16:

[0095] During further flights of the drone 38 and according to a third step 100.c of the method two second pluralities of images of the visual features 28, 30 and, respectively, of the visual features 40, 42 are taken by the image capturing device 12 at second points of time. Each one of the second pluralities also comprises several, for example 10, images. As the drone 38 flies along the building element 16, each one of these images is taken from another position of the monitoring system 10 and, in particular, of the image capturing device 12. Analogously to the first pluralities also the second pluralities of images are stored in the long-term storage.

[0096] The autonomous flights of the drone 38 may be controlled by the computing device 14. Additionally or as an alternative, they may also be controlled or supervised at distance by a separate computing device, e. g. a computing device being part of a cloud-based system.

[0097] In another subsequent step 100.d the computer program product 15 causes the computing device 14 to process the images of the first and the second pluralities of images in order to derive at least one of the distances between the two pairs of visual features 28, 30 and, respectively, 44, 46 and the relative orientations of the two pairs of visual features 28, 30 and, respectively, 44, 46 within the first pluralities of images and within the second pluralities of images.

[0098] In a last step 100.e within each repetition of steps the computer program product 15 causes the computing device 14 to deduce states of the building element 16 for each of the pairs of first and second regions 24, 26 and, respectively, 40, 42 from at least one of the differences between the distances and the differences between the relative orientations. Thus, each of these states represents a healthiness of the respective part of the building element 16 at the respective second point of time. An overall state is then defined as critical if at least one of the states is critical, otherwise the overall state is defined as uncritical.

[0099] By storing these states or the overall state to the long-term storage a chronological history of the healthiness of the building element 16 to be monitored is built up over time.

[0100] If the overall state is critical, the computing device 14 triggers a predefined action. For example, it may order fully-automatically a service team to examine and repair the building element 16.

[0101] In order to process the images of the first and the second pluralities of images according to step 100.d, at first, all visual features, in particular their fiducials or patterns, are detected within the respective images. Secondly, reference points, preferably center points, of the visual features are computed.

[0102] Secondly, in particular, in a setup with the image capturing device comprising only a single image sensor, an additional IMU sensor can be used to determine a scale of the coordinate system. In case of multiple image sensors a calibrated image capturing device can be used to determine the scale.

[0103] Thirdly, corresponding detected points in the images are used to establish 2D to 3D point correspondences. Using at least three 2D to 3D point correspondences, relative poses (orientations and translations) can be estimated using a perspective-n-point algorithm. This process is repeated for each image taken at one point of time. In a case where a natural feature is used, the 3D point data of the visual feature as well as the pose of the image capturing device are simultaneously estimated using a structure-from-motion (SfM)-algorithm or a simultaneous localization and mapping- (SLAM-) algorithm.

[0104] All detected visual features in the images with their corresponding 3D coordinate estimations and the poses of the image capturing device are jointly optimized in a bundle adjustment process. During this optimization process, also the intrinsic calibration parameters of the image capturing device can be refined.

[0105] Fourthly, after the optimization of the 3D coordinate estimations and the poses of the image capturing device, relative distances between two visual features are computed and used for further comparisons with relative distances at different points of time.

REFERENCE SIGNS

[0106] 10 monitoring system

[0107] 12 image capturing device

[0108] 14 computing device

[0109] 15 computer program product

[0110] 16 building element

[0111] 18 baseplate

[0112] 20 underground

[0113] 22 connection element

[0114] 23 connection element

[0115] 24 first region

[0116] 26 second region

[0117] 28 visual feature

[0118] 30 visual feature

[0119] 32 rod

[0120] 34 tip

[0121] 36 image sensor

[0122] 38 drone

[0123] 40 first region

[0124] 42 second region

[0125] 44 visual feature

[0126] 46 visual feature

[0127] 100 method

[0128] 100.a step

[0129] 100.b step

[0130] 100.c step

[0131] 100.d step

[0132] 100.e step

[0133] I distance