Automotive construction machine and method for displaying the surroundings of an automotive construction machine

Abstract

The invention relates to an automotive construction machine with a transport device extending in the working direction beyond the machine frame, in particular a road milling machine or surface miner. Moreover, the invention relates to a method for displaying an image of the surroundings of an automotive construction machine. The construction machine according to the invention has an image display device for displaying a bird's eye view image of the surroundings of the construction machine, which image display device has a camera system with a plurality of cameras for recording individual overlapping image regions of the construction machine surroundings from different image recording positions, and an image processing system. The image processing system is configured in such a way that image details of the individual image regions are joined together to form a total image from a bird's eye view. The camera system has at least one camera which is arranged on the transport device in such a way that the viewing direction of the camera is directed in the direction of the machine frame, and at least one further camera which is arranged on the machine frame in such a way that the viewing direction of the camera is directed away from the machine frame. Using the cameras directed into and away from the machine frame, the machine operator obtains a complete surround view, which includes the region covered by the transport device in the working direction in front of the construction machine.

Claims

1. An automotive construction machine, comprising: a machine frame having a working direction; a working drum supported from the machine frame for removing ground material; a camera system configured to record image details of overlapping individual image regions of surroundings of the construction machine, the camera system including: at least one camera arranged such that a viewing direction of the camera is directed substantially toward the machine frame; and at least one additional camera arranged on the machine frame such that a viewing direction of the additional camera is directed substantially away from the machine frame; an image processing system configured such that the image details of the overlapping individual image regions are joined together to form a total image of the recorded surroundings of the construction machine from a bird's eye view; and a display unit configured to display the total image.

2. The construction machine of claim 1, wherein: the at least one camera arranged such that a viewing direction of the camera is directed substantially toward the machine frame is arranged on a transport conveyor extending forward or rearward beyond the machine frame to convey removed ground material.

3. The construction machine of claim 2, wherein: the at least one camera arranged on the transport conveyor is arranged such that at least a part of the individual image region recorded by the at least one camera arranged on the transport conveyor is located below the machine frame.

4. The construction machine of claim 3, wherein: the at least one camera arranged on the transport conveyor is arranged such that the part of the individual image region located below the machine frame is also located in a region of at least one front running gear.

5. The construction machine of claim 2, wherein: the transport conveyor further includes a jib having a lower side, and a conveyor belt arranged on the jib, and the at least one camera arranged on the transport conveyor is arranged on the lower side of the jib.

6. The construction machine of claim 5, wherein: the jib extends in a longitudinal direction; and the at least one camera arranged on the lower side of the jib has a camera axis extending substantially in the longitudinal direction of the jib.

7. The construction machine of claim 5, wherein: the machine frame has first and second opposite sides and a frame longitudinal axis; the jib has a jib longitudinal axis; and the at least one camera arranged on the transport conveyor includes two cameras arranged on the lower side of the jib, the two cameras being directed at angles to the jib longitudinal axis such that when the jib longitudinal axis is parallel with the frame longitudinal axis one of the two cameras is directed substantially toward the first side of the machine frame and the other of the two cameras is directed substantially toward the second side of the machine frame.

8. The construction machine of claim 1, wherein: the machine frame has a front side, a rear side, a left side, and a right side; and the at least one additional camera arranged on the machine frame includes: a left-hand camera arranged on the machine frame to record a left-hand image region; a right-hand camera arranged on the machine frame to record a right-hand image region; and a rear camera arranged on the machine frame to record a rear image region.

9. The construction machine of claim 8, wherein: the left-hand image region overlaps with an image region recorded by the at least one camera arranged on a transport conveyor extending forward or rearward beyond the machine frame to convey removed ground material; and the right-hand image region overlaps with an image region recorded by the at least one camera arranged on the transport conveyor.

10. The construction machine of claim 9, wherein: the image processing system is configured such that the individual image regions recorded by the left-hand camera, the right-hand camera and the at least one camera arranged on the transport conveyor are joined together to form the total image having a left portion located to the left of the machine frame, a right portion located to the right of the machine frame, a lower portion located below the machine frame and a front portion located in front of the machine frame.

11. The construction machine of claim 1, wherein: a transport conveyor is pivotally arranged on the machine frame to pivot about a pivot axis perpendicular to a plane of the machine frame; the machine further includes a position detector configured to detect a pivoting position of the transport conveyor relative to the machine frame; and the image processing system is further configured such that a course of stitching between the overlapping individual image regions is determined depending on the pivoting position of the transport conveyor.

12. The construction machine of claim 11, wherein: the position detector is configured to produce position data describing the pivoting position; and the image processing system is further configured such that the course of stitching is located within overlapping portions of the individual image regions.

13. The construction machine of claim 12, wherein: the image processing system is further configured such that contours of the individual image regions are determined based on the position data, and such that the overlapping portions of the individual image regions are determined based on the contours.

14. A method of displaying an image of the surroundings of an automotive construction machine, the construction machine including a machine frame and a working drum supported from the machine frame for removing ground material, the method comprising: recording overlapping individual image regions of the surroundings of the construction machine using a plurality of cameras including at least one camera directed toward the machine frame, and including at least one additional camera arranged on the machine frame and directed away from the machine frame; and forming a total image of the recorded surroundings of the construction machine from a bird's eye view by joining together image details of the overlapping individual image regions.

15. The method of claim 14, wherein: the recording step further comprises recording one of the individual image regions with the at least one camera directed toward the machine frame such that a part of the one image region is located below the machine frame.

16. The method of claim 15, wherein: the recording step further comprises the one image region located below the machine frame also being located in an area of at least one front running gear.

17. The method of claim 15, wherein: the recording step further comprises the at least one camera directed toward the machine frame being arranged on a lower side of a transport conveyor such that a camera axis of the at least one camera arranged on the transport conveyor runs in a longitudinal direction of the transport conveyor.

18. The method of claim 14, wherein the recording step further comprises: the at least one camera directed toward the machine frame being arranged on a transport conveyor and including two cameras arranged on a lower side of the transport conveyor and arranged at angles to a longitudinal direction of the transport conveyor such that a viewing direction of one of the two cameras is directed toward one side of the machine frame and a viewing direction of the other of the two cameras is directed towards another side of the machine frame.

19. The method of claim 14, wherein the forming step further comprises: detecting a pivoting position of a transport conveyor relative to the machine frame about an axis running perpendicular to a plane of the machine frame; and determining a course of stitching between image details of the overlapping individual image regions depending on the pivoting position of the transport conveyor.

20. The method of claim 19, wherein: the determining step further comprises determining the course of stitching such that the stitching is located within overlapping portions of the overlapping individual image regions.

21. The method of claim 20, wherein: the determining step further comprises determining contours of the individual image regions and determining the overlapping portions of the overlapping individual image regions based on the contours.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Embodiments of the invention will be described in detail below with reference to the drawings, in which:

(2) FIG. 1 is a side view of an embodiment of an automotive construction machine,

(3) FIG. 2 is a plan view of the construction machine of FIG. 1,

(4) FIG. 3 is a simplified schematic view of the components of the image display device of the construction machine,

(5) FIG. 4 shows an embodiment of the construction machine with a camera arranged on the transport device, the camera axis of which runs in the longitudinal direction of the transport device, the longitudinal axis of the transport device being located on the longitudinal axis of the machine frame,

(6) FIG. 5 shows the construction machine of FIG. 4, wherein the transport device is pivoted to one side of the machine frame,

(7) FIG. 6 shows the viewing angle of a camera arranged on the machine frame,

(8) FIG. 7 shows the viewing angle of a camera arranged on the transport device,

(9) FIG. 8 shows an embodiment of the construction machine with two cameras arranged on the transport device, the camera axes of which are located in a sloping manner with respect to the longitudinal axis of the transport device, the longitudinal axis of the transport device being located on the longitudinal axis of the machine frame, and

(10) FIG. 9 shows the construction machine of FIG. 8, wherein the transport device is pivoted to one side of the machine frame.

DETAILED DESCRIPTION

(11) As an example of an automotive construction machine, FIGS. 1 and 2 show, in a side and plan view, a road milling machine for milling road surfaces, this being a front loader road milling machine. The road milling machine may, however, also be a rear loader road milling machine.

(12) The construction machine has a machine frame 2 carried by a chassis 1, on which a working device 3 is arranged. The working device 3 has a working roller, this being a milling roller. The milling roller 4, only indicated in FIG. 1, is arranged in a milling roller housing 5. On the left-hand and right-hand side in the working direction A, the milling roller housing 5 is closed by an edge protector 6. The milling roller housing 5 is closed by a hold-down device on the front side in the working direction A, and by a stripper device on the rear side, which devices cannot be seen in FIG. 1. The control stand 7 with a control panel 8 for the machine operator is located on the machine frame above the milling roller housing 5. A display unit 9 with a display 9A is located on the control panel 8. The milled-off milling product is removed using a transport device 10 extending in the working direction A a long way forward beyond the machine frame 2. The transport device 10 has an elongate jib 11, which is pivotably fastened on the two sides on the machine frame 2 in the working direction in front of the control stand 7 about an axis running perpendicular to the plane of the machine frame. Moreover, the jib 11 is pivotable about an axis running parallel to the plane of the machine frame 2, so the jib can be moved up and down. A conveyor belt 12 for conveying the milled-off material is located on the upper side of the transport device 10.

(13) The construction machine has, in the working direction A, a front left-hand running gear 11A and a front right-hand running gear 11B and a rear left-hand running gear 12A and a rear right-hand running gear 12B, with which are associated a front left-hand and right-hand lifting device 13A,B in the working direction A and a rear left-hand and right-hand lifting device 14A,B, so that by retracting and extending the lifting devices, the height and incline of the machine frame 2 in relation to the ground surface B can be changed. The running gears of the construction machine can be both crawler tracks and wheels.

(14) FIG. 3 is a simplified schematic view of the components of the construction machine. The construction machine has a position detection device 15 for detecting the pivoting position of the transport device 10 in the horizontal plane, which device can be a component of a central control and computing unit 16 of the construction machine. The horizontal pivoting position of the transport device 10 can be detected using suitable sensors, for example angle transmitters, distance sensors etc., or can be determined from data obtained from the drive means, not shown, for pivoting the transport device, for example from the lifting position of the piston/cylinder arrangements.

(15) The position detection device 15 has a computing unit 15B receiving the data from the sensor 15A, which computing unit is configured in such a way that the angle in the horizontal plane between the longitudinal axis of the transport device 10 and the longitudinal axis of the machine frame 2 is determined from data, for example from the angle transmitter 15A (pivoting position). The position detection device 15 can also detect the angle in the vertical plane between the longitudinal axis of the transport device 10 and the longitudinal axis of the machine frame 2 (height).

(16) Moreover, the construction machine has an image display device 17 for displaying a bird's eye view image of the surroundings of the construction machine, which image display device has a camera system 18 and an image processing system 19. The image is displayed on the display 9A of the display unit 9 on the control stand 7 in the field of vision of the machine operator.

(17) The structure and function of the image display device 17 will be described in detail below. FIGS. 4 and 5 show a first embodiment and FIGS. 8 and 9 show a second embodiment of the construction machine, which only differ from one another with respect to the number of cameras in the camera system.

(18) The image display device 17 of the construction machine has a camera system 18 with a plurality of cameras 18A, 18B, 18C, 18D in the first embodiment or 18A, 18B, 18C, 18D, 18D in the second embodiment for recording individual image regions of the surroundings of the construction machine from different image recording positions. The cameras in each case record a specific image region of the ground surface B. The position and size of the image region recorded by the camera depend on the arrangement and orientation of the camera and the camera lens system, in particular the focal length of the lens of the camera. The cameras are arranged and configured in such a way that the image regions recorded by the cameras overlap.

(19) In the present embodiments, a lateral left-hand camera 18A is preferably located centrally between the left-hand front and rear corner regions of the machine frame 2, a lateral right-hand camera 18B is preferably located centrally between the right-hand front and rear corner regions of the machine frame 2, and a rear camera 18C is preferably located centrally between the rear corner regions of the machine frame. No cameras are located at the front corner regions of the machine frame.

(20) It is assumed that each camera 18A,B,C would record a substantially rectangular image detail of the ground surface B if the viewing direction of the camera (camera axis) was orthogonal to the ground surface. Because of the setting angle, a trapezoidal image region of the ground surface is recorded by the camera, however, i.e. the region of the site located outside the trapezoidal region is not detected by the camera.

(21) The cameras 18A,B,C are arranged on the machine frame 2 in such a way that the viewing directions 29 of the cameras are directed away from the machine frame 2. The camera axes 29 (viewing directions) of the lateral cameras 18A and 18B and of the rear camera 18C are preferably orthogonal to the longitudinal sides or the narrow side of the machine frame 2, the lateral cameras 18A, B recording a trapezoidal lateral image region 20A, 20B and the rear camera 18C recording a trapezoidal rear image region 20C. The camera axes 29 may, however, also be directed forward or rearward in order to detect a larger front or rear site region. The image regions can then, however, not be described by isosceles trapeziums when there is a parallel orientation of the machine frame to the ground surface B. FIG. 6 shows the viewing direction (camera axis 29) of a camera 18A,B,C on the machine frame 2, which is directed away from the machine frame 2.

(22) Instead of two cameras at the front corner regions of the machine frame, the camera system 18 of the first embodiment has a front camera 18D, which is arranged on the lower side of the jib 11 of the transport device 10. The camera 18D is fastened to a point on the longitudinal axis of the jib 11, which is located in a region of the central third between the two ends of the jib. The fastening point may, however, also be located in the region of the front or rear third of the jib. However, the fastening point should not be located in the region of the free end of the jib, as the free end of the jib is temporarily located over the loading area of the transport vehicle during operation of the construction machine. If the camera is fastened to the free end of the jib, a temporarily limited view is only to be expected, however, during the loading process but not while the construction machine is being moved.

(23) The front camera 18D is arranged on the lower side of the transport device 10 in such a way that the camera axis is located on the longitudinal axis of the transport device and that the viewing direction of the camera is directed in the direction of the machine frame 2. The front camera 18D is therefore a camera viewing rearward. FIG. 7 shows the viewing direction (camera axis 29) of the camera 18D on the transport device 10, which is directed in the direction of the machine frame 2.

(24) The angle between the camera axis 29 of the front camera 18D and the plane of the machine frame 2 is preferably such that the camera records a front image region in the working direction, which extends into a portion below the machine frame 2 (FIG. 7). The front camera 18D thus records a site portion 21, which could be detected completely from a bird's eye view if this region were not covered by the transport device 10, and a portion 22, which cannot be detected from the view from an observation point located above the machine frame 2. As the transport device 10 is height-adjustable, the angle between the camera axis 29 and the plane of the machine frame 2 also changes. The setting angle of the front camera 18D is therefore such that in a predetermined position of the transport device 10, which corresponds to the usual operating position, the two image portions 21, 22 include the desired areas (FIG. 7).

(25) FIG. 4 shows the construction machine, the longitudinal axis of the transport device 10 being located on the longitudinal axis of the machine frame 2, while FIG. 5 shows the construction machine, the transport device 10 being pivoted into the operating position, for example to the left-hand side. FIGS. 4 and 5 show that with the pivoting position of the transport device, the position of the image region 20D recorded by the front camera 18D also changes.

(26) The image region 20D of the front camera 18D overlaps with the image region 20A of the lateral left-hand camera 18A and the image region 20B of the lateral right-hand camera 18B, and the image region 20A of the left-hand camera 18A and the image region 20B of the right-hand camera 18B overlap with the image region 20C of the rear camera 18C. A front left-hand overlapping region 23A of the image regions 20A, 20D of the left-hand camera 18A and the front camera 18D and a front right-hand overlapping region 23B of the image regions 20B, 20D of the right-hand camera 18D and the front camera 18D are produced. Furthermore, a rear left-hand overlapping region 24A of the image regions 20A, 20C of the left-hand camera 18A and of the rear camera 18C and a rear right-hand overlapping region 24B of the image regions 20B, 20C of the right-hand camera 18B and of the rear camera 18C are produced.

(27) The image processing system 19 is preferably a data processing unit (CPU), on which a data processing program (software) runs. The position detection device 15 is connected to the image processing system 19 by a data line 25, so that the image processing system 19 can receive the position data of the position detection device.

(28) It is assumed for simplification that the machine frame 2 is located in an orientation parallel to the surface of the ground or that the incline of the machine frame in relation to the site surface is negligible.

(29) The image processing system 19 determines, depending on the pivoting position of the transport device 10, the position of the contours, shown by dotted lines, of the image region 20D of the front camera 18D, the position of which changes upon a pivoting movement of the transport device. Moreover, the image processing system 19 determines the position of the contours, shown by dotted lines, of the image regions 20A, 20B, 20C of the left-hand and right-hand camera 18A, 18B and of the rear camera 18C. The position of the straight contours can be described in a Cartesian X/Y coordinate system, which is located on the surface of the ground B, by the starting and end points thereof. The coordinates of these points are calculated in the image processing system 19 according to an algorithm, which takes into account the pivoting position of the transport device.

(30) Once the position of the image regions 20A, B,C,D, which are described by the coordinates of the starting and end points of the contours, is known, the image processing system 19 determines the position of the overlapping regions 23A, 23B, 24A, 24B of the mutually adjoining image regions 20A,B,C,D. The overlapping regions 23A, 23B, 24A, 24B may, for example, be determined by calculating an intersection of adjacent image regions 20A,B,C,D.

(31) The image processing system 19 is configured in such a way that image details are determined in the individual image regions 20A, 20B, i.e. suitable details from the recorded image regions 20A,B,C,D of the ground surface B, which can be completely joined together to form a total image 27 from a bird's eye view. When joining together the image details, stitching is produced, i.e. boundary lines between the image details or intersecting lines of the image regions. The course of the stitching depends on the position of the overlapping trapezoidal image details.

(32) The image processing system, in the X/Y coordinate system, determines the coordinates of the starting and end points of the contours of the front left-hand overlapping region 23A, the front right-hand overlapping region 23B, the rear left-hand overlapping region 24A and the rear right-hand overlapping region 24B. The overlapping regions are shown by hatched areas in FIGS. 4 and 5. The image processing system 19 thereupon determines the course of the stitching, the following criteria being taken into account.

(33) The course of the rear stitching in the working direction is determined by the image processing system 19 in such a way that the stitching between the left-hand and right-hand image regions 20A, 20B and the rear image region 20C runs within the rear left-hand and right-hand overlapping regions 24A, 24B. The image processing system 19 determines the course of the stitching in such a way that the stitching stretches from the rear, left-hand and right-hand corner points of the machine frame 2 up to the intersection points of the contours of the adjacent image regions 20A, 20C or 20B, 20C. A rear left-hand stitching 25A and a rear right-hand stitching 25B are produced, which are shown in FIG. 5.

(34) For the front image region 20D and the left-hand and right-hand image regions 20A, 20B, the image processing system 19, depending on the pivoting position of the transport device 20, determines a front left-hand stitching 25C running within the front left-hand overlapping region 23A and a front right-hand stitching 25D running within the front right-hand overlapping region 23B.

(35) The front left-hand stitching 25C extends between the intersection points S.sub.1 and S.sub.2 of the contours of the front image region 20D and of the left-hand image region 20A, and the front right-hand stitching 25D extends between the intersection points S.sub.3 and S.sub.4 of the contours of the front image region 20D and of the right-hand image region 20B. Located between the two stitchings 25C and 25D is the image detail 26C of the front image region 20D, which is joined together with the associated image details 26A and 26B of the left-hand and right-hand image regions 20A, 20B. For this purpose, the outer portions of the front image region 20D are cut off along the stitching 25C and 25D and the inner portions of the left-hand and right-hand image regions 20A, 20B are cut off along the stitching 25C and 25D.

(36) The position of the starting and end points of the stitching are determined in the X/Y coordinate system by coordinates of the starting and end points of the stitching.

(37) The image processing system 19 produces a total image 27 in a bird's eye view by joining together all the image details. It is shown that the total image 27 comprises a portion 27A located on the left-hand longitudinal side of the machine frame 2 and a portion 27B located on the right-hand longitudinal side of the machine frame 2. Moreover, the total image 27 comprises a portion 27C located below the machine frame 2, a portion 20D located in front of the machine frame 2 and a portion 20E located behind the machine frame 2.

(38) The stitching for the front image region 20D and the left-hand and right-hand image regions 20A, 20B can also be determined in such a way that individual stitching or all the stitching does not run within the contours or overlapping regions but along them. For example, instead of the left-hand front stitching 25C within the overlapping region 25C, a stitching 25C running between the intersection points S.sub.1 and S.sub.2 along the outer contour of the front image region 20D can be determined. For example, instead of the right-hand front stitching 25D within the overlapping region 25D, a stitching 25D running between the intersection points S.sub.3 and S.sub.4 along the outer contour of the right-hand image region 20B can be determined, for example. It is also possible to determine the stitching for the rear image region and the left-hand and right-hand image regions taking into account other criteria.

(39) FIGS. 8 and 9 show an alternative embodiment, which differs from the embodiment described with reference to FIGS. 4 and 5 in that the camera system 18 has two front cameras 18D and 18D, which are arranged on the transport device 10. The mutually corresponding parts are therefore provided with the same reference numerals. The left-hand front camera 18D is fastened to the lower side of the left-hand longitudinal side and the right-hand front camera 18D is fastened to the lower side of the right-hand longitudinal side of the jib 11 in such a way that the viewing direction of the left-hand camera 18D is directed in the direction of the left-hand side and the viewing direction of the right-hand camera 18D is directed in the direction of the right-hand side of the machine frame 2 when the longitudinal axis of the transport device 10 is on the longitudinal axis of the machine frame 2 (FIG. 8). The two cameras 18D, 18D are fastened to a portion of the jib 11, which is located in the central third of the jib. The image regions 20D and 20D of the front left-hand and right-hand cameras 18D, 18D overlap in a front central overlapping region 28. As in the embodiment with only one camera, the image processing system 19 determines the contours of the front left-hand overlapping region 23A, of the front right-hand overlapping region 23B, of the rear left-hand overlapping region 24A and of the rear right-hand overlapping region 24B. Moreover, the image processing system 19 determines the contours of the front central overlapping region 28. Thereupon, as in the embodiment with only one camera, the rear left-hand and right-hand stitching 24A, 24B is determined. The stitching for the front image regions 20D and 20D of the two cameras 18D, 18D and the left-hand and right-hand image regions 20A, 20B are determined according to the same criteria in such a way that the individual image regions can be joined together to form a complete total image.

(40) The image processing system 19 can determine the course of the stitching and therefore the position and size of the image details, which are joined together to form the total image, in such a way that as large a region as possible of the surroundings is captured completely with all the cameras. For example, the view forward can be extended at the cost of the view rearward or vice versa. However, the view to the left-hand side can also be extended at the cost of the view to the right-hand side or vice versa.

(41) The total image can then be displayed in full size on the display 9 of the display unit 8. However, it is also possible to display only one detail of the total image, for example a rectangular image detail.