METHOD FOR SECURING A STARTING MOVEMENT OF A SEMI-AUTOMATED OR FULLY AUTOMATED VEHICLE

Abstract

A method for securing a starting movement of a semi-automated or fully automated vehicle, the vehicle including at least one imaging sensor, which is configured to capture images of a close-up range of the vehicle. A recognition of objects in the close-up range of the vehicle is carried out. A starting movement of the vehicle is prevented if an object has been recognized.

Claims

1-14. (canceled)

15. A method for securing a starting movement of a semi-automated or fully automated vehicle, the vehicle including at least one imaging sensor, the imaging sensor including a near-field camera which is configured to capture images of a close-up range of the vehicle, the method comprising the following steps: carrying out a recognition of objects in the close-up range of the vehicle prior to a starting movement of the vehicle when the vehicle is stationary by carrying out a combination of at least two of the following steps: a) capturing an instantaneous image of the close-up range of the vehicle using the imaging sensor and recognizing objects located in the close-up range of the vehicle by comparing the instantaneous image with a previously stored comparison image, an at risk object in the close-up range being deduced when the comparison indicates that a structure of the comparison image is not completely visible, but is covered in parts by an object, and/or b) temporally successively capturing at least two instantaneous images using the imaging sensor and recognizing moving objects in the close-up range of the vehicle by comparing the temporally successively captured images and evaluating optical flow or another differential method, and/or c) changing a spatial position of the imaging sensor and capturing at least two images at respectively different spatial positions of the imaging sensor and recognizing objects in the close-up range of the vehicle by analyzing changes, by using a structure-from-motion analysis, of the images captured at the different positions of the imaging sensor; and preventing the starting movement of the vehicle when at least one object has been recognized.

16. The method as recited in claim 15, wherein, if no object has been recognized by the steps a) and/or b) and/or c) when vehicle is stationary, a slow starting movement of the vehicle takes place, images of the close-up range of the vehicle being captured using the imaging sensor and recognizing objects in the close-up range of the vehicle by an analysis, by a structure-from-motion analysis, of image changes of the images captured during the slow starting movement of the vehicle, a further travel of the vehicle being stopped when an object has been recognized.

17. The method as recited in claim 16, wherein the vehicle initially drives in a direction opposite an unparking direction and subsequently drives slowly in the unparking direction, the unparking direction being a driving direction in which a regular unparking process of the vehicle takes place.

18. The method as recited in claim 15, wherein the structure of the comparison image includes a vehicle contour and an object is recognized when the vehicle contour is not completely visible in the instantaneous image, but is covered in parts by an object.

19. The method as recited in claim 15, wherein the structure of the comparison image includes a ground structure, and an object is recognized when the ground structure is not completely visible in the instantaneous image, but is covered in parts by an object.

20. The method as recited in claim 15, wherein objects are additionally recognized in the images captured by the imaging sensor using methods of digital image processing, and are classified based on their shape.

21. The method as recited in claim 15, wherein one or multiple distances between the vehicle and objects in the surroundings of the vehicle is also determined using at least one distance sensor situated at the vehicle, which is configured as an ultrasonic sensor and/or a radar sensor and/or a LIDAR sensor.

22. A device configured to secure a starting movement of a semi-automated or fully automated vehicle, comprising: at least one imaging sensor including a camera; an evaluation unit; and a memory unit; wherein the evaluation unit is configured to evaluate images captured by the imaging sensor of a close-up range of a vehicle and, based on the evaluation, to recognize objects in the close-up range of the vehicle, and: a) the evaluation unit is configured to recognize objects located in the close-up range of the vehicle by comparing an instantaneous image with a comparison image previously stored and present in the memory unit, an object in the close-up range being deduced when the comparison indicates that a structure of the comparison image is not completely visible, but is covered at least in parts by an object; and/or b) the evaluation unit is configured recognize moving objects located in the close-up range of the vehicle by comparing at least two temporally successively captured images by evaluating optical flow; and/or c) the imaging sensor is mountable at the vehicle in such a way that the imaging sensor is adjustable between at least two spatial positions, and the evaluation unit is configured to recognize objects in the close-up range of the vehicle, to compare images captured at different spatial positions and, based on the comparison, to recognize an object in the close-up range of the vehicle; wherein the evaluation unit is configured to output a signal, as a result of which the starting movement of the vehicle is prevented when an object has been recognized by the evaluation unit.

23. The device as recited in claim 22, wherein the evaluation unit is configured to, if no object has been recognized by the steps a) and/or b) and/or c) when the vehicle is stationary, to output a signal for a slow starting movement of the vehicle, and to recognize during the slow starting movement objects in the close-up range of the vehicle by a structure-from-motion analysis of image changes of images captured during the slow starting movement of the vehicle, the evaluation unit being configured to output a signal, as a result of which a further travel of the vehicle is stopped when an object has been recognized by the evaluation unit.

24. The device as recited in claim 22, wherein the imaging sensor is mounted at a folding exterior mirror of a vehicle.

25. The device as recited in claim 22, wherein the device further includes at least one distance sensor, the at least one distance sensor being an ultrasonic sensor and/or a radar sensor and/or a LIDAR sensor.

26. A vehicle configured for a semi-automated or fully automated operation, the vehicle comprising: a device configured to secure a starting movement of a semi-automated or fully automated vehicle, including: at least one imaging sensor including a camera; an evaluation unit; and a memory unit; wherein the evaluation unit is configured to evaluate images captured by the imaging sensor of a close-up range of the vehicle and, based on the evaluation, to recognize objects in the close-up range of the vehicle, and: a) the evaluation unit is configured to recognize objects located in the close-up range of the vehicle by comparing an instantaneous image with a comparison image previously stored and present in the memory unit, an object in the close-up range being deduced when the comparison indicates that a structure of the comparison image is not completely visible, but is covered at least in parts by an object; and/or b) the evaluation unit is configured recognize moving objects located in the close-up range of the vehicle by comparing at least two temporally successively captured images by evaluating optical flow; and/or c) the imaging sensor is mountable at the vehicle in such a way that the imaging sensor is adjustable between at least two spatial positions, and the evaluation unit is configured to recognize objects in the close-up range of the vehicle, to compare images captured at different spatial positions and, based on the comparison, to recognize an object in the close-up range of the vehicle; wherein the evaluation unit is configured to output a signal, as a result of which the starting movement of the vehicle is prevented when an object has been recognized by the evaluation unit.

27. A non-transitory machine-readable memory medium on which is stored a computer program for securing a starting movement of a semi-automated or fully automated vehicle, the vehicle including at least one imaging sensor, the imaging sensor including a near-field camera which is configured to capture images of a close-up range of the vehicle, the computer program, when executed by a computer, causing the computer to perform the following steps: carrying out a recognition of objects in the close-up range of the vehicle prior to a starting movement of the vehicle when the vehicle is stationary by carrying out a combination of at least two of the following steps: a) capturing an instantaneous image of the close-up range of the vehicle using the imaging sensor and recognizing objects located in the close-up range of the vehicle by comparing the instantaneous image with a previously stored comparison image, an at risk object in the close-up range being deduced when the comparison indicates that a structure of the comparison image is not completely visible, but is covered in parts by an object, and/or b) temporally successively capturing at least two instantaneous images using the imaging sensor and recognizing moving objects in the close-up range of the vehicle by comparing the temporally successively captured images and evaluating optical flow or another differential method, and/or c) changing a spatial position of the imaging sensor and capturing at least two images at respectively different spatial positions of the imaging sensor and recognizing objects in the close-up range of the vehicle by analyzing changes, by using a structure-from-motion analysis, of the images captured at the different positions of the imaging sensor; and preventing the starting movement of the vehicle when at least one object has been recognized.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0051] An exemplary embodiment of the present invention is represented in the figures and explained in greater detail below.

[0052] FIGS. 1a) and 1b) schematically represent an automatic valet parking (AVP) process, the situation during the drop-off of the vehicle being represented in FIG. 1a) and the pick-up process being represented in FIG. 1b).

[0053] FIGS. 2a)-2b) show a vehicle according to one possible example embodiment of the present invention. FIG. 2a) shows a side view of the vehicle, FIG. 2b) shows a top view of the vehicle.

[0054] FIGS. 3a), 3b), and 3c) each show a fold-out side mirror including a camera of a vehicle designed according to one embodiment of the present invention in three different positions.

[0055] FIGS. 4a) and 4b) show an object recognition during a slow starting movement of a vehicle according to one preferred embodiment of the present invention.

[0056] FIG. 5 schematically shows a device according to one exemplary embodiment of the present invention.

[0057] FIG. 6 shows a flowchart of a method according to one possible embodiment of the present invention.

[0058] The figures represent only schematically the subject matter of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0059] FIGS. 1a) and 1b) schematically show one example of an application of the method according to the present invention for a parking surroundings 20 of an AVP system, in this example, an AVP system being depicted, in which the control unit of the AVP trip is located in the parking surroundings. The following description is based on this system distribution with the control unit in the parking surroundings. Parking surroundings 20 are constituted here as a parking deck of a parking garage. Parking surroundings 20 include a multitude of parking spaces 22, 23, a central unit 17 and a multitude of stationary sensor units (not shown). Central unit 17 includes a computing device and a communication unit, neither of which is shown separately for the sake of clarity. In one further exemplary embodiment of an AVP system, the control unit for the AVP trip may be located in the vehicle itself.

[0060] Parking surroundings 20 are configured to guide a vehicle 10 starting from a drop-off area 18 to one of parking spaces 22′ and to park it there. If vehicle 10 is needed again, it is guided by parking surroundings 20 to a pick-up area 19, which in this example, is identical to drop-off area 18. In the process, a driver 15 of vehicle 10 leaves vehicle 10 after parking in drop-off area 18 and later receives it again in pick-up area 19.

[0061] For an automatic trip within parking surroundings 20, vehicle 10 includes a control unit, which activates corresponding actuators for the longitudinal guidance and transverse guidance of vehicle 10. The control unit in this case follows a trajectory 40 provided to it by parking surroundings 20. For example, this trajectory 40 is conveyed to vehicle 10 with the aid of a communication unit of central unit 17.

[0062] The present invention is applicable, in particular, when vehicle 10, as shown in FIG. 1b) receives the command to leave parking space 22′ and to drive autonomously, i.e., without a person being situated in vehicle 10, to pick-up area 19. It is possible that in the interim an at risk object 30 has appeared, in particular, in the areas between vehicle 10 and the adjacent vehicles. Vehicle 10 therefore includes at least one camera, preferably, however, multiple cameras.

[0063] One exemplary embodiment of a vehicle 10 is represented in FIGS. 2a) and 2b). In this case, FIG. 2a) shows a side view of vehicle 10 and FIG. 2b) shows a top view from above. Vehicle 10 includes a camera 12 at each of its side mirrors 11. The field of view or the detection area of left-side camera 12 is schematically represented by area 13 and encompasses a close-up range of vehicle 10 on its left side. The right-side field of view is similarly formed and not otherwise shown. The field of view of camera 12 encompasses the respective side of vehicle 10, in particular, vehicle contour 110 on the respective side, as well as roadway 32 on the respective side.

[0064] An at risk object 30, in this example, a child, is located at the left side of vehicle 10 in the area between the doors.

[0065] Camera 12 now captures an instantaneous image of the side area of vehicle 10. The captured instantaneous image is compared with a previously stored comparison image. If the comparison indicates that a structure of the comparison image is not completely visible, but is covered in parts by an object, an at risk object 30 in the detection area 13 of camera 12 is deduced. For this purpose, it may be checked, for example, whether vehicle contour 110 is interrupted or concealed as compared to the comparison image. If a concealment or interruption of vehicle contour 110 is recognized in the instantaneous image, an object 30 is deduced.

[0066] The recognition of an object 30 by checking vehicle contour 110 may fail, however, if object 30 exhibits a distance to vehicle 10 so that vehicle contour 110 is not concealed or interrupted, or if object 30 is located, for example, in the area of the front or rear fender of vehicle 10. A ground structure of the comparison image may therefore alternatively or in addition be compared with the instantaneous image and an object may be recognized if the ground structure in the instantaneous image is not completely visible, but is covered at least in parts by object 30.

[0067] The described recognition of an object 30 by comparing an image captured instantaneously with the aid of camera 12 with a previously stored comparison image may encounter problems if the lighting conditions between the recording of the instantaneous image and the comparison image differ drastically, or if, for example, light reflections or strong shadow casting occurs. In these cases, objects may be recognized even though in reality no object 30 is present (false positive).

[0068] In order to also detect moving objects and to recognize them as such, at least two images may additionally or alternatively be captured temporally in quick succession by camera 12. A moving object in the close-up range of vehicle 10 or in detection area 13 of camera 12 may then be detected by comparing the temporally successively captured images by evaluating with the aid of a differential method or preferably with the aid of optical flow. If an optical flow is established, an at least partially movable object 30 may then be deduced. Problems in the image comparison as a result of changing lighting conditions present no greater problem in this evaluation, since the images to be compared are recorded preferably at a short temporal interval (typically approximately 1 s).

[0069] In order to further improve the object recognition, a change of the spatial position of cameras 12 and capture of at least two images at respectively different spatial positions of a camera 12 may then alternatively or additionally be carried out. The size and position of objects 30 in the close-up range of vehicle 10 is determined by an SfM analysis of the images captured at the different positions of the imaging sensor. The change of the spatial position of camera 12 takes place when vehicle 10 is otherwise at a standstill, for example, by moving side mirror 11, at which camera 12 is situated, automatically into another position, for example, unfolded or folded. One possible embodiment is represented in greater detail in FIG. 3.

[0070] A top view of a detail of a vehicle 10 is represented in FIG. 3a) in the area of a side mirror 11 in a first position of the side mirror. Side mirror 11 is provided in its operating position for observing a rearward area located next to and behind vehicle 10. A camera 12 is situated in or directly at side mirror 11, which is provided, for example, for capturing images for producing a representation of vehicle 10 and of its surroundings. For this purpose, camera 12 has a detection area 13 in the operating position of side mirror 11, which covers a surroundings area extending directly next to vehicle 10 and in the longitudinal direction of vehicle 10 in front of and behind side mirror 11.

[0071] Side mirror 11 is automatically foldable from the operating position shown in FIG. 3a) into the second and third positions in the direction of a vehicle body shown in greater detail in FIGS. 3b) and 3c).

[0072] Camera 12 in the exemplary embodiment shown is situated at a side of side mirror 11 facing away from the vehicle body, in particular, at a transition from an underside of the same to a side surface facing away from the vehicle body. In this case, camera 12 in the completely or partially folded rest positions of side mirror 11 according to FIGS. 3b) and 3c) has a detection area 13′ and 13″, which covers a surroundings area extending in the longitudinal direction of vehicle 10 behind side mirror 11, which overlaps at least partially with detection area 13.

[0073] At least one image of the vehicle surroundings may then be captured by camera 12 at each of the positions or at least two of the positions according to FIGS. 3a), 3b) and 3c). By comparing the images, it is possible to recognize an object 30 in an overlapping area of the images.

[0074] FIG. 4a) shows the situation in which vehicle 10 has received the prompt to start up and no object 30 has been recognized in the close-up range of vehicle 10 during the implementation of a) through c). In one preferred embodiment of the present invention, vehicle 10 may now, as indicated in FIG. 4b), commence a starting movement at a very slow speed. During this slow starting movement, images of the close-up range of vehicle 10 continue to be captured with cameras 12. Object 30 may now be recognized by a structure-from-motion analysis of image changes of the images captured during the slow starting movement of vehicle 10, a further travel of vehicle 10 being stopped once object 30 has been recognized. The vehicle may, in particular, initially drive in the direction opposite the regular unparking direction, in this example, forward, and subsequently drive in the unparking direction, in this example, backward. In the process, vehicle 10 accelerates preferably at less than 0.1 m/s.sup.2 and drives a distance of approximately 5 cm to 10 cm in the respective direction. In the process, at least one first image may be captured during the forward travel, for example. A second image may, for example, be captured when the vehicle has already traveled a particular distance backward during the subsequent backward travel, so that a preferably large spatial distance exists between the positions at which the images are captured. This improves the accuracy of a structure-from-motion analysis.

[0075] FIG. 5 schematically shows a device 2 according to one exemplary embodiment of the present invention. The device includes at least one imaging sensor, in this example, a camera 12, an evaluation unit 25 and a memory unit 27. Evaluation unit 25 is designed to evaluate images captured by camera 12 of a close-up range of vehicle 10 and, based on the evaluation, to recognize objects 30 in a close-up range of vehicle 10, the evaluation unit being configured to recognize objects 30 located in the close-up range of vehicle 10 by comparing an instantaneous image with a comparison image previously stored and present in memory unit 27, an at risk object 30 in the close-up range being deduced if the comparison indicates that a structure of the comparison image is not completely visible, but is covered in parts by an object 30.

[0076] Evaluation unit 25 is further configured to recognize moving objects 30 located in the close-up range of vehicle 10 by comparing at least two temporally successively captured images, by evaluating the optical flow or by applying other differential methods.

[0077] Camera 12 is mountable at a vehicle 10 in such a way that the camera is adjustable between at least two spatial positions, for example, as shown in FIG. 3, by mounting camera 12 in a movable side mirror 11 of vehicle 10. Evaluation unit 25 is also designed, for recognizing objects in a close-up range of the vehicle, to compare images that have been captured at the various spatial positions of camera 12 and, based on the comparison, to recognize an object 30 in the close-up range of the vehicle.

[0078] Evaluation unit 25 is configured to output a signal, which may be further processed, for example, by a control unit of vehicle 10, as a result of which a starting movement is prevented if an object 30 has been recognized by evaluation unit 25.

[0079] The sequence of a method carried out according to one exemplary embodiment of the present invention is represented in FIG. 6. In step 301, a parked vehicle receives the prompt to begin moving. The recognition of objects in a close-up range of the vehicle is subsequently started. For this purpose, an instantaneous image of the close-up range of the vehicle is captured in step 302 with the aid of an imaging sensor of the vehicle. An object in the close-up range of the vehicle is recognized by comparing the captured instantaneous image with a previously stored comparison image, an in particular at risk object in the close-up range being deduced if the comparison indicates that a structure of the comparison image is not completely visible, but is covered in parts by an object. In addition or alternatively, a moving object is recognized in step 303 by the imaging sensor capturing temporally successively at least two images of the close-up range of the vehicle. A moving object in the close-up range of the vehicle is recognized by comparing the temporally successively captured images and by evaluating the optical flow. In addition or alternatively, a spatial position of the imaging sensor relative to the vehicle otherwise at a standstill is changed in step 304, and at least two images are captured at respectively different positions of the imaging sensor. A stationary object in the close-up range of the vehicle is recognized by comparing the images captured at the different positions of the imaging sensors.

[0080] If no object has been recognized in steps 302, 303 and 304, step 305 may optionally be carried out. In this step, a slow starting movement of the vehicle takes place, in particular, at an acceleration of less than 0.1 m/s.sup.2, images of the close-up range of the vehicle being captured with the aid of the imaging sensor. A recognition of non-moving objects in the close-up range of the vehicle takes place by a structure-from-motion analysis of image changes of the images captured during the slow starting movement of the vehicle.

[0081] In step 306, it is queried whether an object has been recognized in one of the previous steps. If an object has been recognized, the starting movement of the vehicle is prevented or a further travel of the vehicle is immediately stopped.

[0082] The present invention is not restricted to the exemplary embodiments described herein and to the aspects highlighted therein. Instead, a multitude of modifications is possible within the scope of the present invention, which fall within the practice routine to those skilled in the art in view of the disclosure herein.