METHOD FOR OPERATING AN AUTOMATICALLY DRIVEN, DRIVERLESS MOTOR VEHICLE AND MONITORING SYSTEM

Abstract

A method for operating an automatically guided, driverless motor vehicle (1), in particular a passenger car, wherein sensor data detected by environmental sensors, including at least one camera (6) of the motor vehicle (1) is evaluated with respect to objects (17) to be taken into account in trajectory planning, which objects (17) can be classified as an obstacle or not an obstacle by means of at least one classifier evaluating the associated sensor data, wherein in case of an object (17) that cannot be classified or cannot be classified with sufficient certainty as an obstacle or not an obstacle and/or in case of at least one object (17) preventing the further drive of the motor vehicle (1) to a current destination, at least one camera image (18) of the respective object (17) is taken with at least one of the at least one camera (6), transmitted to a portable mobile communication device (11) carried along by a user (12) of the motor vehicle (1), and displayed there; an input, classifying the object (17) as an obstacle or not an obstacle, of a user (12) is accepted as classification information; the classification information is transmitted back to the motor vehicle (1) and is taken into account in the further automatic guidance of the motor vehicle (1).

Claims

1. A method, comprising: operating an automatically guided, driverless motor vehicle, in particular a passenger car, wherein sensor data detected by environmental sensors, including at least one camera, of the motor vehicle are evaluated with respect to objects to be taken into account in trajectory planning, which objects can be classified as an obstacle or not an obstacle by means of at least one classifier evaluating the associated sensor data, wherein in case of an object that cannot be classified at all or cannot be classified with sufficient certainty as an obstacle or not an obstacle and/or in case of at least one object preventing the further drive of the motor vehicle to a current destination, at least one camera image of the respective object is taken with at least one of the at least one camera, transmitted to a portable mobile communication device carried along by a user of the motor vehicle, and displayed there; an input, classifying the object as an obstacle or not an obstacle, of a user is accepted as classification information; the classification information is transmitted back to the motor vehicle and is taken into account in the further automatic guidance of the motor vehicle.

2. The method according to claim 1 wherein cellular telephone, in particular a smartphone, is used as the mobile communication device.

3. The method according to claim 1 wherein that the transmission of the camera image and the classification information takes place via a stationary server device outside the motor vehicle and/or at least partially via a mobile radio network and/or a WLAN.

4. The method according to claim 3 wherein digital map data describing the area, in which the motor vehicle is moved, is stored on the server device, wherein the objects classified by a user are stored in the map data with the assigned classification information.

5. The method according to claim 4 wherein classification information of several users for the same object is evaluated statistically to total classification information.

6. The method according to claim 4 wherein the digital map data is transmitted by the server device to the motor vehicle and are taken into account in the automatic guidance of the motor vehicle.

7. The method according to claim 1 wherein at least one piece of additional information, in particular a piece of additional information transmitted with the camera image, regarding the object to be classified is displayed on the mobile communication device.

8. The method according to claim 7 wherein a map of the current environment of the motor vehicle, in which map the location of the object to be classified and/or of additional, already classified objects and/or of the motor vehicle is shown, and/or dimensions, determined by environmental sensors, of the object are used as additional information.

9. The method according to claim 1 wherein the camera image is post-processed, in particular deskewed, for better readability prior to being displayed.

10. The method according to claim 1 wherein a software application is used to execute the steps to be performed on the part of the mobile communication device.

11. The method according to claim 1 wherein cameras covering the environment of the motor vehicle in the entire angular range, in particular at least four cameras provided with a wide-angle lens, are used.

12. A monitoring system, comprising: an automatically guidable motor vehicle, in particular a passenger car, with environmental sensors, including at least one camera, the detected sensor data of which can be evaluated on the part of a control device with respect to objects to be taken into account in trajectory planning, which objects can be classified as an obstacle or not an obstacle by means of at least one classifier evaluating the associated sensor data, and a mobile communication device, designed to execute a method according to claim 1.

Description

[0022] Additional advantages and individual details of the present invention result from the exemplary embodiments described below as well as the drawings. The following is shown:

[0023] FIG. 1 a monitoring system according to the invention,

[0024] FIG. 2 a flow chart of an exemplary embodiment of the method according to the invention,

[0025] FIG. 3 an exemplary driving situation,

[0026] FIG. 4 an exemplary illustration on a mobile communication device, and

[0027] FIG. 5 the statistical evaluation of classification information.

[0028] FIG. 1 shows a monitoring system according to the invention in the form of a schematic diagram, which monitoring system in the present case is used in a parking environment, in particular a parking garage, in which motor vehicles that are appropriately designed can drive driverlessly and in a manner completely automatically guided to their respective parking space and parked there or removed from there. Naturally, the method can however also be transferred to other areas that can be driven by motor vehicles fully automatically.

[0029] A motor vehicle 1 is located in the parking environment 2, which is only schematically depicted here, and is operated fully automatically without a driver, for which purpose an appropriate vehicle system 3 with a control device 4 is used, which is also designed to execute the steps on the motor vehicle side of the method according to the invention and to be explained below. The motor vehicle 1 comprises a plurality of environmental sensors, of which only the radar sensors 5 and six cameras 6 are shown in this case for the sake of clarity, which cameras are respectively provided with a wide-angle lens and can thus cover the environment of the motor vehicle over the entire 360° angular range.

[0030] The sensor data of the environmental sensors are, for example, fused by the control device 4 in an environmental model of the motor vehicle 1, wherein the environmental model in the present case describes different objects in the environment of the motor vehicle 1 by means of the sensor data. By means of classifiers implemented by software in the control device 4, it can be determined whether the objects are considered to be an obstacle or can be driven over or under. Naturally, the classifiers can also deliver additional, more accurate information regarding the objects, such as whether the object is another parked motor vehicle, a traffic sign, a pillar, and the like, if appropriate classifiers are provided.

[0031] Based on the classified object in the environmental model, an automatic guidance of the motor vehicle 1 in the parking environment 2 is possible. This is furthermore supported by digital map data transmitted by a server device 7, which also belongs to the monitoring system, serves as back end server, and is assigned as infrastructure device to the parking environment 2, said map data being held available there in a storage device 8. Between the server device 7 and the motor vehicle 1 therefore exists a communication link 9, in the present case a WLAN connection, which is established upon the motor vehicle 1 driving into the parking environment 2. Naturally, other types of communication links are also conceivable for the communication link 9.

[0032] A communication link 10 to a mobile communication device 11 of a user 12 of the motor vehicle 1 can also be established via the server device 7 or directly by the motor vehicle 1, wherein the communication link 10 is in the present case established via a mobile radio network. Accordingly, the communication device 11, which is carried by the user 12, is a cellular telephone, specifically a smartphone. This smartphone, as is basically known, comprises a display 14, on which information can be displayed. On the mobile communication device 11, a software application 13 (“app”) is installed, which is also used within the framework of the method according to the invention and now to be explained.

[0033] In the now illustrated exemplary embodiment of the method according to the invention, cf. FIG. 2, it is assumed that any communication between the motor vehicle 1 and the mobile communication device 11 takes place via the server device 7, i.e. the communication links 9 and 10, for which purpose the motor vehicle 1 can communicate to the server device 7 a communication address of the communication device 11 in the mobile radio network, for example, i.e. specifically a telephone number, but also possibly a user ID that was generated when the connection to the motor vehicle 1 was established.

[0034] It should furthermore be noted in advance that upon motor vehicle 1 driving into the parking environment 2, the server device 7 provided the motor vehicle 1 with the digital map data, available in the server device 8, of the parking environment 2 in order to support the fully automatic, driverless operation of the motor vehicle 1 by means of the vehicle system 3.

[0035] FIG. 2 now shows a flow chart of an exemplary embodiment of the method according to the invention. During the fully automatic operation of the motor vehicle 1, it is continuously checked in a step S1 whether an inquiry criterion for the inquiry regarding an object to be classified by the user 12 is fulfilled. In doing so, two inquiry criteria are taken into consideration in the present case. The first inquiry criterion checks whether an object detected by the environmental sensors cannot be classified or cannot be classified reliably enough, i.e. whether none of the classifiers in the control device 4 delivered a result that classified the object as an obstacle or not an obstacle, or whether a possible classification is not reliable enough, i.e. a threshold value for a reliability check is fallen below. The second inquiry criterion is always triggered if at least one object blocks the further drive of the motor vehicle 1 such that a current destination would no longer be reachable. In this case, the object to be classified by a user, the at least one object impeding the further drive, in particular the at least one object that was classified with the lowest certainty and/or that as the only object or as one of a few objects impedes the further drive. Naturally, appropriate improvements of the inquiry criterion are provided so that not all objects in the environment of the motor vehicle 1 are selected as objects to be classified by the user.

[0036] FIG. 3 shows a driving situation of the motor vehicle 1 within the parking environment 2 by way of example. Seen can be parking spaces separated by columns 15 and at least partially occupied by additional motor vehicles 16. In front of the motor vehicle 1 is located an object 17 that cannot be classified, in the present case a heap of leaves. Since the heap of leaves 17 cannot be classified by the control device 4 of the motor vehicle 1, it is unclear whether it constitutes an obstacle for the further drive. The first inquiry criterion in step S1 is thus fulfilled. In the present case, the second inquiry criterion in step S1 is however also fulfilled by way of example since the motor vehicle 1 obviously cannot classify the object 17 as being able to be driven over without danger, i.e. as “not an obstacle,” and cannot therefore continue its route since the object 17 cannot be driven around on the left or on the right side. In other words, a “getting stuck” of the motor vehicle is imminent since no alternate trajectory can be calculated.

[0037] If the at least one inquiry criterion in step S1 is fulfilled, communication information regarding the object 17 to be classified by the user 12 is compiled in a step S2. This communication information includes on the one hand a camera image of the camera 6 directed toward the front, which image was deskewed appropriately for better readability since the properties of the respective wide-angle lens are known. Additional information regarding the object 17, which were also collected by the environmental sensors, are furthermore added, such as dimensions of the object 17; finally, a map of the current environment of the motor vehicle 1 is added to the communication information as additional information, which map shows the position of the motor vehicle 1, of the object 17, as well as of additional objects. This communication information is initially transmitted in step S2 to the server device 7, which appropriately stores a reference to the inquiry and uses the communication link 10 to furthermore transmit the communication information to the mobile communication device 11, which takes place in step S3.

[0038] In step S4, the software application 13 accepts the communication information, outputs optical and/or acoustic and/or tactile reference information to the user 12 so that said user is made aware of the inquiry, and generates an illustration for the display 14 as shown by way of example in FIG. 4. Obviously, a large part of the illustration on the display 14 is taken up by the camera image 18. Shown smaller are the map 19 as well as other additional information 20, such as the dimensions. Below the camera image 18 are illustrated two operating elements 21, 22, by means of which the user can classify the object 17, which can also be highlighted in the camera image 18. If the user 12 operates operating element 21, a classification information is generated that describes the object 17 as an obstacle; if said user operates operating element 22, a classification information is generated that classifies the object 17 as “not an obstacle.” Naturally, additional functions can also be provided by the software application 13, such as an enlargement of the map 19, a display of other additional information, and the like.

[0039] In step S5, the input of the user 12 is therefore accepted via the operating elements 21, 22 and the classification information is generated, which is initially transmitted back to the server device 7 in step S6. In the server device 7, the classification information is stored in a step S7 with the associated object 17 in the storage device 8, is further evaluated, which is to be explained further with reference to FIG. 5, and is transmitted to the motor vehicle 1 via the communication link 9.

[0040] In the motor vehicle 1, the classification information is appropriately taken into account for the further automatic guidance of the motor vehicle 1 in step S8. In the case illustrated in FIG. 3, the driver will presumably mark the heap of leaves, i.e. the object 17, as not an obstacle, i.e. being able to be driven over, so that the automatic driving operation can be continued.

[0041] FIG. 5 finally shows a flow chart of the statistical evaluation of classification information, which concerns the same object 17 to be classified, in the server device 7. In a step S9, it is checked there whether a sufficient quantity of classification information of different users 12 is available in order to be able to carry out a statistical evaluation. If this is the case, it is checked in step S10 which classification most of the users 12 have carried out and whether the percentage of the users 12 who have carried out this classification exceeds a threshold value. If the latter is the case, total classification information 23 is generated, stored in the digital map data in a manner associated with the object 17, and thus made available to other motor vehicles.

[0042] In this case, step S3 in FIG. 1 can be extended in that if total classification information 23 regarding the object 17 is already available, this information is immediately transmitted back to the motor vehicle 1 as response so that the inquiry of the user is no longer necessary and it can be continued directly with step S8.