Method and Device for Multi-Sensor Data Fusion For Automated and Autonomous Vehicles

Abstract

A method estimates a course of a roadway in a vicinity of a vehicle based on a state function describing the course of the roadway, wherein the state function includes a clothoid spline. The method includes providing ambient measured data describing the course of the roadway at a current position of the vehicle, where the ambient measured data includes a polynomial function. The method also includes transforming the state function and the ambient measured data into a common coordinate system, and checking the ambient measured data for an error. If no error is detected, then the state function is adapted based on the ambient measured data in the common coordinate system. If an error is detected, then the error is stored.

Claims

1.-16. (canceled)

17. A computer-implemented method for estimating a course of a roadway in a vicinity of a vehicle on the basis of a state function describing the course of the roadway, said state function encompassing a clothoid spline, said method comprising: providing ambient measured data describing the course of the roadway at a current position of the vehicle, said ambient measured data including at least one polynomial function; transforming the state function and the ambient measured data into a common coordinate system; and checking the ambient measured data for an error; wherein if no error is detected, adapting the state function on the basis of the ambient measured data in the common coordinate system, and if an error is detected, storing the error.

18. The computer-implemented method as claimed in claim 17, wherein checking the ambient measured data for the error comprises ascertaining a deviation between a value ascertained in a prediction step of a Kalman filter and the ambient data, and subsequently comparing the deviation with a predetermined threshold value.

19. The computer-implemented method as claimed in claim 18, wherein the stored error is provided to a receiving unit arranged external to the vehicle.

20. The computer-implemented method as claimed in claim 19, wherein the stored error is utilized for correcting a roadway marking and/or correcting map data and/or improving a lane-marking recognition function and/or improving an autonomous driving function.

21. The computer-implemented method as claimed in claim 17, wherein the stored error is sent to a receiving unit arranged outside the vehicle.

22. The computer-implemented method as claimed in claim 17, wherein the stored error is utilized for a purpose of correcting a roadway marking and/or for a purpose of correcting map data and/or for a purpose of improving a lane-marking recognition function and/or for a purpose of improving an autonomous driving function.

23. The computer-implemented method as claimed in claim 17, wherein the ambient measured data are captured by at least one camera.

24. The computer-implemented method as claimed in claim 17, wherein the ambient measured data are provided from a map.

25. The computer-implemented method as claimed in claim 17, wherein first ambient measured data are captured by at least one camera and second ambient measured data are provided from a map.

26. The computer-implemented method as claimed in claim 17, wherein the common coordinate system includes spatial coordinates.

27. The computer-implemented method as claimed in claim 17, wherein adapting the state function includes determining sample points, wherein in the case of several sample points in each instance there is a constant curve length between adjacent sample points.

28. The computer-implemented method as claimed in claim 17, wherein the common coordinate system includes curvature values.

29. The computer-implemented method as claimed in claim 17, wherein adapting the state function to the ambient measured data includes providing for adapting at least one curvature value of the state function to at least one other curvature value of the ambient measured data.

30. The computer-implemented method as claimed in claim 17, wherein the state function characterizes a course of a roadway marking of a roadway.

31. The computer-implemented method as claimed in claim 30, further including ascertaining a course of a lane center by transforming the state function.

32. The computer-implemented method as claimed in claim 17, further including providing measured validation data and validating the estimate of the course of the roadway based at least in part on the validation data.

33. A device for estimating the course of a roadway in the vicinity of the vehicle, comprising: one or more interfaces which have been designed to capture ambient measured data; a processing unit configured to execute the computer-implemented method as claimed in claim 17.

34. A vehicle including the device as claimed in claim 33.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Embodiments will be elucidated in more detail below with reference to the accompanying figures. Shown are:

[0042] FIG. 1 a schematic representation of a multi-lane highway with a vehicle located thereon.

[0043] FIG. 2 an example of a clothoid spline with n clothoid segments.

[0044] FIG. 3 a schematic flowchart of the computer-implemented method for estimating the course of a roadway in the vicinity of the vehicle.

[0045] FIG. 4 a flowchart of a multi-sensor data fusion.

[0046] FIG. 5 a schematic representation of a method for sampling the clothoid spline and the ambient measured data.

[0047] FIG. 6 a graphical comparison of two approximation methods for approximating the clothoid spline.

[0048] FIG. 7 a graphical comparison of two sampling methods.

[0049] FIG. 8 a schematic illustration of a roadway marked by roadway markings.

[0050] FIG. 9 a representation of the map data and the vehicle located on the roadway.

[0051] FIG. 10 a schematic illustration of the vehicle with a device for estimating the course of a roadway.

DETAILED DESCRIPTION

[0052] Various embodiments will now be described more fully with reference to the accompanying drawings in which a few embodiments have been represented. In the figures, the thickness dimensions of lines, layers and/or regions may have been represented in exaggerated manner for the sake of clarity.

[0053] In the field of concepts for autonomous driving, it can be of great significance for the control of vehicles driving autonomously, for instance, to estimate the environment and the course of a roadway. For instance, the vehicle can be controlled in such a manner that it regulates steering, acceleration, speed and other vehicle parameters or driving parameters autonomously, without involvement of a driver, in such a way that, for instance, the vehicle follows the estimated course of a roadway safely and/or can avoid obstacles.

[0054] In FIG. 1 a schematic roadway course is shown, by way of example, of the roadway 14 on which the vehicle 11 is located and/or moving. Embodiments described herein may have been designed to estimate the course of the roadway in two translatory directions—that is to say, in two dimensions. FIG. 1 shows the roadway from a plan view in the two dimensions. The vehicle 11 is located in the middle one of three lanes of the roadway 14, which, for example, are permitted to be driven along in one or both directions. In some advantageous embodiments described herein, a course of the roadway 14 or of the lane, for instance, can be estimated. In a broader sense, by the “course of the roadway” in the present disclosure both the course of the roadway and the course of the lane may be meant.

[0055] A center of the lane and/or a course of the center of the lane may have been denoted by a center line 12. Usually, however, highways do not have a center line 12, so the lane center is not “visible”. The individual lanes are typically each delimited by roadway markings such as a boundary line 13.

[0056] Usual methods from the prior art determine the course of a roadway, for instance, by reference to a roadway model based on polynomials or polynomial splines that may result directly, for instance, from photographs taken by the camera or from the digital map.

[0057] In the course of construction planning, however, by reason of ride comfort and for safety reasons the roadway 14 is typically designed in accordance with a spline (clothoid spline) of contiguous clothoid segments. One advantage of such a design of the roadway 14 is a linear change in a curvature of the roadway, so that curves can be driven through with a high degree of ride comfort and, for the driver of vehicles traveling non-autonomously, can be assessed well. By reason of a style of construction based on the clothoid spline, it may prove advantageous to assume a roadway model based on a clothoid spline when estimating the course of a roadway. For example, the accuracy and reliability of the estimation can consequently be enhanced.

[0058] The roadway model of the roadway 14, which is based on a clothoid spline 20, is shown by way of example in FIG. 2. The clothoid spline 20 comprises the clothoid segments 21 and the transition-points 22 between two clothoid segments 21. The clothoid segments 21 correspond to parts of clothoid functions.

[0059] The clothoid functions, or curvature values c(s) of the clothoid functions, can be mapped as a function of the arc length s in accordance with:

[00001]$\begin{array}{cc}c\left(s\right)=c_0+c_{1,m}*s& \left(1\right)\end{array}$

[0060] Here, c.sub.0 stands for an initial curvature and c.sub.1,m stands for a constant change in curvature of the mth clothoid segment 41 with arc length s. By reference to (1) it can be seen that the curvature value c(s) of the clothoid function changes linearly with c.sub.1,m.

[0061] In the case of the clothoid spline 20 represented in FIG. 2, an orientation θ.sub.0 and a y-axis intercept y.sub.0 of a clothoid segment 21 can additionally be specified. In general, each of the clothoid segments 21 of the clothoid spline 20 can accordingly be completely characterized by the parameters c.sub.0, c.sub.1, θ.sub.0 and y.sub.0. For the clothoid spline 20 with n (n∈) clothoid segments 21, this would mean that the clothoid spline 20 can be described by 4*n parameters.

[0062] The clothoid spline 20 shown in FIG. 2 satisfies connection conditions between the clothoid segments 21 that can reduce a number of parameters necessary for describing the clothoid spline 20, so that a computational effort for determining the clothoid spline 20 is reduced. The clothoid spline 20 is, for instance, continuous, differentiable at the transition-points 22, and the curvature values of two consecutive clothoid segments 21 are equivalent at the transition-points 22. Under the stated connection conditions, the number of parameters necessary for describing the clothoid spline 20 can be reduced to n+3. The clothoid spline 20 can therefore be completely characterized by a reduced parameter set (c.

[0063] The clothoid spline 20 is able to map the course of the roadway 14, for instance both in front of and behind the vehicle 11, to a good approximation. For a determination of the clothoid spline 20, ambient measured data pertaining to several—where appropriate, diverse—data sources or sensors are drawn upon. Particularly in the case where use is made of different types of sensor, there may be a need to determine the clothoid spline 20 in such a manner that the ambient measured data pertaining to the diverse sensors also enter into the determination of the clothoid spline 20. This need can be met, for instance, by embodiments of the computer-implemented method 30 represented in FIG. 3 for estimating the course of a roadway.

[0064] The computer-implemented method 30 for estimating the course of a roadway in the vicinity of the vehicle is based on a determination of the state function describing the course of the roadway. The state function encompasses the clothoid spline 20.

[0065] The computer-implemented method 30 includes the provision 31 of ambient measured data that describe the course of the roadway at a current position of the vehicle 11. The ambient measured data include the at least one polynomial function. The computer-implemented method 30 includes, moreover, the transforming 32 of the state function and of the ambient measured data into the common coordinate system and the adapting 33 of the state function on the basis of the ambient measured data in the common coordinate system.

[0066] The ambient measured data pertaining to the vehicle's own sensorics, such as the camera or several cameras, are able to describe, as in a present embodiment, a course of the boundary lines 13 or the course of the lane center 12 of the roadway by reference to a polynomial function or a polynomial spline.

[0067] The HD map provided by a map manufacturer can, in addition, be drawn upon for the purpose of estimating the course of a roadway. On the basis of waypoints of the roadway, which can be taken from the HD map, ambient measured data can be ascertained. The waypoints may denote, for instance, the center of the lane. The map data resulting from this usually describe the course of the roadway by reference to a further polynomial spline.

[0068] Real-time filters, such as a Kalman filter, it may, for instance, be possible to update the state function, or, to be more exact, the clothoid spline 20, with the ambient measured data, provided that the state function and the ambient measured data are located in the common coordinate system. For instance, the state function is present, as in representation (1), in a state space that encompasses curvature values. The ambient measured data are present, for instance, in a measurement space with spatial coordinates. The state function can be mapped by transforming 32, for instance in the common coordinate system that may encompass curvature values or spatial coordinates.

[0069] In the common coordinate system the Kalman filter can, for instance, draw upon the ambient measured data in order to perform an adapting 33 of the clothoid spline 20 to the ambient measured data.

[0070] The computer-implemented method 30 can, for instance, describe an individual recursion step of the recursive method. The recursive method comprises, for instance, several consecutive recursion steps which each serve for estimating the course of a roadway. The recursive method can be realized, for instance, by the Kalman filter or by other real-time filters. In some embodiments, the Kalman filter has proved advantageous.

[0071] Input variables of the individual recursion steps are the state function from one of the preceding recursion steps and the ambient measured data that can be captured by the provision 31 of the ambient measured data by means of the at least one camera and the HD map.

[0072] A capturing and the transforming 32 of the state function of a preceding recursion step characterize a first phase of the Kalman filter. This phase is designated as prediction.

[0073] At the time of the prediction carried out by the Kalman filter, an estimate for a current state of the clothoid spline 20 can be ascertained. For this purpose, the clothoid spline can be subjected to state dynamics. The state dynamics are determined, for instance, by a motion of the vehicle 11. For instance, if the vehicle 11 is moving along the roadway 14 the clothoid spline 20 can be extended in front of the vehicle 11 and shortened behind the vehicle 11.

[0074] Since the clothoid spline 20 is usually composed of individual segments 21, the clothoid spline 20 cannot, for instance, be continuously extended or shortened in each recursion step but only in the case where the vehicle 11 is traveling over one of the transition-points 22.

[0075] In each recursion step, a check can additionally take place 33a as to whether there is an error in the underlying ambient data. The check can be made by a deviation between a value of the state function ascertained in a prediction step of a Kalman filter and the current ambient data being ascertained and by this deviation being compared with a predetermined threshold value. If the ascertained deviation is greater than a predetermined (upper) threshold value or less than a predetermined (lower) threshold value, there is an error. This error can subsequently be stored 33b and not used any further for the state function—that is to say, not in a phase designated as innovation. The storing may also include a marking of a faulty lane marking corresponding to the error. Furthermore, the error and corresponding data, such as the marked faulty lane marking, can subsequently be sent to a receiving unit arranged outside the vehicle. Depending upon the error, the transmitted data can easily be utilized, for instance for a highways authority for the purpose of correcting the faulty lane marking on a highway and/or to for a vehicle manufacturer for the purpose of improving a camera/lidar lane-marking detection function, in particular for a manual labeling for the purpose of improving learning algorithms. Furthermore, the transmitted data can be utilized to decide whether an autonomous driving mode is to be deactivated at this faulty lane marking. Consequently these data can be utilized to enhance the reliability and safety of autonomous driving functions. Furthermore, the sent data can be checked, in order to detect whether the error has arisen by reason of faulty data pertaining to a camera or by reason of faulty data pertaining to a digital map.

[0076] If there is no error, the following described process takes place.

[0077] The state function of the preceding recursion step is updated by adapting 33 to the ambient measured data in each recursion step. The adapting 33 corresponds to a second phase of the Kalman filter, designated as innovation. For the adapting 33, the state function is compared, for instance, with the ambient measured data. For instance, for this purpose individual values of the state function are compared with other values of the ambient measured data. Depending upon the measurement space, these values may be, for instance, spatial coordinates or curvature values of the state function and of the ambient measured data. Usually, the values of the state function have a fuzziness or, to be more exact, the values of the ambient measured data have a measurement uncertainty, which in each instance can be represented by a probability distribution such as, for example, a Gaussian distribution. In the second phase of the Kalman filter, the adapting 33 of the state function can be undertaken with the state function and with the ambient measured data as input variables. A weighting of the input variables at the time of the adapting 33 of the state function may be dependent on the measurement uncertainties, or on the fuzziness. The lower the measurement uncertainty of the ambient measured data, the more intensely can, for instance, the state function ascertained at the time of the prediction be approximated to the ambient measured data. The greater the measurement uncertainty of the ambient measured data 14, the lower the weighting of the ambient measured data can be at the time of the adapting 33. The fuzziness of the state function that is present as input variable is based, for instance, on mean values and specifications relating to the scatter (for example, covariances) of the parameters of the clothoid spline 20. The mean values and covariances can be ascertained from the parameters of preceding recursion steps. The fuzziness of the state function may have been determined, for instance, by the covariances. In the case of high covariance, the state function acquired at the time of the prediction may have been given a low weighting.

[0078] In contrast, in the case of a low covariance the state function is weighted heavily. Accordingly, depending upon fuzziness and measurement uncertainty, an updated state function or clothoid spline 20 results from the ambient measured data and from the state function serving as input variable. In a subsequent recursion step, the last-updated state function can be drawn upon once again for the prediction.

[0079] The ambient data pertaining to individual sensors (sensorics provided by the map manufacturer, and the vehicle's own sensorics) cannot determine the course of a roadway exactly. The ambient measured data pertaining to the sensorics may, for example, be noisy or faulty in some cases. The data fusion of the ambient data pertaining to several different types of sensor (data sources), on the other hand, can guarantee a robust, highly available estimation, largely unaffected by error, of the course of a roadway. This concept of multi-sensor data fusion is illustrated in FIG. 3.

[0080] At the time of the multi-sensor data fusion, data—in particular, ambient measured data—pertaining to a plurality of sensors can be merged to form the roadway model. In the embodiment represented in FIG. 4, the ambient measured data can be provided by means of the camera 41 and the HD map 43.

[0081] The camera 41 may, for instance, have been attached to the vehicle and directed in the direction of travel. As already mentioned, from the photographs taken by the camera 41 the roadway markings, the roadway boundaries and the course thereof can be represented approximately in the form of one or more consecutive polynomial functions by means of an image-processing application. Typically, a range or a measurement radius of the camera 41 is limited by obstacles or by an optical range of the camera 41.

[0082] The HD map 43 is typically based on output variables pertaining to sensorics provided by a map manufacturer 42. By interpolation of the waypoints that can be taken from the HD map, one or more connected polynomial functions can be determined, in order to describe the course of a roadway by approximation. The HD map 43 or corresponding map data may be present, for instance, on a storage medium which has been fitted to the vehicle 11.

[0083] Alternatively or additionally, the HD map 43 or the map data could be communicated from a transmitter to a receiving module of the vehicle 11 or of a device for estimating the course of a roadway.

[0084] By adapting 33 of the state function in the second phase of the Kalman filter, the ambient measured data pertaining to the HD map 43 and to the camera 41 enter into—for instance, in accordance with the multi-sensor data fusion 40—the estimating of the course of the roadway or of the roadway model 45 which may be represented by the state function or by the clothoid spline 20. Even though the embodiment described herein makes provision for a use of a single camera, further embodiments may include a plurality of cameras 41 which, where appropriate, may have been oriented in various directions.

[0085] With the Kalman filter, the clothoid spline 20 can, for instance amongst other things, be adapted to ambient data pertaining to the camera 41. In the following, the data fusion 44 of the ambient measured data pertaining to the camera 41 will be considered in more detail with the aid of the curves 20 and 50 represented in FIG. 5.

[0086] For the adapting 33 of the state function 20 in the second phase of the Kalman filter, the state function 20—that is to say, the clothoid spline 20—and the ambient measured data 50 are transformed into a point space. This means that both the ambient measured data 50 and the clothoid spline 20 can each be described by a plurality of points in the point space. The points can be ascertained from the clothoid spline 20 and the ambient measured data 50 by a sampling method. Therefore the points are also called sample points 51 and 52.

[0087] It is not possible to ascertain sample points 51 or spatial coordinates of sample points 51 by reference to the usual representation (1) for the clothoid spline 20. Therefore the transforming 32 of the clothoid spline 20 is necessary. For the transforming 32 of the clothoid spline 20, a measurement model can be ascertained, with the aid of which the clothoid spline 20 can be transformed into the measurement space of the ambient measured data 50, so that the clothoid spline 20 can be represented by spatial coordinates.

[0088] There are several measurement models for transforming 32 the clothoid spline 20 in order to represent the latter by spatial coordinates instead of by arc lengths and curvature values as in (1). In the following, two measurement models that can be drawn upon for the purpose of transforming 32 the clothoid spline 20 will be considered, by way of example, by reference to the illustrations 60-1 and 60-2 represented in FIG. 6.

[0089] A first measurement model for the transforming 32 of the clothoid spline 20 makes provision for an approximate representation of the clothoid spline 20, by the clothoid segments 21 being approximated by third-degree polynomials 62. This representation is shown in illustration 60-1. The third-degree polynomial 62 can be represented as follows:

[00002]$\begin{array}{cc}y\left(x\right)={\mathrm{ax}}^3+{\mathrm{bx}}^2+\mathrm{cx}+d& \left(2\right)\end{array}$

a, b, c, and d correspond to parameters that determine a shape of polynomial 62. For the approximate representation of the clothoid segment 21, parameters a, b, c, and d can be replaced by parameters c.sub.0, c.sub.1,m, θ.sub.0 and y.sub.0 of the clothoid segment 21 in the following way:

[00003]$\begin{array}{cc}a=c_{1,m},b=c_0,c=\mathrm{atan}\left({\theta }_0\right),d=y_0& \left(3\right)\end{array}$

[0090] By insertion of the parameters according to (3), polynomial 62, for instance, is obtained which maps by approximation a course of the clothoid or, for instance, of the clothoid segment 21.

[0091] Sample points 61 consequently correspond approximately to sample points 51. By virtue of the approximate representation of the clothoid segment 21 by (2) and (3), spatial coordinates can therefore be assigned to each sample point 61. The clothoid spline 20 in the Kalman filter can be represented in the point space with the aid of the first measurement model described herein. With sample points 61 and sample points 52 of the point space as input variables, the Kalman filter can perform an approximation of polynomial 62 to the ambient measured data 50. In this connection, values for the parameters c.sub.0, θ.sub.0, y.sub.0, and c.sub.1,1 . . . c.sub.1,n of the clothoid spline 20 can be ascertained. With the aid of insertion of the values, the estimate of the course of a roadway can be determined that results from the adapting 33 of the clothoid spline 20 to the ambient measured data 50 pertaining to the camera 41.

[0092] In the case of slight curvatures, the clothoid segments 21 can be approximated well by the described measurement model according to (2) and (3). In the case of intense curvatures, an accuracy of the approximate representation of the clothoid spline 20 according to (2) and (3) may be insufficient to guarantee a high degree of accuracy for the adapting 33 of the clothoid spline 20 to the ambient measured data 50.

[0093] For a better approximation in the case of intense curvatures, a second measurement model instead of the first measurement model can be applied to the clothoid spline 20. For instance, the clothoid segment 21 can be represented by the parameter representation of the clothoid function.

[0094] This is illustrated in illustration 60-2. In the case of the parameter representation—for instance, of the clothoid segment 21—each sample point 63 is represented in a vector representation. A vector of such a sample point 63 comprises, for instance, two components that can be expressed by Fresnel integrals. In order to reduce a numerical effort for computation of the Fresnel integrals, a fifth-order Taylor expansion of the Fresnel integrals, for instance, can be used instead of the Fresnel integrals. From this approximation of the clothoid segment 21, a function 64, for instance, may result, the function 64 corresponding to a Taylor polynomial 64 by reason of the fifth-order Taylor expansion. As can be seen in FIG. 6, this polynomial 64 describes the course of the clothoid segment 21 relatively accurately. The sample points 63 determined with the aid of the second measurement model describe the clothoid segment 21 and/or the clothoid spline 20 to a good approximation, even in the case of intense curvatures. The sample points 63 generated by the second measurement model can better represent the clothoid spline by approximation, particularly in the case of large curvatures, than the sample points 61 generated by the first measurement model. Therefore the course of a roadway can usually be determined with greater accuracy with the aid of the second measurement model than with the first measurement model. In general, both measurement models can be employed for estimating the course of a roadway.

[0095] Varying sampling methods can be used for a determination of sample points 61 and 63 of the respective measurement models and of sample points 52 of the ambient measured data 50. Two possibilities for the sampling method are illustrated in FIG. 7. In a first, usual sampling method (on the left in FIG. 7), sample points 61, 63 and 52 can be chosen in such a way that a spacing of the sample points in the x-direction is constant. It has become evident that this sampling method is not optimal for an intensely curved course of a roadway when adapting 33 the clothoid spline 20, or, to be more exact, the polynomials 62 and 64, with the Kalman filter. With this sampling method, by reason of systematic errors based on geometrical conditions in the case of intense curvatures of the course of a roadway, the distances between sample points 61 or 63 and sample points 52 can be relatively large, even though the polynomial 62 or 64 corresponds approximately to the ambient measured data 50. The Kalman filter would then shift the clothoid spline 20 “too intensely” at the time of the innovation. For instance, in an example represented in the diagram on the left in FIG. 7, the clothoid spline 20 would be shifted down too much, since the spacing d of sample points 61 or 63 from sample points 52 is relatively large.

[0096] Such a systematic error can be reduced by choosing a second sampling method (on the right in FIG. 7) along the lines of further embodiments. In the case of the sampling method illustrated in the diagram on the right in FIG. 7, sample points 61, 63 and 52 are chosen in such a way that a constant arc length Δs lies between adjacent sample points 61, 63 and 52 of the respective polynomials 62, 64 or of the ambient measured data 50. In this way, even in the case of intense curvatures of the course of a roadway, sample points 61, 63 and 52 are situated in such a way that the spacing d between sample points 61 or 63 and sample points 52 is, as a result, for instance, smaller than in the diagram on the left in FIG. 7. Correspondingly, the clothoid spline 20 is, for instance, shifted downward less intensely by the Kalman filter. The adapting 33 of the clothoid spline 20 with the Kalman filter by using the second sampling method can achieve a higher accuracy in comparison with the adapting 33 by using the first sampling method. In addition, the second sampling method has a stabilizing effect on a function of the Kalman filter and hence also has a stabilizing effect on a lateral guidance behavior in the course of controlling the vehicle, since the clothoid spline 20 is corrected slightly by using the second sampling method in the course of the adapting 33. Hence it may be the case that the state function is adapted less “intensely” in the individual recursion steps and is therefore designated as “more stable”.

[0097] In some embodiments, the camera 41 can capture the course of roadway markings 12 or 13, and the image-processing application may have been designed to detect such roadway markings 12 or 13 and to describe them approximately by a polynomial or a polynomial spline. Roadway markings 13 denote boundaries of the roadway. Roadway marking 12 denotes the center line of the roadway, which in some cases is not visible. A schematic illustration of a roadway denoted by roadway markings 12 and 13 is illustrated in FIG. 8. By reference to the course of the roadway markings and/or 13, a roadway width or lane width can, for instance, be ascertained from photographs taken by the camera 41 with the aid of the suitable image-processing application. In some cases, the estimate of the course of the roadway may correspond to the course of roadway markings 13. This may be advantageous for the control of the vehicle. From the courses of two roadway markings 13 which delimit the lane to the right and to the left, it is possible, for instance, to ascertain the lane width or to derive the course of the center line 12 therefrom.

[0098] In the course of the control of the vehicle it may be necessary, under certain circumstances, to ascertain the course of the lane center. In the absence of a center line 12, a course of the lane center cannot be ascertained directly by data fusion 44 of the ambient measured data 50 pertaining to the camera. In such a case, the course of the roadway markings or roadway boundaries 13 can be ascertained by means of data fusion 44, and, proceeding therefrom, taking the lane width into consideration, the course of the lane center or of the center line 12 can be derived. The lane width of the roadway to be determined can either be taken from a data record available to the vehicle or can be determined by reference to the course of roadway markings 13. For a determination of the course of the center line 12, the clothoid spline 20 that, for instance, describes the course of the middle of the lane can be transformed in such a way that a transformed clothoid spline describes the course of one of the roadway markings 13, in order to enable an adapting 33 of the clothoid spline 20 to the ambient measured data pertaining to the camera. After adapting 33 of the clothoid spline 20, the clothoid spline that, in turn, characterizes the course of the lane center can be determined by inverse transforming. For geometrical reasons, for the transforming and inverse transforming it may not be sufficient to shift the clothoid spline 20 in translatory manner—that is to say, in the x- and y-directions.

[0099] For this purpose it may, for instance, be necessary to adapt the parameters c.sub.0, c.sub.1,m in addition by means of a mathematical method in such a way that the transformed clothoid spline corresponds to the course of the center line 12 and hence runs parallel to the roadway boundaries 13.

[0100] Alternatively, for the adapting 33 of the clothoid spline 20 the ambient measured data pertaining to the camera can be transformed, taking a lane width into consideration, in such a way that they approximately characterize the course of the center of the roadway. The Kalman filter can then perform an adapting 33 of the clothoid spline 20 to the transformed measured data. For the multi-sensor data fusion illustrated schematically in FIG. 4, ambient measured data pertaining to a further data source, different from the camera, are usually drawn upon. In the embodiment shown in FIG. 4, the map data pertaining to the HD map 43 are used for this purpose. An example of such map data is shown in FIG. 9. The sensorics provided by the map manufacturer 42 yield waypoints 91 that are situated on the roadway of the vehicle 11. The waypoints can be connected by interpolation, taking further connection conditions into consideration. At the time of the interpolation, the waypoints can be connected by polynomials 92, for instance. In this way, a polynomial spline 90 can be generated from several polynomial functions 92. The polynomial spline 90 may, for instance, correspond to the map data.

[0101] In some embodiments described herein, the polynomial spline may have been represented as a function in spatial coordinates. The clothoid spline 20, which at the time of the data fusion 44 can be adapted to the polynomial spline 90 by the Kalman filter, is usually present, here too, as a parameter set of the parameters y.sub.0, θ.sub.0, c.sub.0 and c.sub.1,1 . . . c.sub.1,n which are able to map the clothoid spline 20 with the aid of the curvature values c(s) resulting from (1).

[0102] In comparison with the ambient measured data 50 pertaining to the camera 41, the map data can already be ascertained long before the camera 41 is able to capture the roadway. For instance, the course of the roadway 14 after intensely curved and/or non-observable curves can be estimated. The map data can, for instance, be used in order to determine well the course of the roadway far ahead of the vehicle 11 by approximation.

[0103] An orientation of the vehicle sometimes cannot be ascertained, or cannot be ascertained accurately, by reference to the map data. The orientation of the vehicle with respect to the roadway 14 typically cannot be determined from position data pertaining to the vehicle 11 and from the polynomial spline 90, since no indication of the orientation of the vehicle 11 can result therefrom. Usually, therefore, spatial coordinates of the map data are not drawn upon in the Kalman filter in order to adapt the clothoid spline 20 to the map data. In this connection, a further measurement model may be employed, in which the map data are represented in a measurement space that encompasses a coordinate system with curvature values.

[0104] For the aforementioned data fusion 44 in the measurement space with the coordinate system with curvature values, it is, for instance, advantageous to represent the clothoid spline 20 in accordance with (1). Consequently the state space of the clothoid spline 20 may already correspond to the measurement space.

[0105] The polynomial spline 90 provided by the HD map 43 is usually present as a mapping in a coordinate system with spatial coordinates. In advantageous practical forms disclosed herein, transforming 32 the map data into the measurement space with the coordinate system with curvature values may therefore be necessary. A suitable mapping of the map data may be, for instance, a function of the arc length s of the map data, in which case a curvature value k of the map data results as a function of the arc length s.

[0106] For the adapting 33 of the clothoid spline 20 to the polynomial spline 90, the curvature values of the clothoid spline 20 are approximated to the curvature values k of the polynomial spline 90, for instance at the time of the innovation carried out by the Kalman filter. For instance, for this purpose the curvature values k at the waypoints of the map data are taken into consideration. The adapting 33 can be undertaken by adapting the parameters y.sub.0, θ.sub.0, c.sub.0 and c.sub.1,1 . . . c.sub.1,n.

[0107] One advantage of the described data fusion 44 by reference to the curvature values k is that this data fusion 44 is robust in relation to angle errors and offset errors. On the assumption that a localization ascertains a lane in which the vehicle 11 is actually located, the course of the roadway can, for instance, be estimated accurately by reference to the curvature values k, even if the vehicle 11 is not located precisely in the center of the lane or is not aligned parallel to the center line 12.

[0108] In advantageous versions herein, the computer-implemented method may include the multi-sensor data fusion 40. Advantages may result from the multi-sensor data fusion 40 of ambient measured data 50 and map data.

[0109] At the time of the data fusion 44 of the ambient measured data, the position and orientation of the vehicle 11 with respect to the roadway 14 can, for instance, be ascertained. In the course of the control of the vehicle 11, a straying of the vehicle 11 from the roadway 14, for instance, can consequently be prevented.

[0110] With the aid of the data fusion 44 of the map data, the course of the roadway can be determined far in advance, so that the control of the vehicle 11 can, for instance, reduce a speed of the vehicle early enough in order to drive safely through an intensely curved curve that cannot be observed.

[0111] Moreover, the data fusion 44 offers robustness in relation to angle errors and offset errors in the estimation of the course of a roadway, as a result of which errors of measurement of the estimation can be reduced. The computer-implemented method 30 can be used for the control of any ground-based vehicles 11. An example of such a vehicle 11 is shown in FIG. 10. The vehicle 11 may have been equipped with a device 100 that serves for estimating the course of a roadway. The device 100 comprises a processing unit (not shown here) and several interfaces for capturing ambient measured data. One such interface may be, for instance, a connection of the processing unit to one or more cameras 41. The cameras 41 may, for example, have been oriented in the direction of travel, contrary to a direction of travel, or to the side, and may enable a provision 31 of the ambient measured data 50. Further interfaces may connect the processing unit to a storage medium 103 such as, for example, a hard-disk drive or an optical memory and, additionally or alternatively, to a receiving unit 102. The provision 31 of map data 90 can be made possible by the map data 90 being present on the storage medium 103 or being capable of being received by the receiving unit 102 and provided to the processing unit. For instance, the map data can be transmitted to the receiving unit from a transmitter 110. With the ambient measured data 50 and the map data, the device 100 can determine an estimate of the course of a roadway by executing the computer-implemented method 10.

[0112] For this purpose, the processing unit has been designed to execute the transforming 32 of the clothoid spline 20 and of the ambient measured data 50 and/or of the map data. Moreover, the processing unit has been configured for the adapting 33 of the clothoid spline 20 to the ambient measured data. The processing unit may be, for example, a processor, a microcontroller, a field-programmable gate array (FPGA), a computer or a programmable hardware component.

[0113] The aspects and features that have been described together with one or more of the previously detailed examples and figures can also be combined with one or more of the other examples, in order to replace a like feature of the other example or in order to introduce the feature into the other example.

[0114] Examples may furthermore be, or relate to, a computer program with program code for executing one or more of the above methods when the computer program is executed on a computer or processor. Steps, operations or processes of various methods described above can be executed by programmed computers or processors. Examples may also cover program-storage devices, for example digital data-storage media that are machine-readable, processor-readable or computer-readable, and machine-executable, processor-executable or computer-executable programs of instructions. The instructions perform some or all of the steps of the methods described above, or bring about the execution thereof. The program-storage devices may include, or be, for example, digital memories, magnetic storage media such as, for instance, magnetic disks and magnetic tapes, hard-disk drives or optically readable digital data-storage media. Further examples may also cover computers, processors or control units that have been programmed to execute the steps of the methods described above, or (field-)programmable logic arrays ((F)PLAs) or (field-)programmable gate arrays ((F)PGAs) that have been programmed to execute the steps of the methods described above.

[0115] Only the principles of the disclosure are presented by the description and drawings. Furthermore, all the examples specified here are expressly intended to serve, in principle, for illustrative purposes only, in order to assist the reader in comprehending the principles of the disclosure and the concepts contributed by the inventor(s) for further development of the technology. All the statements made herein about principles, aspects and examples of the disclosure, as well as specific examples of the same, encompass the equivalents thereof.

[0116] A function block designated as “means for . . . ” executing a particular function may relate to a circuit that has been designed to execute a particular function. Consequently a “means for something” may have been implemented as a “means designed for or suitable for something”, for example a module or a circuit designed for or suitable for the respective task. Functions of various elements shown in the figures, inclusive of each function block designated as “means”, “means for providing a signal”, “means for generating a signal”, etc. may have been implemented in the form of dedicated hardware, for example “a signal-provider”, “a signal-processing unit”, “a processor”, “a control system” etc., and also as hardware capable of executing software in conjunction with associated software. In the case of provision by a processor, the functions may have been provided by a single dedicated processor, by a single processor used collectively, or by a plurality of individual processors, some or all of which can be used collectively.

[0117] However, the term “processor” or “control system” is by no means limited to hardware that is exclusively capable of executing software, but may encompass digital signal-processor hardware (DSP hardware), network processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), read-only memory (ROM) for storing software, random-access memory (RAM) and non-volatile storage. Other hardware—conventional and/or customer-specific—may also have been included.

[0118] A block diagram may represent, for example, a rough circuit diagram that implements the principles of the disclosure. Similarly, a flow diagram, a flowchart, a state-transition diagram, a pseudocode and such like may represent various processes, operations or steps which, for example, are substantially represented in computer-readable medium and accordingly executed by a computer or processor, irrespective of whether such a computer or processor has been shown explicitly. Methods disclosed in the description or in the claims can be implemented by a module that exhibits a means for executing each one of the respective steps of these methods.

[0119] It will be understood that the disclosure of several steps, processes, operations or functions disclosed in the description or in the claims is not to be construed as being in the particular sequence, unless this has been stated explicitly or implicitly elsewhere, for example for technical reasons. Therefore these are not limited to a particular sequence by the disclosure of several steps or functions, unless these steps or functions are not interchangeable for technical reasons. Furthermore, in some examples a single step, function, process or operation may include several sub-steps, sub-functions, sub-processes or sub-operations, and/or may be broken up into the same. Such sub-steps may have been included and may be part of the disclosure of this single step, unless they have been explicitly excluded.

[0120] Furthermore, the following claims have hereby been incorporated into the detailed description, where each claim may stand on its own as a separate example. Whilst each claim may stand on its own as a separate example, it should be noted that, even though a dependent claim in the claims may refer to a particular combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject-matter of any other dependent or independent claim. Such combinations are proposed here explicitly, unless it has been stated that a particular combination is not intended.

[0121] Furthermore, features of one claim are also intended to have been included for any other independent claim, even if this claim has not been made directly dependent on the independent claim.

LIST OF REFERENCE SYMBOLS

[0122] 11 vehicle [0123] 12 center line [0124] 13 roadway boundary [0125] 14 roadway [0126] 20 clothoid spline [0127] 21 clothoid segment [0128] 22 transition-point [0129] 30 computer-implemented method [0130] 31 provision of the ambient measured data [0131] 32 transforming of the state function and of the ambient measured data [0132] 33 adapting of the state function [0133] 33a checking the ambient measured data for an error [0134] 33b storing the error [0135] 33c utilizing the error [0136] 40 multi-sensor data fusion [0137] 41 camera [0138] 42 sensorics provided by the map manufacturer [0139] 43 HD map [0140] 44 data fusion [0141] 45 roadway model [0142] 50 ambient measured data pertaining to the camera [0143] 51 sample points of the clothoid spline [0144] 52 sample points of the ambient measured data pertaining to the camera [0145] 60-1 illustration of the first sampling method [0146] 60-2 illustration of the second sampling method [0147] 61 sample point of the polynomial [0148] 62 polynomial [0149] 63 sample point of the Taylor polynomial [0150] 64 Taylor polynomial [0151] 90 polynomial spline [0152] 91 waypoint [0153] 92 polynomial function [0154] 100 device [0155] 102 receiving unit [0156] 103 storage medium [0157] 110 transmitter