METHOD FOR THE QUALIFICATION OF A CONTROL WITH THE AID OF A CLOSED-LOOP SIMULATION PROCESS

Abstract

A computer-implemented method for comparing generated data sequences for an at least semi-automated driving of a mobile platform, which were generated with the aid of a closed-loop simulation process, and a recorded data sequence of a trip of the mobile platform, controlled in at least semi-automated fashion, for the qualification of the control. The method includes: providing the recorded data sequence, which is based on a multiplicity of determinants, of trips of the mobile platform controlled in at least semi-automated fashion; providing a multitude of generated data sequences, which are based on the multiplicity of determinants, of simulated trips, which were generated with the aid of the closed-loop simulation process; providing similarity limits and a similarity metric for the respective determinant; comparing the recorded data sequence to each individual generated data sequence of the multitude of recorded data sequences.

Claims

1. A computer-implemented method for comparing generated data sequences for an at least semi-automated driving of a mobile platform, which were generated utilizing a closed-loop simulation process, and a recorded data sequence of a trip of the mobile platform controlled in at least semi-automated fashion, for qualification of the control, the method comprising the following steps: providing the recorded data sequence, which is based on a multiplicity of determinants, of trips of the mobile platform controlled in at least semi-automated fashion; providing a multitude of generated data sequences of simulated trips which were generated utilizing the closed-loop simulation process, which are based on the multiplicity of determinants; providing respective similarity limits and a respective similarity metric for each respective determinant; comparing the recorded data sequence to each generated data sequence of the multitude of recorded data sequences, by: determining a similarity of initial values of at least one determinant of the recorded data sequence to a determinant of the generated data sequence of the multitude of generated data sequences using the respective similarity limits, determining a similarity of a time characteristic of the at least one determinant of the recorded data sequence to the determinant of each generated data sequence of the multitude of generated data sequences using the respective similarity metric and the respective similarity limits; and depending on the determined similarity of the initial values of the at least one determinant and the determined similarity of the time characteristic of the at least one determinant, categorizing the recorded data sequence into a first evaluation class for determining a further determinant for a qualification of the control and/or categorizing the recorded data sequence into a second evaluation class for determining highly sensitive behavior of the control, to qualify the control of the at least semi-automated mobile platform using the computer-implemented method.

2. The computer-implemented method as recited in claim 1, wherein the recorded data sequence is categorized as a function of the determined similarity of the initial values of a multiplicity of selected determinants and the determined similarity of the time characteristic of the multiplicity of selected determinants.

3. The computer-implemented method as recited in claim 1, wherein a multitude of recorded data sequences is provided, and the method is carried out for each of the multitude of recorded data sequences.

4. The computer-implemented method as recited in claim 1, wherein the method is carried out for a multitude of recorded data sequences, and wherein each of the recorded data sequences is categorized into the first evaluation class or into the second evaluation class.

5. The computer-implemented method as recited in claim 4, wherein: each recorded data sequence of the multitude of recorded data sequences is categorized into the first evaluation class when, for all generated data sequences, their initial values for each of a number of selected determinants are within the respective similarity limit, and their similarity of the time characteristic for each of the number of selected determinants is outside of the respective similarity limits; and/or the recorded data sequence of the multitude of recorded data sequences is categorized into the second evaluation class when, for all generated data sequences, their initial values for each of the number of selected determinants are within the respective similarity limit and their similarity of the time characteristic for the at least one determinant of the number of selected determinants is within the respective similarity limits, and when their similarity of the time characteristic for at least one determinant of the number of selected determinants is outside of the respective similarity limits; and/or the recorded data sequence of the multitude of recorded data sequences is categorized into a third evaluation class when, for all generated data sequences whose initial values for each of a number of selected determinants are within the respective similarity limit, their similarity of the time characteristic for each of the number of selected determinants is within the respective similarity limits.

6. The computer-implemented as recited in claim 5, wherein the closed-loop simulation process simulates the recorded data sequence of the trip of the mobile platform, controlled in at least semi-automated fashion, sufficiently accurately for a qualification when the recorded data sequence is categorized into the third evaluation class.

7. The computer-implemented method as recited in claim 5, wherein the recorded data sequence of the multitude of recorded data sequences is categorized into a fourth evaluation class when, for all generated data sequences, their initial values for each of the number of selected determinants are determined outside of the respective similarity limits.

8. The computer-implemented method as recited in claim 5, wherein the closed-loop simulation process simulates trips of the mobile platform, controlled in at least semi-automated fashion, sufficiently accurately for a qualification when the multitude of the recorded data sequences are categorized in the third evaluation class.

9. The computer-implemented method as recited in claim 7, wherein the closed-loop simulation process simulates trips of the mobile platform, controlled in at least semi-automated fashion, sufficiently accurately for a qualification when the multitude of recorded data sequences are categorized in the third evaluation class or the fourth evaluation class.

10. The computer-implemented method as recited in claim 1, wherein the steps of the method are repeated when the recorded data sequence was classified with the second evaluation class, and the similarity limits provided have narrower limits for determining the similarity of the initial values of at least one determinant of the recorded data sequence to the determinant of each generated data sequence of the multitude of generated data sequences, in order to compare the evaluation class of the repetition of the method to the evaluation class of a previous implementation of the method.

11. The computer-implemented method as recited in claim 1, wherein the steps of the method are repeated with a further determinant for the recorded data sequence and the generated data sequences when the recorded data sequence is categorized with the first evaluation class and/or the second evaluation class, the further determinant being supplied using a candidate list for further determinants, in order to compare the evaluation class of the repetition to the evaluation class of a previous implementation of the method.

12. The computer-implemented method as recited in claim 10, wherein when the evaluation class of the repetition of the method is the same as the evaluation class of the previous implementation of the method, and the recorded data sequence is classified with the second evaluation class, at least one new recorded data sequence of a trip of the mobile platform, controlled in at least semi-automated fashion, is required, whose selected determinants correspond to the classified recorded data sequence, in order to examine a multistability of the control method.

13. The computer-implemented method as recited in claim 10, wherein when the evaluation class of the repetition is the same as an evaluation class of the previous implementation of the method, it is determined whether one of the determinants of the generated data sequences exceeds a safety-related value.

14. The computer-implemented method as recited in claim 1, wherein the method is used for qualification and/or verification of the control of the at least semi-automated mobile platform.

15. A non-transitory computer-readable storage medium on which is stored a computer program for comparing generated data sequences for an at least semi-automated driving of a mobile platform, which were generated utilizing a closed-loop simulation process, and a recorded data sequence of a trip of the mobile platform controlled in at least semi-automated fashion, for qualification of the control, the computer program, when executed by a computer, causing the computer to perform the following steps: providing the recorded data sequence, which is based on a multiplicity of determinants, of trips of the mobile platform controlled in at least semi-automated fashion; providing a multitude of generated data sequences of simulated trips which were generated utilizing the closed-loop simulation process, which are based on the multiplicity of determinants; providing respective similarity limits and a respective similarity metric for each respective determinant; comparing the recorded data sequence to each generated data sequence of the multitude of recorded data sequences, by: determining a similarity of initial values of at least one determinant of the recorded data sequence to a determinant of the generated data sequence of the multitude of generated data sequences using the respective similarity limits, determining a similarity of a time characteristic of the at least one determinant of the recorded data sequence to the determinant of each generated data sequence of the multitude of generated data sequences using the respective similarity metric and the respective similarity limits; and depending on the determined similarity of the initial values of the at least one determinant and the determined similarity of the time characteristic of the at least one determinant, categorizing the recorded data sequence into a first evaluation class for determining a further determinant for a qualification of the control and/or categorizing the recorded data sequence into a second evaluation class for determining highly sensitive behavior of the control, to qualify the control of the at least semi-automated mobile platform using the computer-implemented method.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Exemplary embodiments of the present invention are represented with reference to FIG. 1 and are explained in greater detail in the following.

[0091] FIG. 1 shows a data flow chart of the computer-implemented method for comparing generated data sequences and recorded data sequences, according to an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0092] FIG. 1 shows schematically the computer-implemented method for comparing generated data sequences 102 for an at least semi-automated driving of a mobile platform, which were generated with the aid of a closed-loop simulation process, and a recorded data sequence 104 of a trip of the mobile platform, controlled in at least semi-automated fashion, for the qualification of the control of the mobile platform.

[0093] A multitude of data sequences 102, generated by a closed-loop simulation process, may be supplied from candidate data sequences 101 and combined with a multitude of supplied recorded data sequences 104 in a step 103, in order to compare them

[0094] Safety goals 106 may be derived for generated data sequences 102 from provided safety goals 108, and derived safety goals 107 may also be provided for recorded data sequences 104.

[0095] Based on provided similarity limits 207 and similarity metrics 203, in step 204, a similarity of initial values of at least one determinant of a recorded data sequence 104 to the determinant of each generated data sequence 102 of the multitude of generated data sequences may be determined with the aid of respective similarity limits 207.

[0096] Furthermore, in step 204, a similarity of the time characteristic of the at least one determinant of a recorded data sequence 104 to the determinant of each generated data sequence of the multitude of generated data sequences 102 is determined with the aid of the respective similarity metric and the respective similarity limits.

[0097] If in a step 202, a further determinant is to be added for the comparison, an assigned similarity metric 201 and assigned similarity limits 205 may be determined for a further determinant from a provided candidate list.

[0098] In this context, the box for step 204 in FIG. 1 sketches two-dimensionally on a horizontal axis, a difference of initial values dP of the corresponding determinants of the respective generated data sequence and the respective recorded data sequence, in order to represent a similarity of the initial values. Plotted on the vertical axis is a difference between a similarity metric and the respective similarity limits of a time characteristic of the at least one determinant of the respective generated data sequence and the respective recorded data sequence, in order to represent a similarity of the time characteristic.

[0099] Depending on the determined similarity of the initial values of the at least one determinant and the determined similarity of the time characteristic of the at least one determinant, in step 301, the recorded data sequence is categorized into a first evaluation class A and/or a second evaluation class B, in order to qualify the control of the mobile platform. In this context, with a recorded data sequence 104 which is categorized into first evaluation class A, a further determinant may be determined, and alternatively or additionally, with a recorded data sequence 104 which is categorized into second evaluation class B, a highly sensitive behavior of the control of the mobile platform may be determined.

[0100] In this context, recorded data sequence 104 of the multitude of recorded data sequences is categorized into first evaluation class A if, for all generated data sequences 102, their initial values for each of a number of selected determinants are within the respective similarity limit, and their similarity of the time characteristic for each of the number of selected determinants is outside of the respective similarity limits.

[0101] Recorded data sequence 104 of the multitude of recorded data sequences is categorized into second evaluation class B if, for all generated data sequences 102, their initial values for each of the number of selected determinants are within the respective similarity limit, and their similarity of the time characteristic for at least one determinant of the number of selected determinants is within the respective similarity limits and if their similarity of the time characteristic for the at least one determinant of the number of selected determinants is outside of the respective similarity limits.

[0102] Recorded data sequences 104 are categorized into a third evaluation class C if, for all generated data sequences 102, their initial values for each of a number of selected determinants are within the respective similarity limit, and if for all generated data sequences 102, their similarity of the time characteristic for each of the number of selected determinants is within the respective similarity limits.

[0103] Recorded data sequences 104 are categorized into a fourth evaluation class D if, for all generated data sequences 102, their initial values for each of a number of selected determinants are outside of the respective similarity limit.

[0104] In a step 311, recorded data sequences 104, which are categorized into third evaluation class C, may be evaluated with a robustness value in accordance with a number of generated data sequences 102 with which they are categorized into third evaluation class C. A generated data sequence 102 which has the least deviation from recorded data sequence 104 may be determined as proxy for the number of generated data sequences 102 that are categorized with recorded data sequence 104 into third evaluation class C.

[0105] Using an additional determinant, which may be determined with the aid of a candidate list for additional determinants in step 322, recorded data sequences 104 which are categorized into first evaluation class A may run through the computer-implemented method again with the additional determinant, starting with step 103, in order to be categorized again, because namely, the simulation process may be carried out in more detailed fashion with the additional determinant, permitting better simulation of recorded data sequence 104. The result may be that with the additional determinant, recorded data sequence 104 is categorized into third evaluation class C.

[0106] If recorded data sequence 104 with the additional determinant continues to be categorized into first evaluation class A, in step 323, with the aid of a comparison to safety-related values, it may be checked whether one of the determinants selected for the comparison of generated data sequence 102 and recorded data sequence 104 is safety-related.

[0107] For recorded data sequences 104 which are categorized into second evaluation class B, in a step 342, narrower limits may be set for the similarity limits provided for determining the similarity of the initial values of at least one determinant of recorded data sequence 104 to the determinant of each generated data sequence of the multitude of generated data sequences 102, so as to compare the evaluation class upon repeating the computer-implemented method, to the evaluation class of a previous implementation of the method. Such a shift of the x-axis is sketched in diagram 345, which results from the fact that recorded data sequence 104 is re-categorized in step 301.

[0108] Due to the narrower limits for determining the similarity of the initial values, in a step 343, it may be determined that for the multitude of generated data sequences with this recorded data sequence, a categorization 344 into third evaluation class C results.

[0109] If, in spite of the narrower limits for determining the similarity of the initial values, this recorded data sequence 104 continues to be categorized into second evaluation class B, in step 345, the computer-implemented method may be run through again with an additional determinant, beginning with step 204, in order to be categorized once again, because namely, the simulation process may be carried out in more detailed fashion with the additional determinant, permitting better simulation of recorded data sequence 104. The result may be that with the additional determinant, recorded data sequence 104 is categorized into third evaluation class C.

[0110] If with the additional determinant, recorded data sequence 104 continues to be categorized into second evaluation class B, in step 346, with the aid of a comparison to safety-related values, it may be checked whether one of the determinants selected for the comparison of generated data sequence 102 and recorded data sequence 104 is safety-related.

[0111] In the case of recorded data sequences 104 which are classified into fourth evaluation class D, none of the multitude of generated data sequences 102 is suitable for corresponding recorded data sequence 104, so that in a step 331, a new simulation process may be developed and carried out, or trips of the mobile platform corresponding to the selected determinants may be carried out.

[0112] For an overview as to the range in which the selected determinants, such as a range of parameters, for example, the computer-implemented method may be employed, the corresponding selected determinants of generated data sequences 102, which are classified with recorded data sequences 104 into third evaluation class C or fourth evaluation class D, may themselves be transferred directly into a parameter space.

[0113] Decisive in the transfer is the respective value of the corresponding determinant of the recorded data sequence vP of each one of the comparison (e.g.,V1 vs every Sx). In this connection, parameter points of third evaluation class C may be assigned to a domain A, and parameter points of fourth evaluation class D may be assigned to a domain C. Customary parameter-space methods such as kernel density estimation or classification may be used for this purpose.

[0114] For the parameter points of first evaluation class A and second evaluation class B, the methods described above may be carried out with narrower limits and/or additional determinants prior to the corresponding entering into the parameter space.