Method and apparatus for evaluating and selecting signal comparison metrics

Abstract

A method for evaluating a simulation model. In the method, for selected test cases, a first performance index is calculated in the simulation model. For the same test case, a second performance index is ascertained in a real test environment. For each of the test cases, a difference is calculated between the first performance index and the second performance index, and a signal metric is determined. For each of the signal metrics, an interrelation between the difference and the respective signal metric is investigated. The signal metric that exhibits the closest interrelation with the difference is selected.

Claims

1. A method for evaluating a simulation model in an at least semiautonomous robot or vehicle, the method comprising the following steps: calculating in the simulation model, for each of selected test cases, a respective first performance index; ascertaining in a real test environment, for each of the same selected test cases, a respective second performance index; for each of the selected test cases, calculating a respective difference between the respective first performance index and the respective second performance index and determining a respective signal metric; for each of the respective signal metrics, determining a respective interrelation between the respective difference and the respective signal metric; and selecting the respective metric that exhibits a closest interrelation with the respective difference.

2. The method as recited in claim 1, wherein the signal metrics relate to at least one of the following: a signal strength; and/or a phase shift; and/or a correlation.

3. The method as recited in claim 1, wherein the test cases are selected using one of the following methods: a random method; or an uncertainty quantification; or a search-based test method.

4. The method as recited in claim 1, further comprising the following step: varying the first performance index by modifying parameters of the simulation model.

5. The method as recited in claim 1, further comprising the following step: varying the second performance index by repeated the test cases.

6. The method as recited in claim 1, wherein: for each of the test cases: if both the respective difference and the respective signal metric are constant, the respective interrelation is defined as one; otherwise if the difference or the signal metric is constant, the respective interrelation is defined as zero; and otherwise, the respective interrelation is defined by a relationship indicator of the respective difference and the respective signal metric.

7. The method as recited in claim 6, wherein the relationship indicator is a correlation coefficient.

8. The method s recited in claim 1, wherein, depending on the signal metric selected, an automatic correction of errors, recognized based on the signal metric, of a system modeled by the simulation model, is effected.

9. A non-transitory machine-readable storage medium on which is stored a computer program for evaluating a simulation model in an at least semiautonomous robot or vehicle, the computer program, when executed by a computer, causing the computer to perform the following steps: calculating in the simulation model, for each of selected test cases, a respective first performance index; ascertaining in a real test environment, for each of the same selected test cases, a respective second performance index; for each of the selected test cases, calculating a respective difference between the respective first performance index and the respective second performance index and determining a respective signal metric; for each of the respective signal metrics, determining a respective interrelation between the respective difference and the respective signal metric; and selecting the respective metric that exhibits a closest interrelation with the respective difference.

10. An apparatus configured to evaluate a simulation model in an at least semiautonomous robot or vehicle, the apparatus being configured to: calculate in the simulation model, for each of selected test cases, a respective first performance index; ascertain in a real test environment, for each of the same selected test cases, a respective second performance index; for each of the selected test cases, calculate a respective difference between the respective first performance index and the respective second performance index and determine a respective signal metric; for each of the respective signal metrics, determine a respective interrelation between the respective difference and the respective signal metric; and selecting the respective metric that exhibits a closest interrelation with the respective difference.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Exemplifying embodiments of the present invention are explained in further detail below and are depicted in the figures.

(2) FIG. 1 is a visualization of the difference between a signal metric and a KPI.

(3) FIG. 2 is a data flow diagram of the method, which is representative of different algorithmic embodiments.

(4) FIGS. 3 to 7 show instances with different relationships between ΔKPI and SM, and the resulting interrelation.

(5) FIGS. 8 and 9 are exemplifying illustrations of the calculations of the method (FIG. 2) according to a possible embodiment, in which the simulated signal S is maintained and only the measured signals m.sub.i vary.

(6) FIG. 10 schematically shows a workstation according to a second embodiment of the invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

(7) A calculation according to the present invention is illustrated by FIG. 2 and is based on the feature of calculating on the one hand varying values ΔKPI and on the other hand varying signal metrics (26) for selected test cases (21, 29), by varying the output signals from simulation (22) and from observation (23) of various real measurements. The approach according to the present invention further provides for calculation of the interrelation (27) between the values calculated for ΔKPI and the signal metrics. The signal metric that exhibits the closest interrelation with ΔKPI is selected (28). The ΔKPI values refer here to the difference (25) between the performance index (24) calculated in the simulation model (22) and, on the other hand, ascertained in the real test environment (23).

(8) A variation of the simulation outputs is achieved by varying certain simulation parameters, e.g., input variables. The variation of the measurements can be achieved by repeating experiments or by way of multiple experiments under different conditions, for example using different parameters.

(9) As already mentioned, a signal metric SM.sub.k maps two signals onto one real value (SM: S×S.fwdarw.); in contrast thereto, a KPI maps a signal, and optionally the original SUT inputs X, onto a real value (KPI: S.fwdarw.). The functions SM and KPI thus possess different signatures, and the interrelation between ΔKPI (ΔKPI: S×S.fwdarw.) and SM is therefore calculated.

(10) The usual definition of the correlation is unsuitable, however, since it is possible (unlike in the ideal case depicted in FIG. 3) for ΔKPI or SM to be constant despite different simulation signals and different measurements. In that case the variance Var(ΔKPI) or Var(SM) is equal to zero, so that a corresponding correlation coefficient is undefined. In consideration of such cases, as depicted in FIGS. 4 to 7, and of numerical instabilities, a modified correlation proves to be useful:

(11) $\begin{array}{cc}{\mathrm{Corr}}^{\prime }\left(A,B\right):=\{\begin{array}{cc}\mathrm{Corr}\left(A,B\right),& \mathrm{Var}\left(\Delta \mathrm{KPI}\right)\ne 0⩓\mathrm{Var}\left(\mathrm{SM}\right)\ne 0\\ 1,& \mathrm{Var}\left(\Delta \mathrm{KPI}\right)=\mathrm{Var}\left(\mathrm{SM}\right)=0\\ 0,& \mathrm{Var}\left(\Delta \mathrm{KPI}\right)=0\oplus \mathrm{Var}\left(\mathrm{SM}\right)=0\end{array}& \left(\mathrm{Equation}1\right)\end{array}$
where ⊕ is the exclusive-OR operator.

(12) Be it noted that Equation 1 can also use other functions, for example the covariance, with the modifications described.

(13) This method (20) can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a control device as indicated by the schematic depiction of FIG. 10.

(14) Example embodiments of the present invention are further described in the following paragraphs.

(15) Paragraph 1. A method (20) for evaluating a simulation model (22) in particular of an at least semiautonomous robot or vehicle, characterized by the following features: for selected test cases (21), a first performance index (24) is calculated in the simulation model (22); for the same test cases (21), a second performance index (24) is ascertained in a real test environment (23); for each of the test cases (21), a difference (25) is calculated between the first performance index (24) and the second performance index (24), and a signal metric (26) is determined; for each of the signal metrics (26), an interrelation (27) between the difference (25) and the respective signal metric (26) is investigated; and the signal metric (26) that exhibits the closest interrelation (27) with the difference (25) is selected (28).

(16) Paragraph 2. The method (20) as recited in Paragraph 1, wherein the signal metrics (26) relate to at least one of the following: a signal strength; a phase shift; or a correlation.

(17) Paragraph 3. The method (20) as recited in Paragraph 1 or 2, wherein the test cases (21) are selected using one of the following methods: a random method; an uncertainty quantification; or a search-based test method.

(18) Paragraph 4. The method (20) as recited in one of Paragraphs 1 to 3, characterized by the following feature: the first performance index (24) is varied by the fact that parameters of the simulation model (22) are modified.

(19) Paragraph 5. The method (20) as recited in one of Paragraphs 1 to 4, characterized by the following feature: the second performance index (24) is varied by the fact that the test cases (21) are repeated.

(20) Paragraph 6. The method (20) as recited in one of Paragraphs 1 to 5, characterized by the following features: if both the difference (25) and the signal metric (26) are constant, the interrelation (27) is defined as one; otherwise, if the difference (25) or the signal metric (26) is constant, the interrelation (27) is defined as zero; and otherwise the interrelation (27) is defined by a relationship indicator of the difference (25) and the signal metric (26).

(21) Paragraph 7. The method (20) as recited in Paragraph 6, characterized by the following feature: the relationship indicator is a correlation coefficient.

(22) Paragraph 8. The method (20) as recited in one of Paragraphs 1 to 7, wherein depending on the signal metric (26) selected, an automatic correction of errors, recognized based on the signal metric (26), of a system modeled by the simulation model (22), is effected.

(23) Paragraph 9. A computer program that is configured to execute the method (20) as recited in one of Paragraphs 1 to 8.

(24) Paragraph 10. A machine-readable storage medium on which the computer program as recited in Paragraph 9 is stored.

(25) Paragraph 11. An apparatus that is configured to execute the method (20) as recited in one of Paragraphs 1 to 8.