Method and apparatus for testing a system, for selecting real tests, and for testing systems with machine learning components

Abstract

A method or testing a system. Input parameters of the system are divided into a first group and a second group. Using a first method, a first selection is made from among the input parameter assignments of the first group. Using a second method, a second selection is made from among the input parameter assignments of the second group. A characteristic value is calculated from the second selection. The first selection is adapted depending on the characteristic value.

Claims

1. A computer-implemented method for testing a system, comprising the following steps: dividing input parameters of the system into a first group and a second group; selecting, using a first method, a first selection from among the input parameters assigned to the first group, wherein the first method includes search-based testing in which an optimization technique is used to select the first selection in which an objective of the optimization technique is to minimize a performance of the system; selecting, using a second method, a second selection from among the input parameters assigned to the second group, wherein the second method selects the second selection based on a specific probability distribution; calculating a characteristic value from the second selection; adapting the first selection depending on the characteristic value; and performing a verification method with filtering of statistically irrelevant errors.

2. The method as recited in claim 1, wherein the system is embedded in an at least semiautonomous robot or vehicle.

3. The method as recited in claim 1, wherein the input parameters of the first group are subject to boundary conditions; and the input parameters of the second group are subject to limitations.

4. The method as recited in claim 1, further comprising the following step: simulating the system based on the first selection and the second selection.

5. The method as recited in claim 4, wherein the simulation encompasses an environment of the system.

6. The method as recited in claim 4, wherein the simulation supplies a performance evaluation of the system.

7. The method as recited in claim 1, wherein the dividing of the input parameters is accomplished manually.

8. The method as recited in claim 1, wherein the first group is smaller than the second group.

9. The method as recited in claim 1, wherein the verification method includes testing or worst-case-oriented methods.

10. The method as recited in claim 1, wherein the statistically irrelevant errors include adversarial examples which occur in a context of machine learning and computer vision.

11. The method as recited in claim 1, wherein an automatic improvement of errors of the system recognized in a test occurs depending on the test.

12. A non-transitory machine-readable memory medium on which is stored a computer program for testing a system, the computer program, when executed by a computer, causing the computer to perform the following steps: dividing input parameters of the system into a first group and a second group; selecting, using a first method, a first selection from among the input parameters assigned to the first group, wherein the first method includes search-based testing in which an optimization technique is used to select the first selection in which an objective of the optimization technique is to minimize a performance of the system; selecting, using a second method, a second selection from among the input parameters assigned to the second group, wherein the second method selects the second selection based on a specific probability distribution; calculating a characteristic value from the second selection; adapting the first selection depending on the characteristic value; and performing a verification method with filtering of statistically irrelevant errors.

13. An apparatus for testing a system, the apparatus configured to: divide input parameters of the system into a first group and a second group; select, using a first method, a first selection from among the input parameters assigned to the first group, wherein the first method includes search-based testing in which an optimization technique is used to select the first selection in which an objective of the optimization technique is to minimize a performance of the system; select, using a second method, a second selection from among the input parameters assigned to the second group, wherein the second method selects the second selection based on a specific probability distribution; calculate a characteristic value from the second selection; adapt the first selection depending on the characteristic value; and perform a verification method with filtering of statistically irrelevant errors.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Exemplifying embodiments of the present invention are depicted in the figures and are explained in further detail in the description below.

(2) FIG. 1 is a flow chart of a method according to a first embodiment of the present invention.

(3) FIG. 2 schematically shows the approach according to the present invention.

(4) FIG. 3 shows a workstation according to a second embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

(5) FIG. 1 illustrates a method (10) according to the present invention which will now be explained with reference to the block diagram of FIG. 2. The method provides that the set of input parameters Z of the SUT (reference character 20—FIG. 2) and its environment (reference character 27—FIG. 2) are divided into two groups X and Y of parameters (process 11—FIG. 1), and they are then investigated using two methods A and B. Method A is a test method that concentrates on the worst case and that creates a sample (reference character 21—FIG. 2) over the values of X (process 12—FIG. 1); and method B is a probabilistic method that creates a sample (reference character 22—FIG. 2) over the values of Y (process 13—FIG. 1). From this selection (22), a statistical characteristic value (reference character 23—FIG. 2) is calculated (process 14—FIG. 1) and is used in turn to influence the selection of X (process 15—FIG. 1). The probabilistic method B thus calculates, as a result, a mathematical projection of Z onto X which is used by method A.

(6) For this purpose, an the parameters Z are divided into the aforesaid two groups X and Y of parameters, where X∪Y=Z. Typically, but not necessarily, the number of parameters X is less than Y, i.e., |X|<|Y|. Parameters X are subject to boundary conditions (reference character 24—FIG. 2), and parameters Y are subject to limitations (reference character 25—FIG. 2), which in turn can contain hard boundary conditions or a distribution that can be predefined explicitly as a probability distribution function (PDF) or implicitly via a sampling method (e.g. environmental conditions).

(7) The example method can be summarized by the following algorithm:

(8) r1=[]

(9) if not A_TestEndX (r1): x=A_GenTestX (r1,XBoundaryConditions) r2=[]/empty list

(10) if not B_TestEndY (r2): y=B_GenSampleY (r2,YLimitations) r2=r2.append (CompleteSUT(x,y))))

(11) r1=Statistics(r2, x)

(12) endresult=sort(r1)

(13) A candidate for method A (A_TestEndX, A_GenTestX) is the aforementioned search-based testing. A candidate for B (B_TestEndY, B_GenSampleY) is uncertainty quantification that is also described above.

(14) The “CompleteSUT” function (reference character 26—FIG. 2) represents the SUT (20) together with its virtual environment (27), possible interference models, and a function (28) that evaluates its behavior or its outputs, for example in the form of a performance monitor, a test oracle, or simply an output signal selector. With the exception of the SUT (20) itself, however, the sub-components (27, 28) of this simulation (26) are optional.

(15) The “Statistics” function (reference character 23—FIG. 2) is a combination of the results r2 for a fixed x and a variable y; this is to be understood as a projection of y onto the current x. Examples of a suitable characteristic value (23) are a minimum, average, expected value, standard deviation, difference between maximum and minimum, or probability of default. The variable r1 represents a list or other data structure of tuples that link each value x to the corresponding statistical result.

(16) The functions “A_TestEndX” and “B_TestEndY” can be defined, for example, according to the following pseudocode: “|r1|<MaxSamplesA” and “|r2|<MaxSamplesB”. More-complex methods (e.g., coverage-based methods) are also possible.

(17) The statistical evaluations (23) with the associated parameter assignments X are combined in a function (reference character 29) and presented to the user as a result. Manifestations of this function are, for example, a sorting, a selection, or a visualization of the text cases based on the calculated statistics.

(18) The final result is a sorted list of the statistical results, which defines a prioritization of the test scenarios over X.

(19) The algorithm effectively searches for an allocation of X in which variations of Y result in the worst statistical value or in which the statistical sensitivity of the model is greatest. Because X is contained in the complete test space Z, it can be understood as a test scenario having variable parameters Y.

(20) With regard to the first of the challenges outlined above, the parameters X are typically inputs that can be controlled without difficulty in the real test, i.e., so to speak, “free” parameters such as the steering input or acceleration of an automobile. The parameters Y, however, are typically difficult to control—e.g. friction of the wheels, engine temperature, or wind conditions—but it is assumed that they too are considered in the simulation model (26). The output of the algorithm is a prioritization of test scenarios for the real environment which are to be regarded as being presumably the most critical in view of the statistics used.

(21) With regard to the second challenge, consider the utilization case of computer vision using the example of automated driving. The input of a relevant algorithm is typically an image, and its output corresponds to a classification of the objects visible in that image. Consider further here the case in which the input into the algorithm derives from an environment (27) that can either be simulated with the aid of three-dimensional computer graphics or recorded in real life using a camera.

(22) In this case the user selects the parameters X that describe the scenario, e.g., based on traffic circumstances, objects in the image, or time of day. The user further selects the parameters Y that can be varied in each scenario, e.g., camera position and orientation, intrinsic camera parameters, and the position and orientation of objects in the scene. The variations in the parameters Y can be regarded as a calculation of the probability of the occurrence of adversarial examples in a scenario.

(23) The algorithm according to the present invention supplies the scenarios that are most critical for the variations in Y. The safety of various operating sectors of an autonomous vehicle can thereby be determined or evaluated.

(24) With regard to the third challenge, test problems having many (for example, 50) parameters are difficult because of the problem of “state space explosion.” The approach described helps solve this problem by subdividing Z in such a way that |X|<<|Y|, e.g. |X|=5 and |Y|=45. The user selects the most important parameters as X, and less important parameters as Y. This approach allows the parameters X and Y to be dealt with using two different sample methods, and projects the results of the Y variation onto the X space. A coarse analysis of the Y space and a detailed analysis of the X space are thus carried out.

(25) This method (10) can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a workstation (30) as illustrated by the schematic depiction of FIG. 3.