MEASURING THE PERFORMANCE OF RADAR, ULTRASOUND OR AUDIO CLASSIFIERS

Abstract

A method for measuring the performance of a classifier for radar, ultrasound or audio spectra. The classifier is configured to map a radar, ultrasound or audio spectrum to a set of classification scores with respect to classes of a given classification. The method includes: providing a set of test radar, ultrasound or audio spectra that form part of, and/or define, a common distribution or manifold; obtaining at least one evaluation spectrum that is a modification of at least one test spectrum with substantially the same semantic content as this at least one test spectrum, and/or does not form part of the common distribution or manifold; mapping, using the classifier, the at least one evaluation spectrum to a set of evaluation classification scores; and determining the performance based on the set of evaluation classification scores, and/or on a further outcome produced by the classifier during the processing of the evaluation spectrum.

Claims

1. A method for measuring performance of a classifier for radar, ultrasound, or audio spectra, the spectrum includes a dependence of at least one measurement quantity that has been derived from a radar, ultrasound, or audio signal on spatial coordinates, and the classifier is configured to map a radar, ultrasound, or audio spectrum to a set of classification scores with respect to classes of a given classification, the method comprising the following steps: providing a set of test radar, ultrasound, or audio spectra that form part of, and/or define, a common distribution or manifold; obtaining at least one evaluation spectrum that: is a modification of at least one test spectrum with substantially the same semantic content as the at least one test spectrum, and/or does not form part of the common distribution or manifold; mapping, using the classifier, the at least one evaluation spectrum to the set of classification scores; and determining the performance based on the set of classification scores, and/or on a further outcome produced by the classifier during processing of the evaluation spectrum; wherein the determining is based at least in part on a comparison between an outcome of the classifier for the evaluation spectrum and an outcome that the classifier has outputted or should output for: the test spectrum from which the evaluation spectrum has been derived, and/or at least one other test spectrum from the set of test spectra.

2. The method of claim 1, wherein the outcome that is used for the comparison includes: at least one classification score and/or confidence, and/or; a rating of at least one classification score by a loss function; and/or a classification accuracy; and/or an expected calibration error.

3. The method of claim 1, wherein the obtaining of the at least one evaluation spectrum includes: applying at least one perturbation to the at least one test spectrum, thereby generating a perturbed spectrum; and determining the evaluation spectrum from the at least one perturbed spectrum.

4. The method of claim 3, further comprising: specifically choosing a perturbation that is likely to occur during the acquisition of a radar, ultrasound or audio signal with at least one sensor, and/or during signal processing that derives the at least one measurement quantity from the signal.

5. The method of claim 3, wherein the at least one perturbation includes: multiplying the test spectrum with a scalar constant; and/or multiplying values in the test spectrum with noise samples drawn from a random distribution; and/or shifting the test spectrum with respect to at least one spatial coordinate; and/or downsampling the test spectrum and then scaling it back to its original size; and/or cutting out a portion of the test spectrum and then scaling the portion to an original size of the test spectrum; and/or smoothing the test spectrum.

6. The method of claim 3, wherein the performance is determined as a function of a strength of the applied perturbation.

7. The method of claim 1, wherein, the smaller a difference determined during the comparison is, the better the performance is determined to be.

8. The method of claim 1, wherein the performance is determined based at least in part on a distinguishing performance of the classifier in distinguishing between spectra that do not form part of the common distribution or manifold and spectra that form part of the common distribution or manifold.

9. The method of claim 1, wherein the performance is determined based at least in part on a uniformity of the evaluation classification scores outputted by the classifier for an evaluation spectrum that does not form part of the common distribution or manifold.

10. A method for training a classifier for radar, ultrasound, or audio spectra, comprising the following steps: setting at least one hyperparameter that affects an architecture of the classifier, and/or the behavior of the training of the classifier; providing training spectra that are labelled with ground truth classification scores; training the classifier with an objective that, when given the training spectra, the classifier maps the training spectra to the ground truth classification scores; measuring the performance of the trained classifier by: providing a set of test radar, ultrasound, or audio spectra that form part of, and/or define, a common distribution or manifold; obtaining at least one evaluation spectrum that: is a modification of at least one test spectrum with substantially the same semantic content as the at least one test spectrum, and/or does not form part of the common distribution or manifold; mapping, using the classifier, the at least one evaluation spectrum to the set of classification scores; and determining the performance based on the set of classification scores, and/or on a further outcome produced by the classifier during processing of the evaluation spectrum; wherein the determining is based at least in part on a comparison between an outcome of the classifier for the evaluation spectrum and an outcome that the classifier has outputted or should output for: the test spectrum from which the evaluation spectrum has been derived, and/or at least one other test spectrum from the set of test spectra; optimizing the at least one hyperparameter with an objective that, when the classifier is trained and its performance is measured again, the performance is likely to improve.

11. A method, comprising the following steps: providing a classifier for radar, ultrasound, or audio spectra; training the classifier by: setting at least one hyperparameter that affects an architecture of the classifier, and/or the behavior of the training of the classifier; providing training spectra that are labelled with ground truth classification scores; training the classifier with an objective that, when given the training spectra, the classifier maps the training spectra to the ground truth classification scores; measuring the performance of the trained classifier by: providing a set of test radar, ultrasound, or audio spectra that form part of, and/or define, a common distribution or manifold; obtaining at least one evaluation spectrum that: is a modification of at least one test spectrum with substantially the same semantic content as the at least one test spectrum, and/or does not form part of the common distribution or manifold; mapping, using the classifier, the at least one evaluation spectrum to the set of classification scores; and determining the performance based on the set of classification scores, and/or on a further outcome produced by the classifier during processing of the evaluation spectrum; wherein the determining is based at least in part on a comparison between an outcome of the classifier for the evaluation spectrum and an outcome that the classifier has outputted or should output for: the test spectrum from which the evaluation spectrum has been derived, and/or at least one other test spectrum from the set of test spectra; optimizing the at least one hyperparameter with an objective that, when the classifier is trained and its performance is measured again, the performance is likely to improve; acquiring, using at least one radar, ultrasound or audio sensor carried by a vehicle, at least one radar, ultrasound, or audio spectrum; mapping, using the trained classifier, the at least one radar, ultrasound or audio spectrum to classification scores; determining an actuation signal based at least in part on the classification scores; and actuating the vehicle with the actuation signal.

12. A non-transitory machine-readable storage medium on which is stored a computer program including machine-readable instructions for measuring performance of a classifier for radar, ultrasound, or audio spectra, the spectrum includes a dependence of at least one measurement quantity that has been derived from a radar, ultrasound, or audio signal on spatial coordinates, and the classifier is configured to map a radar, ultrasound, or audio spectrum to a set of classification scores with respect to classes of a given classification, the instructions, when executed by one or more computers, causing the one or more computers to perform the following steps: providing a set of test radar, ultrasound, or audio spectra that form part of, and/or define, a common distribution or manifold; obtaining at least one evaluation spectrum that: is a modification of at least one test spectrum with substantially the same semantic content as the at least one test spectrum, and/or does not form part of the common distribution or manifold; mapping, using the classifier, the at least one evaluation spectrum to the set of classification scores; and determining the performance based on the set of classification scores, and/or on a further outcome produced by the classifier during processing of the evaluation spectrum; wherein the determining is based at least in part on a comparison between an outcome of the classifier for the evaluation spectrum and an outcome that the classifier has outputted or should output for: the test spectrum from which the evaluation spectrum has been derived, and/or at least one other test spectrum from the set of test spectra.

13. One or more computers configured to measure performance of a classifier for radar, ultrasound, or audio spectra, the spectrum includes a dependence of at least one measurement quantity that has been derived from a radar, ultrasound, or audio signal on spatial coordinates, and the classifier is configured to map a radar, ultrasound, or audio spectrum to a set of classification scores with respect to classes of a given classification, the one or more computers configured to: provide a set of test radar, ultrasound, or audio spectra that form part of, and/or define, a common distribution or manifold; obtain at least one evaluation spectrum that: is a modification of at least one test spectrum with substantially the same semantic content as the at least one test spectrum, and/or does not form part of the common distribution or manifold; map, using the classifier, the at least one evaluation spectrum to the set of classification scores; and determine the performance based on the set of classification scores, and/or on a further outcome produced by the classifier during processing of the evaluation spectrum; wherein the determining is based at least in part on a comparison between an outcome of the classifier for the evaluation spectrum and an outcome that the classifier has outputted or should output for: the test spectrum from which the evaluation spectrum has been derived, and/or at least one other test spectrum from the set of test spectra.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG. 1 shows an exemplary embodiment of the method 100 for measuring the performance 6 of a classifier 1, in accordance with the present invention.

[0052] FIG. 2 shows an exemplary embodiment of the method 200 for training the classifier 1, in accordance with the present invention.

[0053] FIG. 3 shows an exemplary embodiment of the method 300 with the complete sequence of actions up to and including the actuating of a vehicle 50, in accordance with the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0054] FIG. 1 is a schematic flow chart of an exemplary embodiment of the method 100 for measuring the performance 6 of the classifier 1.

[0055] In step 110, a set of test radar, ultrasound or audio spectra 2 is provided.

[0056] In step 120, at least one evaluation spectrum 4 is obtained. This evaluation spectrum 4 is a modification of at least one test spectrum 2 with substantially the same semantic content as this at least one test spectrum 2, and/or it does not form part of the common distribution or manifold.

[0057] In particular, according to block 121, the obtaining 120 of the evaluation spectrum 4 may comprise applying at least one perturbation to at least one test spectrum. This produces a perturbed spectrum. From this at least one perturbed spectrum, the evaluation spectrum 4 may then be determined according to block 122.

[0058] According to block 121a, the perturbation may be specifically chosen to be a perturbation that is likely to occur during the acquisition of a radar, ultrasound or audio signal with at least one sensor 10, and/or during the signal processing that derives the at least one measurement quantity of the radar, ultrasound or audio spectrum from said signal.

[0059] In step 130, the given classifier 1 maps the at least one evaluation spectrum 4 to a set of evaluation classification scores 5.

[0060] In step 140, the sought performance 6 is determined based on the set of evaluation classification scores (5), and/or on a further outcome produced by the given classifier 1 during the processing of the evaluation spectrum 4.

[0061] In particular, according to block 141, this determining 140 may be based at least in part on a comparison between an outcome of the classifier 1 for the evaluation spectrum 4 and an outcome that the classifier 1 has outputted or should output for [0062] a test spectrum 2 from which the evaluation spectrum 4 has been derived, and/or [0063] at least one other test spectrum 2 from the given set of test spectra.

[0064] One example of an outcome that the classifier 1 “should output” for a test spectrum 2 is a ground truth label associated with this test spectrum 2.

[0065] According to block 142, the sought performance 6 may be determined as a function of a strength of an applied perturbation.

[0066] According to block 143, the sought performance 6 may be determined based at least in part on a distinguishing performance of the classifier 1 in distinguishing between spectra that do not form part of the common distribution or manifold and spectra that form part of the common distribution or manifold.

[0067] According to block 144, the sought performance 6 may be determined based at least in part on the uniformity of the evaluation classification scores 5 outputted by the classifier 1 for an evaluation spectrum 4 that does not form part of the common distribution or manifold.

[0068] FIG. 2 is a schematic flow chart of an embodiment of the method 200 for training a classifier 1 for radar, ultrasound or audio spectra.

[0069] In step 210, at least one hyperparameter 7 is set. This hyperparameter 7 affects the architecture of the classifier 1, and/or the behavior of the training of this classifier 1.

[0070] In step 220, training spectra 2a that are labelled with ground truth classification scores 2b are provided.

[0071] In step 230, the classifier 1 is trained with the objective that, when given the training spectra 2a, it maps them to the ground truth classification scores 2b. The trained classifier is labelled with the reference sign 1*.

[0072] In step 240, the performance 6 of the trained classifier 1* is measured with the method 100 described above.

[0073] In step 250, the at least one hyperparameter 7 is optimized with the objective that, when the classifier 1 is trained in step 230 and its performance 6 is measured in step 240 again, this performance 6 is likely to improve. This optimization may be terminated according to any suitable termination criterion. The finally obtained optimized value of the hyperparameter 7 is labelled with the reference sign 7*.

[0074] FIG. 3 is a schematic flow chart of an embodiment of the method 300 with the complete sequence of actions.

[0075] In step 310, a classifier 1 for radar, ultrasound or audio spectra 2 is provided.

[0076] In step 320, the classifier 1 is trained with the method 200 described above.

[0077] In step 330, at least one radar, ultrasound or audio spectrum 2 is acquired using at least one radar, ultrasound or audio sensor 10 that is carried by a vehicle 50.

[0078] In step 340, the at least one radar, ultrasound or audio spectrum 2 is mapped to classification scores 3 using the trained classifier 1*.

[0079] In step 350, based at least in part on the classification scores 3, an actuation signal 350a is determined.

[0080] In step 360, the vehicle 50 is actuated with the actuation signal 350a.