METHOD FOR CLASSIFYING AN OBJECT TO BE DETECTED WITH AT LEAST ONE ULTRASONIC SENSOR

Abstract

A method for classifying an object to be detected with at least one ultrasonic sensor. The method includes: transmitting a first signal using the ultrasonic sensor to the object; receiving a second signal using the ultrasonic sensor, wherein the second signal is a backscattered signal from the object; processing the second signal into a digital signal; extracting a selected signal portion from the digital signal, the selected signal portion representing a relevant and time-limited time segment from the digital signal; transforming the selected signal portion into a two-dimensional feature vector; feeding the two-dimensional feature vector into a neural network as at least one input variable; determining object class information for the object using the neural network, wherein, based on the at least one input variable, the neural network produces an output variable which indicates a probability value for at least one defined object class for the one object.

Claims

1. A method for classifying an object to be detected with at least one ultrasonic sensor, comprising the following steps: transmitting a first signal, using the at least one ultrasonic sensor, to the object; receiving a second signal using the at least one ultrasonic sensor, wherein the second signal is a backscattered signal from the object; processing the second signal into a digital signal; extracting a selected signal portion from the digital signal, wherein the selected signal portion represents a relevant and time-limited time segment from the digital signal; transforming the selected signal portion into a two-dimensional feature vector; feeding the at least one two-dimensional feature vector into a neural network as at least one input variable; and determining object class information for the object using the neural network, wherein, based on the at least one input variable, the neural network produces an output variable which indicates a probability value for at least one defined object class for the one object.

2. The method according to claim 1, further comprising feeding at least one second input variable into the neural network, wherein the second input variable includes distance information which represents a distance between the at least one ultrasonic sensor and the one object.

3. The method according to claim 2, wherein the distance information is fed as an intermediate feed into a posterior classifier portion of fully connected layers of the neural network.

4. The method according to claim 1, wherein the neural network includes at least one first convolutional layer with a non-square filter kernel, a narrow side of which extends along a time dimension of the feature vector, and a second convolutional layer with a non-square filter kernel, a narrow side of which extends along a frequency dimension of the feature vector.

5. The method according to claim 1, wherein the selected signal portion from the digital signal represents a fixed time period from an ascertained starting point of the digital signal.

6. The method according to claim 1, wherein the selected signal portion is ascertained from the digital signal using a sliding window approach or a pulse-echo method, wherein a window which is shorter than an entire recording length of the digital signal is shifted over the entire recording length of the digital signal.

7. The method according to claim 1, wherein the step of processing the second signal into a digital signal includes filtering the digital signal to improve an S/N ratio.

8. The method according to claim 1, wherein the second signal is configured as an analog signal.

9. A detection system configured to classify an object to be detected, comprising: at least one ultrasonic sensor which is configured to: transmit a first signal to the object; receiving a second signal, wherein the second signal is a backscattered signal from the object; process the second signal into a digital signal; extract a selected signal portion from the digital signal, wherein the selected signal portion represents a relevant and time-limited time segment from the digital signal; transform the selected signal portion into a two-dimensional feature vector; feed the at least one two-dimensional feature vector into a neural network as at least one input variable; and determine object class information for the object using the neural network, wherein, based on the at least one input variable, the neural network produces an output variable which indicates a probability value for at least one defined object class for the one object, wherein the at least one ultrasonic sensor (50) can be used in a vehicle.

10. A non-transitory machine-readable data carrier on which is stored a computer program including machine-readable instructions for classifying an object to be detected with at least one ultrasonic sensor, the instructions, when executed on one or more computers and/or compute instances, causing the one or more computers and/or compute instances to perform the following steps: transmitting a first signal, using the at least one ultrasonic sensor, to the object; receiving a second signal using the at least one ultrasonic sensor, wherein the second signal is a backscattered signal from the object; processing the second signal into a digital signal; extracting a selected signal portion from the digital signal, wherein the selected signal portion represents a relevant and time-limited time segment from the digital signal; transforming the selected signal portion into a two-dimensional feature vector; feeding the at least one two-dimensional feature vector into a neural network as at least one input variable; and determining object class information for the object using the neural network, wherein, based on the at least one input variable, the neural network produces an output variable which indicates a probability value for at least one defined object class for the one object.

11. One or more computers and/or compute instances configured to classify an object to be detected with at least one ultrasonic sensor, the one or more computers and/or compute instances being configured to: transmit a first signal, using the at least one ultrasonic sensor, to the object; receive a second signal using the at least one ultrasonic sensor, wherein the second signal is a backscattered signal from the object; process the second signal into a digital signal; extract a selected signal portion from the digital signal, wherein the selected signal portion represents a relevant and time-limited time segment from the digital signal; transform the selected signal portion into a two-dimensional feature vector; feed the at least one two-dimensional feature vector into a neural network as at least one input variable; and determine object class information for the object using the neural network, wherein, based on the at least one input variable, the neural network produces an output variable which indicates a probability value for at least one defined object class for the one object.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG. 1 shows a schematic flowchart of the method 100 for classifying an object 10 to be detected with at least one ultrasonic sensor 50, according to an example embodiment of the present invention.

[0033] FIG. 2 shows a schematic illustration of a detection system 200 for carrying out the method 100 according to an example embodiment of the present invention.

[0034] FIG. 3 shows a schematic illustration of training a neural network 9 using the method 100 according to an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0035] FIG. 1 shows a schematic flowchart of the method 100 for classifying an object 10 to be detected with at least one ultrasonic sensor 50.

[0036] In Step 102, a first signal 1 is transmitted by means of the at least one ultrasonic sensor 50 to the object 10.

[0037] In Step 106, a second signal 2 is received by means of the at least one ultrasonic sensor 50, wherein the second signal 2 is a backscattered signal from the object 10. The second signal 2 is preferably analog signal. The analog signal 2 can be a high-resolution time signal obtained after A/D sampling, which represents the sound pressure in compliance with the Nyquist theorem. Simply stated, the analog signal 2 is therefore the sound pressure of the backscatter from the object 10 in the air.

[0038] In Step 108, the second signal 2 is processed into a digital signal 3. The processing of the second signal 2 can involve an analog/digital (AD) conversion, for example. The Step 108 of processing the second signal 2 into a digital signal 3 can also include filtering the digital signal 3 (advantageously after the A/D converter stage) to improve an S/N ratio. The filtering can be carried out using appropriate-seeming filters, such as low-pass filters, high-pass filters or band-pass filters.

[0039] In Step 110, a selected signal portion 4 is extracted from the digital signal 3, wherein the selected signal portion 4 represents a relevant and time-limited time segment from the digital signal 3. The selected signal portion 4 could optionally be subjected to a plausibility check to ensure a required quality of the extracted signal portion 4.

[0040] The selected signal portion 4 from the digital signal 3 represents a fixed time period from an ascertained starting point of the digital signal 3. The start time can be set using the pulse-echo method.

[0041] The selected signal portion 4 from the digital signal 3 can be ascertained using a sliding window approach or a pulse-echo method. This involves shifting a window which is shorter than the entire recording length of the digital signal 3 over an entire recording length of the digital signal 3, over the entire time or recording length of the signal 3. The sliding window approach is provided as an alternative to known echo-pulse methods for setting start times. However, the window length has to be set to a fixed value in both cases.

[0042] Or, in other words: in the sliding window approach, a window which is shorter than the entire length of the recording of the digital signal 3 is shifted over the entire recording length of the digital signal 3. A plurality of fixed start times can be implemented. This ensures that the individual window sections overlap, so that each window later receives an assigned classification.

[0043] In Step 114, the selected signal portion 4 is transformed 114 into a two-dimensional feature vector 5. The feature vector can alternatively also be multidimensional.

[0044] In Step 116, the at least one two-dimensional feature vector 5 is fed into a neural network 9 as at least one input variable 6. The neural network 9 can be contained both in the ultrasonic sensor 50 and in an associated control unit. If the neural network 9 is a component of a control unit (for example in a vehicle), the configuration of ultrasonic sensor 50 can be as simple as possible, which makes the ultrasonic sensor more cost-efficient.

[0045] The step 116 of feeding can involve providing at least one further input variable 7 to the neural network 9. The at least second input variable 7 includes distance information which represents a distance between the at least one ultrasonic sensor 50 and the one object 10.

[0046] It is important that feeding an additional feature distance to the object as the second input variable 7 does not take place before the first CNN layers, but rather as an intermediate feed into the posterior classifier portion of the fully connected layers. With respect to ultrasound, the properties of the objects to be classified are strongly influenced by the distance.

[0047] Therefore, this feature significantly improves the correct classification rates for the object 10.

[0048] In Step 118, object class information 12 for the one object 10 is determined using the neural network 9, wherein, based on the at least one input variable 6, the neural network 9 produces an output variable 8 which indicates a probability value for at least one defined object class for the one object 10.

[0049] FIG. 2 shows a schematic illustration of a detection system 200 for carrying out the method 100 according to an embodiment of the present invention. The detection system 200 comprises an ultrasonic sensor 50 that transmits a first signal 1 to the object 10 to be detected and classified. This first signal 1 is reflected by the object 10 and reaches the ultrasonic sensor 50 in the form of the second signal 2, which is processed in the above-described manner according to the method 100 according to the invention in order to classify the object 10.

[0050] In this context, it can generally be said that the ultrasonic time signal (=raw signal) backscattered by the object 10 and acquired by means of sensors is being used. The second signal 2 corresponds to the raw signal.

[0051] Downstream signal processing and calculation steps as explained above, then enable a classification of the object 10 into defined object classes. The objective of a precise and ultrasound-based object classification is achieved by the present invention through the following combination of: [0052] (A) special CNN (convolutional neural network) architecture, [0053] (B) ultrasound-specific feature extraction in preprocessing, for example using wavelet transformation, [0054] (C) (optional) feeding of an additional feature distance to the object not before the first CNN layers but as an intermediate feed into the posterior classifier portion of the neural network, and [0055] (D) use of physically motivated data augmentation methods.

[0056] With respect to the special CNN architecture, the following can be said:

[0057] Ultrasound-based object classification in vehicles and simple electronic devices in real time requires a small CNN. The CNN architecture therefore has to always be specifically adapted to the task in order to be able to achieve very good correct classification rates despite a small CNN.

[0058] This is achieved with the following architecture, for example: [0059] 1. convolutional layer, 5?7 kernel, padding (low level features), ReLU [0060] 2. pooling, 2?2 [0061] 3. convolutional layer, 1?5 kernel, padding (time domain features), ReLU [0062] 4. convolutional layer, 5?1 kernel, padding (frequency domain features), ReLU [0063] 5. pooling, 2?2 [0064] 6. convolutional layer, 3?3 kernel, padding (high level features), ReLU [0065] 7. pooling, 2?2 [0066] 8. convolutional layer, 3?3 kernel, padding (high level features), ReLU [0067] 9. concatenation of flattened feature maps and distance feature [0068] 10. fully connected layer, ReLU [0069] 11. dropout layer [0070] 12. fully connected layer, Softmax/Sigmoid

[0071] The number of neurons in the fully connected layers depends on the selected input variable of the time-frequency representation and the desired number of classes.

[0072] The special features in the CNN architecture of the neural network according to the invention can be presented as follows: [0073] Input is a time-frequency representation of 64?32 pixels, for instance. This achieves a very compact and meaningful representation of features in terms of temporal progression and frequency content. [0074] Specific layers: Use of a separated convolutional layer for time and frequency correlations. This is achieved with special non-square convolution kernels with an atypically small number of trainable parameters, e.g. 1?5 or 5?1, which are not common in image processing or other areas.

[0075] Using the method according to the invention makes the probabilities of assignment to one of the object classes available as the output variables of the CNN.

[0076] FIG. 3 shows a schematic illustration of training a neural network 9 or a classification of an object using a neural network using the method 100 according to an embodiment example of the present invention. After appropriate training, the neural network 9 is also used to carry out an object classification for an object. A first input variable 6 and, if necessary, at least one second input variable 7 are provided to the neural network 9. These two input variables 6, 7 are processed by the neural network 9 according to the method 100 as explained above, so that an output variable 8 representing object class information 12 for the detected object 10 is output by the neural network 9. The neural network 9 can be part of the ultrasonic sensor 50, but can also be located outside the ultrasonic sensor 50, for example, as part of a cloud or network-based connection or external connection to a data processing device, such as a control unit in a vehicle.

[0077] Additional data augmentation methods can be used to make the neural network 9 robust to interference from the outside. These can include: [0078] Distance-dependent white noise to modify the signal-to-noise ratio, [0079] Doppler shifting to simulate movements of the vehicle or objects, [0080] shifts in the time domain to simulate small transit time differences such as those typically caused by random influences in sound fields, [0081] masking of time values or frequency lines to simulate a loss of information on the transmission path, [0082] combining object backscatter from different objects to simulate the presence of multiple objects. [0083] part of this procedure according to the invention is the linking of augmentation methods both in the time domain before training and in the frequency domain during training.