Lossy data compressor for vehicle control systems

Abstract

A lossy data compressor for physical measurement data, comprising a parametrized mapping network hat, when applied to a measurement data point x in a space X, produces a point z in a lower-dimensional manifold Z, and configured to provide a point z on manifold Z as output in response to receiving a data point x as input, wherein the manifold Z is a continuous hypersurface that only admits fully continuous paths between any two points on the hypersurface; and the parameters θ of the mapping network are trainable or trained towards an objective that comprises minimizing, on the manifold Z, a distance between a given prior distribution P.sub.Z and a distribution P.sub.Q induced on manifold Z by mapping a given set P.sub.D of physical measurement data from X onto Z using the mapping network, according to a given distance measure.

Claims

1. A sensor arrangement configured to provide compressed physical measurement data characterizing an environment sensed by the sensor arrangement, the sensor arrangement comprising: a sensor, wherein the sensor is configured to sense the environment; and a lossy data compressor comprising a processor arranged for obtaining from the sensor raw physical measurement data produced by the sensor and characterizing the sensed environment, wherein the sensor arrangement is configured to generate the compressed physical measurement data by the a processor executing as a parametrized neural network that, when applied to a data point x of the physical measurement data that lives in a space X, produces a point z in a predefined Riemannian manifold Z that has a lower dimensionality than the space X and that provides the point z on the manifold Z as output in response to receiving the data point x as input; wherein: the manifold Z is a continuous hypersurface that only admits fully continuous paths between any two points on the hypersurface; parameters θ of the neural network are trainable or trained towards an objective that comprises minimizing, on the manifold Z, a distance measure, with respect to both a distance between distributions and a distance between individual points, between a given prior distribution P.sub.Z and a distribution P.sub.Q induced on manifold Z by mapping a given set P.sub.D of physical measurement data from space X onto manifold Z using the neural network; and the processor is configured to output the generated compressed physical measurement data.

2. A system for classifying objects or situations in an environment of a vehicle, comprising: at least one sensor that is configured to obtain physical measurement data defined within a space X from at least part of the environment of the vehicle; at least one lossy data compressor that is: communicatively coupled to the at least one sensor to receive the physical measurement data from the at least one sensor; and configured to produce output based on the physical measurement data received from the at least one sensor; and a classifier module communicatively coupled to the lossy data compressor and configured to classify the output of the lossy data compressor as including an indication of presence of at least one predefined object or at least one predefined situation in the environment of the vehicle; and a decision module; wherein: the decision module is configured to determine, based on the classification, a modified trajectory of the vehicle or a modified status of an automated driving functionality of the vehicle; the lossy data compressor includes a parametrized neural network that, when applied to a data point x of the physical measurement data, produces a point z in a predefined input Riemannian manifold Z that has a lower dimensionality than the space X and that provides the point z on the manifold Z as part of the output in response to receiving the data point x as input; the manifold Z is a continuous hypersurface that only admits fully continuous paths between any two points on the hypersurface; and parameters θ of the neural network are trainable or trained towards an objective that comprises minimizing, on the manifold Z, a distance measure, with respect to both a distance between distributions and a distance between individual points, between a given prior distribution P.sub.Z and a distribution P.sub.Q induced on manifold Z by mapping a given set P.sub.D of physical measurement data from space X onto manifold Z using the neural network.

3. The system of claim 2, wherein the manifold Z is a hypersphere or a hyperellipsoid.

4. The system of claim 2, wherein the distance measure comprises a Wasserstein distance.

5. The system of claim 2, wherein the neural network comprises at least two instances of a single same sandwich structure that includes a plurality of different layers and wherein each of the layers is configured to perform at least one of calculation of an exponential function, matrix multiplication, element-wise division of matrices, and computation of a Frobenius product.

6. The system of claim 2, wherein the given prior distribution P.sub.Z comprises at least two distinct clusters on the manifold Z.

7. The system of claim 2, wherein the classifier module is communicatively coupled to the lossy data compressor via a shared-medium bus network to which further systems of the vehicle are coupled, and each of the at least one lossy data compressor is communicatively coupled to a respective one or more of the at least one sensor via a dedicated broadband connection.

8. The system of claim 2, wherein: the decision module is configured to determine, based on the classification, whether it is necessary to change from a prior trajectory of the vehicle, or to at least partially deactivate the automated driving functionality, to avoid adverse consequences for the vehicle, the driver of the vehicle, or another entity predicted to be caused by the predefined object or situation in the environment of the vehicle; and the system further comprises an actuation module communicatively coupled with the decision module and configured to, in response to said determination of the necessity being positive, actuate a power train of the vehicle, and/or actuate a braking system of the vehicle, and/or actuate a steering system of the vehicle, and/or actuate a warning device of the vehicle to emit a warning that is physically perceptible by a driver of the vehicle, and/or cause at least partial deactivation of the automated driving functionality of the vehicle.

9. A method for operating a vehicle comprising: providing a lossy data compressor in the vehicle; the lossy data compressor of the vehicle producing output based on physical measurement data defined within a space X and received from at least one sensor that produces the physical measurement data by sensing an environment of the vehicle; a classifier module of the vehicle classifying the output produced by the lossy data compressor as including an indication of presence of at least one predefined object or at least one predefined situation in the environment of the vehicle; and a decision module of the vehicle providing, based on the classification, a modified trajectory of the vehicle or a modified status of an automated driving functionality of the vehicle; wherein: the lossy data compressor including a parametrized neural network that, when applied to each of a plurality of data points x of the physical measurement data, produces respective corresponding points z in a predefined input Riemannian manifold Z that has a lower dimensionality than the space X and that provides the points z on the manifold Z as part of the output in response to receiving the data points x as input; the manifold Z is a continuous hypersurface that only admits fully continuous paths between any two points on the hypersurface; parameters θ of the neural network are trainable or trained towards an objective that comprises minimizing, on the manifold Z, a distance measure, with respect to both a distance between distributions and a distance between individual points, between a given prior distribution P.sub.Z and a distribution P.sub.Q induced on manifold Z by mapping a given set P.sub.D of physical measurement data from space X onto manifold Z using the neural network; and the providing of the lossy data compressor includes: setting up an objective function for the minimizing of the distance measure, the objective function being a weighted sum of a distance and an entropy of a doubly stochastic matrix P specifying a probability that a randomly generated label belongs to one of the data points x in P.sub.D; and iteratively performing the following until a predetermined termination criterion is satisfied: minimizing the objective function with respect to the doubly stochastic matrix P to find an optimal matrix P; and minimizing the objective function with respect to the parameters θ of the neural network to find optimal parameters θ of the neural network.

10. The method of claim 9, wherein the objective function comprises a Frobenius dot product of the doubly stochastic matrix P and a cost matrix C that assigns a cost value to every combination of a data point x.sub.i in P.sub.D and a corresponding feature z.sub.j on the manifold Z.

11. The method of claim 10, wherein the cost value corresponds to a Euclidean or geodesic absolute distance between the point on the manifold Z to which the data point x.sub.i is mapped and the corresponding feature z.sub.i.

12. The method of claim 10, wherein the minimizing with respect to the matrix P is performed using a Sinkhorn algorithm.

13. The method of claim 9, wherein the minimizing with respect to the parameters θ of the neural network is performed by stochastic gradient descent on the parameters θ of the neural network.

14. At least one non-transitory machine readable storage medium on which are stored instructions that are executable by at least one processor and that, when executed by the at least one processor, cause the at least one processor to perform a method for a vehicle, the method comprising: receiving from at least one sensor physical measurement data that is defined within a space X and that produced by the one sensor by sensing at least part of an environment of the vehicle; producing output based on the physical measurement data received from the at least one sensor; classifying the output as including an indication of presence of at least one predefined object or at least one predefined situation in the environment of the vehicle; and providing, based on the classification, a modified trajectory of the vehicle or a modified status of an automated driving functionality of the vehicle; wherein: the production of the output includes executing a parametrized neural network that, when applied to a data point x of the physical measurement data, produces a point z in a predefined Riemannian manifold Z that has a lower dimensionality than the space X and that provides the point z on the manifold Z as part of the output in response to receiving a data point x as input; wherein: the manifold Z is a continuous hypersurface that only admits fully continuous paths between any two points on the hypersurface; and parameters θ of the mapping network are trainable or trained towards an objective that comprises minimizing, on the manifold Z, a distance measure, with respect to both a distance between distributions and distance between individual points, between a given prior distribution P.sub.Z and a distribution P.sub.Q induced on manifold Z by mapping a given set P.sub.D of physical measurement data from space X onto manifold Z using the neural network.

15. A method of manufacturing a sensor arrangement configured to provide compressed physical measurement data characterizing an environment sensed by the sensor arrangement, the method comprising: providing a processor that executes as a parametrized lossy data compressing neural network that, when applied to each of a plurality of data points x of raw physical measurement data that lives in a space X, produces, as the compressed physical measurement data, respective corresponding points z in a predefined input Riemannian manifold Z that has a lower dimensionality than the space X and that provides the points z on the manifold Z as part of output in response to receiving the data points x as input; provide a sensor that includes hardware to sense the environment and thereby generate the raw physical measurement data characterizing the sensed environment; and communicatively couple the processor to the sensor for the processor to obtain the raw physical measurement data from the sensor in order to produce the compressed physical measurement data using the raw physical measurement data; wherein: the manifold Z is a continuous hypersurface that only admits fully continuous paths between any two points on the hypersurface; parameters θ of the neural network are trainable or trained so that the processor, executing as the lossy data compressing neural network, minimizes, on the manifold Z, a distance measure, with respect to both a distance between distributions and a distance between individual points, between a given prior distribution P.sub.Z and a distribution P.sub.Q induced on manifold Z by mapping a given set P.sub.D of physical measurement data from space X onto manifold Z; and the providing of the processor that executes as the lossy data compressing neural network includes: setting up an objective function for the minimizing of the distance measure, the objective function being a weighted sum of a distance and an entropy of a doubly stochastic matrix P specifying a probability that a randomly generated label belongs to one of the data points x in P.sub.D; and the processor iteratively performing the following until a predetermined termination criterion is satisfied: minimizing the objective function with respect to the doubly stochastic matrix P to find an optimal matrix P; and minimizing the objective function with respect to the parameters θ of the neural network to find optimal parameters θ of the neural network.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 shows an exemplary embodiment of the lossy data compressor 1.

(2) FIG. 2 shows an exemplary neural network 2a within mapping network 2 of lossy data compressor 1.

(3) FIG. 3 shows an illustration of the advantage of preserving semantic similarity.

(4) FIG. 4 shows an exemplary classification system 51 in vehicle 50.

(5) FIG. 5 shows an exemplary embodiment of the method 100.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

(6) FIG. 1 illustrates an exemplary embodiment of the lossy data compressor 1. A mapping module 2, parametrized by a tuple of parameters θ, maps data points x from a training set P.sub.D, which is a subset of the space X, to points z on lower-dimensional manifold Z. All physical measurement data 3 live in the space X. The manifold Z is defined by a given prior P.sub.Z. In the example illustrated in FIG. 1, the manifold Z is a sphere, and the prior P.sub.Z is a random distribution of points scattered across the surface of this sphere.

(7) All points z that are the result of mapping a data point x from the set P.sub.D make up the set z(P.sub.D). However, the distribution P.sub.Q induced by the mapping includes many more points (drawn as hollow circles) over and above the concrete set z(P.sub.D).

(8) FIG. 2 shows an exemplary neural network 2a in a mapping network 2. The actual mapping of a data point x to a data point z is performed by layers 20a-20e drawn in dashed contours. These layers 20a-20e are parametrized by the tuple of parameters θ.

(9) In addition, the neural network 2a comprises two instances 21a and 21b of layer sandwiches. The layers in these sandwiches are fixed (i.e., they do not depend on the parameters θ) and are therefore drawn in solid contours. The first sandwich 21a comprises layers 22a, 23a and 24a. The second sandwich 21b comprises layers 22b, 23b and 24b. As denoted by the identical shadings, layer 22b performs the same functionality as layer 22a, layer 23b performs the same functionality as layer 23a, and layer 24b performs the same functionality as layer 24a. The combination of the layer sandwiches 21a and 21b is used to compute the error signal E that is a measure for the entropy-relaxed Wasserstein distance between the distribution P.sub.Q induced on Z and the prior distribution P.sub.Z.

(10) During training of the neural network 2a, the parameters θ are adjusted to minimize the error signal E.

(11) FIG. 3a shows an exemplary set P.sub.D of physical measurement data 3 in the form of handwritten numbers x. FIGS. 3b to 3d show different mappings of the data points x to lower-dimensional manifold.

(12) FIG. 3b shows a first mapping onto a band-shaped lower-dimensional manifold Y. The manifold has been assigned reference sign Y instead of Z because it does not fulfill the condition that it only admits fully continuous paths between any two points on this manifold Y. Rather, a path from one point to another point may lead to areas that are off the band. The mapping to this manifold Y is not well-behaved because points relating to different numbers appear in a wild mixture.

(13) FIG. 3c shows a second mapping onto a spherical lower-dimensional manifold Z. Here, there are no discontinuous paths between points on the manifold Z that leave the manifold. Consequently, the surface is smooth. However, the points z relating to different numbers are still mixed.

(14) In FIG. 3d, the same manifold Z is used. In addition, the mapping has been performed with the objective that the induced distribution P.sub.Q shall match the given prior distribution P.sub.Z. Consequently, the points z appear clustered on the manifold Z according to the numbers to which they relate. What is more, it appears that distances in the original space X have been preserved after mapping to the manifold Z, and that semantic similarity has been preserved in the form of “closeness” on the manifold Z: The instance of the number “1” that is most similar to a “7” is quite close to the corresponding instance of the number “7” that is most similar to a “1”.

(15) FIG. 4 shows an exemplary vehicle 50 that is fitted with a system 51 for classifying objects or situations in its environment. The vehicle 50 is fitted with four camera sensors 52a-52d that monitor different parts 53a-53d of the environment of the vehicle 50. Each sensor 52a-52d is connected to a corresponding lossy data compressor 1a-1d via a respective dedicated broadband connection 56a-56d and receives physical measurement data 3 over this connection 56a-56d.

(16) The vehicle 50 is equipped with a CAN bus as a shared medium bus network 55. The lossy data compressors 1a-1d are connected to this network 55 and use it to forward the compressed data to the classifier module 54. The classifier module 54 forwards the result of the classification to the decision module 57 via the network 55. If the decision module 57 finds that there is a need to change the trajectory 50a of the vehicle 50, or to at least partially disable automated driving functionality, the actuation module 58 is notified. The actuation module 58 then actuates the power train 59a, the braking system 59b, and/or the steering system 59c, to this effect. The actuation module 58 may also actuate a warning device 59d of the vehicle 50 to emit a warning 59e that is physically perceptible by the driver of the vehicle 50.

(17) The figure of the classifier module 54, the decision module 57 and the actuation module 58 in distinct places within the vehicle 50 is not limiting in the sense that such placing is required. Rather, these modules may also be combined, e.g., into one single control unit. The exemplary placing in FIG. 4 is illustrative to show that the presence of the shared-medium bus network 55 in the vehicle 50 provides a high degree of freedom as to the placement of components.

(18) FIG. 5 shows an exemplary embodiment of the method 100 for manufacturing the lossy data compressor 1. The main aspect of this manufacturing is the training of the mapping network 2 within the lossy data compressor 1.

(19) In a first step 110, an objective function is set up for the minimizing of the distance between the prior distribution P.sub.Z and the distribution P.sub.Q induced on manifold Z by the mapping of the physical measurement data set P.sub.D. This objective function is minimized in the following in an alternating manner.

(20) In step 120, the objective function is minimized with respect to the doubly stochastic matrix P to find an optimal matrix P.

(21) In step 130, the objective function is minimized with respect to the parameters θ of the mapping network to find optimal parameters θ of the mapping network.

(22) It is then checked in diamond 140a whether a predetermined termination criterion is satisfied. Such a termination criterion may, for example, be formulated in terms of a threshold for the absolute value of the objective function, or for the relative change of the value of the objective function from one iteration to the next. If the termination criterion is met (logical value 1), the method 100 terminates. If the termination criterion is not met (logical value 0), then, in step 140, the method 100 branches back to the minimizing with respect to P according to step 120.