DEVICE AND METHOD FOR CLASSIFYING IMAGES AND ACCESSING THE ROBUSTNESS OF THE CLASSIFICATION

Abstract

A computer-implemented method for determining an output signal characterizing a first classification of an input image into a class from a plurality of classes. The output signal further characterizes a second classification of a robustness of the first classification against an attack with an adversarial patch.

Claims

1-15. (canceled)

16. A computer-implemented method for determining an output signal characterizing a first classification of an input image into a class from a plurality of classes, wherein the output signal further characterizes a second classification of a robustness of the first classification against an attack with an adversarial patch, wherein determining the output signal comprises the following steps: determining a plurality of first score maps, wherein each of the first score maps corresponds to a respective class from the plurality of classes and includes a plurality of area classifications, wherein each of the area classifications characterizes for an area of a plurality of areas of the image whether the area belongs to the class or not; determining a plurality of first class scores, wherein each first class score of the plurality of first class scores corresponds to a class from the plurality of classes and is determined by aggregating the area classifications of the first score map corresponding to the class that characterize a classification into the class; determining a second class score, wherein the second class score is a minimum class score that can be obtained when an adversarial patch is applied to the input image for changing the area classification of the first score map corresponding to a first class that corresponds to a largest first class score; determining a plurality of third class scores, wherein each of the third class scores corresponds to a class different from the first class and is a maximum class score that can be obtained when the adversarial patch is applied to the input image for changing the area classifications of the first score map of the class; determining, based on the second class score being larger than or equal to all of the third class scores, the output signal such that the output signal characterizes the first classification of the input image based on the plurality of first class scores and that the output signal characterizes the second classification of the first classification as robust against the adversarial patch; determining, based on the second class score being smaller than at least one of the third class scores, the output signal such that the output signal characterizes the first classification of the input image based on the plurality of first class scores and that the output signal characterizes a second classification of the first classification as not robust against the adversarial patch.

17. The method according to claim 16, wherein the aggregating the area classifications is achieved by determining the sum of area classifications that characterize the classification of the class that the score map corresponds with.

18. The method according to claim 17, wherein the aggregating is achieved by a neural network, wherein the neural network is configured to accept the score map as input and provide the class score as output and wherein the neural network includes only positive weights and non-decreasing activation functions.

19. The method according to claim 17, further comprising: training the neural network, wherein the training of the neural network is achieved by adversarial training.

20. The method according to claim 17, wherein in the step of determining the second class score, the second class score is determined by subtracting a maximum amount of area classifications the adversarial patch can change from the largest first class score.

21. The method according to claim 17, wherein in the step of determining the plurality of third class scores, a third class score is a determined by adding a maximum amount of area classifications the adversarial patch can change to a first class score.

22. The method according to claim 16, wherein the determining of the second class score includes: determining a plurality of second score maps for the class corresponding to the largest first class score, wherein for each possible position of an adversarial patch in the input image a second score map is determined, wherein for each second score map, an area classification is determined to not characterize the class when the adversarial patch extends into the area corresponding to the area classification; aggregating each second score map to determine a plurality of intermediate class scores; providing the smallest intermediate class score from the plurality of intermediate class scores as the second class score.

23. The method according to claim 16, wherein in the step of determining the plurality of third class scores, each of the third class scores is determined by: determining a plurality of second score maps for the class corresponding to the third class score, wherein for each possible position of an adversarial patch in the input image a second score map is determined, wherein for each second score map an area classification is determined to characterize the class if the adversarial patch extends into the area corresponding to the area classification; aggregate each second score map to determine a plurality of intermediate class scores; providing a largest intermediate class score from the plurality of intermediate class scores as the second class score.

24. The method according to claim 22, wherein the plurality of first score maps and/or the plurality of second score maps is determined by a convolutional neural network.

25. The method according to claim 22, wherein the convolutional neural network determines each first score map by determining an output of an output layer of the convolutional neural network for the input image and applying a Heaviside function to the output.

26. The method according to claim 24, wherein the method further includes training the convolutional neural network, wherein training includes: determining a plurality of first score maps for a training image by providing the image to the convolutional neural network; determining a plurality of first class scores by aggregating each first score map from the previously determined plurality of score maps; determining a loss value $l=\underset{c\ne c_t}{\max }\max \left(0,{\hat{y}}_c+m-{\hat{y}}_{c_t}\right),$ wherein ŷ.sub.c is a c-th class score of the plurality of first class scores, m is a value characterizing a predefined margin and ŷ.sub.c.sub.t is the first class score for a desired class of the training image (x.sub.i); determining a gradient for a plurality of parameters of the convolutional neural network using backpropagation, wherein the convolutional neural network includes a Heaviside function, a gradient of the Heaviside function is replaced by a gradient of a Sigmoid function; adapting the parameters of the convolutional neural network according to the gradient.

27. The method according to claim 16, wherein a device is controlled in accordance with the output signal.

28. An image classifier configured to determine an output signal characterizing a first classification of an input image into a class from a plurality of classes, wherein the output signal further characterizes a second classification of a robustness of the first classification against an attack with an adversarial patch, wherein the image classifier is configured to: determine a plurality of first score maps, wherein each of the first score maps corresponds to a respective class from the plurality of classes and includes a plurality of area classifications, wherein each of the area classifications characterizes for an area of a plurality of areas of the image whether the area belongs to the class or not; determine a plurality of first class scores, wherein each first class score of the plurality of first class scores corresponds to a class from the plurality of classes and is determined by aggregating the area classifications of the first score map corresponding to the class that characterize a classification into the class; determine a second class score, wherein the second class score is a minimum class score that can be obtained when an adversarial patch is applied to the input image for changing the area classification of the first score map corresponding to a first class that corresponds to a largest first class score; determine a plurality of third class scores, wherein each of the third class scores corresponds to a class different from the first class and is a maximum class score that can be obtained when the adversarial patch is applied to the input image for changing the area classifications of the first score map of the class; determine, based on the second class score being larger than or equal to all of the third class scores, the output signal such that the output signal characterizes the first classification of the input image based on the plurality of first class scores and that the output signal characterizes the second classification of the first classification as robust against the adversarial patch; and determine, based on the second class score being smaller than at least one of the third class scores, the output signal such that the output signal characterizes the first classification of the input image based on the plurality of first class scores and that the output signal characterizes a second classification of the first classification as not robust against the adversarial patch; wherein the image classifier comprises: a convolutional neural network; and an aggregation unit which is configured to determine an aggregation of a score map.

29. A non-transitory machine-readable storage medium on which is stored a computer program for determining an output signal characterizing a first classification of an input image into a class from a plurality of classes, wherein the output signal further characterizes a second classification of a robustness of the first classification against an attack with an adversarial patch, wherein the computer program, when executed by a computer, determines the output signal by: determining a plurality of first score maps, wherein each of the first score maps corresponds to a respective class from the plurality of classes and includes a plurality of area classifications, wherein each of the area classifications characterizes for an area of a plurality of areas of the image whether the area belongs to the class or not; determining a plurality of first class scores, wherein each first class score of the plurality of first class scores corresponds to a class from the plurality of classes and is determined by aggregating the area classifications of the first score map corresponding to the class that characterize a classification into the class; determining a second class score, wherein the second class score is a minimum class score that can be obtained when an adversarial patch is applied to the input image for changing the area classification of the first score map corresponding to a first class that corresponds to a largest first class score; determining a plurality of third class scores, wherein each of the third class scores corresponds to a class different from the first class and is a maximum class score that can be obtained when the adversarial patch is applied to the input image for changing the area classifications of the first score map of the class; determining, based on the second class score being larger than or equal to all of the third class scores, the output signal such that the output signal characterizes the first classification of the input image based on the plurality of first class scores and that the output signal characterizes the second classification of the first classification as robust against the adversarial patch; determining, based on the second class score being smaller than at least one of the third class scores, the output signal such that the output signal characterizes the first classification of the input image based on the plurality of first class scores and that the output signal characterizes a second classification of the first classification as not robust against the adversarial patch.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0065] FIG. 1 an image classifier, according to an example embodiment of the present invention.

[0066] FIG. 2 shows a control system comprising the image classifier controlling an actuator in its environment, according to an example embodiment of the present invention.

[0067] FIG. 3 shows the control system controlling an at least partially autonomous robot, according to an example embodiment of the present invention.

[0068] FIG. 4 shows the control system controlling an automated personal assistant, according to an example embodiment of the present invention.

[0069] FIG. 5 shows the control system controlling a medical analysis system, according to an example embodiment of the present invention.

[0070] FIG. 6 shows a training system for training the image classifier, according to an example embodiment of the present invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0071] Shown in FIG. 1 is an embodiment of an image classifier (60) for determining a first classification (c) of an input image (x) and a second classification (r) characterizing a robustness of the first classification (c).

[0072] The image classifier (60) receives the input image (x) in a score map unit (61). The score map unit (61) is configured to determine a plurality of score maps (M) from the input image (x). Preferably, the score map unit (61) comprises a convolutional neural network, which takes the input image (x) as input and provides the plurality of score maps (M). For this, the neural network may be configured to provide a tensor representing the plurality of score maps (M), wherein the tensor is of a predefined height, width and depth. Each matrix of the tensor along the depth dimension may represent a score map of the plurality of score maps, wherein each dimension corresponds to a specific class. In other words, each matrix along the depth dimension of the tensor may represent the score map for a specific class. The tensor (M) comprises area classifications (i.e., the elements of the tensor (M)), which may be 0 or 1. A 0 may indicate that an area classification characterizes an area as not belonging to the class corresponding to the score map the area classification belongs to. Likewise, a 1 may indicate that an area classification characterizes an area as belonging to the class corresponding to the score map the area classification belongs to. In other words, the tensor (M) may be a tensor of zeros and ones. In further embodiments, this dichotomous classification may also be expressed by two other values, e.g., as binary variables (true or false). In even further embodiments, the tensor (M) may also comprise values in the interval from (and including) 0 to 1, e.g., probability values. Preferably, the receptive field of the output layer of the convolutional neural network that provides the tensor covers a small area of the input image (x). For example, the convolutional neural network may be chosen such that the final layer has a receptive field of 9-by-9, 17-by-17 or 33-by-33 pixels in the input image (x).

[0073] Preferably, the convolutional neural network comprises a Heaviside function as activation function in the output layer providing the tensor of score maps (M). This way, the tensor (M) comprises only zeros and ones. In further embodiments, it can be imagined that a different activation or no activation is used in the output layer and that the result of the output layer is then compared against a predefined threshold to determine the tensor or score maps (M).

[0074] The tensor (M) is then forwarded to a classification unit (63), which aggregates the score maps of the tensor (M). In the embodiment, the classification unit (63) performs the aggregation by summing the elements of each matrix along the depth of the tensor, i.e., by determining the sum of each score map in the tensor (M). The result is a plurality of first class scores (c). The first class scores (c) may be understood as characterizing a first classification of the input image (x). In further embodiments, instead of summing the score maps, a second convolutional neural network may be used, which takes the tensor (M) as input and provides the plurality of first class scores (c). In these further embodiments, the second convolutional neural network is configured to resemble a non-decreasing function. That means, that the second convolutional neural network only employs non-decreasing functions as activation functions and comprises only positive weights.

[0075] The tensor (M) is also forwarded to a robustness unit (62). The robustness unit (62) is configured to determine, whether the classification characterized by the first class scores (c) can be altered if an adversarial patch is applied to the input image (x) or not. Here, applying an adversarial patch to the input image (x) may be understood as either changing the input image (x) directly by overlaying it with the adversarial patch. Alternatively, applying an adversarial patch to the input image (x) may also be understood as applying the adversarial patch to the scene the input image (x) is being sensed from.

[0076] The robustness unit (62) takes the tensor (M) as input and provides a robustness classification (r) of the plurality of first class scores (c). For this, the robustness unit determines a maximum amount of area classifications the adversarial patch can change. In the embodiment, this is achieved by determining the amount of area classifications of a score map the adversarial patch can change. As each area classifications depend only on an area of the input image (x), the adversarial patch can only change the area classification if it extends into the area or covers the area. The maximum amount of area classification of a score map the adversarial patch can change is hence equal to an amount of areas the adversarial patch can cover or extend into in the input image (x). Having obtained the amount, the robustness unit (62) determines a second classification (r), wherein the second classification (r) classifies the first classification as robust if the largest first class score from the plurality of first class scores (c) is larger than or equal to the sum of the second largest first class score from the plurality of first class scores (c) and twice the maximum amount. If the largest first class score from the plurality of first class scores (c) is smaller than the sum of the second largest first class score from the plurality of first class scores (c) and twice the maximum amount, the second classification (r) is determined to characterize the first classification as non-robust.

[0077] The plurality of first class scores (c) and the robustness classification (r) may then be provided as an output signal (y) of the image classifier (60). In further embodiments, an optional conversion unit (64) of the image classifier (60) may process the first class scores (c) and/or the robustness classification (r) and output the result as output signal (y). For example, it can be imagined that instead of the plurality of class scores (c), a class corresponding to the largest class score is provided in the output signal (y) along with the robustness classification (r). Alternatively, the conversion unit may provide the plurality of first class scores (c) or the class as output signal (y) if the robustness classification (r) characterizes a robust classification. In this case, the robustness classification (r) characterizes a non-robust classification, the output signal (y) may be set to characterize no definitive classification result. This may, for example, be in the form of setting the output signal (y) such that it characterizes a rejected classification.

[0078] Shown in FIG. 2 is an embodiment of an actuator (10) in its environment (20). The actuator (10) interacts with a control system (40). The actuator (10) and its environment (20) will be jointly called actuator system. At preferably evenly spaced points in time, a sensor (30) senses a condition of the actuator system. The sensor (30) may comprise several sensors. Preferably, the sensor (30) is an optical sensor that takes images of the environment (20). An output signal (S) of the sensor (30) (or, in case the sensor (30) comprises a plurality of sensors, an output signal (S) for each of the sensors) which encodes the sensed condition is transmitted to the control system (40).

[0079] Thereby, the control system (40) receives a stream of sensor signals (S). It then computes a series of actuator control commands (A) depending on the stream of sensor signals (S), which are then transmitted to the actuator (10).

[0080] The control system (40) receives the stream of sensor signals (S) of the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signals (S) into input images (x). Alternatively, in case of no receiving unit (50), each sensor signal (S) may directly be taken as an input image (x). The input image (x) may, for example, be given as an excerpt from the sensor signal (S). Alternatively, the sensor signal (S) may be processed to yield the input image (x). The input image (x) comprises image data corresponding to an image recorded by the sensor (30). In other words, the input image (x) is provided in accordance with the sensor signal (S).

[0081] The input image (x) is then passed on to the image classifier (60).

[0082] The image classifier (60) is parametrized by parameters (□□, which are stored in and provided by a parameter storage (St.sub.1).

[0083] The image classifier (60) determines an output signal (y) from the input images (x). The output signal (y) comprises information that assigns one or more labels to the input image (x). The output signal (y) is transmitted to an optional conversion unit (80), which converts the output signal (y) into the control commands (A). The actuator control commands (A) are then transmitted to the actuator (10) for controlling the actuator (10) accordingly. Alternatively, the output signal (y) may directly be taken as actuator control commands (A).

[0084] The actuator (10) receives actuator control commands (A), is controlled accordingly and carries out an action corresponding to the actuator control commands (A). The actuator (10) may comprise a control logic which transforms an actuator control command (A) into a further control command, which is then used to control actuator (10).

[0085] In further embodiments, the control system (40) may comprise a sensor (30). In even further embodiments, the control system (40) alternatively or additionally may comprise an actuator (10).

[0086] In further embodiments, it can be envisioned that the control system (40) controls a display (10a) instead of or in addition to the actuator (10). The display may, for example, display the classification characterized by the output signal (y) and/or whether the classification is robust.

[0087] Furthermore, the control system (40) may comprise a processor (45) (or a plurality of processors) and at least one machine-readable storage medium (46) on which instructions are stored which, if carried out, cause the control system (40) to carry out a method according to one aspect of the invention.

[0088] FIG. 3 shows an embodiment in which the control system (40) is used to control an at least partially autonomous robot, e.g., an at least partially autonomous vehicle (100).

[0089] The sensor (30) may comprise one or more video sensors and/or one or more radar sensors and/or one or more ultrasonic sensors and/or one or more LiDAR sensors and or one or more position sensors (like e.g. GPS). Some or all of these sensors are preferably but not necessarily integrated in the vehicle (100).

[0090] Alternatively or additionally, the sensor (30) may comprise an information system for determining a state of the actuator system. One example for such an information system is a weather information system which determines a present or future state of the weather in the environment (20).

[0091] The image classifier (60) may be configured to identify whether the vehicle is currently located in an urban environment, a rural environment or on a highway in order to decide whether an automatic operation of the vehicle should be allowed. For example, it can be imagined that automatic operation of the vehicle is only allowed if the vehicle is located on a highway. It can be further imagined that the automatic operation is only enabled if besides a classification of the environment as “on a highway” the output signal (y) of the image classifier (60) also indicates a robust classification, i.e., a classification that cannot be altered by an adversarial patch.

[0092] The actuator (10), which is preferably integrated in the vehicle (100), may be given by a brake, a propulsion system, an engine, a drivetrain, or a steering of the vehicle 100. Actuator control commands (A) may be determined such that the actuator (or actuators) (10) is/are controlled such that vehicle (100) avoids collisions with the detected objects. The detected objects may also be classified according to what the classifier (60) deems them most likely to be, e.g., pedestrians or trees, and the actuator control commands (A) may be determined depending on the classification.

[0093] In further embodiments, the at least partially autonomous robot may be given by another mobile robot (not shown), which may, for example, move by flying, swimming, diving or stepping. The mobile robot may, inter alia, be an at least partially autonomous lawn mower, or an at least partially autonomous cleaning robot. In all of the above embodiments, the actuator command control (A) may be determined such that propulsion unit and/or steering and/or brake of the mobile robot are controlled such that the mobile robot may avoid collisions with said identified objects.

[0094] Shown in FIG. 4 is an embodiment in which the control system (40) is used for controlling an automated personal assistant (250). The sensor (30) may be an optic sensor, e.g., for receiving video images of a gestures of a user (249). Alternatively, the sensor (30) may also be an audio sensor, e.g., for receiving a voice command of the user (249).

[0095] The control system (40) then determines actuator control commands (A) for controlling the automated personal assistant (250). The actuator control commands (A) are determined in accordance with the sensor signal (S) of the sensor (30). The sensor signal (S) is transmitted to the control system (40). For example, the image classifier (60) may be configured to, e.g., carry out a gesture recognition based on at least one image of the user (249). The control system (40) may then determine an actuator control command (A) for transmission to the automated personal assistant (250). It then transmits the actuator control command (A) to the automated personal assistant (250).

[0096] For example, the actuator control command (A) may be determined in accordance with the identified user gesture recognized by the image classifier (60). It may comprise information that causes the automated personal assistant (250) to retrieve information from a database and output this retrieved information in a form suitable for reception by the user (249).

[0097] In further embodiments, it may be envisioned that instead of the automated personal assistant (250), the control system (40) controls a domestic appliance (not shown) controlled in accordance with the identified user gesture. The domestic appliance may be a washing machine, a stove, an oven, a microwave or a dishwasher.

[0098] Shown in FIG. 5 is an embodiment of a medical analysis system (600) being controlled by the control system (40). The medical analysis system (600) is supplied with a microarray (601), wherein the microarray comprises a plurality of spots (602, also known as features) which have been exposed to a medical specimen. The medical specimen may, for example, be a human specimen or an animal specimen, e.g., obtained from a swab.

[0099] The microarray (601) may be a DNA microarray or a protein microarray.

[0100] The sensor (30) is configured to sense the microarray (601). The sensor (30) is preferably an optical sensor such as a video sensor.

[0101] The image classifier (60) is configured to classify a result of the specimen based on an input image (x) of the microarray supplied by the sensor (30). In particular, the image classifier (60) may be configured to determine whether the microarray (601) indicates the presence of a virus in the specimen.

[0102] The control signal (A) may then be chosen such that the display (10a) shows the result of the classification.

[0103] FIG. 6 shows an embodiment of a training system (140) for training the image classifier (60) of the control system (40) by means of a training data set (T). The training data set (T) comprises a plurality of input images (x.sub.i) which are used for training the classifier (60), wherein the training data set (T) further comprises, for each input image (x.sub.i), a desired output signal (y.sub.i), which corresponds to the input image (x.sub.i) and characterizes a desired classification of the input image (x.sub.i).

[0104] For training, a training data unit (150) accesses a computer-implemented database (St.sub.2), the database (St.sub.2) providing the training data set (T). The training data unit (150) determines from the training data set (T) preferably randomly at least one input image (x.sub.i) and the desired output signal (y.sub.i) corresponding to the input image (x.sub.i) and transmits the input image (x.sub.i) to the image classifier (60). The image classifier (60) determines an output signal (ŷ.sub.i) based on the input image (x.sub.i) comprising at least a plurality of first class scores obtained for the input image (x.sub.i).

[0105] The desired output signal (y.sub.i) and the determined output signal (ŷ.sub.i) are transmitted to a modification unit (180).

[0106] Based on the desired output signal (y.sub.i) and the determined output signal (ŷ.sub.i), the modification unit (180) then determines new parameters (Φ′) for the image classifier (60). For this purpose, the modification unit (180) compares the desired output signal (y.sub.i) and the determined output signal (ŷ.sub.i) using a loss function. The loss function determines a first loss value that characterizes how far the determined output signal (ŷ.sub.i) deviates from the desired output signal (y.sub.i). In the given embodiment, a margin loss

[00003]$l=\underset{c\ne c_t}{\max }\max \left(0,{\hat{y}}_c+m-{\hat{y}}_{c_t}\right),$

is used as loss function, wherein ŷ.sub.c is the c-th class score of the plurality of first class scores determined from the image classifier (60), m is a value characterizing a predefined margin and ŷ.sub.c.sub.t is the first class score for a desired class of the training image (x.sub.i).

[0107] In further embodiments, other loss functions may be used, especially conventional loss functions for multiclass classification, e.g., multinomial cross entropy loss.

[0108] The modification unit (180) determines the new parameters (Φ′) based on the first loss value. In the given embodiment, this is done using a gradient descent method, preferably stochastic gradient descent, Adam, or AdamW. If the image classifier (60) comprises a Heaviside function, the gradient of the Heaviside function is replaced with the gradient of a sigmoid function.

[0109] In other preferred embodiments, the described training is repeated iteratively for a predefined number of iteration steps or repeated iteratively until the first loss value falls below a predefined threshold value. Alternatively or additionally, it is also possible that the training is terminated when an average first loss value with respect to a test or validation data set falls below a predefined threshold value. In at least one of the iterations the new parameters (Φ′) determined in a previous iteration are used as parameters (Φ) of the classifier (60).

[0110] In this embodiment, the parameters (Φ) of the image classifier (60) to be updated are the trainable parameters of the convolutional neural network of the image classifier (60). In further embodiments, the image classifier (60) may comprise a second convolutional neural network whose parameters may also belong to the parameters to be updated of the image classifier (60), either in addition or alternatively to the parameters of the convolutional neural network.

[0111] Furthermore, the training system (140) may comprise at least one processor (145) and at least one machine-readable storage medium (146) containing instructions which, when executed by the processor (145), cause the training system (140) to execute a training method according to one of the aspects of the invention.

[0112] The term “computer” may be understood as covering any devices for the processing of pre-defined calculation rules. These calculation rules can be in the form of software, hardware or a mixture of software and hardware.

[0113] In general, a plurality can be understood to be indexed, that is, each element of the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements contained in the plurality. Preferably, if a plurality has N elements, wherein N is the number of elements in the plurality, the elements are assigned the integers from 1 to N. It may also be understood that elements of the plurality can be accessed by their index.