A VISION SYSTEM FOR A MOTOR VEHICLE AND A METHOD OF TRAINING

Abstract

A vision system (1) for a motor vehicle, includes a sensing device (40) adapted to sense data (99), a data processing device (7) adapted to process the sensed data (99), wherein the data processing device (7) includes a controller (30) adapted to switch a power mode of the data processing device (7) between a low-power mode and a high-power mode, wherein a high-power application (110) can be executed in the high-power mode. The data processing device (7), in a low-power mode, is adapted to execute a power mode classifier (20), and the power mode classifier (20) is adapted to classify any input data (99) from the sensing device (40) as requiring executing the high-power application (110) or not, and to output a high-power mode request (130) to the controller (30) to switch the power mode of the data processing device (7) from the low-power mode to the high-power mode in case the power mode classifier (20) has classified the input data (99) as requiring executing the high-power application (110).

Claims

1. A vision system for a motor vehicle, comprising: a sensing device adapted to sense data, a data processing device adapted to process the data, wherein the data processing device comprises a controller adapted to switch a power mode of the data processing device between a low-power mode and a high-power mode, wherein a high-power application can be executed in the high-power mode, wherein the data processing device, in the low-power mode, is adapted to execute a power mode classifier, and the power mode classifier is adapted to classify the data from the sensing device as requiring executing the high-power application or not, and to output a high-power mode request to the controller to switch the power mode of the data processing device from the low-power mode to the high-power mode in a case the power mode classifier has classified the data as requiring executing the high-power application.

2. The vision system as claimed in claim 1, further comprising the data processing device, in the low-power mode, is adapted to execute a low-power application, wherein the low-power application is separate from the power mode classifier.

3. The vision system as claimed in claim 1, further comprising the data processing device, in the low-power mode, is adapted to deactivate the high-power application which requires the high-power mode.

4. The vision system as claimed in claim 1, further comprising that the power mode classifier is adapted to output a low power request to the controller to switch the power mode of the data processing device from the high-power mode to the low-power mode on the basis of the input data in case the power mode classifier has classified the data as not requiring executing the high-power application.

5. The vision system as claimed in claim 1, further comprising the power mode classifier is adapted to output a high-power mode request after a designated amount of time.

6. The vision system as claimed in claim 1, further comprising the sensing device comprises at least one of: an imaging apparatus, a lidar device, a radar device.

7. The vision system as claimed in claim 1, further comprising the high-power mode is additionally externally activatable by a control signal from an external electronic control unit.

8. The vision system as claimed in claim 1, further comprising, the power mode classifier comprises at least one of: a deep neural network, a convolutional neural network, a boosting classifier, a support vector machine.

9. The vision system as claimed in claim 1, further comprising the data processing device comprises an energy efficient accelerator, in the form of a convolutional neural network accelerator, to execute the power mode classifier.

10. The vision system as claimed in claim 1, further comprising the power mode classifier is adapted to output the high-power mode request on the basis of at least two images recorded by the sensing device.

11. The vision system as claimed in claim 1, further comprising the controller is adapted to transmit a low-power application control signal to switch the low-power application or the power mode classifier on or off.

12. A method of training a power mode classifier for a vision system according to claim 1, further comprising the step of: generating ground truth data for training the power mode classifier, wherein the ground truth data comprise pieces of input sensor data corresponding to a motor vehicle and labels indicating for each piece of input sensor data whether it requires switching to the high power mode or not.

13. The method as claimed in claim 12, further comprising labelling the pieces of input sensor data using an algorithm without compute and heat constraints which can determine whether the motor vehicle underlying the input sensor data needs to be started or stopped.

14. The method as claimed in claim 12, further comprising the step of: based on the movement of the motor vehicle in the input sensor data, instances where the motor vehicle is stationary for a certain amount of time is selected and used as examples of when to switch between the power modes.

15. The method as claimed in claim 12 further comprising the step of: assuming that the low power mode should be activated for a fixed set of scenes and scenarios, manually labelling when to switch between the power modes.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In the following the invention shall be illustrated on the basis of preferred embodiments with reference to the accompanying drawings, wherein:

[0032] FIG. 1 shows a schematic diagram of a vision system according to the present invention; and

[0033] FIG. 2 shows a flow diagram of a vision method according to an embodiment of the present invention.

DETAILED DESCRIPTION

[0034] In FIG. 1, a vision system 1 mounted in a motor vehicle comprises a sensing device 40 for sensing data of the motor vehicle and/or a region surrounding 6 the motor vehicle. The sensing device 40 can be coupled to an on-board data processing device 7 adapted to process the sensed data received as input data 99 from the sensing device 40.

[0035] In this embodiment, the sensing device 40 comprises an imaging apparatus for capturing images of the region surrounding 6 the motor vehicle, for example a region in front of the motor vehicle. The imaging apparatus comprises one or more optical imaging devices 4, in particular cameras, preferably operating in the visible and/or infrared wavelength range, where infrared covers near IR with wavelengths below 5 microns and/or far IR with wave-lengths beyond 5 microns.

[0036] In some embodiments the imaging apparatus comprises a plurality of imaging devices 4 in particular forming a stereo imaging apparatus. In other embodiments only one imaging device 4 forming a mono imaging apparatus can be used.

[0037] In still other embodiments or additionally, the sensing device 40 and/or imaging apparatus comprises a lidar system to obtain depth data of the surrounding 6. The sensing device 40 can comprise a radar system to obtain information on the distance between the motor vehicle and objects in the surrounding 6 of the motor vehicle. In these embodiments, the data processing device 7 can be in communication with emitting devices for the emission of electromagnetic radiation, in particular of light and/or laser beams.

[0038] As shown in FIG. 1, the data processing device 7 is electrically connected with an electronic control unit 41. The electronic control unit 41 can be a central electronic control unit 41 of the motor vehicle and/or can provide the data processing device 7 with any input data. In particular, the input data from the electronic control unit 41 can comprise vehicle information, such as speed and/or position, time, and/or signals based on passenger and/or driver action. Preferably, the input data from the electronic control unit 41 comprises a control signal 98 to control the power mode of the data processing device 7.

[0039] The sensing device 40 and/or the electronic control unit 41 provides input data 99 and/or the control signal 98 to the data processing device 7 for further data processing as explained with reference to FIG. 2.

[0040] As shown in FIG. 1, the data processing device 7 preferably has access to, or comprises, a data memory 16 to store for example sensed data and/or images. The data processing device 7 and the memory device 16 are preferably realised in a second on-board electronic control unit (ECU) possible separate from the electronic control unit 41 and may be connected to the sensing device 40 and/or the imaging device 4 via a separate cable or a vehicle data bus. In another embodiment the second ECU and one or more of the imaging devices 4 can be integrated into a single unit, where a one box solution including the second ECU, the electronic control unit 41 and/or all imaging devices 4 can be preferred.

[0041] The data processing device 7 is preferably a digital device which is programmed or programmable and preferably comprises a microprocessor, a microcontroller, a digital signal processor (DSP), and/or a microprocessor part in a System-On-Chip (SoC) device. The data processing device 7, or part of its functions, can be realized by a System-On-Chip (SoC) device comprising, for example, FPGA, DSP, ARM and/or microprocessor functionality.

[0042] The data processing device 7 preferably can comprise an energy efficient accelerator, like a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC), or an FPGA and/or ASIC part in a System-On-Chip (SoC) device, for executing a power mode classifier 20. In particular, the power mode classifier 20 can be or comprise a deep neural network, a convolutional neural network, a boosting classifier, and/or a support vector machine executed in the energy efficient accelerator.

[0043] The data processing device 7 may comprise dedicated hardware for performing certain functions, for example for controlling the capture of images by the imaging apparatus 40, receiving the electrical signal 99 containing the image information from the imaging apparatus 40, rectifying or warping pairs of left/right images into alignment and/or creating disparity or depth images.

[0044] The data processing device 7 is advantageously adapted to activate adjust and/or control at least one safety device 19 through a control signal 25 in order to cause a corresponding action. The safety device 19 may comprise at least one active safety device and/or at least one passive safety device. In particular, the safety device 19 may comprise one or more of: at least one safety belt tensioner, at least one passenger airbag, one or more restraint systems such as occupant airbags, a hood lifter, an electronic stability system, at least one dynamic vehicle control system, such as a brake control system and/or a steering control system, a speed control system; a display device to display information relating to a detected object; a warning device adapted to provide a warning to a driver by suitable optical, acoustical and/or haptic warning signals.

[0045] All steps from imaging, image processing to possible activation or control of safety device 19 are performed automatically and continuously during driving in real time. The invention is applicable to autonomous driving, where the vehicle is an autonomous vehicle adapted to drive partly or fully autonomously or automatically, and driving actions of the driver are partially and/or completely replaced or executed by the vehicle.

[0046] FIG. 2 shows a flow diagram of a vision method for a vision system as for example described with reference to FIG. 1. The data processing device 7 can operate in a low-power mode or in a high-power mode as will be explained with reference to FIG. 2.

[0047] Image and data processing carried out by the processing de-vice 7 advantageously comprises the execution of a power mode classifier 20, a low-power application 100 and a high-power application 110.

[0048] Input data 99 can be transmitted to the high-power application 110, the low-power application 100 and/or the power mode classifier 20, preferably depending on the activated power mode.

[0049] The input data 99 can in particular comprise input of sensor data sensed by a sensing device 40 and/or control signals by an electronic control unit 41. The input data 99 can be obtained from, for example, a camera, a radar system, a lidar system, or any combination thereof. The input data 99 can comprise at least two images, i.e., multiple images, such as the current image and a previously captured image that is stored in a memory 16.

[0050] The high-power application 110 receives the input data 99, in particular if the high-power mode of the data processing device 7 is activated.

[0051] The high-power application 110 can comprise identifying and preferably also classifying possible objects and/or object candidates for example in front of the motor vehicle, such as pedestrians, other vehicles, bicyclists and/or large animals, tracking over time the position of objects and/or object candidates identified in the captured images, and activating or controlling the at least one safety device 19 depending on an estimation performed with respect to a identified, classified and/or tracked object and/or object candidate, for example on an estimated collision probability. The high-power application 110 can comprise any applications that are relevant for the motor vehicle and/or employ full functionality of the vision system 1. The full functionality can constitute a driver assistance system and/or an autonomous driving system.

[0052] The high-power application 110 transmits high-power application output 112 to an output interface 113, thus forming the control signal 25 as explained with reference to FIG. 1. The high-power application output 113 can comprise any application specific signal. Based on the high-power application output 112, the vision system 1 and/or the motor vehicle can display information to a passenger, activate safety functions, execute driving actions, etc.

[0053] The high-power application 110 leads to the use of high-power resources of the data processing device 7 which demand high power and generate heat which heats the vision system 1 and the data processing device 7. In order to reduce the heat that is produced by the data processing device 7 the high-power application 110 can transmit a low-power mode request 120 to a controller 30 which is adapted to switch the power mode of the data processing device 7 between the low-power mode and the high-power mode. The high-power application 110 signals the controller 30 if the power mode of the data processing device 7 should be switched to low-power mode. The high-power application 110 can determine when no high-power application 110 is needed. For example, when the vehicle is stationary.

[0054] The low-power application 100 can comprise passenger information, passenger entertainment and/or simple safety device functionality which lead to less heat generation than the high-power application 110.

[0055] The low-power application 100 receives the input data 99, in particular if the low-power mode of the data processing device 7 is activated. The low-power application 100 leads to less generation of heat by the data processing device 7 than the high-power application 110 as the low-power application 100 is preferably executed in low-power mode.

[0056] The low-power application 100 can transmit low-power application output 102 to a low-power output interface 114. The low-power application output 102 of the low-power application 100 is optional as indicated by the dotted line in FIG. 2. The low-power application output 102 and/or the low-power output interface 114 may be implemented compatible with the output interface 113 of the high-power application 110 and/or the control signal 25 to ensure output interface compatibility.

[0057] The classifier 20 uses the input data 99 of the sensed data, preferably comprising images recorded by the sensing device 40 and/or the imaging device 4. In particular, the classifier 20 can utilize multiple images, such as a current image and a previous image. The previous image can be the most previous image of a sequence of images or an image that was obtained when the vision system switched to low-power mode.

[0058] The data processing device 7 is adapted to execute the power mode classifier 20, and the power mode classifier 20 is adapted to classify any input data 99 from the sensing device 40 as requiring executing a high-power application 110 or not, and to output a high-power mode request 130 to the controller 30 to switch the power mode of the data processing device 7 from the low-power mode to the high-power mode in case the power mode classifier 20 has classified the input data 99 as requiring executing a high-power application 110.

[0059] To activate the high-power mode, the power mode classifier 20 can transmit a high-power mode request 130 to the controller 30. In this case, the low-power power mode classifier 20 signals the controller 30 if the power mode should be switched to high-power mode.

[0060] To deactivate the high-power mode, the power mode classifier 20 can transmit a power low request 131 to the controller 30. In this case, the power mode classifier 20 signals the controller 30 if the power mode should be switched to low-power mode.

[0061] When the controller 30 receives a request 120, 130, 131, i.e., the low-power mode request 120, the power low request 131, or the high-power mode request 130, the controller 30 determines when to switch power modes and/or whether to switch an application 110 100 and/or the power mode classifier 20 on or off. The controller 30 signals the power mode classifier 20 and/or the applications 110, 100, i.e., the high-power application 110 and/or the low-power application 100, to switch on or off. Specifically, the controller 30 can transmit a high-power application control signal 111 to switch the high-power application 110 on or off and a low-power application control signal 101 to switch the low-power application 100 and/or the power mode classifier 20 on or off.

[0062] Preferably, the high-power application control signal 111 and the low-power application control signal 101 are arranged so that only the high-power application 110 is switched on or off while the low-power application 100 and/or the power mode classifier 20 is complementary switched off or on. I.e., only either the low-power application 100 and/or the power mode classifier 20 is switched on or the high-power application 110 is switched on at the same time.

[0063] The power mode of the data processing device 7 can depend on the high-power application control signal 111 and/or the low-power application control signal 101. For example, if the high-power application control signal 111 switches the high-power application 110 off, the low-power mode is activated and/or the high-power mode is deactivated. If the high-power application control signal 111 switches the high-power application 110 on, the low-power mode is deactivated and/or the high-power mode is activated.

[0064] If, for example, the low-power application control signal 101 switches the low-power application 100 and/or the power mode classifier 20 on, the low-power mode can be activated and if the high-power application 110 is additionally switched off, the high-power mode be deactivated. If the low-power application control signal 101 switches the low-power application 100 and/or the power mode classifier 20 off, the low-power mode can be deactivated. Deactivating the low-power mode and/or switching the low-power application 100 and/or the power mode classifier 20 off is optional.

[0065] Preferably, the power mode of the data processing device 7 is switched together with the high-power application 110, the low-power application 100, and/or the power mode classifier 20. I.e., the power mode is switched to low-power mode if the high-power application 110 is switched off, and the low-power application 100 and/or the power mode classifier 20 are switched on. The power mode can be switched to high-power mode if the high-power application 110 is switched on, and/or the low-power application 100 and/or the power mode classifier 20 are switched off.

[0066] The data processing device 7 comprises the controller 30 adapted to switch the power mode of the data processing device 7 between the low-power mode and the high-power mode.

[0067] The controller 30 can receive an external control signals 98 from an electronic control unit 41 to request power mode switching by a control input external to the controller 30. The external control signal 98 by the electronic control unit 41 can be prioritized over the input data 99 from the sensing device 40 and/or the output of the power mode classifier 20. The control signal 98 of the electronic control unit 41 can in particular forcibly switch the power mode of the data processing device 7 to high-power mode by being transmitted to the controller 30 and triggering a high-power application control signal 111 to switch the high-power application 110 and/or the high-power mode on.

[0068] The classifier 20 is a trained classifier 20 which is preferably trained on a manufacturers site and/or by the manufacturer. To train the classifier 20, ground truth data is generated, wherein the ground truth data comprise pieces of input sensor data corresponding to a motor vehicle, and labels indicating for each piece of input sensor data whether it requires switching to high power mode or not. For each piece of input sensor data labels can be assigned which are either true or false labels indicating if the classifier 20 should signal that the high-power mode should be activated and/or that the input data 99 requires executing a high-power application 110, i.e., the high-power mode request 130 should be transmitted to the controller 30. Labels can automatically be assigned by a high-power algorithm without compute and heat constraints which can determine whether the motor vehicle underlying the input sensor data needs to be started and/or stopped and/or the high-power mode should be switched on or off.

[0069] While the above description constitutes the preferred embodiment of the present invention, it will be appreciated that the invention is susceptible to modification, variation and change without departing from the proper scope and fair meaning of the accompanying claims.