A VISION SYSTEM AND METHOD FOR A MOTOR VEHICLE

Abstract

A vision system for a motor vehicle comprises an imaging apparatus (11) adapted to capture images (30) from a surrounding of the motor vehicle, and a data processing unit (14) adapted to perform image processing on images (30) captured by said imaging apparatus (11). The data processing unit (14) comprises a traffic sign detector (31) adapted to detect traffic signs in images (30) captured by said imaging apparatus (11) through image processing, a decision section (35) and a traffic sign estimator (36) that is adapted to estimate validity information (37) of one or more traffic signs in an image (30) captured by said imaging apparatus (11).

Claims

1. A system, comprising: an imaging apparatus configured to capture images from a surrounding of a motor vehicle; and at least one processor configured to: detect one or more traffic signs in each of the images captured by the imaging apparatus; and generate validity information of one or more traffic signs in each of the images captured by the imaging apparatus.

2. The system of claim 1, wherein the at least one processor is configured to generate the validity information based on an entire image of each the images captured by the imaging apparatus.

3. The system of claim 1, wherein the at least one processor is configured to: output a control signal based in part on the validity information.

4. The system of claim 3, wherein the at least one processor is configured to: generate traffic sign classifier information associated with the detected one or more traffic signs in each of the images captured by the imaging apparatus; and output the control signal based on the traffic sign classifier information.

5. The system of claim 4, wherein the at least one processor is configured to: compare the traffic sign classifier information and the validity information to output the control signal; ignore at least one of the one or more detected traffic signs; and output the control signal to: suggest an alternative action to the driver; or signal to the driver to take over control of the vehicle.

6. The system of claim 4, wherein the traffic sign classifier information and the validity information each include one or more traffic sign interpretations, and wherein the at least one processor is configured to: compare the traffic sign classifier information and the validity information; determine, based on the comparison, whether a discrepancy exists between a traffic sign interpretation of the one or more traffic sign interpretations of the traffic sign classifier information and the corresponding traffic sign interpretation of the one or more traffic sign interpretations of the validity information; and for each determined discrepancy and based on the associated traffic sign classifier information and the validity information, determines determine which of the one or more traffic sign interpretations included in the traffic sign classifier information and the validity information is appropriate.

7. The system of claim 6, wherein each of the one or more traffic sign interpretations of the validity information is associated with a probable traffic speed sign, and wherein the at least one processor is configured to determine a traffic sign interpretation of the one or more traffic sign interpretations associated with a traffic speed sign with a lowest speed is appropriate.

8. The system of claim 6, wherein the at least one processor is configured to cause the motor vehicle to perform a suitable action in conformity with the traffic sign interpretation that is considered appropriate.

9. The system of claim 4, wherein the one or more traffic signs includes a road marking, and wherein the at least one processor is configured to compare the validity information associated with the road marking with corresponding traffic sign classifier information associated with the road marking.

10. The system of claim 4, wherein the at least one processor is further configured to turn off an autonomous driving system of the motor vehicle and to return control to a driver of the motor vehicle based on a determination that there is an inconsistency between the traffic sign classifier information and the validity information.

11. The system of claim 4, wherein the at least one processor is configured to apply a trained classifier to each of the images captured by the imaging apparatus to generate the traffic sign classifier information.

12. A computer-implemented method comprising: capturing images from a surrounding of a motor vehicle; detecting one or more traffic signs in each of the images captured by the imaging apparatus; and generating validity information of one or more traffic signs in each of the images captured by the imaging apparatus.

13. The computer-implemented method of claim 12, wherein generating the validity information is based on an entire image of each of the images captured by the imaging apparatus.

14. The computer-implemented method as of claim 12, further comprising: outputting a control signal based in part on the validity information.

15. The computer-implemented method of claim 14, further comprising: generating traffic sign classifier information associated with the detected one or more traffic signs in each of the images captured by the imaging apparatus; and wherein outputting the control signal is based further in part on the traffic sign classifier information.

16. The computer-implemented method of claim 15, wherein outputting the control signal further includes combining the traffic sign classifier information and the validity information.

17. The computer-implemented method of claim 15, wherein the traffic sign classifier information and the validity information each include one or more traffic sign interpretations, and wherein outputting the control signal further includes: comparing the traffic sign classifier information and the validity information; determining whether a discrepancy exists between a traffic sign interpretation of the one or more traffic sign interpretations of the traffic sign classifier information and the corresponding traffic sign interpretation of the one or more traffic sign interpretations of the validity information based on the comparison; and for each determined discrepancy and based on the associated traffic sign classifier information and the validity information, determining which of the one or more traffic sign interpretations included in the traffic sign classifier information and the validity information is appropriate.

18. The computer-implemented method of claim 17, wherein each of the one or more traffic sign interpretations of the validity information is associated with a probable traffic speed sign, and wherein the computer-implemented method further comprises, determining a traffic sign interpretation of the one or more traffic sign interpretations associated with a traffic speed sign with a lowest speed is appropriate.

19. The computer-implemented method of claim 17, further comprising causing the motor vehicle to perform a suitable action in conformity with the traffic sign interpretation considered appropriate.

20. A non-transitory, machine-readable medium having stored thereon a plurality of executable instructions, that when executed by a processor, the plurality of executable instructions comprising instructions to: capture images from a surrounding of a motor vehicle; detecting one or more traffic signs in each of the images captured by the imaging apparatus; and generating validity information of one or more traffic signs in each of the images captured by the imaging apparatus.

Description

[0020] In the following the invention shall be illustrated on the basis of preferred embodiments with reference to the accompanying drawings, wherein:

[0021] FIG. 1 shows a schematic drawing of a vision system; and

[0022] FIG. 2 shows a diagram illustrating functional elements in the data processing unit of the vision system.

[0023] The vision system 10 is preferably an on-board vision system 10 which is mounted, or to be mounted, in or to a motor vehicle. The vision system 10 comprises an imaging apparatus 11 for capturing images of a region surrounding the motor vehicle, for example a region in front of the motor vehicle. The imaging apparatus 11, or parts thereof, may be mounted for example behind the vehicle windscreen or windshield, in a vehicle headlight, and/or in the radiator grille. Preferably the imaging apparatus 11 comprises one or more optical imaging devices 12, in particular cameras, preferably operating in the visible wavelength range, or in the infrared wavelength range, or in both visible and infrared wavelength range. In some embodiments the imaging apparatus 11 comprises a plurality of imaging devices 12 in particular forming a stereo imaging apparatus 11. In other embodiments only one imaging device 12 forming a mono imaging apparatus 11 can be used. Each imaging devices 12 preferably is a fixed-focus camera, where the focal length f of the lens objective is constant and cannot be varied.

[0024] The imaging apparatus 11 is coupled to a data processing unit 14 (or electronic control unit, ECU) which is preferably an on-board data processing unit 14. The data processing unit 14 is adapted to process the image data received from the imaging apparatus 11. The data processing unit 14 is preferably a digital device which is programmed or programmable and preferably comprises a microprocessor, a microcontroller, a digital signal processor (DSP), and/or a microprocessor part in a System-On-Chip (SoC) device, and preferably has access to, or comprises, a digital data memory 25. The data processing unit 14 may comprise a dedicated hardware device, like a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), a Graphics Processing Unit (GPU) or an FPGA and/or ASIC and/or GPU part in a System-On-Chip (SoC) device, for performing certain functions, for example controlling the capture of images by the imaging apparatus 11, receiving the signal containing the image information from the imaging apparatus 11, rectifying or warping pairs of left/right images into alignment and/or creating disparity or depth images. The data processing unit 14 may be connected to the imaging apparatus 11 via a separate cable or a vehicle data bus. In another embodiment the ECU and one or more of the imaging devices 12 can be integrated into a single unit, where a one box solution including the ECU and all imaging devices 12 can be preferred. All steps from imaging, image processing to possible activation or control of a safety device 18 are performed automatically and continuously during driving in real time.

[0025] In another embodiment, the above described image processing, or parts thereof, are performed in the cloud. Consequently, the data processing unit 14, or parts thereof, may be realized by cloud processing resources.

[0026] Image and data processing carried out in the data processing unit 14 advantageously comprises identifying and preferably also classifying possible objects (object candidates) in front of the motor vehicle, such as pedestrians, other vehicles, bicyclists and/or large animals, tracking over time the position of objects or object candidates identified in the captured images, and activating or controlling at least one safety device 18 depending on an estimation performed with respect to a tracked object, for example on an estimated collision probability.

[0027] The safety device 18 may comprise at least one active safety device and/or at least one passive safety device. In particular, the safety device 18 may comprise one or more of: at least one safety belt tensioner, at least one passenger airbag, one or more restraint systems such as occupant airbags, a hood lifter, an electronic stability system, at least one dynamic vehicle control system, such as a brake control system and/or a steering control system, a speed control system; a display device to display information relating to a detected object; a warning device adapted to provide a warning to a driver by suitable optical, acoustical and/or haptic warning signals.

[0028] In the following, a process of traffic sign verification under the present invention is explained with reference to FIG. 2. All method steps related to the functional units 31-38 are performed in real time during driving in the data processing unit 14.

[0029] Images 30 captured by the imaging apparatus 11 of a motor vehicle are forwarded to a traffic sign detector/classifier 31, 33, which is known per se, and in parallel also to an inventive holistic traffic sign estimator 36.

[0030] The traffic sign detector 31 is adapted to detect traffic signs in the input images. Traffic signs 32 detected by the traffic sign detector 31 are forwarded to a traffic sign classifier 33 adapted to classify a detected traffic sign into one or more of a predefined number of categories. The traffic sign classifier 33 is known per se, and usually performs classification on a small image patch in a so-called bounding box closely around a detected traffic sign. Classified traffic signs 34 are forwarded to a decision section 35. The traffic sign detector 31 and/or classifier 33 may perform tracking a detected traffic sign over a plurality of image frames. The traffic sign detector 31 and the traffic sign classifier 33 may be a single unit adapted to detect and classify traffic signs simultaneously.

[0031] The holistic traffic sign estimator 36 has been trained in advance, and is adapted to output, for each entire image from the imaging apparatus 11, validity information 37 of one or more traffic signs in the input image 30, for example a probability 37 that one or more specific, i.e. predefined, traffic signs is present in the input image 30. More specifically, the holistic traffic sign estimator 36 can estimate and output validity information 37, like a probability or a validity/invalidity flag value, for each of a plurality of predefined traffic signs to be present in the input image 30. The one or more estimated validity values or probabilities 37 are forwarded to the decision section 35. The decision section 35 compares or combines the validity information 37 provided by the traffic sign estimator 36 with information 34 provided by the traffic sign detector 31 and/or traffic sign classifier 33, and initiates a suitable action.

[0032] In the following, a practical example is discussed where the holistic traffic sign estimator 36 is restricted to estimating speed signs, and therefore is a holistic speed sign estimator 36. Specifically, the holistic speed sign estimator 36 is a trained classifier, and may be adapted to classify an entire input image into one or more of, for example, five categories:

[0033] containing a 30 km/h speed sign, containing a 50 km/h speed sign, containing a 60 km/h speed sign, containing an 80 km/h speed sign, containing none of these speed signs. It goes without saying that the number of speed signs can be different from five, and/or the speed signs which can be estimated from by the holistic speed sign estimator 36 can involve other speed signs than the above mentioned.

[0034] It may be assumed that the traffic sign detector/classifier 31/33 detects and identifies an 80 km/h speed sign in a particular input image 30. The holistic speed sign estimator 36 estimates the following probabilities for the input image: 5% for 30 km/h speed sign, 15% for 50 km/h speed sign, 40% for 60 km/h speed sign, 30% for 80% speed sign, and 10% for none of these speed signs.

[0035] The decision section 35 compares or combines the above probabilities 37 with the finding by the speed sign detector/classifier 31, 33, and can initiate one or more of the following actions based on this comparison.

[0036] (i) The decision section 35 may accept the detected traffic sign in the further processing, either as an 80 km/h traffic sign as classified by the speed sign detector/classifier 31, 33, or as a 60 km/h speed sign as estimated (with highest probability) by the holistic speed estimator 36.

[0037] (ii) The decision section 35 may ignore the detected traffic sign in the further processing.

[0038] (iii) The decision section 35 may output a control signal 38 to a signaling device 18 (see FIG. 1) to suggest an alternative action to the driver, like taking care of speed limits.

[0039] (iv) The decision section 35 may output a control signal 38 to the signaling device 18 to signal to the driver to take over control of the vehicle.

[0040] (v) The decision section 35, determining an inconsistency between the speed sign (80 km/h) detected and classified by the speed sign detector/classifier 31, 33 and the speed sign (60 km/h) having the highest probability according to the holistic speed sign estimator 36, may determine which of the sign interpretations offered by the traffic sign detector/classifier 31, 33 and the traffic sign estimator 36 is the appropriate/most appropriate . In one embodiment, the speed sign (60 km/h) having the highest probability according to the holistic speed sign estimator 36 may be considered appropriate/most appropriate. In a preferred embodiment, the speed sign (50 km/h) with the lowest speed and a probability over a predetermined threshold (for example 10%) is considered appropriate/most appropriate, here disregarding the 30 km/h speed sign the probability of which is too low to be considered true. The decision section initiates a suitable action based on the appropriate/most appropriate speed sign, for example braking the motor vehicle to decelerate it to the speed of the appropriate/most appropriate speed sign.

[0041] As is evident from the above, the data processing unit 14 preferably comprises two different classifiers, namely the conventional traffic sign classifier 33 performing classification only on a small image patch around a detected traffic sign, and the inventive holistic traffic sign estimator 36 which advantageously performs classification on an entire input image.