Device for enabling a vehicle to automatically resume moving

Abstract

A processing unit with at least one image that relates to a scene external to a vehicle, processes the image to extract low level features. A low level feature is defined such that an object in the scene cannot be characterized on the basis of at least one low level feature in a single image of the at least one scene. The processing unit determines if at least a part of an object is present in a part of the scene, the determination comprises an analysis of the low level features. An output unit outputs information that there is something present in the scene.

Claims

1. A device for enabling a vehicle to automatically resume moving, the device comprising: an input unit; a processing unit coupled to the input unit; and an output unit coupled to the processing unit; wherein, the input unit is configured to provide the processing unit with at least one image, the at least one image relating to a scene external to a vehicle; wherein, the processing unit is configured to process the at least one image to extract low level features and high level features; wherein a low level feature is an object in the scene that cannot be characterized on the basis of at least one low level feature in a single image of the at least one image; wherein a high level feature is an object in the scene that can be characterized on the basis of at least one high level feature in a single image of the at least one image; wherein, the processing unit is configured to determine if at least a part of an object is present in a part of the scene, the determination comprising an analysis of the low level features and of the high level features; and wherein, the output unit is configured to output information that there is something present in the scene.

2. The device of claim 1, wherein the processing unit is configured to implement an algorithm based on artificial intelligence to analyze the low level features.

3. The device of claim 1, wherein the high level features comprise a target vehicle, and wherein the part of the scene within which the processing unit is configured to determine that at least a part of an object is present is determined as the region between the vehicle and the target vehicle.

4. The device of claim 1, wherein the analysis of the low level features comprises determining at least one potential object in the part of the scene and determining at least one confidence level, such that a potential object has an associated confidence level, and wherein a potential object is determined to be the at least part of the object when the confidence level for that potential object is above a threshold value.

5. The device of claim 4, wherein the at least one image comprises a first image and a second image, and wherein confidence levels determined from the first image are updated on the basis of confidence levels determined from the second image.

6. The device of claim 5, wherein the processing unit is configured to track potential objects across the first image and second image.

7. The device of claim 1, wherein at least one of the at least one image was acquired by at least one camera.

8. The device of claim 1, wherein at least one of the at least one image was acquired by a radar sensor.

9. The device of claim 1, wherein the at least part of the object is within a distance of 2.5 m from an outer periphery of the vehicle.

10. The device of claim 1, wherein the low level features comprises one or more of: colour information; edges; gradients; optic flow; optic flow clusters, saliency information.

11. The device according to claim 1, wherein the at least one image was captured whilst the vehicle was stationary.

12. A system for enabling a vehicle to automatically resume moving, the system comprising: at least one sensor system configured to be located within a vehicle and to acquire the at least one image relating to a scene external to the vehicle viewed; a device for enabling a vehicle to automatically resume moving, the device is configured to be located within the vehicle, the device comprising: an input unit; a processing unit coupled to the input unit; and an output unit coupled to the processing unit; wherein, the input unit is configured to provide the processing unit with the at least one image; wherein, the processing unit is configured to process the at least one image to extract low level features and high level features; wherein a low level feature is an object in the scene that cannot be characterized on the basis of at least one low level feature in a single image of the at least one image; wherein a high level feature is defined such that an object in the scene can be characterized on the basis of at least one high level feature in a single image of the at least one image; wherein, the processing unit is configured to determine if at least a part of an object is present in a part of the scene, the determination comprising an analysis of the low level features and of the high level features; and wherein, the output unit is configured to output information that there is something present in the scene.

13. A method for enabling a vehicle to automatically resume moving, the method comprising: providing a processing unit with at least one image, the at least one image relating to a scene external to a vehicle; processing with the processing unit the at least one image to extract low level features, a low level feature is an object in the scene that cannot be characterized on the basis of at least one low level feature in a single image of the at least one image; processing with the processing unit the at least one image to extract high level features, a high level feature is an object in the scene characterized on the basis of at least one high level feature in a single image of the at least one image; determining with the processing unit if at least a part of an object is present in a part of the scene, the determination comprising an analysis of the low level features and the high level features; and outputting with an output unit output information that there is something present in the scene.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) Exemplary embodiments will be described in the following with reference to the following drawings:

(2) FIG. 1 shows a schematic set up of an example of a device for enabling a vehicle to automatically resume moving;

(3) FIG. 2 shows a schematic set up of an example of a system for enabling a vehicle to automatically resume moving;

(4) FIG. 3 shows a method for enabling a vehicle to automatically resume moving;

(5) FIG. 4 shows an example of an Ego vehicle behind a target vehicle, the observation area and possibly intruding object; and

(6) FIG. 5 shows a flow chart of an example of an algorithm used for enabling a vehicle to automatically resume moving.

DETAILED DESCRIPTION OF EMBODIMENTS

(7) FIG. 1 shows an example of a device 10 for enabling a vehicle to automatically resume moving. The device 10 comprises an input unit 20, a processing unit 30, and an output unit 40. The input unit 20 is configured to provide the processing unit 30 with at least one image, via wired or wireless communication. The at least one image relates to a scene external to a vehicle. The processing unit 30 is configured to process the at least one image to extract low level features. A low level feature is defined such that an object in the scene cannot be characterised on the basis of at least one low level feature in a single image of the at least one image. The processing unit 30 is also configured to determine if at least a part of an object is present in a part of the scene. The determination comprises an analysis of the low level features. The output unit 40 is configured to output information that there is something present in the scene.

(8) In an example, the processing unit is configured to implement a decision tree analysis algorithm to analyse the low level features.

(9) According to an example, the processing unit is configured to implement an algorithm based on artificial intelligence to analyse the low level features.

(10) In an example, the algorithm based on artificial intelligence is a machine learning algorithm. In an example, the algorithm based on artificial intelligence is a neural network. In an example, the algorithm based on artificial intelligence is a heat map algorithm.

(11) According to an example, the processing unit is configured to process the at least one image to extract high level features. A high level feature is defined such that an object in the scene can be characterised on the basis of at least one high level feature in a single image of the at least one image. The determination if at least a part of an object is present in the scene then also comprises an analysis of the high level features.

(12) In an example, high level features can be extracted from objects such as pedestrians.

(13) According to an example, the high level features comprises a target vehicle. The part of the scene within which the processing unit is configured to determine that at least a part of an object is present is determined as the region between the vehicle and the target vehicle.

(14) In an example, high-level features can be extracted from objects such as pedestrians entering the area between target and ego vehicle as well if they are visible entirely.

(15) In an example, the target vehicle is a car, or lorry, or motorcycle. In an example, the ego vehicle is a car, or lorry, or motorcycle.

(16) According to an example, the analysis of the low level features comprises determining at least one potential object in the part of the scene and determining at least one confidence level, such that a potential object has an associated confidence level. A potential object is then determined to be the at least part of the object when the confidence level for that potential object is above a threshold value.

(17) In an example, the threshold is manually optimized and/or machine learned. Thus, thresholds could be set that are object specific, such that if there is a chance that the analysis of the low level features tend to indicate the possibility that a child could be in the clearance space the threshold could be set lower than if there is a potential for an adult in the clearance space. The threshold can also take account of where the potential object is within the clearance space as well as taking into account the potential type of object. For example, if the object could potentially be a child the threshold could be set low irrespective of where that object is within the clearance space, thereby ensuring that any movement forward of the vehicle would not panic the child. However, if the object could potentially be an adult, the threshold level could be varied depending upon where within that space the object is located.

(18) According to an example, the at least one image comprises a first image and a second image, and wherein confidence levels determined from the first image are updated on the basis of confidence levels determined from the second image.

(19) In an example, the at least one image comprises n images, and wherein confidence levels determined from the first image are updated on the basis of confidence levels determined from an nth image.

(20) According to an example, the processing unit is configured to track potential objects across the first image and second image.

(21) According to an example, at least one of the at least one image was acquired by at least one camera.

(22) According to an example, at least one of the at least one image was acquired by a radar sensor.

(23) According to an example, the at least part of the object is within a distance of 2.5 m from an outer periphery of the vehicle.

(24) According to an example, the at least one image was captured whilst the vehicle was stationary.

(25) According to an example, the low level features comprises one or more of: colour information; edges; gradients; optic flow; optic flow clusters, saliency information.

(26) In an example, low level features can be individual ones of colour information; edges; gradients; optic flow; optic flow clusters. In an example, mid-level features can be formed from a combination of more than one of: colour information; edges; gradients; optic flow; optic flow clusters. The mid-level features are however still not high level features, in that an object in the scene cannot be characterised on the basis of such a mid level feature in a single image of the at least one image.

(27) FIG. 2 shows an example of a system 100 for enabling a vehicle to automatically resume moving. The system 100 comprises at least one sensor system 110, and a device 10 for enabling a vehicle to automatically resume moving as described with respect to FIG. 1. The device 10 is configured to be located within a vehicle 120. The at least one sensor system 110 is configured to be located within the vehicle 120 and the at least sensor system 110 is configured to acquire the at least one image relating to a scene external to the vehicle viewed.

(28) In an example, the at least one sensor system 110 is the input unit 20.

(29) FIG. 3 shows a method 200 for enabling a vehicle to automatically resume moving in its basic steps. The method 200 comprises:

(30) in a providing step 210, also referred to as step a), providing a processing unit 30 with at least one image, the at least one image relating to a scene external to a vehicle;

(31) in a processing step 220, also referred to as step b), processing with the processing unit the at least one image to extract low level features; wherein a low level feature is defined such that an object in the scene cannot be characterised on the basis of at least one low level feature in a single image of the at least one scene;

(32) in a determining step 230, also referred to as step c), determining with the processing unit if at least a part of an object is present in a part of the scene, the determination comprising an analysis of the low level features; and in an outputting step 240, also referred to as step d), outputting with an output unit 40 output information that there is something present in the scene.

(33) In an example, step c) comprises implementing an algorithm based on artificial intelligence to analyse the low level features.

(34) In an example, step b) comprises processing the at least one image to extract high level features, wherein a high level feature is defined such that an object in the scene can be characterised on the basis of at least one high level feature in a single image of the at least one image; and step c) comprises an analysis of the high level features.

(35) In an example, the high level features comprises a target vehicle, and wherein the part of the scene within which the processing unit is configured to determine that at least a part of an object is present is determined as the region between the vehicle and the target vehicle.

(36) In an example, the analysis of the low level features in step c) comprises determining at least one potential object in the part of the scene and determining at least one confidence level, such that a potential object has an associated confidence level, and wherein a potential object is determined to be the at least part of the object when the confidence level for that potential object is above a threshold value.

(37) In an example, the at least one image comprises a first image and a second image, and wherein confidence levels determined from the first image are updated on the basis of confidence levels determined from the second image.

(38) In an example, the processing unit is configured to track potential objects across the first image and second image.

(39) The device, system and method are now described in further detail with respect to FIGS. 4-5.

(40) FIG. 4 shows an example, of an arbitrary object (pedestrian) entering a monitored area between an ego vehicle and the target vehicle. A camera based approach is used to detect objects, which can be very close to the ego vehicle, and which are either located in the observation area (or clearance space) between the ego vehicle and the target vehicle or about to enter the observation area. A radar-based approach could also be utilised, and a camera and radar based approach could be utilised.

(41) The present approach to determining if an object is present in the observation area with a low false negative rate can be best understood by first considering an ADAS such as enhanced brake assist (EBA), which classifies an object with a high degree of confidence (here called high-level object features) in order for action to be taken. Information within the scene, which can be considered to be low-level object features and medium level object features which cannot be used to classify an object, and therefore are not considered by an EBA system, are now processed within the present approach to determine if an object is within or about to enter the observation area. Therefore, considering an EBA system the object detection/characterization process can be described as follows:

(42) 1) Define n patches of a certain size and slide them over the entire image

(43) 2) For each position of a patch, extract low-level features such as colors, edges, gradients, etc.

(44) 3) Arrange all features in a smart manner to form a feature descriptor (mid-level features). Here mid level features are a combination of low-level features (motifs), hypotheses, optic flow, flow clusters.

(45) 4) Use this descriptor to compute the probability of the patch to contain an object of interest. The patch will become a hypothesis if this probability is larger than a certain threshold (mid-level features)

(46) 5) Since the patches are slid all over the image, many hypotheses are produced for the same object. .fwdarw.Perform non-maximum suppression on the hypotheses to just keep the one with the highest probability value, ideally representing the object to detect.

(47) 6) Use the descriptor of the remaining hypothesis to classify the object as either vehicle, pedestrian, etc., and compute the confidence of this object to belong to the class identified.

(48) However, now in the current approach steps 1-4 are carried out, and new step 5 takes all the low level features and mid level features and processes these to determine if there could be an object present. In other words, intermediate processing steps and the information available at those steps which is not used in a complete ADAS solution, is now directly utilised to indicate the presence of an object in the clearance space. Objects that are very close to the ego vehicle (e.g. less than 2.5 m), cannot be seen in their entirety, are either covered/hidden by the ego vehicle, have parts that are outside of the camera's field of view, are a very small object such as children whose bodies may be covered or obscured by the ego vehicle, can now be detected. This enables an ACC system to inhibit ACC Auto Go when such an object have been detected, and conversely also enables full ACC Stop and Go functionality when no such object has been detected. In this manner clearance confirmation can be reliably generated with a low false negative rate.

(49) FIG. 5 shows a simplified flowchart of the algorithm that enables autonomous resume in an ACC system without further confirmation from the driver. A camera system, which requires imagery which is processed in the current approach, operates at a frame rate of 16 frames per second with the cycle time period between 2 consecutive frames of 66 ms. Different cameras can operate a different cycle times. A processor within the camera system, or a separate processor processing imagery from the camera system, performs close object detection at each cycle and calculates the clearance confirmation by explicitly using mid and low level features of the intermediate processing steps as discussed above. The clearance confirmation then enables an indication that the observation area was not violated by a crossing object when object that was already present in that area.

(50) With continued reference to FIG. 5, the clearance confirmation is computed in the following basic steps: 1. identification of the target vehicle a. identification of the target vehicle to determine the observation area to monitor for entering or present objects b. the actual target vehicle is also observed while driving and in standstill in order to identify a change of the target vehicle, when for example other vehicles cutting between the ego vehicle and the actual target vehicle c. as part of a and b, a trace of the currently tracked target vehicle is monitored, where the trace is the track (route) of a detected vehicle formed by the locations of the vehicle in space and time. An interrupted or completely aborted trace of the target vehicle either during driving or standstill is used as an indicator of a change of the target vehicle 2. analysis of the movement of the ego vehicle and target vehicle to identify a deceleration to standstill and a potential automatic resume afterwards. 3. Monitoring of the observation area after standstill a. as discussed above high-level objects are detected and classified, which can be used for example in an EBA system. However, as discussed above low-level and medium level information is used to determine if there is potential object and the clearance space. The high-level information or objects can also be used in this process. b. The final result of potential object detection is an object list that is used as an indicator for violations of the observation area (clearance space) c. the object list contains objects detected with high confidence, which as discussed above can be used to determine information relating to the target area and information relating to the target vehicle. This high confidence information can also be used to determine if there is an object in the clearance space d. however, the object list now contains additional objects which can be called hypotheses, which have not reached high confidence and therefore are not relevant for other system functions such as EBA. These hypotheses have either been newly initialised (detected) or have been updated with low confidence classify responses (a previously detected low or medium confidence object can have its confidence level updated on the basis of the presently acquired frame). e. With respect to EBA, a confidence level for the hypotheses is updated in every cycle according to associated classify responses (or the lack thereof), and an object hypothesis will either be advanced to a high confidence object or will be deleted if it is no longer reinforced by classify responses. f. However, low and mid-level features such as low confidence classify responses or flow clusters are now explicitly used in order to determine if they could be an object in the clearance space. Therefore, a low confidence classify response which will not in itself result in the initialisation of an object contained in the object list can on the basis of a continuous presence of such low confidence classify responses in a certain area indicate that there is an object in the clearance space at the position. Therefore, low confidence classify responses are accumulated in standstill for a certain period of time. If certain areas contain conspicuous clusters of such responses that can be considered as a low-level feature for the clearance confirmation. In other words, low and medium level information that does not have enough associated confidence to detect and classify an object, for use in for example an EBA, can be used to help inform the ACC that they could be something in the clearance space. Therefore, if an object is very small and/or is covered partially by the ego vehicle and cannot be classified as whole, then there may be still enough hypotheses indicating a potential violation of the clearance space. g. Low-level features includes features such as colour extracted from regions, edges, gradients et cetera and mid-level features includes the combination of low-level features, hypotheses, flow, flow clusters et cetera. Therefore, optical flow and flow clusters are just another form of feature that can be used instead or additionally to determine if they could be an object in the clearance space. The sudden presence of optical flow and flow clusters can indicate (directional) motion and that an object is entering the critical space. Thus, flow clusters can be considered to be the mono camera equivalent of clustered disparity data. If some close by flow vectors have a similar direction and magnitude and cover sufficient area of the image to match intruding objects, those can be clustered to create a generic object hypothesis. Tracked generic objects can then be used as a low-level feature for the clearance confirmation. h. Saliency information can be used as a feature to identify changes in the monitored area, indicating a potential intrusion object i. in addition, if stereo information is available, generic object detection based on clustered disparity data can be used in addition to that discussed above. 4. Computation of clearance confirmation a. low confidence outputs coming from acquired imagery is then used within a neural network machine learning algorithm to determine if there is an object or potential object in the clearance space. The neural network is trained using ground truth information, in other words using imagery that has already been analysed by human. The neural network then operates in real time on acquired imagery to determine if there is a possible object in the clearance space, in order to provide clearance confirmation within an ACC system. As discussed above the high confidence information can also be used within the neural network machine learning algorithm. Rather than use a neural network machine learning algorithm, a handcrafted decision tree algorithm can be utilised. b. To improve the false positive rate a de bouncing mechanism is utilised. This means that an intrusion object is observed to several cycles and/or risk value is cumulated according to the type of detected featureless reliability. Only if a certain threshold (either manually optimised machine learned) is reached will automatic resumption of vehicle motion be revoked. c. As discussed above, the high level object list can be used for direct observation of the target vehicle. Before the clearance detection and subsequently the auto-go feature is activated the current target vehicle that is being followed has to be identified. As soon as the target vehicle has been identified and a stopping maneuver is performed the detection of the target vehicle is expected to remain present. If the track of the target object is lost this can be interpreted as a sign that the target vehicle is now occluded (for example due to someone walking in front if the sensor on the ego vehicle). In this case the clearance for auto-go is also rejected.

(51) In addition, or in replacement, to a camera sensor, radar or ultrasonic sensors can be utilized to determine if potential objects are in the clearance space in order to provide clearance confirmation in the manner described above.

(52) In another exemplary embodiment, a computer program or computer program element is provided that is characterized by being configured to execute the method steps of the method according to one of the preceding embodiments, on an appropriate system.

(53) The computer program element might therefore be stored on a computer unit, which might also be part of an embodiment. This computing unit may be configured to perform or induce performing of the steps of the method described above. Moreover, it may be configured to operate the components of the above described apparatus and/or system. The computing unit can be configured to operate automatically and/or to execute the orders of a user. A computer program may be loaded into a working memory of a data processor. The data processor may thus be equipped to carry out the method according to one of the preceding embodiments.

(54) According to a further exemplary embodiment of the present invention, a computer readable medium, such as a CD-ROM, is presented wherein the computer readable medium has a computer program element stored on it which computer program element is described by the preceding section.

(55) It has to be noted that embodiments of the invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described with reference to the device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise notified, in addition to any combination of features belonging to one type of subject matter also any combination between features relating to different subject matters is considered to be disclosed with this application. However, all features can be combined providing synergetic effects that are more than the simple summation of the features.

(56) While the invention has been illustrated and described in detail in the drawings and foregoing description, such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing a claimed invention, from a study of the drawings, the disclosure, and the dependent claims.

(57) In the claims, the word comprising does not exclude other elements or steps, and the indefinite article a or an does not exclude a plurality. A single processor or other unit may fulfill the functions of several items re-cited in the claims. The mere fact that certain measures are re-cited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any reference signs in the claims should not be construed as limiting the scope.