Method and control device for identifying an object in a piece of image information

Abstract

A method for identifying an object in a piece of image information representing a scene in a detection range of a camera during a situation includes: a step of reading in; a step of selecting; and a step of searching. In the step of reading in, the piece of image information and at least one parameter representing the situation are read in. In the step of selecting, a feature combination of an object class of the object, which is predictably identifiable in the situation, is selected using the parameter. In the step of searching, the feature combination is searched for in the piece of image information to identify the object.

Claims

1. A method for identifying an object in a piece of image information representing a scene in a detection range of a camera during a situation, the method comprising: reading in, by a data interface of a control device, the piece of image information from the camera and at least one parameter representing the situation; selecting, by the control device, a feature combination of an object class of the object, which is predictably identifiable in the situation, using the parameter; searching, by the control device, for the feature combination in the piece of image information to identify the object; and using the identification of the object in a real-time driver assistance system in a vehicle; wherein an integral image is read in as the piece of image information, the integral image representing at least one of a line-by-line and a column-by-column integration of brightness values of pixels of a camera image of the scene; and wherein the searching includes at least a first search step and a second search step, and wherein a first feature from the feature combination is used for checking in the first search step, and subsequently at least one further feature from the feature combination is used for checking in the second search step.

2. The method as recited in claim 1, wherein the at least one further feature for checking is selected from the feature combination in the second search step using a result of the first search step.

3. The method as recited in claim 1, wherein an identification probability for the object is provided in the searching, and wherein the greater an identification probability is provided, the more features of the feature combination are identifiable in the piece of image information.

4. The method as recited in claim 1, wherein pixels of the piece of image information are marked as the object in the searching when a minimum number of features of the feature combination is identified in the pixels.

5. The method as recited in claim 1, further comprising: identifying the situation using at least one of (i) the scene in the piece of image information, (ii) vehicle data, and (iii) environmental conditions.

6. The method as recited in claim 1, wherein: in the step of reading in, a piece of class information of the object class to be identified is additionally read in; and in the step of selecting, the feature combination representing the object class is selected from multiple different feature combinations using the piece of class information.

7. The method as recited in claim 1, wherein the feature combination includes a piece of information about at least one area sum of the object to be identified.

8. The method as recited in claim 1, wherein the control device is control device of a navigation control system of the vehicle.

9. The method as recited in claim 1, wherein the control device is a control device of a camera system.

10. The method as recited in claim 1, wherein the at least one parameter includes a speed of the vehicle.

11. A control device for identifying an object in a piece of image information representing a scene in a detection range of a camera during a situation, the control device comprising: an interface configured for reading in the piece of image information from the camera and at least one parameter representing the situation, the reading in including reading in an integral image as the piece of image information, the integral image representing at least one of a line-by-line and a column-by-column integration of brightness values of pixels of a camera image of the scene; a selection unit configured for selecting a feature combination of an object class of the object, which is predictably identifiable in the situation, using the parameter; and a searching unit configured for searching for the feature combination in the piece of image information to identify the object, the searching including at least a first search and a second search, and wherein a first feature from the feature combination is used for checking in the first search, and subsequently at least one further feature from the feature combination is used for checking in the second search; wherein the identification of the object is forwarded to and used by a real-time driver assistance system in a vehicle.

12. A non-transitory, computer-readable data storage medium storing a computer program having program codes which, when executed on a computer, perform a method for identifying an object in a piece of image information representing a scene in a detection range of a camera during a situation, the method comprising: reading in, by a data interface of a control device, the piece of image information from the camera and at least one parameter representing the situation; selecting a feature combination of an object class of the object, which is predictably identifiable in the situation, using the parameter; and searching for the feature combination in the piece of image information to identify the object; and using the identification of the object in a real-time driver assistance system in a vehicle; wherein an integral image is read in as the piece of image information, the integral image representing at least one of a line-by-line and a column-by-column integration of brightness values of pixels of a camera image of the scene; and wherein the searching includes at least a first search step and a second search step, and wherein a first feature from the feature combination is used for checking in the first search step, and subsequently at least one further feature from the feature combination is used for checking in the second search step.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 shows a representation of a vehicle including a control device for identifying an object in a piece of image information according to one exemplary embodiment of the present invention.

(2) FIG. 2 shows a graphic representation of a method sequence for identifying an object in a piece of image information according to one exemplary embodiment of the present invention.

(3) FIG. 3 shows a block diagram of a control device for identifying an object in a piece of image information according to one exemplary embodiment of the present invention.

(4) FIG. 4 shows a flow chart of a method for identifying an object in a piece of image information according to one exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

(5) In the following description of favorable exemplary embodiments of the present invention, identical or similar reference numerals are used for similarly acting elements shown in the different figures, and a repeated description of these elements is dispensed with.

(6) FIG. 1 shows a representation of a vehicle 100 including a control device 102 for identifying an object in a piece of image information 104 according to one exemplary embodiment of the present invention. Piece of image information 104 is generated by a camera 106, which is installed in vehicle 100 here and directed at a detection range ahead of vehicle 100. Camera 106 is designed to depict a scene 108 in the detection range in piece of image information 104. Control device 102 includes an interface for piece of image information 104 and an interface for at least one parameter 110. Parameter 110 represents a situation in which camera 106 detects scene 108.

(7) In one exemplary embodiment, parameter 110 is a vehicle speed.

(8) In one exemplary embodiment, parameter 110 is an illumination intensity.

(9) In one exemplary embodiment, parameter 110 is a weather condition.

(10) Control device 102 is designed to identify the object using parameter 110. For this purpose, control device 102 selects a feature combination which fits parameter 110 best from a number of different feature combinations of an object class to be identified. In this way, for example, different features may be used for object identification in the dark than during daylight. The object may also have a different appearance in rain than in sunshine. By focusing on the respective best suited features, reliable object identification is made possible. A position of the identified object in scene 108 is forwarded to other vehicle control devices 112 for further use.

(11) In one exemplary embodiment, control device 102 is implemented in an on-board computer of vehicle 100.

(12) In one exemplary embodiment, control device 102 is implemented in a camera control device of vehicle 100.

(13) In one exemplary embodiment, control device 102 is implemented in a navigation control device of vehicle 100.

(14) Piece of image information 104 of scene 108 including objects in front of camera system 106 is read in via a data interface. Vehicle and system data, such as a speed of vehicle 100, are also read in as parameter 110 via the interface.

(15) In other words, FIG. 1 according to one exemplary embodiment shows a use of FPGA-based (field programmable gate array) weighted area sums for object identification in video images.

(16) Objects in camera images may be identified from the combination of sums via surface areas and their linear linkages. These object identifiers may be trained with the aid of learning methods to achieve a good identification rate at a low false alarm rate. The approach presented here reduces the arithmetic complexity of these methods, improves their suitability for the use with real-time driver assistance systems (DAS) and enables a high processing speed and a prompt response to objects from the real world.

(17) The approach presented here introduces a method for FPGAs 102 which is able to meet the object identification task in real time and thus allows its use in the automotive DAS environment in the first place.

(18) For this purpose, specialized area sum features and an FPGA-based processing unit 102 for these features are introduced here, which allow a real time object identification. The great advantage of the proposed approach is its reloadability at run time, which allows a situation-dependent identification of objects, e.g., switching in the event of a change in the weather or lighting conditions.

(19) Camera system 106 is attached in vehicle 100 and records present scene 108 ahead of vehicle 100. Camera system 106 may include one or multiple camera sensors. The system transmits its image data 104 to one or multiple processing units 102 having an installed FPGA via an interface, which may have a wired or wireless design. For example, the FPGA may also be integrated into on-board computer 112, combination instrument 112, camera control device 112 and/or navigation system 112.

(20) FIG. 2 shows a graphic representation of a method sequence 200 for identifying an object in a piece of image information 104 according to one exemplary embodiment of the present invention. Method sequence 200 may be carried out on a control device 102, as it is shown in FIG. 1. In the method according to one exemplary embodiment of the present invention, piece of image information 104 is processed from an input image 202 with the aid of an integral image generation 204 here. Feature descriptions 206 are stored in a memory 208 of control device 102. Memory 208 is an FPGA-internal memory 208 here. For a feature calculation 210, piece of image information 104 and feature descriptions 206 from memory 208 are accessed. The results of feature calculation 210 are checked via a cascading and linkage 212 and are output as results 214 when check 212 is successful.

(21) In one exemplary embodiment, the data processing is carried out on FPGA 102 as shown as a block diagram in FIG. 2. Description 206 of the area sum features is loaded into internal memory 208 of FPGA 102. Input image 202 of the camera system is read into FPGA 102. A summed area table (integral image) 104 is generated 204. Feature descriptions 206 stored in FPGA 102 are applied to the data of summed area table 104 for feature calculation 210. The results of feature calculations 210 are linked and subsequently cascaded 212. In this way, a response 214 on whether or not an object is present may be supplied for every image position. Results 214 of the analysis are output for further processing.

(22) In one exemplary embodiment, feature descriptions 206 are written into FPGA-internal memory 208 prior to the processing of an image 104. Prior to their use on FPGA 102, feature descriptions 206 have been trained for certain objects. To be able to detect different object classes in consecutive images 104, it is possible to also newly write feature descriptions 206 into FPGA-internal memory 208 prior to each new image 104.

(23) In one exemplary embodiment, summed area table (integral image) 104 is calculated based on input image 202 and due to its properties makes a simplified calculation basis possible in the subsequent processing 210.

(24) In one exemplary embodiment, feature calculation 210 analyzes feature descriptions 206 stored in the FPGA-internal memory 208 and extracts the characterized features from summed area table (integral image) 104.

(25) In one exemplary embodiment, linkage 212 of the previously calculated features is established by feature descriptions 206 in FPGA-internal memory 208. A tree structure is created by linkage 212 of the features, which is combined and cascaded in stages. After every stage, it is decided whether the cascade is continued or aborted. When all stages of the cascade have been successfully passed through, a possible object is situated at the present position. The hardware classification is calculated for every pixel position of the input image and thus determined for every pixel position, whether a possible object exists or not. Resolution losses due to filter matrices are taken into consideration.

(26) In one exemplary embodiment, a feature description 206 shall, figuratively speaking, be understood to mean a table which in the first column contains a consecutive number for distinguishing different features for a classification. In the following columns, this table then contains values for feature comparisons which make a partial or complete identification of an object in an image 104 possible. The final column then contains an instruction for further processing. This instruction may be a new line of the column, for example, or else an output of the result.

(27) In one exemplary embodiment, an integral image 104 shall be understood to mean an image which is calculated based on an image 202 recorded by a camera. The pixel value of integral image 104 is formed as a function of all pixel values above it and to its left. From this it is derived that the pixel value in the bottom right pixel is an average across all pixel values of the image. It is further derived that the pixel value of the top left pixel corresponds to that of original image 202.

(28) The elements referred to as feature descriptions 206 in the approach described here may be replaced during a run time of the PGA. This, in turn, may advantageously be made dependent on external parameters, such as weather conditions and/or lighting conditions, for example.

(29) FIG. 3 shows a block diagram of a control device 102 for identifying an object in a piece of image information according to one exemplary embodiment of the present invention. Control device 102 includes an interface 300 for reading in, a unit 302 for selecting, and a unit 304 for searching. The piece of image information represents a scene in a detection range of a camera during a situation. Interface 300 is designed to read in the piece of image information and at least one parameter representing the situation. Unit 302 for selecting is designed to select a feature combination of an object class of the object, which is predictably identifiable in the situation, using the parameter. Unit 304 for searching is designed to search for the feature combination in the piece of image information to identify the object.

(30) Control device 102 is in particular designed to be used in a vehicle, as it is shown in FIG. 1.

(31) FIG. 4 shows a flow chart of a method 400 for identifying an object in a piece of image information according to one exemplary embodiment of the present invention. Method 400 includes a step 402 of reading in, a step 404 of selecting, and a step 406 of searching. The piece of image information represents a scene in a detection range of a camera during a situation. In step 402 of reading in, the piece of image information and at least one parameter representing the situation are read in. In step 404 of selecting, a feature combination of an object class of the object, which is predictably identifiable in the situation, is selected using the parameter. In step 406 of searching, the feature combination is searched for in the piece of image information to identify the object.

(32) In one exemplary embodiment, an integral image is read in as a piece of image information in step 402 of reading in. Here, the integral image represents a line by line and/or column by column integration of brightness values of pixels of a camera image of the scene.

(33) In one exemplary embodiment, a first feature from the feature combination is used for checking in step 406 of searching in a first search step. Thereafter, at least one further feature from the feature combination is used for checking in at least one further search step.

(34) In one exemplary embodiment, the further feature for checking is selected from the feature combination in the further search step using a result of the preceding search step.

(35) In one exemplary embodiment, an identification probability for the object is provided in step 406 of searching. The more features of the feature combination are identifiable in the piece of image information, the greater an identification probability is provided.

(36) In one exemplary embodiment, pixels of the piece of image information are marked as the object in step 406 of searching when a minimum number of features of the feature combination is identified in the pixels.

(37) In one exemplary embodiment, the method includes a step of identifying the situation. The situation is identified using the scene in the piece of image information and/or vehicle data and/or environmental conditions.

(38) In one exemplary embodiment, a piece of class information of the object class to be identified is furthermore read in in step 402 of reading in. In step 404 of selecting, the feature combination representing the object class is selected from multiple different feature combinations using the piece of class information.

(39) In one exemplary embodiment, the feature combination includes a piece of information about at least one area sum of the object to be identified.

(40) The described exemplary embodiments shown in the figures are selected only by way of example. Different exemplary embodiments may be combined with each other completely or with respect to individual features. It is also possible to supplement one exemplary embodiment with features of another exemplary embodiment.

(41) Moreover, method steps according to the present invention may be carried out repeatedly and in a different order than the one described.

(42) If one exemplary embodiment includes an “and/or” linkage between a first feature and a second feature, this should be read in such a way that the exemplary embodiment according to one specific embodiment includes both the first feature and the second feature, and according to an additional specific embodiment includes either only the first feature or only the second feature.