METHOD FOR CAPTURING IMAGE MATERIAL FOR MONITORING IMAGE-ANALYSING SYSTEMS, DEVICE AND VEHICLE FOR USE IN THE METHOD AND COMPUTER PROGRAM

Abstract

Technologies and techniques for capturing image material for monitoring image-analyzing systems, wherein an object is monitored to determine if it has been correctly recognized by the image-analyzing system in respect of time or location. The images captured are recorded in a memory. When a discrepancy is determined, in object recognition beyond a tolerance limit over a temporal or locational reference, select images or image sections are archived for more precise inspection.

Claims

1-15. (canceled)

16. A method for capturing image material for an image analyzing system for a vehicle, comprising: receiving image data in a memory of the image analyzing system; executing, via a processor of the image analyzing system, first object recognition processing on the received image data; processing, via the processor, a recognized object to determine if one or more differences of the recognized object to a temporal or spatial reference exceeds a tolerance limit; and archiving at least portions of the image data for second object recognition processing if the one or more differences of the recognized object to a temporal or spatial reference exceeds the tolerance limit, wherein the second object recognition processing is more precise than the first object recognition processing.

17. The method of claim 16, wherein processing the recognized object comprises calculating a trajectory of a moving object, and wherein archiving at least the portion of the image data comprises archiving the image data based on the calculated trajectory.

18. The method of claim 17, further comprising performing a reverse calculation using the calculated trajectory to calculate a number of images or image sections in the image data in which the moving object may be visible, even if the moving object cannot be recognized via the first object recognition processing.

19. The method of claim 18, wherein a distance to the recognized object at which the object recognition takes place via the first object recognition processing is checked in order to determine whether the object recognition takes place at a correct time or place, and wherein a standard recognition distance is established, indicating a starting distance at which the second object recognition should recognize the object.

20. The method of claim 16, further comprising receiving position data for the recognized object, and processing the position data to determine a distance between the vehicle and the recognized object, for determining the one or more differences in the first object recognition processing.

21. The method of claim 20, wherein the determined distance comprises at least one of a standard recognition distance and an object recognition distance, wherein processing the recognized object comprises calculating a trajectory of a moving object, and wherein the archiving at least portions of the image data comprises transmitting the at least portions of the image data between the standard recognition distance and the object recognition distance to one of the memory of the image analyzing system or an external memory.

22. The method of claim 21, wherein the archiving at least portions of the image data comprises archiving at least portions of the image data with a higher quality and a lower quality.

23. An image analyzing system for capturing image material for a vehicle, comprising: a memory configured to receive image data; a processor, operatively coupled to the memory, wherein the processor and memory are configured to: executing, via a processor of the image analyzing system, first object recognition processing on the received image data; processing, via the processor, a recognized object to determine if one or more differences of the recognized object to a temporal or spatial reference exceeds a tolerance limit; and archiving at least portions of the image data for second object recognition processing if the one or more differences of the recognized object to a temporal or spatial reference exceeds the tolerance limit, wherein the second object recognition processing is more precise than the first object recognition processing.

24. The image analyzing system of claim 23, wherein the processor and memory are configured to process the recognized object by calculating a trajectory of a moving object, and wherein archiving at least the portion of the image data comprises archiving the image data based on the calculated trajectory.

25. The image analyzing system of claim 24, wherein the processor and memory are configured to perform a reverse calculation using the calculated trajectory to calculate a number of images or image sections in the image data in which the moving object may be visible, even if the moving object cannot be recognized via the first object recognition processing.

26. The image analyzing system of claim 25, wherein the processor and memory are configured to determine a distance to the recognized object at which the object recognition takes place via the first object recognition processing and checked in order to determine whether the object recognition takes place at a correct time or place, and wherein a standard recognition distance is established, indicating a starting distance at which the second object recognition should recognize the object.

27. The image analyzing system of claim 23, wherein the processor and memory are configured to receive position data for the recognized object, and process the position data to determine a distance between the vehicle and the recognized object, for determining the one or more differences in the first object recognition processing.

28. The image analyzing system of claim 27, wherein the determined distance comprises at least one of a standard recognition distance and an object recognition distance, wherein processing the recognized object comprises calculating a trajectory of a moving object, and wherein the archiving at least portions of the image data comprises transmitting the at least portions of the image data between the standard recognition distance and the object recognition distance to one of the memory of the image analyzing system or an external memory.

29. The image analyzing system of claim 28, wherein the processor and memory are configured to archive at least portions of the image data by archiving at least portions of the image data with a higher quality and a lower quality.

30. A computer-readable medium having stored therein instructions executable by one or more processors of an image analyzing system for a vehicle, to: receive image data in a memory of the image analyzing system; execute first object recognition processing on the received image data; process a recognized object to determine if one or more differences of the recognized object to a temporal or spatial reference exceeds a tolerance limit; and archive at least portions of the image data for second object recognition processing if the one or more differences of the recognized object to a temporal or spatial reference exceeds the tolerance limit, wherein the second object recognition processing is more precise than the first object recognition processing.

31. The computer-readable medium of claim 30, wherein the instructions are further configured to process the recognized object by calculating a trajectory of a moving object, and wherein archiving at least the portion of the image data comprises archiving the image data based on the calculated trajectory.

32. The computer-readable medium of claim 31, wherein the instructions are further configured to perform a reverse calculation using the calculated trajectory to calculate a number of images or image sections in the image data in which the moving object may be visible, even if the moving object cannot be recognized via the first object recognition processing.

33. The computer-readable medium of claim 32, wherein a distance to the recognized object at which the object recognition takes place via the first object recognition processing is checked in order to determine whether the object recognition takes place at a correct time or place, and wherein a standard recognition distance is established, indicating a starting distance at which the second object recognition should recognize the object.

34. The computer-readable medium of claim 30, wherein the instructions are further configured to receive position data for the recognized object, and processing the position data to determine a distance between the vehicle and the recognized object, for determining the one or more differences in the first object recognition processing.

35. The computer-readable medium of claim 34, wherein the determined distance comprises at least one of a standard recognition distance and an object recognition distance, wherein processing the recognized object comprises calculating a trajectory of a moving object, and wherein the archiving at least portions of the image data comprises transmitting the at least portions of the image data between the standard recognition distance and the object recognition distance to one of the memory of the image analyzing system or an external memory.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Exemplary embodiments of the present disclosure are shown in the drawings, and shall be explained below in greater detail in reference to these drawings.

[0060] Therein:

[0061] FIG. 1 shows a typical cockpit in a vehicle;

[0062] FIG. 2 shows an illustration of the principles of the various communication possibilities provided in a vehicle for external communication;

[0063] FIG. 3 shows a block diagram of the onboard electronics in the vehicle;

[0064] FIG. 4 shows a first illustration of a first driving situation in order to explain the problems in checking the functioning of an image analysis system in the vehicle under some aspects of the present disclosure;

[0065] FIG. 5 shows a second illustration of the first driving situation, wherein the second illustration shows the driving situation at an earlier point in time under some aspects of the present disclosure; and

[0066] FIG. 6 shows a flow chart for a program for capturing image material for checking the functioning of the image analysis system in the first driving situation under some aspects of the present disclosure;

[0067] FIG. 7 shows a first illustration of a second driving situation in order to explain the problems in checking the functioning of an image analysis system in the vehicle under some aspects of the present disclosure;

[0068] FIG. 8 shows a second illustration of the second driving situation, wherein the second illustration of the second driving situation shows the driving situation at an earlier point in time under some aspects of the present disclosure;

[0069] FIG. 9 shows a flow chart for a program for capturing image material for checking the functioning of the image analysis system in the second driving situation under some aspects of the present disclosure;

[0070] FIG. 10 shows an illustration of the variation in image quality for archiving image material on the basis of environmental conditions under some aspects of the present disclosure; and

[0071] FIG. 11 shows an illustration of the variation in recording times for archiving image material on the basis of environmental conditions under some aspects of the present disclosure.

DETAILED DESCRIPTION

[0072] The detailed description illustrates the principles of the disclosure according to the present disclosure. It is to be understood that persons skilled in the art are capable of conceiving of various assemblies which may not be explicitly described herein, but embody the principles of the disclosure according to the present disclosure, and are likewise protected in their scope.

[0073] FIG. 1 shows a typical cockpit of a vehicle 10. This is illustrated in the figure as a passenger automobile. Any other vehicle can also be regarded as the vehicle 10. Examples of other vehicles are: buses, utility vehicles, in particular trucks, agricultural machines, construction machines, rail vehicles, etc. The use of the present disclosure in general is possible with land vehicles, rail vehicles, watercraft and aircraft.

[0074] Two display units for an infotainment system are shown in the cockpit. These include a touch-sensitive screen 30, which is installed in the central console, and the instrument cluster 110, which is in the dashboard. The central console is not in the field of vision of the driver while driving. For this reason, additional information is not displayed on the display unit 30 while driving.

[0075] The touch-sensitive screen 30 is used for operating functions in the vehicle 10. By way of example, a radio, navigation system, music playback and/or air conditioning, other electronic devices, or other comfort functions or applications in the vehicle 10 can be controlled therewith. On the whole, this is frequently referred to as an “infotainment system.” An infotainment system in a motor vehicle, in particular a passenger vehicle, refers on the whole to a radio, navigation system, hands-free speakerphone, advanced driver-assistance system, and other functions in a central operating panel. The term, “infotainment” is a portmanteau, combining the words “information,” and “entertainment.” Mainly, the touch-sensitive screen 30 (“touchscreen”) is used to operate the infotainment system, wherein this touchscreen 30 can be easily seen and operated in particular by a driver of the vehicle 10, but also by a passenger in the vehicle 10. Mechanical operating elements, e.g., buttons, dials, or combinations thereof, such as push-button knobs, can be located in an input unit 50 beneath the touchscreen 3. Typically, parts of the infotainment system can also be operated from the steering wheel. This is not shown separately, but is regarded as part of the input unit 50.

[0076] FIG. 1 also shows a head-up display 20. The head-up display 20 is attached behind the instrument cluster 110 in the region of the dashboard, from the perspective of the driver. This is an image projection unit. By projecting it onto the windshield, additional information is shown in the field of vision of the driver. This additional information appears as though it is projected onto a projection surface 21 at a distance of 7-15 meters in front of the vehicle 10. The real world still remains visible, however, through this projection surface 21. A virtual environment is basically generated with the display of the additional information. The virtual environment is theoretically superimposed on the real world, and contains virtual objects that help and inform the driver while driving. This is only projected onto part of the windshield, however, such that the additional information cannot be placed arbitrarily in the driver's field of vision. This manner of display is also referred to as “augmented reality.”

[0077] FIG. 2 shows a system architecture for mobile vehicle communication. The vehicles 10 are equipped with an onboard communication module 160 comprising corresponding antenna units, such that it can participate in various types of vehicle communication, e.g., V2V and V2X. FIG. 1 shows that the vehicle 10 can communicate with a mobile base station 210 for a mobile communication provider.

[0078] Such a base station 210 can be an eNodeB base station from an LTE mobile communication provider (Long Term Evolution). The base station 210 and the corresponding equipment are part of a mobile communication network that comprises numerous mobile cells, wherein each cell is operated by a base station 210.

[0079] The base station 210 is located near a main street on which the vehicles 10 travel. In the terminology of LTE, a mobile end device corresponds to user equipment UE, which enables a user to access network services, connecting the user to UTRAN or Evolved-UTRAN via the mobile interface. This user equipment typically comprises a smartphone. These mobile end devices are used by the passengers in the vehicles 10. The vehicles 10 are also each equipped with an onboard communication module 160. This onboard communication module 160 can also be equipped with a WLAN p-module, in order to participate in an ad hoc V2X communication mode. V2V and V2X communication is also supported by the new 5.sup.th generation of mobile communication systems. This mobile interface therein is referred to as a PC5 interface. With regard to the LTE mobile communication system, the Evolved UMTS Terrestrial Radio Access Network E-UTRAN from LTE is composed of numerous eNodeBs, which provide the E-UTRA user levels (PDCP/RLC/MAC/PHY) and the control levels (RRC). The eNodeBs are connected to one another by means of the so-called X2 interfaces. The eNodeBs are also connected to the EPC (Evolved Packet Core) 200 via the so-called 51 interfaces.

[0080] From this general architecture, FIG. 2 shows that the basis station 210 is connected to the EPC 200 via the 51 interface, and the EPC 200 is connected to the internet 300. A backend server 320, to and from which the vehicles 10 can send and receive messages, is also connected to the internet 300. In this case, the backend server 320 can be located in a computing center for the vehicle manufacturer. Lastly, a street infrastructure station 310 is also shown. This can be illustrated by a unit on the street, commonly referred to in the art as a “road side unit,” RSU 310. To simplify the implementation thereof, it is assumed that all of the components are assigned an internet address, typically in the form of an IPv6 address, such that the packets that transport messages between the components can be routed accordingly. The various interfaces specified above are standardized. Reference is made in this regard to the corresponding LTE specifications, which are published.

[0081] FIG. 3 shows a schematic illustration of a block diagram for the onboard electronics 200, to which the infotainment system in the vehicle 10 also belongs. The touchscreen 30, a computer 40, an input unit 50, and a memory 60 are used to operate the infotainment system. The touchscreen 30 comprises both a display screen for displaying changing graphic information, as well as an operating surface (touch-sensitive layer) located on top thereof, with which the user can input commands. It can be an LCD touchscreen.

[0082] The touchscreen 30 is connected to the computer 40 via a data line 70. The data line can be configured according to the LVDS standard (Low Voltage Differential Signaling). The touchscreen 30 receives control data for controlling the display surface in the touchscreen 30 from the computer 40 via the data line 70. The input unit is indicated by the reference numeral 50. The aforementioned operating elements, such as buttons, dials, sliders or push-button knobs, with which an operator can make inputs via the menu navigation. An input refers in general to the selection of a menu option, such as the changing of a parameter, i.e., switching a function on and off, etc.

[0083] The memory 50 is connected to the computer 40 via a data line 80. A pictogram index and/or symbol index is store in the memory, containing pictograms and/or symbols for the possible displaying of additional information.

[0084] The other parts of the infotainment system, e.g., a camera 150, radio 140, navigation unit 130, telephone 120, and instrument cluster 110, are connected to the device for operating the infotainment system via a data bus 100. High-speed variants of the CAN bus according to the ISO standard 11898-2 can be used as the data bus. Alternatively, an ethernet-based bus system such as BroadR-Reach could also be used. Bus systems using optical fibers for data transfer can also be used. Examples thereof are the MOST bus (Media Oriented System Transport) or the D2B bus (Domestic Digital Bus). A vehicle measurement unit 170 is also connected to the data bus 100. This vehicle measurement unit 170 is used to detect movement by the vehicle, in particular accelerations thereof. It can be a conventional IMU unit (Inertial Measurement Unit). An IMU unit typically contains acceleration sensors and rotational rate sensors such as a laser gyroscope or a magneto-gyroscope. The vehicle measurement unit 170 can be regarded as part of the odometry for the vehicle 10. These also include wheel speed sensors.

[0085] It should also be noted here that the camera 150 can be a conventional video camera. In this case, it records 25 fps, corresponding to the interlace recording mode of 50 half-images/second. Alternatively, a special camera can be used, that records more frames per second, in order to increase the precision in recognizing objects that move faster, or which record light in a spectrum other than that of visible light. Numerous cameras can be used for environmental observation. The aforementioned radar or lidar systems 152 and 154 can also be used for scanning the environment, or expanding the scanning thereof. The vehicle 10 is equipped with the communication module for wireless internal and external communication.

[0086] The camera 150 is used primarily for object recognition. Typical objects that are to be recognized are traffic signs, leading, surrounding, and parked vehicles, and other road users, intersections, turns, potholes, etc. If an object of potential significance is recognized, information can be output via the infotainment system. Typically, symbols for the recognized traffic signs are shown. This can also be a warning message, if the object poses a hazard. The information is projected directly in the field of vision by the HUD 20. An example of an object that poses a danger is the case in which the image analysis of the images provided by the camera 150 indicates that a vehicle is approaching an intersection from the right that the vehicle itself is also approaching. The image analysis takes place in the computer 40. Known algorithms for object recognition can be used for this. As a result, a danger symbol is shown where the vehicle is located. The display thereof takes place such that the danger symbol does not conceal the vehicle, because the driver would otherwise not be able to determine precisely what the danger is.

[0087] The object recognition algorithms are processed by the computer 40. The number of images that can be analyzed per second depends on the capacity of the computer. A function for automated driving typically has various phases: in the perception phase, data from various environment sensors are processed, and combined on a high-performance platform. The vehicle must also be located on a very precise digital map, on the basis of its GNSS position. The data are used to generate a three-dimensional model of the environment, and provide information regarding dynamic and stationary objects, as well as open spaces surrounding the vehicle (object list). In some circumstances, the precision of the GNSS position in insufficient. For this reason, odometry data in the vehicle are also drawn on, in order to improve the precision of the position determination.

[0088] A motor control unit is indicated by the reference numeral 181. This reference numeral 182 corresponds to an ESP control unit, and the reference numeral 183 refers to a transmission control unit. There can also be other control units in the vehicle, such as an additional driving dynamic control unit (for vehicles with an electrically adjustable suspension), an airbag control unit, etc. The interconnection of these control units, all of which can be attributed to the category of the drive train, normally takes place with the CAN bus system (Controller Area Network) 104, which is standardized according to the ISO standard, usually as ISO d11898-1. Some of the sensors 171 to 173 in a motor vehicle, which are not normally connected to individual control units, are connected to the bus system 104, and their sensor data are sent to the individual control units via the bus. Examples of sensors in a motor vehicle are wheel speed sensors, steering angle sensors, acceleration sensors, rotational rate sensors, tire pressure sensors, distance sensors, knock sensors, air quality sensors, etc. In particular, the wheel speed sensors and the steering angle sensors are part of the odometry for the vehicle. The acceleration sensors and rotational rate sensors can also be connected directly to the vehicle measurement unit 170.

[0089] FIGS. 4 and 5 illustrate a simplified functioning of an image analysis system in one embodiment, as it is used in vehicles. These show an image recorded by the front camera 150 on the vehicle 10. This is an example of traffic sign recognition. FIG. 4 shows the image recorded when the vehicle is close to the traffic sign 15. FIG. 5 shows an image recorded at a greater distance to the traffic sign 15, indicating a no passing zone. The traffic sign recognition system is required to recognize a traffic sign under normal conditions at a standard recognition distance of 80 meters to the traffic sign, as the vehicle 10 approaches the traffic sign 15. FIG. 4 shows the image in the situation in which the vehicle has reached a distance of 20 meters. In FIG. 5, the vehicle 10 is still 40 meters form the traffic sign 15. According to the image analysis, the traffic sign is first recognized when the vehicle is 40 meters away from the traffic sign 15. The checking function of the image analysis system establishes that the traffic sign recognition is late under these conditions. The checking function then executes a test, with which a preliminary analysis of the recorded image takes place. This shows that a truck 13 can be seen in front of the vehicle in the image. As can be seen in FIG. 5, the traffic sign 15 is just to the right of the truck 13. The checking function then establishes that the traffic sign 15 is concealed by the truck at a greater distance. For this reason, the images recorded between 80 meters and the actual traffic sign recognition at 40 meters are archived at a lower quality.

[0090] FIGS. 4 and 5 can also be used to explain another erroneous recognition situation. In these examples, a traffic sign 15 indicating a no passing zone for trucks is first recognized at a distance of 40 meters. At a distance of 20 meters to the traffic sign 15, it is first recognized that this traffic sign 15 indicates an absolute no passing zone. In this situation, the checking function reacts as follows: by storing the marked area 17 of the recorded image, it is archived in the images of a higher quality. The rest of the images are recorded with a lower quality. This is the case for all images recorded between 40 meters and 20 meters thereto.

[0091] FIG. 6 shows a flow chart for a program with which the checking function is implemented under some aspects of the present disclosure. This variation of the program is intended for the second variation, in which an erroneous recognition of the traffic sign first takes place, and a correct recognition first takes place when the vehicle is closer to the traffic sign. The start of the program is indicated by the reference numeral 310. The algorithm for traffic sign recognition is processed in step 312. This is an object recognition algorithm, which carries out pattern recognition on the basis of patterns stored in a table. All of the valid traffic signs are known and their patterns can be stored in a table. Typically, the pattern recognition is improved by a folding operation, in which the recorded images are folded together with the known patterns. These algorithms are known and available to the person skilled in the art. If a traffic sign is recognized in this manner, the distance at which it is recognized is also determined and stored.

[0092] It is checked in step 314 whether or not the traffic sign recognition complies with the standard. In this case, the traffic sign recognition was erroneous for two reasons. First, the recognition did not take place at the standard distance of 80 meters from the traffic sign. Furthermore, a limited no passing was first recognized, which was then revised to an absolute no passing as the sign was approached. If a traffic sign recognition complying with the standard is established in this step, no images need to be recorded for control purposes, and the program stops at step 322. In this case, however, an erroneous recognition was detected. At this point, the program continues in step 316. A calculation of problematic image takes place in step 316. Because the standard recognition distance is not obtained, the calculation can take place as follows. The first recognition takes place at 40 meters. The correct recognition first takes place at 20 meters. The images of interest are therefore those recorded between the standard recognition distance of 80 meters and 20 meters.

[0093] It is also checked whether the erroneous recognition can be explained in step 316. It is determined through image analysis that a truck 13 is in front of the vehicle, and it has presumably concealed the traffic sign 15 until the first recognition thereof. It is thus concluded that the images recorded between 80 meters and 40 meters are less important for the subsequent check. These images are therefore recorded at a lower quality.

[0094] The problematic image sections are calculated in step 318. In the case of the traffic sign recognition in FIGS. 4 and 5, the image section that is marked around the traffic sign 15 is selected as relevant. This image section is therefore selected with a higher quality at a distance ranging from 40 to 20 meters.

[0095] The problematic images and problematic image sections are then sent from the memory 60 to the communication module 160, and the communication module 160 sends these images to the backend server 320 via mobile communication in step 320. They are archived there. The archived images are later analyzed by experts or by machines with artificial intelligence in the computing center. The check has the purpose of detecting potential problems in the image analysis system. The results of the checks can be used to improve the analysis algorithms. Ideally, the improved analysis algorithm can then be returned to the vehicle 10 via an OTA (Over the Air) download, and installed therein. It may also be the case that a systemic error is detected in the check, which is due to the camera 150 being out of adjustment. In this case, a message can be sent to the vehicle 10, in order to inform the driver that the vehicle needs to be brought in for service. The program stops at step 322.

[0096] An example is shown in FIG. 7, in which a vehicle approaches a road having right of way. Another vehicle 12 on this road is first recognized prior to turning onto the road by analyzing the images and/or the data from the radar and lidar sensors 154, 152. The autonomous vehicle 10 must stop quickly for this reason. This may be uncomfortable for the passengers. This abrupt braking should be avoided, particularly with autonomous vehicles, in order to give the passengers a feeling of security. At least in the test phase, this is classified as not desirable by the onboard electronics. The same can take place in the normal operation, i.e. after the test phase. There are numerous possibilities for this. A first possibility is that this classification is made if a predefined acceleration value is exceeded. Another possibility is that the classification is made if the occupants or driver have a negative reaction that the driver/occupant state detection system recognizes. Another possibility for this classification is if the minimum distance to the object recognition is not maintained (in the case of a vehicle approaching from the right).

[0097] In order to reduce the amount of image data that needs to be stored, only those image sections from the ring buffer are stored or sent in which the object can be found.

[0098] FIG. 8 shows an image of the same situation illustrated in FIG. 7, at an earlier point in time. In this example, the vehicle 12 approaching from the right is further away from the intersection. FIG. 8 even shows an image at a point in time at which the vehicle 12 is not yet recognized by the object recognition as an approaching vehicle. This can be for a number of reasons. The vehicle may still be too far away, and the object recognition algorithm may not yet be able to recognize the vehicle as such from the available image information. Or the vehicle 12 may be hidden by other objects (e.g., trees or bushes).

[0099] The reason for why the approaching vehicle 12 is recognized so late should be determined through subsequent analysis. For this reason, it is determined which images recorded by the camera, and potentially by other imaging sensors, should be archived for subsequent analysis. As a result, the amount of data that is to be archived should be reduced as much as possible.

[0100] How this takes place is explained below in reference to FIG. 9. In this example, FIG. 9 shows a flow chart for a program with which the checking function is implemented in another embodiment of the present disclosure. The program start is indicated by the reference numeral 340. The program is started when a hazard situation has been recognized, in which a moving object is recognized too late. The trajectory of the moving object that has been recognized too late is estimated in step 342. This takes place in the following manner: the current speed and acceleration of the object is determined at the point in time when the object is recognized from numerous successive video images and drawing on the various sensor data form the observer vehicle 10. A trajectory of the object is estimated therewith, and potentially drawing on the geometry of the route known from the map in the navigation system 130. In conjunction with the estimation of the movement of the vehicle, an image section 14 can be determined in the recorded image data for the point in time in the past, within which the object (not yet recognized at this point in time) must have been located. This image section 14 is also referred to as a “bounding box.” It can be assumed that the bounding box would be larger at earlier points in time, than at the points in time that are not so far in the past. The size of the bounding box is affected by the object parameter (e.g., size) at the time of the recognition, as well as by the uncertainties, possible alternatives in the trajectory, known sensor errors/imprecisions, and empirically determined factors.

[0101] The estimated trajectory is used for a reverse calculation in step 344, in order to determine a number of problematic images with which it is to be checked why the approaching vehicle 12 was not recognized therein. The distance between the observing vehicle 10 and the approaching vehicle 12 is also taken into account therein. If the distance exceeds a predefined limit, there is no need to archive any earlier images. For the various environment sensors, the respective distances until which the desired objects can still be recognized are known. The program then continues at step 346. A calculation of the number of problematic images takes place in step 346.

[0102] The respective bounding boxes for the number of problematic images are calculated in step 348. This also takes the estimated trajectory into account.

[0103] In another variant, it is also determined whether not recognizing an object can be due to a possible concealment of the relevant object by other, possibly correctly recognized objects (other vehicles, houses/walls/noise protection walls, trees, bushes, forests) known from navigation data. In this case, the images may not be transmitted. The time period for problematic images may be limited in that the assumed object trajectory is obvious, or in that the relevant object was not yet within range of the camera at an earlier point in time.

[0104] The problematic image sections 14 are then sent from the ring buffer in the memory 60 to the communication module 160 in step 348, and the communication module 160 sends these image data to the backend server 320 via a mobile connection. They are archived there. The archived images are later analyzed by experts or using machines with artificial intelligence in the computing center. The check is used to detect potential problems in the image analysis system. The results of the checks can be used to improve the analysis algorithms. Ideally, the improved analysis algorithm can be returned to the vehicle 10 via an OTA download (Over the Air), and installed therein. It may also be the case that a systemic error is detected in the check, which is due to the camera 150 being out of adjustment. In this case, a message can be sent to the vehicle 10, in order to inform the driver that the vehicle needs to be brought in for service. The program stops at step 350.

[0105] The number and quality of the images that are to be archived may be affected by various factors. In addition to the cases presented above, the following factors may also be specified:

[0106] Weather Conditions

[0107] With adverse weather conditions, vision is severely limited. This may be such that it is no longer possible to recognize any objects. In this case, other environment detection sensors such as radar and lidar sensors may provide better results. In any case, the number of images or image sections that are to be archived can be limited with adverse weather conditions, because the objects in question cannot be detected at the standard recognition distance.

[0108] Position Precision

[0109] The precision of the position determination based on GNSS signals and odometry signals can also be dependent on the weather. It may also depend on other factors. One example thereof is the environment in which the vehicle is travelling. In a city, satellite signal reception may be limited by construction. This can also be the case when travelling overland. Satellite signals can be weakened in forests. There may also be poor reception in mountains, due to tectonics. In these cases, it is therefore proposed that more images or image sections are captured. It is then more probable that the relevant sections of the route will also be recorded, despite imprecise positioning.

[0110] Quality of the Road Surface

[0111] The quality of the road surface can also have a similar impact. Strong vibrations occur on cobblestones, such that the images that are recorded may be blurred. On slick surfaces, due to ice, snow or rain, a friction coefficient is determined by the drive-slippage control. If the slippage is large enough, the odometry data are no longer reliable, and these effects should be taken into account in the effects impacting the positioning precision.

[0112] Time of Day

[0113] Image quality varies significantly over the course of the day. A distinction should be made at least between daytime and nighttime. Data from radar or lidar sensors should be used more than camera data at night.

[0114] Traffic Conditions

[0115] A distinction can be made here between urban traffic, highway traffic, and rural traffic. In urban traffic, a particularly precise detection of traffic signs is important. In this case, the number of captured images may be increased. In slow traffic on highways, the number of captured images can be reduced.

[0116] Temporary storage of the images or image sections 14 that are to be archived takes place in the vehicle 10. The memory 60 can be used for this. With the practical implementation there is a ring buffer configured for this in the memory 60. This is managed such that the newly captured images are successively written into the free memory area in the ring buffer. When the free memory area is full, the already used part of the ring buffer is overwritten. By using the allocated memory area as a ring buffer, the oldest part of memory is always overwritten. The images or image sections 14 can be stored in a compressed or uncompressed format. It is recommended, however, to use a lossless compression format, to avoid loss of any relevant image content. One example of a lossless compression format if the FFmpeg codec. The images are stored in a corresponding file format. Various data container formats can be used for this. One example thereof is the ADTF format, which was developed for the automotive industry, and the TIFF format. Other container formats for storing image and sound data are MPEG, Ogg, Audio Video Interleave, DIVX, Quicktime, Matroska, etc. If the images are sent to a backend server 320 while the vehicle is underway, it may be sufficient to configure the ring buffer for a 20 second recording period. There can also be a separate memory in the vehicle on which the images are archived. A USB hard drive can be used for this, which can be removed to connect it to a computer which is then used to analyze the archived images.

[0117] FIG. 10 shows how the amount of data that is to be archived can be reduced by reducing the image quality. The amount of image data for the various image qualities: full HD, with a resolution of 1920×1080 pixels, SD, with a resolution of 640×480 pixels, and VCD, with a resolution of 352×288 pixels, are listed from top to bottom.

[0118] FIG. 11 shows the effects of varying the number of images or image sections 17 on the basis of the positioning precision or weather conditions, etc., under some aspects of the present disclosure. In this example, if the length of recording is increased to a distance of 160 meters due to positioning precision, there is twice as much data to be archived. If the distance is reduced to 40 meters, there is only half as much.

[0119] All of the examples contained herein, as well as the certain formulations, are not to be limited to the specific examples listed herein. The person skilled in the art knows that the block diagrams shown herein represent a conceptual view of an exemplary circuit arrangement. Similarly, it should also be noted that a flow chart, state transition diagram, pseudocode, etc. represents various variations for illustrating the processes that are substantially stored in computer-readable media, and can thus be executed by a computer or processor. The objects specified in the claims can also expressly be people.

[0120] It should be noted that the proposed method, and the associated devices can be implemented in the form of software, hardware, firmware, special processors, or a combination thereof. Special processors can comprise application-specific integrated circuits (ASICs), reduced instruction set computers (RISCs) and or field-programmable gate arrays (FPICs). The proposed method and device are preferably implemented as a combination of hardware and software. The software is preferably installed as an application on a program memory. This is typically a computer platform-based machine, which contains hardware such as one or more central processing units (CPUs), a random-access memory (RAM), and one or more input/output (I/O) interface(s). An operating system is also typically installed on the computer platform. The various processes and functions described herein can be part of the application, or a part that is executed via the operating system.

[0121] The disclosure is not limited to the exemplary embodiments described herein. There is room for adjustments and modifications, which the person skilled in the art would also regard as belonging to the disclosure due to his expert knowledge.

[0122] The present disclosure has been explained in greater detail in reference to the exemplary embodiments using its use in vehicles, by way of example. Reference is also made here to the use thereof in airplanes and helicopters, e.g., for landing maneuvers or when used for searches, etc.

[0123] The present disclosure can also be used in remote controlled devices such as drones and robots that make significant use of image analysis. Other application possibilities include smartphones, tablets, personal assistants, or data glasses.

LIST OF REFERENCE SYMBOLS

[0124] 10 observer vehicle [0125] 12 approaching vehicle [0126] 13 leading vehicle [0127] 14 first relevant image section [0128] 15 traffic sign [0129] 17 second relevant image section [0130] 20 head-up display [0131] 30 touchscreen [0132] 40 computer [0133] 50 input unit [0134] 60 memory [0135] 70 data line to display [0136] 80 data line to memory [0137] 90 data line to input unit [0138] 100 data bus [0139] 110 instrument cluster [0140] 120 telephone [0141] 130 navigator [0142] 140 radio [0143] 142 gateway [0144] 144 onboard diagnosis interface [0145] 150 camera [0146] 160 communication module [0147] 170 vehicle measurement unit [0148] 171 sensor 1 [0149] 172 sensor 2 [0150] 173 sensor 3 [0151] 180 control unit for automatic driving function [0152] 181 motor control unit [0153] 182 ESP control unit [0154] 183 transmission control unit [0155] 200 evolved packet core [0156] 210 base station [0157] 300 internet [0158] 310 roadside unit [0159] 320 backend server [0160] 330-336 various program steps in a first computer program [0161] 340-350 various program steps in a second computer program [0162] Uu wireless interface for communication UE and eNodeB [0163] PC5 wireless interface for vehicle direct communication