Abstract
Method for monitoring a computer vision system (CVS), said computer vision system (CVS) being part of a vehicle control system (VCS) of a vehicle (1000) that is used to maneuver said vehicle (1000) in 3D-space (3000), said computer vision system (CVS) being configured to monitor a surrounding area of the vehicle in real time and said computer vision monitor (CVM) monitoring the behavior of the computer vision system (CVS), comprising the steps of a.) providing the computer vision monitor (CVM) with information concerning a position (LM_POS) of at least one landmark (2000) in the 3D-space (3000), wherein said information is provided by a source, said source being independent of the computer vision system (CVS), b.) providing the computer vision monitor (CVM) with information concerning a current position (CUR_POS) of the vehicle (1000), c.) selecting based on steps a.) and b.) at least one landmark which falls within the range of vision of the computer vision system (CVS), d.) classifying the computer vision system (CVS) as being faulty when the computer vision system (CVS) fails to detect a configurable number of selected landmarks (2000).
Claims
1. A method for monitoring a computer vision system (CVS) by a computer vision monitor (CVM), said computer vision system (CVS) being part of a vehicle control system (VCS) of a vehicle (1000) that is used to maneuver said vehicle (1000) in 3D-space (3000), said computer vision system (CVS) being configured to monitor a surrounding area of the vehicle in real time and said computer vision monitor (CVM) monitoring the behavior of the computer vision system (CVS), the method comprising the steps of a.) providing the computer vision monitor (CVM) with information concerning an expected position (LM_POS) of at least one landmark (2000) with regard to the 3D-space (3000), wherein said information is provided by a source, said source being independent of the computer vision system (CVS), b.) providing the computer vision monitor (CVM) with information concerning a current position (CUR_POS) of the vehicle (1000) with regard to the 3D-space, c.) selecting based on steps a.) and b.) at least one landmark which falls within the range of vision of the computer vision system (CVS), d.) classifying the computer vision system (CVS) as being faulty when the computer vision system (CVS) fails to detect a configurable number of selected landmarks (2000).
2. The method of claim 1, wherein in step d) the computer vision monitor (CVM) uses the information of steps a) and b) to determine an expectancy value with reference to at least one selected land mark (2000) and wherein said expectancy value is compared with information provided by the computer vision system (CVS), wherein the computer vision monitor classifies the computer vision system (CVS) as being faulty when the difference between the expectancy value and the information provided by the computer vision system (CVS) exceeds a predetermined threshold.
3. The method of claim 1, wherein the computer vision monitor (CVM) uses natural landmarks.
4. The method of claim 1, wherein artificial landmarks are explicitly placed in the 3D-space as part of the computer vision monitor (CVM) method.
5. The method of claim 1, wherein in case that the computer vision monitor (CVM) detects a failure of the computer vision system (CVS) the vehicle control system (VCS) brings the vehicle into a safe state.
6. The method of claim 1, wherein in step a.) knowledge of the position (LM_POS) of at least one landmark (2000) is provided by a landmark maintenance center (4000), said landmark maintenance center (4000) being configured to document misbehaviors as reported by one or many computer vision monitors CVM of one or many vehicles, wherein the vehicle communicates/reports the computer vision system (CVS) detected failures and corresponding landmarks (2000) to the landmark maintenance center (4000).
7. The method of claim 6, wherein the maintenance center (4000) triggers a maintenance activity, such as sending a repair crew to the damaged landmark's site.
8. The method of claim 1, wherein in step b.) knowledge of the current position (CUR_POS) of the vehicle (1000) is provided by means of a Global Positioning System (GPS) system.
9. The method of claim 1, wherein in step b.) knowledge of the current position (CUR_POS) of the vehicle (1000) is provided by a landmark (2000), in particular by means of a wireless connection.
10. The method of claim 1, wherein in step a.) knowledge of the position (LM_POS) of at least one landmark (2000) is provided by a landmark (2000), in particular by means of a wireless connection.
11. A system for monitoring a computer vision system (CVS) comprising a computer vision monitor (CVM), said computer vision system (CVS) being part of a vehicle control system (VCS) of a vehicle (1000), said computer vision system (CVS) being configured to monitor a surrounding area of the vehicle in real time, said system being configured to perform the method of claim 1.
Description
BRIEF DESCRIPTION OF FIGURES
[0025] In the following we discuss several exemplary embodiments of the invention with reference to the attached drawings. it is emphasized that these embodiments are given for illustrative purpose and are not to be construed as limiting the invention.
[0026] FIG. 1 depicts a 3D-space in which a landmark is positioned and a vehicle moves around.
[0027] FIG. 2 depicts relations between the elements related to the computer vision system.
[0028] FIG. 3 depicts a computer vision monitor method according to the invention.
[0029] FIG. 4 depicts the interaction between the computer vision monitor and the vehicle control system in more detail.
[0030] FIG. 5 depicts a 3D-space together with a vehicle and a landmark.
[0031] FIG. 6 depicts an extended realization of a computer vision monitor.
[0032] FIG. 7 depicts another extended realization of the computer vision monitor.
[0033] FIG. 8 depicts an exemplary operation of a landmark maintenance center.
[0034] FIG. 9 depicts a vehicle equipped with a computer vision monitoring system according to the invention.
[0035] FIG. 10 depicts a vehicle equipped with another variant of a computer vision monitoring system according to the invention.
EXEMPLARY EMBODIMENTS
[0036] In the following we discuss exemplary embodiments of many possible embodiments of the invention, which can be freely combined unless stated otherwise.
[0037] In FIG. 1 a 3D-space 3000 is depicted in which a landmark 2000 is positioned and in which a vehicle 1000 moves around. The position LM_POS of the selected landmark 2000 is known to the vehicle 1000. Examples for landmarks 2000 include geographic entities like a hill, a mountain, or courses of rivers, road signs, or visuals on a road or next to a road, or buildings or monuments. The vehicle 1000 may for example obtain the knowledge of the position LM_POS of the associated landmark 2000 from a source, said source being independent of the computer vision system CVS. This source can comprise a vehicle-local storage medium such as a flash-drive, hard disk, etc. The vehicle 1000 may also obtain the knowledge of the position LM_POS of the associated landmark 2000 from a remote location, for example a data center, via a wireless connection. Furthermore, the vehicle 1000 has means to establish its current location CUR_POS in the 3D-space 3000, e.g., by means of the Global Positioning System (GPS). The landmarks 2000 can be existing landmarks, such as traffic signs, geological factors, etc. or landmarks particularly placed in the 3D-space as part of the computer vision monitoring CVM method. For example, the landmarks 2000 can be dedicated road signs or other visuals on a road or next to a road installed in the 3D-space 3000 that are especially installed for the computer vision monitoring method CVM, so called artificial landmarks. An example for an artificial landmark is a board having a particular shape or containing a particular symbol, which can be easily recognized by a computer vision system CVS. Such symbols can be geometric forms as rectangles or triangles having a strong contrast to surrounding space. The symbols can be for example in white colour on dark background or vice versa. The board can be shaped like a road sign. Examples for such road signs or other visuals are signs or visuals that visualize an individual person, or groups of people, or one or a multitude of vehicles.
[0038] In FIG. 2 the relations between the vehicle 1000, the vehicle control system VCS, the computer vision system CVS, the computer vision monitor CVM, a communication subsystem CSS, as well as, vehicle actuators are depicted: [0039] The vehicle 1000 incorporates a vehicle control system VCS. [0040] The vehicle control system VCS incorporates a computer vision system CVS and a computer vision monitor CVM. The computer vision system CVS being able to monitor at least parts of the surrounding of the vehicle 1000 in real-time, i.e., it is capable to capture and process images acquired of said parts of the surrounding of the vehicle fast enough such that maneuvering actions of the vehicle 1000 can be deduced from the captured and processed images. [0041] The vehicle control system VCS communicates with vehicle actuators VAC using a communication subsystem CSS.
[0042] In FIG. 3 the computer vision monitor method is depicted in detail. The method includes the following steps: [0043] CVM_001: Assessing the current vehicle position CUR_POS, e.g., by means of GPS [0044] CVM_002: Selecting a landmark 2000 in the range of the computer vision system CVS of the vehicle control system VCS [0045] CVM_003: Evaluating whether the computer vision system CVS detects the landmark 2000 selected in CVM_002, [0046] CVM_004: The CVM classifying the computer vision system CVS as being faulty when the CVS fails to detect one or a defined multitude of selected landmarks 2000 in step CVM_002. In particular, a detection fault can recognized as such, when the computer vision monitor CVM calculates an expectancy value with reference to at least one selected landmark 2000 falling in the range of vision of the computer vision system CVS, wherein said expectancy value is compared with information provided by the computer vision system CVS, and the difference between the expectancy value and the information provided by the computer vision system CVS exceeds a predetermined threshold. Such a threshold be defined as for example by a time criteria: In case the position of a vehicle is in proximity to a specific landmark 2000, said landmark falling within the range of vision of the CVS, the computer vision system CVS can be classified as being faulty in case the computer vision system CVS fails to recognize the landmark 2000 within a particular period of time, for example 10 ms. Also, another criterion for a threshold can be given by taking the time into consideration in which a particular landmark 2000 is detected by the computer vision system CVS. In case a specific landmark 2000 has already left the range of vision of a CVS (as a consequence of vehicle movement) this landmark 2000 should not be recognized by the CVS anymore. If the computer vision system CVS still signals to recognized a landmark 2000 being already out of the range of vision of the CVS, the computer vision system CVS can be classified as being faulty (“system freeze”). In a preferred embodiment landmarks a placed in a proximity to each other, that allows the computer vision system to recognize at least two landmarks 2000 at the same time. [0047] CVM_005: The CVM reporting the unexpected CVS behavior to the vehicle control system VCS for further processing.
[0048] In FIG. 4 the interaction between the computer vision monitor CVM and the vehicle control system is depicted in more detail: [0049] VCS_001: the VCS collects information of the CVS misbehavior, e.g., the CVM reports that the CVS failed to detect one or a defined multitude of consecutive landmarks 2000 [0050] VCS_002: once the number and/or type of reported CVS misbehaviors reaches a given threshold (for example one, two, three, or more), the VCS triggers some vehicle 1000 action or a multitude of vehicle 1000 actions, for example, [0051] it signals to stop the vehicle 1000, or [0052] it disables the CVS system and it notifies the vehicle 1000 operator that the camera vision system CVS is disabled.
[0053] In FIG. 5 again a 3D-space 3000 is depicted together with a vehicle 1000 and a landmark 2000, in addition, in this 3D-space 3000 also a landmark maintenance center 4000 is depicted, said land mark maintenance center 4000 providing the vehicle with knowledge of the position of landmarks 2000. The vehicle 1000 is capable of communicating directly or indirectly with a landmark maintenance center 4000, e.g., using one or many wireless communication link or links, for example following telecom standards such as 3GPP or IT standards such as IEEE 802.11 or some following or upcoming standards.
[0054] In FIG. 6 an extended realization of the computer vision monitor CVM is depicted. CVM_006: when the CVM detects an unexpected CVS behavior, it reports the CVS misbehavior, for example that the CVS failed to detect one, two, or a multitude of the landmarks 2000, to the landmark maintenance center 4000. Reporting allows the landmark maintenance center 4000 to identify issues with landmarks 2000, e.g., a landmark 2000 may be permanently damaged and, thus, not recognizable by a computer vision system CVS.
[0055] In FIG. 7 another extended realization of the computer vision monitor CVM is depicted. CVM_007: the landmark maintenance center 4000 informs the CVM of the current status of landmarks 2000. For doing this the landmark maintenance center 4000 may take the vehicle position CUR_POS into account, e.g., to deliver information only for landmarks in the surrounding of the vehicle 1000.
[0056] In FIG. 8 an example operation of the landmark maintenance center 4000 is described: [0057] 4001: the landmark maintenance center 4000 collects the CVS misbehaviors as reported by one or many computer vision monitors CVM of one or many vehicles 1000 [0058] 4002: based on the collected information, the landmark maintenance center 4000 identifies problematic landmarks 2000, e.g., a landmark 2000 for which several vehicles 1000 report a CVS misbehavior can be identified to be damaged. [0059] 4003: the computer vision monitors CVM and/or the vehicle control systems VCS are informed that the identified landmark 2000 may be damaged. [0060] 4004: the landmark maintenance center 4000 may trigger a maintenance activity, such as sending a repair crew to the damaged landmark's site.
[0061] In FIG. 9 an example vehicle 1000 is depicted that realizes a computer vision monitor CVM to monitor the correct behavior of a computer vision system CVS. In the example in FIG. 9 the vehicle obtains knowledge of the current position CUR_POS of the vehicle 1000 by means of GPS (global positioning system). Furthermore, the vehicle 1000 obtains knowledge about landmarks 2000 in the surrounding of the vehicle (and in particular their position LM_POS) from a digital map DM that is locally stored in the vehicle 1000.
[0062] In FIG. 10 another example of a vehicle 1000 is depicted that realizes a computer vision monitor CVM to monitor the correct behavior of a computer vision system CVS. In the example in FIG. 10 the vehicle obtains knowledge of its current position CUR_POS and the existence of landmarks 2000 in the surrounding of the vehicle 1000 and their position LM_POS from the landmarks 2000 themselves, for example by means of a wireless connection WL. A landmark 2000, may thus instruct a vehicle 1000 of the landmarks 2000 existence by transmitting information over a wireless communication channel to the vehicle 1000, where the transmitted information can be interpreted by the vehicle 1000 as CUR_POS and LM_POS.
