Teleoperable Vehicle and System

Abstract

A system and a vehicle 100 of a system are described. The vehicle 100 is enabled to provide an autonomous driving mode, a manual driving mode and a vehicle-to-vehicle teleoperation driving mode. The vehicle 100 may send a support request if it detects an incident that prevents the use of the autonomous driving mode. FIG. 1a

Claims

1. A vehicle (100) comprising - a control device (10) which provides an autonomous driving mode, a manual driving mode and a vehicle-to-vehicle teleoperation driving mode; - wherein, in the autonomous driving mode, the driving of the vehicle (100) is controllable automatically, in the manual driving mode, the driving of the vehicle (100) is controllable by a driver, and, in the vehicle-to-vehicle teleoperation driving mode, the driving of the vehicle (100) is controllable by a driver who is located in another vehicle (150-158), - a support request unit (11) configured to detect an incident that prevents the use of the autonomous driving mode and transmit a support request through a communication device (103) to a driver of another vehicle (150-158) when detecting the incident.

2. The vehicle according to claim 1, wherein the vehicle (100) further includes sensors (101) to sense the surroundings, and the support request includes incident information about the detected incident which prevents the use of the autonomous driving mode.

3. The vehicle according to claim 1 or 2, wherein the vehicle further includes sensors (101) to sense the surroundings of the vehicle (100), and the control device (10) further configured to determine a type of the detected incident based on sensor data from the sensors (101).

4. The vehicle according to at least one of the preceding claims, wherein the support request includes an indication whether on-site assistance or vehicle-to-vehicle teleoperation driving is required depending on the kind of incident which the support request unit (11) detects and determines based on sensor data, environmental data, or position data of the ego vehicle, wherein, - in case a blocked path situation has been determined by the support request unit (11), the support request unit transmits the support request which does not include the indication, - in case a high risk driving area has been determined by the support request unit (11), the support request unit (11) transmits the support request which includes the indication that vehicle-to-vehicle teleoperation driving is requested, - in case a malfunction of a component of the vehicle (100) has been determined by the support request unit (11), the support request unit (11) transmits the support request which includes the indication that on-site assistance is requested, - in case a bad weather situation has been determined by the support request unit (11), the support request unit (11) transmits the support request which includes the indication that vehicle-to-vehicle teleoperation driving is preferable.

5. The vehicle according to at least one of the preceding claims, wherein the support request unit (11) issues the support request either via unicast or multicast, wherein the unicast or multicast sending mode is selected based on a predetermined sending policy and/or the kind of the detected incident that are stored in the storage space (17).

6. The vehicle according to at least one of the preceding claims, wherein the vehicle (100) further has a management table (20) stored in the storage space (17) and provided to contain, for the ego vehicle and other vehicles, vehicle identification information, operation mode information, driver on board information, and driver status information, and the support request unit (11) is configured to send the support request based on the information of the management table (20) and a location information of each other vehicle (150-158).

7. The vehicle according to at least one of the preceding claims, wherein the vehicle (100) comprises a priority table that includes priority information about possible support request recipients, and, if the support request is to be sent via unicast, the support request is sent to the recipient with the highest priority in the priority table.

8. The vehicle according to at least one of the preceding claims, wherein the management table (20) and/or the priority table is updated, respectively, when a preset change event has occurred.

9. The vehicle according to at least one of the preceding claims, wherein the vehicle (100) further includes - a communication device (103) which allows transmitting and receiving data wireless, - a cockpit (102) including a driving device (104) configured to output driving commands to the vehicle (100) or to another vehicle (150-158), and a display (105) configured to selectively display the surroundings and vehicle control data of the vehicle (100) or of another vehicle (150-158).

10. The vehicle according to at least one of the preceding claims, wherein the vehicle (100) further includes an interface configured to connect with a mobile device of the driver for data exchange and when the mobile device of the driver is connected to the interface, the support request of another vehicle (150-158) is displayed on a screen of the mobile device.

11. The vehicle according to at least one of the preceding claims, wherein the control device (10) of the vehicle (100) sends a rejection command in reply to a support request of another vehicle (150-158) if the management table (20) indicates that the operator status of the vehicle (100) is not available.

12. A system including a plurality of vehicles according to at least one of claims 1 to 11, wherein in some vehicles (100) a driver is present and in other vehicles (100) no driver is present, wherein the vehicles (100) are communicably connected with each other via a network.

13. The system including a teleoperation center for providing a teleoperation center teleoperation of one or more of the vehicles (100), and the management table (20) includes another entry indicating the status of the operator in the teleoperation center.

Description

BRIEF DESCRIPTION OF DRAWINGS

[0049] FIGS. 1A to 1C show schematic examples of a vehicle, a controller and a cockpit of the vehicle.

[0050] FIG. 2 shows two examples for a system, each system including a different kind of vehicles.

[0051] FIG. 3A shows a system_including trains, and in FIG. 3B shows a method for solving in incident.

[0052] FIG. 4 shows a management table.

[0053] FIG. 5 shows a method for selecting a recipient of a support request.

[0054] FIG. 6 shows an example for a multicast support request when a high risk area is detected.

[0055] FIG. 7 shows an example for a unicast support request when a high risk area is detected.

[0056] FIG. 8 shows a multicast support request for a blocked path incident.

[0057] FIG. 9 shows a multicast support request for a blocked path incident.

[0058] FIG. 10 shows a multicast support request for a blocked path incident.

[0059] FIG. 11 shows a procedure when a vehicle waits for on-site assistance.

[0060] FIG. 12 shows a procedure when a component malfunction is detected.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0061] In the following, preferred aspects and examples will be described in more detail with reference to the accompanying figures. Same or similar features in different drawings and examples are referred to by similar reference numerals. It is to be understood that the detailed description below relating to various preferred aspects and preferred examples are not to be meant as limiting the scope of the present disclosure.

General Description of the Present Disclosure and Preferred Aspects

[0062] The future of transportation will probably see the merge of two different paradigms: autonomous operation and teleoperation of vehicles. A (semi-)autonomous vehicle may operate autonomously 99.99 % of the time, but sometimes it may require human assistance to move forward or solve a problem. Such incidents or abnormal situations will be of very different nature and it is not possible to classify all possible types of incidents beforehand in a predetermined number of situations for which a predetermined action can be applied.

[0063] The present disclosure proposes an efficient way to handle these incidents flexibly, especially those which are not predetermined or which have not been foreseen within a list of predetermined incidents, by sending control requests to human assistors, also called “operator-teleoperators”, who can solve the issue through teleoperation and/or on-site (also called “presential”) assistance. The assistors are usually, within this disclosure, drivers of other vehicles. Hence, the herein-proposed solution leverages on-site assistance and remote assistance for autonomous and semi-autonomous vehicles with the objective to provide a more efficient, flexible and reliable service. As noted above, preferred possibilities for human assistance can be: Teleoperation (e.g.: controlling the vehicle in a crowded area); and on-site assistance (e.g.: remove a plastic bag that occludes a sensor of a vehicle, removing a blocking object in the driving path of a vehicle).

[0064] The system described, in particular, includes a plurality of participating/grouped vehicles, which shall preferably mean that the vehicles of the system are connected communicably with each other within the system. This may be achieved by a communication network, preferably a wireless network. Hence, here it is disclosed, in particular, a system in which the cockpit of the participating vehicles is configured for teleoperation with interchangeable modes for classic human operation and teleoperation. The teleoperation mode is (mainly) a V2V teleoperation mode, which means that, e.g., a vehicle “A” is teleoperated from another vehicle “B”. The vehicles within the system all have the same or similar configuration with regard to the operation modes and the respective software and hardware needed for realizing the V2V teleoperation. The vehicles are preferably vehicles which are configured to drive autonomously whenever possible, that means, when no incident has happened/has been detected which prevents the autonomous driving.

[0065] It is a further technical advantage, that the herein proposed solution allows to exploit autonomous and semi-autonomous operation of the vehicles more efficiently since human operators/teleoperators are closer to the service vehicles than in a teleoperation center paradigm where vehicles are operated/controlled from a remotely located teleoperation center (i.e., C2V teleoperation) and which does not include the possibility for V2V teleoperation. Teleoperation shall mean, as usually known, that a vehicle is controlled/driven from a person which is not the driver of said vehicle, but a remotely located person. V2V teleoperation shall mean that the person being in another vehicle takes over the control of the V2V teleoperated vehicle; the latter will often be named “ego/own vehicle” in the following, while the vehicle in which the remote operator/driver is located may be named “other vehicle” in the following. C2V teleoperation shall mean that a driver, which is rather an operator in this case, will be in present in a remote stationary center which is equipped to operate a vehicle from said center.

[0066] With regard to the hardware configuration of the vehicles of the herein-described system, preferably, the teleoperation of an autonomous or semi-autonomous vehicle from another similar vehicle is improved by providing the same or at least similar cockpit equipment, at least with regard to the V2V teleoperation control equipment, to each vehicle of the herein proposed system. This supports a driver who takes over V2V teleoperation of a vehicle to perform the teleoperation safely and without the need for a time-consuming training. Each vehicle in the system may be preferably enabled to be used for teleoperation either as master or slave vehicle. With regard to the data flow during V2V teleoperation, it is preferred that information from different sensors and processing algorithms is sent from the V2V teleoperated vehicle to the remote displays of the teleoperator’s vehicle, and likewise control actions from the teleoperator’s cockpit are linked to the actuators of the teleoperated vehicle. In summary, every vehicle can be at least: Operated manually by a driver (classic operation), operate (semi-)autonomously, V2V teleoperated from another vehicle, and used to V2V teleoperate another vehicle.

[0067] A specific implementation example of the present disclosure may relate, as a non-limiting example, to a railway system including a plurality of railway vehicles. For example, the railway system may be a rail transportation line with high frequency of vehicle movements, the line may be circular or bidirectional without further particularities, such as it may be the case for urban metro lines or the like. This non-limiting example will be used to describe a possible incident handling method further below.

Configuration of Vehicle(s) of the System According to the Present Disclosure

[0068] FIG. 1a shows a schematic vehicle 100 of the present disclosure, FIG. 1b shows a schematic representation of a possible configuration of a controller 1 of said vehicle 100 including different control devices and units which will be explained later, and FIG. 1c shows a schematic representation of a (teleoperation) cockpit 102 of the vehicle 100.

[0069] The controller 1 as shown by FIGS. 1a, 1b can provide different technical functions via devices, modules or units which may be integrated into a single controller hardware or which may be separate entities. Also the distribution of devices and the like as shown in FIG. 1b is not mandatory and the devices/units and the like may also be arranged differently, such as some of the devices/units shown may be merged or further distributed over other entities. In other words, the architecture as shown is schematic and the units and devices may be arranged differently without departing from their intended technical function. The devices/units/modules and the like which are shown may be configured by hardware and/or software, preferably they are configured by one or more processors and internal and external storage space(s) for storing the computer programs and for saving databases, and the like. Further, the controller 1 may be a single unit or may be split in a plurality of units which may also be locally distributed over the vehicle 100. In FIG. 1a, the controller is shown as a single unit as one possible example.

[0070] With regard to the configuration of the controller 1 and connected components of the vehicle 100, a possible configuration example is shown by FIG. 1b and different technical functionalities can be organized within different hardware devices/units/modules, such as:

[0071] 1) AD ECU 12 (Autonomous Driving Electronic Control Unit) refers to the computer or set of computers that perform the bulk of the calculations required for perception, localization and trajectory planning required for autonomous operation of the vehicle. The AD ECU 12 can operate in accordance with known principles of autonomous control of vehicles.

[0072] 2) The controller 1 may be connected communicably to one or more sensors 101 as shown in FIG. 1a. The list of sensors 101 is defined, among others, by the autonomous operation requirements and therefore will vary according to the type of vehicles and deployment scenario, it may include passive optical sensors 101, such as camera, stereo camera, event-based camera, active optical sensors 101, such as like LiDAR, RADAR, SONAR, GNSS / INS and IMU, and the like.

[0073] 3) The communication infrastructure (not shown) of the system may include private wireless area networks (WAN) when the application allows it. e.g. in case of public transportation system in a city, vehicles 100, 150-158 operating in a mine, or the like, or the use of public networks, preferably having a high-speed bandwidth, like 5G, through telecom providers. The controller 1 may include or may be connected to a communication device 103 which allows to connect the vehicle 100 communicably to the above communication infrastructure so that the vehicle 100 can exchange data with other vehicles of the system and, if available, a teleoperation center or a remote server (not shown).

[0074] 4) The control device or teleoperation ECU 10 refers to the computers and routers in charge of communication with other vehicles, its responsibilities may include sharing information on the status of the different vehicles and the location of human assistors, together with the transmission of information for teleoperation. The teleoperation ECU 10 functionality may also be integrated into other units so that a dedicated teleoperation ECU would not be necessary. Further said teleoperation ECU, also called control device 10, may integrate the technical functionality for selecting and providing different control modes of the vehicles 100, 150-158 and preferably, a support request unit 11, may be a sub-unit of said control device 10.

[0075] 5) Controls or driving devices 104, as shown in FIG. 1c, refer to all the devices which are present on a vehicle driven by a human for controlling the vehicle manually. Here, FIG. 1c shows control sticks 104a and buttons 104b.

[0076] 6) A cockpit 102 for teleoperation and/or for the driver of the ego vehicle is also provided which includes display(s) 105 on which sensor data are displayed and which are required to allow a human to teleoperate a remote vehicle 150-158 from the cockpit of his current vehicle 100 (where he is on board, which may be operating autonomously or be at stop), and to drive the current vehicle 100 in standard operation (human driving) with minimal changes among both modes. A possible implementation could use the same displays 105 for teleoperation and standard/manual operation, without the traditional windshield window to avoid interference of sensations to the human operator. Likewise, the cockpit 102 may be as much isolated as possible from vehicle vibrations to avoid confusing sensations while teleoperating a remote vehicle 150-158 from an autonomous vehicle in movement. A possible schematic of a cockpit 102 is shown by FIG. 1c which shows, exemplarily, control buttons 104b, driving sticks 104a, and a display 105.

[0077] Further, the controller 1 of the vehicle 100, as shown in FIG. 1b, may include a storage device 17 for storing data locally in the vehicle 100 and other control units 13-16 which may have different technical tasks, such as communication and sensor data control, engine control, brake control, trained artificial intelligence (Al) or machine learning unit (ML), and the like. Further, one of the units 13-16 may also be an interface for establishing a communication connection between a mobile computing device of the driver with the cockpit 102 or the vehicle 100.

[0078] Summarizing, the vehicle 100 may include a control setup which is based on known autonomously-driving vehicles of the respective type, such as trains or cars, and in addition the controller 1 or the control device 10 may include the additional control modes for V2V teleoperation and for performing the detection of an incident and for issuing support requests as well as the hardware to enable V2V teleoperation of another vehicle 150-158, including driving devices 104, communication device(s) 103 and a cockpit 102 for V2V teleoperation, which may be shared with the normal cockpit for the manual driving mode in a preferred example. A support request unit 11 may be integrated into the controller 1 as a sub-unit or may be part of one of the other units, such as the teleoperation ECU 10. The support request unit 11 may preferably be configured to detect incidents, to determine a type of the incident (if possible automatically) and to issue a support request as well as to handle incoming support requests. The support request unit 11 or any other part of the controller 1 may include a trained Al or ML unit which were trained to detect incidents and/or to determine the type of an incident. The training data used for such purpose may be gained from test drives with the real vehicles or in a simulator, wherein the test driver highlights incidents during the test drives so that the AI/ML can be trained. Otherwise, general computer program algorithms and procedures can also be used, i.e. without the use of an Al or ML, for the detection of incidents and the determination of their type in accordance with known principles. For example, a blocked driving path could be detected if the respective sensors 101 of the vehicle 100 detect that an object is placed in the future driving path or if sensors measurements are out of a normal range, it may be concluded that they have a malfunction, and so on. The respective programs can be stored in a storage space 17 of the vehicle 100, such as shown in FIG. 1b, or they can be stored remotely, e.g. in a remote computer (“cloud computer”).

[0079] FIG. 2 provides two examples of two different systems which fall under the present subject-matter, one system including public transportation trains, the other system shows a system including trucks. However, the general principle is the same irrespective of the type of the vehicle 100 and it is indicated schematically for both example systems of FIG. 2 that there are vehicles 151, 152 which do not include a driver and that there is a vehicle 100 in which a driver is present. Of course, the system can include less than three vehicles and, preferably, much more vehicles than the three shown in the examples of FIG. 2. Further, the preferred ratio of unmanned vehicles 100, 150-158 to vehicles 100, 150-158 in which a driver is present can be flexible and can be determined on a day to day basis. For example, if it is expected that a lot of support requests may be issued on a specific day, e.g. due to expected bad weather or the like, a public transportation provider could schedule more drivers on said day compared to other days during which less/normal amount of support requests are expected.

[0080] FIG. 3 shows a possible implementation of a method which may be performed by a computer program product run by one of the units/devices of the vehicle 100, preferably within the support request unit 11 or the control device 10. The application scenario concerns a circular public transportation line with 10 vehicles 100, 150-158 and 5 driver-teleoperators (or driver or operator) as shown in FIG. 3a. Here, vehicles 150, 152, 154, 155, 157 include a driver, the vehicle 100 will face an incident being the “ego” vehicle, and vehicles 151, 153, 156 and 158 do not have a driver. The number of vehicles, drivers, and the type of system is an example and may be different. The vehicles 100, 150-158 run autonomously which is the default mode (no incident). Upon the detection of an incident preventing autonomous operation, the vehicle 100 facing an incident sends a support request for assistance. Operator-teleoperators may receive the support request (briefly: request) from their current vehicle 100 or from a remote one 150-158. The available operator-teleoperator may accept the request, if the incident cannot be solved by teleoperation the operator-teleoperator may raise a request for on-site assistance that is sent to the most relevant operator-teleoperator to the respective vehicle. e.g. the closest one. A more complex policy for sending and receiving assistance requests may be proposed for the particular implementation depending on: a) the number of vehicles versus the number of assistors, b) the frequency and nature of requests, c) the proximity of assistors to remote vehicles.

Operating Method for the Generation and Distribution of Support Requests

[0081] FIG. 3b shows a preferred method. At S1, all vehicles 100, 150-158 operate autonomously as the default operation mode. Upon an incident that prevents safe autonomous operation of the ego vehicle 100, a support request is generated and transmitted (S2) to human assistors which are drivers of other vehicles 150-158. The support request preferably contains basic information on the incident, such as the vehicle identification (ID), the vehicle location, the type of incident, a priority score, and/or the like. This information may be sent by the support request unit 11 of the vehicle 100 which requires assistance to another vehicle 150-158 and/or to a mobile device (not shown) of a driver of a vehicle 150-158, like a cell phone through a telecom operator, and/or through a private wireless area network (WAN) deployed specifically for the transportation system. Requests can be sent to a single recipient or to several ones via unicast sending mode according to proximity and pertinence, or to all available assistors via multicast sending mode. Here, an assistor is preferably a driver (operator or operator-teleoperator) of another vehicle 150-158. Different request policies may be implemented for sending the requests according to the present disclosure. For instance, a single target recipient for the request may be targeted, e.g. the most relevant assistor in the example of FIG. 3a would be the closest one 155 to the vehicle 100 in the opposite direction of movement, in other words: the assistor who is already on board if there is one, or the assistor on board the following vehicle otherwise, when the assistor does not accept the request after a given time, the request can be re-assigned to the following most relevant assistor and so on. The arrangement of assistors’ relevance for a given vehicle may also be calculated by an algorithm upon control request generation considering different factors, like proximity of assistors to the ego vehicle sending the request. For that, the location of the assistors can be determined using the WiFi connection of their mobile devices and a WiFi router on each vehicle. Control requests may also be assigned a priority score to distribute them efficiently to the available assistors.

[0082] If the recipient of the support request finds out that teleoperation can solve the incident, the V2V teleoperation is requested by the recipient of the support request and, if accepted by the ego vehicle 100, activated (S3, S4). If, however, in step S3 the answer to the question whether teleoperation can solve the problem, it is proceeded with S5 which is another question as to whether the incident happened at the ego vehicle 100, if “yes” and if a driver is on board of the ego vehicle 100, the driver may provide on-site assistance himself/herself. If, however, the driver of another vehicle 150-158 is required to provide on-site assistance, e.g. because the ego vehicle 100 does not have a driver, the other vehicle 155 will drive to the ego vehicle 100 (S7) and then provide on-site assistance (S6). Further, the dotted arrow in FIG. 3b indicates that after the on-site assistance it is also possible to continue (temporarily) with teleoperation according to S4. Otherwise, the procedure continues with the reinitiated autonomous driving S8.

[0083] One option for realizing an efficient support request distribution according to the present disclosure may rely on providing a management table 20 to the storage space 17 of each vehicle 100, 150-158 or to a remote server (not shown), which is shown in FIG. 4. Said management table 20 may provide an overview accessible for each vehicle 100, 150-158 about the ID 21 of each vehicle 100, 150-158 in the system, its operation mode 22, data about whether an operator is present 23, and the status of the operator 24. Further, if C2V teleoperation is additionally provided, the management table 20 may also contain a section 25 with regard to the operators in the center and their status/availability.

[0084] Using the information from the management table 20 as shown in an example of FIG. 4, the above described method of determining the preferred/primary target recipient can be processed including mainly the procedural steps (P) as shown by FIG. 5, i.e. receiving location data of each vehicle and calculating each distance between the ego vehicle 100 and each other vehicle 150-158 (step P1), check available operators and rank them by relevance (step P2), e.g. proximity/distance, and determine the most relevant (e.g. closest) available operator and send the request to the corresponding vehicle 150-158 having the determined ID 21 (step P3).

[0085] Further, a priority table which is not shown may also be provided in which priorities for the target recipients are included so that the information of the priority table may also be used in combination with the management table information or alone. For example, if a priority table is provided, the steps of FIG. 5 may not include the determination of the distance/relevance but the most relevant target recipient may be determined based on the priority table information.

[0086] Further, the sending mode may be determined case by case, e.g., by the teleoperation ECU 11 or any other component of the vehicle 100, or it may be predetermined by the system supervisor for a certain time, e.g. one day or the like, or permanently, e.g. only unicast sending mode is used. The adaption of the sending mode may be determined based on sending policies which may include the examples as follows: [0087] a) On-site assistance is required as often as teleoperation => Unicast, send request to the closest assistor available (or assistor on a following vehicle); [0088] b) High number of assistance requests (e.g. > 100 per day & vehicle), even need of teleoperation & On-site assistance => Multiple unicast targeting the most relevant assistors; [0089] c) Most requests can be solved by teleoperation => Multicast (minimize number of human assistors); [0090] d) Most requests can be solved by teleoperation and big system size (e.g. > 100 vehicles) and high number of assistance requests (e.g. > 100 per day & vehicle) => Multicast (minimize number of human assistors), and combine with teleoperation center [0091] e) Bad weather => Assign more human assistors on duty to be able to handle more assistance requests

Operation of the Acceptance of Support Requests

[0092] Upon a support request reception, the recipient/assistor may accept it or reject it using his mobile device. Alternatively, the support request is indicated on the display of the cockpit and can be accepted or rejected via the cockpit or the display thereof. Upon request acceptance the assistor will go to the cockpit 102, if not already present, of the vehicle (not the ego vehicle which sent the support request) and run the procedure to control the requesting vehicle (note that the requesting vehicle may also be the same one where the assistor is on board). If the requesting vehicle 100 is not the same one where the assistor is on board, the assistor may run the V2V tele-operation procedure on the cockpit 102 of his vehicle to connect the displays 105 and cockpit 102 to the remote vehicle 100 that raised the control request. He/she may then solve the issue by V2V teleoperation, or drive towards the requesting vehicle 100 to provide on-site assistance (e.g., as shown in the example provided in FIG. 3). If a teleoperation center exists (not shown), the assistor may control a remote vehicle either from another vehicle 150-158 or from the teleoperation center.

Types of Control Requests

[0093] Incidents preventing autonomous operation may have different nature that can be used to classify them on different types. However, the system is flexible and non-classified incidents may also be handled because the recipient of a support request receives information enabling to select the best possible solution for overcoming the incident/problem. However, predetermined incidents may be saved in the storage space 17 of the vehicles 100 or of a remote server of the system for increasing response speed in case of predefined incidents happening.

[0094] 1) High risk area incident: The support request may be generated according to an implemented operation policy which may differ between busy urban areas, like city center, commercial areas, residential areas, rural areas and countryside with little human activity. A high risk area may preferably relate to an elevated risk of difficult driving operation, e.g. due to high traffic or many people possibly blocking the drive path, etc. As an example, a support request can be generated in a busy commercial area upon a certain risk score is met, such risk score may be a function of the number of dynamic objects traversing the path at a given distance range in front of the moving autonomous vehicle 100, the number of dynamic objects in the proximity of the planned path, the velocity of the autonomous vehicle 100, and the relative velocities of the dynamic objects in the proximity of the autonomous vehicle 100, among others. Such policies may be implemented through a risk score algorithm or risk score maps stored in the storage space, so that a support request is generated when a risk score threshold is reached. The procedure to compute the risk score will depend on the area where the vehicle 100 is (different areas may be mapped and classified, where the autonomous vehicles may be localized with respect to the map either with global positioning like GPS, GNSS, or with local positioning, e.g. map-based localization). The procedure based on which the risk score is computed may also consider the type of vehicles, such as tram, train, shuttle bus, etc., and the types of risks of the areas of deployment.

[0095] 2) Blocked path: an autonomous vehicle 100, 150-158 may stop due to blockage of its path. A policy on how to react to this situation can be implemented considering the type of vehicle, such as tram, train, shuttle bus, etc., and the current location on the map. For instance, a waiting time can be programmed before sending a request to let the blockage resolve (e.g. a car blocking the path). The generation of a support request may also be dependent on whether the blocked autonomous vehicle 100, 150-158 is also blocking other vehicles, e.g., at an intersection.

[0096] 3) Component malfunctioning: may be generated when a component, such as a sensor 101, is not working properly. For instance, a vital sensor 101 is occluded by an unknown object (e.g. a plastic bag carried by the wind), and such an incident highlights in particular the benefits of the present disclosure enabling efficient resolution for an incident. Other examples of component malfunctioning which may lead to a respective support request may be generated when a component is not responding, when two redundant devices or modules do not provide corresponding outputs, or when the communication latency required for teleoperation is above a certain threshold. Component malfunctioning control requests will often require on-site assistance, nonetheless, the assistor can start addressing the issue remotely while his current vehicle 150-158 drives autonomously towards the requesting vehicle 100, or perform minimal V2V teleoperation tasks to move the requesting vehicle 100 away of a blocked intersection.

[0097] Preferably, the support request issued from a vehicle 100 includes an indicator which indicates the determined type of the incident, if it has been possible to determine the type of the incident by the control device 10 or a sub-unit thereof.

[0098] Further, if the vehicle 100, 150-158 does not provide automated determination of the type of an incident or in case the type could not be determined, the information send with the support request enables the recipient to determine the incident and to select the apt actions for resolving the issue. Hence, there is no situation with which the vehicle 100, 150-158 and the system described herein cannot deal which ensures quick reaction to incidents and reliable service at any time.

[0099] As noted above, in case of predetermined incident types, there can also be predetermined action policies, especially in view of the support request(s). Such predefined policies enable to increase the reaction/response speed because, e.g., the detection and identification of an incident may be processed more quickly by the control device 10 compared to an unforeseen incident. FIGS. 6 to 12 show examples of preferred action policies, i.e. methods, which address some possible incidents and the preferred exemplary actions to solve them.

[0100] FIG. 6 shows an example method for addressing the detection of a high risk area when a multicast sending mode is preset or selected for transmitting the support request. Schematically, FIG. 6 shows a system including seven vehicles 100, 150-158, in FIG. 6: #1 to #7, and vehicle #6 detects to be in or in front of a high risk area and therefore issues a respective support request via multicast. This is depicted by the broken arrows which point to each vehicle in the FIG. 6. Then, in this example, two operators/drivers, which are possible assistors, reply to the support request by accepting it, namely the drivers of vehicles #1 and #5. This is indicated by the arrows pointing back to the vehicle #6. Further, the operator/driver of vehicle #7 is not available/busy, but he/she does not need to respond because the system expects only a single acceptance. After having received an acceptance, or more, the ego vehicle 100 which has requested assistance enables the recipient of the request who has accepted it to perform assistance, e.g. in this case by way of V2V teleoperation. In the present example of FIG. 6, since two drivers accepted the support request, one is selected and the other driver is informed via a notification that the acceptance has been rejected. In the FIG. 6 this is the case for the driver of vehicle #1. The selection may be automated based on a predetermined policy, e.g., the faster acceptance reply is selected or the acceptance from the nearest other vehicle 150-158. After the V2V teleoperation has been enabled for the driver/operator of vehicle #5, the management table, and if necessary and provided the priority table, is updated on the remote server or in each vehicle of the system to indicate that the driver of vehicle #5 is busy now and to update the status of the vehicle #6. Subsequently, V2V teleoperation starts immediately and the driver/operator of vehicle #5 can navigate the vehicle #6 remotely through the high risk area, while the vehicle #5 continues autonomous driving.

[0101] Another example is shown in FIG. 7 relating to a high risk area incident as well, however, in this example a unicast sending mode is used. In this example, different to the procedure in FIG. 6, the controller 1 or a sub-unit of the vehicle #6 looks up a priority table not shown which provides the ranking of target recipients, such as a ranking of vehicle IDs. In this case of FIG. 7, the one with the highest priority, vehicle #7, has indicated in the management table 20 that the driver is busy, therefore the vehicle #5 is selected having the second highest priority and being available. Therefore, vehicle #5 is determined as the unicast target recipient of the support request and since, in the next step, the driver/operator accepts it, the V2V teleoperation is enabled and started and the tables are updated, in particular the management table 20.

[0102] In FIG. 8, an example is shown in case a blocked path is detected and a respective support request is transmitted via multicast to the vehicles 150-158. As in FIG. 6, two operators/drivers accept the request and one operator does not respond because being busy. Other vehicles 150-158 are not specifically shown. Then, optionally upon further request of the possible assistor, further information for determining the exact problem may be submitted to the assistor so that he/she can determine the best possible response. In the present example, the assistor decides that the problem can be solved by V2V teleoperation so a respective teleoperation enabling request can be submitted to the vehicle #6 which enables the V2V teleoperation accordingly. Afterwards, the same further steps as described before can be performed.

[0103] FIG. 9 shows a blocked path incident which will lead to on-site assistance in a multicast sending mode. Specifically, the vehicle #6 detects a blocked path and issues the support request with the indication. Then, as in FIG. 8, the flow proceeds likewise, however, the driver of the vehicle #5 decides, after checking the sensor information included in the support request data, that on-site assistance is preferable and indicates this to the vehicle #6. Respectively, the tables, especially the management table 20, is updated to include the new information, such as the changed status of the driver of vehicle #5. Vehicle #6 waits for on-site assistance afterwards.

[0104] FIG. 10 shows another procedure for a blocked path incident handling which relies on multicast issuance of the support request and unicast assignment of on-site assistance. Specifically, the support request is send by multicast to all vehicles #1 to #5 and #7. Then, operators of #1 and #5 accept the support request and the driver of #7 is busy. Afterwards the tables, i.e. the management tables 20 of all vehicles, are updated and the sensor information is sent to the operator of vehicle #1 while #5 is informed that the acceptance is denied. The driver of #1 checks the information and recommends on-site assistance which is notified to the vehicle #6. The management tables 20 are updated again and then it is determined based on a priority table (not shown) in the vehicles which operator should take over the on-site assistance. Since the closest one may be the fastest, the closest will be picked. The priority table may show the ranking of the closest operators, e.g., in a railway situation where the distances are predetermined. For example, in the present case the priority table may include #7 as the first, being however busy. Therefore, the next one is selected, which is, in this example, #5. Then, a request for on-site assistance is sent to the vehicle #5 and the operator accepts it. The tables are then updated again and vehicle #6 waits for the arrival of the assistance.

[0105] Further, FIG. 11 shows a possible procedure which is operated during the waiting for on-site assistance. Specifically, vehicle #6 waits for the arrival of vehicle #1 and may send further sensor information to the vehicle #1 in the meantime. The information sent may help the assistor to familiarize with the problem while his current vehicle drives autonomously towards the blocked one. Notice that the blockage might resolve also during the assistor trip (either by teleoperation or by external reasons: improvement of traffic conditions). Then, vehicle #6 arrives at vehicle #1 and the driver fixes a blockage or performs other acts for traversing the problem so that the autonomous driving can be resumed afterwards. During the above steps, further, the tables of the vehicle may be updated at different situations, e.g., as shown in FIG. 11.

[0106] FIG. 12 shows a flow of steps for a unicast sending of a component malfunctioning. If vehicle #6 has a malfunction which is detected it sends a support request by unicast according to a priority table (not shown) saved in the storage space 17 or in a remote server. The operator of vehicle #5, which has been the recipient of the support request, accepts the request and starts moving to the site of vehicle #5. As shown, at different positions of time, the management tables 20 are updated for all vehicles. Further, due to the long time required to arrive at destination, the information about the operator on board is updated before the operator arrives. Each vehicle keeps updated information about all vehicles. Also the tables can be updated when the assistor has arrived. Further, the vehicle #6 keeps sending sensor information to the vehicle #5 while moving to the site so that the assistor can familiarize with the problem while the vehicle drives autonomously towards the blocked one. Notice that the blockage might resolve also during the assistor trip. The assistor, after arrival, fixes the malfunction and then the autonomous operation can be resumed.

General Procedures for the Return to Autonomous Operation

[0107] After the assistor has performed the necessary actions to operate the requesting vehicle 100 upon an incident, the assistor returns the control back to the default autonomous operation by performing the required action on the cockpit 102 for that purpose. For that, after the vehicle 100 sends a support request, it may display in the cockpit 102 of the assistor a visible signal when the autonomous operation may take safe control again. This signal helps the assistor to make the decision on returning control to the autonomous vehicle 100. Depending on the application, a requesting vehicle 100 waiting for an assistor to take control may cancel the control request when it is not needed anymore (e.g. blocked path resolved) and continue the autonomous operation.

[0108] It is noted that for example the herein described system may be formed by different vehicles and networks, however, a schematic arrangement of vehicles 100, 150-158 grouped in the system and communicably connected is shown by FIG. 3b.

[0109] The proposed disclosure has, among others, technical benefits, such as vehicles are operated more efficiently, reducing frequency and duration of disruptions to improve service quality. The human operator can use its current vehicle 150-158 to move towards a vehicle 100 when on-site assistance is required.

[0110] Further, a number of vehicles 100, 150-158 (V) can be operated by a number of operator/teleoperators (O) where O <= V, for better efficiency and lower operation cost. Traditional driving remains possible if required by law or by eventuality (e.g. bad weather). More amenable workload that combines teleoperation with physical tasks (depending on the automation level, more diversity of tasks can be assigned to the operators to reduce boredom/drowsiness). Training of operator-teleoperators remains similar to traditional driving. Faster and more relevant on-site assistance because operator-teleoperators are constantly on site and know the context). No need of a teleoperation centre which reduces the cost of the system (related to facility, hardware, maintenance), improves security (distributed system instead of centralized), however, such center can be integrated within the proposed solution without reducing the scope of this invention. For instance, a teleoperation or control center may be integrated as an additional safety and management layer for big transportation systems with hundreds of vehicles (e.g. public transportation of big cities like Paris or Tokyo), where assistors may be on a waiting room (control centre), and/or may also teleoperate vehicles from another vehicle and from the teleoperation centre.

[0111] As will be appreciated by one of skill in the art, the present disclosure, as described hereinabove and the accompanying figures, may be embodied as a method (e.g., a computer-implemented process or any other process), apparatus (including a device, machine, system, computer program product, and/or any other apparatus), or a combination of the foregoing. Aspects/Examples of the present disclosure may be a software entirely (including firmware, resident software, micro-code, etc.), or a combination of software and hardware aspects that may be referred to as a “system”. Furthermore, the present disclosure may take the form of a computer program product on a computer-readable medium having computer-executable program code embodied in the medium.

[0112] It should be noted that arrows may be used in drawings to represent communication, transfer, or other activity involving two or more entities. If present, double-ended arrows generally indicate that activity may occur in both directions (e.g., a command/request in one direction with a corresponding reply back in the other direction, or peer-to-peer communications initiated by either entity), although in some situations, activity may not necessarily occur in both directions.

[0113] Single-ended arrows generally indicate activity exclusively or predominantly in one direction, although it should be noted that, in certain situations, such directional activity actually may involve activities in both directions (e.g., a message from a sender to a receiver and an acknowledgement back from the receiver to the sender, or establishment of a connection prior to a transfer and termination of the connection following the transfer). Thus, the type of arrow used in a particular drawing to represent a particular activity is exemplary and should not be seen as limiting.

[0114] The present disclosure may be described with reference to flowchart illustrations and/or block diagrams of methods and apparatuses, and with reference to a number of sample views of a graphical user interface generated by the methods and/or apparatuses. It will be understood that each block of the flowchart illustrations and/or block diagrams, and/or combinations of blocks in the flowchart illustrations and/or block diagrams, as well as the graphical user interface, can be implemented by computer-executable program code.

[0115] The computer-executable program code may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a particular machine, such that the program code, which executes via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts/outputs specified in the flowchart, figures, and/or written description.

[0116] The computer-executable program code may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the program code stored in the computer readable memory produce an article of manufacture including instruction means which implement the function/act/output specified in the flowchart, block diagram block(s), figures, and/or written description.

[0117] The computer-executable program code may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the program code which executes on the computer or other programmable apparatus provides steps for implementing the functions/acts/outputs specified in the flowchart, figures, and/or written description. Alternatively, computer program implemented steps or acts may be combined with operator or human implemented steps or acts in order to carry out an embodiment of the disclosure.

[0118] It should be noted that terms such as “server” and “processor” may be used herein to describe devices that may be used in certain aspects of the present disclosure and should not be construed to limit the present disclosure to any particular device type unless the context otherwise requires. Thus, a device may include, without limitation, a bridge, router, bridge-router (brouter), switch, node, server, computer, appliance, or other type of device. Such devices typically include one or more network interfaces for communicating over a communication network and a processor (e.g., a microprocessor with memory and other peripherals and/or application-specific hardware) configured accordingly to perform device functions.

[0119] Communication networks generally may include public and/or private networks; may include local-area, wide-area, metropolitan-area, storage, and/or other types of networks; and may employ communication technologies including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies.

[0120] It should also be noted that devices may use communication protocols and messages (e.g., messages created, transmitted, received, stored, and/or processed by the device), and such messages may be conveyed by a communication network or medium.

[0121] Unless the context otherwise requires, the present disclosure should not be construed as being limited to any particular communication message type, communication message format, or communication protocol. Thus, a communication message generally may include, without limitation, a frame, packet, datagram, user datagram, cell, or other type of communication message.

[0122] Unless the context requires otherwise, references to specific communication protocols are exemplary, and it should be understood that alternatives may, as appropriate, employ variations of such communication protocols (e.g., modifications or extensions of the protocol that may be made from time-to-time) or other protocols either known or developed in the future.

[0123] It should also be noted that logic flows may be described herein to demonstrate various aspects of the disclosure, and should not be construed to limit the present disclosure to any particular logic flow or logic implementation. The described logic may be partitioned into different logic blocks (e.g., programs, modules, functions, or subroutines) without changing the overall results or otherwise departing from the true scope of the disclosure.

[0124] Often, logic elements may be added, modified, omitted, performed in a different order, or implemented using different logic constructs (e.g., logic gates, looping primitives, conditional logic, and other logic constructs) without changing the overall results or otherwise departing from the scope of the disclosure.

[0125] The present disclosure may be embodied in many different forms, including, but in no way limited to, a graphical processing unit as well as computer program logic for use with a processor (e.g., a microprocessor, microcontroller, digital signal processor, or general purpose computer), programmable logic for use with a programmable logic device (e.g., a Field Programmable Gate Array (FPGA) or other PLD), discrete components, integrated circuitry (e.g., an Application Specific Integrated Circuit (ASIC)), or any other means including any combination thereof. Computer program logic implementing some or all of the described functionality is typically implemented as a set of computer program instructions that is converted into a computer executable form, stored as such in a computer readable medium, and executed by a microprocessor under the control of an operating system. Hardware-based logic implementing some or all of the described functionality may be implemented using one or more appropriately configured FPGAs.

[0126] Computer program logic implementing all or part of the functionality previously described herein may be embodied in various forms, including, but in no way limited to, a source code form, a computer executable form, and various intermediate forms (e.g., forms generated by an assembler, compiler, linker, or locator).

[0127] Source code may include a series of computer program instructions implemented in any of various programming languages (e.g., an object code, an assembly language, or a high-level language such as Fortran, python, C, C++, JAVA, JavaScript or HTML) for use with various operating systems or operating environments. The source code may define and use various data structures and communication messages. The source code may be in a computer executable form (e.g., via an interpreter), or the source code maybe converted (e.g., via a translator, assembler, or compiler) into a computer executable form.

[0128] Computer-executable program code for carrying out operations of embodiments of the present disclosure may be written in an object oriented, scripted or unscripted programming language such as Java, Perl, Smalltalk, C++, or the like. However, the computer program code for carrying out operations of aspects of the present disclosure may also be written in conventional procedural programming languages, such as the “C” programming language or similar programming languages.

[0129] Computer program logic implementing all or part of the functionality previously described herein may be executed at different times on a single processor (e.g., concurrently) or may be executed at the same or different times on multiple processors and may run under a single operating system process/thread or under different operating system processes/threads.

[0130] Thus, the term “computer process” refers generally to the execution of a set of computer program instructions regardless of whether different computer processes are executed on the same or different processors and regardless of whether different computer processes run under the same operating system process/thread or different operating system processes/threads.

[0131] The computer program may be fixed in any form (e.g., source code form, computer executable form, or an intermediate form) either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), a PC card (e.g., PCMCIA card), or other memory device.

[0132] The computer program may be fixed in any form in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies.

[0133] The computer program may be distributed in any form as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web).

[0134] Hardware logic (including programmable logic for use with a programmable logic device) implementing all or part of the functionality previously described herein may be designed using traditional manual methods, or may be designed, captured, simulated, or documented electronically using various tools, such as Computer Aided Design (CAD), a hardware description language (e.g., VHDL or AHDL), or a PLD programming language (e.g., PALASM, ABEL, or CUPL).

[0135] Any suitable computer readable medium may be utilized. The computer readable medium may be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or medium.

[0136] More specific examples of the computer readable medium include, but are not limited to, an electrical connection having one or more wires or other tangible storage medium such as a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a compact disc read-only memory (CD-ROM), or other optical or magnetic storage device.

[0137] Programmable logic may be fixed either permanently or transitorily in a tangible storage medium, such as a semiconductor memory device (e.g., a RAM, ROM, PROM, EEPROM, or Flash-Programmable RAM), a magnetic memory device (e.g., a diskette or fixed disk), an optical memory device (e.g., a CD-ROM), or other memory device.

[0138] The programmable logic may be fixed in a signal that is transmittable to a computer using any of various communication technologies, including, but in no way limited to, analog technologies, digital technologies, optical technologies, wireless technologies (e.g., Bluetooth), networking technologies, and internetworking technologies.

[0139] The programmable logic may be distributed as a removable storage medium with accompanying printed or electronic documentation (e.g., shrink wrapped software), preloaded with a computer system (e.g., on system ROM or fixed disk), or distributed from a server or electronic bulletin board over the communication system (e.g., the Internet or World Wide Web). Of course, some embodiments of the disclosure may be implemented as a combination of both software (e.g., a computer program product) and hardware. Still other aspects of the present disclosure are implemented as entirely hardware, or entirely software.

[0140] While certain exemplary aspects have been described and shown in the accompanying drawings, it is to be understood that such embodiments are merely illustrative of and are not restrictive on the broad disclosure, and that the aspects of the present disclosure are not limited to the specific constructions and arrangements shown and described, since various other changes, combinations, omissions, modifications and substitutions, in addition to those set forth in the above paragraphs, are possible.

[0141] Those skilled in the art will appreciate that various adaptations, modifications, and/or combination of the just described aspects and examples can be configured. Therefore, it is to be understood that, within the scope of the appended claims, the disclosure may be practiced other than as specifically described herein. For example, unless expressly stated otherwise, the steps of processes described herein may be performed in orders different from those described herein and one or more steps may be combined, split, or performed simultaneously. Those skilled in the art will also appreciate, in view of this disclosure, that different aspects or examples of the disclosure described herein may be combined to form other aspects or examples of the disclosure. [0142] Controller 1 [0143] Control Device/Teleoperation control unit 10 [0144] Support Request unit 11 [0145] Autonomous Driving Control unit 12 [0146] Other Electronic Control unit(s) 13-16 [0147] Data Storage Space 17 [0148] Management Table 20 [0149] Vehicle ID (data) 21 [0150] Operation mode (data) 22 [0151] Operator present (data) 23 [0152] Operator status (data) 24 [0153] C2V data 25 [0154] Vehicle 100 [0155] Sensors 101 [0156] Cockpit 102 [0157] Communication Device 103 [0158] Control/Driving Device 104, 104a, 104b [0159] Display 105 [0160] Communication Device/Transceiver 103 [0161] Other Vehicle(s) 150-158