CONTROL OF HOST DEVICE USING THREE-DIMENSIONAL POSITION AND VELOCITY

Abstract

A system and method of controlling operation of a host device in real-time, the host device operatively connected to an optical device and a radar device. The optical device is configured to obtain visual data of at least one object. The object is located at an incline, relative to the host device, the incline being characterized by an elevation angle () and an azimuth angle (). The radar device is configured to obtain radar data, including a radial distance (r) of the object from the host device, the azimuth angle (), and a range rate (dr/dt). The controller is programmed to determine a time-to-contact for the host device and the object based at least partially on a 3-D position and 3-D velocity vector. The operation of the host device is controlled based at least partially on the time-to-contact.

Claims

1. A system of controlling operation of a host device in real-time, the system comprising: an optical device operatively connected to the host device and configured to obtain visual data of at least one object; wherein the at least one object is located at an incline relative to the host device, the incline being characterized by an elevation angle () and an azimuth angle (); a radar device operatively connected to the host device and configured to obtain radar data, including the azimuth angle (), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt); a controller operatively connected to the host device and including a processor and tangible, non-transitory memory on which is recorded instructions, execution of the instructions by the processor causing the controller to: determine a 3-D position matrix (r, , ) and a 3-D rate of change matrix (dr/dt, d/dt, d/dt) based on the visual data and the radar data; and control operation of the host device based at least partially on the 3-D position matrix (r, , ) and the 3-D rate of change matrix (dr/dt, d/dt, d/dt).

2. The system of claim 1, wherein prior to determining the 3-D position matrix (r, , ) and the 3-D rate of change matrix (dr/dt, d/dt, d/dt), the controller is configured to: obtain respective bounding boxes around the at least one object in the visual data; determine an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (d/dt) and a change rate of the azimuth angle (d/dt); and match the respective bounding boxes with the radar data.

3. The system of claim 2, wherein the controller is configured to: obtain a 3-D position vector (X, Y, Z) and a 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z) from the 3-D position matrix (r, , ) and the 3-D change rate matrix (dr/dt, d/dt, d/dt); and determine a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z).

4. The system of claim 3, wherein the controller is configured to: communicate with a remote module such that the remote module is not physically connected to the controller; transfer a location coordinate of the host device to the remote module; receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate; and determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.

5. The system of claim 3, wherein the controlling operation of the host device includes: sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold; and wherein the first predefined threshold is greater than the second predefined threshold.

6. The system of claim 5, wherein: the host device includes an automatic braking module configured to decelerate the host device; and controlling operation of the host device includes activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.

7. The system of claim 3, wherein: the controller includes an association module configured to perform the matching of the respective bounding boxes with the radar data.

8. The system of claim 3, wherein: the controller includes a Kalman filter module configured to perform the combining of the visual data and the radar data.

9. The system of claim 3, wherein the time-to-contact (t) is defined as:
t={square root over ((X/v.sub.x).sup.2+(Y/v.sub.y).sup.2+(Z/v.sub.z).sup.2)}

10. The system of claim 3, wherein: the respective bounding boxes include a first bounding box B.sub.1 characterized by an initial position (u,v); the optical flow is characterized by a first displacement change (u) of the respective bounding boxes in row coordinates and a second displacement change (v) of the respective bounding boxes in column coordinates, both over a predefined unit of time (t); and the 3-D position matrix (r, , ) and the 3-D change rate matrix (dr/dt, d/dt, d/dt) include respective components determined as a function of the optical flow, the initial position (u,v), an optical focal length (f) and the predefined unit of time (t) such that:
=inverse tangent(u/f);
=inverse tangent(v/f);
(d/dt)=v/(f*t); and
(d/dt)=u/(f*t).

11. The system of claim 3, wherein respective components of the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z) are defined as:
X=r sin cos ;
Y=r sin sin ;
Z=r cos ;
v.sub.x=[(dr/dt)sin cos +r(d/dt)cos cos r(d/dt)sin sin ];
v.sub.y=[(dr/dt)sin e sin +r(d/dt)cos sin +r(d/dt)sin cos ]; and
v.sub.z=[(dr/dt)cos r(d/dt)sin ].

12. A method of controlling operation of a host device in real-time, the host device operatively connected to an optical device and a radar device, the method comprising: obtaining visual data of at least one object; wherein the at least one object is located at an incline relative to the host device, the incline being characterized by an elevation angle () and an azimuth angle (); obtaining radar data via the radar device, the radar data including the azimuth angle (), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt); obtaining respective bounding boxes around the at least one object in the sequence of images in the visual data; determining an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (d/dt) and a change rate of the azimuth angle (d/dt); matching the respective bounding boxes with the radar data; determining a 3-D position matrix (r, , ) and a 3-D rate of change matrix (dr/dt, d/dt, d/dt) based on a combination of the visual data and the radar data; obtaining a 3-D position vector (X, Y, Z) and a 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z) from the 3-D position matrix (r, , ) and the 3-D change rate matrix (dr/dt, d/dt, d/dt); determining a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z); and controlling operation of the host device based at least partially on the time-to-contact.

13. The method of claim 12, wherein the controlling operation of the host device includes: sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold.

14. The method of claim 13, wherein the host device includes an automatic braking module configured to decelerate the host device and controlling operation of the host device includes: activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.

15. The method of claim 12, wherein the time-to-contact (t) is defined as:
t={square root over ((X/v.sub.x).sup.2+(Y/v.sub.y).sup.2+(Z/v.sub.z).sup.2)}

16. A system of controlling operation of a host device in real-time, the system comprising: an optical device operatively connected to the host device and configured to obtain visual data of at least one object; wherein the at least one object is located at an incline relative to the host device, the incline being characterized by an elevation angle () and an azimuth angle (); a radar device operatively connected to the host device and configured to obtain radar data, including the azimuth angle (), a radial distance (r) of the at least one object from the host device and a change rate of the range (dr/dt); a controller operatively connected to the host device and including a processor and tangible, non-transitory memory on which is recorded instructions, execution of the instructions by the processor causing the controller to: obtain respective bounding boxes around the at least one object in the visual data; determine an optical flow of the respective bounding boxes, including determining a change rate of the elevation angle (d/dt) and a change rate of the azimuth angle (d/dt); match the respective bounding boxes with the radar data; determine a 3-D position matrix (r, , ) and a 3-D rate of change matrix (dr/dt, d/dt, d/dt) by combining the visual data and the radar data; and determine a time-to-contact for the host device and the at least one object based at least partially on the 3-D position vector (X, Y, Z) and the 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z); and control operation of the host device based at least partially on the time-to-contact.

17. The system of claim 16, wherein the controller is configured to: communicate with a remote module such that the remote module is not physically connected to the controller; transfer a location coordinate of the host device to the remote module; receive a location assessment factor (LAF) from the remote module, based at least partially on the location coordinate; and determine the time-to-contact only if the location assessment factor (LAF) is at or above a threshold value.

18. The system of claim 17, wherein the host device includes an automatic braking module configured to decelerate the host device, and the controlling of the operation of the host device includes: sending a signal to a user of the host device, if the time-to-contact is less than a first predefined threshold but greater than a second predefined threshold; and activating the automatic braking module, if the time-to-contact is less than the second predefined threshold.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0011] FIG. 1 is a schematic rear view of an example host device;

[0012] FIG. 2 is a schematic top view of the host device of FIG. 1; and

[0013] FIG. 3 is a schematic flow diagram for a method of controlling operation of the host device of FIG. 1.

DETAILED DESCRIPTION

[0014] Referring to the drawings, wherein like reference numbers refer to like components, FIGS. 1-2 schematically illustrate a system 10 of controlling operation of a host device 12 in real-time. FIG. 1 is a schematic rear view of an example host device 12. FIG. 2 is a schematic top view of the host device 12. The host device 12 may be a mobile platform, such as, but not limited to, standard passenger car, sport utility vehicle, light truck, heavy duty vehicle, ATV, minivan, bus, transit vehicle, bicycle, robot, farm implement, sports-related equipment, boat, plane, train or other transportation device. The host device 12 may take many different forms and include multiple and/or alternate components and facilities.

[0015] Referring to FIG. 1, an optical device 16 is operatively connected to the host device 12 and configured to obtain visual data of at least one object 20 surrounding the host device 12. Referring to FIG. 2, a radar device 18 is operatively connected to the host device 12 and configured to obtain radar data of the object 20. Referring to FIG. 2, the system 10 is configured to obtain three-dimensional (hereinafter 3-D) position and velocity information of the object 20, by conjoining the radar data and the visual data. It is to be appreciated that the optical device 16 and the radar device 18 may be mounted at any position relative to the host device 12. In one embodiment, the optical device 16 is positioned at a side of the host device 12. In one embodiment, the radar device 18 is mounted at the rear of the host device 12 and at its center of gravity.

[0016] Referring to FIG. 1, a controller C is operatively connected to the host device 12 and includes at least one processor P and at least one memory M (or non-transitory, tangible computer readable storage medium) on which are recorded instructions for executing method 100, shown in FIG. 3, for controlling operation of the host device 12. The memory M can store controller-executable instruction sets, and the processor P can execute the controller-executable instruction sets stored in the memory M.

[0017] Referring to FIGS. 1-2, the controller C may be configured to communicate with a remote module 30 that is not physically connected to the controller C, via a wireless network 32. In one embodiment, the host device 12 is configured to receive and transmit wireless communication to the remote module 30 through a navigation sensor 34, shown in FIG. 1. In another embodiment, the host device 12 is configured to receive and transmit wireless communication to the remote module 30 through a mobile application 36, shown in FIG. 2. The mobile application 36 may be built into and run on an infotainment system 38 of the host device 12. The circuitry and components of a remote server 30 and mobile application 36 (apps) available to those skilled in the art may be employed. The mobile application 36 may be integral with or physically connected (e.g. wired) to the controller C, such that it has physical access to the data in the controller C.

[0018] Referring to FIG. 2, the remote module 30 may be operatively connected to a geographical database 50, via the wireless network 32. The geographical database 50 may be any public or commercial source of information available to those skilled in the art, such as for example, Google Earth. The remote module 30 may include at least one processor 40 and at least one memory 42 (or any non-transitory, tangible computer readable storage medium). The memory 42 can store controller-executable instruction sets, and the processor 40 can execute the controller-executable instruction sets stored in the memory 42. Referring to FIG. 1, the host device 12 may include an automatic braking module 60 configured to reduce motion or decelerate the host device 12.

[0019] Referring now to FIG. 3, a flowchart of the method 100 stored on and executable by the controller C of FIG. 1 is shown. Method 100 need not be applied in the specific order recited herein. Furthermore, it is to be understood that some steps may be eliminated. The start and end of the method 100 are indicated by S and E, respectively.

[0020] In block 102A of FIG. 3, the controller C is programmed to obtain the visual data. Referring to FIG. 1, the visual data includes a sequence of images, for example, a first image 22A and second image 22B of the object 20, at time t.sub.A and t.sub.B, respectively, in a first coordinate system 24. The object 20 is located at an incline relative to the host device 12. The incline is characterized by an elevation angle () (shown in FIG. 1) and an azimuth angle () (shown in FIG. 2).

[0021] In block 102B of FIG. 3, the controller C is programmed to obtain the radar data, in a second coordinate system 26 (axes labeled XYZ). The radar device 18 may include an embedded transmitter producing electromagnetic waves in the radio or microwave domain, a transmitting antenna, a receiving antenna, a receiver and an integrated processor. The electromagnetic waves reflect off the object 20 and return to the receiver, providing information about the location and speed of the object 20. The radar data includes a radial distance (r) of the object 20 from the host device 12, the azimuth angle (), and a range rate (dr/dt).

[0022] In block 104, the controller C is programmed to obtain respective bounding boxes around the object 20 in each of the images in the visual data. Referring to FIG. 1, a first bounding box B.sub.1 and a second bounding box B.sub.2, are obtained for the first and second images 22A, 22B, respectively. Each bounding box may be defined as the minimum or smallest bounding or enclosing box for a set of points (defining the object 20) in three dimensions. In other words, the bounding box in each of the sequence of images is the box with the smallest volume within which all the points defining the object 20 lie. Referring to FIG. 1, the controller C may include a deep learning module (shown as DL in FIG. 1) to detect and localize each bounding box. The controller C may include a graphics processing unit employed by those skilled in the art to perform this task in real time.

[0023] In block 104 of FIG. 3, the controller C is also programmed to determine an optical flow (F) of the respective bounding boxes, such as first and second bounding boxes B.sub.1, B.sub.2. The optical flow (F) is characterized by a first displacement change (u) of the first and second bounding boxes B.sub.1, B.sub.2 in row coordinates or pixels and a second displacement change (v) of the first and second bounding boxes B.sub.1, B.sub.2 in column coordinates or pixels, both over the predefined time period (t=t.sub.Bt.sub.A). Referring to FIG. 1, the position of the first bounding box B.sub.1 may be characterized by an initial position (u, v), i.e., the row and column coordinates at time twin the first coordinate system 24. The position of the second bounding box B.sub.2 may be characterized by the second position (u+u, v+v), i.e., the row and column coordinates at time t.sub.B in the first coordinate system 24. It is to be appreciated that the reference positions may be selected at any part of the respective bounding box: front, rear, top or bottom.

[0024] Block 104 includes determining the change rate of the elevation angle (d/dt) and the change rate of the azimuth angle (d/dt). The elevation angle (), azimuth angle (), the change rate of the elevation angle (d/dt) and the change rate of the azimuth angle (d/dt) may be determined as a function of the optical flow, the initial position (u,v), an optical focal length (f) .e., the projection of the radial distance (r) on the X-Y plane in the first coordinate system 24 and the predefined unit of time (t) such that:

(d/dt)=u/(f*t);

(d/dt)=v/(f*t);

=inverse tangent(u/f); and

=inverse tangent(v/f).

[0025] In block 106 of FIG. 3, the controller C is programmed to match the respective bounding boxes with the radar data. As noted above, the radar data includes a radial distance (r) of the object 20 from the host device 12, the azimuth angle (), and a range rate (dr/dt). The controller C may include an association module (AM) (see FIG. 1) configured to match the respective bounding boxes in the visual data with the radar data.

[0026] In block 108 of FIG. 3, the controller C is programmed to determine a 3-D position matrix (r, , ) and a 3-D rate of change matrix (dr/dt, d/dt, d/dt) by combining the visual data and the radar data. The controller C may include a Kalman filter module (KF) (see FIG. 1) to combine the visual data and the radar data. The Kalman filter module (KF) works in a recursive fashion and can run in real time, using the current state, the previously calculated state and its uncertainty matrix. The Kalman filter module (KF) may work in a two-step process. For example, in a first step, the Kalman filter module (KF) produces estimates of the current state variables, along with their uncertainties. Once the outcome of the next measurement (having a discrete amount of error such as random noise) is observed, these estimates are updated using a weighted average, with more weight being given to estimates with higher certainty.

[0027] In block 110 of FIG. 3, the controller C is programmed to determine a 3-D position vector (x, y, z) and a 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z) based at least partially on the 3-D position matrix (r, , ) and the 3-D rate of change matrix (dr/dt, d/dt, d/dt). The respective components of the 3-D position vector (x, y, z) and the 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z) may be obtained as:

x=r sin cos ;y=r sin sin ;z=r cos ;

v.sub.x=[(dr/dt)sin cos +r(d/dt)cos cos r(d/dt)sin sin ];

v.sub.y=[(dr/dt)sin sin +r(d/dt)cos sin +r(d/dt)sin cos ];

v.sub.z=[(dr/dt)cos r(d/dt)sin ].

[0028] From block 110, the method 100 may proceed to block 111 (or block 114 directly, as shown by line 112). In block 111, the controller C is programmed to transmit a location coordinate (i.e., geophysical coordinates) of the host device 12 to the remote module 30 (see FIG. 1) and receive a location assessment factor (LAF) from the remote module 30. The remote module 30 is configured to determine the location assessment factor (LAF) based at least partially on the location coordinate and a look-up table or repository in the memory 42. The look-up table is configured to store location assessment factors (LAF) for various location coordinates. For example, if the location coordinate is an intersection or the location coordinate indicates a school in proximity, the location assessment factor (LAF) may be selected to be above the threshold value. If the location coordinate indicates an ordinary street, the location assessment factor (LAF) may be selected to be below the threshold value.

[0029] From block 111, the method 100 proceeds to block 113, where the controller C is programmed to determine if the location assessment factor (LAF) is at or above a threshold value (e.g. 0.5). In one embodiment, the controller C is programmed to proceed to block 114 and determine the time-to-contact as long as the location assessment factor (LAF) is at or above the threshold value.

[0030] As noted above, the method 100 may proceed to block 114 directly from block 110 or from block 113. In block 114 of FIG. 3, the controller C is programmed to determine a time-to-contact for the host device and the object based at least partially on the 3-D position vector (x, y, z) and the 3-D velocity vector (v.sub.x, v.sub.y, v.sub.z). The time-to-contact (denoted by t) may be determined as:

t={square root over ((X/v.sub.x).sup.2+(Y/v.sub.y).sup.2+(Z/v.sub.z).sup.2)}

[0031] In block 116 of FIG. 3, the controller C is programmed to control operation of the host device 12 based on the time-to-contact. Block 116 includes sub-blocks 118, 120, 122 and 124. In block 118, the controller C is programmed to determine if the time-to-contact is less than a first predefined threshold (t.sub.1). If so, the method 100 proceeds to block 120. If not, the method 100 is ended. In block 120, the controller C is programmed to determine if the time-to-contact is less than a second predefined threshold (t.sub.2). The second predefined threshold is less than the first predefined threshold (t.sub.2<t.sub.1). Per block 122, controlling operation of the host device 12 includes activating the automatic braking module 60, if the time-to-contact is less than the second predefined threshold (t.sub.2). Per block 124, controlling operation of the host device 12 includes sending a signal to a user of the host device 12, if the time-to-contact is less than the first predefined threshold (t.sub.1) but greater than the second predefined threshold (t.sub.2).

[0032] The controller C (and execution of the method 100) improves the functioning of the host device 12 by precisely obtaining time-to-contact between the host device 12 and the respective ones of the plurality of objects 20 and controlling operation of the host device 12 based at least partially on the time-to-contact. The controller C of FIG. 1 may be an integral portion of, or a separate module operatively connected to, other controllers of the host device 12.

[0033] The controller C includes a computer-readable medium (also referred to as a processor-readable medium), including a non-transitory (e.g., tangible) medium that participates in providing data (e.g., instructions) that may be read by a computer (e.g., by a processor of a computer). Such a medium may take many forms, including, but not limited to, non-volatile media and volatile media. Non-volatile media may include, for example, optical or magnetic disks and other persistent memory. Volatile media may include, for example, dynamic random access memory (DRAM), which may constitute a main memory. Such instructions may be transmitted by one or more transmission media, including coaxial cables, copper wire and fiber optics, including the wires that comprise a system bus coupled to a processor of a computer. Some forms of computer-readable media include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, other magnetic media, a CD-ROM, DVD, other optical media, punch cards, paper tape, other physical media with patterns of holes, a RAM, a PROM, an EPROM, a FLASH-EEPROM, other memory chips or cartridges, or other media from which a computer can read.

[0034] Look-up tables, databases, data repositories or other data stores described herein may include various kinds of mechanisms for storing, accessing, and retrieving various kinds of data, including a hierarchical database, a set of files in a file system, an application database in a proprietary format, a relational database management system (RDBMS), etc. Each such data store may be included within a computing device employing a computer operating system such as one of those mentioned above, and may be accessed via a network in one or more of a variety of manners. A file system may be accessible from a computer operating system, and may include files stored in various formats. An RDBMS may employ the Structured Query Language (SQL) in addition to a language for creating, storing, editing, and executing stored procedures, such as the PL/SQL language mentioned above.

[0035] The detailed description and the drawings or FIGS. are supportive and descriptive of the disclosure, but the scope of the disclosure is defined solely by the claims. While some of the best modes and other embodiments for carrying out the claimed disclosure have been described in detail, various alternative designs and embodiments exist for practicing the disclosure defined in the appended claims. Furthermore, the embodiments shown in the drawings or the characteristics of various embodiments mentioned in the present description are not necessarily to be understood as embodiments independent of each other. Rather, it is possible that each of the characteristics described in one of the examples of an embodiment can be combined with one or a plurality of other desired characteristics from other embodiments, resulting in other embodiments not described in words or by reference to the drawings. Accordingly, such other embodiments fall within the framework of the scope of the appended claims.