System for sensing and responding to a lateral blind spot of a mobile carrier and method thereof

Abstract

The present application is to provide a system for sensing and responding to a lateral blind spot of a mobile carrier and method thereof, which is applied for a mobile carrier during moving to a parking place. Firstly, a light scan unit and a depth image capture unit are used to scan a plurality of surrounding objects and capture a plurality of object depth images of the surrounding objects, and then a plurality of screened images are obtained according to a moving route of the mobile carrier for further obtaining correspondingly a plurality of forecasted lines to generate corresponded notice message for noting driver or ADAS. Due to the objects corresponding to the screened images and located on a blind position which is at one side of the mobile carrier, the notice message provides the driver preventing from the ignored danger by ignoring the blind position.

Claims

1. A method for sensing and responding to a lateral blind spot of a mobile carrier, the mobile carrier comprising a host, a light scanner, and an image extraction unit, said host connected electrically to said light scanner and said image extraction unit, and said method comprising the following steps of: said host executing a parking command indicated of said host generating a positioning message according to a relative location or an absolute location of said mobile carrier with respective to a parking space located at one side of said mobile carrier; said host acquiring a first moving route indicated of said mobile carrier parking to said parking space according to said positioning message and a location message of said parking space; said host adopting said light scanner scanning the one or more objects located at and corresponding to the parking space, and said host adopting said image extraction unit extracting one or more object images corresponding to said one or more objects, and said one or more objects corresponding to a lateral blind spot of said mobile carrier; said host filtering said one or more object images using an image optical flow method according to first moving route and giving the corresponding one or more object images as one or more filtered images according to said first moving route; said host generating one or more predicted routes according to the corresponding one or more object vectors of said one or more filtered images; and said host modifying said first moving route according to said one or more predicted routes and generating a second moving route correspondingly; wherein in said step of said host modifying said first moving route according to said one or more predicted routes and generates a second moving route correspondingly, said host judges if a first effective area of said parking space is shrunk to a second effective area according to said one or more predicted routes, said first effective area is greater than a carrier size of said mobile carrier, said second effective area is smaller than said carrier size, and when said first effective area is shrunk to said second effective area, said second moving route is indicated of said mobile carrier parking to a portion of said parking space.

2. The method for sensing and responding to a lateral blind spot of a mobile carrier of claim 1, where in said step in which said light scanner scans one or more objects at said parking space according to said first moving route and said image extraction unit extracts one or more object images correspondingly, said light scanner further scans said one or more objects surrounding said parking space and said image extraction unit extracts t said corresponding one or more object images surrounding said parking space.

3. The method for sensing and responding to a lateral blind spot of a mobile carrier of claim 1, where in said step in which said host adopts an image optical flow method to classify said one or more object images, said host extracts a plurality of three-dimensional images according to said one or more filtered images and classifies said one or more object images using said image optical flow method according to said positioning message.

4. The method for sensing and responding to a lateral blind spot of a mobile carrier of claim 1, where in said step in which said host modifies said first moving route according to said one or more predicted routes and generates a second moving route correspondingly, said host calculates according to a corresponding radius difference between inner wheels and a turning angle of said first moving route and said one or more predicted path routes and then modifies said first moving route and generates said second moving route correspondingly.

5. A system for sensing and responding to a lateral blind spot of a mobile carrier comprising: a host, disposed in said mobile carrier, executing a parking command according to a parking space located at one side of said mobile carrier while said host generating a positioning message according to a relative location or an absolute location of said mobile carrier with respective to said parking space, and said host acquiring a first moving route indicated of said mobile carrier parking to said parking space according to said positioning message and a corresponding location message of said parking space; a light scanner, disposed on said side of said mobile carrier, scanning one or more objects located at and corresponding to said parking space according to said first moving route, and said one or more objects corresponding to a lateral blind spot of said mobile carrier; and an image extraction unit, disposed on said side of said mobile carrier and adjacent to said light scanner, connected electrically to said host, and extracting one or more object images corresponding to said one or more objects; where said host executes an image optical flow method according to said first moving route for filtering said one or more object images and giving one or more filtered images according to said first moving route; said host generates one or more predicted routes according to one or more object vectors of said one or more filtered images; and said host modifies said first moving route according to said one or more predicted routes and generates a second moving route correspondingly, said host judges if a first effective area of the parking space is shrunk to a second effective area according to said one or more predicted routes, said first effective area is greater than a carrier size of said mobile carrier, said second effective area is smaller than said carrier size, and when said first effective area is shrunk to said second effective area, said second moving route is indicated of said mobile carrier parking to a portion of said parking space.

6. A system for sensing and responding to a lateral blind spot of a mobile carrier system of claim 5, wherein said light scanner is a lidar or a radar scanner.

7. A system for sensing and responding to a lateral blind spot of a mobile carrier of claim 5, wherein said host calculates according to a corresponding radius difference between inner wheels and a turning angle of said first moving route and said one or more predicted routes and then modifies said first moving route and generates said second moving route correspondingly.

8. A system for sensing and responding to a lateral blind spot of a mobile carrier of claim 5, wherein a location of the lateral blind spot is a blind spot region corresponding to said parking space of said mobile carrier and defined by the intelligent transport system ISO 17387.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 shows a flowchart according to an embodiment of the present application;

(2) FIG. 2A to FIG. 2F show schematic diagrams of partial steps according to an embodiment of the present application;

(3) FIG. 3 shows a schematic diagram of perspective projection method according to an embodiment of the present application;

(4) FIG. 4 shows a schematic diagram of parking in a portion of the parking space according to an embodiment of the present application; and

(5) FIG. 5 shows a schematic diagram of parking in a parking space according to an embodiment of the present application.

DETAILED DESCRIPTION

(6) Since the radar system according to the prior art and dash cams cannot provide prediction of lateral blind spots of a mobile carrier, the present application provides a system for sensing and responding to a lateral blind spot of a mobile carrier and the method thereof for avoiding the dangerous situations caused by later blind spots of a mobile carrier.

(7) In the following, the properties and the accompanying system of the mobile carrier warning sensor fusion system and the method thereof according to the present application will be further illustrated.

(8) First, please refer to FIG. 1, which shows a flowchart according to an embodiment of the present application. As shown in the figure, according to the method for sensing and responding to a lateral blind spot of a mobile carrier of the present application, the host should execute the following steps: Step S10: Judging if the mobile carrier turns and moves to a parking space; Step S12: Generating a positioning message according to the relative location or absolute location of the mobile carrier with respective to the parking space; Step S122: The host generating a first moving route according to the positioning message and a location message of the parking space; Step S14: A light scanning unit scanning the corresponding objects at or surrounding the parking space and an image extraction unit extracting the corresponding object images; Step S16: Classifying the object images using an image optical flow method and giving the corresponding filtered images of the first moving route; Step S18: Generating a predicted route according to the corresponding object vectors of the filtered images; and Step S20: Adjusting the first moving route according to the predicted route and generating a corresponding second moving route.

(9) Please refer to FIG. 2A to FIG. 2E, which illustrate the accompanying identification system 1 for the method for sensing and responding to a lateral blind spot of a mobile carrier according to the present application. The system 1 comprises a host 10, a light scanning unit 20, and an image extraction unit 30. The host 10 according to the present embodiment is an automotive computer that includes a processing unit 12 and a memory 14. Nonetheless, the present application is not limited to the embodiment. The host 10 according to the present application can be a server, a notebook computer, a tablet computer, or any electronic device with image processing capability. The light scanning unit 20 according to the present embodiment is a lidar or a laser scanner. The image extraction unit 30 according to the present embodiment is a color image extraction unit, for example, an automotive CMOS image sensor. The host 10 uses the processing unit 12 to execute an operational program P for receiving the image data IMG generated by the image extraction unit 30 and performing image processing. The host 10 is disposed in a mobile carrier V. The light scanning unit 20 and image extraction unit 30 are disposed on one side of the mobile carrier V. The host 10 is connected electrically to the light scanning unit 20 and the image extraction unit 30. An image extraction angle of the image extraction unit 30 according to the present embodiment is between 120 and 170 degrees and extracts object images with 10 meters. Besides, the host 10 is further connected electrically to a positioning unit 40.

(10) In the step S10, as shown in FIG. 2A, the host 10 judges if a parking command CMD is executed. In other words, the host judges the mobile carrier V turns and heads for a parking space 50. If not, the host 10 continue to judge if there is a parking command by re-executing the step S10. When there is a parking command CMD, the step S12 is executed. Please refer to FIG. 2A and FIG. 2B. According to the present embodiment, a positioning message 42 generated by the positioning unit 40 is transmitted to the processing unit 12 of the host 10. The positioning unit 40 generates the positioning message 42 to the processing unit 12 according to the absolute location of the mobile carrier V and the parking space 50. Then the processing unit 12 generates a corresponding first moving route L1 of the mobile carrier V according to the positioning message 42 and the parking space 50 in the step S122. For example, the first moving route L1 is the mobile carrier V turns and heads for the parking space 50. The first moving route L1 is a predetermined route for the mobile carrier V to move to the parking space 50. Thereby, according to the present embodiment, the step S14 will be executed subsequently. In addition to using the positioning unit 40 to provide the positioning message 42 of absolute location, the light scanning unit 20 can perform optical scanning on one side of the mobile carrier V or even 10 to 50 meters surrounding the mobile carrier V for providing the positioning message 42 of relative location. In other words, the light scanning unit 20 acquires the positioning result for the space surrounding the mobile carrier V and hence providing the positioning message 42 corresponding to the parking space 50 with respect to the mobile carrier V.

(11) The host 10 executes the step S14. Please refer again to FIG. 2A and FIG. 2B. The host 10 uses the light scanning unit 20 to perform optical scanning on one side of the mobile carrier V, especially on the parking space 50, according to the first moving route L1. It also scans the surroundings of the parking space 50. Namely, the light scanning unit 20 scans the objects corresponding to the parking space 50. The scanning method of the light scanning unit 20 is to generate one or more optical gratings 22 to one or more objects. According to the present embodiment, the objects includes a first object VO1 and a second object VO2, which will produce reflection light 32 according to the optical grating 22 to the image extraction unit 30 and hence producing a plurality of object images OBJ correspondingly. According to the present embodiment, the light scanning unit 20 is a lidar. A plurality of parallel stripes of light, particularly, vertical laser light, form the optical grating 22. The image extraction unit 30 extracts the corresponding reflection light 32 of the optical grating 22 and generates the corresponding object images OBJ of the reflection light 32. In addition, the light scanning unit 20 according to the present application can further be a laser scanner which achieves the effect of a lidar by a plurality of laser scans. The processing unit 12 executes the operational program P for processing the object images OBJ extracted by the image extraction unit 30 and hence highlighting the object images OBJ corresponding to the first object VO1 and the second object VO2, as well as performing image stitching or color and greyscale calibration on the object images OBJ for subsequent spatial identification.

(12) The location of the lateral blind spot is a blind spot region corresponding to the parking space of the mobile carrier V and defined by the intelligent transport system ISO 17387. For the first object VO1 or the second object VO2 in the blind spots, the light scanning unit 20 and the image extraction unit 30 can assist to extract the unaware places. In addition, the ADAS also needs a more complete image extraction for identifying lateral objects, such as pedestrians, cars, bus stops, traffic labels, or traffic lights, or even the A-pillars, which are the visual direction that always induces blind spots.

(13) Next, in the step S16, as shown in FIG. 2C, the processing unit 12 executes an image optical flow method L for filtering the object images OBJ and giving the filtered images IMG. In other words, the processing unit 12 filters the corresponding objects according the first moving route L1 of the mobile carrier V and acquiring the corresponding filtered images IMG. For example, if the object is an roadside object or car, the processing unit 12 will not take its corresponding object image OBJ into consideration and the corresponding object image OBJ will not be labeled as one of the filtered images. As shown in FIG. 2B, the object VO includes the first object VO1 and the second object VO2. The second object VO2 is a roadside car and hence will not influence the first moving route L1 of the mobile carrier V. Thereby, the object image OBJ of the second object VO2 will not be labeled as a filtered image IMG. That is to say, the object image OBJ of the first object VO1 will be filtered and become a filtered image IMG. The processing unit 12 according to the present embodiment executes the operational program P to extract a three-dimensional (3D) image V3D of the first object VO1 and performs spatial identification according to the three-dimensional image V3D. Namely, the host 10 performs spatial identification according to the three-dimensional image V3D and uses the positioning message 42 provided by the positioning unit 40 to confirm that the second object VO2 is a parked car and not moved. In addition, the first object VO1 according to the present embodiment is the person taking the mobile carrier V. Nonetheless, the present application is not limited to the embodiment. Alternatively, the first object VO1 can be a moving car.

(14) In the step S18, please refer to FIG. 2B and FIG. 2D, the host 10 executes the operational program P and performs a prediction operation according to the filtered images IMG for predicting the predicted route ML corresponding to the first object VO1 of the filtered images IMG. The processing unit 12 performs the prediction operation according to the positioning message 42 and the corresponding object vectors of the filtered images IMG to give the corresponding route data MLD of the filtered images IMG. The route data MLD correspond to the predicted route ML shown in FIG. 2B. The corresponding object vectors of the filtered images IMG can be a zero vector, representing a stationary object influencing the first moving route L1.

(15) In the step S20, please refer to FIG. 2B and FIG. 2E, the host 10 executes the operational program P and refers to the first moving route L1 of the mobile carrier V to give first moving data L1D, for example, the turning angle and the radius difference between inner wheels. The first moving data L1D is calculated with the route data MLD given in the step S18 to generate a second moving route L2. The host 10 will adjust the first moving data L1D according to the route data MLD and hence adjusting the first moving route L1 of the mobile carrier V for further generating second moving data L2D of the second moving route L2, for example, delaying moving, changing the inserting angle of the mobile carrier V into the parking space 50, or changing the parking space 50. In addition to displaying on a display unit (not shown in the figures) for notifying the driver of the mobile carrier V with the dangerous situation at the blind spot on one side of the mobile carrier V, the second moving route L2 generated by the host 10 according to the present application can be further applied to the ADAS for intervening drivers' the driving behaviors for avoiding danger.

(16) The equations for calculating the radius difference between inner wheels include:

(17) $\begin{array}{cc}a=\sqrt{R^2-L^2}-\frac{d_2}{2}& \left(1\right)\end{array}$$\begin{array}{cc}\cos =\frac{a+\frac{d_2}{2}}{R}& \left(2\right)\end{array}$$\begin{array}{cc}b=\sqrt{R^2+\left(\frac{d_2}{2}\right)-d_{1}R\cos }& \left(3\right)\end{array}$$\begin{array}{cc}m=b-a& \left(4\right)\end{array}$

(18) R is the turning radius of the mobile carrier V; L is the wheelbase; d.sub.1 is the distance between front wheels; d.sub.2 is the distance between rear wheels; is the angle between the midpoint of the front and rear axles of the mobile carrier V and the center of the turning circle; a is the moving radius of the central line of the inner rear wheel; b is the moving radius of the central line of the inner front wheel; and m is the radius difference of inner wheel of a non-trailer carrier.

(19) As shown in FIG. 3, by using the perspective projection method, the image point P.sub.0 extracted by the image extraction unit 30 includes a first image point P.sub.1 and a second image point P.sub.2. The coordinates (x, y) of the first image point P.sub.1 are located in the first domain DM1; the coordinates (x, y) of the second image point P.sub.2 are located in the second domain DM2. Thereby, the relation between the first image point P.sub.1 and the second image point P.sub.2 extracted by the image extraction unit 30 can be expressed by the following equations:

(20) $\begin{array}{cc}{x}^{}=\frac{m_{0}x+m_{1}y+m_2}{m_{6}x+m_{7}y+1}& \left(5\right)\end{array}$$\begin{array}{cc}{y}^{}=\frac{m_{3}x+m_{4}y+m_5}{m_{6}x+m_{7}y+1}& \left(6\right)\end{array}$

(21) (x,y) is the first image point P.sub.1; (x, y) is the second image point P.sub.2; m.sub.0, m.sub.1, . . . m.sub.7 are the parameters relevant to the image extraction unit 30, including the focal length, the turning angle, and sizing parameters. The image points can be expanded to a plurality of image point pairs. Then the Levenberg-Marquardt algorithm can be used to perform nonlinear minimization and giving the optimum values of m.sub.1 to m.sub.7, which is used as the optimum focal length for the image extraction unit 30.

(22) The above image optical flow method L adopts the Lucas-Kanade optical flow algorithm for estimating obstacles. The image difference method is used first. Then the image constraint equation is expanded by the Taylor equation:

(23) $\begin{array}{cc}I\left(x+x,y+y,z+z,t+t\right)=I\left(x,y,z,t\right)+\frac{I}{x}x+\frac{I}{y}y+\frac{I}{z}z+\frac{I}{t}t+H.O.T.& \left(7\right)\end{array}$
where H.O.T. means higher order terms in the equation and can be neglected for infinitesimal displacement. According to the equation, we can get:

(24) $\begin{array}{cc}\frac{I}{x}x+\frac{I}{y}y+\frac{I}{z}z+\frac{I}{t}t=0\mathrm{or}& \left(8\right)\end{array}$$\begin{array}{cc}\frac{I}{x}\frac{x}{t}+\frac{I}{y}\frac{y}{t}+\frac{I}{z}\frac{z}{t}+\frac{I}{t}\frac{t}{t}=0& \left(9\right)\end{array}$
and giving:

(25) $\begin{array}{cc}\frac{I}{x}{V}_x+\frac{I}{y}{V}_y+\frac{I}{z}{V}_z+\frac{I}{t}=0& \left(10\right)\end{array}$

(26) V.sub.x, V.sub.y, V.sub.z are formed by x, y, z in the optical flow vector I(x,y,z,t).

(27) $\frac{I}{x},\frac{I}{y},\frac{I}{z},\mathrm{and}\frac{I}{t}$
are the partial derivatives of the image with respective to the corresponding directions at the point (x,y,z,t). Thereby, equation (10) can be converted to the following equation:
I.sub.xV.sub.x+I.sub.yV.sub.y+I.sub.zV.sub.z=I.sub.t(11)

(28) Rewriting equation (11) as:
I.sup.T.Math.{right arrow over (V)}=I.sub.t(12)

(29) Since equation (10) contains three unknowns (Vx,Vy,Vz), the subsequent algorithm can solve for the unknowns.

(30) First, assume that the optical flow vector (V.sub.x, V.sub.y, V.sub.z) is constant in a small m*m*m (m>1) cube. Then, according to the voxel 1 . . . n, n=m.sup.3, the following equation set will be given:

(31) $\begin{array}{cc}{I}_{x_1}{V}_x+I_{y1}{V}_y+I_{z_1}{V}_z=-I_{t_1}& \left(13\right)\end{array}$$I_{x_2}{V}_x+I_{y2}{V}_y+I_{z_2}{V}_z=-I_{t_2}$$.Math.$$I_{x_n}{V}_x+I_{yn}{V}_y+I_{z_n}{V}_z=-I_{t_n}$

(32) The above equation contain three unknowns and form an overdetermined equation set, meaning there is redundancy therein. The equation set can be represented as:

(33) $\begin{array}{cc}\left[\begin{array}{ccc}{I}_{x1}& I_{y1}& I_{z1}\\ I_{x2}& I_{y2}& I_{z2}\\ .Math.& .Math.& .Math.\\ I_{x_n}& I_{y_n}& I_{z_n}\end{array}\right]\left[\begin{array}{c}{V}_x\\ V_y\\ V_z\end{array}\right]=\left[\begin{array}{c}-I_{t_1}\\ -I_{t_2}\\ .Math.\\ -I_{t_n}\end{array}\right]& \left(14\right)\end{array}$
Denote (14) as:
A{right arrow over (v)}=b(15)

(34) To solve this overdetermined problem, equation (15) adopts the least square method to give:
A.sup.TA{right arrow over (v)}=A.sup.T(b)(16)
{right arrow over (v)}=(A.sup.TA).sup.1A.sup.T(b)(17)
We can get:

(35) $\begin{array}{cc}\left[\begin{array}{c}{V}_x\\ V_y\\ V_z\end{array}\right]={\left[\begin{array}{ccc}.Math.I_{x_i}^2& .Math.I_{x_i}{I}_{y_i}& .Math.I_{x_i}{I}_{z_i}\\ .Math.I_{x_i}{I}_{y_i}& .Math.I_{y_i}^2& .Math.I_{x_i}{I}_{z_i}\\ .Math.I_{x_i}{I}_{z_i}& .Math.I_{y_i}{I}_{z_i}& .Math.I_{z_i}^2\end{array}\right]}^{-1}\left[\begin{array}{c}-.Math.I_{x_i}{I}_{t_i}\\ -.Math.I_{y_i}{I}_{t_i}\\ -.Math.I_{z_i}{I}_{t_i}\end{array}\right]& \left(18\right)\end{array}$

(36) Substituting the result of equation (18) into equation (10) for estimating acceleration vector information and distance information of one or more objects. Thereby, the one or more objects can be classified and their route can be predicted. For example, the object image OBJ of the first object VO1 is classified as a filtered image IMG, and the predicted route ML of the first object VO1 is predicted.

(37) In addition, as shown in FIG. 4, the host can further get a first effective area A1 of the parking space 50 and a carrier size S, namely, the visual length and width, of the mobile carrier V. In the step S20, the processing unit 12 of the host 10 can judge if the first effective area A1 is shrunk to a second effective area A2. The first effective area A1 is greater than the carrier size S; the second effective area A2 is smaller than the carrier size S. When the processing unit 12 of the host 10 judges that the first effective area A1 is shrunk to a second effective area A2, the processing unit 12 adjusts the second moving data L2D so that the second moving route L2 guides the mobile carrier V to park to a portion of the parking space 50. For example, one of the first objects VO1 is located on a side edge of the parking space 50 and shrinking the effective area of the parking space 50 to 80% and smaller than the carrier size S. A portion of the mobile carrier V is located on or even exceeding the edge of the parking space 50. As shown in FIG. 5, when the processing unit 12 judges that the effective area of the parking space 50 is not changed, the processing unit 12 maintains the second moving data L2D and the second moving route L2 guides the mobile carrier V to parking into the parking space.

(38) To sum up, the present application provides a system for sensing and responding to a lateral blind spot of a mobile carrier and the method thereof. The host acquires the object images of a plurality of objects on one side of a mobile carrier for classifying and giving filtered images. Then prediction calculations are performed on the corresponding objects of the filtered images to give predicted route. The predicted route is calculated with the moving route of the mobile carrier to give a second moving route. Besides, the host can further adjust the moving data according to the route data for avoiding dangerous situations.