IMAGE PROCESSING DEVICE, IMAGE PROCESSING METHOD, AND STORAGE MEDIUM

Abstract

An image processing device acquires a probability value and corresponding coordinates by using a learned model that outputs a probability value indicating an existence probability of a road boundary for each of coordinates in an image in response to an input of the image representing an area in front of a mobile object, captured by a camera mounted on the mobile object, rearranges a plurality of coordinates in descending order of the probability value, iteratively updates parameters of a function that approximates the road boundary in the image using the plurality of coordinates in the rearranged order, and executes driving control or driving assistance of the mobile object on the basis of the road boundary approximated by the function defined by the updated parameters.

Claims

1. An image processing device comprising: a storage medium configured to store an instruction readable by a computer and a processor that is connected to the storage medium, wherein the processor executes the instruction readable by the computer, thereby acquiring a probability value and corresponding coordinates by using a learned model that outputs a probability value indicating an existence probability of a road boundary for each of coordinates in an image in response to an input of the image representing an area in front of a mobile object, captured by a camera mounted on the mobile object, rearranging a plurality of coordinates in descending order of the probability value, iteratively updating parameters of a function that approximates the road boundary in the image using the plurality of coordinates in the rearranged order, and executing driving control or driving assistance of the mobile object on the basis of the road boundary approximated by the function defined by the updated parameters.

2. The image processing device according to claim 1, wherein the hardware processor acquires coordinates of the image for which the probability value is equal to or greater than a threshold value as candidate points of the road boundary.

3. The image processing device according to claim 1, wherein the hardware processor iteratively updates parameters of a function that approximates a road boundary by dividing the image at predetermined intervals, iteratively updating parameters of an element function that approximates the road boundary in each divided area, and combining the parameters.

4. The image processing device according to claim 1, wherein the hardware processor iteratively updates the parameters to minimize an error between a component of the coordinates in a first direction and an estimated value of the component in the first direction calculated on the basis of a component of the coordinates in a second direction.

5. An image processing method comprising: by a computer, acquiring a probability value and corresponding coordinates by using a learned model that outputs a probability value indicating an existence probability of a road boundary for each of coordinates in an image in response to an input of the image representing an area in front of a mobile object, captured by a camera mounted on the mobile object; rearranging a plurality of coordinates in descending order of the probability value; iteratively updating parameters of a function that approximates the road boundary in the image using the plurality of coordinates in the rearranged order, and executing driving control or driving assistance of the mobile object on the basis of the road boundary approximated by the function defined by the updated parameters.

6. A computer-readable non-transitory storage medium that stores a program causing a computer to execute: acquiring a probability value and corresponding coordinates by using a learned model that outputs a probability value indicating an existence probability of a road boundary for each of coordinates in an image in response to an input of the image representing an area in front of a mobile object, captured by a camera mounted on the mobile object, rearranging a plurality of coordinates in descending order of the probability value, iteratively updating parameters of a function that approximates the road boundary in the image using the plurality of coordinates in the rearranged order, and executing driving control or driving assistance of the mobile object on the basis of the road boundary approximated by the function defined by the updated parameters.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0014] FIG. 1 is a diagram which shows an example of an environment in which an image processing device mounted in a host vehicle M is used.

[0015] FIG. 2 is a diagram which shows an example of a configuration of the image processing device.

[0016] FIG. 3 is a diagram which shows an example of a method in which a candidate point extraction unit extracts candidate points on a track boundary.

[0017] FIG. 4 is a diagram which shows an example of a method for rearranging candidate points on a track boundary.

[0018] FIG. 5 is a diagram for describing an outline of a track boundary model updated by a model parameter updating unit.

[0019] FIG. 6 is a diagram which shows an example of a flow of model parameter update processing executed by a model parameter updating unit.

[0020] FIG. 7 is a graph for describing a method of calculating a reference value of a model parameter.

[0021] FIG. 8 is a sequence diagram which shows an example of a flow of processing executed by the image processing device.

DESCRIPTION OF EMBODIMENT

[0022] Hereinafter, embodiments of an image processing device, an image processing method, and a storage medium of the present invention will be described with reference to the drawings. In the present embodiment, the image processing device is, for example, a terminal device such as a smartphone having a camera and a display. However, the present invention is not limited to such a configuration, and the image processing device may be a computer device as long as it receives at least an image captured by a camera, processes it, and outputs a result of the processing to a display. In this case, functions of the present invention are realized by the camera, the display, and the image processing device in cooperation.

[0023] [Configuration]

[0024] FIG. 1 is a diagram which shows an example of an environment in which an image processing device 100 mounted in a host vehicle M is used. The host vehicle M is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electric power generated by a generator connected to the internal combustion engine, or electric power discharged from a secondary battery or a fuel cell.

[0025] As shown in FIG. 1, the image processing device 100 is installed in the host vehicle M so that the camera 10 can capture an image of an area in front of the host vehicle M in a traveling direction. The image processing device 100 is held by, for example, an in-vehicle holder (not shown) attached to a dashboard of the host vehicle M, and captures an image of the area in front of the host vehicle M. The host vehicle M is an example of a “mobile object.” In the following description, in the present embodiment, an example in which the image processing device 100 is mounted in the host vehicle M, which is a mobile object, is described, but more generally, the mobile object includes devices with cameras mounted in a vehicle such as dash cameras and smartphones.

[0026] FIG. 2 is a diagram which shows an example of a configuration of the image processing device 100. As shown in FIG. 2, the image processing device 100 includes, for example, a camera 10, a display unit 20, a candidate point extraction unit 110, a model parameter updating unit 120, and a driving control unit 130. The candidate point extraction unit 110, the model parameter updating unit 120, and the driving control unit 130 are realized by a hardware processor such as a central processing unit (CPU) executing a program (software). Some or all of these components may be realized by hardware (a circuit unit; including circuitry) such as large scale integration (LSI), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a graphics processing unit (GPU) or by software and hardware in cooperation. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as a hard disk drive (HDD) or flash memory, or may be stored in a removable storage medium (a non-transitory storage medium) such as a DVD or CD-ROM and may be installed by the storage medium being attached to a drive device. The camera 10 is, for example, a digital camera using a solid-state imaging device such as a charge coupled device (CCD) or complementary metal oxide semiconductor (CMOS). The display unit 20 is, for example, a display device such as a touch panel or liquid crystal display.

[0027] [Extraction of Candidate Point]

[0028] The candidate point extraction unit 110 extracts candidate points of a boundary of a track (a track boundary) on which the host vehicle M travels, on the basis of an image representing the area in front of the host vehicle M captured by the camera 10. FIG. 3 is a diagram which shows an example of a method in which the candidate point extraction unit 110 extracts candidate points on a track boundary. As shown in FIG. 3, when the candidate point extraction unit 110 acquires an image captured by the camera 10, it inputs the acquired image into a learned model (deep neural network; DNN) that has been learned to output a probability value (0 or more and 1 or less) indicating whether each pixel (coordinates) of the image is a track boundary according to the input of the image. The candidate point extraction unit 110 extracts pixels for which the output probability values take positive values as candidate points of the track boundary. Alternatively, the candidate point extraction unit 110 may also extract pixels for which the output probability values are equal to or greater than a threshold value (for example, 0.5) as candidate points of the track boundary.

[0029] [Update of Model Parameter]

[0030] The model parameter updating unit 120 rearranges the candidate points of the track boundary extracted by the candidate point extraction unit 110 in descending order of the probability value, and uses the candidate points of the track boundary in descending order of the probability value to update model parameters of a track boundary model to be described below. FIG. 4 is a diagram which shows an example of a method of rearranging the candidate points of a track boundary. In FIG. 4, k indicates a time point in a recognition cycle for recognizing a track boundary, N(k) indicates the number of candidate points of the track boundary acquired at the time point k, x′(n,k) indicates an x-coordinate of each candidate point before rearrangement, y′(n,k) indicates a y-coordinate of each candidate point before rearrangement, x(n,k) indicates an x-coordinate of each candidate point after rearrangement, and y(n,k) indicates a y-coordinate of each candidate point after rearrangement. As will be described below, by iteratively updating the model parameters of the track boundary model using the candidate points of the track boundary in descending order of the probability value, even if down-sampling occurs within one recognition cycle, it is possible to output a track boundary model with high accuracy even in a shorter time.

[0031] FIG. 5 is a diagram for describing an outline of the track boundary model updated by the model parameter updating unit 120. The model parameter updating unit 120 obtains an estimated value y.sub._hat(n,k) of the track boundary by iteratively substituting the candidate points of the track boundary rearranged in descending order of the probability value into the track boundary model defined by the following Equation (1).

[00001]$\begin{array}{cc}& \mathrm{Equation}\left(1\right)\end{array}$$y_{\_\mathrm{hat}}\left(n,k\right)=\underset{i=1}{\overset{m}{.Math.}}{w}_i\left(x\left(n,k\right)\right)\left(a_i\left(n,k\right){x\left(n,k\right)}^2+b_i\left(n,k\right)x\left(n,k\right)+c_i\left(n,k\right)\right)=\underset{i=1}{\overset{m}{.Math.}}{w}_i\left(x\left(n,k\right)\right){{\theta }_i\left(n,k\right)}^T\zeta \left(n,k\right)=\underset{i=1}{\overset{m}{.Math.}}{w}_i\left(x\left(n,k\right)\right)y_{\_\mathrm{hat}\mathrm{\_i}}\left(n,k\right)={{\theta }_i\left(n,k\right)}^T\zeta \left(n,k\right)$$\left(i=1~m\right)$

[0032] In Equation (1), a.sub.i(n,k), b.sub.i(n,k), and c.sub.i(n,k) represent a second order coefficient, a first order coefficient, and a constant term of a quadratic function (which may be referred to hereinafter as an “element function”) that approximates a track boundary in an image, respectively, and w.sub.i represents a weighting function that outputs a weight of 0 or more and 1 or less according to an x-coordinate of the input candidate point. More specifically, the weighting function is defined by the following Equations (2) to (4). In Equations (2) to (4), the values of x.sub.wi (i=1 to m) are fixed values set in advance. A sum of w.sub.i(x) (i=1 to m) is set to always be one.

[00002]$\begin{array}{cc}{w}_1\left(x\right)=\{\begin{array}{cc}1& \left(x\left(k\right)\leqq x_{w1}\right)\\ \frac{x_{w2}-x\left(k\right)}{x_{w2}-x_{w1}}& \left(x_{w1}<x\left(k\right){\mathrm{xx}}_{w2}\right)\\ 0& \left(x_{w2}\leqq x\left(k\right)\right)\end{array}& \mathrm{Equation}\left(2\right)\end{array}$$\left(i=1\right)$$\begin{array}{cc}{w}_i\left(x\right)=\{\begin{array}{cc}0& \left(x\left(k\right)\leqq x_{\mathrm{wi}-1}\right)\\ \frac{x\left(k\right)-x_{\mathrm{wi}-1}}{x_{\mathrm{wi}}-x_{\mathrm{wi}-1}}& \left(x_{\mathrm{wi}-1}<x\left(k\right)\leqq x_{\mathrm{wi}}\right)\\ \frac{x_{\mathrm{wi}+1}-x\left(k\right)}{x_{\mathrm{wi}+1}-x_{\mathrm{wi}}}& \left(x_{\mathrm{wi}}<x\left(k\right)<x_{\mathrm{wi}+1}\right)\\ 0& \left(x_{\mathrm{wi}+1}\leqq x\left(k\right)\right)\end{array}& \mathrm{Equation}\left(3\right)\end{array}$$\left(i=2~m-1\right)$$\begin{array}{cc}{w}_m\left(x\right)=\{\begin{array}{cc}0& \left(x\left(k\right)\leqq x_{\mathrm{wm}-1}\right)\\ \frac{x\left(k\right)-x_{\mathrm{wm}-1}}{x_{\mathrm{wm}}-x_{\mathrm{wm}-1}}& \left(x_{\mathrm{wm}-1}<x\left(k\right){\mathrm{xx}}_{\mathrm{wm}}\right)\\ 1& \left(x_{\mathrm{wm}}\leqq x\left(k\right)\right)\end{array}& \mathrm{Equation}\left(4\right)\end{array}$$\left(i=m\right)$

[0033] FIG. 5 shows element functions f.sub.1, f.sub.2, f.sub.3 and weighting functions w.sub.1, w.sub.2, w.sub.3 when m=3 in equations (1) to (4). That is, the track boundary model in the present embodiment divides an image into a plurality of areas based on the x-coordinate, approximates a track boundary by a quadratic function for each divided area, and approximates the track boundary in the entire image by combining the quadratic function for each area.

[0034] In this manner, compared to a conventional method (a least-squares method of a batch calculation method) in which the candidate points of a track boundary in an image are approximated by a single quadratic function, the track boundary is approximated by a quadratic function for each partial area of the image, and the approximated quadratic functions are combined to obtain a final approximation curve in the present embodiment, so that the track boundary in the image can be expressed with higher accuracy.

[0035] Note that, in the present embodiment, values and the number of x-coordinates x.sub.wi (i=1 to m) for dividing the area are fixed values set in advance. However, the present invention is not limited to such a configuration, and the values and number of x-coordinates x.sub.wi for dividing the area may be set as, for example, the number of clusters obtained by clustering the extracted candidate points and boundary points thereof.

[0036] Next, model parameter update processing executed by the model parameter updating unit 120 will be described with reference to FIG. 6. FIG. 6 is a diagram which shows an example of a flow of the model parameter update processing executed by the model parameter updating unit 120.

[0037] First, the model parameter updating unit 120 rearranges the candidate points (x′(n,k) and y′(n,k)) of the track boundary extracted by the candidate point extraction unit 110 in descending order of the probability value, and obtains candidate points (x(n,k) and y(n,k)). The model parameter updating unit 120 generates a vector ζ(n,k)=[x(n,k).sup.2,x(n,k),1] based on the candidate point x(n,k), and substitutes x(n,k) into the weighting function w.sub.i to obtain a weighting value w.sub.i(x(n,k)).

[0038] Next, the model parameter updating unit 120 calculates an inner product of the vector ζ(n,k) and model parameters θ.sub.i(n,k)=[a.sub.i(n,k),b.sub.i(n,k),c.sub.i(n,k)] and multiplies it by the weighting value w.sub.i(x(n,k)), thereby obtaining element functions f.sub.i(x)=w.sub.i(x(n,k))(a.sub.i(n,k)×(n,k).sup.2+b.sub.i(n,k)×(n,k)+c.sub.i(n,k)) for each area. The model parameter updating unit 120 obtains an output estimated value y.sub._hat(n,k) represented by Equation (1) by taking a sum of the elemental functions f.sub.i(x) for each area. Initial values of the model parameters θ.sub.i(n,k)=[a.sub.i(n,k),b.sub.i(n,k),c.sub.i(n,k)] to be multiplied at this time will be described below.

[0039] The model parameter updating unit 120 then calculates an identification error between the output estimated value y.sub._hat(n,k) and a y-coordinate y(n,k) of a candidate point as e.sub.id(n,k)=w.sub.i(x(n,k))(y(n,k)−y.sub._hat(n,k)). The identification error can be reflected for each area by multiplying the error between the output estimated value y.sub._hat(n,k) and the y-coordinate y(n,k) of the candidate point by the weighting value w.sub.i(x(n,k)). The model parameter updating unit 120 defines an adaptive gain K.sub.p for correcting the model parameter θ.sub.i(n,k) in a direction of decreasing a squared error of the identification error e.sub.id(n,k) by the following Equation (5).

[00003]$\begin{array}{cc}{K}_P\left(n,k\right)=\frac{P^{\prime }\left(n-1,k\right)\zeta \left(n,k\right)}{1+{\zeta \left(n,k\right)}^T{P}^{\prime }\left(n-1,k\right)\zeta \left(n,k\right)}& \mathrm{Equation}\left(5\right)\end{array}$

[0040] In Equation (5), P′(n,k) represents a covariance matrix of 3 rows and 3 columns and is a matrix defined by Equation (6) below.

[00004]$\begin{array}{cc}{P}^{\prime }\left(n,k\right)=\frac{1}{{\lambda }_1}\left(1-\frac{{\lambda }_2{P}^{\prime }\left(n-1,k\right)\zeta \left(n,k\right){\zeta \left(n,k\right)}^T}{{\lambda }_1+{\lambda }_2{\zeta \left(n,k\right)}^T{P}^{\prime }\left(n-1,k\right)\zeta \left(n,k\right)}\right)P^{\prime }\left(n-1,k\right)& \mathrm{Equation}\left(6\right)\end{array}$

[0041] In Equation (6), I represents a unit matrix of 3 rows and 3 columns, and λ.sub.1 and λ.sub.2 represent setting parameters of an iterative identification algorithm. λ.sub.1 and λ.sub.2 are constant values greater than 0 and equal to or less than 1. They are set as λ.sub.1=1 and λ.sub.2=1 when a least squares method is applied, set as λ.sub.1=λ(0<λ≤1) 1 and λ.sub.2=1 when a weighted least squares method is applied, and set as λ.sub.1=1 and λ.sub.2=0 when a fixed gain method is applied. When the fixed gain method is applied, the adaptive gain K.sub.p is represented by the following Equations (7) and (8).

[00005]$\begin{array}{cc}{K}_P\left(n,k\right)=\frac{P\zeta \left(n,k\right)}{1+{\zeta \left(n,k\right)}^{T}P\zeta \left(n,k\right)}& \mathrm{Equation}\left(7\right)\end{array}$$\begin{array}{cc}P=\left[\begin{array}{ccc}{P}_1& 0& 0\\ 0& P_2& 0\\ 0& 0& P_3\end{array}\right]& \mathrm{Equation}\left(8\right)\end{array}$

[0042] In Equation (8), P represents an identification gain matrix. P.sub.1, P.sub.2, and P.sub.3 represent identification gains and are positive fixed values. The model parameter updating unit 120 multiplies the identification error e.sub.id(n,k) by the adaptive gain K.sub.p, thereby obtaining a correction amount ddθ.sub.i(n,k) of the model parameter represented by the following equation (9).

ddθ.sub.i(n,k)=K.sub.p(n,k)e.sub.id_i(n,k) (i=1˜m)  Equation (9)

[0043] Next, the model parameter updating unit 120 multiplies a previous value dθ.sub.i (n−1,k) of the final correction amount, which will be described below, by a forgetting gain Δ.sub.fgt, and adds the obtained multiplied value to the correction amount ddθ.sub.i(n,k) of the model parameter of Equation (9), thereby obtaining a correction amount dθ.sub.raw_i(n,k) of the model parameter represented by the following Equation (10). In this manner, by defining the correction amount dθ.sub.raw_i(n,k) using a value obtained by multiplying the final correction amount dθ.sub.i(n−1,k) previously obtained by the forgetting gain Δ.sub.fgt, abrupt fluctuations of the track boundary model can be curbed.

dθ.sub.raw_i(n,k)=Δ.sub.fgtdθ.sub.i(n−1,k)+K.sub.p(n,k)e.sub.id_i(n,k)   Equation (10)

[0044] In Equation (10), the forgetting gain Δ.sub.fgt is a diagonal matrix of 3 rows and 3 columns represented by the following Equation (11). In Equation (11), δ.sub.fgt_1, δ.sub.fgt_2, and δ.sub.fgt_3 are constant values that satisfy 0<δ.sub.fgt_1, δ.sub.fgt_2<1, and δ.sub.fgt_3=1. That is, the forgetting gain Δ.sub.fgt is set to apply a forgetting effect to a.sub.i(n,k) and b.sub.i(n,k).

[00006]$\begin{array}{cc}{\Delta }_{\mathrm{fgt}}=\left[\begin{array}{ccc}{\delta }_{\mathrm{fgt}\_1}& 0& 0\\ 0& {\delta }_{\mathrm{fgt}\_2}& 0\\ 0& 0& {\delta }_{\mathrm{fgt}\_3}\end{array}\right]& \mathrm{Equation}\left(11\right)\end{array}$

[0045] Next, the model parameter updating unit 120 performs limiter processing (an example of a “constraint condition”) represented by the following Equations (12) to (14) on the correction amount dθ.sub.raw_i(n,k) of the model parameter to correct the correction amount of the model parameter and obtain a final correction amount dθ.sub.i(n,k) represented by the following Equation (15). In Equations (12) to (14), da.sub.L, da.sub.H, db.sub.L, db.sub.H, dc.sub.L, and dc.sub.H are fixed values set in advance, and are set to prevent the track boundary model from being in an unrealistic shape.

[00007]$\begin{array}{cc}{\mathrm{da}}_i\left(n,k\right)=\{\begin{array}{cc}{\mathrm{da}}_L\left(n,k\right)& \left({\mathrm{da}}_{\mathrm{raw}\_i}\left(n,k\right)<{\mathrm{da}}_L\right)\\ {\mathrm{da}}_{\mathrm{raw}\_i}\left(n,k\right)& \left({\mathrm{da}}_L\leqq {\mathrm{da}}_{\mathrm{raw}\_i}\left(n,k\right)\leqq {\mathrm{da}}_H\right)\\ {\mathrm{da}}_H\left(n,k\right)& \left({\mathrm{da}}_H<{\mathrm{da}}_{\mathrm{raw}\_i}\left(n,k\right)\right)\end{array}& \mathrm{Equation}\left(12\right)\end{array}$$\begin{array}{cc}{\mathrm{db}}_i\left(n,k\right)=\{\begin{array}{cc}{\mathrm{db}}_L\left(n,k\right)& \left({\mathrm{db}}_{\mathrm{raw}\_i}\left(n,k\right)<{\mathrm{db}}_L\right)\\ {\mathrm{db}}_{\mathrm{raw}\_i}\left(n,k\right)& \left({\mathrm{db}}_L\leqq {\mathrm{db}}_{\mathrm{raw}\_i}\left(n,k\right)\leqq {\mathrm{db}}_H\right)\\ {\mathrm{db}}_H\left(n,k\right)& \left({\mathrm{db}}_H<{\mathrm{db}}_{\mathrm{raw}\_i}\left(n,k\right)\right)\end{array}& \mathrm{Equation}\left(13\right)\end{array}$$\begin{array}{cc}{\mathrm{dc}}_i\left(n,k\right)=\{\begin{array}{cc}{\mathrm{dc}}_L\left(n,k\right)& \left({\mathrm{dc}}_{\mathrm{raw}\_i}\left(n,k\right)<{\mathrm{dc}}_L\right)\\ {\mathrm{dc}}_{\mathrm{raw}\_i}\left(n,k\right)& \left({\mathrm{dc}}_L\leqq {\mathrm{dc}}_{\mathrm{raw}\_i}\left(n,k\right)\leqq {\mathrm{dc}}_H\right)\\ {\mathrm{dc}}_H\left(n,k\right)& \left({\mathrm{dc}}_H<{\mathrm{dc}}_{\mathrm{raw}\_i}\left(n,k\right)\right)\end{array}& \mathrm{Equation}\left(14\right)\end{array}$$\begin{array}{cc}{d\theta }_i\left(n,k\right)=\left[\begin{array}{c}{\mathrm{da}}_i\left(n,k\right)\\ {\mathrm{db}}_i\left(n,k\right)\\ {\mathrm{dc}}_i\left(n,k\right)\end{array}\right]& \mathrm{Equation}\left(15\right)\end{array}$

[0046] Next, the model parameter updating unit 120 adds a reference value θ.sub.base_i(n,k) of the model parameter represented by the following Equation (16) to the correction amount dθ.sub.i(n,k), thereby obtaining a current model parameter value θ.sub.i(n,k) represented by the following Equation (17). The calculated current model parameter value θ.sub.i(n,k) is used as an identification value (an initial value) for calculating the model parameter θ.sub.i(n+1,k) for a next input value n+1.

[00008]$\begin{array}{cc}{\theta }_{\mathrm{base}\_i}\left(n,k\right)=\left[\begin{array}{c}{a}_{\mathrm{base}\_i}\left(n,k\right)\\ b_{\mathrm{base}\_i}\left(n,k\right)\\ c_{\mathrm{base}\_i}\left(n,k\right)\end{array}\right]& \mathrm{Equation}\left(16\right)\end{array}$
θ.sub.i(n,k)=θ.sub.base_i(n,k)+dθ.sub.i(n,k)   Equation (17)

[0047] [Reference Value Calculation of Model Parameter]

[0048] Next, a method for calculating the reference value θ.sub.base_i(n,k) of the model parameter will be described with reference to FIG. 7. FIG. 7 is a graph for describing a method of calculating a reference value of the model parameter. The model parameter updating unit 120 sets the reference value θ.sub.base_i(n,k) of the model parameter using the following Equations (18) to (20).

a.sub.base_i(n,k)=0  Equation (18)

c.sub.base_i(n,k)=0  Equation (19)

[00009]$\begin{array}{cc}{b}_{\mathrm{base}\_i}\left(n,k\right)=g\left(c_i\left(n-1,k\right)\right)={\alpha c}_i\left(n-1,k\right)+\beta & \mathrm{Equation}\left(20\right)\end{array}$

[0049] Equations (18) and (19) represent initial values of model parameters a.sub.base_i(n,k) and c.sub.base_i(n,k), respectively. As shown in Equations (18) and (19), since the track boundary model can be curved in both left and right directions, the initial value of the model parameter a.sub.base_i(n,k) may be zero, and since a model parameter c.sub.i, corresponding to a y-intercept of the track boundary model can be in both left and right directions, the initial value of the model parameter c.sub.base_i(n,k) may be zero.

[0050] In Equation (20), c.sub.i(n−1,k) represents a previously calculated identification value of the model parameter c.sub.i, and a function g represents a scaling function that gives a straight line passing through the identification value (that is, the y-intercept) of the model parameter c.sub.i, and a vanishing point VP of an image. That is, as shown in a left part of FIG. 7, as the identification value c.sub.i(n−1,k) takes a larger value, the reference value b.sub.base_i(n,k) of the model parameter takes a smaller value, and, as shown in a right part of FIG. 7, the straight line inclines to the left. By setting the reference value θ.sub.base_i(n,k) of the model parameter using the methods described above, it is possible to prevent the track boundary model from being in an unrealistic shape even when the number of DNN output values N(k) in a certain recognition cycle is extremely small due to, for example, an event such as bad weather or dark visibility conditions.

[0051] When the model parameter updating unit 120 determines a track boundary model for each recognition cycle, the driving control unit 130 performs automated driving or driving assistance of the host vehicle M on the basis of the determined track boundary model. More specifically, for example, the driving control unit 130 performs bird's-eye view conversion on a track boundary model in a camera coordinate system to obtain a track boundary model in a bird's-eye view coordinate system. The driving control unit 130 generates a target trajectory and an action plan of the host vehicle M using the track boundary model in the bird's-eye view coordinate system, and causes the host vehicle M to travel according to the generated target trajectory and action plan. In addition, for example, the driving control unit 130 uses the track boundary model in the bird's-eye view coordinate system to assist with steering or to issue a warning such that the host vehicle does not deviate from the determined track boundary model when an occupant of the host vehicle M performs manual driving.

[0052] FIG. 8 is a sequence diagram which shows an example of a flow of processing executed by the image processing device 100. As shown in FIG. 8, at a time point k−2, the image processing device 100 rearranges DNN output values x′(1,k−2), y′(1,k−2), x′(2,k−2), y′(2,k−2), . . . , x′(N(k−2),k−2)), y′(N(k−2),N(k−2)) in descending order of the probability value, and obtains x(1,k−2), y(1,k−2), x(2,k−2), y(2,k−2), . . . , x(N(k−2), k−2), and y(N(k−2), k−2). The image processing device 100 constitutes a vector ζ(n,k−2)=[x(n,k−2).sup.2,x(n,k−2),1] in descending order of the probability value, and inputs it to the iterative identification algorithm as shown in FIG. 6 to iteratively update the model parameter values θ.sub.i(n,k−2).

[0053] When a resampling timing T.sub.ds has elapsed while the model parameter value θi(n,k−2) is being updated, the image processing device 100 performs down-sampling on a model parameter value (for example, θ.sub.i(N(k−2)−1,k−2)) at a time when the resampling timing T.sub.ds has elapsed, and determines the result as the final model parameter value in a recognition cycle k−2. The image processing device 100 causes the display unit 20 to display a track boundary model set with the determined model parameter value θ.sub.i(N(k−2)−1,k−2). As described above, unlike the least-squares method of the batch calculation method, in the present embodiment, the model parameter value θ.sub.i(n,k) is iteratively updated by using the output values in descending order of the probability value, thereby it is possible to estimate a track boundary model with high accuracy even when the data amount is large and calculation cannot be completed by the least-squares method of the batch calculation method.

[0054] The image processing device 100, in a recognition cycle k−1, rearranges DNN output values x′(1,k−1), y′(1,k−1), x′(2,k−1), y′(2,k−1), . . . , x′(N(k−1),k−1), y′(N(k−1),k−1) in descending order of the probability value, and obtains x(1,k−1), y(1,k−1), x(2,k−1), y(2,k−1), . . . , x(N(k−1),k−1), and y(n(k−1),k−1). The image processing device 100 constitutes a vector ζ(n,k−1)=[x(n,k−1).sup.2,x(n,k−1), 1] in descending order of the probability value, and updates the model parameter value θ.sub.i(n,k−1) by inputting it to the iterative identification algorithm as shown in FIG. 6. At this time, the image processing device 100 uses the model parameter value θ.sub.i(N(k−2)−1,k−2)) determined in a recognition cycle k−2, as an initial value for calculating the estimated value y.sub._hat(1,k−1). When the model parameter value θ.sub.i(n,k−1) has been updated to the output values of x(N(k−1),k−1) and y(N(k−1),k−1), the image processing device 100 holds the model parameter value θ.sub.i(N(k−1),k−1) until a next recognition cycle k, and causes the display unit 20 to display a track boundary model with the held model parameter value θ.sub.i(N(k−1),k−1) set. Thereafter, in the recognition cycle k, the image processing device 100 similarly updates the model parameter value θ.sub.i(n,k) by using DNN output values x′(1,k), y′(1,k), x′(2,k), y′(2,k), . . . , x′(N(k),k), and y′(N(k),k). At this time, the image processing device 100 uses the model parameter value θ.sub.i(N(k−1),k−1) determined in the recognition cycle k−1 as an initial value for calculating the estimated value y.sub._hat(1,k).

[0055] According to the present embodiment described above, the image processing device divides an area in front of a host vehicle on a coordinate system based on the host vehicle at predetermined intervals, generates a function for approximating a road marking in each area on the basis of a probability value indicating an existence probability of the road marking for each of coordinates in each area obtained by the division and corresponding coordinates, and generates a function that approximates a road marking in the front area by combining the functions generated for each area. As a result, it is possible to estimate road markings that can be stably used for traveling control of mobile objects.

[0056] The embodiment described above can be expressed as follows.

[0057] An image processing device includes a storage medium that stores an instruction readable by a computer and a processor that is connected to the storage medium, in which the processor executes an instruction readable by the computer, thereby acquiring a probability value and corresponding coordinates by using a learned model that outputs a probability value indicating an existence probability of a road boundary for each of coordinates in an image in response to an input of the image representing an area in front of a mobile object, captured by a camera mounted on the mobile object, rearranging a plurality of coordinates in descending order of the probability value, iteratively updating parameters of a function that approximates the road boundary in the image using a plurality of coordinates in the rearranged order, and executing driving control or driving assistance of the mobile object on the basis of the road boundary approximated by the function defined by the updated parameters.

[0058] As described above, a mode for implementing the present invention has been described using the embodiments, but the present invention is not limited to such embodiments at all, and various modifications and replacements can be added within a range not departing from the gist of the present invention.