ELECTRO-HYDRAULIC VARIFOCAL LENS-BASED METHOD FOR TRACKING THREE-DIMENSIONAL TRAJECTORY OF MOVING OBJECT

Abstract

The present disclosure discloses an electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object. The method includes the following steps of: (1) obtaining a functional relation between a focusing control current and camera's intrinsic parameters; (2) obtaining a functional relation between focusing control currents of the electro-hydraulic varifocal lens and an optimal object distance; (3) initializing an object tracking algorithm, and taking an object tracking box as a subsequent focusing window; (4) carrying out first autofocusing, recording a focusing control current value after the autofocusing is completed, as well as a size and center point coordinates of the object tracking box; (5) calculating and recording coordinates of the object in 3D space; and (6) repeating steps (4) and (5) for the same object, and sequentially connecting the recorded coordinates of the object in 3D space into a trajectory.

Claims

1. An electro-hydraulic varifocal lens-based method for tracking a three-dimensional (3D) trajectory of a moving object, comprising: step 1, calibrating the electro-hydraulic varifocal lens under different focal distances to obtain a functional relation between a focusing control current and camera's intrinsic parameters; step 2, establishing an electro-hydraulic varifocal lens-based optical imaging system model to obtain a functional relation between a focusing control current of the electro-hydraulic varifocal lens and an optimal object distance; step 3, initializing an object tracking algorithm, generating an object tracking box, and selecting a to-be-tracked object; step 4, carrying out autofocusing, and recording a focusing control current after the autofocusing is completed, as well as a size of the object tracking box in an image and center point coordinates after undistortion; step 5, calculating, by a camera projection model, coordinates of the object in 3D space and recording the same; and step 6, repeating steps 4-5 for the same tracked object, and sequentially connecting the recorded coordinates of the object in 3D space into a trajectory.

2. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 1, wherein step 1 specifically comprises calibrating the electro-hydraulic varifocal lens under multiple focusing control currents to obtain the functional relation between the focusing control current and the camera's intrinsic parameters by curve fitting:
(f.sub.x,f.sub.y)=H(I)  (1) wherein f.sub.x and f.sub.y denote equivalent focal distances of a camera in x and y directions of a pixel plane respectively, in a unit of px; and I denotes a focusing control current of an electro-hydraulic varifocal lens; and obtaining coordinates of a camera's optical center on the pixel plane, and a slant parameter between horizontal and vertical edges of a camera's photosensitive element.

3. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 2, wherein step 2 specifically comprises recording an optimal object distance under multiple focusing control currents by using the electro-hydraulic varifocal lens-based optical imaging system model, and conducting curve fitting on the recorded data to obtain a functional relation between focusing control currents of the electro-hydraulic varifocal lens and the optimal object distance:
u=F(I)  (2) wherein u denotes an optimal object distance, and I denotes a focusing control current of the electro-hydraulic varifocal lens.

4. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 3, wherein the autofocusing in step 4 comprises first autofocusing and subsequent autofocusing, and the first autofocusing specifically comprises: (1) searching an initial focusing control current at a certain stride, calculating a sharpness evaluation value of an internal image region of the object tracking box, obtaining a maximum sharpness evaluation value and a focusing control current corresponding to the maximum sharpness evaluation value, and setting a sharpness evaluation threshold:
K=αD.sub.max  (3) wherein α denotes a preset sharpness confidence level, and α<1; K denotes a sharpness evaluation threshold used in the subsequent autofocusing; and D.sub.max denotes a maximum sharpness evaluation value; and (2) after autofocusing is finished, recording a size of the object tracking box in an image and center point coordinates after undistortion.

5. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 4, wherein the subsequent autofocusing specifically comprises: calculating a sharpness evaluation value D.sub.i of the internal image region of the object tracking box; and if D.sub.i≥K, directly recording the focusing control current I.sub.i at this moment, as well as a size size.sub.i of the object tracking box in an image and center point coordinates after undistortion; or if D.sub.i<K, reading a size size.sub.i of the object tracking box in the image at this moment, comparing the size with a size size.sub.i-1 of the object tracking box at last successful focusing, and adjusting a focusing control current to complete focusing; and after the focusing is completed, recording the focusing control current and the size of the object tracking box in the image after focusing and center point coordinates after undistortion.

6. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 5, wherein when D.sub.i<K, the focusing control current is adjusted based on a comparison result of size, and size.sub.i-1, which specifically comprises: if size.sub.i<size.sub.i-1, searching the focusing control current at a certain stride in the direction where the optimal object distance becomes longer, calculating a sharpness evaluation value in the object tracking box, and completing focusing after the sharpness evaluation value is greater than or equal to the sharpness evaluation threshold; or if size.sub.i>size.sub.i-1, searching the focusing control current at a certain stride in the direction where the optimal object distance becomes shorter, calculating a sharpness evaluation value in the object tracking box, and completing focusing after the sharpness evaluation value is greater than or equal to the sharpness evaluation threshold.

7. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 6, wherein the undistortion in step 4 specifically comprises: calculating an undistorted image of a current frame according to a distortion model, and reading and recording center point coordinates of the object tracking box in the undistorted image.

8. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 5, wherein the camera projection model in step 5 is: $\begin{array}{cc}\left(\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right)=\frac{1}{Z_i}\left(\begin{array}{ccc}{f}_{xi}& s& c_x\\ 0& f_{yi}& c_y\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}{X}_i\\ Y_i\\ Z_i\end{array}\right)& \left(4\right)\end{array}$ wherein (x.sub.i, y.sub.i) denote center point coordinates of an object tracking box in an undistorted image, c.sub.x and c.sub.y denote coordinates of a camera's optical center on the pixel plane, s denotes a slant parameter between horizontal and vertical edges of a camera's photosensitive element, f.sub.xi, f.sub.yi, denote equivalent focal distances of a camera in x and y directions corresponding to a focusing control current I.sub.i at this moment respectively, and (X.sub.i, Y.sub.i, Z.sub.i) denote 3D coordinates of a center point of a tracked object; and Z.sub.i=u.sub.i, wherein u.sub.i denotes an optimal object distance corresponding to the focusing control current I.sub.i at this moment.

9. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 4, wherein the sharpness evaluation value is calculated using a Laplacian function, and the Laplacian function is expressed as: $\begin{array}{cc}D\left(f\right)={.Math.}_y{.Math.}_x.Math.G\left(x,y\right).Math.& \left(5\right)\end{array}$ wherein G(x, y) denotes convolution of a Laplacian operator at a pixel point (x, y), and the Laplacian operator is expressed as: $\begin{array}{cc}L=\frac{1}{6}\left[\begin{array}{ccc}1& 4& 1\\ 4& -20& 4\\ 1& 4& 1\end{array}\right].& \left(6\right)\end{array}$

10. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 5, wherein the sharpness evaluation value is calculated using a Laplacian function, and the Laplacian function is expressed as: $\begin{array}{cc}D\left(f\right)={.Math.}_y{.Math.}_x.Math.G\left(x,y\right).Math.& \left(5\right)\end{array}$ wherein G(x, y) denotes convolution of a Laplacian operator at a pixel point (x, y), and the Laplacian operator is expressed as: $\begin{array}{cc}L=\frac{1}{6}\left[\begin{array}{ccc}1& 4& 1\\ 4& -20& 4\\ 1& 4& 1\end{array}\right].& \left(6\right)\end{array}$

11. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 6, wherein the sharpness evaluation value is calculated using a Laplacian function, and the Laplacian function is expressed as:
D(f)=Σ.sub.yΣ.sub.x|G(x,y)|  (5) wherein G(x, y) denotes convolution of a Laplacian operator at a pixel point (x, y), and the Laplacian operator is expressed as: $\begin{array}{cc}L=\frac{1}{6}\left[\begin{array}{ccc}1& 4& 1\\ 4& -20& 4\\ 1& 4& 1\end{array}\right].& \left(6\right)\end{array}$

12. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 7, wherein the sharpness evaluation value is calculated using a Laplacian function, and the Laplacian function is expressed as:
D(f)=Σ.sub.yΣ.sub.x|G(x,y)|  (5) wherein G(x, y) denotes convolution of a Laplacian operator at a pixel point (x, y), and the Laplacian operator is expressed as: $\begin{array}{cc}L=\frac{1}{6}\left[\begin{array}{ccc}1& 4& 1\\ 4& -20& 4\\ 1& 4& 1\end{array}\right].& \left(6\right)\end{array}$

13. The electro-hydraulic varifocal lens-based method for tracking a 3D trajectory of a moving object according to claim 8, wherein the sharpness evaluation value is calculated using a Laplacian function, and the Laplacian function is expressed as:
D(f)=Σ.sub.yΣ.sub.x|G(x,y)|  (5) wherein G(x, y) denotes convolution of a Laplacian operator at a pixel point (x, y), and the Laplacian operator is expressed as: $\begin{array}{cc}L=\frac{1}{6}\left[\begin{array}{ccc}1& 4& 1\\ 4& -20& 4\\ 1& 4& 1\end{array}\right].& \left(6\right)\end{array}$

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG. 1 is a flow chart according to Embodiment 1 of the present disclosure.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiment 1

[0033] The present disclosure provides an electro-hydraulic varifocal lens-based method for tracking a three-dimensional (3D) trajectory of a moving object, including:

[0034] step 1, calibrate, based on Zhang Zhengyou Calibration Method, the electro-hydraulic varifocal lens under different focal distances to obtain a functional relation between a focusing control current and camera's intrinsic parameters;

[0035] step 1 specifically includes calibrating the electro-hydraulic varifocal lens under multiple focusing control currents to obtain corresponding f.sub.x, f.sub.y, and obtaining a functional relation between a focusing control current and camera's intrinsic parameters by curve fitting:

(f.sub.x,f.sub.y)=H(I)  (1)

[0036] where f.sub.x and f.sub.y denote parameters in the camera's intrinsic parameters that change with the focal distance, and as physically defined as equivalent focal distances of a camera in x and Y directions of a pixel plane, in a unit of px; and I denotes a focusing control current of an electro-hydraulic varifocal lens.

[0037] In the meanwhile, obtain camera's intrinsic parameters c.sub.x, c.sub.y, s and distortion parameters k.sub.1, k.sub.2 (only radial distortion is considered in the Zhang Zhengyou Calibration Method) that do not change with the focal distance, where c.sub.x and c.sub.y are physically defined as the coordinates of a camera's optical center on the pixel plane, and s is physically defined as a slant parameter between the horizontal and vertical edges of a camera's photosensitive element, all of which are constants obtainable in calibration.

[0038] step 2, establish, by Zemax software, an electro-hydraulic varifocal lens-based optical imaging system model, and set the radius, thickness, curvature, material and other parameters of the electro-hydraulic varifocal lens used in Zemax software, so as to obtain a functional relation between a focusing control current of the electro-hydraulic varifocal lens and an optimal object distance;

[0039] step 2 specifically includes recording an optimal object distance under multiple focusing control currents by using the electro-hydraulic varifocal lens-based optical imaging system model constructed by the Zemax software, and conducting curve fitting on the recorded data to obtain a functional relation between focusing control currents of the electro-hydraulic varifocal lens and the optimal object distance:

u=F(I)  (2)

[0040] where u denotes an optimal object distance, and I denotes a focusing control current of the electro-hydraulic varifocal lens.

[0041] step 3, initialize an object tracking algorithm, and select a to-be-tracked object, where the object tracking box is taken as a subsequent focusing window; generally, the object tracking algorithm can be divided into: first, correlation filtering methods, such as CSK, KCF/DCF, CN, etc.; second, depth learning methods, such as C-COT, ECO and DLT; both methods can be selected in the present disclosure, and the KCF algorithm is selected in the present embodiment.

[0042] step 4, carry out first autofocusing to make a sharpness evaluation value in an object tracking box in an image greater than a preset threshold K, and record a focusing control current I.sub.i after the autofocusing is completed, as well as a size size.sub.i of the object tracking box in the image and center point coordinates (x.sub.i, y.sub.i) of the object tracking box after undistortion;

[0043] the autofocusing includes first autofocusing and subsequent autofocusing, and the first autofocusing specifically includes: (1) searching an initial focusing control current (focusing control current corresponding to a shortest or longest focal distance) at a certain stride t=(b−a)/1000, where a denotes a minimum focusing control current, and b denotes a maximum focusing control current; and calculating a sharpness evaluation value for an internal image region of the object tracking box, obtaining a maximum sharpness evaluation value D.sub.max and a focusing control current I.sub.1 corresponding to the maximum sharpness evaluation value, and setting a sharpness evaluation threshold:

K=αD.sub.max  (3)

[0044] where α denotes a preset sharpness confidence level (α<1); and K denotes a preset sharpness evaluation threshold used in the subsequent autofocusing.

[0045] The sharpness evaluation value is calculated by the sharpness evaluation function, and the sharpness evaluation function can be commonly used SMD function, EOG function, Roberts function, Tenengrad function, Brenner function, Laplacian function or SML function. For ease of understanding, the Laplacian function is selected for calculation in this embodiment, which is expressed as:

D(f)=Σ.sub.yΣ.sub.x|G(x,y)|  (5)

[0046] where G(x, y) denotes convolution of a Laplacian operator at a pixel point (x, y), and the Laplacian operator is expressed as:

[00004]$\begin{array}{cc}L=\frac{1}{6}\left[\begin{array}{ccc}1& 4& 1\\ 4& -20& 4\\ 1& 4& 1\end{array}\right]& \left(6\right)\end{array}$

[0047] (2) after autofocusing is finished, record a size size.sub.1 of the object tracking box in an image and center point coordinates (x.sub.1, y.sub.1) after undistortion (namely, center point coordinates of the object tracking box in an undistorted image).

[0048] The subsequent autofocusing specifically includes: calculating a sharpness evaluation value D.sub.i of the internal image region of the object tracking box; and if D.sub.i≥K, directly recording the focusing control current I.sub.i at this moment, as well as a size size.sub.i of the object tracking box in an image and center point coordinates (x.sub.i, y.sub.i) of the object tracking box after undistortion; or if D.sub.i<K, reading a size size.sub.i of the object tracking box in the image at this moment, comparing the size with a size size.sub.i-1 of the object tracking box at last successful focusing; if size.sub.i<size.sub.i-1, searching the focusing control current at a certain stride t in the direction where the optimal object distance becomes longer, calculating a sharpness evaluation value in the object tracking box, and completing focusing after the sharpness evaluation value is greater than or equal to the sharpness evaluation threshold; or if size.sub.i>size.sub.i-1, searching the focusing control current at a certain stride tin the direction where the optimal object distance becomes shorter, calculating a sharpness evaluation value in the object tracking box, and completing focusing after the sharpness evaluation value is greater than or equal to the sharpness evaluation threshold; and after the focusing is completed, recording the searched focusing control current I.sub.i and the size size.sub.i of the object tracking box in the image after focusing and center point coordinates (x.sub.i, y.sub.i) after undistortion.

[0049] The undistortion specifically includes: calculating, by a distortion model used in the selected calibration method, an undistorted image of a current frame and reading and recording center point coordinates (x.sub.i, y.sub.i) of the object tracking box in the undistorted image.

[0050] The radial distortion model used in the Zhang Zhengyou Calibration Method is:

x.sub.distorted=x(1+k.sub.1r.sup.2+k.sub.2r.sup.4)  (7)

y.sub.distorted=y(1+k.sub.1r.sup.2+k.sub.2r.sup.4)  (8)

r=√{square root over (x.sup.2+y.sup.2)}  (9)

[0051] where x.sub.distorted, y.sub.distorted denote pixel coordinates after distortion of an image, x and y are ideal pixel coordinates for undistortion, and k.sub.1, k.sub.2: denote distortion parameters obtained by calibration. Calculate, by the above distortion model, an undistorted image of a current frame, and read and record center point coordinates (x.sub.i, y.sub.i) of the object tracking box in the image after undistortion.

[0052] step 5, substitute a focusing control current value I.sub.i, corresponding intrinsic parameters f.sub.xi, f.sub.yi, c.sub.x, c.sub.y, s, distortion parameters k.sub.1 and k.sub.2, and an optimal object distance u and the center point coordinates (x.sub.i, y.sub.i) of the object tracking box after undistortion into a camera projection model, and calculate and record coordinates (X.sub.i, Y.sub.i, Z.sub.i) of the object in 3D space; where the camera projection model is:

[00005]$\begin{array}{cc}\left(\begin{array}{c}{x}_i\\ y_i\\ 1\end{array}\right)=\frac{1}{Z_i}\left(\begin{array}{ccc}{f}_{xi}& s& c_x\\ 0& f_{yi}& c_y\\ 0& 0& 1\end{array}\right)\left(\begin{array}{c}{X}_i\\ Y_i\\ Z_i\end{array}\right)& \left(4\right)\end{array}$

[0053] where (x.sub.i, y.sub.i) denote center point coordinates after undistortion, c.sub.x, c.sub.y, and s are camera's intrinsic parameters obtained during calibration and do not vary with the focal distance, f.sub.xi, f.sub.yi, denote equivalent focal distances of the camera in the x and y directions obtained by substituting a focusing control current I.sub.ti at this moment into the calibration formula (1), and (X.sub.i, Y.sub.i, Z.sub.i) denote 3D coordinates of a center point of the tracked object; and the 3D coordinates (X.sub.i, Y.sub.i, Z.sub.i) of the center point of the tracked object can be calculated by substituting center point coordinates (x.sub.i, y.sub.i) of the object tracking box after undistortion obtained in step 4, the camera's intrinsic parameters obtained during calibration and Z.sub.i=u.sub.i (u.sub.i denotes an optimal object distance by substituting the focusing control current I.sub.i at this moment into formula (2)) into the above camera projection model;

[0054] step 6, repeat steps 4-5 for the same tracked object, and sequentially connect the recorded coordinates of the object in 3D space into a trajectory (equivalent to a 3D trajectory of the tracked moving object).

[0055] The present disclosure does not require stereo vision equipment with complex structure and large size, and can track the 3D trajectory of the object simply using a single camera, which is less costly. According to the present disclosure, the 3D trajectory of the object can be tracked, in the meanwhile, the tracked object can be kept in focus in the image through autofocusing, which improves the stability of the object tracking algorithm used, marking a significant progress compared with the prior art.

Embodiment 2

[0056] Compared with Embodiment 1, step 4 in this embodiment includes: directly calling an undistortion function of OpenCV, introducing distortion parameters k.sub.1, k.sub.2: obtained through calibration, conducting undistorting on an image, calculating an undistorted image of a current frame, and reading and recording center point coordinates (x.sub.i, y.sub.i) of an object tracking box in the image after undistortion.