Method of integrating binocular stereo video scenes with maintaining time consistency

Abstract

A method of integrating binocular stereo video scenes with maintaining time consistency includes: propagating and extracting a contour of moving object of stereo video A; integrating and deformating of parallax between moving object and dynamic scene with time consistency; color blending of moving object and dynamic scene with time consistency where a method of median coordinate fusion is utilized. The method is simple and effective to utilize a small quantity of user interactions to successfully extract moving objects from stereo video which are same in time and as consistent as possible between left view and right view to develop multiple constraint conditions to guide the integration and deformation of parallax of moving object and dynamic scene and allow moving object to conform to the rules of perspective of dynamic scene. Moreover, the deformation result of moving object is smooth and consistent and can avoid effectively the occurrence of dithering phenomenon.

Claims

1. A method of integrating binocular stereo video scenes while maintaining temporal consistency, comprising: propagating and extracting contours of a moving object from a stereo video A, in which in response to a user selecting the contours of the moving object by applying an interactive method such as a brush, extracting the user selected contours of the moving object from a first left frame from stereo video A and inserting the user selected contours of moving object into a first left frame of a stereo video B and further automatically propagating a corresponding image of the moving object from a first right frame from the stereo video A to the first right frame from the stereo video B, the contours of the moving objects for the first left frame and the first right frame from the stereo video A are separately propagated to correspond to the contours of the moving object for all other frames for a left eye view and a right eye view; integrating and deforming of parallax of the contours of the moving object and a dynamic scene of the stereo video B with temporal consistency, in which a position to place the contours of the moving object from the stereo video A into the dynamic scene of the stereo video B is determined based on in response to the user specifying the position, and for each frame a calculated disparity map based on regions of the moving object of the stereo video A and a calculated disparity map based on the specified position of the stereo video B are integrated to produce an integrated disparity map and thereafter the integrated disparity map is used to deform the contours of the moving object in a time-consistent manner to allow it conform to the rules of perspective of the dynamic scene of the stereo video B; and color blending of the contours of the moving object and the dynamic scene of the stereo video B with temporal consistency utilizing a method of a median coordinate fusion to blend the colors of the contours of the moving object and the dynamic scene of the stereo video B.

2. The method of integrating binocular stereo video scene while maintaining temporal consistency according to claim 1, further comprising: the propagating and extracting of the contours of the moving object of stereo video A comprises the sub-steps as follows: calculating separately a scale-invariant feature transform (SIFT) flow field for each frame between the left eye view and the right eye view and the SIFT flow field between two consecutive frames of the left eye view and the right eye view for stereo video A; calculating SIFT feature matching point pairs for the first left frame of stereo video A and the first right frame and utilizing a method of Random Sample Consensus (RANSAC) to remove mismatching; in response to the user applying an interactive mode using the brush to select the contours of the moving object on the first left frame of stereo video A; using the SIFT matching points inside the contours of the moving object in separate calculating of the SIFT flow field and the SIFT matching points of the contours of the moving object defining mesh vertexes to triangulate the contours of the moving object and yielding a mesh of triangles of the contours of the moving object of the first left frame of stereo video A, then a method put forward by Luo et al. is utilized to propagate the mesh of the first left frame of stereo video A to the first right frame of stereo video A through the SIFT feature matching point pairs and the constraints of mesh distortion of the triangles; propagating the mesh of the contours of the moving object of the first left frame and the right frame of stereo video A to the contours of the moving object in each other frame of the stereo video A to complete the extraction of the contours of the moving object; a triangle mesh {T.sub.c.sup.t;t=L, R} being set to denote the collection of all of the triangles of the mesh of the contours of the moving object of current left frame and the current right frame and a vertex collection {v.sub.c.sup.t(i);t=R, L} being set to denote the collection of all of the mesh vertexes, then the propagation of the contours of the moving object solving the coordinates of all the mesh vertexes of the vertex collection {v.sub.n.sup.t(i);t=R, L} of the triangle mesh {T.sub.n.sup.t;t=L, R} of the contours of the moving object of a next left frame and a next right frame, propagating of contours of the moving object through minimization of an energy equation as follows;
E=.sub.1E.sub.1+.sub.2E.sub.2+.sub.3E.sub.3+.sub.4E.sub.4 wherein: E denotes overall energy, E.sub.1 denotes a constraint energy corresponding to inter-frame feature points, E.sub.2 denotes the constraint energy against triangle mesh distortion, E.sub.3 denotes the constraint energy against association relation between the meshes of the left eye view and the right eye view, E.sub.4 denotes the constraint energy against vertical aligning, .sub.1, .sub.2, .sub.3, .sub.4 denote the weights of each energy term; the energy equation comprises four energy terms which indicate separately four types of constraint conditions, wherein: E.sub.1 denotes the constraint energy corresponding to the inter-frame feature points, the mesh vertexes of current frames being propagated to the corresponding frame(s) through SIFT flow field and vertex(s) having high confidence levels being selected for propagation to satisfy the energy term: $E_1=\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^L\left(i\right)-v_n^L\left(s\left(i\right)\right).Math.}^2+\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^R\left(i\right)-v_n^R\left(s\left(i\right)\right).Math.}^2$ the energy term constrains each mesh vertex {v.sub.n.sup.t(i);t=R, L} of a next frame to be as close as possible to the position {v.sub.c.sup.t(s(i));t=R, L} of the next frame to which each vertex collection {v.sub.c.sup.t(i);t=R, L} of the current frame is propagated through SIFT flow field; n.sub.f is the number of reliable vertexes; the vertexes corresponding to the matching point pairs in the calculating separately of the SIFT flow field are preferred as the reliable points and the characteristics of SIFT features matching point pairs are also based to allow the user to select points of high confidence level for propagation; E.sub.2 is the constraint energy against triangle mesh distortion where the shape of the mesh should not change dramatically after propagated from previous frame; three vertexes (v.sub.i, v.sub.j, v.sub.k) for a mesh triangle and the vertex v.sub.i is expressed linearly by other two vertexes as v.sub.j, v.sub.k as v.sub.i=F.sub.s(v.sub.j,v.sub.k), the energy to constrain triangle mesh distortion is: $E_2=\underset{\left(v_n^L\left(i\right),v_n^L\left(j\right),v_n^L\left(k\right)\right)T_n^L}{.Math.}{.Math.v_n^L\left(i\right)-F_S^L\left(v_n^L\left(j\right),v_n^L\left(k\right)\right).Math.}^2+\underset{\left(v_n^R\left(i\right),v_n^R\left(j\right),v_n^R\left(k\right)\right)T_n^R}{.Math.}{.Math.v_n^R\left(i\right)-F_S^R\left(v_n^R\left(j\right),v_n^R\left(k\right)\right).Math.}^2$ F.sub.S.sup.L, F.sub.S.sup.R separately denotes the linear relation between two vertexes of two corresponding triangles of left eye view and the right eye view, the energy term constrains the shape of each triangle in the triangle mesh of next frame to be as close as possible to the shape of corresponding triangle of current frame; E.sub.3 is the constraint energy against an association relation between the meshes of left eye view and the right eye view where the left eye view and the right eye view have a corresponding association relation for a binocular stereo video, such that if the contours of the moving object of left eye view or the right eye view is independently propagated downward, an inconsistency may possibly occur in correspondence between the meshes of left eye view and the right eye view and further cause an inconsistency between the contours of the moving object by the left eye view and the contours of the moving object by the right eye view, and in order to maintain the association relation between the left eye view and the right eye view and guarantee consistency of the contents of the contours of the moving object between by the left eye view and the right eye view, a constraint is needed for the association relation between the left eye view and the right eye view: $E_3=\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^L\left(i\right)-v_n^{R.fwdarw.L}\left(s\left(i\right)\right).Math.}^2+\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^R\left(i\right)-v_n^{L.fwdarw.R}\left(s\left(i\right)\right).Math.}^2$ because the mesh positions of the next frame by the left eye view and the right eye view are unknown yet, so the association relation between them is not established directly, thus, the positions yielded after twice transmission of the coordinates of vertex positions of the meshes of the current frame are set as the direction for constraint optimization, for example, as for the constraint of association relation from the mesh of the left eye view to the mesh of the right eye view, the reliable vertex v.sub.c.sup.L(i) of the mesh of current frame in the left eye view is firstly propagated to the position s(i) of next frame in the left eye view through the SIFT flow field between current frame and next frame in the left eye view and then the s(i) is propagated to the corresponding position v.sub.n.sup.R.fwdarw.L(s(i)) in the right eye view through the SIFT flow field between next frame in the left eye view and next frame in the right eye view, the vertex position v.sub.n.sup.R(i) of next frame in the right eye view and the calculated vertex position v.sub.n.sup.R.fwdarw.L(s(i)) should be as close as possible, similarly, the association relation from the mesh in the right eye view to the mesh in the left eye view is also constrained; E.sub.4 is the constraint energy against vertical aligning, for a binocular stereo image, the corresponding contents in the left eye view and the right eye view are same in the vertically direction of image space, thus the coordinates of y to constrain the corresponding vertexes of the meshes in the left eye view and in the right eye view should be constrained to be similar as much as possible, the energy term here is as follows: $E_4=\underset{i=1}{\overset{n}{.Math.}}{\left(v_n^L\left(i\right)\left[y\right]-v_n^R\left(i\right)\left[y\right]\right)}^2$ these four energy terms constitute together the combined energy constraint against the contour propagation of the contours of the moving object and .sub.1=.sub.3=.sub.4=10, .sub.2=1 are set as a reference weight of each energy term.

3. The method of integrating binocular stereo video scenes while maintaining temporal consistency according to claim 1, further comprising: the integrating and deforming of parallax of the contours of the moving object and the dynamic scene with temporal consistency comprises the sub-steps as follows: calculating the disparity map of each frame of the stereo video A and the stereo video B; utilizing Poisson integration to merge the disparity map in the area corresponding to the contours of the moving object in the stereo video A and the disparity map of the stereo video B to yield an integration disparity map sequence which is denoted as D={d.sub.1, d.sub.2, d.sub.3 . . . }; utilizing a method based on perspective perception deformation to carry out the parallax integration and deformation whose result is then set as the initial solution; adjusting and deforming automatically the parallax with temporal consistency for the contours of the moving object and the dynamic scenes of all other frames; assuming the result of deformation of the contours of the moving objects of frame i is known, then V.sub.i.sup.L,V.sub.i.sup.R is separately set as the vertex collection of the triangle mesh corresponding to the contours of the moving object of frame i in the left eye view and in the right eye view, the result of deformation of the contours of the moving object of frame i+1 and frame i+2 is realized through minimization of the following energy equation:
N=.sub.1N.sub.1+.sub.2N.sub.2+.sub.3N.sub.3 wherein: N denotes the overall energy, N.sub.1 denotes the constraint energy for corresponding parallax, N.sub.2 denotes the positioning constraint energy, N.sub.3 denotes the constraint energy against mesh distortion of the contours of the moving object, and .sub.1, .sub.2, .sub.3 denote the weight of each energy term; the energy equation includes three energy terms which separately denote three types of constraint conditions, where N.sub.1 is the constraint energy for parallax corresponding to constrain that the parallax of each vertex of the mesh of the contours of the moving object to be consistent with the parallax of the corresponding position of integrated disparity map; $N_1=\underset{v_i^L{V}_{i+1}^L,v_i^R{V}_{i+1}^R,v_o^L{\tilde{V}}_{i+1}^L}{.Math.}{\left(v_t^L\left[x\right]-v_t^R\left[x\right]-d_{i+1}\left(v_o^L\right)\right)}^2+\underset{v_i^L{V}_{i+2}^L,v_i^R{V}_{i+2}^R,v_o^L{\tilde{V}}_{i+2}^L}{\overset{}{.Math.}}{\left(v_t^L\left[x\right]-v_t^R\left[x\right]-d_{i+2}\left(v_o^L\right)\right)}^2;$ V.sub.i+1.sup.L, V.sub.i+1.sup.R separately denotes the mesh vertex collection after deformation of the contours of the moving object in frame i for the left eye view and the right eye view; {circumflex over (V)}.sub.i+1.sup.L denotes the mesh vertex collection before moving object deformation of frame i+1 in the left eye view and the symbol representation of frame i+2 is same as that of frame i+1, and v.sub.t.sup.L[x], v.sub.t.sup.R[x] denote the coordinates of x of the corresponding vertex; N.sub.2 is the positioning constraint energy to constrain that the position of geometric center of the contours of the moving object in the left eye view is the same as much as possible as the geometric center of the contours of the moving object in the left eye view before deformation; $N_2={.Math.c_{i+1}^l-\frac{1}{.Math.V^L.Math.}\underset{v_t^L{V}_{i+1}^L}{.Math.}{v}_t^L.Math.}^2+{.Math.c_{i+2}^l-\frac{1}{.Math.V^L.Math.}\underset{v_t^L{V}_{i+2}^L}{.Math.}{v}_t^L.Math.}^2$ c.sub.i+1.sup.l, c.sub.i+2.sup.l are separately the mesh geometry center before the moving object deformation of frame i+1 and frame i+2 in the left eye view; N.sub.3 is the constraint energy against mesh distortion of the contours of the moving object to constrain the deformation of previous and next frames as well as the left and right frames not to be too violent; $N_3=\underset{\left(v_{i+1}^L\left(t\right),v_{i+1}^L\left(j\right),v_{i+1}^L\left(k\right)\right)T_{i+1}^L}{.Math.}{}_{i+1}^L{.Math.v_{i+1}^L\left(t\right)-F^L\left(v_{i+1}^L\left(j\right),v_{i+1}^L\left(k\right)\right).Math.}^2+\underset{\left(v_{i+1}^R\left(t\right),v_{i+1}^R\left(j\right),v_{i+1}^R\left(k\right)\right)T_{i+1}^R}{.Math.}{}_{i+1}^R{.Math.v_{i+1}^R\left(t\right)-F^R\left(v_{i+1}^R\left(j\right),v_{i+1}^R\left(k\right)\right).Math.}^2+\underset{\left(v_{i+2}^L\left(t\right),v_{i+2}^L\left(j\right),v_{i+2}^L\left(k\right)\right)T_{i+2}^L}{.Math.}{}_{i+2}^L{.Math.v_{i+2}^L\left(t\right)-F^L\left(v_{i+2}^L\left(j\right),v_{i+2}^L\left(k\right)\right).Math.}^2+\underset{\left(v_{i+2}^R\left(t\right),v_{i+2}^R\left(j\right),v_{i+2}^R\left(k\right)\right)T_{i+2}^R}{.Math.}{}_{i+2}^R{.Math.v_{i+2}^R\left(t\right)-F^R\left(v_{i+2}^R\left(j\right),v_{i+2}^R\left(k\right)\right).Math.}^2$ T.sub.i+1.sup.L, T.sub.i+1.sup.R are the triangle collection of the triangle mesh of the contours of the moving object of frame i+1, respectively, in the left eye view and the right eye view, T.sub.i+2.sup.L, T.sub.i+2.sup.R is the triangle collection of the triangle mesh of the contours of the moving object of frame i+2 in the left eye view and the right eye view, v(t), v(j), v(k) are separately the three vertexes of the corresponding triangle, F.sup.L, F.sup.R are separately the linear relation between different vertexes of each triangle of the mesh of the contours of the moving object of frame i in the left eye view and the right eye view, .sub.i+1 is the smoothing factor which is calculated with e.sup.F(v.sup.i+1.sup.(j), v.sup.i+1.sup.(k))F(v.sup.i.sup.(j),v.sup.i.sup.(k)).sup.2 and .sub.i+2 is calculated in the same way as .sub.i+1; the energy terms constitute together the combined energy constraint against the moving object deformation with temporal consistency and .sub.1=50, and .sub.2=.sub.3=1 are reference weights of each energy term; and after the adjusting and deforming the parallax, two solutions are generated for the coordinate of each vertex of the mesh corresponding to the contours of the moving object of each frame of the left eye view and the right eye view, the average value of the two solutions is calculated as the final result of deformation.

4. The method of integrating binocular stereo video scenes while maintaining temporal consistency according to claim 1, further comprising: the color blending of the contours of the moving object and dynamic scene with temporal consistency is implemented through: utilizing a method of median coordinate fusion to calculate the internal vertexes of the contours of the moving object mesh against the median coordinate of the boundary vertexes of mesh; utilizing the color difference between the pixel corresponding to the median coordinates and the boundary points of the contours of the moving object and the pixel of the position corresponding to dynamic scene to calculate the color difference of the internal vertexes of the contours of the moving object; the color difference of the pixel points of non-vertex point(s) is obtained through linear interpolation of the color difference among three vertexes of the triangle mesh patch where the pixels of non-vertex lie in; add the color difference and the color value of the original pixel of the contours of the moving object to yield the result of color blending.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

(1) FIG. 1 is the flow chart of the present invention;

(2) FIG. 2 is the schematic diagram of association relation between the propagation object of the contour of moving object according to the present invention;

(3) FIG. 3 is the schematic diagram of association relation between the objects of parallax integration deformation of moving object and dynamic scene according to the present invention;

EMBODIMENT

(4) Now the present invention is further described in detail with the drawings as follows.

(5) FIG. 1 is the flow chart of the present invention, where the method according the present invention comprises three steps:

(6) Step S100-propagation and extraction of contour of moving object of stereo video A: the user applies an interactive method such as a brush to select the contour of a moving object of the first frame of stereo video A by left eye. It is automatically propagated to a corresponding moving object of the first frame by right eye. Referring to this, the contours of moving objects of the first frames of stereo video A by left eye/right eye are separately propagated to corresponding moving objects of each other frame by left eye/right eye to extract moving object(s) of stereo video A.

(7) Step S200-Integration and deformation of parallax of moving object and dynamic scene with time consistency: the user specifies the position to place moving object of stereo video A into dynamic scene of stereo video B. The disparity map of the moving object region of stereo video A and that of stereo video B are integrated and then the result of integration is based to deform the moving object in a time-consistent manner to allow it conform to the rules of perspective of dynamic scene.

(8) Step S300-Color blending of moving object and dynamic scene with time consistency: utilize a method of median coordinate fusion to blend the colors of moving object and dynamic scene.

(9) The propagation and extraction of moving object of stereo video A described in Step S100 comprises the sub-steps as follows:

(10) Step S110-Calculate separately SIFT flow field of each frame between left eye and right eye and SIFT flow field between two consecutive frames by left eye/right eye for stereo video A. Calculate SIFT feature matching point pairs for the first frame of stereo video A by left eye and the first frame by right eye and utilize the method of RANSAC to remove the mismatching.

(11) Step S120-User applies an interactive mode such as brush to select a rough contour of moving object on the first frame stereo video A by left eye. Use SIFT matching points inside the contour of moving object in Step S110 and the points of sampling the contour of moving object as mesh vertexes to triangulate the moving object and yield the mesh of moving object of the first frame of stereo video A by left eye. Then a method put forward by Luo et al. is utilized to propagate the mesh from the first frame of stereo video A by left eye to the first frame of stereo video A by right eye through SIFT feature point matching and the constraints of triangle mesh distortion.

(12) Step S130-Propagate the mesh of moving object of the first frame of stereo video A by left eye/right eye to the moving object of each other frame of stereo video A to complete the extraction of moving object.

(13) Set {T.sub.c.sup.t;t=L, R} to denote the collection of all the triangles of the triangle mesh of the known moving object of current frame by left eye/right eye and set {v.sub.c.sup.t(i);t=R, L} to denote the collection of all the vertexes of current corresponding mesh. Then the propagation of the contour of moving object is to solve the coordinates of all the vertexes of the vertex collection {v.sub.n.sup.t(i);t=R, L} of the triangle mesh {T.sub.n.sup.t;t=L, R} of moving object of next frame by left eye/right eye. The present invention realizes the propagation of contour of moving object through minimization of the energy equation as follows.
E=.sub.1E.sub.1+.sub.2E.sub.2+.sub.3E.sub.3+.sub.4E.sub.4

(14) The energy equation comprises four energy terms which indicate separately four types of constraint conditions. Wherein: E.sub.1 denotes the constraint energy corresponding to the inter-frame feature points. The mesh vertexes of current frame can be propagated to the corresponding frame(s) through SIFT flow and the present invention selects the vertex(s) of high confidence level for propagation. They satisfy the energy term:

(15) $E_1=\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^L\left(i\right)-v_n^L\left(s\left(i\right)\right).Math.}^2+\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^R\left(i\right)-v_n^R\left(s\left(i\right)\right).Math.}^2$

(16) This energy term will constrain each mesh vertex {v.sub.n.sup.t(i);t=R, L} of next frame to be as close as possible to the position {v.sub.c.sup.t(s(i));t=R, L} of next frame to which each vertex {v.sub.c.sup.t(i);t=R, L} of current frame is propagated through SIFT flow. n.sub.f is the number of reliable vertexes. The present invention selects preferably the vertexes corresponding to the matching point pairs in Step S110 as the reliable points or the characteristics of SIFT features are based to allow the user to select the points of high confidence level for propagation.

(17) E.sub.2 is the constraint energy against triangle mesh distortion where the shape of mesh should not change dramatically after propagated from previous frame. There are three vertexes (v.sub.i, v.sub.j, v.sub.k) for a mesh triangle. The vertex v.sub.i is expressed linearly by other two vertexes v.sub.j,v.sub.k as v.sub.i=F.sub.S(v.sub.j, v.sub.k). The energy to constrain triangle mesh distortion is:

(18) $E_2=\underset{\left(v_n^L\left(i\right),v_n^L\left(j\right),v_n^L\left(k\right)\right)T_n^L}{.Math.}{.Math.v_n^L\left(i\right)-F_S^L\left(v_n^L\left(j\right),v_n^L\left(k\right)\right).Math.}^2+\underset{\left(v_n^R\left(i\right),v_n^R\left(j\right),v_n^R\left(k\right)\right)T_n^R}{.Math.}{.Math.v_n^R\left(i\right)-F_S^R\left(v_n^R\left(j\right),v_n^R\left(k\right)\right).Math.}^2$

(19) F.sub.S.sup.L, F.sub.S.sup.R separately denotes the linear relation between two vertexes of two corresponding triangles by left eye and right eye. This energy term constrains the shape of each triangle in the triangle mesh of next frame to be as close as possible to the shape of corresponding triangle of current frame.

(20) E.sup.3 is the constraint energy against an association relation between the meshes of left eye/right eye which is shown as FIG. 2. Left eye and right eye have a corresponding association relation for a binocular stereo video. If the contour of moving object of left eye or right eye is independently propagated downward, an inconsistency may possibly occur the corresponding between the meshes of left eye and right eye and further cause an inconsistency between moving object by left eye and moving object by right eye. In order to maintain the association relation between left eye and right eye and guarantee consistency of the contents of moving object between by left eye and by right eye, a constraint is needed for the association relation between left eye and right eye:

(21) 0$E_3=\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^L\left(i\right)-v_n^{R.fwdarw.L}\left(s\left(i\right)\right).Math.}^2+\underset{i=1}{\overset{n_f}{.Math.}}{.Math.v_n^R\left(i\right)-v_n^{L.fwdarw.R}\left(s\left(i\right)\right).Math.}^2$

(22) Because the mesh positions of next frame by left eye and by right eye are unknown yet, so the association relation between them can't be established directly.

(23) Thus, the present invention sets the positions yielded after twice transmission of the coordinates of vertex positions of the meshes of current frame as the direction for constraint optimization. For example, as for the constraint of association relation from the mesh of left eye to the mesh of right eye, the reliable vertex of the mesh of current frame by left eye is firstly propagated to the position of next frame by left eye through the SIFT flow between current frame and next frame by left eye and then the is propagated to the corresponding position by right eye through the SIFT flow between next frame by left eye and next frame by right eye. The present invention constrains the vertex position of next frame by right eye to be as close as possible to the calculated vertex position. Similarly; the association relation from the mesh by right eye to the mesh by left eye is also constrained.

(24) E.sup.4 is the constraint energy against vertical aligning. For a binocular stereo image, the corresponding contents in left eye and right eye are same in the vertically direction of image space. Thus, the present invention constrains the coordinates of y to be similar as much as possible to the corresponding vertexes of the meshes by left eye and by right eye. The energy term here is as follows:

(25) $E_4=\underset{i=1}{\overset{n}{.Math.}}{\left(v_n^L\left(i\right)\left[y\right]-v_n^R\left(i\right)\left[y\right]\right)}^2$

(26) These four energy terms constitute together the combined energy constraint against the contour propagation of moving object and we use .sub.t=.sub.3=.sub.4=10, .sub.2=1 as the reference weight of each energy term.

(27) FIG. 2 is the schematic diagram of association relation between the propagation object of the contour of moving object according to the present invention. It denotes the constraint of association relation of the mesh of moving object by left eye/right eye.

(28) Wherein: the integration and deformation of parallax of moving object and dynamic scene with time consistency described in Step S200 comprises the substeps as follows:

(29) S210-Calculate the disparity map of each frame of stereo video A and stereo video B. Utilize Poisson integration and other integration methods to merge the disparity map in the area corresponding to moving object in stereo video A and the disparity map of stereo video B to yield an integration disparity map sequence which can be denoted as D={d.sub.1, d.sub.2, d.sub.3 . . . }.

(30) Step S220-Utilize the method based on perspective perception deformation put forward by Luo et al. to carry out the parallax integration and deformation whose result is then set as the initial solution.

(31) Step S230-Adjust and deform automatically the parallax with time consistency for the moving objects and the dynamic scenes of all other frames. Assuming the result of deformation of moving objects of frame i is known. V.sub.i.sup.L, V.sub.i.sup.R is separately set as the vertex collection of the triangle mesh corresponding to moving object of frame i by left eye and by right eye. The deformation result of moving object of frame i+1 and frame i+2 can be realized through minimization of the following energy equation:
N=.sub.1N.sub.1+.sub.2N.sub.2+.sub.3N.sub.3

(32) Wherein: N denotes the overall energy, N.sub.1 denotes the constraint energy for parallax corresponding, N.sub.2 denotes the positioning constraint energy, N.sub.3 denotes the constraint energy against mesh distortion of moving object. .sub.1, .sub.2, .sub.3 denote the weight of each energy term.

(33) The energy equation includes three energy terms which separately denote three types of constraint conditions. Wherein: N.sub.1 is the constraint energy for parallax corresponding to constrain that the parallax of each vertex of the mesh of moving object to be consistent with the parallax of the corresponding position of integrated disparity map.

(34) $N_1=\underset{v_i^L{V}_{i+1}^L,v_i^R{V}_{i+1}^R,v_o^L{\tilde{V}}_{i+1}^L}{.Math.}{\left(v_t^L\left[x\right]-v_t^R\left[x\right]-d_{i+1}\left(v_o^L\right)\right)}^2+\underset{v_i^L{V}_{i+2}^L,v_i^R{V}_{i+2}^R,v_o^L{\tilde{V}}_{i+2}^L}{\overset{}{.Math.}}{\left(v_t^L\left[x\right]-v_t^R\left[x\right]-d_{i+2}\left(v_o^L\right)\right)}^2$

(35) V.sub.i+1.sup.L, V.sub.i+1.sup.R separately denotes the mesh vertex collection after moving object deformation of frame i by left eye/right eye; {circumflex over (V)}.sub.i+1.sup.L denotes the mesh vertex collection before moving object deformation of frame i+1 by left eye. The symbol representation of frame i+2 is same as that of frame i+1. v.sub.t.sup.L[x], v.sub.t.sup.R[x] denote the coordinates of x of the corresponding vertex.

(36) N.sub.2 is the positioning constraint energy to constrain that the position of geometry center of moving object by left eye to be same as much as possible with the geometry center of moving object by left eye before deformation.

(37) $N_2={.Math.c_{i+1}^l-\frac{1}{.Math.V^L.Math.}\underset{v_t^L{V}_{i+1}^L}{.Math.}{v}_t^L.Math.}^2+{.Math.c_{i+2}^l-\frac{1}{.Math.V^L.Math.}\underset{v_t^L{V}_{i+2}^L}{.Math.}{v}_t^L.Math.}^2$

(38) c.sub.i+1.sup.l, c.sub.i+2.sup.l are separately the mesh geometry center before moving object deformation of frame i+1 and frame i+2 by left eye.

(39) N.sub.3 is the constraint energy against mesh distortion of moving object to constrain the deformation of previous and next frames as well as left and right frames not to be violent too much.

(40) $N_3=\underset{\left(v_{i+1}^L\left(t\right),v_{i+1}^L\left(j\right),v_{i+1}^L\left(k\right)\right)T_{i+1}^L}{.Math.}{}_{i+1}^L{.Math.v_{i+1}^L\left(t\right)-F^L\left(v_{i+1}^L\left(j\right),v_{i+1}^L\left(k\right)\right).Math.}^2+\underset{\left(v_{i+1}^R\left(t\right),v_{i+1}^R\left(j\right),v_{i+1}^R\left(k\right)\right)T_{i+1}^R}{.Math.}{}_{i+1}^R{.Math.v_{i+1}^R\left(t\right)-F^R\left(v_{i+1}^R\left(j\right),v_{i+1}^R\left(k\right)\right).Math.}^2+\underset{\left(v_{i+2}^L\left(t\right),v_{i+2}^L\left(j\right),v_{i+2}^L\left(k\right)\right)T_{i+2}^L}{.Math.}{}_{i+2}^L{.Math.v_{i+2}^L\left(t\right)-F^L\left(v_{i+2}^L\left(j\right),v_{i+2}^L\left(k\right)\right).Math.}^2+\underset{\left(v_{i+2}^R\left(t\right),v_{i+2}^R\left(j\right),v_{i+2}^R\left(k\right)\right)T_{i+2}^R}{.Math.}{}_{i+2}^R{.Math.v_{i+2}^R\left(t\right)-F^R\left(v_{i+2}^R\left(j\right),v_{i+2}^R\left(k\right)\right).Math.}^2$

(41) T.sub.i+1.sup.L, T.sub.1+1.sup.R are separately the triangle collection of the triangle mesh of moving object of frame i+1 by left eye/right eye. Similarly, T.sub.i+2.sup.L, T.sub.i+2.sup.R is the triangle collection of the triangle mesh of moving object of frame i+2 by left eye/right eye. v(t), v(j), v(k) are separately the three vertexes of the corresponding triangle. F.sup.L, F.sup.R are separately the linear relation between different vertexes of each triangle of the mesh of moving object of frame i by left eye/right eye. .sub.i+1 is the smoothing factor which is calculated with e.sup.F(v.sup.i+1.sup.(j), v.sup.i+1.sup.(k))F(v.sup.i.sup.(j),v.sup.i.sup.(k)).sup.2 and .sub.i+2 is calculated in the same way.

(42) These three energy terms constitute together the combined energy constraint against the moving object deformation with time consistency. We use .sub.1=50, .sub.2=.sub.3=1 as the reference weight of each energy term.

(43) Step S240-After Step S230, two solutions will be generated for the coordinate of each vertex of the mesh corresponding to the moving object of each frame by left eye/right eye. Then the average value of them is calculated as the final result of deformation.

(44) FIG. 3 is the schematic diagram of association relation between the objects of parallax integration deformation of moving object and dynamic scene according to the present invention.

(45) In conclusion, the present invention discloses a method of integrating binocular stereo video scene with maintaining time consistency which can integrate moving object of stereo video A into dynamic scene of stereo video B and maintain the consistency of color, parallax and time. This method comprises three steps: to utilize an algorithm of SIFT feature matching to automatically propagate the contour of a designated moving object of the first frame of stereo video A by left eye to the corresponding moving object of the first frame of stereo video A by right eye, then to establish a relation of feature matching between moving object of stereo video A by left eye and moving object of stereo video A by right eye, and separately propagate the contour of moving object of the first frame of stereo video A by left eye/right eye to the corresponding moving object of each other frame by left eye/right eye under the combined constraint of SIFT flow and the deformation of the object contour by left eye/right eye to extract the moving object of stereo video A; then place moving object of stereo video A in dynamic scene of stereo video B to carry out the parallax integration deformation between moving object and dynamic scene with maintaining time consistency; finally, utilize a method of median coordinate integration to realize the color blending with time consistency between moving object of stereo video A and dynamic scene of stereo video B. The present invention develops two points of innovations. The first is the method of contour propagation and extraction of moving object from stereo video; the second is the method of integrating color and parallax with time consistency for moving object and dynamic scene. So the present invention can be applied in the fields of video image processing and film and television production, etc.