METHOD FOR THE TRANSFORMATION OF A MOVING IMAGE SEQUENCE AND MOVING IMAGE SEQUENCE TRANSFORMATION DEVICE

Abstract

A method is provided for the transformation of a moving image sequence and a moving image sequence transformation device designed for executing the transformation method. For a current individual image of the moving image sequence, the method forms a transformation basis (1) in which a first individual image, the current individual image, and a second individual image are arranged adjacent to each other. Intersection points (S1, S2) are determined for connecting lines (7, 8) which extend from image points corresponding to each other from the first individual image and the current individual image and from the current individual image to the second individual image, comprising image starting limits and image end limits (15, 16) of the current individual image. A new image point position of an image point in the current individual image results from averaging the intersection points.

Claims

1. Method for the transformation of a moving image sequence, in particular for rectification and stabilization of images, comprising the steps of: providing, in a moving image sequence storage, a moving image sequence with a plurality of individual images (B.sub.−1, B.sub.0, B.sub.1) successive in time and assigned to time intervals (ΔT), which are set by a respective start time (t.sub.-1, t.sub.0, t.sub.1) and end time (t.sub.0, t.sub.1, t.sub.2), wherein each individual image is composed of a plurality (n) of lines (3) that comprise image points, which are in each case assigned to a time (0, Tz, 2Tz) different to one another within the time interval assigned to the individual image, wherein an image point position (x, y) in an image point plane having a first image point dimension (x) and a second image point dimension (y) is assigned to each image point; providing, in a moving image information storage, a moving image information (Δ(B.sub.−1, B.sub.0), Δ(B.sub.0, B.sub.1), which describes a motion flow between individual images of the moving image sequence; transforming the moving image sequence by repeating the steps, for the successive individual images, of: forming a transformation base (1) assigned to a current individual image (B.sub.0) to be treated and having a moving image sequence time axis (21) as a first dimensional axis and a line succession time axis (22) as a second dimensional axis orthogonal to the first dimensional axis by arranging the current individual image (B.sub.0), a first individual image (B.sub.−1), which is different from the current individual image, and a second individual image (B.sub.1), which is different from the current image and from the first individual image, in a chronological order of the respective start times (t.sub.−1, t.sub.0, t.sub.1) and end times (t.sub.0, t.sub.1, t.sub.2) of the individual images (B.sub.−1, B.sub.0, B.sub.1)along the first dimensional axis (21), wherein the lines (3) of these individual images (B.sub.−1, B.sub.0, B.sub.1)assigned to the same relative points of time are arranged next to one another along the second dimension (22); displacing at least one image point (P) of the current individual image (B.sub.0) from a current image point position (xP, yP) to a new image point position (xQ, yQ) by executing, for the respective image point to be displaced, the steps of: determining, for a current image point (P) to be displaced of the current individual image (B.sub.0), a first image point position (xPL, yPL) in the first individual image (B.sub.−1) assigned to the current image point based upon the image motion information (Δ(B.sub.−1, B.sub.0)); determining, for the current image point (P) of the current individual image (B.sub.0), a second image point position (xPR, yPR) in the second individual image (B.sub.1) assigned to the current image based upon the image motion information (ΔB.sub.0, B.sub.−1)); identifying a first interpolation point (S1) in the transformation base (1) assigned to the start time (t.sub.0) of the current individual image based upon the determined first image point position (xPL, yPL) and the current image point position (xP, yP); identifying a second interpolation point (S2) in the transformation base (1) assigned to the end time (t.sub.1) of the current individual image based upon the current image point position (xP, yP) and the determined second image point position (xPR, yPR); determining a new image point position (xQ, yQ) of the current image point (P) of the current individual image (B.sub.0) based upon an average value of the first and the second interpolation point (S1, S2); and detecting an assignment of the current image point (P) and the determined new image point position (xQ, yQ) in a displacement image storage and/or transformation moving image sequence storage.

2. Method according to claim 1, wherein the first interpolation point (S1) is identified based upon a first line (7) running through the first image point position (xPL, yPL) and the current image point position (xP, yP) in the transformation base (1), and/or wherein the second interpolation point (S2) is identified based upon a second line (8) running through the current image point position (xP, yP) and the second image point position (xPR, yPR) in the transformation base (1).

3. Method according to claim 2, wherein the first line (7) and/or the second line (8) are determined by a respective linear, quadratic, or cubic equation.

4. Method according to claim 2, wherein the first interpolation point (S1) is identified as an intersection point of the first line (7) with a start time axis (15) assigned to the start time (t.sub.0) of the current individual image (B.sub.0) and extending in the transformation base along the second dimension, and/or wherein the second interpolation point (S2) is identified as an intersection point of the second line (8) with an end time axis (16) assigned to the end time (t.sub.1) of the current individual image (B.sub.0) and extending in the transformation base along the second dimension.

5. Method according to claim 1, wherein the new image point position (yQ) of the current image point for the second dimension (y) of the image point plane is determined as an average value of the positions of the first (yS1) and second (yS2) intersection point in the second dimension of the transformation base.

6. Method according to claim 1, wherein the new image point position (xQ) of the current image point for the first dimension (x) of the image point plane is determined as an average value of the positions of the first (xS1) and second (xS2) intersection point in a third dimension of the transformation base.

7. Method according to claim 1, wherein determining the new image point position of the current image point is effected by an identification, in particular calculation, of a first correction offset (Δx) in the first dimension of the image point plane, which is identified based upon the first and second image point position each in the first and second dimension of the image point plane, the actual image point position only in the second dimension of the image point plane, and/or by an identification, in particular calculation, of a second correction offset (Δy) in the second dimension of the image point plane, which is identified based upon the first and second image point position in the second dimension only of the image point plane and the current image point position in the second dimension only of the image point plane.

8. Method according to claim 7, wherein identifying the first (Δx) and/or the second (Δy) correction offset is effected further based upon a predetermined speed (v), and/or a predetermined line dimension size (h).

9. Method according to claim 1, wherein the transformation base (1) comprises a third dimension (x), which is orthogonal to the first (t) and second (y) dimension of the transformation base, and wherein the image point planes of the individual images are arranged in the plane orthogonal to the first dimension (t) of the transformation base and thus along the second (y) and third dimension of the transformation base, and in particular the second image point dimension and the second dimension of the transformation base coincide.

10. Method according to claim 1, wherein the image motion information (Δ (B.sub.−1, B.sub.0)) describes a change of a first individual section position assigned to an individual image section in an individual image of the moving image sequence in relation to a second individual section position assigned to the same individual image section in an individual image next in terms of time to the individual image.

11. Method according to claim 1, wherein the first individual image (B.sub.−1) is spaced from the current individual image (B.sub.0) following in the moving image sequence by a first individual image distance of three, individual images, and the second individual image (B.sub.1) is spaced from the current individual image (B.sub.0) preceding in the moving image sequence by a second individual image distance of three individual images.

12. Application method for image rectification and image stabilization of a moving image sequence recorded with a CMOS sensor, comprising an application of the method according to claim 1

13. Moving image sequence transformation device, which is configured for executing the method according to claim 1.

14. Chip with a hardware architecture which is configured for executing the method according to claim 1.

15. Camera unit with a CMOS sensor, which is configured for image rectification and image stabilization, after completion of a recording of a moving image sequence or in real time while recording, of the moving image sequence recorded with the CMOS sensor by applying the method according to claim 1 to the recorded moving image sequence.

16. Computer program product, in particular recorded on a data carrier, including a program code for execution of the method according to claim 1 on a computer executing the program code.

17. Method according to claim 1, wherein the first individual image (B.sub.−1) is located ahead in time from the current individual image, and wherein the second individual image (B.sub.1) is successive in time from the current individual image.

18. Method according to claim 8, wherein the predetermined speed (v) is an individual image detection speed or a row detection speed, of a CMOS sensor having detected the moving image sequence to be transformed, and wherein the predetermined line dimension size (h) is of the CMOS sensor having detected the moving image sequence to be transformed.

19. Method according to claim 10, wherein the first individual section position is an image point, and wherein the change is based upon a vector information.

20. Method according to claim 11, wherein the first individual image distance is exactly one individual image, and wherein second individual image distance is exactly one individual image.

Description

BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Exemplary embodiments of the invention will be explained by means of the attached figures, in which:

[0042] FIG. 1 shows an illustration of a transformation base formed according to the invention with a first individual image, a current individual image and a second individual image;

[0043] FIG. 2 shows a section of the transformation base according to FIG. 1 used for the determination of the new image point position;

[0044] FIG. 3 shows a flow diagram, which schematically illustrates the transformation of a moving image sequence with the method according to the invention;

[0045] FIG. 4 is a flow diagram, which illustrates an embodiment of the step of displacing an image point in greater detail mentioned in FIG. 3;

[0046] FIG. 5 is a flow diagram, which illustrates another embodiment of the step of displacing an image point in greater detail mentioned in FIG. 3.

DETAILED DESCRIPTION

[0047] Throughout the drawings, like reference numerals indicate like or similar components. A repeated, figure-related explanation of components provided with like reference numerals will be omitted for reasons of clarity of the illustration of the invention.

[0048] FIG. 1 shows an exemplary view of a transformation base determined according to the method according the invention, which serves as a basis for determining the new image point position of the image point to be displaced of the current individual image. The transformation base 1 illustrated in FIG. 1 extends along a first dimension and a second dimension of the transformation base, i.e. horizontally along a moving image sequence 21 and vertically along a time succession time axis 22. A third dimension of the transformation base, which extends orthogonal to the first and second dimension, is not illustrated for reasons of a better understanding of the following explanations. Along the moving image sequence 21 as the first dimensional axis of the transformation base 1, start times t.sub.−1, t.sub.0, t.sub.1 of a first individual image B.sub.−1, B.sub.0, B.sub.1 and the associated end times t.sub.0, t.sub.1, t.sub.2 thereof are plotted, the respective individual images extending in each case along the first dimensional axis 21 across a time interval ΔT. In the second dimensional axis, orthogonal to the first dimensional axis and representing a row sequence of an individual image, the individual rows 3 are indicated by horizontal dashed lines running parallel to one another.

[0049] The relative points of time are indicated on the left side of FIG. 1 as multiple integers of Tz, to which is detected a respective row 3 within an image detection time TB of an individual image.

[0050] According to FIG. 1, the first uppermost image row is detected at a relative point of time 0, while the subsequent image row is detected at a relative point of time Tz, and the further subsequent image row is detected at a point of time 2Tz, and the lowest row is detected with the largest delay, i.e. (n−1) times Tz.

[0051] In the transformation base 1 illustrated in FIG. 1, a first individual image B.sub.−1, a current image B.sub.0, and a second individual image B.sub.1, which follow directly one after the other and thus have individual image distances to one another of in each case one, are arranged in the transformation base 1 such that the horizontal direction of each individual image, i.e. in the first image point dimension of each individual image, runs parallel to a row detected at the same relative point of time. The three individual images (B.sub.−1, B.sub.0, B.sub.1) are further arranged in the transformation base 1 such that the vertical direction of these individual images, i.e. the y-axis in the second image point dimension thereof, are oriented to match with the row succession axis 22, i.e. the second dimension of the transformation base. The row-dependent delay time within a respective individual image is illustrated in FIG. 1 by the relative points of time illustrated on the left side of FIG. 1 on the one hand, and on the other hand by a detection delay time diagonal 5 running obliquely through each individual image from top left to at the bottom right. An image point P to be displaced of the current individual image B.sub.0 with the positions or time and image point position tP in the direction of the t-axis of the individual image and yP in the direction of the y-axis of the individual image is thus located on the detection delay time diagonal 5 associated to this individual image B.sub.0. The start in time of the current individual image B.sub.0 is located in the transformation base 1 on the moving image time axis at the position of t.sub.0, and the image start in time or the image start limit or boundary of the current individual image is characterized by the one of the vertically and straight running start time axis 15. Accordingly, the time end of the current individual image is characterized in the transformation base by the vertically running end time axis 16 at end time t.sub.1. The image point positions determined by the respective image motion information in the first individual image B.sub.−1 and the second individual image B.sub.1, which are assigned to the current positions tP, yP, are located in the respective individual image also on the respective detection delay time diagonal 5 with positions or time and image point position tPL and yPL for the first individual image and tPR and yPR for the second individual image.

[0052] The arrow 2 shown in a dashed manner and oriented downward in FIG. 1 is to illustrate the movement of the image points PL, P, PR, which are assigned to one another via the respective image motion information, exemplary underlying the illustration in FIG. 1. Specifically, FIG. 1 is to illustrate a vertically downward and accelerated movement of the individual image content of individual image B.sub.−1 to the current individual image B.sub.0 to the next individual image B.sub.1. According to the acceleration, the displacement of the respective point PL, P or PR increases on the respective detection delay time diagonal 5 toward the bottom right, which illustrates that the respective image point is detected within the individual image time interval ΔT at an increasing later relative point of time within the respective individual image.

[0053] A first line 7 is set as a connecting line from the first time and image point position tPL, yPL of the first individual image B.sub.−1 to the current image point position tP, yP of the current individual image Bo, and this line 7 intersects the start time axis 15 of the current individual image B.sub.0 in the intersection point S1. Similarly, a line 8 was identified or determined in the transformation base 1, which runs through the time and image point position tP, yP of the current individual image B.sub.0 and through the second time and image point position tPR, yPR of the second individual image B.sub.1. This line 8 intersects the end time axis 16 in the intersection point S2. Since in FIG. shows the exemplary case of a vertical acceleration of an image point or the content associated with it, line 8 has a greater slope compared over line 7, which can be clearly discerned from the extension of line 7 shown as a dashed line 7*.

[0054] FIG. 2 is a section focused to the section of the current individual image Bo of the transformation base 1 formed according to the invention. The general explanations with respect to FIG. 1 also apply to FIG. 2. In contrast, FIG. 2 illustrates more specifically how the new time and image point position tQ, yQ is determined based upon the current time and image point position tP, yP. As illustrated in FIG. 2, the first intersection point S1 is determined as the intersection point of the first line 7, which connects the first time and image point position tPL, yPL with the current time and image point position tP, yP, and the time start axis 15 of the current individual image by the time and image point positions tS1, yS1. The intersection point S2 is determined by the time and image point positions tS2, yS2. As illustrated in FIG. 2 in an exemplary manner, the new time and image point position in the current individual image B.sub.0 is determined based upon an average value of the first and second interpolation point or intersection point S1 and S2. The new image point position yQ of the current image point is determined for the second dimension, i.e. the y-axis of the current individual image, as an average value of yS1 and yS2. Thus, the following equations apply:

yQ=(yS1+yS2)/2

or:

yQ=yS1+|yS1−yS2|/2.

or even:

yQ=yS2−|yS1−yS2|/2.

[0055] For the second image point position y, which runs orthogonally in the image to the y-axis and determines the pixel position:

xQ=(xS1+xS2)/2

or:

xQ=xS1+|xS1−xS2|/2.

or even:

xQ=xS2−|xS1−xS2|/2.

[0056] FIG. 2 also shows the embodiment of the method according to the invention, in which averaging the intersection point positions or interpolation point positions is effected in the direction of the t-axis of the individual image Bo. Accordingly, the new time tQ is defined as:

tQ=(tS1+tS2)/2.

[0057] Since the positions of the intersection points S1 and S2 in the direction of the t-axis are always fixedly determined by the start time axis 15 and the end time axis 16 of the current individual image Bo, the new time position in the first axis, i.e. tQ, according to this embodiment is located always on the same point, which however does not apply to the new time position tQ in the second dimensional axis.

[0058] FIG. 2—for the illustration of another embodiment of the method according to the invention—illustrates a correction offset Ay in the direction of the y-axis, respectively the second dimension of the transformation base 1, and FIG. 2 emphasizes the determination of the new time and image point position tQ, xQ, yQ.

xQ:=xP+ΔJ

yQ:=yP+ΔJ

tQ:=tP+ΔJ

[0059] The determination of tQ or Δt in the first dimension of the transformation base, as illustrated in FIG. 2, is a mathematic tool in order to finally be able to calculate the decisive correction amounts in the two-dimensional image point plane from the three-dimensional transformation base.

[0060] The first correction offset Δx and the second correction offset Δy are finally determined, after a mathematic transformation of Δx, Δy and Δt as follows:

Δx=−(xPL.Math.yPR.Math.v.sup.2.Math.yP+xPR.Math.yPL.Math.v.sup.2.Math.yP−h.Math.xPR.Math.v.Math.yP+h.Math.xPL.Math.v.Math.yP−h.Math.xPR.Math.yPL.Math.v+h.sup.2.Math.xPR)/2(yPL.Math.v−h)(yPR.Math.v+h)

Δy=−(2.Math.yPL.Math.yPR.Math.v.sup.2.Math.yP−h.Math.yPR.Math.v.Math.yP+h.Math.yPL.Math.v.Math.yP−h.Math.yPL.Math.yPR.Math.v+h.sup.2.Math.yPR)/2(yPL.Math.v−h)(yPR.Math.v+h)

[0061] The variable v in the above two formulas refers to a predetermined speed, which preferably is a row succession speed and particularly preferably a row detection speed of a CMOS sensor, which previously has detected the moving image sequence to be transformed. The variable h in the above two formulas is a predetermined row dimension size, which is preferably intrinsic to the CMOS sensor, which has previously detected the moving image sequence to be transformed, and particularly preferably the height of the CMOS sensor, i.e. the extension thereof in the second dimension.

[0062] FIG. 3 shows a flow diagram, which illustrates an exemplary course or flow of an embodiment according to the invention of the method for the transformation of a moving image sequence. In a first step, an index I, which in each case points to the current individua image in the type of an index, is set at a start value (I:=0). Subsequently, steps S1β to S50 are iteratively performed for the transformation of the moving image sequence. First, in step S10, three successive individual images B.sub.−1+i, B.sub.0+i, and B.sub.1+i are read out from a moving image sequence storage. Furthermore, in step S20, a first image motion information data set Δ(B.sub.−1+I and B.sub.1+i) which describes the optical flow or optical motion from the first individual image B.sub.−1+i to the current individual image B.sub.0+1, and a second image motion information data set Δ(B.sub.0+i, B.sub.1+i), which describes the optical flow or optical motion from the current individual image B.sub.0+i to the second individual image B.sub.1+i, are read from the image motion information storage. Subsequently, in step S30, the transformation base 1 is formed by arranging the individual images B.sub.−1+i, B.sub.o+i and B.sub.1+i read in step 10 in the chronological order thereof along the moving image sequence time axis. Subsequently, in step S40, a displacement is effected of at least one image point of the individual image B.sub.0+1 from the respective image point-specific current image point position xP, yP to the new image point position xQ, yQ to be determined. Once the image points of the individual image B.sub.0+i selected for the transformation have been displaced toward to the respective new image point positions, the index i (or the counter thereof) for the current individual image is increased by one, followed by the repetition of steps S10 to S50 for the next individual image set by the index i and increased by one.

[0063] The flow diagram in FIG. 4 shows an embodiment according to the invention of step S40 shown in FIG. 3. The realization of step S40 essentially corresponds to the determination of the new positions xQ, yQ via the respective average values of the intersection points positions.

[0064] FIG. 5 shows another embodiment of step S40 stated in FIG. 3 according to the invention. The determination indicated in FIG. 5 in step 47 of the new position xQ, yQ is based upon the determination of the first correction offset Δx and the second correction offset Δy by using the above indicated formulas for the determination of the first or second correction offset Δx and Δy.

[0065] The present invention is not limited to the embodiments shown and described here in an exemplary manner. The scope of protection of the invention is rather defined by the attached claims.