Method for motion estimation based on motion blur

Abstract

A method for motion estimation based on perceived blur due to motion between a first image frame and a second image frame of a digital video sequence comprises estimating a motion vector between the frames for each of a plurality of pixel blocks in the first and second image frames. A patch motion vector is then estimated for each of a plurality of patches of the motion vectors based on one of vectors in each patch and motion vectors in proximate patches. The patch motion vector of its corresponding patch is allocated to each pixel in the first frame. A system and computer program for motion estimation based on perceived blur is also provided.

Claims

1. A method of motion estimation based on perceived blur due to motion between a first frame and a second frame of a digital video sequence using estimates of motion direction and motion extent of pixels between the frames, the method comprising: generating an initial guess frame based on the first frame; blurring pixels in the guess frame as a function of their respective estimated blur directions and blur extents; comparing each blurred pixel with a respective pixel in the first frame to generate a motion variable pixel for each respective pixel; weighting each motion variable pixel; and combining each motion variable pixel and its respective pixel in the initial guess frame.

2. The method of claim 1, wherein the weighting is a function of the respective pixel motion.

3. The method of claim 2, wherein the weighting is an estimate of the edge magnitude of the respective pixel in the guess image in the direction of pixel motion.

4. The method of claim 2 wherein the pixel blurring, comparing, motion variable pixel blurring and weighting, and combining are performed iteratively.

5. The method of claim 4 wherein the pixel blurring, comparing, motion variable pixel blurring and weighting, and combining are performed iteratively until a sum of error falls below a threshold level.

6. The method of claim 4 wherein the pixel blurring, comparing, motion variable pixel blurring and weighting, and combining are performed iteratively a predetermined number of times.

7. The method of claim 4 wherein the pixel blurring, comparing, motion variable pixel blurring and weighting, and combining are performed iteratively until a sum of error fails to change by more than a threshold amount between successive iterations.

8. A computer readable medium embodying a program of instructions executable by the computer to perform the method of claim 1.

9. A method of motion estimation based on perceived blur due to motion between a first frame and a second frame of a digital video sequence comprising: estimating a motion vector between the frames for each of a plurality of pixel blocks in the frames; estimating a cluster motion vector for each of a plurality of clusters of motion vectors based on one of motion vectors in each cluster and motion vectors in proximate clusters; allocating to each pixel in the first frame, the cluster motion vector of its corresponding cluster; generating an initial guess frame based on the first frame; blurring pixels in the guess frame as a function of their respective allocated cluster motion vector; comparing each blurred pixel with a respective pixel in the first frame to generate a motion variable pixel for each respective pixel; weighting each motion variable pixel; and combining each motion variable pixel and its respective pixel in the initial guess frame. A system for compensating for perceived blur due to motion between a first frame and a second frame of a digital video sequence using estimates of motion direction and motion extent of pixels between the frames, the system comprising: a motion blur filter array blurring pixels in an initial guess image, that is based on the first frame, as a function of their respective estimated blur directions and blur extents; a comparator comparing each blurred pixel with a respective pixel in the first frame to generate a motion variable pixel for each respective pixel, said motion blur filter array further weighting each motion variable pixel; and an algorithm combining each motion variable pixel and its respective pixel in the initial guess frame.

10. The system of claim 9, wherein the weighting is a function of the respective pixel motion.

11. The system of claim 9, wherein the weighting is an estimate of the edge magnitude of the respective pixel in the guess image in the direction of pixel motion.

12. The system of claim 9 wherein the initial guess frame is the first frame.

13. The system of claim 9 wherein the initial guess frame is the first frame.

14. The system of claim 9, wherein the motion blur filter array provides weighting based on an estimate of the edge magnitude of the respective pixel in the guess frame in the direction of pixel motion.

15. The system of claim 9 wherein the motion blur filter array, the comparator and the algorithm iteratively perform the blurring, comparing, blurring and weighting, and combining.

16. The system of claim 9 wherein the motion blur filter array, the comparator and the algorithm iteratively perform the blurring, comparing, blurring and weighting, and combining until the sum of error falls below a threshold level.

17. The system of claim 9 wherein the motion blur filter array, the comparator and the algorithm iteratively perform the blurring, comparing, blurring and weighting, and combining a predetermined number of times.

18. The system of claim 9 wherein motion blur filter array, the comparator and the algorithm iteratively perform the blurring, comparing, blurring and weighting, and combining until the sum of error fails to change by more than a threshold amount between successive iterations.

Description

SUMMARY OF THE INVENTION

[0004] According to one aspect there is provided a method of estimating motion of pixels between a first frame and a second frame in a digital video sequence, comprising:

[0005] estimating a motion vector between the frames for each of a plurality of pixel blocks of said first and second frames;

[0006] estimating a patch motion vector for each of a plurality of patches of motion vectors based on one of motion vectors in each patch and or motion vectors in proximate patches; and

[0007] allocating to each pixel in the first frame, the patch motion vector of its corresponding patch.

[0008] The estimating a motion vector may comprise dividing the first and second frames into a plurality of the pixel blocks and wherein said motion vector estimating comprises: for each of the pixel blocks, selecting from a list of motion vector candidates a motion vector that provides the least difference between the pixel block in the first frame and a corresponding pixel block in the second frame.

[0009] Estimating the patch motion vector may comprise assigning patch motion vectors to respective ones of the patches that have a first threshold proportion of motion vectors with substantially the same direction and at least a threshold average extent; and in the event that less than a second threshold proportion of assigned patch motion vectors have substantially the same direction, iteratively assigning the motion vectors of assigned patches to proximate ones of unassigned patches.

[0010] In accordance with another aspect, there is provided a method of estimating motion for perceived blur due to motion between a first frame and a second frame of a digital video sequence comprising:

[0011] estimating a motion vector between the frames for each of a plurality of pixel blocks in the frames;

[0012] estimating a patch motion vector for each of a plurality of patches of motion vectors based on one of motion vectors in each patch and motion vectors in proximate patches;

[0013] allocating to each pixel in the first frame, the patch motion vector of its corresponding patch;

[0014] weighting each pixel; and

[0015] combining each patch motion vector and its respective pixel.

[0016] In accordance with another aspect, there is provided a system for estimating motion of pixels between a first frame and a second frame in a digital video sequence, comprising:

[0017] a block vector calculator determining a motion vector between the frames for each of a plurality of pixel blocks of the first and second frames;

[0018] a patch vector calculator determining a patch motion vector for each of a plurality of patches of motion vectors based on one of motion vectors in each patch and motion vectors of proximate patches; and

[0019] a pixel vector allocator allocating to each pixel in the first frame, the patch motion vector of its corresponding patch.

[0020] In accordance with another aspect, there is provided a computer readable medium including a computer program for estimating motion of pixels between a first frame and a second frame in a digital video sequence, the computer program comprising:

[0021] computer program code for estimating a motion vector between the frames for each of a plurality of pixel blocks of said first and second frames;

[0022] computer program code for estimating a patch motion vector for each of a plurality of patches of motion vectors based on one of motion vectors in each patch and motion vectors in proximate patches; and

[0023] computer program code for allocating to each pixel in the frame, the patch motion vector of its corresponding patch.

[0024] In accordance with another aspect, there is provided a computer readable medium including a computer program for estimation motion based on perceived blur due to motion between a first frame and a second frame of a digital video sequence comprising, the computer program comprising:

[0025] computer program code for estimating a motion vector between the frames for each of a plurality of pixel blocks in the motion blurred frame;

[0026] computer program code for a patch motion vector for each of a plurality of patches of motion vectors based on one of motion vectors in each patch and motion vectors in proximate patches;

[0027] computer program code for allocating to each pixel in the first frame, the patch motion vector of its corresponding patch;

[0028] computer program code for weighting each pixel; and

[0029] computer program code for combining each patch motion vector and its respective pixel.

[0030] The methods and systems described herein provide the benefits of increased sharpness in digital video displayed on a hold-type display, such as for example an LCD device, without suffering excessive noise amplification as is common in many known inverse filtering methods and without requiring modification of the hold-type display hardware itself. Furthermore, by grouping pixels and patching motion vectors, prominent global and local object motion in the frame for the perceived motion blur can be determined quickly.