METHOD AND SYSTEM FOR CAMERA MOTION BLUR REDUCTION
20230179863 · 2023-06-08
Assignee
Inventors
Cpc classification
H04N23/683
ELECTRICITY
H04N23/6845
ELECTRICITY
H04N23/6812
ELECTRICITY
International classification
Abstract
A method for reducing camera motion blur comprises, before acquiring an image frame for a video stream, a camera measurement unit measuring data related to a camera module motion during a time window; determining camera module motion based on the measured data and predicting a camera motion blur during acquisition of the image frame based at least on the determined camera module motion and the lens projection model; determining whether the predicted camera motion blur exceeds a threshold; in response to determining that the predicted camera motion blur exceeds the threshold, determining a reduction of the provisional exposure time determined to acquire the image frame so that the predicted camera motion blur reaches the threshold, determining whether a corresponding increase in the provisional gain determined to acquire the image frame is below a maximum gain value, adjusting the provisional exposure time and gain, and acquiring the image frame.
Claims
1. A method at an image acquisition system for reducing camera motion blur, wherein the image acquisition system comprises a camera module including an image sensor and a lens, the lens being associated with a lens projection model, the method comprising: determining, by one or more processors, a set of provisional exposure parameters to acquire each of successive image frames for a video stream, the provisional exposure parameters including a provisional exposure time in combination with a provisional gain; before acquiring an image frame for a video stream, measuring, by the one or more processors, data related to a camera module motion during a time window; determining, by the one or more processors, the camera module motion based on measured measurement data related to a camera module motion and predicting a camera motion blur during acquisition of the image frame based at least on the camera module motion and the lens projection model; determining, by the one or more processors, whether the predicted camera motion blur exceeds a threshold; in response to determining that the predicted camera motion blur exceeds the threshold, determining, by the one or more processors, a reduction of the provisional exposure time determined to acquire the image frame so that the predicted camera motion blur reaches the threshold, and determining whether a corresponding increase in the provisional gain determined to acquire the image frame is below a maximum gain value, in response to determining that said corresponding increase in the provisional gain is below the maximum gain value, adjusting, by the one or more processors, the provisional exposure time and gain by applying said reduction and corresponding increase; and acquiring the image frame using the adjusted provisional exposure time and gain.
2. The method of claim 1, wherein the lens has a variable focal length and the associated model is a variable lens projection model.
3. The method of claim 1, further comprising: in response to determining that said corresponding increase in the provisional gain exceeds the maximum gain value, adjusting the provisional exposure time and gain by: increasing the provisional gain to reach the maximum gain value, and either applying a corresponding reduction to the provisional exposure time, or applying said reduction of the provisional exposure time so that the predicted camera motion blur reaches the threshold; and acquiring the image frame by using the adjusted exposure time and gain.
4. The method of claim 1, further comprising: in response to determining that the predicted camera motion blur IS below the threshold, adjusting the provisional exposure time and gain by: increasing the provisional exposure time so that the predicted camera motion blur reaches the threshold, and applying a corresponding reduction to the provisional gain; and acquiring the image frame by using the adjusted exposure time and gain.
5. The method of claim 1, wherein the time window corresponds to at least a portion of an exposure time used to acquire a preceding image frame.
6. The method of claim 1, wherein the data measured during the time window comprise angular velocity data of the camera module, wherein determining the camera module motion based on the measured data comprises: transforming the measured angular velocity data into corresponding camera module orientations, and calculating a camera module rotation over at least one portion of the time window; and wherein predicting the camera motion blur comprises: selecting at least one reference point at a first position in an image space, projecting the at least one reference point into a virtual 3D space based on the lens projection model, rotating the projected at least one reference point according to the calculated camera module rotation, projecting back the rotated at least one reference point to a second position in the image space based on the lens projection module, and calculating a camera motion blur corresponding to the calculated camera module rotation as a distance between the first and second positions.
7. The method of claim 6, wherein the camera module rotation is calculated between a start and end of the time window.
8. The method of claim 6, wherein a plurality of camera module rotations are calculated over a plurality of corresponding portions of the time window, and wherein predicting the camera motion blur comprises combining calculated camera motion blurs corresponding to each of the plurality of calculated camera module rotations.
9. The method of claim 6, wherein said at least one reference point includes a center point in the image space.
10. The method of claim 6, wherein the at least one reference point includes at least a first reference point and a second, different, reference point in the image space, and wherein predicting the camera motion blur comprises: selecting the maximum of a first camera motion blur and a second camera motion blur calculated starting from the selection of the first reference point and the second reference point, respectively, in the image space.
11. The method of claim 1, further comprising: determining a camera module motion periodicity based on data measured over a time period preceding said time window; and before performing step c), refining the predicted camera motion blur based on the determined camera module motion periodicity.
12. The method of claim 1, wherein before acquiring the image frame by using the adjusted provisional exposure time and gain, the method further comprises: determining whether the camera module is decelerating; and in response to determining a camera module deceleration, acquiring the image frame using the provisional exposure time and gain.
13. The method of claim 1, further comprising: processing successively acquired image frames, including the video frame acquired with the adjusted provisional exposure time and gain, to generate a stabilized video stream.
14. An image acquisition system comprising: a camera module including at least a lens and an image sensor, the lens being associated with a lens projection model, a motion sensor configured to measure data related to a camera module motion, and a controller configured to determine a set of provisional exposure parameters to acquire each of successive image frames for a video stream, the provisional exposure parameters including a provisional exposure time in combination with a provisional gain; the image acquisition system being configured to perform the steps of claim 1.
15. A computer program product comprising a non-transitory computer readable medium on which instructions are stored which, when executed on an image acquisition system, are configured to cause the image acquisition system to perform the steps of claim 1.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:
[0026]
[0027]
[0028]
[0029]
DETAILED DESCRIPTION
[0030] Referring now to
[0031] The system 10 comprises a camera module 11 including a lens 12, preferably with variable focal length, and an image sensor 13. The lens 12 is associated with a mathematical projection model that is used by the system 10 during operation of the camera blur reduction method. Various lens projection models can be used by the system 10, with the requirement of providing at least a set of two functions describing, respectively, a point forward projection from a 3D virtual world to the 2D image space associated with the camera module 11 and a point backward projection from the 2D image space to the 3D virtual world.
[0032] With reference to
R=√{square root over (X.sup.2+Y.sup.2)}
α=atan 2(R,Z)
k=LUTF(α)
x=F.sub.xkX+O.sub.x
y=F.sub.ykY+O.sup.y
[0033] The lens projection model further includes the following set of equations for projecting back point p to point P:
[0034] With reference back to
[0041] The configuration and functionality of the system 10, and especially of the camera motion blur suppression module 16, will be now disclosed in more detail by referring to the operation of the method 100 illustrated in
[0042] With reference to
[0043] The IMU 15 measures gyroscope data related to the 3D angular velocity of the camera module 11 at different moments during T.sub.exp1 (step 102). The gyroscope data include at least data measured at the beginning (T.sub.start) and end (T.sub.end) of T.sub.exp1.
[0044] The camera trajectory computation module 17 integrates and transforms the measured gyroscope data into corresponding quaternions Q.sub.start . . . Q.sub.end, each representing a camera module orientation at the moment when the corresponding gyroscope data have been measured by the IMU 15 (step 103). It is to be noted that the module 17 can be alternatively configured to transform the measured gyroscope data into any other form suitable for representing camera module orientation.
[0045] The module 17 stores the obtained quaternions in the camera module orientation buffer (step 104).
[0046] Then the method 100 proceeds by determining whether a subsequent image frame has to be acquired for the video frame (step 105).
[0047] In response to a positive determination at step 105, the auto-exposure module 14 determines a set of provisional auto exposure parameters for acquiring the subsequent image frame with a proper, good quality, exposure level (step 106). The determined set of exposure parameters include at least a provisional exposure time T.sub.exp2_provisional in combination with a provisional sensor gain G.sub.2_provisional. It to be appreciated that T.sub.exp2_provisional and G.sub.2_provisional can correspond to or be different than T.sub.exp1 and G.sub.1 used to acquire the starting image frame at step 101, depending on the brightness or other parameters of the scene being the same or different than during T.sub.exp1.
[0048] The camera motion blur level prediction module 18 retrieves from the camera module orientation buffer at least the quaternions Q.sub.start, Q.sub.end representing the camera module orientations at T.sub.start and T.sub.end of T.sub.exp1. Then, the module 18 calculates (step 107) the camera module rotation Δα occurring during T.sub.exp1 as:
Δα=Q_start*conjungate(Q.sub.end).
[0049] Based on the assumption that Δα is a good enough prediction of the camera module rotation that will occur during the planned acquisition of the subsequent image frame, the module 18 predicts a length of a camera motion blur trail occurring during the planned acquisition based on Δα and the lens projection model (step 108).
[0050] In some embodiments, the module 18 predicts the length of the camera motion blur trial based on Δα and the lens projection model by:
[0051] 1) selecting at least two reference points in the 2D image space at different starting positions with coordinates x0, y0 and x0′, y0′, respectively. For example, the two reference points can be selected each closer to a corresponding boundary of the 2D image space (such as two points symmetrical relative to the center of the 2D image space);
[0052] ii) projecting, based on the lens projection model, each of the two reference points into a virtual 3D space at different positions with coordinates X0, Y0, Z0 and X0′, Y0′, Z0′, respectively;
[0053] iii) rotating each of the two projected reference points with the calculated camera module rotation Δα, obtaining two different rotated reference points at coordinates X1, Y1, Z1 and X1′, Y1′, Z1′, respectively;
[0054] iv) projecting back, based on the lens projection model, the two rotated reference points from the virtual 3D space to the 2D image space, at corresponding end positions with coordinates x1, y1 and x1′, y1′, respectively (different that the coordinates x0, y0 and x0′, y0′ of the starting positions of the two selected reference points);
[0055] v) computing camera motion blur trail lengths TrailLength1 and TrailLength2 as the distances between the start and the end positions of the two selected reference points, according to the equations:
TrailLength1=√{square root over ((x0−x1).sup.2+(y0−y1).sup.2)}
TrailLength1=√{square root over ((x0′−x1′).sup.2+(y0′−y1′).sup.2)}
[0056] vi) selecting the maximum of TrailLength1 and TrailLength2 as the prediction of the camera motion blur trail during the acquisition of the subsequent image frame.
[0057] It is to be appreciated using the lens projection model with variable focal lengths F for predicting the camera motion blur trail, is based on the consideration that images acquired with the same kind of camera module rotation and the same exposure time, but with different focal lengths, can contain a significantly different camera motion blur (in particular, a longer focal length camera will generate longer camera motion blur trails).
[0058] It is to be further appreciated that, according to alternative embodiments, a length of the camera motion blur trail can be predicted by operating steps similar to the above disclosed steps ii)-v), but selecting a single reference point (e.g., the center of the image space 2D. However, it is preferable to use at least two reference points for the prediction of the camera motion blur trail, because if the camera module 11 rotates only along the roll axis during T.sub.exp1 and only the center of the 2D image space is selected as reference point, this reference point will not move at all due to the rotation (whereas points closer to the image boundary will move).
[0059] Furthermore, in some alternative embodiments T.sub.exp1 can be divided in smaller time segments for performing the prediction of the camera motion blur trial. In particular, according to these embodiments:
[0060] at step 102, in addition to gyroscope data related to the 3D angular velocity of the camera module 11 at T.sub.start and T.sub.end of T.sub.exp1, the IMU 15 also measures gyroscope data at a plurality of different moments between T.sub.start and T.sub.end, so as to divide T.sub.exp1 in a plurality of N time segments between successive measurements;
[0061] in addition to the quaternions Q.sub.start, Q.sub.end representing camera module orientations at T.sub.start and T.sub.end of Texp1, the module 18 retrieves from the camera orientation buffer a plurality of quaternions representing intermediate camera module orientations corresponding to the plurality of gyroscope data measured between T.sub.start and T.sub.end;
[0062] at step 107, based on the retrieved quaternions, the module 18 calculates a plurality of camera module rotations Δα.sub.1 . . . Δα.sub.N occurring during each of the N segments of T.sub.exp1; and
[0063] at step 108, the module 18 calculates a plurality of camera motion blur trail lengths, TrailLength1 . . . TrailLengthN, each corresponding to one of the plurality of camera module rotations Δα.sub.1 . . . Δα.sub.N. This calculation is performed in a way similar to the above disclosed calculation based on the overall camera module rotation Δα occurring between T.sub.start and T.sub.end of T.sub.exp1 (either performed by selecting two reference points or a single reference point in the 2D image space).
[0064] The module 18 then combines the calculated TrailLength1 . . . TrailLengthN to obtain an overall prediction of the camera motion blur trail length during acquisition of the subsequent image frame, e.g. by computing an average or other statistical combination of TrailLength1 . . . TrailLengthN.
[0065] In this way, a closer (more granular) approximation of the camera module rotation trajectory is obtained than just considering the overall camera module rotation Δα occurring between T.sub.start and T.sub.end of T.sub.exp1. This closer approximation improves the prediction of the camera motion blur trail length, especially in situations where the camera module orientation changes significantly between the start and the end of an exposure time, but close to the end the camera module orientation comes back close to the starting orientation. An example of these situations is illustrated in
[0066] With reference back to
[0067] In response to determining that the predicted camera motion blur trail exceeds the threshold at step 109, the module 18 determines a reduction of T.sub.exp2_provisional so that the predicted camera motion blur trail reaches the threshold (step 110). In practice, if the predicted camera motion blur trail exceeds the threshold by a factor X.sub.1, a corresponding reduction of X.sub.1 is determined for T.sub.exp2_provisional.
[0068] However, in order to preserve the good quality overall exposure level achievable according to the provisional exposure parameters recommended by auto-exposure module 14, before applying the X.sub.1 factor reduction to T.sub.exp2_provisional, the method 100 proceeds by determining whether a corresponding X.sub.1 factor increase can be applied to G.sub.2_provisional without exceeding G.sub.Max (step 111).
[0069] In response to a positive determination, T.sub.exp2_provisional and G.sub.2_provisional are reduced and increased, respectively, by the same X.sub.1 factor (step 114), thus reducing the predicted camera motion blur trail to the acceptable threshold level while maintaining the exposure level recommended by the auto-exposure module 14 (as permitted by G.sub.MAX).
[0070] After step 114, the method 100 proceeds starting again from step 101 where the subsequent image frame for the video stream is acquired using the adjusted T.sub.exp2_provisional and G.sub.2_provisional.
[0071] In some embodiments, an additional filtering is applied to the adjusted T.sub.exp2_provisional at the acquiring the subsequent image frame, so as to smooth out changes and to avoid undesired visual effects in the subsequent image frame caused by sudden exposure changes.
[0072] Furthermore, in some embodiments, before T.sub.exp2_provisional and G.sub.2_provisional are adjusted at step 114, the module 18 can determine whether the camera module 11 is decelerating (step 112). For example, this determination can be based on camera module angular acceleration measurements provided by an accelerometer within the IMU 15 (and/or by an accelerometer of the system 10 separated from the IMU 15). In addition or alternatively, the camera module acceleration/deceleration can be calculated based on the camera module angular velocities measured by the one or more gyroscopes of the IMU 15.
[0073] Based on the assumption that if the camera module 11 is not decelerating the camera module motion will most likely continue during the acquisition of the subsequent image frame, if camera module deceleration is not determined at step 112, T.sub.exp2_provisional and G.sub.2_provisional are adjusted by the same factor X.sub.1 at step 114.
[0074] Based on the assumption that if the camera module 11 is decelerating the camera module motion will most likely stop during the acquisition of the subsequent image frame, if camera module deceleration is determined at step 112, T.sub.exp2_provisional and G.sub.2_provisional are maintained at step 113 (i.e., the adjustments determined by operation of the previous steps 110-111 are not applied). The method 100 then proceeds to start again from step 101, where the subsequent image frame for the video stream is acquired using the unmodified T.sub.exp2_provisional and G.sub.2_provisional.
[0075] With reference back to step 111, in response to determining that a X.sub.1 factor increase in G.sub.2_provisional would exceed G.sub.MAX, the module 18 determines a factor X.sub.2 (less than X.sub.1) to increase G.sub.2_provisional so as to reach G.sub.MAX, and to reduce T.sub.exp2_provisional (step 115). In this way, the proper exposure level recommended by the auto-exposure module 14 is maintained, while reducing the predicted camera motion blur trail (although not enough to reach the threshold).
[0076] Again, in some embodiments, before T.sub.exp2_provisional and G.sub.2_provisional are adjusted at step 114 by applying the X.sub.2 factor, the module 18 determines whether the camera module 11 is decelerating (step 112). If camera module deceleration is not determined at step 112, T.sub.exp2_provisional and G.sub.2_provisional are adjusted by the same factor X.sub.2 at step 114. If camera module deceleration is determined at step 112, T.sub.exp2_provisional and G.sub.2_provisional are maintained unmodified at step 113.
[0077] In alternative embodiments, the module 18 increases G.sub.2_provisional by the factor X.sub.2 so as to reach G.sub.MAX, but still applies the factor X.sub.1 (greater than X.sub.2) to reduce T.sub.exp2_provisional. In this way, the predicted camera motion blur trail is reduced to reach the threshold, but accepting an exposure level below the level recommended by the auto-exposure module 14.
[0078] With reference back to step 109, in response to determining that the predicted camera motion blur trail is below the threshold, the module 18 determines an increase in T.sub.exp2_provisional so that the predicted camera motion blur trail reaches the threshold, and a corresponding reduction to G.sub.2_provisional (step 116). In this way, the predicted camera motion blur trail is maintained at the acceptable threshold level while reducing the noise level in the subsequent image frame. The provisional T.sub.exp2_provisional and G.sub.2_provisional are adjusted accordingly at step 114 before acquiring the subsequent image frame.
[0079] In alternative embodiments, when no relevant camera motion blur risk is detected, T.sub.exp2_provisional and G.sub.2_provisional are maintained without adjustment to acquire the subsequent image frame.
[0080] The operation of the method 100 continues until the system 10 determines, at step 105, that no further image frames have to be acquired for the video stream, and the method ends (step 117).
[0081] In some embodiments, when the IMU 15 has measured data over the exposure time of a sufficient number of successive previous image frames to make a periodicity in the change of the camera module orientation detectable, the method 100 further comprises detecting and using this periodicity to refine the prediction of the camera motion blur trail calculated at step 109 before acquisition of a subsequent image frame.
[0082] This refinement is particularly effective in situations where the camera module undergoes a change in orientation due to strong periodic motions of a portable device including the system 10. Examples of these situations include a user holding the device while walking or running with regular steps, or the device be mounted on an object that undergoes periodic motion (e.g., a mast swaying in the wind).
[0083] The image frames for a video stream acquired as per the operation of the method 100 are stored in the memory 20 of the system 10 (or any other memory accessible by the system 10). The stored image frames are retrieved by the video stabilization module 19 of the system 10 that performs image processing for generating a stabilized video stream in which camera motion blur trails are suppressed or at least limited due to the exposure parameters adjustment performed by the operation of the method 100. Alternatively, video stabilization on the acquired image frames can be performed by a video stabilization module that is external but operatively connected to the system 10 so as to retrieve the image frames from the memory 20.
[0084] In light of the above disclosed operation of the method 100, it will be appreciated that the threshold value used at step 109 can be fixed or can be changed according to user preferences to achieve a desired balance between image sharpness and the amount of visible sensor noise.
[0085] With reference back to the beginning of the method at step 101, it will be further appreciated that the exposure time and gain used for acquiring the first image frame for the video stream can also result from an adjustment of the provisional exposure time and gain recommended by the auto-exposure module 14. In particular, this adjustment can be based on operation of steps similar to the above disclosed method steps 102-114, but with the difference that the gyroscope data provided by the IMU 15 and used to predict a camera module rotation and corresponding camera motion blur trail during the acquisition of the first image frame are measured during a time window preceding the acquisition that does not correspond to the exposure time of a preceding image frame.
[0086] Finally, although in the above disclosed operation of the method 100 the gyroscope data provided by the IMU 15 and used to predict a camera module rotation and corresponding camera motion blur trail during the acquisition of a subsequent image frame are measured during the exposure time used for acquiring a preceding image frame, it will be noted that the gyroscope data can be generated during any measurement time window preceding the acquisition of the subsequent image frame. For example, a measurement time window longer than and including the exposure time of the preceding image frame can be employed, e.g., a measurement time window including the exposure times of a plurality of previously acquired image frames. Furthermore, a time window corresponding to a portion of the exposure time of the preceding image frame, or any other suitable time window different than the exposure time of the preceding image frame, can be employed.