METHOD AND CIRCUITRY FOR EXPOSURE COMPENSATION APPLIED TO HIGH DYNAMIC RANGE VIDEO
20230239577 · 2023-07-27
Inventors
Cpc classification
G06V10/751
PHYSICS
G06V10/25
PHYSICS
H04N23/741
ELECTRICITY
G06V10/60
PHYSICS
G06V10/507
PHYSICS
International classification
H04N23/741
ELECTRICITY
G06V10/25
PHYSICS
G06V10/50
PHYSICS
G06V10/60
PHYSICS
Abstract
A method and a circuitry for exposure compensation applied to a high dynamic range video are provided. The circuitry is adapted to an image-acquisition device. In the method, when a video is received, the pixel values for each of the sequential frames can be obtained. Next, an exposure value ratio between two adjacent frames is obtained. A processor exposure value ratio of an image signal processor can be regarded as an initial exposure value ratio. A fixed adjustment ratio is used to control the image signal processor and an image sensor of the image-acquirement device so as to calculate an exposure value ratio for each of the frames. The exposure value ratio is referred to for performing the high dynamic range compensation for the frames so as to output an HDR video.
Claims
1. A method, comprising: retrieving pixel values of pixels in each frame of sequential frames of a video; obtaining an exposure value ratio between two frames in succession among the sequential frames; acquiring an exposure value ratio for each frame of the sequential frames by using a fixed adjustment ratio to control an image signal processor and an image sensor according to a processor exposure value ratio of the image signal processor; and generating sequential frames that undergone a high dynamic range compensation process by exposure compensation that employs the exposure value ratio for each frame of the sequential frames.
2. The method according to claim 1, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames.
3. The method according to claim 1, wherein, in a beginning of the method, a brightness range threshold is set for excluding overly-dark and overly-bright pixels, and a light-dark ratio threshold is set for excluding one or more potential moving regions from the video.
4. The method according to claim 3, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames.
5. The method according to claim 3, wherein, in the method, two frame exposure values are provided in a high dynamic range process; wherein, when the sequential frames are retrieved from the video, a first frame exposure value ratio is used as the processor exposure value ratio that multiplies the fixed adjustment ratio, and a second frame exposure value ratio is used as the processor exposure that multiplies twice the fixed adjustment ratio; correspondingly, an initial exposure value ratio that corresponds to the first frame exposure value ratio is set in the image sensor, an error is adjusted to be the same as the fixed adjustment ratio, and the exposure value ratio corresponding to the second frame exposure value ratio set in the image sensor is a result of the initial exposure value ratio multiplying the fixed adjustment ratio.
6. The method according to claim 5, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames.
7. The method according to claim 3, wherein, when pixel values of two sequential frames are retrieved from the video, a ratio range and an appropriate brightness range are set between the two sequential frames according to the processor exposure value ratio and the brightness range threshold for selecting a valid pixel pair between the two sequential frames within a reasonable ratio after excluding overexposed or overly-dark pixels, and an exposure compensation is performed upon the HDR video with the valid pixel pair between the two sequential frames.
8. The method according to claim 7, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames.
9. The method according to claim 8, wherein the instant exposure value ratio equals to a previous row exposure value ratio plus an adjustment value for each of the rows.
10. The method according to claim 9, wherein the instant exposure value ratio of a first row is used as the processor exposure value ratio of the image signal processor.
11. The method according to claim 10, wherein, when the exposure value ratio for each row of each of the frames is calculated, a threshold for adjusting the exposure value ratio is used to converge the exposure value ratio for each of the frames.
12. The method according to claim 11, wherein, starting from the processor exposure value ratio initially, the adjustment value of each of the rows and the fixed adjustment ratio are used to converge the exposure value ratio, a removable region with a height in each of the frames is removed after the convergence, and an output image region with another height is taken for outputting the HDR video via an output circuit.
13. A circuitry for exposure compensation applied to a high dynamic range (HDR) video, the circuitry being disposed in an image-acquisition device, comprising: a control circuit used to control operations of an image signal processor and an image sensor of the image-acquisition device, performing a method for exposure compensation applied to a high dynamic range video including: receiving a video, and retrieving pixel values of each frame of sequential frames of the video; obtaining an exposure value ratio between two frames of the sequential frames; using a fixed adjustment ratio to control the image signal processor and the image sensor according to a processor exposure value ratio of the image signal processor for acquiring an exposure value ratio for each frame of the sequential frames; and using the exposure value ratio for each frame of the sequential frames to perform exposure compensation for generating sequential frames undergone a high dynamic range compensation process so as to output an HDR video.
14. The circuitry according to claim 13, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames, and the instant exposure value ratio equals to a previous row exposure value ratio plus an adjustment value for each of the rows.
15. The circuitry according to claim 13, wherein, in a beginning of the method, a brightness range threshold is set for excluding overly-dark and overly-bright pixels, and a light-dark ratio threshold is set for excluding one or more potential moving regions from the video.
16. The circuitry according to claim 15, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames, and the instant exposure value ratio equals to a previous row exposure value ratio plus an adjustment value for each of the rows.
17. The circuitry according to claim 15, wherein, when pixel values of two sequential frames are retrieved from the video, a ratio range and an appropriate brightness range are set between the two sequential frames according to the processor exposure value ratio and the brightness range threshold for selecting a valid pixel pair between the two sequential frames within a reasonable ratio after excluding overexposed or overly-dark pixels, and an exposure compensation is performed upon the HDR video with the valid pixel pair between the two sequential frames.
18. The circuitry according to claim 17, wherein, in units of a row of one of the frames, an instant exposure value ratio is obtained by comparing sums of row pixel values of two of the frames, and the instant exposure value ratio equals to a previous row exposure value ratio plus an adjustment value for each of the rows.
19. The circuitry according to claim 18, wherein, when the exposure value ratio for each row of each of the frame is calculated, a threshold for adjusting the exposure value ratio is used to converge the exposure value ratio for each of the frames.
20. The circuitry according to claim 19, wherein, starting from the processor exposure value ratio initially, the adjustment value of each of the rows and the fixed adjustment ratio are used to converge the exposure value ratio, a removable region with a height in each of the frames is removed after the convergence, and an output image region with another height is taken for outputting the HDR video via an output circuit.
Description
BRIEF DESCRIPTION OF THE DRAWINGS
[0016] The described embodiments may be better understood by reference to the following description and the accompanying drawings, in which:
[0017]
[0018]
[0019]
[0020]
[0021]
[0022]
[0023]
[0024]
[0025]
DETAILED DESCRIPTION OF THE EXEMPLARY EMBODIMENTS
[0026] The present disclosure is more particularly described in the following examples that are intended as illustrative only since numerous modifications and variations therein will be apparent to those skilled in the art. Like numbers in the drawings indicate like components throughout the views. As used in the description herein and throughout the claims that follow, unless the context clearly dictates otherwise, the meaning of “a,” “an” and “the” includes plural reference, and the meaning of “in” includes “in” and “on.” Titles or subtitles can be used herein for the convenience of a reader, which shall have no influence on the scope of the present disclosure.
[0027] The terms used herein generally have their ordinary meanings in the art. In the case of conflict, the present document, including any definitions given herein, will prevail. The same thing can be expressed in more than one way. Alternative language and synonyms can be used for any term(s) discussed herein, and no special significance is to be placed upon whether a term is elaborated or discussed herein. A recital of one or more synonyms does not exclude the use of other synonyms. The use of examples anywhere in this specification including examples of any terms is illustrative only, and in no way limits the scope and meaning of the present disclosure or of any exemplified term. Likewise, the present disclosure is not limited to various embodiments given herein. Numbering terms such as “first,” “second” or “third” can be used to describe various components, signals or the like, which are for distinguishing one component/signal from another one only, and are not intended to, nor should be construed to impose any substantive limitations on the components, signals or the like.
[0028] The present disclosure is related to a method for exposure compensation applied to a high dynamic range (HDR) video and a circuit. When an HDR video is produced, an exposure compensation process is performed according to an exposure value ratio that is detected in real time. In the exposure compensation process, an image sensor and an image signal processor do not rely on a queue to conduct parameter synchronization for avoiding brightness discontinuity and abnormal composition that occur in an intersection region between frames.
[0029] Continuous images are converted to an HDR video through a high dynamic range process. The high dynamic range process uses at least two frames, in which each of the two frames is an image with a height “m” and a width “n”, to render an HDR image. The pixel value (P) of each pixels of the image can be expressed by P.sub.i,j,k based on a weight, a height, and a number of exposed frames, in which “i” indicates a position at an x-direction of the image, “j” indicates a position at a y-direction of the image, and “k” indicates the number of the exposed frames. For implementing the HDR video, two frames including a long exposure frame and a short exposure frame are incorporated. With an image captured by a static camera as an example, in addition to a moving region of the image, the remaining region is static. Accordingly, the pixel values of a previous frame and a next frame have a relationship of P.sub.i,j,1=P.sub.i,j,2*R, in which “P.sub.i,j,1” indicates a pixel value of a first frame, “P.sub.i,j,2” indicates a pixel value of a second frame, and “R” indicates an exposure value ratio between the first frame and the second frame. Further, according to one embodiment of the present disclosure, the exposure compensation process is performed in rows. For a first row of the image (i=1 to n), the exposure value ratio R can be obtained by comparing the sums of pixel values of rows of the two exposed frames. The exposure value ratio R can be calculated by equation 1.
[0030] However, using the two frame exposure values to implement high dynamic range would cause two problems. One of which is that the content in a long exposure frame is different from the content in a short exposure frame due to the pixel pair in the previous frame and the next frame changes in a moving region. Therefore, the above-mentioned method does not apply to the video having the moving region. The other one of which is that an error may occur since the pixel values of the overly-dark or overly-bright region are too small and too close to an overexposure region. Accordingly, the method for exposure compensation applied to the HDR video is provided for eliminating the abovementioned problems.
[0031] A pair of pixel values formed by the previous and next frames can be expressed by “{P.sub.i,j,1, P.sub.i,j,2}.” A processor exposure value ratio R.sub.ISP is configured to be inputted to the image signal processor in advance. Further, in a beginning of the method, a brightness range threshold is set for excluding overly-dark and overly-bright pixels, and a light-dark ratio threshold is set for excluding one or more potential moving regions from the video. An exemplary example of selecting a pixel range being used to render the HDR video is as follows.
[0032] A first frame (P.sub.i,j,1) is configured to be a bright frame. An appropriate brightness range without overexposed region and/or overly-dark region is chosen according to a brightness distribution. For example, a brightness range [1024, 3072] is chosen from a 12-bit image.
[0033] A second frame (P.sub.i,j,2) is configured to be a dark frame. A reasonable range [P.sub.i,j,1/(R.sub.ISP+ε), P.sub.i,j,1/(R.sub.ISP−ε)] of the second frame (P.sub.i,j,2) can be obtained according to the pixel values of the first frame (P.sub.i,j,1), the exposure value ratio (R.sub.ISP) of the image signal processor, and a potential error ratio (ε) Next, in the previous frame and the next frame, only the pixel pair in the bright frame (i.e., first frame P.sub.i,j,1) and the dark frame (i.e., second frame P.sub.i,j,2) that are within the reasonable range are taken. The reasonable range indicates a range where the overly-bright and/or overly-dark pixels are excluded according to a brightness range threshold. Further, a light-dark ratio threshold can also be used for excluding one or more potential moving regions from the video.
[0034] Reference is made to
[0035] In the process, the pixel values (P.sub.i,j,k) of the sequential frames in the video are obtained (step S501). The processor exposure value ratio (R.sub.ISP) of the image signal processor is also obtained (step S503). Next, an instant exposure value ratio (R) of a current frame is calculated (step S505). The brightness range threshold is used to exclude the overly-dark and the overly-bright pixels in each of the frames (step S507). The light-dark ratio threshold is used to exclude the moving region in each of the frames (step S509).
[0036] According to one embodiment of the present disclosure, reference is made to
[0037] Further, in the step S505 of acquiring an instant exposure value ratio R, a calculation for acquiring the instant exposure value ratio for valid pixel pair in each of the frames is performed. Reference is made to
[0038] In the beginning, a predetermined value of an exposure value ratio can be a processor exposure value ratio (R.sub.ISP) of an image signal processor (step S601). Similarly, an instant exposure value ratio (R) of the current frame is obtained (step S603). In one embodiment of the present disclosure, in order to maintain continuity of the picture in the video, the exposure compensation can be performed in units of a row. According to a range having the valid pixel pair shown in
[0039] In an exemplary example, when the exposure value ratio for each of the frames is calculated in units of a row of the frames, a previous row exposure value ratio (R.sub.in) is firstly obtained (step S605). In addition to setting a first row exposure value ratio to be an initial processor exposure value ratio (R.sub.ISP) inputted to the image signal processor, an instant exposure value ratio for each of the rows equals to a previous row exposure value ratio plus an adjustment value for each of the rows, that can be expressed by an equation “R.sub.out=R.sub.in+R.sub.m”, in which “R.sub.out” indicates the instant exposure value ratio, “R.sub.in” indicates the previous row exposure value ratio, and “R.sub.m” is the adjustment value for each of the rows. Accordingly, the instant exposure value ratio R.sub.out can be obtained from the previous row exposure value ratio R.sub.in. Therefore, the method can determine the adjustment value (R.sub.m) in advance (step S607).
[0040] The adjustment value R.sub.m can be exemplarily determined by an equation 2.
R.sub.m=0, if −E≤R−R.sub.in≤E, in which “E” is a threshold for adjusting the ratio; Equation 2:
[0041] R.sub.m=R.sub.m, if R−R.sub.in>E;
[0042] R.sub.m=−R.sub.m, if R−R.sub.in<−E.
[0043] Therefore, the above steps can be used to calculate an instant exposure value ratio (R.sub.out) of a current row (step S609). The exposure value ratio for every row can be obtained by repeating the above steps (step S611).
[0044] In the process of determining the instant exposure value ratio R.sub.out, if a frame is initially configured to be that the processor exposure value ratio R.sub.ISP of the image signal processor is lower than a practical exposure value ratio R.sub.s of the image sensor, the exposure value ratio (R.sub.1) of a first row of the frame is R.sub.1=R.sub.ISP, and the exposure value ratio (R.sub.2) of a second row equals to the processor exposure value ratio (R.sub.ISP) of the image signal processor plus an adjustment value (R.sub.m), namely R.sub.2=R.sub.ISP+R.sub.m. The exposure value ratios of each of the rows in each frame can also be calculated. According to one embodiment of the present disclosure, a threshold for adjusting the exposure value ratio can be incorporated to converge the exposure value ratio of each of the frames, that is, to calculate the exposure value ratio of each of the rows until the exposure value ratio of a row j satisfies the relationship: R.sub.j=R.sub.ISP+j*R.sub.m>R−E, in which “R” is an instant exposure value ratio, “E” is the threshold for adjusting the exposure value ratio. Afterwards, the exposure value ratio enters a stable region that can be a ratio range covering the reasonable ratio 405 shown in
[0045] Afterwards, the multiple converged exposure value ratios of multiple rows are summed up for obtaining the stable exposure value ratio for each frame (step S613).
[0046] According to the embodiments relating to the abovementioned chosen valid pixel pair and the exposure value ratio between the frames, the method for exposure compensation applied to a high dynamic range video can be implemented, and a circuitry applying the method can be referred to in a circuit block diagram shown in
[0047] According to the embodiment of the present disclosure, the control circuit controls operations of the circuit components of the image-acquisition device. When the image-acquisition device is powered on, the lens is driven to capture continuous images sensed by the image sensor so as to generate image signals. The image signals are firstly stored in a memory. The image signal processor then processes the image signals. The circuitry embodies an HDR video exposure compensation circuit. The control circuit performs the method for exposure compensation applied to the HDR video upon the images.
[0048] According one further embodiment of the method for exposure compensation applied to a high dynamic range video, rather than the conventional method that uses a queue to synchronize the image sensor and the image signal processor, the image signal processor of the present disclosure conducts exposure compensation according to the instant exposure value ratio between a previous frame and a next frame. The relevant embodiment can be referred to in a flowchart shown in
[0049] In the method, an initial exposure value ratio is set as the processor exposure value ratio. A fixed adjustment ratio can be used to control both the image sensor and the image signal processor at the same time (step S901). The image data of continuous images can be retrieved from a video. The abovementioned embodiments are referred to so as to obtain a valid pixel pair with respect to each of the frames, and calculate an exposure value ratio in real time. Therefore, the valid pixel pair of each of the frames is acquired so as to calculate an exposure value ratio (step S903). In one of the embodiments of the present disclosure, a predetermined processor exposure value ratio (R.sub.ISP) of the image signal processor is adapted as the exposure value ratio for the zeroth frame (frame 0). Next, the exposure value ratio between the previous frame and the next frame is referred to for obtaining the fixed adjustment ratio (R.sub.adj) for each of the frames. The fixed adjustment ratio R.sub.adj can be used as a fixed ratio that is used to fine-tune the ratio of each of the frames. Therefore, every frame can be controlled to have the same error for acquiring the exposure value ratio for every frame. Since the continuity of the frames of the video is considered, an in-frame exposure value ratio can be fine-tuned in units of a row (step S905).
[0050] With using two frame exposure values to perform a high dynamic range process as an example, when the sequential frames are retrieved from the video a first frame exposure value ratio is used as the processor exposure value ratio (R.sub.ISP) that multiplies the fixed adjustment ratio (R.sub.adj), and a second frame exposure value ratio is used as the processor exposure that multiplies twice the fixed adjustment ratio; correspondingly, an initial exposure value ratio that corresponds to the first frame exposure value ratio is set in the image sensor. Next, an error is adjusted to be the same as the fixed adjustment ratio (R.sub.adj). Therefore, the exposure value ratio corresponding to the second frame exposure value ratio set in the image sensor is a result of the initial exposure value ratio multiplying the fixed adjustment ratio.
[0051] For example, starting with a predetermined processor exposure value ratio (R.sub.ISP), the fixed adjustment ratio R.sub.adj is used to adjust the exposure value ratio for each of the frames. The exposure value ratio of a first frame is R.sub.ISP*R.sub.adj. The image sensor has an initial exposure value ratio R.sub.s. An error can be adjusted to the fixed adjustment ratio R.sub.adj. The exposure value ratio of a second frame is R.sub.ISP*R.sub.adj*R.sub.adj. The exposure value ratio applied to the image sensor is R.sub.s*R.sub.adj, and the error is still a multiple of R.sub.adj. Afterwards, the error sustains until the convergence ends and then the error disappears since the number of error frames of the image signal processor and the image sensor are the same. If the ratio adjustment value for each frame is fixed, then the error for each frame can be fixed without using the processor queue, and the discontinuity in the intersection region during the convergence can be avoided.
[0052] Furthermore, a combination of the process of detecting the exposure value ratio and the process of high dynamic range compensation can compensate an error between the compensated picture and an expected exposure value ratio. In the embodiment of exposure compensation that the in-frame exposure value ratio is fine-tuned in units of a row, the speed of convergence and the rows to be affected are determined based on the adjustment value R.sub.m for each of the rows and the fixed adjustment ratio R.sub.adj.
[0053] Table 1 exemplarily shows outputs of the image sensor. The initial exposure value ratio (i.e., the processor exposure value ratio (R.sub.ISP)) is 4 times. The adjustment value (R.sub.m) is 3.125% (1/32). The fixed adjustment ratio (R.sub.adj) controls the frame requiring 4 rows to be converged progressively if the exposure value ratio of each of the frames raises 12.5% (1/8). According to Table 1, the exposure value ratio error (E.sub.i) for each row in each frame changes until E.sub.5 through the convergence process.
TABLE-US-00001 TABLE 1 frame R.sub.ISP R.sub.m R.sub.s E.sub.1 E.sub.2 E.sub.3 E.sub.4 E.sub.5 . . . 0 4.0 0.125 4.0 1.0 1.0 1.0 1.0 1.0 1 4.5 0.141 4.0 1.125 1.090 1.054 1.019 1.0 2 5.0625 0.158 4.5 1.125 1.090 1.054 1.019 1.0 3 5.695 0.178 5.0625 1.125 1.090 1.054 1.019 1.0 . . .
[0054] Table 1 shows a whole process of convergence of the exposure value ratio, in which the exposure value ratio error (E.sub.i) for each of the rows is fixed, and the brightness of the pixels in each of the rows can be ensured to change stably during the convergence process when the given fixed adjustment ratio R.sub.adj is small enough and the adjustment value R.sub.m for each row is large enough. Therefore, the number of rows to be affected can be limited to a certain number.
[0055] Further, in one further embodiment of the present disclosure, reference is made to
[0056] Thus, when the image outputted by the image sensor has a height m+k, the rows with erroneous exposure value ratio (i.e., the removable region 801) are removed so as to form the output image with the height m after the convergence process. Therefore, the rows with the erroneous exposure value ratio, which are configured to be discarded, can be excluded from the process of convergence (step S907). In one further embodiment, the removable region 801 may not need to be removed but can be made less visible to the human eyes via other manners. Finally, the image being compensated with correct exposure value ratio can be obtained (step S909). After that, a series of sequential frames undergone the high dynamic range compensation are obtained, and an HDR video is outputted via an output circuit of the image-acquisition device (step S911).
[0057] It should be noted that, in the method for exposure compensation applied to a high dynamic range video, a stability of changes in the whole convergence process can be guaranteed when the fixed adjustment ratio R.sub.adj is controlled to be small enough so that the problem of continuity and abnormal composition can be prevented.
[0058] In summation, according to the abovementioned embodiments of the method and the circuitry for exposure compensation applied to a high dynamic range video, the exposure compensation method uses an exposure value ratio of a pixel pair in the frames of a video to perform compensation. Therefore, the image sensor and the image signal processor can take effect synchronously. The method does not use the queue that may cause discontinuity to the intersection region of the images in the conventional technology. In one embodiment (but not limited thereto), the exposure value ratio of the image sensor can be updated in units of a row in real time. The exposure compensation can be performed with an instant detected exposure value ratio for guaranteeing the continuity of the compensated exposure between the rows or the frames. Thus, rather than the queue being used for processing the HDR video, the method uses the proposed exposure value ratio to continuously process the HDR video for ensuring the stable convergence process regardless of the difference in the number of the effective frames between the image sensor and the image signal processor. The problems of abnormal composition and flickering phenomenon on the screen can be prevented when implementing the HDR video.
[0059] The foregoing description of the exemplary embodiments of the disclosure has been presented only for the purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching.
[0060] The embodiments were chosen and described in order to explain the principles of the disclosure and their practical application so as to enable others skilled in the art to utilize the disclosure and various embodiments and with various modifications as are suited to the particular use contemplated. Alternative embodiments will become apparent to those skilled in the art to which the present disclosure pertains without departing from its spirit and scope.